Skip to main content
Version: 3.3.0

Testing Pipes

It's useful to test pipes - or sections of them - before deploying them to actual hardware.

Consider this basic pipe which has a context expansion:

# test.yml
name: test
context:
msg: hello
input:
exec:
command: 'echo {{msg}}'
output:
write: console

We can run it using the Hotrod CLI:

hotrod pipes run --file test.yml
{"_raw":"hello"}

To experiment with overriding the context, define a little file containing some context:

# context.yml
context:
msg: goodbye

Then, use below to override the pipe context:

hotrod pipes run --file test.yml --context-files context.yml
{"_raw":"goodbye"}

Experimenting with pipes in this way builds confidence. For more complicated pipes, it is always easier to troubleshoot problems on a local machine.

The following pipe consumes a file times.txt containing times in Unix epoch format (total seconds since 1970). The pipe must detect any gaps in this time record.

1592928057
1592928058
1592928059
1592928060
1592928061
1592928062
1592928063
1592928064
1592928065
1592928066

The time comes in as a string with the default field _raw, and we rename and convert it to a number.

name: stream
input:
exec:
command: cat times.txt
actions:
- rename:
- _raw: epoch
- convert:
- epoch: num
output:
write: console

Running:

hotrod pipes run -f stream.yml
{"epoch":1592928057}
{"epoch":1592928058}
{"epoch":1592928059}
{"epoch":1592928060}
{"epoch":1592928061}
{"epoch":1592928062}
{"epoch":1592928063}
{"epoch":1592928064}
{"epoch":1592928065}
{"epoch":1592928066}

To find the difference between times, use the stream action just after convert

- stream:
operation: delta
watch: epoch

We now get:

{"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928058,"delta":1,"elapsed":0}
{"epoch":1592928059,"delta":1,"elapsed":0}
{"epoch":1592928060,"delta":1,"elapsed":0}
{"epoch":1592928061,"delta":1,"elapsed":0}
{"epoch":1592928062,"delta":1,"elapsed":0}
{"epoch":1592928063,"delta":1,"elapsed":0}
{"epoch":1592928064,"delta":1,"elapsed":0}
{"epoch":1592928065,"delta":1,"elapsed":0}
{"epoch":1592928066,"delta":1,"elapsed":0}

In this case, to detect gaps, only pass through events where delta is not equal to 1, and it is also not the first event:

    - filter:
condition: delta != 1 and count() > 1

The pipe will output nothing, but we can now provoke output by putting gaps into time.yml. This would be an example of a pipe monitoring data and detecting anomalies.

Therefore the suggested way to build non-trivial pipes is step-by-step in this fashion.

It's possible to override the inputs of a pipe. This is useful for testing pipes which get their input from another source like a live HTTP request, TCP etc. For instance, the version of stream.yml that ends with stream can be fed a single timestamp:

hotrod pipes run -f stream.yml --input 1592928057
{"epoch":1592928057,"delta":1592928057,"elapsed":0}

Using --input @times.txt gives us the same result as before - if the argument starts with '@', it is considered a file which contains the data (this is same behaviour as curl).

Use --output <filename> to save the output to a file.

Tracing a pipe can show you how each step transforms the data. Here are our steps:

name: stream
input:
exec: # step 0
command: cat times.txt
actions:
- rename: # step 1
- _raw: epoch
- convert: # step 2
- epoch: num
- stream: # step 3
operation: delta
watch: epoch
output:
write: console # step 4

Run the last example with tracing enabled:

hotrod pipes run -f stream.yml --input 1592928057 --trace
[TRACE] action-rename step 1
LINE: {"_raw":"1592928057"} -> [{"epoch":"1592928057"}]
[TRACE] action-convert step 2
LINE: {"epoch":"1592928057"} -> [{"epoch":1592928057}]
[TRACE] action-stream step 3
LINE: {"epoch":1592928057} -> [{"epoch":1592928057,"delta":1592928057,"elapsed":0}]
[TRACE] output-exec step 4
LINE: {"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928057,"delta":1592928057,"elapsed":0}