Version: 3.4.0

Testing Pipes

It's useful to test pipes - or sections of them - before deploying them to actual hardware.

Consider this basic pipe which has a context expansion:

# test.yml
name: test
context:
  msg: hello
input:
  exec:
    command: 'echo {{msg}}'
output:
  write: console

We can run it using the Hotrod CLI:

hotrod pipes run --file test.yml
{"_raw":"hello"}

To experiment with overriding the context, define a little file containing some context:

# context.yml
context:
  msg: goodbye

Then, use below to override the pipe context:

hotrod pipes run --file test.yml --context-files context.yml
{"_raw":"goodbye"}

Experimenting with pipes in this way builds confidence. For more complicated pipes, it is always easier to troubleshoot problems on a local machine.

The following pipe consumes a file times.txt containing times in Unix epoch format (total seconds since 1970). The pipe must detect any gaps in this time record.

The time comes in as a string with the default field _raw, and we rename and convert it to a number.

name: stream
input:
  exec:
    command: cat times.txt
actions:
- rename:
  - _raw: epoch
- convert:
  - epoch: num
output:
  write: console

Running:

hotrod pipes run -f stream.yml
{"epoch":1592928057}
{"epoch":1592928058}
{"epoch":1592928059}
{"epoch":1592928060}
{"epoch":1592928061}
{"epoch":1592928062}
{"epoch":1592928063}
{"epoch":1592928064}
{"epoch":1592928065}
{"epoch":1592928066}

To find the difference between times, use the stream action just after convert

- stream:
    operation: delta
    watch: epoch

We now get:

{"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928058,"delta":1,"elapsed":0}
{"epoch":1592928059,"delta":1,"elapsed":0}
{"epoch":1592928060,"delta":1,"elapsed":0}
{"epoch":1592928061,"delta":1,"elapsed":0}
{"epoch":1592928062,"delta":1,"elapsed":0}
{"epoch":1592928063,"delta":1,"elapsed":0}
{"epoch":1592928064,"delta":1,"elapsed":0}
{"epoch":1592928065,"delta":1,"elapsed":0}
{"epoch":1592928066,"delta":1,"elapsed":0}

In this case, to detect gaps, only pass through events where delta is not equal to 1, and it is also not the first event:

    - filter:
        condition: delta != 1 and count() > 1

The pipe will output nothing, but we can now provoke output by putting gaps into time.yml. This would be an example of a pipe monitoring data and detecting anomalies.

Therefore the suggested way to build non-trivial pipes is step-by-step in this fashion.

It's possible to override the inputs of a pipe. This is useful for testing pipes which get their input from another source like a live HTTP request, TCP etc. For instance, the version of stream.yml that ends with stream can be fed a single timestamp:

hotrod pipes run -f stream.yml --input 1592928057
{"epoch":1592928057,"delta":1592928057,"elapsed":0}

Using --input @times.txt gives us the same result as before - if the argument starts with '@', it is considered a file which contains the data (this is same behaviour as curl).

Use --output <filename> to save the output to a file.

Tracing a pipe can show you how each step transforms the data. Here are our steps:

name: stream
input:
  exec: # step 0
    command: cat times.txt
actions:
- rename:  # step 1
  - _raw: epoch
- convert: # step 2
  - epoch: num
- stream: # step 3
    operation: delta
    watch: epoch
output:
  write: console  # step 4    

Run the last example with tracing enabled:

hotrod pipes run -f stream.yml --input 1592928057 --trace
[TRACE]  action-rename step 1
LINE: {"_raw":"1592928057"} -> [{"epoch":"1592928057"}]
[TRACE]  action-convert step 2
LINE: {"epoch":"1592928057"} -> [{"epoch":1592928057}]
[TRACE]  action-stream step 3
LINE: {"epoch":1592928057} -> [{"epoch":1592928057,"delta":1592928057,"elapsed":0}]
[TRACE]  output-exec step 4
LINE: {"epoch":1592928057,"delta":1592928057,"elapsed":0}
{"epoch":1592928057,"delta":1592928057,"elapsed":0}