Action: stream

Create a new field calculated on historical data

Currently, the only operation supported is delta, which gives the difference between the current value and the last value, together with the elapsed time in milliseconds.

If marker is specified, a new event will be created, otherwise the existing event will be enriched

Example

Input:

{"time":"2018-07-09T15:09:06.479Z","data":1}
{"time":"2018-07-09T15:09:07.479Z","data":2}
{"time":"2018-07-09T15:09:08.179Z","data":4}

Pipe Language Snippet:

stream:
  input-time: time
  watch: data
  marker:
    - delta
  operation: delta

Output:

{"_marker":"delta","delta":1,"elapsed":0}
{"time":"2018-07-09T15:09:06.479Z","data":1}
{"_marker":"delta","delta":1,"elapsed":1000}
{"time":"2018-07-09T15:09:07.479Z","data":2}
{"_marker":"delta","delta":2,"elapsed":700}
{"time":"2018-07-09T15:09:08.179Z","data":4}

Field Summary

Field Name	Type	Description	Default
condition	expression	Only run this action if the condition the specified condition is met	-
input-time	iso time field	Use incoming time, instead of current system time	-
watch-field	field	A field to watch for changes	-
operation	string	A stream operation (delta is the only supported value)	-
only-changes	bool	Let elapsed field contain the time in milliseconds since the actual change occurred	false
output-field	field	Use this field to store the difference	delta
elapsed-field	field	Field name to store elapsed time	elapsed
group-by	field	Track differences between field values belonging to different groups	-
marker	(string,string) pair	The marker to use for the newly generated event	-

Fields

condition

Type: expression

Only run this action if the condition the specified condition is met

input-time

Type: iso time field

Use incoming time, instead of current system time

watch-field

Type: field
Alias: watch-field

A field to watch for changes

operation

Type: string

Possible Values: delta

A stream operation (delta is the only supported value)

only-changes

Type: bool

Default: false

Let elapsed field contain the time in milliseconds since the actual change occurred

output-field

Type: field

Default: delta

Use this field to store the difference

elapsed-field

Type: field

Default: elapsed

Field name to store elapsed time

group-by

Type: field

Track differences between field values belonging to different groups

Example: Here the change in state is tracked for each value of name

Input:

{"time":"2020-02-03T15:34:45.149Z","name":"john","state":0}
{"time":"2020-02-03T15:34:46.149Z","name":"jane","state":0}
{"time":"2020-02-03T15:34:47.149Z","name":"john","state":1}
{"time":"2020-02-03T15:34:48.149Z","name":"jane","state":1}
{"time":"2020-02-03T15:34:49.149Z","name":"john","state":1}
{"time":"2020-02-03T15:34:50.149Z","name":"jane","state":1}

Pipe Language Snippet:

stream:
  input-time: time
  watch: state
  group-by: name
  operation: delta

Output:

{"time":"2020-02-03T15:34:45.149Z","name":"john","state":0,"delta":0,"elapsed":0}
{"time":"2020-02-03T15:34:46.149Z","name":"jane","state":0,"delta":0,"elapsed":0}
{"time":"2020-02-03T15:34:47.149Z","name":"john","state":1,"delta":1,"elapsed":2000}
{"time":"2020-02-03T15:34:48.149Z","name":"jane","state":1,"delta":1,"elapsed":2000}
{"time":"2020-02-03T15:34:49.149Z","name":"john","state":1,"delta":0,"elapsed":2000}
{"time":"2020-02-03T15:34:50.149Z","name":"jane","state":1,"delta":0,"elapsed":2000}

marker

Type: (string,string) pair

The marker to use for the newly generated event

Example: With a marker a new event is created

Input:

{"time":"2018-07-09T15:09:06.479Z","data":1}
{"time":"2018-07-09T15:09:07.479Z","data":2}
{"time":"2018-07-09T15:09:08.179Z","data":4}

Pipe Language Snippet:

stream:
  input-time: time
  watch: data
  operation: delta
  marker: [DELTA]

Output:

{"_marker":"DELTA","delta":1,"elapsed":0}
{"time":"2018-07-09T15:09:06.479Z","data":1}
{"_marker":"DELTA","delta":1,"elapsed":1000}
{"time":"2018-07-09T15:09:07.479Z","data":2}
{"_marker":"DELTA","delta":2,"elapsed":700}
{"time":"2018-07-09T15:09:08.179Z","data":4}

Example: Without marker the existing event is enriched

Input:

{"time":"2018-07-09T15:09:06.479Z","data":1}
{"time":"2018-07-09T15:09:07.479Z","data":2}
{"time":"2018-07-09T15:09:08.179Z","data":4}

Pipe Language Snippet:

stream:
  input-time: time
  watch: data
  operation: delta

Output:

{"time":"2018-07-09T15:09:06.479Z","data":1,"delta":1,"elapsed":0}
{"time":"2018-07-09T15:09:07.479Z","data":2,"delta":1,"elapsed":1000}
{"time":"2018-07-09T15:09:08.179Z","data":4,"delta":2,"elapsed":700}