Skip to main content
Version: 3.5.0

Action: generate

Create new events, specifically for alerts

This step requires special attention, because it is a way to create custom events and alerts that are aware of the history of the data.

As a stream of JSON events is read and passed through. generate saves these records in a SQLite database so it can use the full power of SQL to generate historical queries over aggregates such as averages and maximums.

It takes free-form fields, but the values are documented in the fields array below.

See the discussion

Example: A contrived example

Input:

input:
exec:
command: |
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
raw: true
no-strip-linefeeds: true

Pipe Language Snippet:

generate:
high average:
let:
avg: AVG(number, 3s)
when: avg >= 2
add:
- title: average too high
- text: average threshold 2 exceeded over time range=3s interval is higher than threshold

Output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-06T11:21:04.653Z"}
{"number":1}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:29:01.365Z"}

Example: another example

Input:

input:
exec:
command: |
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
raw: true
no-strip-linefeeds: true

Pipe Language Snippet:

generate:
high average:
let:
avg: AVG(number, 3s)
when: avg >= 2
add:
- title: average too high
- text: average threshold=2 exceeded (over time range=3s)

Output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:23:56.271Z"}
{"number":1}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:30:46.524Z"}
{"number":2}

Example: A more real example

Pipe Language Snippet:

generate:
bbox.linkutilisation.incoming:
let:
avg_incoming: AVG(incomingBytesPerInterval, 5m)
max_incoming: MAX(incomingBytesPerInterval, 60m)
threshold: 90.0/100.0
when: (avg_incoming / max_incoming) > threshold
add:
- severity: warning
- kind: alert
- title: "Line Utilisation incoming over 90%"
- text: "average incoming ${avg_incoming}kb close to max incoming ${max_incoming}kb: ratio: ${threshold:1}"

Field Summary

Field NameTypeDescriptionDefault
mapBTreeMapA map of events to generate when conditions are met-
alertAlertA single event to generate when conditions are met-

Fields

map

Type: BTreeMap

A map of events to generate when conditions are met

alert

Type: Alert

A single event to generate when conditions are met

Field NameTypeDescriptionDefault
addarray of (field, value) pairsDescribes the new fields to be added, whenever the event is generated-
whenstringDetermines if an event should be generated-
group-bycomma separated field listComma separated list of fields to group the aggregations by-
let(string, string) mapA section to generate calculations-
at-endTimeoutAdd the generated event at the end of a series-
notificationdurationHow often the event should be generated, to help limit event count-

  add

Type: array of (field, value) pairs

Describes the new fields to be added, whenever the event is generated

  when

Type: string

Determines if an event should be generated

  group-by

Type: comma separated field list

Comma separated list of fields to group the aggregations by

  let

Type: (string, string) map

A section to generate calculations

  at-end

Type: Timeout

Add the generated event at the end of a series

Field NameTypeDescriptionDefault
timeoutdurationEmit the event after a timeout of duration-
markerarray of stringsEmit the event after the specified marker-

  timeout

Type: duration

Emit the event after a timeout of duration

Example

Input:

input:
exec:
command: |
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":3}';
raw: true
no-strip-linefeeds: true

Pipe Language Snippet:

generate:
high.average:
let:
avg: AVG(number, 3s)
when: avg > 2
add:
- title: average too high
- text: average threshold 2 exceeded over time rage=3s interval is higher than threshold
at_end: 2s

Output:

{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":3}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time rage=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:24:16.648Z"}

  marker

Type: array of strings

Emit the event after the specified marker

  notification

Type: duration

How often the event should be generated, to help limit event count

Example: A contrived example

Input:

input:
exec:
command: |
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":3}';
raw: true
no-strip-linefeeds: true

Pipe Language Snippet:

generate:
high.average:
let:
avg: AVG(number, 3s)
when: avg > 2
add:
- title: average too high
- text: average threshold 2 exceeded over time range=3s interval is higher than threshold
notification: 3s

Output:

{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:55:35.893Z"}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":5,"@timestamp":"2022-05-05T14:55:39.886Z"}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":8,"@timestamp":"2022-05-05T14:55:42.888Z"}
{"number":3}
{"number":3}