Skip to main content
Version: 3.4.0

generate

Create new events, specifically for alerts

This step requires special attention, because it is a way to create custom events and alerts that are aware of the history of the data.

As a stream of JSON events is read and passed through. generate saves these records in a SQLite database so it can use the full power of SQL to generate historical queries over aggregates such as averages and maximums.

It takes free-form fields, but the values are documented in the fields array below.

See the discussion

Note: This DSL needs review, for it is different from the other Actions

Example: A contrived example

  • We take the 3s average (avg) over event field, number
  • We create a new event (alert) when threshold is greater than 2
  • add indicates what fields will be added to the new event, should the when be met

input:

input:
exec:
command: |
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
raw: true
no-strip-linefeeds: true

action:

generate:
high average:
let:
avg: AVG(number, 3s)
when: avg >= 2
add:
- title: average too high
- text: average threshold 2 exceeded over time range=3s interval is higher than threshold

output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-06T11:21:04.653Z"}
{"number":1}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:29:01.365Z"}

Example: another example

input:

input:
exec:
command: |
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":1}';
sleep 1;
echo '{"number":2}';
raw: true
no-strip-linefeeds: true

action:

generate:
high average:
let:
avg: AVG(number, 3s)
when: avg >= 2
add:
- title: average too high
- text: average threshold=2 exceeded (over time range=3s)

output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:23:56.271Z"}
{"number":1}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:30:46.524Z"}
{"number":2}

Example: A more real example

  • We take the 5m average over incomingBytesPerInterval (an event field)
  • We take the 60m max value over the same field,
  • We create a new event (alert) when the ratio of the former (avg_incoming) over the latter (max_incoming) is greater than threshold.
  • add indicates what fields will be added to the new event, should the when be met

action:

generate:
bbox.linkutilisation.incoming:
let:
avg_incoming: AVG(incomingBytesPerInterval, 5m)
max_incoming: MAX(incomingBytesPerInterval, 60m)
threshold: 90.0/100.0
when: (avg_incoming / max_incoming) > threshold
add:
- severity: warning
- kind: alert
- title: "Line Utilisation incoming over 90%"
- text: "average incoming ${avg_incoming}kb close to max incoming ${max_incoming}kb: ratio: ${threshold:1}"

Field NameDescriptionTypeDefault
mapA map of events to generate when conditions are metBTreeMap-
alertA single event to generate when conditions are metAlert-

map

A map of events to generate when conditions are met

Type: BTreeMap

alert

A single event to generate when conditions are met

Type: Alert

Field NameDescriptionTypeDefault
addDescribes the new fields to be added, whenever the event is generatedarray of (field, value) pairs-
whenDetermines if an event should be generatedstring-
group-byComma separated list of fields to group the aggregations bycomma separated field list-
letA section to generate calculations(string, string) map-
at-endAdd the generated event at the end of a seriesTimeout-
notificationHow often the event should be generated, to help limit event countduration-

add

Describes the new fields to be added, whenever the event is generated

Type: array of (field, value) pairs

when

Determines if an event should be generated

Type: string

group-by

Comma separated list of fields to group the aggregations by

Type: comma separated field list

let

A section to generate calculations

Type: (string, string) map

at-end

Add the generated event at the end of a series

Type: Timeout

Field NameDescriptionTypeDefault
timeoutEmit the event after a timeout of durationduration-
markerEmit the event after the specified markerarray of strings-

timeout

Emit the event after a timeout of duration

Type: duration

Example

input:

input:
exec:
command: |
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":3}';
raw: true
no-strip-linefeeds: true

action:

generate:
high.average:
let:
avg: AVG(number, 3s)
when: avg > 2
add:
- title: average too high
- text: average threshold 2 exceeded over time rage=3s interval is higher than threshold
at_end: 2s

output:

{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":3}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time rage=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:24:16.648Z"}

marker

Emit the event after the specified marker

Type: array of strings

notification

How often the event should be generated, to help limit event count

Type: duration

Example: A contrived example

  • We take the 3s average (avg) over event field, number
  • We create a new event (alert) when threshold is greater than 2
  • add indicates what fields will be added to the new event, should the when be met
  • We create the event only when notification duration has been exceeded

input:

input:
exec:
command: |
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":2}';
sleep 1;
echo '{"number":3}';
sleep 1;
echo '{"number":3}';
raw: true
no-strip-linefeeds: true

action:

generate:
high.average:
let:
avg: AVG(number, 3s)
when: avg > 2
add:
- title: average too high
- text: average threshold 2 exceeded over time range=3s interval is higher than threshold
notification: 3s

output:

{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:55:35.893Z"}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":5,"@timestamp":"2022-05-05T14:55:39.886Z"}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":8,"@timestamp":"2022-05-05T14:55:42.888Z"}
{"number":3}
{"number":3}