Version: 3.4.0

generate

Create new events, specifically for alerts

This step requires special attention, because it is a way to create custom events and alerts that are aware of the history of the data.

As a stream of JSON events is read and passed through. generate saves these records in a SQLite database so it can use the full power of SQL to generate historical queries over aggregates such as averages and maximums.

It takes free-form fields, but the values are documented in the fields array below.

See the discussion

Note: This DSL needs review, for it is different from the other Actions

Example: A contrived example

We take the 3s average (avg) over event field, number
We create a new event (alert) when threshold is greater than 2
add indicates what fields will be added to the new event, should the when be met

input:

input:
  exec:
    command: |
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
    raw: true
    no-strip-linefeeds: true

action:

generate:
  high average:
    let:
       avg: AVG(number, 3s)
    when: avg >= 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time range=3s interval is higher than threshold

output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-06T11:21:04.653Z"}
{"number":1}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:29:01.365Z"}

Example: another example

input:

input:
  exec:
    command: |
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
    raw: true
    no-strip-linefeeds: true

action:

generate:
  high average:
    let:
       avg: AVG(number, 3s)
    when: avg >= 2
    add:
      - title: average too high
      - text: average threshold=2 exceeded (over time range=3s)

output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:23:56.271Z"}
{"number":1}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:30:46.524Z"}
{"number":2}

Example: A more real example

We take the 5m average over incomingBytesPerInterval (an event field)
We take the 60m max value over the same field,
We create a new event (alert) when the ratio of the former (avg_incoming) over the latter (max_incoming) is greater than threshold.
add indicates what fields will be added to the new event, should the when be met

action:

generate:
  bbox.linkutilisation.incoming:
    let:
       avg_incoming: AVG(incomingBytesPerInterval, 5m)
       max_incoming: MAX(incomingBytesPerInterval, 60m)
       threshold: 90.0/100.0
    when: (avg_incoming / max_incoming) > threshold
    add:
      - severity: warning
      - kind: alert
      - title: "Line Utilisation incoming over 90%"
      - text: "average incoming ${avg_incoming}kb close to max incoming ${max_incoming}kb: ratio: ${threshold:1}"

Field Name	Description	Type	Default
map	A map of events to generate when conditions are met	BTreeMap	-
alert	A single event to generate when conditions are met	Alert	-

map

A map of events to generate when conditions are met

Type: BTreeMap

alert

A single event to generate when conditions are met

Type: Alert

Field Name	Description	Type	Default
add	Describes the new fields to be added, whenever the event is generated	array of (field, value) pairs	-
when	Determines if an event should be generated	string	-
group-by	Comma separated list of fields to group the aggregations by	comma separated field list	-
let	A section to generate calculations	(string, string) map	-
at-end	Add the generated event at the end of a series	Timeout	-
notification	How often the event should be generated, to help limit event count	duration	-

add

Describes the new fields to be added, whenever the event is generated

Type: array of (field, value) pairs

when

Determines if an event should be generated

Type: string

group-by

Comma separated list of fields to group the aggregations by

Type: comma separated field list

let

A section to generate calculations

Type: (string, string) map

at-end

Add the generated event at the end of a series

Type: Timeout

Field Name	Description	Type	Default
timeout	Emit the event after a timeout of `duration`	duration	-
marker	Emit the event after the specified marker	array of strings	-

timeout

Emit the event after a timeout of duration

Type: duration

Example

input:

input:
  exec:
    command: |
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":3}';
    raw: true
    no-strip-linefeeds: true

action:

generate:
  high.average:
    let:
       avg: AVG(number, 3s)
    when: avg > 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time rage=3s interval is higher than threshold
    at_end: 2s

output:

{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":3}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time rage=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:24:16.648Z"}

marker

Emit the event after the specified marker

Type: array of strings

notification

How often the event should be generated, to help limit event count

Type: duration

Example: A contrived example

We take the 3s average (avg) over event field, number
We create a new event (alert) when threshold is greater than 2
add indicates what fields will be added to the new event, should the when be met
We create the event only when notification duration has been exceeded

input:

input:
  exec:
    command: |
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":3}';
    raw: true
    no-strip-linefeeds: true

action:

generate:
  high.average:
    let:
       avg: AVG(number, 3s)
    when: avg > 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time range=3s interval is higher than threshold
    notification: 3s

output:

{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:55:35.893Z"}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":5,"@timestamp":"2022-05-05T14:55:39.886Z"}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":8,"@timestamp":"2022-05-05T14:55:42.888Z"}
{"number":3}
{"number":3}

generate

map​

alert​

add​

when​

group-by​

let​

at-end​

timeout​

marker​

notification​

map

alert

add

when

group-by

let

at-end

timeout

marker

notification