Action: generate

Create new events, specifically for alerts

This step requires special attention, because it is a way to create custom events and alerts that are aware of the history of the data.

As a stream of JSON events is read and passed through. generate saves these records in a SQLite database so it can use the full power of SQL to generate historical queries over aggregates such as averages and maximums.

It takes free-form fields, but the values are documented in the fields array below.

See the discussion

Example: A contrived example

Input:

input:
  exec:
    command: |
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
    raw: true
    no-strip-linefeeds: true

Pipe Language Snippet:

generate:
  high average:
    let:
       avg: AVG(number, 3s)
    when: avg >= 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time range=3s interval is higher than threshold

Output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-06T11:21:04.653Z"}
{"number":1}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:29:01.365Z"}

Example: another example

Input:

input:
  exec:
    command: |
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":1}';
      sleep 1;
      echo '{"number":2}';
    raw: true
    no-strip-linefeeds: true

Pipe Language Snippet:

generate:
  high average:
    let:
       avg: AVG(number, 3s)
    when: avg >= 2
    add:
      - title: average too high
      - text: average threshold=2 exceeded (over time range=3s)

Output:

{"number":1}
{"number":2}
{"number":2}
{"number":3}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:23:56.271Z"}
{"number":1}
{"type":"alert","text":"average threshold=2 exceeded (over time range=3s)","title":"average too high","aggregation_key":"high average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:30:46.524Z"}
{"number":2}

Example: A more real example

Pipe Language Snippet:

generate:
  bbox.linkutilisation.incoming:
    let:
       avg_incoming: AVG(incomingBytesPerInterval, 5m)
       max_incoming: MAX(incomingBytesPerInterval, 60m)
       threshold: 90.0/100.0
    when: (avg_incoming / max_incoming) > threshold
    add:
      - severity: warning
      - kind: alert
      - title: "Line Utilisation incoming over 90%"
      - text: "average incoming ${avg_incoming}kb close to max incoming ${max_incoming}kb: ratio: ${threshold:1}"

Field Summary

Field Name	Type	Description	Default
map	BTreeMap	A map of events to generate when conditions are met	-
alert	Alert	A single event to generate when conditions are met	-

Fields

map

Type: BTreeMap

A map of events to generate when conditions are met

alert

Type: Alert

A single event to generate when conditions are met

Field Name	Type	Description	Default
add	array of (field, value) pairs	Describes the new fields to be added, whenever the event is generated	-
when	string	Determines if an event should be generated	-
group-by	comma separated field list	Comma separated list of fields to group the aggregations by	-
let	(string, string) map	A section to generate calculations	-
at-end	Timeout	Add the generated event at the end of a series	-
notification	duration	How often the event should be generated, to help limit event count	-

  add

Type: array of (field, value) pairs

Describes the new fields to be added, whenever the event is generated

  when

Type: string

Determines if an event should be generated

  group-by

Type: comma separated field list

Comma separated list of fields to group the aggregations by

  let

Type: (string, string) map

A section to generate calculations

  at-end

Type: Timeout

Add the generated event at the end of a series

Field Name	Type	Description	Default
timeout	duration	Emit the event after a timeout of duration	-
marker	array of strings	Emit the event after the specified marker	-

  timeout

Type: duration

Emit the event after a timeout of duration

Example

Input:

input:
  exec:
    command: |
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":3}';
    raw: true
    no-strip-linefeeds: true

Pipe Language Snippet:

generate:
  high.average:
    let:
       avg: AVG(number, 3s)
    when: avg > 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time rage=3s interval is higher than threshold
    at_end: 2s

Output:

{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"number":3}
{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time rage=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2020-03-12T10:24:16.648Z"}

  marker

Type: array of strings

Emit the event after the specified marker

  notification

Type: duration

How often the event should be generated, to help limit event count

Example: A contrived example

Input:

input:
  exec:
    command: |
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":2}';
      sleep 1;
      echo '{"number":3}';
      sleep 1;
      echo '{"number":3}';
    raw: true
    no-strip-linefeeds: true

Pipe Language Snippet:

generate:
  high.average:
    let:
       avg: AVG(number, 3s)
    when: avg > 2
    add:
      - title: average too high
      - text: average threshold 2 exceeded over time range=3s interval is higher than threshold
    notification: 3s

Output:

{"number":3}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":1,"@timestamp":"2022-05-05T14:55:35.893Z"}
{"number":2}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":5,"@timestamp":"2022-05-05T14:55:39.886Z"}
{"number":2}
{"number":3}
{"number":2}
{"type":"alert","text":"average threshold 2 exceeded over time range=3s interval is higher than threshold","title":"average too high","aggregation_key":"high.average","severity":"info","alert_count":8,"@timestamp":"2022-05-05T14:55:42.888Z"}
{"number":3}
{"number":3}