Expansion

There are two kinds of expansion in pipes. The first happens at processing-time involving the context and the second happens at run-time where we extract values from data fields.

Context Expansion

Any pipe has associated parameters, like how often to sample, what network interface to probe, etc. It is a good idea to give these values names, and define them in the context section of a pipe; as described in contexts these parameters can then be overridden.

Double-braces are used to enclose context variables ({{ name }}).

The pipe context predefines some default variables:

• name the name of the agent
• agent the agent id
• pipe the name of the pipe

Context variables can be used to simplify regular expressions

name: temp
context:
  INT: '(\d+)'
input:
  exec:
    command: echo "today date is 2019/07/22"
actions:
- extract:
    remove: true
    pattern: '{{INT}}/{{INT}}/{{INT}}'
    output-fields:
    - year
    - month
    - day
- convert:
    auto: true
output:
    write: console

Context variable definitions may themselves include context variables:

context:
    PORT: 3030
    ADDR: '0.0.0.0:{{PORT}}'

So {{ADDR}} will expand to 0.0.0.0:3030.

It is useful to give names to arbitrary values, but the real power of context variables is that they can be overridden, per-agent, per-pipe, per-tag, etc.

name: ping
context:
    interval: 5s
    address: google.com
input:
    exec:
        command: ping {{address}}
        interval: '{{interval}}'
output:
    write: console

This pipe can be pointed at different sites and made to poll at different rates by controlling the pipe's context.

NOTE changing a pipe's context requires it to be restarted.

'nested' variables can be rendered.

name: temp
context:
    object:
        person:
            name: frodo
            age: 50
input:
    exec:
        command: echo 'hello {{object.person.name}} your age is {{object.person.age}}'
output:
    write: console

(Although object.person.age is a number, a variable expansion always results in a text value)

Likewise, it can access the elements of an array:

name: temp
context:
    array:
    - one
    - two
    - 3
input:
    exec:
        command: echo 'we have {{array.0}}, {{array.1}} and {{array.2}} {{array}}'
        json: true
output:
    write: console

{{array}} by default renders as '"one","two",3' - i.e strings are quoted.

The variable can be followed by an explicit delimiter, so {{array ' '}} will render as '"one" "two" 3'.

A wildcard expression allows a particular field to be extracted from an array of objects

name: templ
context:
    hosts:
    - port: 10
      name: foo
    - port: 20
      name: boo
    - port: 30
      name: bar
input:
    text: host port {{hosts.*.port ' and host port '}}
output:
    write: console
# host port 10 and host port 20 and host port 30

Arrays may be merged:

name: temp
context:
  birds: [swallow,thrush]
  animals: [cat,dog]
input:
  exec:
    command: echo {{merge(birds,animals)}}
output:
    write: console
#  {"_raw":"swallow,thrush,cat,dog"}

Template Expansion

An alternative context expansion method that functions like a full templating language is also provided. These are defined in a special top-level section called templates.

name: templ
context:
    value:
    - 10
    - 20
    - 30
templates:
- name: one
  definition: '{{ value | join(sep=" // ") }}'
input:
    text: 'hello $T{one}'
output:
    write: console
# hello 10 // 20 // 30

The full power of Tera templates is available.

name: snifftest
context:
  interface: eth0
  inverted: false
  OID: '.1.3.6.1.2.1.2.2.1'
templates:
   - name: snmpd
     definition: |
        {% if inverted -%}
            {%- set INIF = 16 -%}
            {%- set OUTIF = 10 -%}
        {%- else -%}
            {%- set INIF = 10 -%}
            {%- set OUTIF = 16 -%}
        {%- endif -%}
        {{OID}}.{{INIF}}.interface {{OID}}.{{OUTIF}}.interface
input:
    text: 'snmpdelta $T{snmpd}'
output:
    write: console
# snmpdelta .1.3.6.1.2.1.2.2.1.10.interface .1.3.6.1.2.1.2.2.1.16.interface

Event Field Expansion (${variable})

At several places in the DSL it is possible to expand strings containing ${field} references.

For example, given {"date":"2018-06-19","time":"05:31:07"} we can construct the full ISO time as below:

- add:
    output-fields:
    - '@timestamp': '${date}T${time}Z'

(As with context expansions, the result is always a string)

Actions where field expansion is allowed:

• add - output-fields, template and template-file
• time - when, output-fields
• exec - command
• output - see outputs below
• inputs

add is special since the field names can also be expanded.

Outputs where field expansion happens:

• exec - command (but only if stdin: false and/or input-field defined)
• http-post - url (but only if batch: 0)
• redis  - key
• http-server - body

Inputs do not normally receive events so event field expansions are not present. But scheduled inputs have access to schedule variables

• exec - command
• http-poll - query values, address and body (if method is POST)
• redis - hash key

However, when run as actions, event field expansions happen.