Field Expansion

There are two kinds of Variable Expansion in Pipes:

1. Server-side processing-time expansion involving context.
2. Run-time expansion where values are extracted from data fields.

Context Expansion

All Pipes have associated parameters such as how often to sample and what network interface to probe, etc. It is advised that you give these values names and define them in the context section of a Pipe.

note

As outlaid in Contexts, these parameters can then be overridden.

Double-braces, {{var}}, are used to enclose context variables.

The Pipe context predefines certain default variables (v3.5.0 upwards):

• Agent name: {{name}}
• Agent ID: {{agent}}
• Pipe name: {{pipe}}

context variables can also be used to simplify regexes:

name: temp

context:
  INT: '(\d+)'

input:
  echo:
    event: echo "today date is 2019/07/22"

actions:
- extract:
    remove: true
    pattern: '{{INT}}/{{INT}}/{{INT}}'
    output-fields:
    - year
    - month
    - day

- convert:
    auto: true

output:
    print: STDOUT

context definitions may include variables:

context:
    PORT: 3030
    URL: 'http://0.0.0.0:{{PORT}}'

So {{URL}} will expand to http://0.0.0.0:3030.

It is useful to give names to arbitrary values, but the real power of context variables is that they can be overridden, according to desired precedence:

name: ping

context:
    interval: 5s
    address: google.com

input:
    exec:
        command: ping {{address}}
        interval: '{{interval}}'

output:
    write: console

The Pipe above can be directed at different sites and can poll at different rates by controlling the Pipe context.

note

Changing the Pipe context, will trigger a Pipe redeployment and a restart.

context values can also be nested:

name: temp

context:
  object:
    person:
      name: frodo
      age: 50

input:
  exec:
    command: echo 'hello {{object.person.name}} your age is {{object.person.age}}'

output:
  print: STDOUT

Although context.object.person.age is a number (int), context expansion always results in a text value (str).

Likewise, elements can be accessed as such:

name: array

context:
    array:
    - one
    - two
    - 3

input:
    exec:
        command: echo 'The elements {{array.0}}, {{array.1}}, and {{array.2}} are for the [{{array}}] array.'
        json: true

output:
    write: console

{{array}} by default renders as "one","two",3 — i.e., strings are quoted.

context variables can be followed by an explicit delimiter. Therefore, {{array ' '}} will render as "one" "two" 3.

A wildcard expression allows a particular field to be extracted from an array of objects, as follows:

name: templ

context:
    hosts:
    - port: 10
      name: foo
    - port: 20
      name: boo
    - port: 30
      name: bar

input:
    text: host port {{hosts.*.port ' and host port '}}

output:
    write: console

# Output:
# host port 10 and host port 20 and host port 30

Arrays may be merged:

name: temp

context:
  birds: [swallow,thrush]
  animals: [cat,dog]

input:
  exec:
    command: echo {{merge(birds,animals)}}

output:
    write: console

# Output:
# {"_raw":"swallow,thrush,cat,dog"}

Template Expansion

A full templating language, Tera, is provided to function like an alternative Context Expansion method. These are defined in a section titled templates:

name: templ

context:
    value:
    - 10
    - 20
    - 30

templates:
- name: one
  definition: '{{ value | join(sep=" // ") }}'

input:
    text: 'hello $T{one}'

output:
    write: console

# Output:
# hello 10 // 20 // 30

The full power of templates can be accessed here:

name: snifftest

context:
  interface: eth0
  inverted: false
  OID: '.1.3.6.1.2.1.2.2.1'

templates:
   - name: snmpd
     definition: |
        {% if inverted -%}
            {%- set INIF = 16 -%}
            {%- set OUTIF = 10 -%}
        {%- else -%}
            {%- set INIF = 10 -%}
            {%- set OUTIF = 16 -%}
        {%- endif -%}
        {{OID}}.{{INIF}}.interface {{OID}}.{{OUTIF}}.interface

input:
    text: 'snmpdelta $T{snmpd}'

output:
    write: console

# Output:
# snmpdelta .1.3.6.1.2.1.2.2.1.10.interface .1.3.6.1.2.1.2.2.1.16.interface

Event Field Expansion

It is possible to expand strings containing ${field} references at several places in Pipe Language. For a date and time, e.g., {"date":"2018-06-19","time":"05:31:07"} we can construct the full ISO time as below:

- add:
    output-fields:
    - '@timestamp': '${date}T${time}Z'

As with Context Expansion, the result is always a text value (str).

actions where Field Expansion is allowed, are constantly being developed and added to Pipe Language.

Several output destinations also support Field Expansion:

• exec: command (only if stdin: false and/or input-field defined)
• http-post: url (only if batch: 0)
• redis: hash key
• http-server: body

input does not normally receive events, therefore, event Field Expansion is not present.

However, a scheduled input has access to scheduled variables:

• exec: command
• http-poll: query values, address, and body (if method is POST)
• redis: hash key

When run as actions, event Field Expansion will occur.