Context

note

Overriding a context variable causes the redeployment of the affected Pipes, with Context Expansion happening on the Server.

There are default context variables always available to a Pipe (v3.5.0 upwards):

• Agent name: {{name}}
• Agent ID: {{agent}}
• Pipe name: {{pipe}}

It is common for a Pipe to enrich its data with the name of the Agent it is running on:

add:
  output-fields:
  - name: '{{name}}'

Priority

A higher-priority context overrides a lower-priority context, with Agent context being the highest priority and local context being the lowest:

• Agent
• Tenant
• Pipe
• Tag
• global
• local

Ping Example

Consider a simple Pipe which pings a site and subsequently uses extract for the ping times:

name: ping

input:
  exec:
    command: 'ping -i 4 google.com'

actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec

  - convert:
      - msec: num

  - add:
      output-fields:
        - site: google.com

output:
  file:
    path: /tmp/{{name}}

The following steps will:

• add the Pipe to the Server and make it available for deployment

• create an Agent named Office using add (see Agent setup

• update the Office Agent by adding the ping Pipe

$> hotrod pipes add --file ping.yml

$> hotrod agents add Office

agent Office has id 7fdfb5e9-bb54-49fc-bdc6-9e9490706cf4

$> hotrod agents update Office --add-pipe ping

[INFO] user=admin adding pipe=ping to target=Office

The Pipe definition has two arbitrary values:

1. The site itself: google.com

2. The ping interval: -i 4

Now add a context section:

name: ping

context:
  site: google.com
  interval: 4

input:
  exec:
    command: 'ping -i {{interval}} {{site}}'

actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec

  - convert:
      - msec: num

  - add:
      output-fields:
        - site: '{{site}}'

output:
  file:
    path: /tmp/{{name}}

Any context values in {{}} will undergo Context Expansion (i.e., the Agent {{name}}).

note

Context Expansion occurs on the Server, the rendered (internal representation) version of the Pipe gets sent to the Agent. Context can originate from the Server (global), Agent, or Pipe.

Now update the Pipe:

$> hotrod pipes update --file ping.yml

Agents affected (Pipe removed/added):
- Office

The resulting output:

$> tail -f /tmp/Office

{"msec":48.3,"site":"google.com"}
{"msec":49,"site":"google.com"}
{"msec":48.7,"site":"google.com"}
...

Overriding

While this a useful way to avoid repetition, the real power of context is the way that it can be overridden:

$> hotrod context global site=yahoo.com

[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed

The resulting output:

$> tail -f /tmp/Office

{"msec":374,"site":"yahoo.com"}
{"msec":306,"site":"yahoo.com"}
{"msec":351,"site":"yahoo.com"}
...

You can now add another Durban Agent and deploy the ping Pipe to it:

$> hotrod agents add Durban

agent Durban has id 8c71be6f-b756-428c-8d35-cbe7be0dcfbe

$> hotrod agents update Durban --add-pipe ping

[INFO] user=admin adding pipe=ping to target=Durban

Known as the local context, it will ping yahoo.com since the global context overrides the original Pipe context.

Confirm the Agent is indeed pinging yahoo.com:

$> tail -f /tmp/Durban

{"msec":387,"site":"yahoo.com"}
{"msec":357,"site":"yahoo.com"}
{"msec":363,"site":"yahoo.com"}

Global Context

Alternately you may provide context in the form of a simple YAML file:

$> cat vars.yml

one: 1
two: 2

$> hotrod context global -f vars.yml

[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed

Show the available global context:

$> hotrod context global

[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Office (0850a5b7-3861-4787-acea-f43de43a7915) deployed

:::warn hotrod pipes run -c vars.yml uptime_enrich.yml results in an error! :::

The resulting error:

2023-04-06T11:46:01.303Z ERROR hotrod > bad pipe: unknown field `one`, expected one of `name`, `metadata`, `trace`, `debug`, `owner`, `env`, `tests`, `hidden`, `tags`, `foreach`, `templates`, `files`, `context`, `context-map`, `input`, `actions`, `output`

For instance, there is something special about Durban and we want it to ping Johannesburg.

$> hotrod agents context Durban site=joburg.co.za

A per-Agent context overrides global context as it is more specialized.

We can now create yet another Agent, Cape Town, and attach the Tag seaside to it, as well as to Durban:

$> hotrod agents add 'Cape Town' --tags seaside

agent Cape Town has id 537c7521-9151-4384-b8c3-0103c417958c

$> hotrod agents update 'Cape Town' --add-pipe ping

[INFO] user=admin adding pipe=ping to target=Cape Town

$> hotrod agents update Durban --add-tag seaside

[INFO] user=admin added tag seaside to target=Durban

$> hotrod agents list

 name      | id                                   | tags      | pipes  | last seen
-----------+--------------------------------------+-----------+--------+-----------
 Office    | a5bb18fc-6d5a-43fc-a5fa-7835f14d6713 |           | ping   |
 Durban    | f98ea805-4210-4ba2-a898-80148e3aaf5a | seaside   | ping   |
 Cape Town | 537c7521-9151-4384-b8c3-0103c417958c | seaside   | ping   |

Tag Context

You can attach context to a tag:

$> hotrod context tag seaside addr=jhb.co.za

[INFO] target Durban (95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b) deployed
[INFO] target Cape Town (ff18d520-f723-4fc0-8f08-e88a1454b1ca) deployed

That way, Cape Town gets jhb.co.za from the tag context while Durban remains joburg.co.za because Agent context always wins:

$> hotrod agents show 'Cape Town'

id: ff18d520-f723-4fc0-8f08-e88a1454b1ca
tenant: default
tags: seaside
pipes: ping

# Context:
# site: jhb.co.za

$> hotrod agents show Durban

id: 95cb8a32-52f4-44a4-bd8a-2d7c7ea1357b
tenant: default
tags: seaside
pipes: ping

# Context:
# site: joburg.co.za

tag context is a useful way for a whole group of Agents to acquire different settings from the global default.

All instances of a Pipe running on different Agents share a Context — settable with Pipes context. In this example, we can point ping at a new site by setting the site in the Pipe context — except in a case where the Agent context overrides.

So, a higher-priority context overrides a lower-priority context, with Agent context being the highest priority and local context being the lowest.

Note that administrators with restricted privileges for a group of Agents, cannot set a Pipe, Tag or global context as this may affect the operation of Pipes running on other Agents.

Multiple Hosts in a Pipe

For instance, if we wanted a Pipe to ping multiple hosts, one solution would be to make copies of ping.yml as ping1.yml, ping2.yml. This would, however, become a tedious exercise as all the files need to be changed together. The foreach feature was designed to handle this easily and transparently:

name: pinga

context:
  sites:
  - google.com
  - yahoo.com

foreach:
  site: sites

input:
  exec:
    command: ping -i 4 {{site}}

actions:
- extract:
    input-field: _raw
    remove: true
    pattern: '=(\S+) ms$'
    output-fields:
      - msec

- convert:
    - msec: num

- add:
    output-fields:
      - site: '{{site}}'

output:
  file:
    path: /tmp/{{name}}

Again, the default hosts go into the context as addresses, but as an array. foreach says that each address should come from this context array.

The result is that two Pipe services are created on the Agent, one with ping -i 2 google.com and another with ping -i 2 yahoo.com. The Pipe services will be called pinga-00 and pinga-01. Effectively, this creates two copies of the Pipe definition for the Agent.

The array may be nested, using the same dot notation for context expansions:

context:
  my_context:
    sites:
    - google.com
    - yahoo.com

foreach:
  site: my_context.sites

Multiple Interfaces in a Pipe

We now want to ping the same hosts, but through either one of the available interfaces on the host:

name: pingaif

context:
  sites:
    - google.com
    - yahoo.com
  interfaces:
    - eth0
    - eth1

foreach:
  site: sites
  interface: interfaces

input:
  exec:
    command: ping -i 2 {{site}} -I {{interface}}

This generates four Pipe services for the possible combinations (pingaif-01, pingaif-02...):

• ping -i 2 google.com -I eth0

• ping -i 2 google.com -I eth1

• ping -i 2 yahoo.com -I eth0

• ping -i 2 yahoo.com -I eth1

Another form of foreach allows you to go over nested arrays in the following context:

name: hosttrio

context:
  hosts:
    - spoof: 192.168.66.38
      agents:
        - 1.1.1.1
        - 2.2.2.2
      port: 22
    - spoof: 192.168.66.39
      agents:
        - 3.3.3.3
        - 4.4.4.4
      port: 22      

foreach:
  host: hosts[agents]

input:
  exec:
    command: echo {{host.spoof}} {{host.agents}}:{{host.port}}
    raw: true

output:
  write: console

# Output:
# 192.168.66.38 1.1.1.1:22
# 192.168.66.38 2.2.2.2:22
# 192.168.66.39 3.3.3.3:22
# 192.168.66.39 4.4.4.4:22

Aliases

An alias refers to an input, output, or a set of actions.

Consider the following enrich.alias:

name: enrich

actions:
- add:
    output-fields:
      - name: '{{name}}'
      - pipe: '{{pipe}}'
      - agent: '{{agent}}'
      - tenant: '{{tenant}}'

- time:
    output-field: '@timestamp'

It looks very much like a Pipe definition, but it does not need to have all three of the components of a Pipe (input, actions and output).

This is because it is substituted into a Pipe definition, note the alias action:

name: uptime_enrich

input:
  exec:
    command: uptime
    interval: 2s

actions:
  - extract:
      input-field: _raw
      pattern: 'load average: (\S+), (\S+), (\S+)'
      output-fields: [m1, m5, m15]

  - alias: enrich

output:
  print: STDOUT

All the site-specific enrichments can be wrapped up in an alias definition and reused in any Pipe. Update aliases with (currently, broken, workaround by including everything in the Pipe definition):

$> hotrod aliases update enrich.alias

info

Aliases can also provide predefined, reusable, definitions for input and output — which is useful for testing.

Then do:

$> hotrod pipes add --file uptime_enrich.yml

$> hotrod agents update Office --add-pipe ping

[INFO] user=admin adding pipe=uptime_enrich to target=Office

This Pipe, is rendered on the Server and deployed to the Agent (Server/Agent context). Inspect the output on the Office Agent host:

$> journalctl -fu hotrod-agent

Apr 05 21:46:43 rocky9.ephemeric.lan hotrod-agent[296602]: {"_raw":" 21:46:43 up 5 days, 2:16, 1 user, load average: 1.00, 1.00, 1.00","m1":"1.00","m5":"1.00","m15":"1.00","name":"Office","tenant":"default","pipe":"uptime_enrich","agent":"4255b0e1-3ea5-4dcf-b6f5-9ead3e91be5d","@timestamp":"2023-04-05T19:46:43.234Z"}

The Pipe can run from the CLI, explicitly specifying the `context` file (see `hotrod pipes run --help`):

:::note
Context Expansion variables like `{{name}}`, `{{tenant}}`, `{{agent}}`, and `{{pipe}}` etc., are _not_ available in the CLI Pipe runtime as they are rendered on the Server.
:::

```sh
$> hotrod pipes run --context-files enrich.alias --file uptime_enrich.yml

{"_raw":" 23:06:33 up 5 days,  3:36,  1 user,  load average: 1.13, 1.06, 1.02","name":"{{name}}","pipe":"{{pipe}}","agent":"{{agent}}","tenant":"{{tenant}}","@timestamp":"2023-04-05T21:06:33.312Z"}
...

danger

The CLI Pipe runtime does not merge, but replaces when an alias file is used for --context-files. Note the above output: the extract action has been replaced with the add and time actions (missing m1, m5, and m15 fields).

note

When running Pipes from the CLI, there is no Server/Agent context, therefore context must be defined in the Pipe definition or in a context file.