Context

Consider the following simple pipe, which pings a site and extracts the ping times:

name: ping
input:
  exec:
    command: 'ping -i 4 google.com'
actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec
  - convert:
      - msec: num
  - add:
      output-fields:
        - site: google.com
output:
  exec:
    command: cat

The following steps can be used to create a new site and add this pipe:

$ hotrod pipes add --file ping.yml
$ hotrod agents add Office
$ hotrod agents update Office --add-pipe ping

This pipe has two arbitrary values: the site itself and the time between pings.

So, add a context section:

name: ping
context:
  addr: google.com
  interval: 4
input:
  exec:
    command: 'ping -i {{interval}} {{addr}}'
actions:
  - extract:
      input-field: _raw
      remove: true
      pattern: '=(\S+) ms$'
      output-fields:
        - msec
  - convert:
      - msec: num
  - add:
      output-fields:
        - site: '{{addr}}'
output:
  exec:
    command: cat

Any context values can be expanded in {{}} (double-braces).

Running this, after hotrod pipes update --file ping.yml to update:

{"msec":48.3,"site":"google.com"}
{"msec":49,"site":"google.com"}
{"msec":48.7,"site":"google.com"}
....

This is a useful way to avoid repetition, but the real power of contexts is that they can be overridden:

$ hotrod context global addr=yahoo.com
# .... run as before
{"msec":374,"site":"yahoo.com"}
{"msec":317,"site":"yahoo.com"}
{"msec":362,"site":"yahoo.com"}
{"msec":306,"site":"yahoo.com"}
{"msec":351,"site":"yahoo.com"}
....

You can now add another site, and deploy the ping pipe to it:

$ hotrod agents add Durban
site Durban has id 8c71be6f-b756-428c-8d35-cbe7be0dcfbe
$ hotrod agents update Durban --add-pipe ping

Again, it will ping "yahoo.com" since the global context overrides the original pipe context - which we call the local context.

You may alternately provide context as a simple YAML file:

$ cat > vars.yml
one: 1
two: 2
$ hotrod context global -f vars.yml

At any point, hotrod context show will display the available contexts.

There is something special about Durban and we want it to ping Johannesburg.

hotrod agents context Durban addr=panoptix.io

A per-agent context overrides global context since it is more specialized.

We can create yet another agent, Cape Town, then attach the tag "seaside" to it, as well as to Durban.

$ hotrod agents add 'Cape Town' --tags seaside
site Cape Town has id 537c7521-9151-4384-b8c3-0103c417958c
$ hotrod agents update 'Cape Town' --add-pipe ping
$ hotrod agents update Durban --add-tag seaside
$ hotrod agents list
 name      | id                                   | tags      | pipes  | last seen
-----------+--------------------------------------+-----------+--------+-----------
 Office    | a5bb18fc-6d5a-43fc-a5fa-7835f14d6713 |           | ping   |
 Joburg    | 9568fdba-f4f6-4d7e-a6ba-dcfcef64cf95 |           | uptime |
 Durban    | f98ea805-4210-4ba2-a898-80148e3aaf5a | seaside   | ping   |
 Cape Town | 537c7521-9151-4384-b8c3-0103c417958c | seaside   | ping   |

You can attach context to a tag:

hotrod context tag seaside addr=jhb.co.za

That way, Cape Town gets "jhb.co.za" from the tag context - Durban remains "panoptix.io" since agent context always wins.

$ hotrod agents show 'Cape Town'
id: 537c7521-9151-4384-b8c3-0103c417958c
tenant: default
tags: seaside
pipes: ping

$ hotrod agents show Durban
id: 8c71be6f-b756-428c-8d35-cbe7be0dcfbe
tenant: default
tags: seaside
pipe: ping
context:
---
addr: panoptix.io

Tag contexts are a useful way for a whole group of agents to acquire different settings from the global default.

All instances of a pipe running on different agents share a context, which is settable with hotrod pipes context. So we can point all our pings at a new address by setting addr in the pipe context, except where the agent context overrides.

To summarize: a higher-priority context overrides a lower-priority context, with agent contexts the highest priority and local context the lowest priority

• agent
• tenant
• pipe
• tag
• global
• local

Note that admins with limited responsibility for a group of agents ("tenant admins') cannot set pipe, tag or global context since this might affect the operation of pipes running on other agents.

There are three context variables always available to a pipe:

• pipe the name of the pipe itself
• name the agent name
• agent the agent identifier
• tenant the tenant (group owning the resource)

So it is common for a pipe to enrich its data with the name of the agent it is running on:

add:
  output-fields:
  - name: '{{name}}'

Please note that overriding a context variable causes the redeployment of the affected pipes, since context expansion happens server-side.

Multiple hosts in a single Pipe

If we wanted an agent to ping multiple hosts, then one solution would be to make copies of ping.yml as ping1.yml, ping2.yml. This would be tedious because all these files need to be changed together. The foreach feature handles this transparently:

name: pinga
context:
  addresses:
  - google.com
  - yahoo.com
foreach:
  addr: addresses
input:
  exec:
    command: ping -i 2 {{addr}}
...

Again, the default hosts go in the context as addresses, but as an array.

foreach here says that each addr should come from this context array.

The result is that two pipe services will be created on the agent, one with ping -i 2 google.com and another with ping -i 2 yahoo.com. The pipe services will be called ping-00 and ping-01.

Effectively this creates two copies of the pipe definition for the agent.

The array may be nested, using the same dot notation for context expansions:

context:
  my_context:
    addresses:
    - google.com
    - yahoo.com
foreach:
  addr: my_context.addresses

We now want to ping the same hosts, but through either one of the available interfaces on the machine:

name: pinga
context:
  addresses:
    - google.com
    - yahoo.com
  interfaces:
    - eth0
    - eth1
foreach:
  addr: addresses
  interface: interfaces
input:
  exec:
    command: ping -i 2 {{addr}} -I {{interface}}

This generates four pipe services for the possible combinations:

• ping -i 2 google.com -I eth0
• ping -i 2 google.com -I eth1
• ping -i 2 yahoo.com -I eth0
• ping -i 2 yahoo.com -I eth1

Another form of foreach allows you to go over nested arrays in the context:

name: test1
context:
  hosts:
    - spoof: 192.168.66.38
      agents:
        - 1.1.1.1
        - 2.2.2.2
      port: 22
    - spoof: 192.168.66.39
      agents:
        - 3.3.3.3
        - 4.4.4.4
      port: 22      
foreach:
  host: hosts[agents]
input:
  exec:
    command: echo {{ host.spoof }} {{ host.agents }}:{{ host.port }}
    raw: true
output:
  write: console
# 192.168.66.38 1.1.1.1:22
# 192.168.66.38 2.2.2.2:22
# 192.168.66.39 3.3.3.3:22
# 192.168.66.39 4.4.4.4:22

Aliases

An alias is a way to refer to an input, output, or set of actions.

For example, consider test.alias:

name: enrich
actions:
- add:
    output-fields:
      - name: '{{name}}'
      - tenant: '{{tenant}}'
- time:
    output-field: '@timestamp'

It looks very much like a pipe definition, except that it does not need to have all three components of a pipe (input, actions, and output).

This is because it is substituted into a pipe definition, so:

name: uptime-enrich
input:
  exec:
    command: uptime
    interval: 2s
actions:
  - extract:
      input-field: _raw
      pattern: 'load average: (\S+), (\S+), (\S+)'
        output-fields: [m1, m5, m15]
  - alias: enrich
output:
    write: console

All the site-specific enrichments can be wrapped up in an alias definition, and reused in any pipe.

Once the alias file has been updated with hotrod aliases update test.alias, uptime-enrich.yml can be loaded in the usual way.

Aliases can also provide 'canned' definitions of inputs and outputs, which is useful for testing.