Skip to main content

Pipe Outline

This section describes the basic structure of Pipe definitions. It assumes you are familiar with the Pipe introduction.

Because a Pipe definition is a YAML document, it's items can technically appear in any order. As a convention, we highly recommend that you order your Pipe definitions in their "natural" order, as described below.

Pipe Names​

Within a Hotrod Server, all Pipes must have a unique name. The name of a Pipe acts as an internal identifier, against which logs, metrics and other events are collected and stored.

name: example-pipe

Pipe names must:

  1. Only contain ASCII letters (a-z, A-Z) and numbers (0-9), underscores (_) and hyphens (-).
  2. Not contain spaces.
  3. Not end with a hyphen followed by a digit, like my-pipe-3. This pattern is internally used for some deployment features.
  4. Not be the word AGENT, which is reserved for special internal use.

Beyond these 4 restrictions, the platform doesn't insist on a particular naming convention. When working with many Pipes on a Server, it's recommended to follow some kind of naming convention that's appropriate for your context.

note

Because Pipe names are unique identifiers on the platform, Pipes can't be renamed. If you need to rename a Pipe, copy it's content into a new Pipe, with a new name.

Local Pipe Context​

Context variables can be defined locally within a Pipe definition. The context section is optional.

# optional
context:
    key: value
    anotherKey: anotherValue

Pipe Input​

Every Pipe definition requires one input specification, which may be any valid input defined in the Pipe Language.

This example uses the files input:

# input is required
input:
    files:
        path: /var/log/auth.log

Pipe Actions​

Pipe definitions may have zero or more actions to apply to events generated by the input. Actions are applied in the order that they are specified, which is a YAML list.

actions:

# first action
- add:
    fields:
        - newField: newValue

# second action
- rename:
    fields:
        - oldName: newName

# additional actions...

While actions generally operate on a single event at a time, certain actions have different behaviour:

  • Some actions may not output any events (like assert, filter, abort)
  • Some actions may consume a single event, and output multiple events (like expand, generate)
  • Some actions may consume multiple events, and output a single event (like transaction, collapse)

Pipe Output​

Every Pipe definition requires one output specification, which may be any valid output defined in the Pipe Language.

This example uses the file output:

# output is required
output:
  file:
    path: /tmp/custom.log