Data Input/Output

This section covers how data enters and exits a Pipe.

Inbound Data

Pipes operate on textual data of any shape or format, whether structured (JSON, CSV, XML, delimited, key-value, etc) or unstructured (typically logs). Various inputs will consume non-textual sources, but they output data in textual form.

Data enters a Pipe as events, which represents the individual parts of a data stream that a Pipe processes. How inputs split inbound data streams are particular to the specific input. This behaviour is configurable on certain inputs, usually with the raw or json options.

By default, all inputs produce events as JSON documents. Some inputs produce a structured event, such as:

{"field": "value", "anotherField": true, "amount": 11235}

When an input produces an unstructured event, it wraps the incoming data into a JSON document with a standard form:

{"_raw": "content of an unstructured input event"}

Pipe actions expects and operates on JSON events, except for the raw action.

Some inputs have options that will produce events in their raw form, preventing the JSON wrapping:

Some unstructured input line
produced in raw form

The special raw action will transform and manipulate non-JSON events, and can convert them to JSON events.

Outbound Data

Pipe outputs produce JSON documents by default. The format in which an event reaches a Pipe's output determines the format of output data. Events reach the output of a Pipe either from the last defined action, or the Pipe input in the case of no actions.

To output non-JSON data, a variety of actions can convert JSON events into other formats.

Events will leave a Pipe in the same order they arrived in. Some outputs support a batch option to group output events by various criteria.