Output: exec

Execute arbitrary commands

Things to note:

• By default, input data streams through the output process, i.e. works like a shell filter. You cannot have ${field} expansions of the command when streaming, so streaming is switched off. Then the command will be executed once for each input line. With input-field you have to explicitly ask for streaming: true for reasons of backward compatibility.
• With batched output you will have access to the fields of the first event in the batch

Field Summary

Field Name	Type	Description	Default
batch	integer or the 'document'	Maximum number of events in an output batch. If 'document' send on end of document	-
timeout	interval	Interval after which the batch is sent, to keep throughput going	100ms
header	templated text	Put a header line before the batch	-
footer	templated text	Put a header line after the last line of the batch	-
use-document-marker	bool	Enrich the pipe metadata with a document marker (for document handling in batch mode)	false
input-field	field	A field from where data is to be read, meaning JSON is assumed	-
command	string	A shell command to be executed	-
streaming	bool	Run the command separately for each event	true
retry	Retry	For operations that could potentially fail	-

Fields

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

header

Type: templated text

Put a header line before the batch

footer

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

input-field

Type: field

A field from where data is to be read, meaning JSON is assumed

Otherwise the whole event is sent

Example

Input:

{"text":"one","number":1}

Pipe Language Snippet:

exec:
  input-field: text
  command: cat

Output:

one

command

Type: string

A shell command to be executed

If streaming is set to false (or input-field by default) then there can be field expansions

Example

Input:

{"text":"one","number":1}

Pipe Language Snippet:

exec:
  command: cat

Output:

{"text":"one","number":1}

streaming

Type: bool
Alias: input
Default: true

Run the command separately for each event

Normally, the command runs continuously and events are passed through

Example: streaming must be explicit if input-field is defined

Input:

{"msg":"hello dolly"}
{"msg":"we are happy"}

Pipe Language Snippet:

exec:
  input-field: msg
  streaming: true
  command: tr 'h' 'H'

Output:

Hello dolly
we are Happy

Example: streaming is implicit if there are field expansions

Input:

{"text":"one","number":1}

Pipe Language Snippet:

exec:
  command: echo "${text}-${number}"

Output:

one-1

retry

Type: Retry

For operations that could potentially fail

Field Name	Type	Description	Default
count	integer	How many attempts to make before declaring failure	-
pause	duration	How long to pause before re-trying	-
forever	bool	Keep trying until success is declared	false

  count

Type: integer

How many attempts to make before declaring failure

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 1

Output:

{"_raw":"one two"}

  pause

Type: duration

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 6
    pause: 10s

Output:

{"_raw":"one two"}

  forever

Type: bool

Default: false

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    forever: true

Output:

{"_raw":"one two"}