Skip to main content
Version: Next

Output: file

Write to a file

Can use field expansions to create the filename. When it changes, you can specify a header and/or footer

Example

Pipe Language Snippet:

file:
path: '/tmp/out-${now_time_fmt %Y-%M-%D}.log
input-field: out
header: Start: ${name}

Field Summary

Field NameTypeDescriptionDefault
batchinteger or the 'document'Maximum number of events in an output batch. If 'document' send on end of document-
timeoutintervalInterval after which the batch is sent, to keep throughput going100ms
headertemplated textPut a header line before the batch-
footertemplated textPut a header line after the last line of the batch-
use-document-markerboolEnrich the pipe metadata with a document marker (for document handling in batch mode)false
retryRetryFor operations that could potentially fail-
pathstringPath to the file to be written-
input-fieldfieldUse the specified field as the content for the file line-
flush-at-endboolWhen the filename changes a new file will be written, force a flush at that pointfalse
compress-afterboolCompress the written file after the filename changesfalse
create-directoriesboolCreate parent directories for the given path if they do not existfalse
append-to-existingboolAppend to an output file if it already existsfalse

Fields

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

Type: templated text

Put a header line before the batch

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

retry

Type: Retry

For operations that could potentially fail

Field NameTypeDescriptionDefault
countintegerHow many attempts to make before declaring failure-
pausedurationHow long to pause before re-trying-
foreverboolKeep trying until success is declaredfalse

  count

Type: integer

How many attempts to make before declaring failure

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
count: 1

Output:

{"_raw":"one two"}

  pause

Type: duration

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
count: 6
pause: 10s

Output:

{"_raw":"one two"}

  forever

Type: bool

Default: false

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
forever: true

Output:

{"_raw":"one two"}

path

Type: string

Path to the file to be written

input-field

Type: field

Use the specified field as the content for the file line

flush-at-end

Type: bool

Default: false

When the filename changes a new file will be written, force a flush at that point

compress-after

Type: bool

Default: false

Compress the written file after the filename changes

create-directories

Type: bool

Default: false

Create parent directories for the given path if they do not exist

append-to-existing

Type: bool

Default: false

Append to an output file if it already exists