Skip to main content
Version: Next

Output: s3

Write events to a S3 bucket file

Field Summary

Field NameTypeDescriptionDefault
retryRetryFor operations that could potentially fail-
batchinteger or the 'document'Maximum number of events in an output batch. If 'document' send on end of document-
timeoutintervalInterval after which the batch is sent, to keep throughput going100ms
headertemplated textPut a header line before the batch-
footertemplated textPut a header line after the last line of the batch-
use-document-markerboolEnrich the pipe metadata with a document marker (for document handling in batch mode)false
input-fieldfieldUse the specified field as the content for the file line-
bucket-namestringBucket Name-
disable-object-name-guidboolDo not append a GUID to the output object namefalse
guid-prefixstringGUID Prefix, will be prepended to the GUID, the default value is "/"-
guid-suffixstringGUID Suffix, will be appended to the GUID if specified-
object-namestringObject Name-
object-name-fieldfieldField containing the Object Name-
regionstringRegion-
endpointstringS3 Endpoint-
access-keystringAccess Key ID-
secret-keystringSecret Key ID-
security-tokenstringSecurity Token-
session-tokenstringSession Token-
preprocessorsOutputPreProcessorPreprocessors (process data before making it available for upload) these processors will be run in the order they are specified-
scubaScubaIf this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance-

Fields

retry

Type: Retry

For operations that could potentially fail

Field NameTypeDescriptionDefault
countintegerHow many attempts to make before declaring failure-
pausedurationHow long to pause before re-trying-
foreverboolKeep trying until success is declaredfalse

  count

Type: integer

How many attempts to make before declaring failure

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
count: 1

Output:

{"_raw":"one two"}

  pause

Type: duration

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
count: 6
pause: 10s

Output:

{"_raw":"one two"}

  forever

Type: bool

Default: false

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
command: echo 'one two'
retry:
forever: true

Output:

{"_raw":"one two"}

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

Type: templated text

Put a header line before the batch

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

input-field

Type: field

Use the specified field as the content for the file line

bucket-name

Type: string

Bucket Name

disable-object-name-guid

Type: bool

Default: false

Do not append a GUID to the output object name

guid-prefix

Type: string

GUID Prefix, will be prepended to the GUID, the default value is "/"

guid-suffix

Type: string

GUID Suffix, will be appended to the GUID if specified

object-name

Type: string

Object Name

object-name-field

Type: field

Field containing the Object Name

region

Type: string

Region

endpoint

Type: string

S3 Endpoint

access-key

Type: string

Access Key ID

secret-key

Type: string

Secret Key ID

security-token

Type: string

Security Token

session-token

Type: string

Session Token

preprocessors

Type: OutputPreProcessor

Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified

Field NameTypeDescriptionDefault
gzipGzip the output data-

  gzip

Gzip the output data

scuba

Type: Scuba

If this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance

Field NameTypeDescriptionDefault
hoststringThe Scuba Lite instance (for example: https://scuba-lite.example.com)-
insecureboolAllow self-signed certificates for Scuba Lite connectionsfalse
tokenstringThe Scuba Lite API token-
table-namestringThe Scuba Lite table name-
file-patternstringThe Scuba Lite file search pattern-
columnsScubaLiteColumns-

  host

Type: string

The Scuba Lite instance (for example: https://scuba-lite.example.com)

  insecure

Type: bool

Default: false

Allow self-signed certificates for Scuba Lite connections

  token

Type: string

The Scuba Lite API token

  table-name

Type: string

The Scuba Lite table name

  file-pattern

Type: string

The Scuba Lite file search pattern

  columns

Type: ScubaLiteColumns

Field NameTypeDescriptionDefault
namestringThe name of the column-
typeScubaLiteColumnTypeThe type of the column-
actorboolSpecify if the column is an actorfalse

  name

Type: string

The name of the column

  type

Type: ScubaLiteColumnType

The type of the column

Field NameTypeDescriptionDefault
string-
timestring-

  string

  time

Type: string

  actor

Type: bool

Default: false

Specify if the column is an actor