Output: s3
Write events to a S3 bucket file
Field Summary
| Field Name | Type | Description | Default |
|---|---|---|---|
| retry | Retry | For operations that could potentially fail | - |
| batch | integer or the 'document' | Maximum number of events in an output batch. If 'document' send on end of document | - |
| timeout | interval | Interval after which the batch is sent, to keep throughput going | 100ms |
| header | templated text | Put a header line before the batch | - |
| footer | templated text | Put a header line after the last line of the batch | - |
| use-document-marker | bool | Enrich the pipe metadata with a document marker (for document handling in batch mode) | false |
| input-field | field | Use the specified field as the content for the file line | - |
| bucket-name | string | Bucket Name | - |
| disable-object-name-guid | bool | Do not append a GUID to the output object name | false |
| guid-prefix | string | GUID Prefix, will be prepended to the GUID, the default value is "/" | - |
| guid-suffix | string | GUID Suffix, will be appended to the GUID if specified | - |
| object-name | string | Object Name | - |
| object-name-field | field | Field containing the Object Name | - |
| region | string | Region | - |
| endpoint | string | S3 Endpoint | - |
| access-key | string | Access Key ID | - |
| secret-key | string | Secret Key ID | - |
| security-token | string | Security Token | - |
| session-token | string | Session Token | - |
| preprocessors | OutputPreProcessor | Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified | - |
| scuba | Scuba | If this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance | - |
Fields
retry
Type: Retry
For operations that could potentially fail
| Field Name | Type | Description | Default |
|---|---|---|---|
| count | integer | How many attempts to make before declaring failure | - |
| pause | duration | How long to pause before re-trying | - |
| forever | bool | Keep trying until success is declared | false |
count
Type: integer
How many attempts to make before declaring failure
Example
Pipe Language Snippet:
exec:
command: echo 'one two'
retry:
count: 1
Output:
{"_raw":"one two"}
pause
Type: duration
How long to pause before re-trying
Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)
Example
Pipe Language Snippet:
exec:
command: echo 'one two'
retry:
count: 6
pause: 10s
Output:
{"_raw":"one two"}
forever
Type: bool
Default: false
Keep trying until success is declared
Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)
Example
Pipe Language Snippet:
exec:
command: echo 'one two'
retry:
forever: true
Output:
{"_raw":"one two"}
batch
Type: integer or the 'document'
Maximum number of events in an output batch. If 'document' send on end of document
timeout
Type: interval
Default: 100ms
Interval after which the batch is sent, to keep throughput going
header
Type: templated text
Put a header line before the batch
footer
Type: templated text
Put a header line after the last line of the batch
use-document-marker
Type: bool
Default: false
Enrich the pipe metadata with a document marker (for document handling in batch mode)
input-field
Type: field
Use the specified field as the content for the file line
bucket-name
Type: string
Bucket Name
disable-object-name-guid
Type: bool
Default: false
Do not append a GUID to the output object name
guid-prefix
Type: string
GUID Prefix, will be prepended to the GUID, the default value is "/"
guid-suffix
Type: string
GUID Suffix, will be appended to the GUID if specified
object-name
Type: string
Object Name
object-name-field
Type: field
Field containing the Object Name
region
Type: string
Region
endpoint
Type: string
S3 Endpoint
access-key
Type: string
Access Key ID
secret-key
Type: string
Secret Key ID
security-token
Type: string
Security Token
session-token
Type: string
Session Token
preprocessors
Type: OutputPreProcessor
Preprocessors (process data before making it available for upload) these processors will be run in the order they are specified
| Field Name | Type | Description | Default |
|---|---|---|---|
| gzip | Gzip the output data | - |
gzip
Gzip the output data
scuba
Type: Scuba
If this Pipe is sending data to Scuba Lite, optionally have the Pipe configure the Scuba Lite instance
| Field Name | Type | Description | Default |
|---|---|---|---|
| host | string | The Scuba Lite instance (for example: https://scuba-lite.example.com) | - |
| insecure | bool | Allow self-signed certificates for Scuba Lite connections | false |
| token | string | The Scuba Lite API token | - |
| table-name | string | The Scuba Lite table name | - |
| file-pattern | string | The Scuba Lite file search pattern | - |
| columns | ScubaLiteColumns | - |
host
Type: string
The Scuba Lite instance (for example: https://scuba-lite.example.com)
insecure
Type: bool
Default: false
Allow self-signed certificates for Scuba Lite connections
token
Type: string
The Scuba Lite API token
table-name
Type: string
The Scuba Lite table name
file-pattern
Type: string
The Scuba Lite file search pattern
columns
Type: ScubaLiteColumns
| Field Name | Type | Description | Default |
|---|---|---|---|
| name | string | The name of the column | - |
| type | ScubaLiteColumnType | The type of the column | - |
| actor | bool | Specify if the column is an actor | false |
name
Type: string
The name of the column
type
Type: ScubaLiteColumnType
The type of the column
| Field Name | Type | Description | Default |
|---|---|---|---|
| string | - | ||
| time | string | - |
string
time
Type: string
actor
Type: bool
Default: false
Specify if the column is an actor