Skip to main content
Version: Next

Input: files

Read from multiple files, in order of creation

Example

Pipe Language Snippet:

files:
path: /var/log/dpkg.*

Field Summary

Field NameTypeDescriptionDefault
pathpathA path with wildcards identifying files to be read-
data-dirpathThe directory where checkpoints are writtenthe platform-specific data directory for the user executing the pipe
encodingencodingThe text encoding for the monitored filesUTF-8
includearray of pathsAny other paths to be included-
excludearray of pathsPaths to be excluded-
ignore-older-thanintegerIgnore files older than this (epoch time in seconds)-
glob-minimum-cooldownintegerPause after scanning for new files (in milliseconds)300ms
stop-reading-afterboolDo not wait for new filesfalse
ignore-line-breaksboolConsume each file as one eventfalse
remove-afterintegerRemove files that are done reading after this period (seconds)-
fingerprintingFingerprinterFile fingerprinting strategyDevInode
oldest-firstboolDefault is youngest firstfalse
file-path-fieldfieldFill this field with the field we are currently reading from-
file-basenameboolThe file-path-field path will be the basename (e.g. '/path/frodo.txt' becomes just 'frodo')false
start-at-beginningboolIgnore stored checkpoints and re-read all specified filesfalse
rawboolDisable 'quoting' of lines with _rawfalse
max-read-sizeintegerWill not attempt to read files larger than this100MB
max-line-sizeintegerWill not attempt to read lines longer than this1MB

Fields

path

Type: path

A path with wildcards identifying files to be read

data-dir

Type: path

Default: the platform-specific data directory for the user executing the pipe

The directory where checkpoints are written

encoding

Type: encoding

Default: UTF-8

The text encoding for the monitored files

include

Type: array of paths

Any other paths to be included

exclude

Type: array of paths

Paths to be excluded

ignore-older-than

Type: integer

Ignore files older than this (epoch time in seconds)

glob-minimum-cooldown

Type: integer
Alias: timeout
Default: 300ms

Pause after scanning for new files (in milliseconds)

stop-reading-after

Type: bool

Default: false

Do not wait for new files

ignore-line-breaks

Type: bool

Default: false

Consume each file as one event

remove-after

Type: integer

Remove files that are done reading after this period (seconds)

fingerprinting

Type: Fingerprinter

Default: DevInode

File fingerprinting strategy

Field NameTypeDescriptionDefault
checksumFingerprinterChecksumFingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes-
first-line-checksumFingerprinterFirstLineChecksumFingerprint strategy that uses the first line of the file to detect if a file has changed-
dev-inodeFingerprinterDevInodeFingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated-

  checksum

Type: FingerprinterChecksum

Fingerprint strategy is set to a checksum of the start of the file determined by the bytes that are skipped (ignored-header-bytes) and then the fingerprint size in bytes

Field NameTypeDescriptionDefault
fingerprint-bytesintegerThe number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified-
ignored-header-bytesintegerThe number of bytes from the start of the file to ignore before starting the fingerprint-

  fingerprint-bytes

Type: integer
Alias: fingerprint-bytes

The number of bytes to use for the fingerprint, starts after ignore-header-bytes if specified

  ignored-header-bytes

Type: integer
Alias: ignored-header-bytes

The number of bytes from the start of the file to ignore before starting the fingerprint

  first-line-checksum

Type: FingerprinterFirstLineChecksum

Fingerprint strategy that uses the first line of the file to detect if a file has changed

Field NameTypeDescriptionDefault
max-line-lengthintegerThe maximum number of bytes to read from the first line of the file to calculate the fingerprint-

  max-line-length

Type: integer
Alias: max-line-length

The maximum number of bytes to read from the first line of the file to calculate the fingerprint

  dev-inode

Type: FingerprinterDevInode

Fingerprint strategy is set to the inode of the file causing the file monitor to detect new files when a file is recreated

oldest-first

Type: bool

Default: false

Default is youngest first

file-path-field

Type: field

Fill this field with the field we are currently reading from

file-basename

Type: bool

Default: false

The file-path-field path will be the basename (e.g. '/path/frodo.txt' becomes just 'frodo')

start-at-beginning

Type: bool

Default: false

Ignore stored checkpoints and re-read all specified files

raw

Type: bool

Default: false

Disable 'quoting' of lines with _raw

max-read-size

Type: integer

Default: 100MB

Will not attempt to read files larger than this

max-line-size

Type: integer

Default: 1MB

Will not attempt to read lines longer than this