Skip to main content
Version: 3.3.0

raw

Operations on raw (non-JSON) data

Field NameDescriptionTypeDefault
conditionOnly run this action if the condition the specified condition is metexpression-
to-jsonWrap plain text as a JSON fieldfield-
extractExtract data from plain text, using a patternPatExtract-
replaceReplace data from plain text, using a patternPatReplace-
discard-untilA pattern that must match before data starts to be processedDiscardUtilEnum-
multilineCombine lines into multi-line eventsMultiline-

condition

Only run this action if the condition the specified condition is met

Type: expression

to-json

Wrap plain text as a JSON field

Type: field

Example

input:

some text

action:

raw:
to-json: some-field

output:

{"some-field":"some text"}

extract

Extract data from plain text, using a pattern

Type: PatExtract

Field NameDescriptionTypeDefault
patternThe pattern to match onregex-
replaceReplacement textstring-
input-fieldField containing datafield-

pattern

The pattern to match on

If replace is not provided, the first match is the output

Note: If there is no match, then we discard the event

Type: regex

Example

input:

num=1
num=2
num=3

action:

raw:
extract:
pattern: 'num=(\d+)'

output:

1
2
3

replace

Replacement text

If pattern is not provided, the whole input is still available as $0.

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

Example

input:

num=1
num=2
num=3

action:

raw:
extract:
pattern: 'num=(\d+)'
replace: we got $1

output:

we got 1
we got 2
we got 3

input-field

Field containing data

Note: In this case, the input must be JSON

Type: field

Example

input:

{"one":"1"}

action:

raw:
extract:
input-field: one
replace: we got $0

output:

{"one":"we got 1"}

replace

Replace data from plain text, using a pattern

The difference between this and extract is that it will never fail

Type: PatReplace

Field NameDescriptionTypeDefault
patternThe pattern to match onregex-
substitutionReplacement textstring-
input-fieldField containing datafield-

pattern

The pattern to match on

Type: regex

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
replace:
pattern: one
substitution: we got $0

output:

{"num":"1"}
{"text":"we got one"}

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
replace:
pattern: (\d+)
substitution: prefix of $1

output:

{"num":"prefix of 1"}
{"text":"one"}

substitution

Replacement text

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

input-field

Field containing data

Type: field

Example

input:

{"one":"hey 1"}

action:

raw:
replace:
input-field: one
pattern: 'hey (\d+)'
substitution: we got $1

output:

{"one":"we got 1"}

discard-until

A pattern that must match before data starts to be processed

All data before then will be discarded. An example of where this is useful, is where the data would start with comments. Use begin-marker-field to reset

Type: DiscardUtilEnum

Example

input:

# some comment
# some other comment
num=1
num=2
num=3

action:

raw:
discard-until: '^num'

output:

num=1
num=2
num=3

Example

input:

{"a":"# comment","first":true}
{"a":"num=1"}
{"a":"num=2"}
{"a":"# comment","first":true}
{"a":"# another"}
{"a":"num=10"}
{"a":"num=11"}

action:

raw:
discard-until:
pattern: '^[^#]'
input-field: a
begin-marker-field: first

output:

{"a":"num=1"}
{"a":"num=2"}
{"a":"num=10"}
{"a":"num=11"}

Field NameDescriptionTypeDefault
stringDiscard until a string is matched against the eventstring-
discard-untilDiscard until a pattern, marker (optionally in a specific field is matched)DiscardUntil-

string

Discard until a string is matched against the event

Type: string

discard-until

Discard until a pattern, marker (optionally in a specific field is matched)

Type: DiscardUntil

Field NameDescriptionTypeDefault
input-fieldUse the specified field to match againstfield-
begin-marker-fieldUse the specified marker as the match conditionfield-
patternUse the specified regex pattern as the match conditionregex-

input-field

Use the specified field to match against

Type: field

begin-marker-field

Use the specified marker as the match condition

Type: field

pattern

Use the specified regex pattern as the match condition

Type: regex

multiline

Combine lines into multi-line events

Similar to how Logstash does it, except that 'pattern' is used instead of 'matches' and 'merge' is used instead of 'what'; we say 'not-matches' instead of 'pattern' plus 'negate'

Type: Multiline

Field NameDescriptionTypeDefault
input-fieldOptional field containing line, otherwise use whole linefield-
mergeDirection to try mergestring-
delimHow to join lines of an event togetherstring\n
matchesThe line must match this patternregex-
not-matchesThe line must NOT match this patternregex-

input-field

Optional field containing line, otherwise use whole line

Type: field

merge

Direction to try merge

Type: string

Example: keep merging matching lines with previous

input:

hello dolly
foo this gets merged
foo this as well
hello jane
foo that gets merged
foo that as well

action:

raw:
multiline:
matches: '^foo'
merge: previous
delim: '|'

output:

hello dolly|foo this gets merged|foo this as well
hello jane|foo that gets merged|foo that as well

delim

How to join lines of an event together

Type: string

matches

The line must match this pattern

Type: regex

Example: merge matching lines with next line

input:

foo again
this gets merged
this does not <---- what happens to stray lines?
foo two
we get this second line
foo three
we get this third line

action:

raw:
multiline:
matches: '^foo'
merge: next
delim: '|'

output:

foo again|this gets merged
this does not <---- what happens to stray lines?
foo two|we get this second line
foo three|we get this third line

not-matches

The line must NOT match this pattern

Must either have 'matches' or 'not-matches'

Type: regex

Example: merge not-matching lines with previous line

input:

FOO header line
a merged line
another merged line
FOO header line 2
a merged line 2
another merged line 2
FOO header line 3
a merged line 3
another merged line 3

action:

raw:
multiline:
not-matches: '^FOO'
merge: previous
delim: '|'

output:

FOO header line|a merged line|another merged line
FOO header line 2|a merged line 2|another merged line 2