raw

Operations on raw (non-JSON) data

Field Name	Description	Type	Default
condition	Only run this action if the condition the specified condition is met	expression	-
to-json	Wrap plain text as a JSON field	field	-
extract	Extract data from plain text, using a pattern	PatExtract	-
replace	Replace data from plain text, using a pattern	PatReplace	-
discard-until	A pattern that must match before data starts to be processed	DiscardUtilEnum	-
multiline	Combine lines into multi-line events	Multiline	-

condition

Only run this action if the condition the specified condition is met

Type: expression

to-json

Wrap plain text as a JSON field

Type: field

Example

input:

some text

action:

raw:
  to-json: some-field

output:

{"some-field":"some text"}

extract

Extract data from plain text, using a pattern

Type: PatExtract

Field Name	Description	Type	Default
pattern	The pattern to match on	regex	-
replace	Replacement text	string	-
input-field	Field containing data	field	-

pattern

The pattern to match on

If replace is not provided, the first match is the output

Note: If there is no match, then we discard the event

Type: regex

Example

input:

num=1
num=2
num=3

action:

raw:
  extract:
    pattern: 'num=(\d+)'

output:

1
2
3

replace

Replacement text

If pattern is not provided, the whole input is still available as $0.

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

Example

input:

num=1
num=2
num=3

action:

raw:
  extract:
    pattern: 'num=(\d+)'
    replace: we got $1

output:

we got 1
we got 2
we got 3

input-field

Field containing data

Note: In this case, the input must be JSON

Type: field

Example

input:

{"one":"1"}

action:

raw:
  extract:
    input-field: one
    replace: we got $0

output:

{"one":"we got 1"}

replace

Replace data from plain text, using a pattern

The difference between this and extract is that it will never fail

Type: PatReplace

Field Name	Description	Type	Default
pattern	The pattern to match on	regex	-
substitution	Replacement text	string	-
input-field	Field containing data	field	-

pattern

The pattern to match on

Type: regex

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
  replace:
    pattern: one
    substitution: we got $0

output:

{"num":"1"}
{"text":"we got one"}

Example

input:

{"num":"1"}
{"text":"one"}

action:

raw:
  replace:
    pattern: (\d+)
    substitution: prefix of $1

output:

{"num":"prefix of 1"}
{"text":"one"}

substitution

Replacement text

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Type: string

input-field

Field containing data

Type: field

Example

input:

{"one":"hey 1"}

action:

raw:
  replace:
    input-field: one
    pattern: 'hey (\d+)'
    substitution: we got $1

output:

{"one":"we got 1"}

discard-until

A pattern that must match before data starts to be processed

All data before then will be discarded. An example of where this is useful, is where the data would start with comments. Use begin-marker-field to reset

Type: DiscardUtilEnum

Example

input:

# some comment
# some other comment
num=1
num=2
num=3

action:

raw:
  discard-until: '^num'

output:

num=1
num=2
num=3

Example

input:

{"a":"# comment","first":true}
{"a":"num=1"}
{"a":"num=2"}
{"a":"# comment","first":true}
{"a":"# another"}                  
{"a":"num=10"}
{"a":"num=11"}

action:

raw:
  discard-until:
    pattern: '^[^#]'
    input-field: a
    begin-marker-field: first

output:

{"a":"num=1"}
{"a":"num=2"}
{"a":"num=10"}
{"a":"num=11"}

Field Name	Description	Type	Default
string	Discard until a string is matched against the event	string	-
discard-until	Discard until a pattern, marker (optionally in a specific field is matched)	DiscardUntil	-

string

Discard until a string is matched against the event

Type: string

discard-until

Discard until a pattern, marker (optionally in a specific field is matched)

Type: DiscardUntil

Field Name	Description	Type	Default
input-field	Use the specified field to match against	field	-
begin-marker-field	Use the specified marker as the match condition	field	-
pattern	Use the specified regex pattern as the match condition	regex	-

input-field

Use the specified field to match against

Type: field

begin-marker-field

Use the specified marker as the match condition

Type: field

pattern

Use the specified regex pattern as the match condition

Type: regex

multiline

Combine lines into multi-line events

Similar to how Logstash does it, except that 'pattern' is used instead of 'matches' and 'merge' is used instead of 'what'; we say 'not-matches' instead of 'pattern' plus 'negate'

Type: Multiline

Field Name	Description	Type	Default
input-field	Optional field containing line, otherwise use whole line	field	-
merge	Direction to try merge	string	-
delim	How to join lines of an event together	string	\n
matches	The line must match this pattern	regex	-
not-matches	The line must NOT match this pattern	regex	-

input-field

Optional field containing line, otherwise use whole line

Type: field

merge

Direction to try merge

Type: string

Example: keep merging matching lines with previous

input:

hello dolly
foo this gets merged
foo this as well
hello jane
foo that gets merged
foo that as well

action:

raw:
  multiline:
    matches: '^foo'
    merge: previous
    delim: '|'

output:

hello dolly|foo this gets merged|foo this as well
hello jane|foo that gets merged|foo that as well

delim

How to join lines of an event together

Type: string

matches

The line must match this pattern

Type: regex

Example: merge matching lines with next line

input:

foo again
this gets merged
this does not      <---- what happens to stray lines?
foo two
we get this second line
foo three
we get this third line

action:

raw:
  multiline:
    matches: '^foo'
    merge: next
    delim: '|'

output:

foo again|this gets merged
this does not      <---- what happens to stray lines?
foo two|we get this second line
foo three|we get this third line

not-matches

The line must NOT match this pattern

Must either have 'matches' or 'not-matches'

Type: regex

Example: merge not-matching lines with previous line

input:

FOO header line
a merged line
another merged line
FOO header line 2
a merged line 2
another merged line 2
FOO header line 3
a merged line 3
another merged line 3

action:

raw:
  multiline:
    not-matches: '^FOO'
    merge: previous
    delim: '|'

output:

FOO header line|a merged line|another merged line
FOO header line 2|a merged line 2|another merged line 2