Action: raw

Operations on raw (non-JSON) data

Field Summary

Field Name	Type	Description	Default
condition	expression	Only run this action if the condition the specified condition is met	-
to-json	field	Wrap plain text as a JSON field	-
extract	PatExtract	Extract data from plain text, using a pattern	-
replace	PatReplace	Replace data from plain text, using a pattern	-
discard-until	DiscardUtilEnum	A pattern that must match before data starts to be processed	-
multiline	Multiline	Combine lines into multi-line events	-

Fields

condition

Type: expression

Only run this action if the condition the specified condition is met

to-json

Type: field

Wrap plain text as a JSON field

Example

Input:

some text

Pipe Language Snippet:

raw:
  to-json: some-field

Output:

{"some-field":"some text"}

extract

Type: PatExtract

Extract data from plain text, using a pattern

Field Name	Type	Description	Default
pattern	regex	The pattern to match on	-
replace	string	Replacement text	-
input-field	field	Field containing data	-

  pattern

Type: regex

The pattern to match on

If replace is not provided, the first match is the output

Example

Input:

num=1
num=2
num=3

Pipe Language Snippet:

raw:
  extract:
    pattern: 'num=(\d+)'

Output:

1
2
3

  replace

Type: string

Replacement text

If pattern is not provided, the whole input is still available as $0.

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Example

Input:

num=1
num=2
num=3

Pipe Language Snippet:

raw:
  extract:
    pattern: 'num=(\d+)'
    replace: we got $1

Output:

we got 1
we got 2
we got 3

  input-field

Type: field

Field containing data

Example

Input:

{"one":"1"}

Pipe Language Snippet:

raw:
  extract:
    input-field: one
    replace: we got $0

Output:

{"one":"we got 1"}

replace

Type: PatReplace

Replace data from plain text, using a pattern

The difference between this and extract is that it will never fail

Field Name	Type	Description	Default
pattern	regex	The pattern to match on	-
substitution	string	Replacement text	-
input-field	field	Field containing data	-

  pattern

Type: regex

The pattern to match on

Example

Input:

{"num":"1"}
{"text":"one"}

Pipe Language Snippet:

raw:
  replace:
    pattern: one
    substitution: we got $0

Output:

{"num":"1"}
{"text":"we got one"}

Example

Input:

{"num":"1"}
{"text":"one"}

Pipe Language Snippet:

raw:
  replace:
    pattern: (\d+)
    substitution: prefix of $1

Output:

{"num":"prefix of 1"}
{"text":"one"}

  substitution

Type: string

Replacement text

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

  input-field

Type: field

Field containing data

Example

Input:

{"one":"hey 1"}

Pipe Language Snippet:

raw:
  replace:
    input-field: one
    pattern: 'hey (\d+)'
    substitution: we got $1

Output:

{"one":"we got 1"}

discard-until

Type: DiscardUtilEnum

A pattern that must match before data starts to be processed

All data before then will be discarded. An example of where this is useful, is where the data would start with comments. Use begin-marker-field to reset

Example

Input:

# some comment
# some other comment
num=1
num=2
num=3

Pipe Language Snippet:

raw:
  discard-until: '^num'

Output:

num=1
num=2
num=3

Example

Input:

{"a":"# comment","first":true}
{"a":"num=1"}
{"a":"num=2"}
{"a":"# comment","first":true}
{"a":"# another"}                  
{"a":"num=10"}
{"a":"num=11"}

Pipe Language Snippet:

raw:
  discard-until:
    pattern: '^[^#]'
    input-field: a
    begin-marker-field: first

Output:

{"a":"num=1"}
{"a":"num=2"}
{"a":"num=10"}
{"a":"num=11"}

Field Name	Type	Description	Default
string	string	Discard until a string is matched against the event	-
discard-until	DiscardUntil	Discard until a pattern, marker (optionally in a specific field is matched)	-

  string

Type: string

Discard until a string is matched against the event

  discard-until

Type: DiscardUntil

Discard until a pattern, marker (optionally in a specific field is matched)

Field Name	Type	Description	Default
input-field	field	Use the specified field to match against	-
begin-marker-field	field	Use the specified marker as the match condition	-
pattern	regex	Use the specified regex pattern as the match condition	-

  input-field

Type: field

Use the specified field to match against

  begin-marker-field

Type: field

Use the specified marker as the match condition

  pattern

Type: regex

Use the specified regex pattern as the match condition

multiline

Type: Multiline

Combine lines into multi-line events

Similar to how Logstash does it, except that 'pattern' is used instead of 'matches' and 'merge' is used instead of 'what'; we say 'not-matches' instead of 'pattern' plus 'negate'

Field Name	Type	Description	Default
input-field	field	Optional field containing line, otherwise use whole line	-
merge	string	Direction to try merge	-
delim	string	How to join lines of an event together	\n
matches	regex	The line must match this pattern	-
not-matches	regex	The line must NOT match this pattern	-

  input-field

Type: field

Optional field containing line, otherwise use whole line

  merge

Type: string

Possible Values: previous, next

Direction to try merge

Example: keep merging matching lines with previous

Input:

hello dolly
foo this gets merged
foo this as well
hello jane
foo that gets merged
foo that as well

Pipe Language Snippet:

raw:
  multiline:
    matches: '^foo'
    merge: previous
    delim: '|'

Output:

hello dolly|foo this gets merged|foo this as well
hello jane|foo that gets merged|foo that as well

  delim

Type: string

Default: \n

How to join lines of an event together

  matches

Type: regex

The line must match this pattern

Example: merge matching lines with next line

Input:

foo again
this gets merged
this does not      <---- what happens to stray lines?
foo two
we get this second line
foo three
we get this third line

Pipe Language Snippet:

raw:
  multiline:
    matches: '^foo'
    merge: next
    delim: '|'

Output:

foo again|this gets merged
this does not      <---- what happens to stray lines?
foo two|we get this second line
foo three|we get this third line

  not-matches

Type: regex

The line must NOT match this pattern

Must either have 'matches' or 'not-matches'

Example: merge not-matching lines with previous line

Input:

FOO header line
a merged line
another merged line
FOO header line 2
a merged line 2
another merged line 2
FOO header line 3
a merged line 3
another merged line 3

Pipe Language Snippet:

raw:
  multiline:
    not-matches: '^FOO'
    merge: previous
    delim: '|'

Output:

FOO header line|a merged line|another merged line
FOO header line 2|a merged line 2|another merged line 2