Skip to main content
Version: 3.5.3

Action: raw

Operations on raw (non-JSON) data

Field Summary

Field NameTypeDescriptionDefault
conditionexpressionOnly run this action if the condition the specified condition is met-
to-jsonfieldWrap plain text as a JSON field-
extractPatExtractExtract data from plain text, using a pattern-
replacePatReplaceReplace data from plain text, using a pattern-
discard-untilDiscardUtilEnumA pattern that must match before data starts to be processed-
multilineMultilineCombine lines into multi-line events-

Fields

condition

Type: expression

Only run this action if the condition the specified condition is met

to-json

Type: field

Wrap plain text as a JSON field

Example

Input:

some text

Pipe Language Snippet:

raw:
to-json: some-field

Output:

{"some-field":"some text"}

extract

Type: PatExtract

Extract data from plain text, using a pattern

Field NameTypeDescriptionDefault
patternregexThe pattern to match on-
replacestringReplacement text-
input-fieldfieldField containing data-

  pattern

Type: regex

The pattern to match on

If replace is not provided, the first match is the output

Example

Input:

num=1
num=2
num=3

Pipe Language Snippet:

raw:
extract:
pattern: 'num=(\d+)'

Output:

1
2
3

  replace

Type: string

Replacement text

If pattern is not provided, the whole input is still available as $0.

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

Example

Input:

num=1
num=2
num=3

Pipe Language Snippet:

raw:
extract:
pattern: 'num=(\d+)'
replace: we got $1

Output:

we got 1
we got 2
we got 3

  input-field

Type: field

Field containing data

Example

Input:

{"one":"1"}

Pipe Language Snippet:

raw:
extract:
input-field: one
replace: we got $0

Output:

{"one":"we got 1"}

replace

Type: PatReplace

Replace data from plain text, using a pattern

The difference between this and extract is that it will never fail

Field NameTypeDescriptionDefault
patternregexThe pattern to match on-
substitutionstringReplacement text-
input-fieldfieldField containing data-

  pattern

Type: regex

The pattern to match on

Example

Input:

{"num":"1"}
{"text":"one"}

Pipe Language Snippet:

raw:
replace:
pattern: one
substitution: we got $0

Output:

{"num":"1"}
{"text":"we got one"}

Example

Input:

{"num":"1"}
{"text":"one"}

Pipe Language Snippet:

raw:
replace:
pattern: (\d+)
substitution: prefix of $1

Output:

{"num":"prefix of 1"}
{"text":"one"}

  substitution

Type: string

Replacement text

Note that $0 refers to entire text, $1 first regex group match, $2 second one, and so on...

  input-field

Type: field

Field containing data

Example

Input:

{"one":"hey 1"}

Pipe Language Snippet:

raw:
replace:
input-field: one
pattern: 'hey (\d+)'
substitution: we got $1

Output:

{"one":"we got 1"}

discard-until

Type: DiscardUtilEnum

A pattern that must match before data starts to be processed

All data before then will be discarded. An example of where this is useful, is where the data would start with comments. Use begin-marker-field to reset

Example

Input:

# some comment
# some other comment
num=1
num=2
num=3

Pipe Language Snippet:

raw:
discard-until: '^num'

Output:

num=1
num=2
num=3

Example

Input:

{"a":"# comment","first":true}
{"a":"num=1"}
{"a":"num=2"}
{"a":"# comment","first":true}
{"a":"# another"}
{"a":"num=10"}
{"a":"num=11"}

Pipe Language Snippet:

raw:
discard-until:
pattern: '^[^#]'
input-field: a
begin-marker-field: first

Output:

{"a":"num=1"}
{"a":"num=2"}
{"a":"num=10"}
{"a":"num=11"}

Field NameTypeDescriptionDefault
stringstringDiscard until a string is matched against the event-
discard-untilDiscardUntilDiscard until a pattern, marker (optionally in a specific field is matched)-

  string

Type: string

Discard until a string is matched against the event

  discard-until

Type: DiscardUntil

Discard until a pattern, marker (optionally in a specific field is matched)

Field NameTypeDescriptionDefault
input-fieldfieldUse the specified field to match against-
begin-marker-fieldfieldUse the specified marker as the match condition-
patternregexUse the specified regex pattern as the match condition-

  input-field

Type: field

Use the specified field to match against

  begin-marker-field

Type: field

Use the specified marker as the match condition

  pattern

Type: regex

Use the specified regex pattern as the match condition

multiline

Type: Multiline

Combine lines into multi-line events

Similar to how Logstash does it, except that 'pattern' is used instead of 'matches' and 'merge' is used instead of 'what'; we say 'not-matches' instead of 'pattern' plus 'negate'

Field NameTypeDescriptionDefault
input-fieldfieldOptional field containing line, otherwise use whole line-
mergestringDirection to try merge-
delimstringHow to join lines of an event together\n
matchesregexThe line must match this pattern-
not-matchesregexThe line must NOT match this pattern-

  input-field

Type: field

Optional field containing line, otherwise use whole line

  merge

Type: string

Possible Values: previous, next

Direction to try merge

Example: keep merging matching lines with previous

Input:

hello dolly
foo this gets merged
foo this as well
hello jane
foo that gets merged
foo that as well

Pipe Language Snippet:

raw:
multiline:
matches: '^foo'
merge: previous
delim: '|'

Output:

hello dolly|foo this gets merged|foo this as well
hello jane|foo that gets merged|foo that as well

  delim

Type: string

Default: \n

How to join lines of an event together

  matches

Type: regex

The line must match this pattern

Example: merge matching lines with next line

Input:

foo again
this gets merged
this does not <---- what happens to stray lines?
foo two
we get this second line
foo three
we get this third line

Pipe Language Snippet:

raw:
multiline:
matches: '^foo'
merge: next
delim: '|'

Output:

foo again|this gets merged
this does not <---- what happens to stray lines?
foo two|we get this second line
foo three|we get this third line

  not-matches

Type: regex

The line must NOT match this pattern

Must either have 'matches' or 'not-matches'

Example: merge not-matching lines with previous line

Input:

FOO header line
a merged line
another merged line
FOO header line 2
a merged line 2
another merged line 2
FOO header line 3
a merged line 3
another merged line 3

Pipe Language Snippet:

raw:
multiline:
not-matches: '^FOO'
merge: previous
delim: '|'

Output:

FOO header line|a merged line|another merged line
FOO header line 2|a merged line 2|another merged line 2