Skip to main content
Version: 3.4.0

expand

Converts simple separated data into JSON

This is called expand because it pulls JSON data (values, key-value pairs or JSON text) from a field.

Field NameDescriptionTypeDefault
conditionOnly run this action if the condition the specified condition is metexpression-
delimThe delimiter (separator) to be usedstring,
warningIf set to true warnigns will no longer be ignoredboolfalse
input-fieldField containing datafield_raw
document-modeWith 'expand array' collapse input-field between doc-start and doc-end into one text eventboolfalse
removeWhether to remove field after its value is usedboolfalse
csvConverts CSV rows into JSON recordsCsv-
key-valueConverts data in key=value formatKeyValue-
eventsConverts a single event into multiple eventsEvents-
xmlExpand a field containing an XML document to a JSON representationXmlEnum-
multilineCollect lines into fieldsarray of (field,regex) pairs-
jsonTreat incoming data as already in JSON formatboolfalse

condition

Only run this action if the condition the specified condition is met

Type: expression

delim

The delimiter (separator) to be used

Type: string

Example

input:

{"row":"1 2"}

action:

expand:
delim: " "
input-field: row
csv:
fields:
- one: num
- two: str

output:

{"row":"1 2","one":1,"two":"2"}

warning

If set to true warnigns will no longer be ignored

Type: bool

input-field

Field containing data

Type: field

Example

input:

{"good":"fast,safe","bad":"slow,dangerous"}

action:

expand:
input-field: good
remove: true
csv:
fields:
- speed: str
- trust: str

output:

{"bad":"slow,dangerous","speed":"fast","trust":"safe"}

document-mode

With 'expand array' collapse input-field between doc-start and doc-end into one text event

Type: bool

remove

Whether to remove field after its value is used

Type: bool

Example

input:

{"row":"1 2"}

action:

expand:
delim: " "
remove: true
input-field: row
csv:
fields:
- one: num
- two: str

output:

{"one":1,"two":"2"}

csv

Converts CSV rows into JSON records

Type: Csv

Field NameDescriptionTypeDefault
relaxed-schemaAllow picking just the first given number fields, and giving those namesboolfalse
headerWhether to treat first line as a CSV headerboolfalse
gen-headersWhether to generate automatic CSV headersboolfalse
autoconvertControl auto-conversion when reading in headersbooltrue
fieldsSpecify fields and their types (str, num, bool, num)array of key-value pairs-
field-fileFile containing "name:type" pairs. Specify fields and their types (str, num, bool, num)path-
header-fieldField containing header (CSV column names)field-
header-field-typesField containing header has types specified (with name:type format)boolfalse
header-field-on-changeWith 'header-field`, only write out headers if columns changeboolfalse
null-valueA substitue string value to be used in the event that a field is nullstring-

relaxed-schema

Allow picking just the first given number fields, and giving those names

Type: bool

Example

input:

{"row":"1,2,3,4,5"}

action:

expand:
input-field: row
remove: true
csv:
relaxed-schema: true
fields:
- first: num
- second: str

output:

{"first":1,"second":"2"}

Whether to treat first line as a CSV header

Type: bool

Example

input:

text,number
one,1
two,2

action:

expand:
input-field: _raw
remove: true
csv:
header: true

output:

{"text":"one","number":1}
{"text":"two","number":2}

gen-headers

Whether to generate automatic CSV headers

Type: bool

Example

input:

one,1

action:

expand:
input-field: _raw
remove: true
csv:
gen-headers: true

output:

{"_0":"one","_1":1}

autoconvert

Control auto-conversion when reading in headers

Type: bool

fields

Specify fields and their types (str, num, bool, num)

Type: array of key-value pairs

Example

input:

{"data":"one,1"}

action:

expand:
input-field: data
csv:
fields:
- text: str
- number: num

output:

{"data":"one,1","text":"one","number":1}

field-file

File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)

Type: path

Example

file: some-file

text:str
number:num

input:

{"data":"one,1"}

action:

expand:
input-field: data
csv:
field-file: some-file

output:

{"data":"one,1","text":"one","number":1}

header-field

Field containing header (CSV column names)

Type: field

Example

input:

{"header":"text,number","row":"one,1"}

action:

expand:
input-field: row
remove: true
csv:
header-field: header

output:

{"text":"one","number":1}

header-field-types

Field containing header has types specified (with name:type format)

Type: bool

Example

input:

{"header":"text:str,number:str","row":"one,1"}

action:

expand:
input-field: row
remove: true
csv:
header-field: header
header-field-types: true

output:

{"text":"one","number":"1"}

header-field-on-change

With 'header-field`, only write out headers if columns change

Type: bool

null-value

A substitue string value to be used in the event that a field is null

Type: string

key-value

Converts data in key=value format

Type: KeyValue

Example: Something basic

input:

one=1 two=2

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
key-value-delim: '='

output:

{"one":"1","two":"2"}

Example: Quoted strings

input:

male="john doe" female="jane doe"

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
key-value-delim: '='

output:

{"male":"john doe","female":"jane doe"}
Field NameDescriptionTypeDefault
autoconvertAutomatically convert to numbers if possibleboolfalse
key-value-delimSet the delimiter between the key and the valuestring=
multipleIn the case of repeated, choose whether to choose first, last, or to place all in an arraystring-

autoconvert

Automatically convert to numbers if possible

Type: bool

Example

input:

one=1 two=2.0

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
autoconvert: true

output:

{"one":1,"two":2}

key-value-delim

Set the delimiter between the key and the value

Type: string

Example

input:

one:1 two:2

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
key-value-delim: ':'

output:

{"one":"1","two":"2"}

multiple

In the case of repeated, choose whether to choose first, last, or to place all in an array

Type: string

Example: first

input:

name=john name=jane

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
multiple: first

output:

{"name":"john"}

Example: last

input:

name=john name=jane

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
multiple: last

output:

{"name":"jane"}

Example: array

input:

name=john name=jane

action:

expand:
input-field: _raw
remove: true
delim: ' '
key-value:
multiple: array

output:

{"name":["john","jane"]}

events

Converts a single event into multiple events

Type: Events

Field NameDescriptionTypeDefault
output-split-fieldA field where resulting split data is to be placedfield-
skip-listAn array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"array of JSON Pointers-

output-split-field

A field where resulting split data is to be placed

Type: field

Example

input:

{"names":"john jane","family":"doe"}

action:

expand:
input-field: names
remove: true
delim: ' '
events:
output-split-field: name

output:

{"family":"doe","name":"john"}
{"family":"doe","name":"jane"}

skip-list

An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"

Type: array of JSON Pointers

Example

input:

{ "array": [{"name": "jane","family":"doe" }, { "name":"john","family":"doe" }] }

action:

expand:
events:
skip-list: []

output:

{"array":{"name":"jane","family":"doe"}}
{"array":{"name":"john","family":"doe"}}

xml

Expand a field containing an XML document to a JSON representation

Type: XmlEnum

Field NameDescriptionTypeDefault
yesSimply expand the field with no further optionsboolfalse
fieldsExpand the field with additional optionsXml-

yes

Simply expand the field with no further options

Type: bool

fields

Expand the field with additional options

Type: Xml

Field NameDescriptionTypeDefault
arraysList of fields in an xml payload to be expanded into separate eventsarray of strings-

arrays

List of fields in an xml payload to be expanded into separate events

Type: array of strings

multiline

Collect lines into fields

Currently the patterns allowed are '\n' (line end) and '\n\n' (blank line)

Type: array of (field,regex) pairs

Example: collecting pairs of lines into fields x and y

input:

one 1
one 2
two 1
two 2

action:

expand:
input-field: _raw
multiline:
- x: '(\n)'
- y: '(\n)'

output:

{"x":"one 1","y":"one 2"}
{"x":"two 1","y":"two 2"}

Example: map first line to x and rest to y up to empty line

input:

one 1
one 2
one 3
one 4

two 1
two 2

action:

expand:
input-field: _raw
delim: "\n"
multiline:
- x: "(\n)"
- y: "(\n\n)"

output:

{"x":"one 1","y":"one 2\none 3\none 4"}
{"x":"two 1","y":"two 2"}

json

Treat incoming data as already in JSON format

Type: bool

Example

input:

{"row":"{\"one\":1,\"two\":2}"}

action:

expand:
input-field: row
remove: true
json: true

output:

{"one":1,"two":2}