expand

Converts simple separated data into JSON

This is called expand because it pulls JSON data (values, key-value pairs or JSON text) from a field.

Field Name	Description	Type	Default
condition	Only run this action if the condition the specified condition is met	expression	-
delim	The delimiter (separator) to be used	string	,
warning	If set to true warnigns will no longer be ignored	bool	false
input-field	Field containing data	field	_raw
document-mode	With 'expand array' collapse input-field between doc-start and doc-end into one text event	bool	false
remove	Whether to remove field after its value is used	bool	false
csv	Converts CSV rows into JSON records	Csv	-
key-value	Converts data in key=value format	KeyValue	-
events	Converts a single event into multiple events	Events	-
xml	Expand a field containing an XML document to a JSON representation	XmlEnum	-
multiline	Collect lines into fields	array of (field,regex) pairs	-
json	Treat incoming data as already in JSON format	bool	false

condition

Only run this action if the condition the specified condition is met

Type: expression

delim

The delimiter (separator) to be used

Type: string

Example

input:

{"row":"1 2"}

action:

expand:
  delim: " "
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

output:

{"row":"1 2","one":1,"two":"2"}

warning

If set to true warnigns will no longer be ignored

Type: bool

input-field

Field containing data

Type: field

Example

input:

{"good":"fast,safe","bad":"slow,dangerous"}

action:

expand:
  input-field: good
  remove: true
  csv:
    fields:
    - speed: str
    - trust: str

output:

{"bad":"slow,dangerous","speed":"fast","trust":"safe"}

document-mode

With 'expand array' collapse input-field between doc-start and doc-end into one text event

Type: bool

remove

Whether to remove field after its value is used

Type: bool

Example

input:

{"row":"1 2"}

action:

expand:
  delim: " "
  remove: true
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

output:

{"one":1,"two":"2"}

csv

Converts CSV rows into JSON records

Type: Csv

Field Name	Description	Type	Default
relaxed-schema	Allow picking just the first given number fields, and giving those names	bool	false
header	Whether to treat first line as a CSV header	bool	false
gen-headers	Whether to generate automatic CSV headers	bool	false
autoconvert	Control auto-conversion when reading in headers	bool	true
fields	Specify fields and their types (str, num, bool, num)	array of key-value pairs	-
field-file	File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)	path	-
header-field	Field containing header (CSV column names)	field	-
header-field-types	Field containing header has types specified (with name:type format)	bool	false
header-field-on-change	With 'header-field`, only write out headers if columns change	bool	false
null-value	A substitue string value to be used in the event that a field is null	string	-

relaxed-schema

Allow picking just the first given number fields, and giving those names

Type: bool

Example

input:

{"row":"1,2,3,4,5"}

action:

expand:
  input-field: row
  remove: true
  csv:
    relaxed-schema: true
    fields:
      - first: num
      - second: str

output:

{"first":1,"second":"2"}

header

Whether to treat first line as a CSV header

Type: bool

Example

input:

text,number
one,1
two,2

action:

expand:
  input-field: _raw
  remove: true
  csv:
    header: true

output:

{"text":"one","number":1}
{"text":"two","number":2}

gen-headers

Whether to generate automatic CSV headers

Type: bool

Example

input:

one,1

action:

expand:
  input-field: _raw
  remove: true
  csv:
    gen-headers: true

output:

{"_0":"one","_1":1}

autoconvert

Control auto-conversion when reading in headers

Type: bool

fields

Specify fields and their types (str, num, bool, num)

Type: array of key-value pairs

Example

input:

{"data":"one,1"}

action:

expand:
  input-field: data
  csv:
    fields:
      - text: str
      - number: num

output:

{"data":"one,1","text":"one","number":1}

field-file

File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)

Type: path

Example

file: some-file

text:str
number:num

input:

{"data":"one,1"}

action:

expand:
  input-field: data
  csv:
    field-file: some-file

output:

{"data":"one,1","text":"one","number":1}

header-field

Field containing header (CSV column names)

Type: field

Example

input:

{"header":"text,number","row":"one,1"}

action:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header

output:

{"text":"one","number":1}

header-field-types

Field containing header has types specified (with name:type format)

Type: bool

Example

input:

{"header":"text:str,number:str","row":"one,1"}

action:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header
    header-field-types: true

output:

{"text":"one","number":"1"}

header-field-on-change

With 'header-field`, only write out headers if columns change

Type: bool

null-value

A substitue string value to be used in the event that a field is null

Type: string

key-value

Converts data in key=value format

Type: KeyValue

Example: Something basic

input:

one=1 two=2

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

output:

{"one":"1","two":"2"}

Example: Quoted strings

input:

male="john doe" female="jane doe"

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

output:

{"male":"john doe","female":"jane doe"}

Field Name	Description	Type	Default
autoconvert	Automatically convert to numbers if possible	bool	false
key-value-delim	Set the delimiter between the key and the value	string	=
multiple	In the case of repeated, choose whether to choose first, last, or to place all in an array	string	-

autoconvert

Automatically convert to numbers if possible

Type: bool

Example

input:

one=1 two=2.0

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    autoconvert: true

output:

{"one":1,"two":2}

key-value-delim

Set the delimiter between the key and the value

Type: string

Example

input:

one:1 two:2

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: ':'

output:

{"one":"1","two":"2"}

multiple

In the case of repeated, choose whether to choose first, last, or to place all in an array

Type: string

Example: first

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: first

output:

{"name":"john"}

Example: last

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: last

output:

{"name":"jane"}

Example: array

input:

name=john name=jane

action:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: array

output:

{"name":["john","jane"]}

events

Converts a single event into multiple events

Type: Events

Field Name	Description	Type	Default
output-split-field	A field where resulting split data is to be placed	field	-
skip-list	An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"	array of JSON Pointers	-

output-split-field

A field where resulting split data is to be placed

Type: field

Example

input:

{"names":"john jane","family":"doe"}

action:

expand:
  input-field: names
  remove: true
  delim: ' '
  events:
    output-split-field: name

output:

{"family":"doe","name":"john"}
{"family":"doe","name":"jane"}

skip-list

An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"

Type: array of JSON Pointers

Example

input:

{ "array": [{"name": "jane","family":"doe" }, { "name":"john","family":"doe" }] }

action:

expand:
  events:
    skip-list: []

output:

{"array":{"name":"jane","family":"doe"}}
{"array":{"name":"john","family":"doe"}}

xml

Expand a field containing an XML document to a JSON representation

Type: XmlEnum

Field Name	Description	Type	Default
yes	Simply expand the field with no further options	bool	false
fields	Expand the field with additional options	Xml	-

yes

Simply expand the field with no further options

Type: bool

fields

Expand the field with additional options

Type: Xml

Field Name	Description	Type	Default
arrays	List of fields in an xml payload to be expanded into separate events	array of strings	-

arrays

List of fields in an xml payload to be expanded into separate events

Type: array of strings

multiline

Collect lines into fields

Currently the patterns allowed are '\n' (line end) and '\n\n' (blank line)

Type: array of (field,regex) pairs

Example: collecting pairs of lines into fields x and y

input:

one 1
one 2
two 1
two 2

action:

expand:
  input-field: _raw
  multiline:
    - x: '(\n)'
    - y: '(\n)'

output:

{"x":"one 1","y":"one 2"}
{"x":"two 1","y":"two 2"}

Example: map first line to x and rest to y up to empty line

input:

one 1
one 2
one 3
one 4

two 1
two 2

action:

expand:
  input-field: _raw
  delim: "\n"
  multiline:
    - x: "(\n)"
    - y: "(\n\n)"

output:

{"x":"one 1","y":"one 2\none 3\none 4"}
{"x":"two 1","y":"two 2"}

json

Treat incoming data as already in JSON format

Type: bool

Example

input:

{"row":"{\"one\":1,\"two\":2}"}

action:

expand:
  input-field: row
  remove: true
  json: true

output:

{"one":1,"two":2}