Action: expand

Converts simple separated data into JSON

This is called expand because it pulls JSON data (values, key-value pairs or JSON text) from a field.

Field Summary

Field Name	Type	Description	Default
condition	expression	Only run this action if the condition the specified condition is met	-
delim	string	The delimiter (separator) to be used	,
warning	bool	If set to true warnigns will no longer be ignored	false
input-field	field	Field containing data	_raw
document-mode	bool	With 'expand array' collapse input-field between doc-start and doc-end into one text event	false
remove	bool	Whether to remove field after its value is used	false
csv	Csv	Converts CSV rows into JSON records	-
key-value	KeyValue	Converts data in key=value format	-
events	Events	Converts a single event into multiple events	-
xml	XmlEnum	Expand a field containing an XML document to a JSON representation	-
multiline	array of (field,regex) pairs	Collect lines into fields	-
json	bool	Treat incoming data as already in JSON format	false

Fields

condition

Type: expression

Only run this action if the condition the specified condition is met

delim

Type: string

Default: ,

The delimiter (separator) to be used

Example

Input:

{"row":"1 2"}

Pipe Language Snippet:

expand:
  delim: " "
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

Output:

{"row":"1 2","one":1,"two":"2"}

warning

Type: bool

Default: false

If set to true warnigns will no longer be ignored

input-field

Type: field

Default: _raw

Field containing data

Example

Input:

{"good":"fast,safe","bad":"slow,dangerous"}

Pipe Language Snippet:

expand:
  input-field: good
  remove: true
  csv:
    fields:
    - speed: str
    - trust: str

Output:

{"bad":"slow,dangerous","speed":"fast","trust":"safe"}

document-mode

Type: bool

Default: false

With 'expand array' collapse input-field between doc-start and doc-end into one text event

remove

Type: bool
Alias: remove-field
Default: false

Whether to remove field after its value is used

Example

Input:

{"row":"1 2"}

Pipe Language Snippet:

expand:
  delim: " "
  remove: true
  input-field: row
  csv:
    fields:
    - one: num
    - two: str

Output:

{"one":1,"two":"2"}

csv

Type: Csv

Converts CSV rows into JSON records

Field Name	Type	Description	Default
relaxed-schema	bool	Allow picking just the first given number fields, and giving those names	false
header	bool	Whether to treat first line as a CSV header	false
gen-headers	bool	Whether to generate automatic CSV headers	false
autoconvert	bool	Control auto-conversion when reading in headers	true
fields	array of key-value pairs	Specify fields and their types (str, num, bool, num)	-
field-file	path	File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)	-
header-field	field	Field containing header (CSV column names)	-
header-field-types	bool	Field containing header has types specified (with name:type format)	false
header-field-on-change	bool	With 'header-field`, only write out headers if columns change	false
null-value	string	A substitue string value to be used in the event that a field is null	-

  relaxed-schema

Type: bool

Default: false

Allow picking just the first given number fields, and giving those names

Example

Input:

{"row":"1,2,3,4,5"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    relaxed-schema: true
    fields:
      - first: num
      - second: str

Output:

{"first":1,"second":"2"}

  header

Type: bool

Default: false

Whether to treat first line as a CSV header

Example

Input:

text,number
one,1
two,2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  csv:
    header: true

Output:

{"text":"one","number":1}
{"text":"two","number":2}

  gen-headers

Type: bool

Default: false

Whether to generate automatic CSV headers

Example

Input:

one,1

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  csv:
    gen-headers: true

Output:

{"_0":"one","_1":1}

  autoconvert

Type: bool

Default: true

Control auto-conversion when reading in headers

  fields

Type: array of key-value pairs

Specify fields and their types (str, num, bool, num)

Example

Input:

{"data":"one,1"}

Pipe Language Snippet:

expand:
  input-field: data
  csv:
    fields:
      - text: str
      - number: num

Output:

{"data":"one,1","text":"one","number":1}

  field-file

Type: path

File containing "name:type" pairs. Specify fields and their types (str, num, bool, num)

Example

File: some-file

text:str
number:num

Input:

{"data":"one,1"}

Pipe Language Snippet:

expand:
  input-field: data
  csv:
    field-file: some-file

Output:

{"data":"one,1","text":"one","number":1}

  header-field

Type: field

Field containing header (CSV column names)

Example

Input:

{"header":"text,number","row":"one,1"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header

Output:

{"text":"one","number":1}

  header-field-types

Type: bool

Default: false

Field containing header has types specified (with name:type format)

Example

Input:

{"header":"text:str,number:str","row":"one,1"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  csv:
    header-field: header
    header-field-types: true

Output:

{"text":"one","number":"1"}

  header-field-on-change

Type: bool

Default: false

With 'header-field`, only write out headers if columns change

  null-value

Type: string

A substitue string value to be used in the event that a field is null

key-value

Type: KeyValue

Converts data in key=value format

Example: Something basic

Input:

one=1 two=2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

Output:

{"one":"1","two":"2"}

Example: Quoted strings

Input:

male="john doe" female="jane doe"

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: '='

Output:

{"male":"john doe","female":"jane doe"}

Field Name	Type	Description	Default
autoconvert	bool	Automatically convert to numbers if possible	false
key-value-delim	string	Set the delimiter between the key and the value	=
multiple	string	In the case of repeated, choose whether to choose first, last, or to place all in an array	-

  autoconvert

Type: bool

Default: false

Automatically convert to numbers if possible

Example

Input:

one=1 two=2.0

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    autoconvert: true

Output:

{"one":1,"two":2}

  key-value-delim

Type: string

Default: =

Set the delimiter between the key and the value

Example

Input:

one:1 two:2

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    key-value-delim: ':'

Output:

{"one":"1","two":"2"}

  multiple

Type: string

Possible Values: first, last, array

In the case of repeated, choose whether to choose first, last, or to place all in an array

Example: first

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: first

Output:

{"name":"john"}

Example: last

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: last

Output:

{"name":"jane"}

Example: array

Input:

name=john name=jane

Pipe Language Snippet:

expand:
  input-field: _raw
  remove: true
  delim: ' '
  key-value:
    multiple: array

Output:

{"name":["john","jane"]}

events

Type: Events

Converts a single event into multiple events

Field Name	Type	Description	Default
output-split-field	field	A field where resulting split data is to be placed	-
skip-list	array of JSON Pointers	An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"	-
exclude-non-empty-arrays	bool		false

  output-split-field

Type: field

A field where resulting split data is to be placed

Example

Input:

{"names":"john jane","family":"doe"}

Pipe Language Snippet:

expand:
  input-field: names
  remove: true
  delim: ' '
  events:
    output-split-field: name

Output:

{"family":"doe","name":"john"}
{"family":"doe","name":"jane"}

  skip-list

Type: array of JSON Pointers

An array (in JSON Pointer RFC6901 format) that should be skipped. For examle: "/array"

Example

Input:

{ "array": [{"name": "jane","family":"doe" }, { "name":"john","family":"doe" }] }

Pipe Language Snippet:

expand:
  events:
    skip-list: []

Output:

{"array":{"name":"jane","family":"doe"}}
{"array":{"name":"john","family":"doe"}}

  exclude-non-empty-arrays

Type: bool

Default: false

xml

Type: XmlEnum

Expand a field containing an XML document to a JSON representation

Field Name	Type	Description	Default
yes	bool	Simply expand the field with no further options	false
fields	Xml	Expand the field with additional options	-

  yes

Type: bool

Default: false

Simply expand the field with no further options

  fields

Type: Xml

Expand the field with additional options

Field Name	Type	Description	Default
arrays	array of strings	List of fields in an xml payload to be expanded into separate events	-

  arrays

Type: array of strings

List of fields in an xml payload to be expanded into separate events

multiline

Type: array of (field,regex) pairs

Collect lines into fields

Currently the patterns allowed are '\n' (line end) and '\n\n' (blank line)

Example: collecting pairs of lines into fields x and y

Input:

one 1
one 2
two 1
two 2

Pipe Language Snippet:

expand:
  input-field: _raw
  multiline:
    - x: '(\n)'
    - y: '(\n)'

Output:

{"x":"one 1","y":"one 2"}
{"x":"two 1","y":"two 2"}

Example: map first line to x and rest to y up to empty line

Input:

one 1
one 2
one 3
one 4

two 1
two 2

Pipe Language Snippet:

expand:
  input-field: _raw
  delim: "\n"
  multiline:
    - x: "(\n)"
    - y: "(\n\n)"

Output:

{"x":"one 1","y":"one 2\none 3\none 4"}
{"x":"two 1","y":"two 2"}

json

Type: bool

Default: false

Treat incoming data as already in JSON format

Example

Input:

{"row":"{\"one\":1,\"two\":2}"}

Pipe Language Snippet:

expand:
  input-field: row
  remove: true
  json: true

Output:

{"one":1,"two":2}