Version: Next

Action: enrich

Allows using CSV lookup to enrich data

This step requires special attention, as it needs to have its files attached to the pipe that will use the table lookup mechanisms. This is convenient as updated tables can be bundled with your pipes and managed centrally.

Please see the discussion and the example provided at the end of the Table Lookup section.

Example: Add known network ports to events

File: lookup.csv

port,service
22,ssh
80,http
443,https

Input:

{"port":22}
{"port":80}
{"port":100}
{"port":443}

Pipe Language Snippet:

enrich:
  lookup-file: lookup.csv
  match:
    - type: num
      event-field: port
      lookup-field: port
  add:
    event-field: service
    lookup-field: service

Output:

{"port":22,"service":"ssh"}
{"port":80,"service":"http"}
{"port":100}
{"port":443,"service":"https"}

Field Summary

Field Name	Type	Description	Default
condition	expression	Only run this action if the condition the specified condition is met	-
lookup-file	path	A file containing lookups	-
dynamic	bool	Do not panic if `lookup-file` does not immediately exist at pipe creation time	false
add	AddEnum	Detail on what to add to the event, based on the match.

Fields

`condition`

Type: expression

Only run this action if the condition the specified condition is met

`lookup-file`

Type: path

A file containing lookups

`dynamic`

Type: bool

Default: false

Do not panic if lookup-file does not immediately exist at pipe creation time

`add`

Type: AddEnum

Detail on what to add to the event, based on the match. If there is no default value, then the output field will not be added to the event.

Field Name	Type	Description	Default
add	Add	A field value to add to the event	-
add-multiple	AddMultiple	Field values to add to the event	-

`add`

Type: Add

A field value to add to the event

Field Name	Type	Description	Default
event-field	field	Field name to be added to the event	-
lookup-field	field	Field (CSV header) to lookup data to be place in event-field	-
default-value	field	YAML formatted default value if the event is empty	-

`event-field`

Type: field

Field name to be added to the event

`lookup-field`

Type: field

Field (CSV header) to lookup data to be place in event-field

`default-value`

Type: field

YAML formatted default value if the event is empty

Example: type=cidr

File: lookup7.csv

office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24

Input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

Pipe Language Snippet:

enrich:
  lookup-file: lookup7.csv
  match:
    - type: cidr
      event-field: address
      lookup-field: network
  add:
    event-field: office
    lookup-field: office
    default-value: unknown

Output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Headquarters"}

`add-multiple`

Type: AddMultiple

Field values to add to the event

Field Name	Type	Description	Default
event-fields	array of (field,value) pairs	Add multiple fields to a single event based on a single match, providing a default	-

`event-fields`

Type: array of (field,value) pairs

Add multiple fields to a single event based on a single match, providing a default

The limitation with this shortcut is that the lookup field name should be the same as the event field

Example: type=cidr

File: lookup8.csv

office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24

Input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

Pipe Language Snippet:

enrich:
  lookup-file: lookup8.csv
  match:
    - type: cidr
      event-field: address
      lookup-field: network
  add:
    event-fields:
      - office: unknown

Output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Headquarters"}

`match`

Type: Match

Detail on what to match on, associating event fields and lookup fields

Field Name	Type	Description	Default
type	string	Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list	-
event-field	field	Event field matched	-
lookup-field	field	Lookup field matched	-

`type`

Type: string

Possible Values: str, num, cidr, ip, num-range, num-list, str-list

Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list

Example: matching strings

File: lookup1.csv

port,service
22,ssh
80,http
443,https

Input:

{"service":"ssh"}
{"service":"http"}
{"service":"unknown"}
{"service":"https"}

Pipe Language Snippet:

enrich:
    lookup-file: lookup1.csv
    match:
      - type: str
        event-field: service
        lookup-field: service
    add:
      event-field: port
      lookup-field: port

Output:

{"service":"ssh","port":"22"}
{"service":"http","port":"80"}
{"service":"unknown"}
{"service":"https","port":"443"}

Example: matching CIDR

File: lookup2.csv

office,network
Customer,192.168.26.0/28
Headquarters,192.168.85.0/24

Input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

Pipe Language Snippet:

enrich:
    lookup-file: lookup2.csv
    match:
      - type: cidr
        event-field: address
        lookup-field: network
    add:
      event-field: office
      lookup-field: office

Output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100","office":"Headquarters"}

Example: IP address (if event match.event-field does not match ip address format, the event will be filtered out)

File: lookup3.csv

fqdn,address
domain.io,206.189.28.194

Input:

{"address":"192.168.26.10"}
{"address":"206.189.28.194"}

Pipe Language Snippet:

enrich:
    lookup-file: lookup3.csv
    match:
      - type: ip
        event-field: address
        lookup-field: address
    add:
      event-field: fqdn
      lookup-field: fqdn

Output:

{"address":"192.168.26.10"}
{"address":"206.189.28.194","fqdn":"domain.io"}

Example: Type is a range of numbers

File: lookup4.csv

range,grouping
0-3,small numbers
3-10,larger numbers

Input:

{"number":1}
{"number":10}
{"number":100}

Pipe Language Snippet:

enrich:
    lookup-file: lookup4.csv
    match:
      - type: num-range
        event-field: number
        lookup-field: range
    add:
      event-field: grouping
      lookup-field: grouping

Output:

{"number":1,"grouping":"small numbers"}
{"number":10,"grouping":"larger numbers"}
{"number":100}

Example: Type is a list of strings

File: lookup5.csv

lists,grouping
"zero,two,four",even
"one,three,five",odd

Input:

{"number":"one"}
{"number":"two"}
{"number":"three"}

Pipe Language Snippet:

enrich:
    lookup-file: lookup5.csv
    match:
      - type: str-list
        event-field: number
        lookup-field: lists
    add:
      event-field: grouping
      lookup-field: grouping

Output:

{"number":"one","grouping":"odd"}
{"number":"two","grouping":"even"}
{"number":"three","grouping":"odd"}

Example: CDIR with multiple matches

File: lookup6.csv

source,destination,label
168.26.0/24,192.168.26.0/24,sameSide
168.85.0/24,192.168.85.0/24,sameSide
168.26.0/24,192.168.85.0/24,sameSide
168.26.0/24,0.0.0.0/0,outbound
168.85.0/24,0.0.0.0/0,outbound
0.0.0/0,192.168.26.0/24,inbound
0.0.0/0,192.168.85.0/24,inbound
0.0.0/0,0.0.0.0/0,unknown

Input:

{"src":"192.168.26.10","dst":"192.168.26.11"}
{"src":"192.168.26.10","dst":"192.168.85.10"}
{"src":"192.168.85.10","dst":"192.168.85.11"}
{"src":"192.168.26.10","dst":"192.168.86.10"}
{"src":"192.168.86.10","dst":"192.168.26.10"}
{"src":"192.168.86.10","dst":"192.168.86.11"}

Pipe Language Snippet:

enrich:
    lookup-file: lookup6.csv
    match:
     - type: cidr
       event-field: src
       lookup-field: source
     - type: cidr
       event-field: dst
       lookup-field: destination
    add:
      event-field: traffic-direction
      lookup-field: label

Output:

{"src":"192.168.26.10","dst":"192.168.26.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.85.10","traffic-direction":"sameSide"}
{"src":"192.168.85.10","dst":"192.168.85.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.86.10","traffic-direction":"outbound"}
{"src":"192.168.86.10","dst":"192.168.26.10","traffic-direction":"inbound"}
{"src":"192.168.86.10","dst":"192.168.86.11","traffic-direction":"unknown"}

`event-field`

Type: field

Event field matched

`lookup-field`

Type: field

Lookup field matched

Action: enrich

Field Summary​

Fields​

condition​

lookup-file​

dynamic​

add​

add​

event-field​

lookup-field​

default-value​

add-multiple​

event-fields​

match​

type​

event-field​

lookup-field​

Field Summary

Fields

`condition`

`lookup-file`

`dynamic`

`add`

`add`

`event-field`

`lookup-field`

`default-value`

`add-multiple`

`event-fields`

`match`

`type`

`event-field`

`lookup-field`