Skip to main content
Version: 3.4.0

enrich

Allows using CSV lookup to enrich data

This step requires special attention, as it needs to have its files attached to the pipe that will use the table lookup mechanisms. This is convenient as updated tables can be bundled with your pipes and managed centrally.

Please see the discussion and the example provided at the end of the Table Lookup section.

Example: Add known network ports to events

file: lookup.csv

port,service
22,ssh
80,http
443,https

input:

{"port":22}
{"port":80}
{"port":100}
{"port":443}

action:

enrich:
lookup-file: lookup.csv
match:
- type: num
event-field: port
lookup-field: port
add:
event-field: service
lookup-field: service

output:

{"port":22,"service":"ssh"}
{"port":80,"service":"http"}
{"port":100}
{"port":443,"service":"https"}

Field NameDescriptionTypeDefault
conditionOnly run this action if the condition the specified condition is metexpression-
lookup-fileA file containing lookupspath-
dynamicDo not panic if lookup-file does not immediately exist at pipe creation timeboolfalse
addDetail on what to add to the event, based on the match.

If there is no default value, then the output field will not be added to the event. | AddEnum | - | |match| Detail on what to match on, associating event fields and lookup fields | Match | - |

condition

Only run this action if the condition the specified condition is met

Type: expression

lookup-file

A file containing lookups

Type: path

dynamic

Do not panic if lookup-file does not immediately exist at pipe creation time

Available from Hotrod: 2.8

Type: bool

add

Detail on what to add to the event, based on the match. If there is no default value, then the output field will not be added to the event.

Type: AddEnum

Field NameDescriptionTypeDefault
addA field value to add to the eventAdd-
add-multipleField values to add to the eventAddMultiple-

add

A field value to add to the event

Type: Add

Field NameDescriptionTypeDefault
event-fieldField name to be added to the eventfield-
lookup-fieldField (CSV header) to lookup data to be place in event-fieldfield-
default-valueYAML formatted default value if the event is emptyfield-

event-field

Field name to be added to the event

Type: field

lookup-field

Field (CSV header) to lookup data to be place in event-field

Type: field

default-value

YAML formatted default value if the event is empty

Type: field

Example: type=cidr

file: lookup7.csv

office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24

input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

action:

enrich:
lookup-file: lookup7.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office
default-value: unknown

output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Panoptix"}

add-multiple

Field values to add to the event

Type: AddMultiple

Field NameDescriptionTypeDefault
event-fieldsAdd multiple fields to a single event based on a single match, providing a defaultarray of (field,value) pairs-

event-fields

Add multiple fields to a single event based on a single match, providing a default

The limitation with this shortcut is that the lookup field name should be the same as the event field

Type: array of (field,value) pairs

Example: type=cidr

file: lookup8.csv

office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24

input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

action:

enrich:
lookup-file: lookup8.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-fields:
- office: unknown

output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Panoptix"}

match

Detail on what to match on, associating event fields and lookup fields

Type: Match

Field NameDescriptionTypeDefault
typeType of match, one of the following: str, num, cidr, ip, num-range, num-list, str-liststring-
event-fieldEvent field matchedfield-
lookup-fieldLookup field matchedfield-

type

Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list

Type: string

Example: matching strings

file: lookup1.csv

port,service
22,ssh
80,http
443,https

input:

{"service":"ssh"}
{"service":"http"}
{"service":"unknown"}
{"service":"https"}

action:

enrich:
lookup-file: lookup1.csv
match:
- type: str
event-field: service
lookup-field: service
add:
event-field: port
lookup-field: port

output:

{"service":"ssh","port":"22"}
{"service":"http","port":"80"}
{"service":"unknown"}
{"service":"https","port":"443"}

Example: matching CIDR

file: lookup2.csv

office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24

input:

{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}

action:

enrich:
lookup-file: lookup2.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office

output:

{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100","office":"Panoptix"}

Example: IP address (if event match.event-field does not match ip address format, the event will be filtered out)

file: lookup3.csv

fqdn,address
panoptix.io,206.189.28.194

input:

{"address":"192.168.26.10"}
{"address":"206.189.28.194"}

action:

enrich:
lookup-file: lookup3.csv
match:
- type: ip
event-field: address
lookup-field: address
add:
event-field: fqdn
lookup-field: fqdn

output:

{"address":"192.168.26.10"}
{"address":"206.189.28.194","fqdn":"panoptix.io"}

Example: Type is a range of numbers

file: lookup4.csv

range,grouping
0-3,small numbers
3-10,larger numbers

input:

{"number":1}
{"number":10}
{"number":100}

action:

enrich:
lookup-file: lookup4.csv
match:
- type: num-range
event-field: number
lookup-field: range
add:
event-field: grouping
lookup-field: grouping

output:

{"number":1,"grouping":"small numbers"}
{"number":10,"grouping":"larger numbers"}
{"number":100}

Example: Type is a list of strings

file: lookup5.csv

lists,grouping
"zero,two,four",even
"one,three,five",odd

input:

{"number":"one"}
{"number":"two"}
{"number":"three"}

action:

enrich:
lookup-file: lookup5.csv
match:
- type: str-list
event-field: number
lookup-field: lists
add:
event-field: grouping
lookup-field: grouping

output:

{"number":"one","grouping":"odd"}
{"number":"two","grouping":"even"}
{"number":"three","grouping":"odd"}

Example: CDIR with multiple matches

file: lookup6.csv

source,destination,label
192.168.26.0/24,192.168.26.0/24,sameSide
192.168.85.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,0.0.0.0/0,outbound
192.168.85.0/24,0.0.0.0/0,outbound
0.0.0.0/0,192.168.26.0/24,inbound
0.0.0.0/0,192.168.85.0/24,inbound
0.0.0.0/0,0.0.0.0/0,unknown

input:

{"src":"192.168.26.10","dst":"192.168.26.11"}
{"src":"192.168.26.10","dst":"192.168.85.10"}
{"src":"192.168.85.10","dst":"192.168.85.11"}
{"src":"192.168.26.10","dst":"192.168.86.10"}
{"src":"192.168.86.10","dst":"192.168.26.10"}
{"src":"192.168.86.10","dst":"192.168.86.11"}

action:

enrich:
lookup-file: lookup6.csv
match:
- type: cidr
event-field: src
lookup-field: source
- type: cidr
event-field: dst
lookup-field: destination
add:
event-field: traffic-direction
lookup-field: label

output:

{"src":"192.168.26.10","dst":"192.168.26.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.85.10","traffic-direction":"sameSide"}
{"src":"192.168.85.10","dst":"192.168.85.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.86.10","traffic-direction":"outbound"}
{"src":"192.168.86.10","dst":"192.168.26.10","traffic-direction":"inbound"}
{"src":"192.168.86.10","dst":"192.168.86.11","traffic-direction":"unknown"}

event-field

Event field matched

Type: field

lookup-field

Lookup field matched

Type: field