enrich
Allows using CSV lookup to enrich data
This step requires special attention, as it needs to have its files attached to the pipe that will use the table lookup mechanisms. This is convenient as updated tables can be bundled with your pipes and managed centrally.
Please see the discussion and the example provided
at the end of the Table Lookup section.
Example: Add known network ports to events
file: lookup.csv
port,service
22,ssh
80,http
443,https
input:
{"port":22}
{"port":80}
{"port":100}
{"port":443}
action:
enrich:
lookup-file: lookup.csv
match:
- type: num
event-field: port
lookup-field: port
add:
event-field: service
lookup-field: service
output:
{"port":22,"service":"ssh"}
{"port":80,"service":"http"}
{"port":100}
{"port":443,"service":"https"}
| Field Name | Description | Type | Default |
|---|---|---|---|
| condition | Only run this action if the condition the specified condition is met | expression | - |
| lookup-file | A file containing lookups | path | - |
| dynamic | Do not panic if lookup-file does not immediately exist at pipe creation time | bool | false |
| add | Detail on what to add to the event, based on the match. |
If there is no default value, then the output field will not be added to the event. | AddEnum | - | |match| Detail on what to match on, associating event fields and lookup fields | Match | - |
condition
Only run this action if the condition the specified condition is met
Type: expression
lookup-file
A file containing lookups
Type: path
dynamic
Do not panic if lookup-file does not immediately exist at pipe creation time
Available from Hotrod: 2.8
Type: bool
add
Detail on what to add to the event, based on the match. If there is no default value, then the output field will not be added to the event.
Type: AddEnum
| Field Name | Description | Type | Default |
|---|---|---|---|
| add | A field value to add to the event | Add | - |
| add-multiple | Field values to add to the event | AddMultiple | - |
add
A field value to add to the event
Type: Add
| Field Name | Description | Type | Default |
|---|---|---|---|
| event-field | Field name to be added to the event | field | - |
| lookup-field | Field (CSV header) to lookup data to be place in event-field | field | - |
| default-value | YAML formatted default value if the event is empty | field | - |
event-field
Field name to be added to the event
Type: field
lookup-field
Field (CSV header) to lookup data to be place in event-field
Type: field
default-value
YAML formatted default value if the event is empty
Type: field
Example: type=cidr
file: lookup7.csv
office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24
input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
action:
enrich:
lookup-file: lookup7.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office
default-value: unknown
output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Panoptix"}
add-multiple
Field values to add to the event
Type: AddMultiple
| Field Name | Description | Type | Default |
|---|---|---|---|
| event-fields | Add multiple fields to a single event based on a single match, providing a default | array of (field,value) pairs | - |
event-fields
Add multiple fields to a single event based on a single match, providing a default
The limitation with this shortcut is that the lookup field name should be the same as the event field
Type: array of (field,value) pairs
Example: type=cidr
file: lookup8.csv
office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24
input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
action:
enrich:
lookup-file: lookup8.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-fields:
- office: unknown
output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100","office":"unknown"}
{"address":"192.168.85.100","office":"Panoptix"}
match
Detail on what to match on, associating event fields and lookup fields
Type: Match
| Field Name | Description | Type | Default |
|---|---|---|---|
| type | Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list | string | - |
| event-field | Event field matched | field | - |
| lookup-field | Lookup field matched | field | - |
type
Type of match, one of the following: str, num, cidr, ip, num-range, num-list, str-list
Type: string
Example: matching strings
file: lookup1.csv
port,service
22,ssh
80,http
443,https
input:
{"service":"ssh"}
{"service":"http"}
{"service":"unknown"}
{"service":"https"}
action:
enrich:
lookup-file: lookup1.csv
match:
- type: str
event-field: service
lookup-field: service
add:
event-field: port
lookup-field: port
output:
{"service":"ssh","port":"22"}
{"service":"http","port":"80"}
{"service":"unknown"}
{"service":"https","port":"443"}
Example: matching CIDR
file: lookup2.csv
office,network
Customer,192.168.26.0/28
Panoptix,192.168.85.0/24
input:
{"address":"192.168.26.10"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100"}
action:
enrich:
lookup-file: lookup2.csv
match:
- type: cidr
event-field: address
lookup-field: network
add:
event-field: office
lookup-field: office
output:
{"address":"192.168.26.10","office":"Customer"}
{"address":"192.168.26.100"}
{"address":"192.168.85.100","office":"Panoptix"}
Example: IP address (if event match.event-field does not match ip address format, the event will be filtered out)
file: lookup3.csv
fqdn,address
panoptix.io,206.189.28.194
input:
{"address":"192.168.26.10"}
{"address":"206.189.28.194"}
action:
enrich:
lookup-file: lookup3.csv
match:
- type: ip
event-field: address
lookup-field: address
add:
event-field: fqdn
lookup-field: fqdn
output:
{"address":"192.168.26.10"}
{"address":"206.189.28.194","fqdn":"panoptix.io"}
Example: Type is a range of numbers
file: lookup4.csv
range,grouping
0-3,small numbers
3-10,larger numbers
input:
{"number":1}
{"number":10}
{"number":100}
action:
enrich:
lookup-file: lookup4.csv
match:
- type: num-range
event-field: number
lookup-field: range
add:
event-field: grouping
lookup-field: grouping
output:
{"number":1,"grouping":"small numbers"}
{"number":10,"grouping":"larger numbers"}
{"number":100}
Example: Type is a list of strings
file: lookup5.csv
lists,grouping
"zero,two,four",even
"one,three,five",odd
input:
{"number":"one"}
{"number":"two"}
{"number":"three"}
action:
enrich:
lookup-file: lookup5.csv
match:
- type: str-list
event-field: number
lookup-field: lists
add:
event-field: grouping
lookup-field: grouping
output:
{"number":"one","grouping":"odd"}
{"number":"two","grouping":"even"}
{"number":"three","grouping":"odd"}
Example: CDIR with multiple matches
file: lookup6.csv
source,destination,label
192.168.26.0/24,192.168.26.0/24,sameSide
192.168.85.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,192.168.85.0/24,sameSide
192.168.26.0/24,0.0.0.0/0,outbound
192.168.85.0/24,0.0.0.0/0,outbound
0.0.0.0/0,192.168.26.0/24,inbound
0.0.0.0/0,192.168.85.0/24,inbound
0.0.0.0/0,0.0.0.0/0,unknown
input:
{"src":"192.168.26.10","dst":"192.168.26.11"}
{"src":"192.168.26.10","dst":"192.168.85.10"}
{"src":"192.168.85.10","dst":"192.168.85.11"}
{"src":"192.168.26.10","dst":"192.168.86.10"}
{"src":"192.168.86.10","dst":"192.168.26.10"}
{"src":"192.168.86.10","dst":"192.168.86.11"}
action:
enrich:
lookup-file: lookup6.csv
match:
- type: cidr
event-field: src
lookup-field: source
- type: cidr
event-field: dst
lookup-field: destination
add:
event-field: traffic-direction
lookup-field: label
output:
{"src":"192.168.26.10","dst":"192.168.26.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.85.10","traffic-direction":"sameSide"}
{"src":"192.168.85.10","dst":"192.168.85.11","traffic-direction":"sameSide"}
{"src":"192.168.26.10","dst":"192.168.86.10","traffic-direction":"outbound"}
{"src":"192.168.86.10","dst":"192.168.26.10","traffic-direction":"inbound"}
{"src":"192.168.86.10","dst":"192.168.86.11","traffic-direction":"unknown"}
event-field
Event field matched
Type: field
lookup-field
Lookup field matched
Type: field