Scripting in Pipes

script, like add is mostly used to add new fields to events (enrichment). The difference is that script can calculate fields and can conditionally add them.

These expressions are in Lua format, e.g conditions like a > 2 and b < 1 and expressions like 2*a - 1. All event fields are available in these expressions. If the data is not 'flat' then nested fields are accessed with a period and arrays are indexed with [] E.g if our event is {"a":{"b":1},"c":[10,20,30]}) then the value of 1 here can be accessed with a.b. Array values are accessed starting at one so c[1] == 10.

NOTE For this to work field names must be valid Lua variables - start with a letter, and contain only letters, digits, and underscores. This restriction also applies to extract so we recommend this for all field names.

You can use regular Lua keywords like function or end as field names, except for true,false,nil,or,and.

let versus set

With let, you give a list of variable-expression pairs, but with set you provide constants. Context expansions are always available.

# input: {"a":10}
- script:
    let:
    - c: 2*a - 1
    set:
    - site: '{{name}}'
# output: {"a":10,"c":19,"site":"NAME"}

script set is essentially what add output-fields does, except with add you can add structured data like arrays and objects.

Why set? Because setting constants with let is more tricky, for example - name: '"hello"' - notice the double quotes!

Both script and add will not override existing fields by default. Force this with override: true

There may be a condition field - the condition must be true before any fields are written.

The 'pseudo-field' _E is special and refers to the whole event, so this will make a copy of the event and put it into the res field:

- script:
    let:
    - res: _E

Extra Functions

• round(x) returns the nearest integer to a floating point number, like round(tmillis/1000). Useful for converting bytes to kB, milliseconds since epoch to seconds since epoch, etc.
• count() counter since pipe start
• random(n) a random integer between 1 and n
• pick_random(...) returns one of multiple arguments randomly
• sum(acc,val,cond) and avg(acc,val,cond) running sum and average of value val.
• sec_s() will return seconds since epoch, sec_ms() milliseconds since epoch.
• cidr(addr, spec) will match an IPv4 network address against a CIDR specification like '10.0.0.0/24'.
• ip2asn uses the Team Cymru services to match IP addresses to domain owner.
• cond(condition, value1, value2) is a useful function that will return value1 if condition is true, otherwise returns value2. E.g. status: cond(istat > 0,"ok","error").
• condn(...) is like cond for multiple conditions and values
• array(...) makes its arguments into an array
• len(v) length of array or number of characters in text
• map(...) makes an object
• hashes:
  • md5(txt)
  • sha1(txt)
  • sha256(txt)
  • sha512(txt)
• uuid() returns a Unique Identifier each time
• encrypt(txt,key) encrypt text using AES-128-CBC with a key and encode as Base64
• decrypt(txt,key) decrypt result of encrypt using same key
• emit(v) write out data directly

NOTE Any field with one of these names will overwrite the function!

Examples

The first argument of sum is the 'accumulator' where we keep the count and can be any name that isn't a field. (We have this because there may be more than one field being summed.)

(Note that filter is the only other place where Lua expressions occur.)

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: counter % 5 == 0
# output
    {"counter":5,"sumi":15}
    {"counter":10,"sumi":55}
    {"counter":15,"sumi":120}
    {"counter":20,"sumi":210}
    ....

sum has an extra argument - if present, it will collect while it is true:

# input:
{"val":10,"ok":true}
{"val":10,"ok":true}
{"val":5,"ok":false}
{"val":50,"ok":false}    
{"val":10,"ok":true}
{"val":10,"ok":true}
  - script:
      let:
      - s: sum("s",val,ok)
# output
{"val":10,"ok":true,"s":10}
{"val":10,"ok":true,"s":20}
{"val":5,"ok":false,"s":5}
{"val":50,"ok":false,"s":50}
{"val":10,"ok":true,"s":10}
{"val":10,"ok":true,"s":20}

map and array are a convenient way to build 'deep' objects.

# input
{"a":1,"b":2}
  - script:
      let:
      - res: |
          map (
            "one",a+b,
            "two",a-b,
            "three",a*b,
            "four",array(a,b)
          )
# output
{"a":1,"b":2,"res":{"four":[1,2],"three":2,"two":-1,"one":3}}

You will notice that order is not preserved with Lua objects!

condn is useful for more complicated conditions. You can use it for simple lookups.

# input
{"a":1}
{"a":2}
{"a":3}
  - script:
      let:
      - size: |
          condn(
            a < 2, "small",
            a == 2,"medium",
            a > 2, "large"
          )
# output
{"a":1,"size":"small"}
{"a":2,"size":"medium"}
{"a":3,"size":"large"}            

Useful Standard Lua Functions

Processing Text

• string.upper(txt) convert text to upper-case
• string.lower(txt) convert text to lower-case
• string.sub(txt,istart,iend) 'substring' of text
• string.find(txt,patt) find if a pattern matches
• table.concat(txt,sep) join an array as text

name: upper
input:
  text: '{"msg":"hello dolly","arr":[1,2,3],"num":3.1423}'
actions:
- script:
    overwrite: true
    let:
    - msg: string.upper(msg)
    - first: string.sub(msg,1,4)
    - n: string.len(msg)
    - arr: table.concat(arr," ")
    - num: string.format("%.02f",num)
output:
  write: console
# {"msg":"HELLO DOLLY","arr":"1.0 2.0 3.0","num":"3.14","first":"HELL","n":11}

If you are comfortable with Lua, then saying - msg: 'msg:upper()' works just as well, but we do need to use quotes because YAML gets confused by values containing colons.

Notice that you can join strings together ('concatenate') using the .. operator as usual:

# input
{"first":"jonas","second":"whale"}
actions:
- script:
    let:
    - full_name: first .. " " .. second
# output
{"first":"jonas","second":"whale","full_name":"jonas whale"}    

Mathematical Functions

• math.ceil(x) nearest greater integer (e.g. 1.4 -> 2)
• math.floor(x) nearest smaller integer (e.g. 1.4 -> 1)
• math.min(...) smallest of multiple values
• math.max(...) largest of multiple values
• math.log(x,b) log of x with base b
• math.exp(x) exponential
• math.sin, math.cos ... trigonometric functions (in radians)

See the Lua manual for the full list of available functions.

NOTE Some functionality has been removed for safe sandboxing. The global functions require, dofile and load are not present.

Extending script using Lua

If there is a file init.lua in the pipe directory, it will be loaded into the Lua state. Any global functions or variables become available:

-- init.lua
    function every(n)
       return count() % n == 0
    end

Then the counter example can be made improved as below:

- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: every(5)

init.lua will be loaded automatically, but you have to copy it together with the pipe. A complete pipe could look like this:

name: sumi
files:
- init.lua
input:
    exec:
        command: echo foo
        interval: 100ms
actions:
- remove: [_raw]
- script:
    let:
    - counter: count()
    - sumi: sum("s",counter)
- filter:
    condition: every(5)
output:
    write: console

A more interesting example involves unpacking the result of a Prometheus query. The task is to take a structured JSON document and create events for each retrieved data point - expanding:

{
    "status":"success",
    "data":{
        "resultType":"matrix",
        "result":[
            {
                "metric":{
                    "__name__":"probe_success",
                    "instance":"178.62.49.144:20000",
                    "job":"APP-F4ZASD-JD-STE-RU-3033"},
                    "values":[[1571997720,"0"],[1571997780,"0"],...]
                }
            },
            ...
        ]
    }
}

init.lua contains a function which is passed the whole document and uses the emit function to directly write out events:

function unpack_prom(result)
    for _,res in ipairs(result) do
        local t = res.metric
        for _,pair in ipairs(res.values) do
            t.time = pair[1]
            t.value = pair[2]
            emit(t)
        end        
    end
    return 'ok'    
end

And this pipe exercises it:

- script:
    let:
    - a: unpack_prom(data.result)
- remove: [status,data]
# output
{"instance":"178.62.49.144:20000","__name__":"probe_success","time":1571997720,"value":"0","job":"APP-F4ZASD-JD-STE-RU-3033"}
....

Here we are interested in the side effect of the function.