HTTP Inputs and Outputs

Hotrod has several inputs and outputs that make it convenient to integrate with other servers over HTTP.

http-server input

This provides a HTTP server, which can be used to define a webhook. There is both an input and an output variant; both of them run in the background listening for HTTP GET or POST requests.

By default every request to this endpoint results in a JSON event as below:

• method: the HTTP method (GET,POST,etc)
• url: the full URL
• query: the query parameters (e.g. "?p=foo&q=boo") as an object ({"p":foo,"q":boo"})
• headers: the HTTP headers as an object
• body: the received body

The only required field is address. Running curl http://localhost:3030 results in:

input:
    http-server:
        address: 0.0.0.0:3030
# {"body":"","method":"GET","query":{},"url":"/","headers":{"accept":"*/*","host":"localhost:3030","user-agent":"curl/7.58.0"}}        

If only the body is needed, set only-body: true. This will be quoted with _raw unless json-body: true.

This server supports secure HTTPS connections. cert is path to TLS certificate and key is path to TLS key.

It's possible to return a fixed response with any request using custom-response, and optionally specifying its type with content-type.

http-server cannot be an action because it is not internally scheduled; we wait for external events.

http-poll input

http-poll does a HTTP(S) GET or a POST request. The only required field is the address. Like with exec the response is quoted with _raw and can be passed through directly with raw: true. Using headers and query full URIs can be built up with ease. You may provide a body.

Send request as POST with method: post.

Polling the server above as follows:

input:
    http-poll:
        address: http://localhost:3030/bar
        interval: 2s
        query:
        - foo: hello
        headers:
        - content-type: text/plain

And its response will be:

{"body":"","method":"GET","query":{"foo":"hello"},"url":"/bar","headers":{"accept":"*/*","accept-encoding":"gzip","content-type":"text/plain","host":"localhost:3030","user-agent":"reqwest/0.9.24"}}

This is a scheduled input, so schedule variables are available in address and query as field expansions.

It is available as an action under input. As in input, dollar-curly field expansions are possible, and you can use this to enrich data.

http-server output

http-server is very similar to the input form, except that it is fed events by the pipe. When an outside agent queries it, it will get the last event that was generated by the pipe. So there is no history: if it is polled at a slower rate than the pipe, then data will be lost.

For instance, this pipe executes the 'uptime' command repeatedly and extracts the last number (the 5-minute load average).

name: server
input:
    exec:
        command: uptime
        interval: 5s
actions:
- extract:
    remove: true
    pattern: '(\S+)$'
    output-fields: [min5]
output:
    http-server:
        address: 0.0.0.0:3030
        content-type: application/json
        path: /uptime

We can now use curl to scrape the server:

$ curl -v http://localhost:3030/uptime
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3030 (#0)
> GET /uptime HTTP/1.1
> Host: localhost:3030
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< transfer-encoding: chunked
< date: Wed, 15 May 2019 06:48:33 GMT
<
* Connection #0 to host localhost left intact
{"min5":"0.49"}

path is optional - without it, it will serve the same content with any URL. Likewise, content-type can be optionally specified.

You can GET the output as often as you like, but if you scrape faster than 5s in this case, you will get the last value created by the pipe.

By default, the body returned is the whole event generated by the pipe. body allows you to specify exactly what is returned. Here the body is the value of the _raw field.

http-server:
    address: 0.0.0.0:3030
    body: '${_raw}'

body can contain any field expansions.

As with the input, you can set TLS certificate and key with

    cert: /path/to/my.cert
    key: /path/to/my.key

http-post output

http-post is the output equivalent to http-poll; the pipe's events are sent as HTTP(s) requests to a server.

batch is the maximum number of events in each POST request; if we wait for more than timeout, then the request is posted with less than this number. Note that timeout is a interval ('100ms', '1s', '1m'). The default is 100ms.

This allows maximum throughput and ensuring minimum waiting on data.

output:
    http-post:
        batch: 50
        timeout: 300ms
        url: http://whatever:port/parms

Without the batch, it does a single unbuffered POST request, and some useful expansions become possible. By default, the body of the POST is the whole record, but can be provided with body-field which is a field name to be used instead (just like input-field with exec.)

The url can contain field expansions. These will be properly URL-encoded if needed.

name: poster
input:
    text: '{"msg":"hello dolly","path":"/foo", "query":"new dawn", "lines": "/one/two"}'
output:
    http-post:
        body-field: msg
        url: 'http://localhost:3030${path}?q=${query}&p=${lines}'

You may add HTTP headers:

input:
    http-post:
        url: http://localhost:3030
        headers:
        - content-type: text/plain

Without headers, the content-type defaults to application/json.

Running as an action under output allows sending data to multiple outputs. As an action, the original data will be passed through unchanged.