HTTP Inputs and Outputs

For convenient integration with other servers over HTTP, a Pipe features HTTP-related input and output.

http-server Input

This provides a HTTP server, used to define a webhook. Both the input and output variants run in the background listening for HTTP GET or POST requests.

By default, every request to this endpoint results in a JSON event as below:

• method: the HTTP method (GET, POST, etc.)
• url: the full URL
• query: the query parameters (e.g., "?p=foo&q=boo" as an object {"p":foo,"q":boo"})
• headers: the HTTP headers as an object
• body: the received body

The only required field is address. Running curl http://localhost:3030 results in:

input:
  http-server:
    address: 127.0.0.1:3030

# Output: {"body":"","method":"GET","query":{},"url":"/","headers":{"accept":"*/*","host":"localhost:3030","user-agent":"curl/7.58.0"}}        

If only the body is needed, set only-body: true, which will be quoted with _raw unless json-body: true is present.

The server supports HTTPS. cert is a path to a TLS certificate and key is a path to a TLS key.

custom-response returns a fixed response with any request, with the option to specify its type using content-type.

http-server cannot be an action because it is not internally scheduled, so instead we wait for external events.

http-poll Input

http-poll performs a HTTP(S) GET or POST request, with address as the only required field. As with exec, the response is quoted with _raw and can be passed through directly with raw: true. It is easy to construct a complete URI using headers and query and you can also provide a body.

Send requests as POST with method: post.

Polling the server above as follows:

input:
  http-poll:
    address: http://localhost:3030/bar
    interval: 2s
    query:
    - foo: hello
    headers:
    - content-type: text/plain

Its response will be:

{"body":"","method":"GET","query":{"foo":"hello"},"url":"/bar","headers":{"accept":"*/*","accept-encoding":"gzip","content-type":"text/plain","host":"localhost:3030","user-agent":"reqwest/0.9.24"}}

This is a scheduled input, so schedule variables are available in address and query as Field Expansion.

http-poll is available as an action under input. It is possible to use ${} Field Expansion to enrich data.

http-server Output

http-server output differs from the input form because events are received from a Pipe. When queried by an outside Agent, it will get the last event that was generated by the Pipe. This means there is no queue: if it is polled at a slower rate than the Pipe, then data will be lost.

The Pipe below executes the uptime command every 5s and extracts the last value (the 15-minute load average). The URL path is optional and it will serve the same content, regardless of URL if unspecified. Specifying content-type is also optional.

name: http_server

input:
  exec:
    command: uptime
    interval: 5s

actions:
- extract:
    remove: true
    pattern: '(\S+)$'
    output-fields: [min15]

output:
  http-server:
    address: 127.0.0.1:3030
    content-type: application/json
    path: /uptime

We can now use curl to GET the uptime:

$> curl -4v http://localhost:3030/uptime

*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3030 (#0)
> GET /uptime HTTP/1.1
> Host: localhost:3030
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< transfer-encoding: chunked
< date: Wed, 15 May 2019 06:48:33 GMT
<
* Connection #0 to host localhost left intact
{"min15":"0.49"}

You can GET the output as often as you like, however, if you scrape faster than 5s you will get the last value created by the Pipe.

By default, the body returned is the whole event generated by the Pipe but this can be specified using body. In this instance, body is the value of the _raw field.

http-server:
  address: 127.0.0.1:3030
  body: '${_raw}'

body can contain any Field Expansion.

For HTTPS, specify the TLS cert and key using:

  cert: /path/to/my.cert
  key: /path/to/my.key

http-post Output

http-post is the output equivalent to http-poll and the Pipe events are sent as HTTP(S) requests to a server.

batch is the maximum number of events in each POST request. If we wait for more than the timeout qualifier, then the request is posted with less than this number.

note

timeout is an interval (100ms, 1s, 1m) and the default is 100ms.

This allows for maximum throughput and ensures minimum time spent waiting on data.

output:
    http-post:
        batch: 50
        timeout: 300ms
        url: http://whatever:port/parms

Without batch, it performs a single unbuffered POST request, allowing for numerous useful expansions. By default, the body of the POST is the entire record, but this can be adjusted using the field name body-field (just like input-field with exec).

url contains Field Expansion which can be URL-encoded if needed.

name: poster

input:
    text: '{"msg":"hello dolly","path":"/foo", "query":"new dawn", "lines": "/one/two"}'

output:
    http-post:
        body-field: msg
        url: 'http://localhost:3030${path}?q=${query}&p=${lines}'

You may add HTTP headers:

input:
    http-post:
        url: http://localhost:3030
        headers:
        - content-type: text/plain

If headers are not added, the content-type defaults to application/json.

Running this as an action under output allows the data to be sent to multiple destinations. In doing so, the original data can pass through unchanged as an action.