Version: Next

Input: http-poll

Run HTTP queries (GET and POST)

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: true

Output:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d24b6-4b01cb2acaf9e7a531b44d74"
  },
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/get"
}

Field Summary

Field Name	Type	Description	Default
when	message_filter	Fire this input when a specific internal message occurs	-
interval	duration	How often to run the command	-
cron	cron	How often to run the command. Note that unlike standard Cron, Pipes use a Cron syntax that includes a column for seconds. See full discussion	-
immediate	bool	Run as soon as invoked, instead of waiting for the specified cron interval	false
random-offset	duration	Sets a random offset to the schedule, then sticks to it	0s
window	Window	For resources that need a time window to be specified	-
block	bool	Block further input schedules from triggering if the pipe output is retrying	false
retry	Retry	For operations that could potentially fail	-
batch	Batch	For when a number of output events need to be marked as belonging to a distinct group	-
url	url	The URL for this request	-
raw	bool	Use for when data is text or already JSON	false
document-mode	bool	Collect all the output together as a single document (will affect batching on the output)	false
ignore-line-breaks	bool	Do not treat separate lines as distinct events	false
headers	array of maps	Headers to send with the query	-
query	array of maps	Query parameters to send with the query	-
auth	Auth	Basic HTTP authentication (user:pass)	-
body	string	The payload to send	-
body-file	filename	Same as `body` but contents as a file.	-
method	string	HTTP method to use for the query (either 'get' or 'post')	get
timeout	string	Timeout for the request (the default is 30 seconds)	-
response	ResponseResult	Names for parts of response	-
insecure	bool	Ignore TLS certificate validation errors (This is unsafe to use)	false

Fields

`when`

Type: message_filter

Fire this input when a specific internal message occurs

This field overloads time-based scheduling with a scheduler that fires on matching messages.

Example

Pipe Language Snippet:

input:
  http-poll:
    when:
      message-received:
        filter-type:
          - pipe-idle
    url: "http://localhost:8888"
    raw: true
    ignore-line-breaks: true

`interval`

Type: duration

How often to run the command

By default, interval: 0s which means: once. Note that scheduled inputs set document markers. See full discussion

Example

Pipe Language Snippet:

exec:
  command: echo 'once a day'
  interval: 1d

`cron`

Type: cron

How often to run the command. Note that unlike standard Cron, Pipes use a Cron syntax that includes a column for seconds. See full discussion

Example: Once a day

Pipe Language Snippet:

exec:
  command: echo 'once a day'
  cron: '0 0 0 * * *'

Example: Once a day, using a convenient shortcut

Pipe Language Snippet:

exec:
  command: echo 'once a day'
  cron: '@daily'

`immediate`

Type: bool

Default: false

Run as soon as invoked, instead of waiting for the specified cron interval

Example: Run immediately on invocation, and thereafter at 10h every morning

Pipe Language Snippet:

exec:
  command: echo 'hello'
  immediate: true
  cron: '0 0 10 * * *'

`random-offset`

Type: duration

Default: 0s

Sets a random offset to the schedule, then sticks to it

This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00

Example: Would fire up to a minute after every hour

Pipe Language Snippet:

exec:
  command: echo 'hello'
  random-offset: 1m
  cron: '0 0 * * * *'

`window`

Type: Window

For resources that need a time window to be specified

Field Name	Type	Description	Default
size	duration	Window size	-
offset	duration	Window offset	0s
start-time	time	Allows the windowing to start at a specified time	-
highwatermark-file	path	Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted	-

`size`

Type: duration

Window size

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  window:
    size: 1m

`offset`

Type: duration

Default: 0s

Window offset

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  window:
    size: 1m
    offset: 10s

`start-time`

Type: time

Allows the windowing to start at a specified time

It should in the following format: 2019-07-10 18:45:00.000 +0200

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  window:
    size: 1m
    start-time: 10s

`highwatermark-file`

Type: path

Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  window:
    size: 1m
    highwatermark-file:: /tmp/mark.txt

`block`

Type: bool

Default: false

Block further input schedules from triggering if the pipe output is retrying

`retry`

Type: Retry

For operations that could potentially fail

Field Name	Type	Description	Default
count	integer	How many attempts to make before declaring failure	-
pause	duration	How long to pause before re-trying	-
forever	bool	Keep trying until success is declared	false

`count`

Type: integer

How many attempts to make before declaring failure

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 1

Output:

{"_raw":"one two"}

`pause`

Type: duration

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    count: 6
    pause: 10s

Output:

{"_raw":"one two"}

`forever`

Type: bool

Default: false

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Example

Pipe Language Snippet:

exec:
  command: echo 'one two'
  retry:
    forever: true

Output:

{"_raw":"one two"}

`batch`

Type: Batch

For when a number of output events need to be marked as belonging to a distinct group

Field Name	Type	Description	Default
uuid-field	field	Field where generated uuid, the unique marker for the group, will be stored	-
invocation-time-field	field	Field where invocation time will be stored	-
completion-time-field	field	Field where completion (end of execution) time will be stored	-
begin-marker-field	field	Field used to mark first event in the group	-
end-marker-field	field	Field used to mark last event in the group	-
line-count-field	field	Field used to store the line count of the batch	-
line-num-field	field	Field used to store the line number of the batch	-

`uuid-field`

Type: field

Field where generated uuid, the unique marker for the group, will be stored

Example

Pipe Language Snippet:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    uuid-field: marker
  interval: 1m

Output:

{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}

Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)

Pipe Language Snippet:

exec:
  command: echo foo
  no-strip-linefeeds: true
  count: 3
  batch:
    uuid-field: marker
  interval: 1m

Output:

{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}

`invocation-time-field`

Type: field

Field where invocation time will be stored

Example

Pipe Language Snippet:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
  interval: 1m

Output:

{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}

`completion-time-field`

Type: field

Field where completion (end of execution) time will be stored

Example

Pipe Language Snippet:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    invocation-time-field: begin
    completion-time-field: end
  interval: 1m

Output:

{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}

`begin-marker-field`

Type: field

Field used to mark first event in the group

Example

Pipe Language Snippet:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
  interval: 1m

Output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}

`end-marker-field`

Type: field

Field used to mark last event in the group

Example

Pipe Language Snippet:

exec:
  command: |
    for n in $(seq 3)
    do
      echo $n
    done
  no-strip-linefeeds: true
  batch:
    begin-marker-field: begin
    end-marker-field: end
  interval: 1m

Output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}

`line-count-field`

Type: field

Field used to store the line count of the batch

`line-num-field`

Type: field

Field used to store the line number of the batch

`url`

Type: url
Alias: address

The URL for this request

`raw`

Type: bool
Alias: json
Default: false

Use for when data is text or already JSON

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: true

Output:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d24b6-4b01cb2acaf9e7a531b44d74"
  },
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/get"
}

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: "false"

Output:

{"_raw":"{"}
{"_raw":"  \"args\": {}, "}
{"_raw":"  \"headers\": {"}
{"_raw":"    \"Accept\": \"*/*\", "}
{"_raw":"    \"Accept-Encoding\": \"gzip\", "}
{"_raw":"    \"Content-Type\": \"application/json\", "}
{"_raw":"    \"Host\": \"httpbin.org\", "}
{"_raw":"    \"User-Agent\": \"reqwest/0.9.22\", "}
{"_raw":"    \"X-Amzn-Trace-Id\": \"Root=1-5e4d2a2d-0a1a9703fbf66d686b1ded16\""}
{"_raw":"  }, "}
{"_raw":"  \"origin\": \"159.65.88.207\", "}
{"_raw":"  \"url\": \"http://httpbin.org/get\""}
{"_raw":"}"}

`document-mode`

Type: bool

Default: false

Collect all the output together as a single document (will affect batching on the output)

`ignore-line-breaks`

Type: bool
Alias: ignore-linebreaks
Default: false

Do not treat separate lines as distinct events

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  ignore-line-breaks: true

Output:

{"_raw":"{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"gzip\", \n    \"Content-Type\": \"application/json\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"reqwest/0.9.22\", \n    \"X-Amzn-Trace-Id\": \"Root=1-5e4d2aa5-34b8cc9f7cfd296d2f78f4de\"\n  }, \n  \"origin\": \"159.65.88.207\", \n  \"url\": \"http://httpbin.org/get\"\n}\n"}

`headers`

Type: array of maps

Headers to send with the query

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: true
  headers:
    - custom-header: some data

Output:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Content-Type": "application/json",
    "Custom-Header": "some data",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
  },
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/get"
}

`query`

Type: array of maps

Query parameters to send with the query

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: true
  query:
    - one: '1'
    - two: '2'

Output:

{
  "args": {
    "one": "1",
    "two": "2"
  },
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
  },
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/get?one=1&two=2"
}

`auth`

Type: Auth

Basic HTTP authentication (user:pass)

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  raw: true
  auth:
    username: user
    password: pass

Output:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Authorization": "Basic dXNlcjpwYXNz",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
  },
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/get"
}

Field Name	Type	Description	Default
username	string	Name of User	-
password	string	Password	-

`username`

Type: string

Name of User

`password`

Type: string

Password

`body`

Type: string

The payload to send

Only applies to 'post'. "application/json" is default content type

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/post
  raw: true
  body: some content
  method: post

Output:

{
  "args": {},
  "data": "some content",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip",
    "Content-Length": "12",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "reqwest/0.9.22",
    "X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
  },
  "json": null,
  "origin": "159.65.88.207",
  "url": "http://httpbin.org/post"
}

`body-file`

Type: filename

Same as body but contents as a file.

`method`

Type: string

Default: get
Possible Values: get, post

HTTP method to use for the query (either 'get' or 'post')

Example

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/get
  method: get

Example: A POST

Pipe Language Snippet:

http-poll:
  address: http://httpbin.org/post
  method: post

`timeout`

Type: string

Timeout for the request (the default is 30 seconds)

`response`

Type: ResponseResult

Names for parts of response

By default, data is assumed to be JSON, set json to false if not. Next few examples assume localhost returns {"msg":"hello"}

Field Name	Type	Description	Default
json	bool	Return response in JSON format as single line-delimited event	true
status-field	field	Field to store HTTP status code	-
headers-field	field	Field to store response headers	-
response-field	field	Field to store the body of the response	_raw

`json`

Type: bool

Default: true

Return response in JSON format as single line-delimited event

Like raw, but guarantees that the JSON is on one line

Example

Pipe Language Snippet:

http-poll:
  address: http://localhost:3030
  response:
    json: false

Output:

{"_raw":"{\"msg\":\"hello\"}"}

`status-field`

Type: field

Field to store HTTP status code

Example

Pipe Language Snippet:

http-poll:
  address: http://localhost:3030
  response:
    status-field: status

Output:

{"_raw":{"msg":"hello"},"status":200}

`headers-field`

Type: field

Field to store response headers

Example

Pipe Language Snippet:

http-poll:
  address: http://localhost:3030
  response:
    headers-field: response-headers

Output:

{"_raw":{"msg":"hello"},"headers":{"content-length":"15","content-type":"text/plain; charset=utf-8","date":"Fri, 24 Apr 2020 14:05:14 GMT"}}

`response-field`

Type: field

Default: _raw

Field to store the body of the response

Example

Pipe Language Snippet:

http-poll:
  address: http://localhost:3030
  response:
    status-field: status
    response-field: body

Output:

{"body":{"msg":"hello"},"status":200}

`insecure`

Type: bool

Default: false

Ignore TLS certificate validation errors (This is unsafe to use)

Input: http-poll

Field Summary​

Fields​

when​

interval​

cron​

immediate​

random-offset​

window​

size​

offset​

start-time​

highwatermark-file​

block​

retry​

count​

pause​

forever​

batch​

uuid-field​

invocation-time-field​

completion-time-field​

begin-marker-field​

end-marker-field​

line-count-field​

line-num-field​

url​

raw​

document-mode​

ignore-line-breaks​

headers​

query​

auth​

username​

password​

body​

body-file​

method​

timeout​

response​

json​

status-field​

headers-field​

response-field​

insecure​

Field Summary

Fields

`when`

`interval`

`cron`

`immediate`

`random-offset`

`window`

`size`

`offset`

`start-time`

`highwatermark-file`

`block`

`retry`

`count`

`pause`

`forever`

`batch`

`uuid-field`

`invocation-time-field`

`completion-time-field`

`begin-marker-field`

`end-marker-field`

`line-count-field`

`line-num-field`

`url`

`raw`

`document-mode`

`ignore-line-breaks`

`headers`

`query`

`auth`

`username`

`password`

`body`

`body-file`

`method`

`timeout`

`response`

`json`

`status-field`

`headers-field`

`response-field`

`insecure`