Skip to main content
Version: 3.4.0

http-poll

Run HTTP queries (GET and POST)

Example

action:

http-poll:
address: http://httpbin.org/get
raw: true

output:

{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d24b6-4b01cb2acaf9e7a531b44d74"
},
"origin": "159.65.88.207",
"url": "http://httpbin.org/get"
}

Field NameDescriptionTypeDefault
intervalHow often to run the commandduration-
cronHow often to run the command. Note that Hotrod uses a different format than Cron it includes a column for seconds. See full discussioncron-
immediateRun as soon as invoked, instead of waiting for the specified cron intervalboolfalse
random-offsetSets a random offset to the schedule, then sticks to itduration0s
windowFor resources that need a time window to be specifiedWindow-
blockBlock further input schedules from triggering if the pipe output is retryingboolfalse
retryFor operations that could potentially failRetry-
batchFor when a number of output events need to be marked as belonging to a distinct groupBatch-
urlThe URL for this requesturl-
rawUse for when data is text or already JSONboolfalse
document-modeCollect all the output together as a single document (will affect batching on the output)boolfalse
ignore-line-breaksDo not treat separate lines as distinct eventsboolfalse
headersHeaders to send with the queryarray of maps-
queryQuery parameters to send with the queryarray of maps-
authBasic HTTP authentication (user:pass)Auth-
bodyThe payload to sendstring-
body-fileSame as body but contents as a file.filename-
methodHTTP method to use for the query (either 'get' or 'post')stringget
responseNames for parts of responseResponseResult-
insecureIgnore TLS certificate validation errors (This is unsafe to use)boolfalse

interval

How often to run the command

By default, interval: 0s which means: once. Note that scheduled inputs set document markers. See full discussion

Type: duration

Example

action:

exec:
command: echo 'once a day'
interval: 1d

cron

How often to run the command. Note that Hotrod uses a different format than Cron it includes a column for seconds. See full discussion

Type: cron

Example: Once a day

action:

exec:
command: echo 'once a day'
cron: '0 0 0 * * *'

Example: Once a day, using a convenient shortcut

action:

exec:
command: echo 'once a day'
cron: '@daily'

immediate

Run as soon as invoked, instead of waiting for the specified cron interval

Type: bool

Example: Run immediately on invocation, and thereafter at 10h every morning

action:

exec:
command: echo 'hello'
immediate: true
cron: '0 0 10 * * *'

random-offset

Sets a random offset to the schedule, then sticks to it

This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00

Type: duration

Example: Would fire up to a minute after every hour

action:

exec:
command: echo 'hello'
random-offset: 1m
cron: '0 0 * * * *'

window

For resources that need a time window to be specified

Type: Window

Field NameDescriptionTypeDefault
sizeWindow sizeduration-
offsetWindow offsetduration0s
start-timeAllows the windowing to start at a specified timetime-
highwatermark-fileSpecify file where timestamp would be stored in order to resume, for when Pipe has been restartedpath-

size

Window size

Type: duration

Example

action:

exec:
command: echo 'one two'
window:
size: 1m

offset

Window offset

Type: duration

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
offset: 10s

start-time

Allows the windowing to start at a specified time

It should in the following format: 2019-07-10 18:45:00.000 +0200

Type: time

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
start-time: 10s

highwatermark-file

Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted

Type: path

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
highwatermark-file:: /tmp/mark.txt

block

Block further input schedules from triggering if the pipe output is retrying

Type: bool

retry

For operations that could potentially fail

Type: Retry

Field NameDescriptionTypeDefault
countHow many attempts to make before declaring failureinteger-
pauseHow long to pause before re-tryingduration-
foreverKeep trying until success is declaredboolfalse

count

How many attempts to make before declaring failure

Type: integer

Example

action:

exec:
command: echo 'one two'
retry:
count: 1

output:

{"_raw":"one two"}

pause

How long to pause before re-trying

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Type: duration

Example

action:

exec:
command: echo 'one two'
retry:
count: 6
pause: 10s

output:

{"_raw":"one two"}

forever

Keep trying until success is declared

Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)

Type: bool

Example

action:

exec:
command: echo 'one two'
retry:
forever: true

output:

{"_raw":"one two"}

batch

For when a number of output events need to be marked as belonging to a distinct group

Type: Batch

Field NameDescriptionTypeDefault
uuid-fieldField where generated uuid, the unique marker for the group, will be storedfield-
invocation-time-fieldField where invocation time will be storedfield-
completion-time-fieldField where completion (end of execution) time will be storedfield-
begin-marker-fieldField used to mark first event in the groupfield-
end-marker-fieldField used to mark last event in the groupfield-
line-count-fieldField used to store the line count of the batchfield-
line-num-fieldField used to store the line number of the batchfield-

uuid-field

Field where generated uuid, the unique marker for the group, will be stored

Type: field

Example

action:

exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
uuid-field: marker
interval: 1m

output:

{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}

Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)

action:

exec:
command: echo foo
no-strip-linefeeds: true
count: 3
batch:
uuid-field: marker
interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}

invocation-time-field

Field where invocation time will be stored

Type: field

Example

action:

exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
interval: 1m

output:

{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}

completion-time-field

Field where completion (end of execution) time will be stored

Type: field

Example

action:

exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
completion-time-field: end
interval: 1m

output:

{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}

begin-marker-field

Field used to mark first event in the group

Type: field

Example

action:

exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}

end-marker-field

Field used to mark last event in the group

Type: field

Example

action:

exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
end-marker-field: end
interval: 1m

output:

{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}

line-count-field

Field used to store the line count of the batch

Type: field

line-num-field

Field used to store the line number of the batch

Type: field

url

The URL for this request

Type: url

raw

Use for when data is text or already JSON

Type: bool

Example

action:

http-poll:
address: http://httpbin.org/get
raw: true

output:

{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d24b6-4b01cb2acaf9e7a531b44d74"
},
"origin": "159.65.88.207",
"url": "http://httpbin.org/get"
}

Example

action:

http-poll:
address: http://httpbin.org/get
raw: "false"

output:

{"_raw":"{"}
{"_raw":" \"args\": {}, "}
{"_raw":" \"headers\": {"}
{"_raw":" \"Accept\": \"*/*\", "}
{"_raw":" \"Accept-Encoding\": \"gzip\", "}
{"_raw":" \"Content-Type\": \"application/json\", "}
{"_raw":" \"Host\": \"httpbin.org\", "}
{"_raw":" \"User-Agent\": \"reqwest/0.9.22\", "}
{"_raw":" \"X-Amzn-Trace-Id\": \"Root=1-5e4d2a2d-0a1a9703fbf66d686b1ded16\""}
{"_raw":" }, "}
{"_raw":" \"origin\": \"159.65.88.207\", "}
{"_raw":" \"url\": \"http://httpbin.org/get\""}
{"_raw":"}"}

document-mode

Collect all the output together as a single document (will affect batching on the output)

Type: bool

ignore-line-breaks

Do not treat separate lines as distinct events

Type: bool

Example

action:

http-poll:
address: http://httpbin.org/get
ignore-line-breaks: true

output:

{"_raw":"{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\": \"gzip\", \n    \"Content-Type\": \"application/json\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\": \"reqwest/0.9.22\", \n    \"X-Amzn-Trace-Id\": \"Root=1-5e4d2aa5-34b8cc9f7cfd296d2f78f4de\"\n  }, \n  \"origin\": \"159.65.88.207\", \n  \"url\": \"http://httpbin.org/get\"\n}\n"}

headers

Headers to send with the query

Type: array of maps

Example

action:

http-poll:
address: http://httpbin.org/get
raw: true
headers:
- custom-header: some data

output:

{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Content-Type": "application/json",
"Custom-Header": "some data",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
},
"origin": "159.65.88.207",
"url": "http://httpbin.org/get"
}

query

Query parameters to send with the query

Type: array of maps

Example

action:

http-poll:
address: http://httpbin.org/get
raw: true
query:
- one: '1'
- two: '2'

output:

{
"args": {
"one": "1",
"two": "2"
},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
},
"origin": "159.65.88.207",
"url": "http://httpbin.org/get?one=1&two=2"
}

auth

Basic HTTP authentication (user:pass)

Type: Auth

Example

action:

http-poll:
address: http://httpbin.org/get
raw: true
auth:
username: user
password: pass

output:

{
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Authorization": "Basic dXNlcjpwYXNz",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
},
"origin": "159.65.88.207",
"url": "http://httpbin.org/get"
}

Field NameDescriptionTypeDefault
usernameName of Userstring-
passwordPasswordstring-

username

Name of User

Type: string

password

Password

Type: string

body

The payload to send

Only applies to 'post'. "application/json" is default content type

Type: string

Example

action:

http-poll:
address: http://httpbin.org/post
raw: true
body: some content
method: post

output:

{
"args": {},
"data": "some content",
"files": {},
"form": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Content-Length": "12",
"Content-Type": "application/json",
"Host": "httpbin.org",
"User-Agent": "reqwest/0.9.22",
"X-Amzn-Trace-Id": "Root=1-5e4d2c52-9a3ee870293e2d002828bdb0"
},
"json": null,
"origin": "159.65.88.207",
"url": "http://httpbin.org/post"
}

body-file

Same as body but contents as a file.

Type: filename

method

HTTP method to use for the query (either 'get' or 'post')

Type: string

Example

action:

http-poll:
address: http://httpbin.org/get
method: get

Example: A POST

action:

http-poll:
address: http://httpbin.org/post
method: post

response

Names for parts of response

By default, data is assumed to be JSON, set json to false if not. Next few examples assume localhost returns {"msg":"hello"}

Type: ResponseResult

Field NameDescriptionTypeDefault
jsonReturn response in JSON format as single line-delimited eventbooltrue
status-fieldField to store HTTP status codefield-
headers-fieldField to store response headersfield-
response-fieldField to store the body of the responsefield_raw

json

Return response in JSON format as single line-delimited event

Like raw, but guarantees that the JSON is on one line

Type: bool

Example

action:

http-poll:
address: http://localhost:3030
response:
json: false

output:

{"_raw":"{\"msg\":\"hello\"}"}

status-field

Field to store HTTP status code

Type: field

Example

action:

http-poll:
address: http://localhost:3030
response:
status-field: status

output:

{"_raw":{"msg":"hello"},"status":200}

headers-field

Field to store response headers

Type: field

Example

action:

http-poll:
address: http://localhost:3030
response:
headers-field: response-headers

output:

{"_raw":{"msg":"hello"},"headers":{"content-length":"15","content-type":"text/plain; charset=utf-8","date":"Fri, 24 Apr 2020 14:05:14 GMT"}}

response-field

Field to store the body of the response

Type: field

Example

action:

http-poll:
address: http://localhost:3030
response:
status-field: status
response-field: body

output:

{"body":{"msg":"hello"},"status":200}

insecure

Ignore TLS certificate validation errors (This is unsafe to use)

Type: bool