scuba
Run BQL queries against a Scuba API endpoint
| Field Name | Description | Type | Default |
|---|---|---|---|
| interval | How often to run the command | duration | - |
| cron | How often to run the command | cron | - |
| immediate | Run as soon as invoked, instead of waiting for the specified cron interval | bool | false |
| random-offset | Sets a random offset to the schedule, then sticks to it | duration | 0s |
| window | For resources that need a time window to be specified | Window | - |
| block | Block further input schedules from triggering if the pipe output is retrying | bool | false |
| retry | For operations that could potentially fail | Retry | - |
| batch | For when a number of output events need to be marked as belonging to a distinct group | Batch | - |
| base-url | The base URL for this request (https://myserver.myscuba.example:3000) | url | - |
| insecure | Ignore TLS certificate validation errors (This is unsafe to use) | bool | false |
| bql-query | The Scuba BQL query to send to the server | string | - |
| api-key | The API key to be used to query the server | string | - |
interval
How often to run the command
By default, interval: 0s which means: once.
Note that scheduled inputs set document markers.
See full discussion
Type: duration
Example
action:
exec:
command: echo 'once a day'
interval: 1d
cron
How often to run the command
Type: cron
Example: Once a day
action:
exec:
command: echo 'once a day'
cron: '0 0 0 * * *'
Example: Once a day, using a convenient shortcut
action:
exec:
command: echo 'once a day'
cron: '@daily'
immediate
Run as soon as invoked, instead of waiting for the specified cron interval
Type: bool
Example: Run immediately on invocation, and thereafter at 10h every morning
action:
exec:
command: echo 'hello'
immediate: true
cron: '0 0 10 * * *'
random-offset
Sets a random offset to the schedule, then sticks to it
This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00
Type: duration
Example: Would fire up to a minute after every hour
action:
exec:
command: echo 'hello'
random-offset: 1m
cron: '0 0 * * * *'
window
For resources that need a time window to be specified
Type: Window
| Field Name | Description | Type | Default |
|---|---|---|---|
| size | Window size | duration | - |
| offset | Window offset | duration | 0s |
| start-time | Allows the windowing to start at a specified time | time | - |
| highwatermark-file | Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted | path | - |
size
Window size
Type: duration
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
offset
Window offset
Type: duration
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
offset: 10s
start-time
Allows the windowing to start at a specified time
It should in the following format: 2019-07-10 18:45:00.000 +0200
Type: time
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
start-time: 10s
highwatermark-file
Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted
Type: path
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
highwatermark-file:: /tmp/mark.txt
block
Block further input schedules from triggering if the pipe output is retrying
Type: bool
retry
For operations that could potentially fail
Type: Retry
| Field Name | Description | Type | Default |
|---|---|---|---|
| count | How many attempts to make before declaring failure | integer | - |
| pause | How long to pause before re-trying | duration | - |
| forever | Keep trying until success is declared | bool | false |
count
How many attempts to make before declaring failure
Type: integer
Example
action:
exec:
command: echo 'one two'
retry:
count: 1
output:
{"_raw":"one two"}
pause
How long to pause before re-trying
Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)
Type: duration
Example
action:
exec:
command: echo 'one two'
retry:
count: 6
pause: 10s
output:
{"_raw":"one two"}
forever
Keep trying until success is declared
Accepts human-friendly formats, like 1m (for 1 minute) and 4h (for 4 hours)
Type: bool
Example
action:
exec:
command: echo 'one two'
retry:
forever: true
output:
{"_raw":"one two"}
batch
For when a number of output events need to be marked as belonging to a distinct group
Type: Batch
| Field Name | Description | Type | Default |
|---|---|---|---|
| uuid-field | Field where generated uuid, the unique marker for the group, will be stored | field | - |
| invocation-time-field | Field where invocation time will be stored | field | - |
| completion-time-field | Field where completion (end of execution) time will be stored | field | - |
| begin-marker-field | Field used to mark first event in the group | field | - |
| end-marker-field | Field used to mark last event in the group | field | - |
| line-count-field | Field used to store the line count of the batch | field | - |
| line-num-field | Field used to store the line number of the batch | field | - |
uuid-field
Field where generated uuid, the unique marker for the group, will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
uuid-field: marker
interval: 1m
output:
{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)
action:
exec:
command: echo foo
no-strip-linefeeds: true
count: 3
batch:
uuid-field: marker
interval: 1m
output:
{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}
invocation-time-field
Field where invocation time will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
interval: 1m
output:
{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}
completion-time-field
Field where completion (end of execution) time will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
completion-time-field: end
interval: 1m
output:
{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}
begin-marker-field
Field used to mark first event in the group
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
interval: 1m
output:
{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}
end-marker-field
Field used to mark last event in the group
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
end-marker-field: end
interval: 1m
output:
{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}
line-count-field
Field used to store the line count of the batch
Type: field
line-num-field
Field used to store the line number of the batch
Type: field
base-url
The base URL for this request (https://myserver.myscuba.example:3000)
Type: url
insecure
Ignore TLS certificate validation errors (This is unsafe to use)
Type: bool
bql-query
The Scuba BQL query to send to the server
Type: string
api-key
The API key to be used to query the server
Type: string