sql
Query a SQL database
Currently only Postgres and Sqlite (embedded) are supported
| Field Name | Description | Type | Default |
|---|---|---|---|
| interval | How often to run the command | duration | - |
| cron | How often to run the command | cron | - |
| immediate | Run as soon as invoked, instead of waiting for the specified cron interval | bool | false |
| random-offset | Sets a random offset to the schedule, then sticks to it | duration | 0s |
| window | For resources that need a time window to be specified | Window | - |
| block | Block further input schedules from triggering if the pipe output is retrying | bool | false |
| query | An SQL SELECT statement | string | - |
| source | SQL connection options | SqlSources | - |
interval
How often to run the command
By default, interval: 0s which means: once.
Note that scheduled inputs set document markers.
See full discussion
Type: duration
Example
action:
exec:
command: echo 'once a day'
interval: 1d
cron
How often to run the command
Type: cron
Example: Once a day
action:
exec:
command: echo 'once a day'
cron: '0 0 0 * * *'
Example: Once a day, using a convenient shortcut
action:
exec:
command: echo 'once a day'
cron: '@daily'
immediate
Run as soon as invoked, instead of waiting for the specified cron interval
Type: bool
Example: Run immediately on invocation, and thereafter at 10h every morning
action:
exec:
command: echo 'hello'
immediate: true
cron: '0 0 10 * * *'
random-offset
Sets a random offset to the schedule, then sticks to it
This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00
Type: duration
Example: Would fire up to a minute after every hour
action:
exec:
command: echo 'hello'
random-offset: 1m
cron: '0 0 * * * *'
window
For resources that need a time window to be specified
Type: Window
| Field Name | Description | Type | Default |
|---|---|---|---|
| size | Window size | duration | - |
| offset | Window offset | duration | 0s |
| start-time | Allows the windowing to start at a specified time | time | - |
| highwatermark-file | Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted | path | - |
size
Window size
Type: duration
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
offset
Window offset
Type: duration
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
offset: 10s
start-time
Allows the windowing to start at a specified time
It should in the following format: 2019-07-10 18:45:00.000 +0200
Type: time
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
start-time: 10s
highwatermark-file
Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted
Type: path
Example
action:
exec:
command: echo 'one two'
window:
size: 1m
highwatermark-file:: /tmp/mark.txt
block
Block further input schedules from triggering if the pipe output is retrying
Type: bool
query
An SQL SELECT statement
Type: string
source
SQL connection options
Type: SqlSources
| Field Name | Description | Type | Default |
|---|---|---|---|
| postgres | Specify kind and connection for an SQL database | SqlSource | - |
| sqlite | Specify kind and connection for an SQL database | SqlSource | - |
| my-sql | Specify kind and connection for an SQL database | SqlSource | - |
| odbc | SQL options | OdbcSqlSource | - |
postgres
Specify kind and connection for an SQL database
Type: SqlSource
| Field Name | Description | Type | Default |
|---|---|---|---|
| connection | For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres" | connection-string | - |
connection
For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres"
Type: connection-string
sqlite
Specify kind and connection for an SQL database
Type: SqlSource
| Field Name | Description | Type | Default |
|---|---|---|---|
| connection | For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres" | connection-string | - |
connection
For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres"
Type: connection-string
my-sql
Specify kind and connection for an SQL database
Type: SqlSource
| Field Name | Description | Type | Default |
|---|---|---|---|
| connection | For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres" | connection-string | - |
connection
For 'sqlite', a path to a file, for 'postgres' strings like "host=localhost user=postgres"
Type: connection-string
odbc
SQL options
Type: OdbcSqlSource
| Field Name | Description | Type | Default |
|---|---|---|---|
| quick | Simple connection to ODBC (assumes system or user configuration) | DsnFields | - |
| connection-string | Create a compatible connection string to connect to ODBC this may vary between ODBC drivers | DriverFields | - |
| max-str-len | The maximum string buffer to allocate when parsing returned strings this is specified in bytes and should be large enough to accommodate the largest possible string returned | integer | - |
| max-bin-len | The maximum string buffer to allocate when parsing returned binary blobs this is specified in bytes and should be large enough to accommodate the largest possible binary blobs returned | integer | - |
quick
Simple connection to ODBC (assumes system or user configuration)
Type: DsnFields
| Field Name | Description | Type | Default |
|---|---|---|---|
| dsn | The configured DSN name | string | - |
| username | The username for the ODBC connection | string | - |
| password | The password for the ODBC connection | string | - |
dsn
The configured DSN name
Type: string
username
The username for the ODBC connection
Type: string
password
The password for the ODBC connection
Type: string
connection-string
Create a compatible connection string to connect to ODBC this may vary between ODBC drivers
Type: DriverFields
| Field Name | Description | Type | Default |
|---|---|---|---|
| driver | The ODBC driver to use | string | - |
| attributes | Attributes in a "String: Value" format | Map | - |
driver
The ODBC driver to use
Type: string
attributes
Attributes in a "String: Value" format
Type: Map
max-str-len
The maximum string buffer to allocate when parsing returned strings this is specified in bytes and should be large enough to accommodate the largest possible string returned
Type: integer
max-bin-len
The maximum string buffer to allocate when parsing returned binary blobs this is specified in bytes and should be large enough to accommodate the largest possible binary blobs returned
Type: integer