amqp
Read from AMQP queues
| Field Name | Description | Type | Default |
|---|---|---|---|
| batch | For when a number of output events need to be marked as belonging to a distinct group | Batch | - |
| uri | AMQP address to listen to | url | amqp://localhost// |
| queue | An AMQP Queue | Queue | - |
| routing-key | Used in conjunction with topic exchange, to route data | string | - |
| exchange | An AMQP Exchange | Exchange | - |
| persistent | Let messages be cached and survive server restart | bool | false |
| durable | Let Queue or Exchange survive server restart | bool | false |
| raw | If true, treat incoming events as JSON, else wrap in _raw field ({"_raw":...}) | bool | false |
| ignore-line-breaks | Do not treat separate lines as distinct events | bool | false |
batch
For when a number of output events need to be marked as belonging to a distinct group
Type: Batch
| Field Name | Description | Type | Default |
|---|---|---|---|
| uuid-field | Field where generated uuid, the unique marker for the group, will be stored | field | - |
| invocation-time-field | Field where invocation time will be stored | field | - |
| completion-time-field | Field where completion (end of execution) time will be stored | field | - |
| begin-marker-field | Field used to mark first event in the group | field | - |
| end-marker-field | Field used to mark last event in the group | field | - |
| line-count-field | Field used to store the line count of the batch | field | - |
| line-num-field | Field used to store the line number of the batch | field | - |
uuid-field
Field where generated uuid, the unique marker for the group, will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
uuid-field: marker
interval: 1m
output:
{"_raw":"foo","line-count":3,"line-num":1,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"f3308aa9-6f56-4cc1-8782-c4231ff254b8"}
Example: For cases where event count is known, a simple counter is used, instead of uuid (useful for testing)
action:
exec:
command: echo foo
no-strip-linefeeds: true
count: 3
batch:
uuid-field: marker
interval: 1m
output:
{"_raw":"1","line-count":3,"line-num":1,"marker":"1"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"1"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"1"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"2"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"2"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"2"}
{"_raw":"1","line-count":3,"line-num":1,"marker":"3"}
{"_raw":"2","line-count":3,"line-num":2,"marker":"3"}
{"_raw":"3","line-count":3,"line-num":3,"marker":"3"}
invocation-time-field
Field where invocation time will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
interval: 1m
output:
{"_raw":"1","line-count":3,"line-num":1,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"2","line-count":3,"line-num":2,"begin":"2020-01-17T09:55:09.135Z"}
{"_raw":"3","line-count":3,"line-num":3,"begin":"2020-01-17T09:55:09.135Z"}
completion-time-field
Field where completion (end of execution) time will be stored
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
invocation-time-field: begin
completion-time-field: end
interval: 1m
output:
{"_raw":"1","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":1}
{"_raw":"2","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":2}
{"_raw":"3","begin":"2020-01-17T10:02:14.302Z","end":"2020-01-17T10:02:14.301Z","line-count":3,"line-num":3}
begin-marker-field
Field used to mark first event in the group
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
interval: 1m
output:
{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","line-count":3,"line-num":3}
end-marker-field
Field used to mark last event in the group
Type: field
Example
action:
exec:
command: |
for n in $(seq 3)
do
echo $n
done
no-strip-linefeeds: true
batch:
begin-marker-field: begin
end-marker-field: end
interval: 1m
output:
{"_raw":"1","begin":true,"line-count":3,"line-num":1}
{"_raw":"2","line-count":3,"line-num":2}
{"_raw":"3","end":true,"line-count":3,"line-num":3}
line-count-field
Field used to store the line count of the batch
Type: field
line-num-field
Field used to store the line number of the batch
Type: field
uri
AMQP address to listen to
Type: url
Example
action:
amqp:
queue:
name: queue-name
uri: amqp://localhost//
queue
An AMQP Queue
Type: Queue
| Field Name | Description | Type | Default |
|---|---|---|---|
| name | Queue name | string | - |
| passive | Assume Queue has already been declared, else error | bool | false |
name
Queue name
Type: string
Example
action:
amqp:
queue:
name: queue-name
passive
Assume Queue has already been declared, else error
Type: bool
Example
action:
amqp:
queue:
name: some-queue
passive: true
routing-key
Used in conjunction with topic exchange, to route data
Type: string
exchange
An AMQP Exchange
Type: Exchange
| Field Name | Description | Type | Default |
|---|---|---|---|
| name | Exchange name | string | - |
| type | Exchange type | string | - |
| passive | Assume Exchange has already been declared, else error | bool | false |
name
Exchange name
Type: string
Example
action:
amqp:
exchange:
name: some-exchange
type: fanout
type
Exchange type
There are 2 supported types:
- fanout: For when same messages are to be consumed by multiple comsumers
- topic: Topic-based pub-sub
- direct: For when routing keys are to be used
Type: string
Example
action:
amqp:
exchange:
name: some-exchange
type: topic
routing-key: some.key
passive
Assume Exchange has already been declared, else error
Type: bool
Example
action:
amqp:
exchange:
name: some-exchange
passive: true
persistent
Let messages be cached and survive server restart
Type: bool
Example
action:
amqp:
queue:
name: some-queue
persistent: true
durable
Let Queue or Exchange survive server restart
Type: bool
Example
action:
amqp:
queue:
name: some-queue
durable: true
raw
If true, treat incoming events as JSON, else wrap in _raw field ({"_raw":...})
Type: bool
Example
action:
amqp:
queue:
name: queue-name
raw: true
ignore-line-breaks
Do not treat separate lines as distinct events
Type: bool
Example
action:
amqp:
queue:
name: some-queue
ignore-line-breaks: true