Skip to main content
Version: 3.3.1

kafka

Consume events from one or more Kafka topics

Example

action:

kafka:
broker: localhost:9092
topics: [test]
group: frodo

output:

{
"_raw":"{\"name\":\"peteNUM\",\"age\":43,\"birthdate\":\"1965-04-01\"}",
"headers":{"foo":"bar"},
"offset":6,"partition":0,
"timestamp":1616057768091,
"topic":"test"
}

Field NameDescriptionTypeDefault
brokerThe [host:port] pair for the Kafka Brokerhost:port-
topicsThe list of Kafka topics to subscribe toarray of strings-
groupThe group-id of the subscriptionstring-
jsonDo not quote with _raw (e.g. when data is already JSON)boolfalse
ignore-line-breaksDo not break up the message by line breaksboolfalse
plainJust pass through the payload of the messageboolfalse
avroChoose to use Apache Avro encoding for input and outputAvro-

broker

The [host:port] pair for the Kafka Broker

Type: host:port

topics

The list of Kafka topics to subscribe to

Type: array of strings

group

The group-id of the subscription

Type: string

json

Do not quote with _raw (e.g. when data is already JSON)

Type: bool

Example

action:

kafka:
broker: localhost:9092
topics: [test]
json: true
group: frodo

output:

{"name":"peteNUM","age":43,"birthdate":"1965-04-01"}

ignore-line-breaks

Do not break up the message by line breaks

Type: bool

plain

Just pass through the payload of the message

Type: bool

avro

Choose to use Apache Avro encoding for input and output

Type: Avro

Field NameDescriptionTypeDefault
schema-fileFile containing Avro schema in JSON formatpath-
schema-jsonAvro schema in JSON formatstring-
kindChoose the type of objects that are being readstringsingle
compressedChoose a compression strategystringgzip

schema-file

File containing Avro schema in JSON format

Type: path

schema-json

Avro schema in JSON format

Type: string

kind

Choose the type of objects that are being read

Type: string

compressed

Choose a compression strategy

Type: string