kafka

Consume events from one or more Kafka topics

Example

action:

kafka:
  broker: localhost:9092
  topics: [test]
  group: frodo

output:

{
  "_raw":"{\"name\":\"peteNUM\",\"age\":43,\"birthdate\":\"1965-04-01\"}",
  "headers":{"foo":"bar"},
  "offset":6,"partition":0,
  "timestamp":1616057768091,
  "topic":"test"
}

Field Name	Description	Type	Default
broker	The [host:port] pair for the Kafka Broker	host:port	-
topics	The list of Kafka topics to subscribe to	array of strings	-
group	The group-id of the subscription	string	-
json	Do not quote with _raw (e.g. when data is already JSON)	bool	false
ignore-line-breaks	Do not break up the message by line breaks	bool	false
plain	Just pass through the payload of the message	bool	false
avro	Choose to use Apache Avro encoding for input and output	Avro	-

broker

The [host:port] pair for the Kafka Broker

Type: host:port

topics

The list of Kafka topics to subscribe to

Type: array of strings

group

The group-id of the subscription

Type: string

json

Do not quote with _raw (e.g. when data is already JSON)

Type: bool

Example

action:

kafka:
  broker: localhost:9092
  topics: [test]
  json: true
  group: frodo

output:

{"name":"peteNUM","age":43,"birthdate":"1965-04-01"}

ignore-line-breaks

Do not break up the message by line breaks

Type: bool

plain

Just pass through the payload of the message

Type: bool

avro

Choose to use Apache Avro encoding for input and output

Type: Avro

Field Name	Description	Type	Default
schema-file	File containing Avro schema in JSON format	path	-
schema-json	Avro schema in JSON format	string	-
kind	Choose the type of objects that are being read	string	single
compressed	Choose a compression strategy	string	gzip

schema-file

File containing Avro schema in JSON format

Type: path

schema-json

Avro schema in JSON format

Type: string

kind

Choose the type of objects that are being read

Type: string

compressed

Choose a compression strategy

Type: string