Skip to main content
Version: 3.5.0

Input: kafka

Consume events from one or more Kafka topics

Example

Pipe Language Snippet:

kafka:
broker: localhost:9092
topics: [test]
group: frodo

Output:

{
"_raw":"{\"name\":\"peteNUM\",\"age\":43,\"birthdate\":\"1965-04-01\"}",
"headers":{"foo":"bar"},
"offset":6,"partition":0,
"timestamp":1616057768091,
"topic":"test"
}

Field Summary

Field NameTypeDescriptionDefault
brokerhost:portThe [host:port] pair for the Kafka Broker-
topicsarray of stringsThe list of Kafka topics to subscribe to-
groupstringThe group-id of the subscription-
jsonboolDo not quote with _raw (e.g. when data is already JSON)false
configStringMapA list of optional Kafka consumer configuration parameters-
ignore-line-breaksboolDo not break up the message by line breaksfalse
plainboolJust pass through the payload of the messagefalse
avroAvroChoose to use Apache Avro encoding for input and output-

Fields

broker

Type: host:port

The [host:port] pair for the Kafka Broker

topics

Type: array of strings

The list of Kafka topics to subscribe to

group

Type: string

The group-id of the subscription

json

Type: bool
Alias: json
Default: false

Do not quote with _raw (e.g. when data is already JSON)

Example

Pipe Language Snippet:

kafka:
broker: localhost:9092
topics: [test]
json: true
group: frodo

Output:

{"name":"peteNUM","age":43,"birthdate":"1965-04-01"}

config

Type: StringMap

A list of optional Kafka consumer configuration parameters

ignore-line-breaks

Type: bool
Alias: ignore-linebreaks
Default: false

Do not break up the message by line breaks

plain

Type: bool

Default: false

Just pass through the payload of the message

avro

Type: Avro

Choose to use Apache Avro encoding for input and output

Field NameTypeDescriptionDefault
schema-filepathFile containing Avro schema in JSON format-
schema-jsonstringAvro schema in JSON format-
kindstringChoose the type of objects that are being readsingle
compressedstringChoose a compression strategygzip

  schema-file

Type: path

File containing Avro schema in JSON format

  schema-json

Type: string

Avro schema in JSON format

  kind

Type: string

Default: single
Possible Values: single, container

Choose the type of objects that are being read

  compressed

Type: string

Default: gzip
Possible Values: gzip

Choose a compression strategy