Input: kafka

Consume events from one or more Kafka topics

Example

Pipe Language Snippet:

kafka:
  broker: localhost:9092
  topics: [test]
  group: frodo

Output:

{
  "_raw":"{\"name\":\"peteNUM\",\"age\":43,\"birthdate\":\"1965-04-01\"}",
  "headers":{"foo":"bar"},
  "offset":6,"partition":0,
  "timestamp":1616057768091,
  "topic":"test"
}

Field Summary

Field Name	Type	Description	Default
broker	host:port	The [host:port] pair for the Kafka Broker	-
topics	array of strings	The list of Kafka topics to subscribe to	-
group	string	The group-id of the subscription	-
json	bool	Do not quote with _raw (e.g. when data is already JSON)	false
config	StringMap	A list of optional Kafka consumer configuration parameters	-
ignore-line-breaks	bool	Do not break up the message by line breaks	false
plain	bool	Just pass through the payload of the message	false
avro	Avro	Choose to use Apache Avro encoding for input and output	-

Fields

broker

Type: host:port

The [host:port] pair for the Kafka Broker

topics

Type: array of strings

The list of Kafka topics to subscribe to

group

Type: string

The group-id of the subscription

json

Type: bool
Alias: json
Default: false

Do not quote with _raw (e.g. when data is already JSON)

Example

Pipe Language Snippet:

kafka:
  broker: localhost:9092
  topics: [test]
  json: true
  group: frodo

Output:

{"name":"peteNUM","age":43,"birthdate":"1965-04-01"}

config

Type: StringMap

A list of optional Kafka consumer configuration parameters

ignore-line-breaks

Type: bool
Alias: ignore-linebreaks
Default: false

Do not break up the message by line breaks

plain

Type: bool

Default: false

Just pass through the payload of the message

avro

Type: Avro

Choose to use Apache Avro encoding for input and output

Field Name	Type	Description	Default
schema-file	path	File containing Avro schema in JSON format	-
schema-json	string	Avro schema in JSON format	-
kind	string	Choose the type of objects that are being read	single
compressed	string	Choose a compression strategy	gzip

  schema-file

Type: path

File containing Avro schema in JSON format

  schema-json

Type: string

Avro schema in JSON format

  kind

Type: string

Default: single
Possible Values: single, container

Choose the type of objects that are being read

  compressed

Type: string

Default: gzip
Possible Values: gzip

Choose a compression strategy