Skip to main content
Version: 3.4.0

kafka

Write to a Kafka topic

Example

action:

kafka:
broker: localhost:9092
topic: test

Field NameDescriptionTypeDefault
batchMaximum number of events in an output batch. If 'document' send on end of documentinteger or the 'document'-
timeoutInterval after which the batch is sent, to keep throughput goinginterval100ms
headerPut a header line before the batchtemplated text-
footerPut a header line after the last line of the batchtemplated text-
use-document-markerEnrich the pipe metadata with a document marker (for document handling in batch mode)boolfalse
brokerThe [host:port] pair for the Kafka Brokerhost:port-
topicThe name of an existing Kafka topic to subscribe tostring-
headersOptional headers for the messagearray of key-value pairs-
input-fieldSend the data contained in this fieldfield-
keyOptional key for the messagestringnone
message-timeoutTime to wait for deliveryduration500ms
avroChoose to use Apache Avro encoding for input and outputAvro-

batch

Maximum number of events in an output batch. If 'document' send on end of document

Type: integer or the 'document'

timeout

Interval after which the batch is sent, to keep throughput going

Type: interval

Put a header line before the batch

Type: templated text

Put a header line after the last line of the batch

Type: templated text

use-document-marker

Enrich the pipe metadata with a document marker (for document handling in batch mode)

Type: bool

broker

The [host:port] pair for the Kafka Broker

Type: host:port

topic

The name of an existing Kafka topic to subscribe to

Type: string

headers

Optional headers for the message

Type: array of key-value pairs

Example

action:

kafka:
broker: localhost:9092
topic: test
headers:
- foo: bar

input-field

Send the data contained in this field

Type: field

key

Optional key for the message

Type: string

message-timeout

Time to wait for delivery

Type: duration

avro

Choose to use Apache Avro encoding for input and output

Type: Avro

Field NameDescriptionTypeDefault
schema-fileFile containing Avro schema in JSON formatpath-
schema-jsonAvro schema in JSON formatstring-
kindChoose the type of objects that are being readstringsingle
compressedChoose a compression strategystringgzip

schema-file

File containing Avro schema in JSON format

Type: path

schema-json

Avro schema in JSON format

Type: string

kind

Choose the type of objects that are being read

Type: string

compressed

Choose a compression strategy

Type: string