kafka

Write to a Kafka topic

Example

action:

kafka:
  broker: localhost:9092
  topic: test

Field Name	Description	Type	Default
batch	Maximum number of events in an output batch. If 'document' send on end of document	integer or the 'document'	-
timeout	Interval after which the batch is sent, to keep throughput going	interval	100ms
header	Put a header line before the batch	templated text	-
footer	Put a header line after the last line of the batch	templated text	-
use-document-marker	Enrich the pipe metadata with a document marker (for document handling in batch mode)	bool	false
broker	The [host:port] pair for the Kafka Broker	host:port	-
topic	The name of an existing Kafka topic to subscribe to	string	-
headers	Optional headers for the message	array of key-value pairs	-
input-field	Send the data contained in this field	field	-
key	Optional key for the message	string	none
message-timeout	Time to wait for delivery	duration	500ms
avro	Choose to use Apache Avro encoding for input and output	Avro	-

batch

Maximum number of events in an output batch. If 'document' send on end of document

Type: integer or the 'document'

timeout

Interval after which the batch is sent, to keep throughput going

Type: interval

header

Put a header line before the batch

Type: templated text

footer

Put a header line after the last line of the batch

Type: templated text

use-document-marker

Enrich the pipe metadata with a document marker (for document handling in batch mode)

Type: bool

broker

The [host:port] pair for the Kafka Broker

Type: host:port

topic

The name of an existing Kafka topic to subscribe to

Type: string

headers

Optional headers for the message

Type: array of key-value pairs

Example

action:

kafka:
  broker: localhost:9092
  topic: test
  headers:
  - foo: bar

input-field

Send the data contained in this field

Type: field

key

Optional key for the message

Type: string

message-timeout

Time to wait for delivery

Type: duration

avro

Choose to use Apache Avro encoding for input and output

Type: Avro

Field Name	Description	Type	Default
schema-file	File containing Avro schema in JSON format	path	-
schema-json	Avro schema in JSON format	string	-
kind	Choose the type of objects that are being read	string	single
compressed	Choose a compression strategy	string	gzip

schema-file

File containing Avro schema in JSON format

Type: path

schema-json

Avro schema in JSON format

Type: string

kind

Choose the type of objects that are being read

Type: string

compressed

Choose a compression strategy

Type: string