Output: kafka

Write to a Kafka topic

Example

Pipe Language Snippet:

kafka:
  broker: localhost:9092
  topic: test

Field Summary

Field Name	Type	Description	Default
batch	integer or the 'document'	Maximum number of events in an output batch. If 'document' send on end of document	-
timeout	interval	Interval after which the batch is sent, to keep throughput going	100ms
header	templated text	Put a header line before the batch	-
footer	templated text	Put a header line after the last line of the batch	-
use-document-marker	bool	Enrich the pipe metadata with a document marker (for document handling in batch mode)	false
broker	host:port	The [host:port] pair for the Kafka Broker	-
topic	string	The name of an existing Kafka topic to subscribe to	-
headers	array of key-value pairs	Optional headers for the message	-
input-field	field	Send the data contained in this field	-
key	string	Optional key for the message	none
message-timeout	duration	Time to wait for delivery	500ms
avro	Avro	Choose to use Apache Avro encoding for input and output	-
config	StringMap	A list of optional Kafka producer configuration parameters	-

Fields

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

header

Type: templated text

Put a header line before the batch

footer

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

broker

Type: host:port

The [host:port] pair for the Kafka Broker

topic

Type: string

The name of an existing Kafka topic to subscribe to

headers

Type: array of key-value pairs

Optional headers for the message

Example

Pipe Language Snippet:

kafka:
  broker: localhost:9092
  topic: test
  headers:
  - foo: bar

input-field

Type: field

Send the data contained in this field

key

Type: string

Default: none

Optional key for the message

message-timeout

Type: duration

Default: 500ms

Time to wait for delivery

avro

Type: Avro

Choose to use Apache Avro encoding for input and output

Field Name	Type	Description	Default
schema-file	path	File containing Avro schema in JSON format	-
schema-json	string	Avro schema in JSON format	-
kind	string	Choose the type of objects that are being read	single
compressed	string	Choose a compression strategy	gzip

  schema-file

Type: path

File containing Avro schema in JSON format

  schema-json

Type: string

Avro schema in JSON format

  kind

Type: string

Default: single
Possible Values: single, container

Choose the type of objects that are being read

  compressed

Type: string

Default: gzip
Possible Values: gzip

Choose a compression strategy

config

Type: StringMap

A list of optional Kafka producer configuration parameters