Skip to main content
Version: Next

Output: kafka

Write to a Kafka topic

Example

Pipe Language Snippet:

kafka:
broker: localhost:9092
topic: test

Field Summary

Field NameTypeDescriptionDefault
batchinteger or the 'document'Maximum number of events in an output batch. If 'document' send on end of document-
timeoutintervalInterval after which the batch is sent, to keep throughput going100ms
headertemplated textPut a header line before the batch-
footertemplated textPut a header line after the last line of the batch-
use-document-markerboolEnrich the pipe metadata with a document marker (for document handling in batch mode)false
brokerhost:portThe [host:port] pair for the Kafka Broker-
topicstringThe name of an existing Kafka topic to subscribe to-
headersarray of key-value pairsOptional headers for the message-
input-fieldfieldSend the data contained in this field-
keystringOptional key for the messagenone
message-timeoutdurationTime to wait for delivery500ms
avroAvroChoose to use Apache Avro encoding for input and output-
configStringMapA list of optional Kafka producer configuration parameters-

Fields

batch

Type: integer or the 'document'

Maximum number of events in an output batch. If 'document' send on end of document

timeout

Type: interval

Default: 100ms

Interval after which the batch is sent, to keep throughput going

Type: templated text

Put a header line before the batch

Type: templated text

Put a header line after the last line of the batch

use-document-marker

Type: bool

Default: false

Enrich the pipe metadata with a document marker (for document handling in batch mode)

broker

Type: host:port

The [host:port] pair for the Kafka Broker

topic

Type: string

The name of an existing Kafka topic to subscribe to

headers

Type: array of key-value pairs

Optional headers for the message

Example

Pipe Language Snippet:

kafka:
broker: localhost:9092
topic: test
headers:
- foo: bar

input-field

Type: field

Send the data contained in this field

key

Type: string

Default: none

Optional key for the message

message-timeout

Type: duration

Default: 500ms

Time to wait for delivery

avro

Type: Avro

Choose to use Apache Avro encoding for input and output

Field NameTypeDescriptionDefault
schema-filepathFile containing Avro schema in JSON format-
schema-jsonstringAvro schema in JSON format-
kindstringChoose the type of objects that are being readsingle
compressedstringChoose a compression strategygzip

  schema-file

Type: path

File containing Avro schema in JSON format

  schema-json

Type: string

Avro schema in JSON format

  kind

Type: string

Default: single
Possible Values: single, container

Choose the type of objects that are being read

  compressed

Type: string

Default: gzip
Possible Values: gzip

Choose a compression strategy

config

Type: StringMap

A list of optional Kafka producer configuration parameters