Output: kafka
Write to a Kafka topic
Example
Pipe Language Snippet:
kafka:
broker: localhost:9092
topic: test
Field Summary
| Field Name | Type | Description | Default |
|---|---|---|---|
| batch | integer or the 'document' | Maximum number of events in an output batch. If 'document' send on end of document | - |
| timeout | interval | Interval after which the batch is sent, to keep throughput going | 100ms |
| header | templated text | Put a header line before the batch | - |
| footer | templated text | Put a header line after the last line of the batch | - |
| use-document-marker | bool | Enrich the pipe metadata with a document marker (for document handling in batch mode) | false |
| broker | host:port | The [host:port] pair for the Kafka Broker | - |
| topic | string | The name of an existing Kafka topic to subscribe to | - |
| headers | array of key-value pairs | Optional headers for the message | - |
| input-field | field | Send the data contained in this field | - |
| key | string | Optional key for the message | none |
| message-timeout | duration | Time to wait for delivery | 500ms |
| avro | Avro | Choose to use Apache Avro encoding for input and output | - |
| config | StringMap | A list of optional Kafka producer configuration parameters | - |
Fields
batch
Type: integer or the 'document'
Maximum number of events in an output batch. If 'document' send on end of document
timeout
Type: interval
Default: 100ms
Interval after which the batch is sent, to keep throughput going
header
Type: templated text
Put a header line before the batch
footer
Type: templated text
Put a header line after the last line of the batch
use-document-marker
Type: bool
Default: false
Enrich the pipe metadata with a document marker (for document handling in batch mode)
broker
Type: host:port
The [host:port] pair for the Kafka Broker
topic
Type: string
The name of an existing Kafka topic to subscribe to
headers
Type: array of key-value pairs
Optional headers for the message
Example
Pipe Language Snippet:
kafka:
broker: localhost:9092
topic: test
headers:
- foo: bar
input-field
Type: field
Send the data contained in this field
key
Type: string
Default: none
Optional key for the message
message-timeout
Type: duration
Default: 500ms
Time to wait for delivery
avro
Type: Avro
Choose to use Apache Avro encoding for input and output
| Field Name | Type | Description | Default |
|---|---|---|---|
| schema-file | path | File containing Avro schema in JSON format | - |
| schema-json | string | Avro schema in JSON format | - |
| kind | string | Choose the type of objects that are being read | single |
| compressed | string | Choose a compression strategy | gzip |
schema-file
Type: path
File containing Avro schema in JSON format
schema-json
Type: string
Avro schema in JSON format
kind
Type: string
Default: single
Possible Values: single, container
Choose the type of objects that are being read
compressed
Type: string
Default: gzip
Possible Values: gzip
Choose a compression strategy
config
Type: StringMap
A list of optional Kafka producer configuration parameters