Skip to main content
Version: Next

Splunk HEC

The Splunk HEC output provides a convenient mechanism to send properly formatted events to a Splunk HEC input. HOTROD abstracts away all the required operations for submitting events to the Splunk API.

Simple message with timestamp

A simple message with added timestamp, remove: true removes mytime after the field value has been placed into the time field. If there is an existing time field then time-field: time is not required.

name: splunk-hec-eventlog
input:
echo:
json: true
ignore-linebreaks: false
event: |
{"message":"Hello, World!"}
actions:
- time:
output-field: mytime
output-format: epoch_frac_secs
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
host: server.hotrod.local
sourcetype: hotrod:hpr
source: pipe
time-field: mytime
remove: true
warn

If there is no time field in the submitted event, the Splunk timestamp is time received for said event.

Only url (possibly with insecure: true) and hec-token options are mandatory:

...
output:
splunk-hec:
insecure: true
url: https://splunk.hotrod.local:8088/services/collector/event
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697

If host, source, sourcetype, and time are omitted, Splunk indexes the following:

```plaintext host = splunk.hotrod.local:8088 source = http:hotrod_events sourcetype = httpevent _index = hotrod_events _time = 2023-07-24T18:11:44.000+00:00 ```
  • host being the configured hostname of the Splunk host
  • source being the configured index (default), or indexes (allowed indexes) in the HEC inputs.conf
  • sourcetype being the Splunk default if no sourcetype field was extracted
  • index being the configured default index
  • time being received time

Example inputs.conf:

...
[http://hotrod_events]
disabled = 0
host = aa9aa88978a2
index = hotrod_events
indexes = hotrod_events
token = 05c0cde0-ecf0-4576-bd93-819d33529697

Added or extracted fields as metadata

Here the fields have been specified for demonstrative purposes but in a real-world example, extracted from the ingested event:

name: splunk-hec-eventlog
input:
echo:
json: true
ignore-linebreaks: false
event: |
{"message":"Hello, World!"}
actions:
- time:
output-field: mytime
output-format: epoch_frac_secs
- add:
fields:
- myhost: myhost.hotrod.local
- myindex: "hotrod_events"
- mysource: "pipe"
- mysourcetype: "hotrod:message"
- mytoken: 05c0cde0-ecf0-4576-bd93-819d33529697
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
host-field: myhost
source-field: mysource
sourcetype-field: mysourcetype
hec-token-field: mytoken
time-field: mytime
remove: true

As before, all specified *-field: my* fields are removed.

An event value as payload

The event payload is the event-field: myevent value:

name: splunk-hec-eventlog
input:
echo:
json: true
ignore-linebreaks: false
event: |
{"message":"Hello, World!"}
actions:
- time:
output-field: mytime
output-format: epoch_frac_secs
- add:
fields:
- myevent: Something happened.
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
host: splunk.hotrod.local
sourcetype: hotrod:hpr
source: pipe
event-field: myevent
note

The below event-field: myevent formats are supported.

actions:
- add:
fields:
- myevent: Something happened.
actions:
- add:
fields:
- myevent: {"message":{"what":"Something happened.","severity":"INFO"}}
warn

host-field, source-field, sourcetype-field, and hec-field are mutually exclusive with host, source, sourcetype and hec-token.

danger

The below event-field: myevent formats are unsupported and result in duplicate host, source, and sourcetype Splunk fields with time not appearing as _time.

actions:
- add:
fields:
- myevent: {"message":{"what":"Something happened.","severity":"INFO"},"host":"foo.hotrod.local","source":"foo","sourcetype":"hotrod:foo","time":1689106057.777}
actions:
- add:
fields:
- myevent: {"message":"Something happened.","severity":"INFO","host":"foo.hotrod.local","source":"foo","sourcetype":"hotrod:foo","time":1689106057.777}

Duplicate source fields problem

The problematic Pipe:

name: splunk-hec-eventlog
input:
echo:
json: true
event: |
{"message":"Hello, World!","source":"pipe"}
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
host: myhost.hotrod.local
source: pipe
sourcetype: hotrod:message
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697

Resulting in duplicate source fields per Splunk event:

The POST event is correctly formatted as shown in the debug log (note the nested and top-level source fields):

2023-07-20T14:21:43.250Z DEBUG pipeline::webhook          > post:
https://splunk.hotrod.local:8088/services/collector/event
{"authorization": "Splunk 05c0cde0-ecf0-4576-bd93-819d33529697", "content-type": "application/json"}
"{\"event\":{\"message\":\"Hello, World!\",\"source\":\"pipe\"},\"host\":\"myhost.hotrod.local\",\"source\":\"pipe\",\"sourcetype\":\"hotrod:message\"}"

According to the Splunk docs: only top-level fields are treated as metadata. So in this case, the 1st, nested, "source":"pipe" field is not a Splunk metadata source field because it is not top-level. This could be a Splunk bug. But, the docs do not explicitly state that this scenario is either supported or unsupported. None of the examples allude to or mention the handling of duplicate top-level and nested, recognised metadata fields. If we ratify the spec, then it stands to reason that it should be supported.

Workaround the duplicate source field problem by renaming the source field to mysource or specifiying source-field: source to remove the nested source field and create a top-level source metadata field instead:

name: splunk-hec-eventlog
input:
echo:
json: true
event: |
{"message":"Hello, World!","source":"announcement"}
actions:
- rename:
- source: mysource
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
host: myhost.hotrod.local
source: pipe
sourcetype: hotrod:message
name: splunk-hec-eventlog
input:
echo:
json: true
event: |
{"message":"Hello, World!","source":"pipe"}
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
host: myhost.hotrod.local
source-field: source
sourcetype: hotrod:message
remove: true

Colon in sourcetype broken

According to Splunk docs, sourcetype does support a colon as a naming convention. sourcetype: hotrod:message causes duplicate field values in Splunk:

...
output:
splunk-hec:
url: https://splunk.hotrod.local:8088/services/collector/event
insecure: true
hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
host: myhost.hotrod.local
source-field: source
sourcetype: hotrod:message
remove: true

source":"pipe:pipe does not cause duplicates.

Workaround: don't use a colon.