Splunk HEC

The Splunk HEC output provides a convenient mechanism to send properly formatted events to a Splunk HEC input. HOTROD abstracts away all the required operations for submitting events to the Splunk API.

Simple message with timestamp

A simple message with added timestamp, remove: true removes mytime after the field value has been placed into the time field. If there is an existing time field then time-field: time is not required.

name: splunk-hec-eventlog
input:
  echo:
    json: true
    ignore-linebreaks: false
    event: |
      {"message":"Hello, World!"}
actions:
  - time:
      output-field: mytime
      output-format: epoch_frac_secs
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
    host: server.hotrod.local
    sourcetype: hotrod:hpr
    source: pipe
    time-field: mytime
    remove: true

warn

If there is no time field in the submitted event, the Splunk timestamp is time received for said event.

Only url (possibly with insecure: true) and hec-token options are mandatory:

...
output:
  splunk-hec:
    insecure: true
    url: https://splunk.hotrod.local:8088/services/collector/event
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697

If host, source, sourcetype, and time are omitted, Splunk indexes the following:

```plaintext host = splunk.hotrod.local:8088 source = http:hotrod_events sourcetype = httpevent _index = hotrod_events _time = 2023-07-24T18:11:44.000+00:00 ```

• host being the configured hostname of the Splunk host
• source being the configured index (default), or indexes (allowed indexes) in the HEC inputs.conf
• sourcetype being the Splunk default if no sourcetype field was extracted
• index being the configured default index
• time being received time

Example inputs.conf:

...
[http://hotrod_events]
disabled = 0
host = aa9aa88978a2
index = hotrod_events
indexes = hotrod_events
token = 05c0cde0-ecf0-4576-bd93-819d33529697

Added or extracted fields as metadata

Here the fields have been specified for demonstrative purposes but in a real-world example, extracted from the ingested event:

name: splunk-hec-eventlog
input:
  echo:
    json: true
    ignore-linebreaks: false
    event: |
      {"message":"Hello, World!"}
actions:
  - time:
      output-field: mytime
      output-format: epoch_frac_secs
  - add:
      fields:
      - myhost: myhost.hotrod.local
      - myindex: "hotrod_events"
      - mysource: "pipe"
      - mysourcetype: "hotrod:message"
      - mytoken: 05c0cde0-ecf0-4576-bd93-819d33529697
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    host-field: myhost
    source-field: mysource
    sourcetype-field: mysourcetype
    hec-token-field: mytoken
    time-field: mytime
    remove: true

As before, all specified *-field: my* fields are removed.

An event value as payload

The event payload is the event-field: myevent value:

name: splunk-hec-eventlog
input:
  echo:
    json: true
    ignore-linebreaks: false
    event: |
      {"message":"Hello, World!"}
actions:
  - time:
      output-field: mytime
      output-format: epoch_frac_secs
  - add:
      fields:
      - myevent: Something happened.
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
    host: splunk.hotrod.local
    sourcetype: hotrod:hpr
    source: pipe
    event-field: myevent

note

The below event-field: myevent formats are supported.

actions:
  - add:
      fields:
      - myevent: Something happened.

actions:
  - add:
      fields:
      - myevent: {"message":{"what":"Something happened.","severity":"INFO"}}

warn

host-field, source-field, sourcetype-field, and hec-field are mutually exclusive with host, source, sourcetype and hec-token.

danger

The below event-field: myevent formats are unsupported and result in duplicate host, source, and sourcetype Splunk fields with time not appearing as _time.

actions:
  - add:
      fields:
      - myevent: {"message":{"what":"Something happened.","severity":"INFO"},"host":"foo.hotrod.local","source":"foo","sourcetype":"hotrod:foo","time":1689106057.777}

actions:
  - add:
      fields:
      - myevent: {"message":"Something happened.","severity":"INFO","host":"foo.hotrod.local","source":"foo","sourcetype":"hotrod:foo","time":1689106057.777}

Duplicate source fields problem

The problematic Pipe:

name: splunk-hec-eventlog
input:
  echo:
    json: true
    event: |
      {"message":"Hello, World!","source":"pipe"}
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    host: myhost.hotrod.local
    source: pipe
    sourcetype: hotrod:message
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697

Resulting in duplicate source fields per Splunk event:

The POST event is correctly formatted as shown in the debug log (note the nested and top-level source fields):

2023-07-20T14:21:43.250Z DEBUG pipeline::webhook          > post:
https://splunk.hotrod.local:8088/services/collector/event
{"authorization": "Splunk 05c0cde0-ecf0-4576-bd93-819d33529697", "content-type": "application/json"}
"{\"event\":{\"message\":\"Hello, World!\",\"source\":\"pipe\"},\"host\":\"myhost.hotrod.local\",\"source\":\"pipe\",\"sourcetype\":\"hotrod:message\"}"

According to the Splunk docs: only top-level fields are treated as metadata. So in this case, the 1st, nested, "source":"pipe" field is not a Splunk metadata source field because it is not top-level. This could be a Splunk bug. But, the docs do not explicitly state that this scenario is either supported or unsupported. None of the examples allude to or mention the handling of duplicate top-level and nested, recognised metadata fields. If we ratify the spec, then it stands to reason that it should be supported.

Workaround the duplicate source field problem by renaming the source field to mysource or specifiying source-field: source to remove the nested source field and create a top-level source metadata field instead:

name: splunk-hec-eventlog
input:
  echo:
    json: true
    event: |
      {"message":"Hello, World!","source":"announcement"}
actions:
  - rename:
    - source: mysource 
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
    host: myhost.hotrod.local
    source: pipe
    sourcetype: hotrod:message

name: splunk-hec-eventlog
input:
  echo:
    json: true
    event: |
      {"message":"Hello, World!","source":"pipe"}
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
    host: myhost.hotrod.local
    source-field: source
    sourcetype: hotrod:message
    remove: true

Colon in sourcetype broken

According to Splunk docs, sourcetype does support a colon as a naming convention. sourcetype: hotrod:message causes duplicate field values in Splunk:

...
output:
  splunk-hec:
    url: https://splunk.hotrod.local:8088/services/collector/event
    insecure: true
    hec-token: 05c0cde0-ecf0-4576-bd93-819d33529697
    host: myhost.hotrod.local
    source-field: source
    sourcetype: hotrod:message
    remove: true

source":"pipe:pipe does not cause duplicates.

Workaround: don't use a colon.