Working with Time

Converting Date and Time

the time action can convert between strings and times, and output time in the desired format.

If there is no input time, the current date and time is returned.

- time:
    output-field: '@timestamp`
# {"@timestamp":"2020-04-16T11:28:25.503Z"}

The default output format is ISO UTC with fractional seconds, e.g '2019-02-19T09:27:03.943Z'.

Here are some of the formats used to convert and display date/time:

• 'default_iso' - the default '%Y-%m-%dT%H:%M:%S%.3fZ'
• 'epoch_secs' - seconds since Unix epoch
• 'epoch_msecs' - milliseconds since epoch
• epoch_frac_secs' - seconds since epoch with fractional seconds
• '%Y' - full year, four digits (e.g. 2019)
• '%m' - month number (01-12)
• '%d` - day number (01-31)
• '%F' - ISO year-month-day, short for '%Y-%m-%d'
• '%H' - hour number (00-23)
• '%M' - minute number (00-59)
• '%S' - second number (00-60)
• '%T' - short for '%H:%M:%S'
• '%.3f' - fractional second (e.g. .026)
• '%s' - seconds since Unix epoch
• '%z' offset from local time to UTC (e.g +0200)

The 'epoch_*' shortcuts also ensure that the output value is numerical.

A full list is here.

Getting time in another format:

- time:
    output-format: '%F %z'
    output-field: '@timestamp'
# {"@timestamp":"2019-02-19 +0000"}

(format is an alias for output-format)

The event data may contain time information:

# {"time":1550568549}
- time:
    input-field: time
    input-format: '%s'
    output-format: '%F %z'
    output-field: '@timestamp'
# output:
# {"time":1550568549,"@timestamp":"2019-02-19 +0000"}

Data may contain timestamps in different forms (for instance, logs) so multiple formats can be specified with input-formats:

- time:
    input-field: tspec
    input-formats:
    - epoch_secs
    - default_iso

Note that all output times are in UTC by default.

Classifying Time of Day

time when is a classifier; if the time matches a range of time intervals, then a tag named by output-tag will be added to the record. An input time needs to be provided.

For example:

# json: {"@timestamp":"2018-07-09T15:09:06.479Z"}
- time:
    input-field: '@timestamp'
    when:
        - 'mon-fri:16:00-20:00'
        - 'sat:10:00-11:00'
    output-tag: tag=busy
# output: {"@timestamp":"2018-07-09T15:09:06.479Z","tag":"busy"}

time-range-start-field and time-range-length-field can be specified with when, which gives the time (in seconds since epoch, Unix-style) of the start of the range and the length in seconds of the range. So the first could be the start of the working day, and the second is the length of the working day.

when can either be a list of time ranges or a string with time ranges separated by commas. This is useful if the time ranges are available as a field, e.g. after a lookup with enrich.

- time:
    when: 'mon-fri:09:00-17:00,sat:09:00-11:00'

This form of time when allows for field expansions like ${sunday}.

You can use output-fields instead of output-tag which works like add output-fields.

Timezones

By default, output times are assumed to be UTC, and input times are local (unless the input format is 'default_iso' which is explicitly UTC.)

Use input-timezone and output-timezone to be completely explicit. The timezones are in IANA format

For Docker pipes (used on hotrod-server agent), the default timezone is UTC. If you want to change to something else, you'd have to add this to your pipe definition:

context:
  docker-compose:
    environment:
      - TZ=Africa/Johannesburg

When Only Time is Available

time insists that the input is valid date + time. This causes a problem if the input time lacks an explicit date, as in:

$ uptime
 11:34:17 up 2 days,  2:12,  1 user,  load average: 0.35, 0.35, 0.32

The workaround is to find out the date and combine it with the time so it is a full date-time.

(This is not sensible if there is a serious lag between when the data was generated and when it was processed, of course.)

# {"time":"11:34:17"}
# get the date part
- time:
    output-field: date
    format: '%Y-%m-%d'
# combine the date with the time
- add:
    output-fields:
    - full-time: '${date} ${time}'
# and parse this new field
- time:
    input-field: full-time
    input-format: '%Y-%m-%d %H:%M:%S'
    output-field: '@timestamp'   

This is a little tedious, but it shows the power of combining operations using pipes.