Working with Time

Converting Date and Time

The time action can convert between strings and times as well as output time, in the desired format.

If there is no input time, the current date and time is returned:

- time:
    output-field: '@timestamp`

# Output: {"@timestamp":"2020-04-16T11:28:25.503Z"}

The default output format is ISO UTC with fractional seconds, e.g., 2019-02-19T09:27:03.943Z.

The various formats used to convert and display date and time are:

• 'default_iso' - the default '%Y-%m-%dT%H:%M:%S%.3fZ'
• 'epoch_secs' - seconds since Unix epoch
• 'epoch_msecs' - milliseconds since epoch
• 'epoch_frac_secs' - seconds since epoch with fractional seconds
• '%Y' - full year, four digits (e.g. 2019)
• '%m' - month number (01-12)
• '%d` - day number (01-31)
• '%F' - ISO year-month-day, short for '%Y-%m-%d'
• '%H' - hour number (00-23)
• '%M' - minute number (00-59)
• '%S' - second number (00-60)
• '%T' - short for '%H:%M:%S'
• '%.3f' - fractional second (e.g. .026)
• '%s' - seconds since Unix epoch
• '%z' offset from local time to UTC (e.g +0200)

The epoch_* shortcuts ensure a numerical output value.

Find the full list of date and time conversion formats here.

Here’s how to get time in another format (format is an alias for output-format):

- time:
    output-format: '%F %z'
    output-field: '@timestamp'
# {"@timestamp":"2019-02-19 +0000"}

The event data may contain time information, as illustrated below:

# Input: {"time":1550568549}

- time:
    input-field: time
    input-format: '%s'
    output-format: '%F %z'
    output-field: '@timestamp'

# Output: {"time":1550568549,"@timestamp":"2019-02-19 +0000"}

Note that the data may contain timestamps in different forms (for instance, logs). Multiple formats can be specified using input-formats:

- time:
    input-field: tspec
    input-formats:
    - epoch_secs
    - default_iso

All output times default to UTC.

Classifying Time of Day

The command time.when is known as a classifier. If the specified time matches a range of time intervals, a Tag named by output-tag is added to the record. An input time must also be provided, for example:

# Input: {"@timestamp":"2018-07-09T15:09:06.479Z"}

- time:
    input-field: '@timestamp'
    when:
        - 'mon-fri:16:00-20:00'
        - 'sat:10:00-11:00'
    output-tag: tag=busy

# Output: {"@timestamp":"2018-07-09T15:09:06.479Z","tag":"busy"}

time-range-start-field and time-range-length-field can be specified using the when command, which gives the time (in seconds since the epoch) of the start and length of the range.

time.when can be either a list of ranges or a string of comma-separated ranges. This is useful if the ranges are available as a field, e.g., after a lookup with enrich:

- time:
    when: 'mon-fri:09:00-17:00,sat:09:00-11:00'

This form of time.when allows for Field Expansion like ${sunday}.

Here, you can use output-fields instead of output-tag.

Timezones

As stated previously, output times default to UTC and input times are local (except in cases where the input format is default_iso which is explicitly UTC.)

To specify output and input times, use input-time zone and output-time zone. These timezones are in IANA format.

The default time zone for Docker Pipes (used by the built-in Agent) is UTC. Changing this requires additions to your Pipe definition:

context:
  docker-compose:
    environment:
      - TZ=Africa/Johannesburg

When Only Time is Available

time insists that the input is a valid date and time. This can be an issue if the input time lacks an explicit date, as seen below:

$> uptime

11:34:17 up 2 days,  2:12,  1 user,  load average: 0.35, 0.35, 0.32

To remedy this, find the date and combine it with the time so it becomes a full date-time format.

note

This workaround is not feasible in instances where there is serious lag between when the data was generated and when it was processed.

Below, we get the date part, combine the date and time, and parse the new field:

# Input: {"time":"11:34:17"}

# Get the date part.
- time:
    output-field: date
    format: '%Y-%m-%d'

# Combine date and time.
- add:
    output-fields:
    - full-time: '${date} ${time}'

# Parse this new field.
- time:
    input-field: full-time
    input-format: '%Y-%m-%d %H:%M:%S'
    output-field: '@timestamp'