Skip to main content
Version: 3.4.0

azure-blob

Send data to a Microsoft Azure Storage Blob (Block Storage)

Available from Hotrod: 3.1

Field NameDescriptionTypeDefault
intervalHow often to run the commandduration-
cronHow often to run the command. Note that Hotrod uses a different format than Cron it includes a column for seconds. See full discussioncron-
immediateRun as soon as invoked, instead of waiting for the specified cron intervalboolfalse
random-offsetSets a random offset to the schedule, then sticks to itduration0s
windowFor resources that need a time window to be specifiedWindow-
blockBlock further input schedules from triggering if the pipe output is retryingboolfalse
container-nameThe storage service container for created blobsstring-
blob-namesThe name for the blobarray of strings-
blob-name-fieldThe field that a blob name from an operation should be stored infield-
creation-time-fieldThe field that the blob creation time should be stored infield-
last-modified-fieldThe field that the blob last modified time should be stored infield-
content-length-fieldThe field that the blob content length information should be stored infield-
content-type-fieldThe field that the blob content type information should be stored infield-
content-md5-fieldThe field that the blob content md5 should be stored infield-
data-fieldA field that the blob data should be nested infield-
storage-accountThe Storage Account Name to be used (credential)string-
storage-master-keyThe Storage Master Key to be used (credential)string-
timestamp-modeDerive a timestamp for this blob for filtering purposes based on the selected strategy.AzureBlobTimestampMode-
maximum-ageRemove any blobs older than this many seconds from the candidate listMaxAgeSpecifier-
modeThe operating mode for this inputAzureBlobInputMode-
fingerprintingEnable object fingerprinting, which will cause a object to only be downloaded onceboolfalse
maximum-fingerprint-ageRemove any object fingerprints older than this from the trackerMaxAgeSpecifier30 days

interval

How often to run the command

By default, interval: 0s which means: once. Note that scheduled inputs set document markers. See full discussion

Type: duration

Example

action:

exec:
command: echo 'once a day'
interval: 1d

cron

How often to run the command. Note that Hotrod uses a different format than Cron it includes a column for seconds. See full discussion

Type: cron

Example: Once a day

action:

exec:
command: echo 'once a day'
cron: '0 0 0 * * *'

Example: Once a day, using a convenient shortcut

action:

exec:
command: echo 'once a day'
cron: '@daily'

immediate

Run as soon as invoked, instead of waiting for the specified cron interval

Type: bool

Example: Run immediately on invocation, and thereafter at 10h every morning

action:

exec:
command: echo 'hello'
immediate: true
cron: '0 0 10 * * *'

random-offset

Sets a random offset to the schedule, then sticks to it

This can help avoid the thundering herd problem, where you do not, for example, want to overload some service at 00:00:00

Type: duration

Example: Would fire up to a minute after every hour

action:

exec:
command: echo 'hello'
random-offset: 1m
cron: '0 0 * * * *'

window

For resources that need a time window to be specified

Type: Window

Field NameDescriptionTypeDefault
sizeWindow sizeduration-
offsetWindow offsetduration0s
start-timeAllows the windowing to start at a specified timetime-
highwatermark-fileSpecify file where timestamp would be stored in order to resume, for when Pipe has been restartedpath-

size

Window size

Type: duration

Example

action:

exec:
command: echo 'one two'
window:
size: 1m

offset

Window offset

Type: duration

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
offset: 10s

start-time

Allows the windowing to start at a specified time

It should in the following format: 2019-07-10 18:45:00.000 +0200

Type: time

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
start-time: 10s

highwatermark-file

Specify file where timestamp would be stored in order to resume, for when Pipe has been restarted

Type: path

Example

action:

exec:
command: echo 'one two'
window:
size: 1m
highwatermark-file:: /tmp/mark.txt

block

Block further input schedules from triggering if the pipe output is retrying

Type: bool

container-name

The storage service container for created blobs

Type: string

blob-names

The name for the blob

Type: array of strings

blob-name-field

The field that a blob name from an operation should be stored in

Type: field

creation-time-field

The field that the blob creation time should be stored in

Type: field

last-modified-field

The field that the blob last modified time should be stored in

Type: field

content-length-field

The field that the blob content length information should be stored in

Type: field

content-type-field

The field that the blob content type information should be stored in

Type: field

content-md5-field

The field that the blob content md5 should be stored in

Type: field

data-field

A field that the blob data should be nested in

Type: field

storage-account

The Storage Account Name to be used (credential)

Type: string

storage-master-key

The Storage Master Key to be used (credential)

Type: string

timestamp-mode

Derive a timestamp for this blob for filtering purposes based on the selected strategy.

Type: AzureBlobTimestampMode

Field NameDescriptionTypeDefault
noneThe default mode, do not filter blobs based on timestamps--
creation-timeFilter blobs on the creation-time timestamp reported by the service--
last-modifiedFilter blobs on the last-modified timestamp reported by the service--
blob-name-patternFilter blobs on the timestamp derived from the blob name for example: blob-name-pattern: =(?P<Y>[\\d]{4,4})-(?P<m>[\\d]{2,2})-(?P<d>[\\d]{2,2})/ string-

none

The default mode, do not filter blobs based on timestamps

creation-time

Filter blobs on the creation-time timestamp reported by the service

last-modified

Filter blobs on the last-modified timestamp reported by the service

blob-name-pattern

Filter blobs on the timestamp derived from the blob name for example: blob-name-pattern: =(?P<Y>[\\d]{4,4})-(?P<m>[\\d]{2,2})-(?P<d>[\\d]{2,2})/

Type: string

maximum-age

Remove any blobs older than this many seconds from the candidate list

Type: MaxAgeSpecifier

Field NameDescriptionTypeDefault
secondsSpecify the maximum age in number of secondsinteger-
durationSpecify the maximum age as a human readable duration (example: 1 hour)string-

seconds

Specify the maximum age in number of seconds

Type: integer

duration

Specify the maximum age as a human readable duration (example: 1 hour)

Type: string

mode

The operating mode for this input

Type: AzureBlobInputMode

Field NameDescriptionTypeDefault
list-blobsList Blobs--
download-blobDownload Given Blobs--
list-and-download-blobsList Blobs and Download--

list-blobs

List Blobs

download-blob

Download Given Blobs

list-and-download-blobs

List Blobs and Download

fingerprinting

Enable object fingerprinting, which will cause a object to only be downloaded once

Type: bool

maximum-fingerprint-age

Remove any object fingerprints older than this from the tracker

Type: MaxAgeSpecifier

Field NameDescriptionTypeDefault
secondsSpecify the maximum age in number of secondsinteger-
durationSpecify the maximum age as a human readable duration (example: 1 hour)string-

seconds

Specify the maximum age in number of seconds

Type: integer

duration

Specify the maximum age as a human readable duration (example: 1 hour)

Type: string