Agent Overview
Hotrod Agents are lightweight processes responsible for providing the distributed runtime environment for Pipes. Each Agent is configured with a single upstream Server.
Agents can be installed on any supported operating system. Agents must be able to reach the HTTP API of their configured Server. Agents have practically no maintenance beyond their initial configuration, apart from the occasional version update, as and when required.
The Server features an internal Agent, which is useful for testing, as well as light workloads.
From version 3.5 the internal Server Agent is always enabled when starting the Server (with hotrod run server).
Prior to version 3.5 the internal Server Agent is optional, and may be enabled with the command hotrod run stand-alone.
Identifying And Authenticating Agents
The Server identifies Agents via a unique ID, which may have an associated name. All communication between Agent and Server are authenticated. Different authentication methods are available, to support both manual and automated provisioning of Agents.
Pipe Assignment and Deployment
The Server is responsible for assigning and deploying Pipes to Agents. Pipe assignment can be done either manually, or automatically via a tag-based mechanism. The Server is further responsible for generating and providing Pipe payloads to Agents, which may include additional files. A signature-based mechanism provides security guarantees that Agents run the correct Pipe definitions.
New versions of Pipe definitions are automatically and transparently redeployed to Agents, when they are changed on the Server.
The Server does not directly communicate with Agents. It provides each Agents with a manifest of Pipes assigned to it. The Agent then retrieves and synchronizes it's Pipes from the Server, based on the Server-provided manifest.
Pipe Scheduler
Agents are primarily a scheduler for running Pipes, with an internal scheduler. Agents can also be configured to leverage systemd for running Pipes, in cases where systemd is available on the Agent's host machine.
Agent Telemetry
An Agent forwards telemetry to their upstream Server, in the form of logs, metrics and other metadata, both from within the Agent and from the Pipes that the Agent runs.
The data processed by a Pipe does not traverse the Server in any way. The Pipe only generates telemetry and metadata about the data it processes internally, which it sends to the Server via the Agent it runs on.