Skip to main content
Version: 3.4.0

Hotrod Architecture

This section is an overview of Hotrod's design. As a distributed system, Hotrod as an architecture that enables processing data at scale, while focusing on keeping the management thereof as simple as possible.

Hotrod Components​

Here's an overview of Hotrod's components, and how they work together.

Pipes​

Pipes are the foundation of Hotrod. A Pipe - or a data processing pipeline - represents a specific data processing workload, and is described with the Hotrod Pipe Language.

Agents​

Hotrod Agents are the runtime for Pipes. An Agent schedules and executes them. Agents can run anywhere - on VM's in the cloud, or in your own on-premise environment or local network.

Server​

Hotrod Server coordinates the allocation of Pipes to Agents, and provides other centralized management and monitoring capabilities, through CLI, HTTP API, and a web-based UI that includes a pipe editor to design, test and deploy pipes.

Both the CLI and UI utilizes Hotrod Server's HTTP API, which can be used directly for automation and introspection into Hotrod from other systems.

Deployment Model​

A Hotrod deployment consists of a single Hotrod Server, which manages one or more Hotrod Agents. A single server can manage hundreds of agents, which are in turn responsible for executing pipes.

Agents can be co-located with Hotrod Server and other agents. They can also run on another machine, or a container.

Hotrod is deployed as a single binary. The same binary is used for running both Hotrod Server, Hotrod Agents, and Hotrod CLI, all of which are configured through environment variables and/or CLI arguments on startup.

Agent-Server Communication​

Every Hotrod Agent is configured to attach to one Hotrod Server. An agent communicates with its server via HTTP (with TLS). Agents periodically communicates with the Hotrod Server they are attached to, in order to:

  • Report the state of pipes they are running
  • Check for configuration changes which includes:
  • Send logs and metrics

The frequency of this Agent-Server communication is configurable, and defaults to 5 seconds. Hotrod Server does not initiate requests to agents. Therefore, agents do not need to be network-accessible from Hotrod Server. But Hotrod Server's HTTP API must be network-accessible from agents.

Agent-Server Authentication​

Hotrod Server always enforces authentication for any agents attempting to communicate with it. There are two ways in which Hotrod Server can authenticate agents.

Agent API Keys​

This is the default behaviour. Each agent must be added to Hotrod Server first, before the server will allow incoming communication from the agent. Additionally, an API must be created on Hotrod Server, for use on one or more agents. Agents are then configured with an Agent ID and an API key at startup.

Auto-Enrolment​

In some cases and deployment scenarios, manually adding agents to Hotrod Server are not desirable. Auto-enrolment provides an alternative authentication scheme. Hotrod Server is configured with an auto-enrolment key at startup. This same key is then used for all agent configuration. All agents then use this auto-enrolment key for authentication with Hotrod Server. Hotrod Server will automatically create internal agent entries for new agents.