Skip to main content

Overview

Hotrod is a data engineering platform that simplifies creating, managing, running and scaling data processing pipelines.

What is Data Engineering?​

Data engineering describes the technical and operational domain of handling data. It typically includes:

  • Collection & ingestion
  • Cleaning & validation
  • Transformation
  • Moving data between sources and destinations
  • Building and managing data processing pipelines for the above tasks

What is a Data Processing Pipeline?​

A data processing pipeline moves data from a data source to a data destination, often with some intermediate processing or transformation.

This idea represents most kinds of data processing and system integrations. In today's data-driven world, capabilities around collection, processing, transforming and ingesting datasets, feeds and streams is becoming increasingly important for orgnanizations large and small.

What is Hotrod?​

Hotrod is a platform that simplifies a variety of data engineering problems, through providing:

  • A consistent and simplified means to define data processing Pipelines
  • A runtime to execute those Pipelines
  • Agents to run Pipelines in a distributed manner, and which enables scaling and data processing at the edge
  • A Server that centrally manages Agents and Pipelines

Hotrod was originally developed to solve two particular challenges well:

  • Collecting and processing data from many geographically distributed systems, servers and networks
  • Centrally ingesting that data into data analysis systems

Hotrod has since become a general purpose data engineering platform. It offers data engineers the leverage to solve their problems faster and more consistently, while also reducing the complexity of handling infrastructure, code and related peripheral details.

The fundamental ideas behind Hotrod's design are:

Versatility
Data engineering problems are continuously changing. Therefore, data engineers need versatile tools that can adapt to changing needs. Being able to create your own connectors is more important than having some standard set of connectors.

Simplify Data Engineering
Hotrod provides a higher-level interface to handle data processing workloads. This allows data engineers to focus on outcomes, rather than handling implementation details. The Hotrod Pipe Runtime handles the latter.

Data Ownership
Hotrod is a self-hosted platform. You can run it anywhere you like; cloud, on-prem or on-site, or any combination of these. No third parties are involved in your data engineering workloads, giving you maximum control over the management of your data.

Process at the Edge
Large-scale data engineering problems often benefit from the ability to move data collection and processing out over many servers, sometimes geographically distributed. Hotrod simplifies running and managing data engineering workloads at the edge.

Hotrod focuses narrowly on data engineering solutions. As such, it's not a data analytics solution, nor a data store or database. While Hotrod is not a replacement for either, it can significantly simplify moving data between such systems.