Server Setup

The HOTROD executable is installed and available on the system, set up the Server as follows:

1. Create a system account.

2. Create a data directory.

3. Create systemd files.

4. Start the Server.

By the end of this section, you should be able to access the Server via a browser.

Create a system account

Create a system account for the Server to run under:

$> sudo adduser --system --home /var/lib/hotrod-server --disabled-login --group hotrod

danger

Do not run the Server with root user privileges.

Create a Data Directory

The Server requires a data directory to store Pipes, logs, and metric data.

The hotrod user home directory is /var/lib/hotrod-server and it will also serve as the data directory.

danger

Secure environments require 0700 permissions on the data directory!

If a different data directory is required, create it with the appropriate ownership and permissions. For example:

$> sudo mkdir /data/hotrod

$> sudo chown hotrod:hotrod /data/hotrod

Create systemd Files

Create a systemd service unit file:

$> sudo vi /etc/systemd/system/hotrod-server.service

The file must contain the following:

[Unit]
Description=Hotrod Server
After=network.target auditd.service

[Service]
EnvironmentFile=/etc/default/hotrod-server
User=hotrod
Group=hotrod
ExecStart=/usr/sbin/hotrod run server
Restart=on-failure
RestartSec=60

[Install]
WantedBy=multi-user.target

note

From v3.5.0, running the built-in Agent is mandatory.

Create an environment file for the EnvironmentFile setting:

$> sudo vi /etc/default/hotrod-server

Here, the Server is configured through either hotrod run server options or environment variables. In this case, we'll be using the latter.

tip

See hotrod run server --help for startup options and their environment variable equivalents and the reference.

At a minimum, the Server needs HOTROD_STAGING_DIR:

HOTROD_STAGING_DIR=/var/lib/hotrod-server
HOTROD_LICENSE_EULA_ACCEPT=yes
HOTROD_ADMIN_INIT_PASSWORD=ChangeMeVerySoon

note

If hotrod user home directory is not HOTROD_STAGING_DIR, set accordingly.

We've added 2 additional environment variables:

• HOTROD_LICENSE_EULA_ACCEPT=yes prevents the one-time prompt for accepting the EULA.

• HOTROD_ADMIN_INIT_PASSWORD provides an initial password for the Server admin user.

Upon first initialization of the Server user database, if HOTROD_ADMIN_INIT_PASSWORD is unset, a random password will be generated in the Server STDOUT output (see journalctl -u hotrod-server).

note

Whether using HOTROD_ADMIN_INIT_PASSWORD, or the randomly generated password on first run, change the admin password immediately!

note

Besides the configured data directory (HOTROD_STAGING_DIR), the Server will persist some minimal state information to the home directory of the user that the Server runs as (~/.local/share/hotrod*/). This directory must be writable by the Server.

Once you have saved the service unit file, reload systemd:

$> sudo systemctl daemon-reload

To start the Server at boot, enable the service with:

$> sudo systemctl enable hotrod-server

Finally, start the Server:

$> sudo systemctl start hotrod-server

Verify that the Server started successfully:

$> systemctl status hotrod-server

It's a good idea to inspect the startup output, which will contain the admin user password if it wasn't set with HOTROD_ADMIN_INIT_PASSWORD:

$> journalctl -u hotrod-server

The Server will be listening on HOTROD_BIND_ADDRESS (default 127.0.0.1:3000).

tip

The Server can be configured to use TLS, by providing a certificate and key file. See hotrod server run --help. When using a reverse proxy for TLS termination, like Caddy or Nginx, it's recommended to configure the appropriate HTTP client address headers, for logging purposes.

If your cert uses subjectAltName, you must have an entry matching the cert CN. In the below CSR it is server.hotrod.local:

openssl req -new -nodes -out server.hotrod.local.csr -newkey rsa:4096 -keyout server.hotrod.local.key -subj '/CN=server.hotrod.local/C=ZA/ST=Gauteng/L=Johannesburg/O=Hotrod'

The matching entry DNS.1 = server.hotrod.local:

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = server.hotrod.local
IP.1 = 192.168.235.10

If no subjectAltName, the CN will suffice for successful cert verification, else an error occurs (see journalctl -u hotrod-agent):

Jun 21 20:17:51 splunk hotrod[13738]: 2023-06-21T20:17:51.613Z ERROR hotrod > Failed to start agent RunAgentError("Failed to instantiate agent id: AgentInitialisationError("Failed to auto enroll agent: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("server.hotrod.local")), port: Some(3002), path: "/api/v1/agents/enroll", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 337047686, library: "SSL routines", function: "tls_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1921 }]))) }, X509VerifyResult { code: 62, error: "**_Hostname mismatch_**" })) }")")

TL; DR: X509VerifyResult { code: 62, error: "**_Hostname mismatch_**" }.

When testing with curl -v, the output is indicative of what failed:

* subjectAltName does not match server.hotrod.local

openssl s_client works without issues.

Go to http://localhost:3000 in a browser. Log in with the username admin and the appropriate password.

At this point, the Server is ready to start serving Agents.