Monitors

A monitor is a [[monitors]] entry with an id, a name, a kind and a probing cadence (interval_secs, timeout_secs). Failed probes are retried once (after 1s) before anything is recorded - override with probe_retries = 0..5 - so a one-off network blip never pollutes the history or the error budget. Every failure that survives its retry is logged with its reason and shown as a tooltip on the status dot (plus last_error in the API).

Kinds

http (default)

target is a URL; up = 2xx, or exactly expected_status if set. TLS certificate expiry is checked automatically for https:// targets.

[[monitors]]
id = "api"
name = "API"
target = "https://api.example.com/health"
interval_secs = 30
timeout_secs = 10
# expected_status = 200
# degraded_over_ms = 800    # up but slower than this = degraded (yellow)
# headers = { Authorization = "Bearer ${TOKEN}" }
# proxy = "socks5://127.0.0.1:9050"

Body assertions turn a reachability check into a correctness check:

keyword = "operational"   # body must contain this (keyword_invert = true → must NOT)
json_query = "$.status"   # JSONPath (RFC 9535) against a JSON body
json_expected = "ok"      # the queried value must equal this (omit = must match a node)
number_regex = 'in stock: (\d+)'  # extract a number from the body (first capture
                          # group, or the whole match) - down when nothing matches
number_min = 1            # ... or when the number is below this (inclusive)
number_max = 500          # ... or above this (inclusive)
max_body_kb = 256         # cap on the body read for assertions (default 1 MiB)

number_regex is the assertion for pages that render a gauge inline with no JSON endpoint behind them - a queue depth, a stock level, a sensor’s “receivers online” counter. When the page is HTML, anchor the pattern on the surrounding markup: number_regex = 'class="shipCount">(\d+)<' with number_min = 1 goes down the moment the counter reads zero (or disappears entirely).

Redirects are followed (up to 10), but configured headers - which may carry credentials - are only re-attached while the redirect stays on the original origin, so a compromised target can’t bounce your API key to another host.

tcp

target is host:port; up = the TCP connect succeeds.

[[monitors]]
id = "database"
name = "Database"
kind = "tcp"
target = "db.example.com:5432"
interval_secs = 60
timeout_secs = 5

icmp

target is a host or IP (no port); up = an echo reply within the timeout. Uses an unprivileged datagram socket - no CAP_NET_RAW, rootless-Docker friendly, IPv4 and IPv6. If the socket is unavailable the monitor reports down with a clear reason naming the sysctl to fix (net.ipv4.ping_group_range).

dns

target is a hostname to resolve. Without dns_expected any non-empty answer counts as up - CDN and round-robin answers rotate constantly, so alerting on mere change would flap. With it, the answer is pinned (hijack detection), compared order-insensitively:

[[monitors]]
id = "dns-check"
name = "DNS Check"
kind = "dns"
target = "example.com"
interval_secs = 300
dns_record = "A"             # A (default), AAAA, CNAME, MX, NS, TXT, SRV, SOA, PTR
dns_expected = "1.2.3.4"     # comma-separated, order-insensitive
dns_resolver = "8.8.8.8:53"  # optional; default: system resolver

exec

Run any external check following the monitoring-plugins convention: exit 0 = up, 1 = degraded, anything else = down; the first stdout line is the message (the |perfdata tail is stripped). The whole Nagios/Icinga plugin ecosystem - thousands of checks: RAID, disks, exotic certificates, SNMP - becomes usable from Hora.

[[monitors]]
id = "raid"
name = "RAID"
kind = "exec"
command = ["check_raid", "--no-sudo"]   # a raw argv - no shell, ever
interval_secs = 300
timeout_secs = 30                       # the plugin is SIGKILLed past this

The security model is the HORA_EXEC_DIR environment variable - deliberately not a config key. The config is hot-reloaded, so it alone must never be able to run code: enabling exec probes takes deployment-level access on top of config access. With the variable set:

• command[0] is resolved strictly inside that directory - a bare file name, validated at load, canonicalized at run time (a symlink planted in the directory that points outside is refused);
• no shell is involved: command is an argv array, so nothing is ever interpolated or injected;
• the plugin gets a scrubbed environment (PATH, HOME, LANG, TZ): the daemon’s own environment carries your notification tokens, and no plugin has any business reading them;
• output is bounded and drained (a chatty plugin never deadlocks on a full pipe), and a stuck plugin is killed at the monitor’s timeout.

Without HORA_EXEC_DIR, a config declaring an exec monitor fails at load - and hora doctor verifies the directory and that every configured plugin is actually present.

Recipe: watching other containers from a rootless Hora

In rootless Docker, mounting the Docker socket only grants your user’s rights - not root on the host - which makes this pattern reasonable:

# checks/check_container - exit 0 if the container reports healthy
#!/bin/sh
state=$(curl -sf --unix-socket /var/run/docker.sock \
  "http://localhost/containers/$1/json" | jq -r '.State.Health.Status // .State.Status')
case "$state" in
  healthy|running) echo "container $1 $state"; exit 0 ;;
  *) echo "container $1 $state"; exit 2 ;;
esac

docker run -d --name hora \
  -v ./hora-config:/etc/hora \
  -v ./checks:/etc/hora/checks:ro \
  -e HORA_EXEC_DIR=/etc/hora/checks \
  -v $XDG_RUNTIME_DIR/docker.sock:/var/run/docker.sock:ro \
  my-hora-with-tools

with a one-line derived image for the tools your scripts need:

FROM ghcr.io/uplg/hora:latest
RUN apk add --no-cache curl jq monitoring-plugins

monitoring-plugins alone brings check_disk, check_load, check_smtp, check_snmp and friends - each one a command = ["check_..."] away from uptime bars, SLOs and the weekly digest.

Exec monitors take no target, are excluded from multi-vantage confirmation (a local check has no remote vantage), and their failure detail collapses to “plugin check failed” for anonymous viewers unless public_error_detail opts in - plugin output names devices and container ids.

push (heartbeat)

No target - the job calls Hora. Down when no ping arrives within interval_secs:

curl -fsS -X POST -H "X-Push-Token: ${BACKUP_TOKEN}" \
  "https://status.example.com/api/push/nightly-backup"

The cron-aware variant declares when the job runs, and alerts only when a scheduled run misses its grace window - a 03:00 backup pinging at 03:05 is fine, one still silent at 03:30 + grace is down:

[[monitors]]
id = "nightly-backup"
name = "Nightly backup"
kind = "push"
interval_secs = 60           # with `schedule`, just the re-evaluation cadence
push_token = "${BACKUP_TOKEN}"
schedule = "0 3 * * *"       # five-field cron, UTC
grace_secs = 1800            # how late a ping may be (default 30m)

Optional query parameters: ?status=up|down|degraded, msg=... (recorded with the heartbeat), ping=<ms> (round-trip latency).

Dual-stack verification

dual_stack = true

probes IPv4 and IPv6 separately (concurrently) and requires both: the classic silent failure is a service whose IPv6 has been dead for weeks behind a healthy IPv4 (or the reverse), invisible to every single-connection check. One broken family confirms down with the culprit named - “IPv6 failing: connection timed out (IPv4 ok)” - and the surviving family’s latency recorded; when both answer, the recorded latency is the slower path’s, so degraded_over_ms judges the worst case.

Works for http, tcp and icmp monitors with a hostname target (an IP literal has a single family); cannot be combined with proxy.

The probing host needs both families

In Docker, default bridge networks have no IPv6 - a dual-stack monitor would blame your container’s network, not your service. Enable IPv6 on the daemon/compose network (or use host networking) first.

Topology: groups and dependencies

group = "infra"              # display group on the status page
depends_on = ["db", "cache"] # upstream monitors this one depends on

When a monitor goes down its alert is annotated with root cause vs. symptom: “caused by X” when an upstream it depends on is also down, or “impacts: A, B, C” (the blast radius) when its upstreams are all healthy and it is the root cause. The dependency graph is validated acyclic at load. Dependencies also drive root-cause alert grouping.

TLS certificates

For https:// monitors, expiry is checked automatically and warned alerts.cert_expiry_days in advance (default 14). Optionally pin the public key:

cert_pin = "abc123..."   # SHA-256 of the leaf public key

An unexpected key change - MITM, botched renewal - alerts once per change, with the old and new fingerprints. Disable expiry checking per monitor with check_cert = false.

Mail servers get the same watch: set starttls = "smtp" (or "imap") on a tcp monitor and the watcher negotiates STARTTLS before reading the certificate.

kind = "tcp"
target = "mail.example.com:25"
starttls = "smtp"
ehlo_name = "status.example.com"   # optional

SMTP needs a client identity in EHLO. RFC 5321 requires a fully qualified domain or an address literal, and MX servers enforce it on port 25 (550 Invalid EHLO domain). By default Hora announces the address literal of its end of the connection ([192.0.2.10]), which is valid even in a container whose hostname is a bare id. Set ehlo_name to announce a real name instead.

Domain expiry (RDAP)

The natural sibling of the TLS warnings: “your domain expires in 14 days”.

domain_expiry = "example.com"

The registered domain (not the subdomain the target uses - that’s why it is explicit rather than derived) is checked once a day against the registry over RDAP - JSON over HTTP, no whois parsing - and an alert fires alerts.domain_expiry_days (default 14) before it expires, with the same edge-triggered, maintenance-muted policy as certificates.

Visibility

public = false               # hide from unauthenticated viewers
public_error_detail = true   # publish raw failure reasons to anonymous viewers

A monitor with public = false disappears from the unauthenticated status page, API, badges and history (its latency endpoint answers 404, exactly like a missing monitor); the viewer token (server.auth_token) reveals the full view. By default anonymous viewers see only a safe category of a failure (“HTTP 500”, “content check failed”) - the stored detail can carry response snippets, DNS answers or asserted keywords; public_error_detail = true opts a monitor into publishing the full reason (and the failure snapshot).

Latency and SLO fields

degraded_over_ms = 800   # up but slower = degraded
slo_latency_ms = 500     # the 24h p95 is flagged met/breached against this
slo_uptime = 99.9        # availability SLO; shows error budget, arms burn-rate alerts
slo_window_days = 30     # SLO window (default 30)
retention_days = 30      # how long this monitor's raw history is kept

See SLOs & error budgets for how the budget and burn-rate alerts work.