Operations & Self-Healing

The server is built for crash safety on three levels — and for a persistently unhealthy process to terminate itself so the supervisor brings up a fresh one.

Level 1: panics stay local

• A panic in an HTTP handler is contained: the request ends as a 500, the stack trace goes to the log, the process keeps serving.
• A panic in a background job (propagation, reminders, certificates, …) becomes an ordinary error: the job gets the normal attempt backoff (30 s up to 30 min) and runs again later — a broken job can neither hot-loop nor take the process down.
• State is event-sourced in the database: even a hard crash loses no confirmed change; projections and reactors resume from their checkpoint after the restart.

Level 2: health endpoints

Two public endpoints (deliberately unauthenticated — they only reveal up/down) for supervisors and load balancers:

Endpoint	Meaning	Response
GET /healthz	process serves requests (liveness)	200 ok
GET /readyz	database answers (readiness)	200 or 503 with an error

Level 3: watchdog — self-restart on instability

A built-in watchdog probes the process from the inside: database ping, optional heap and goroutine ceilings. Single hiccups are only logged; only several consecutive failures terminate the process cleanly with exit code 2 — the supervisor restarts it (crash-only pattern: a fresh process is the most reliable way back to a known state).

Variable	Meaning	Default
DNSE_WATCHDOG	watchdog active	on
DNSE_WATCHDOG_INTERVAL	probe interval	30s
DNSE_WATCHDOG_FAILS	consecutive failures before restart	4
DNSE_WATCHDOG_HEAP_MB	Go heap ceiling in MB (0 = off)	0
DNSE_WATCHDOG_GOROUTINES	goroutine ceiling (0 = off)	0

Note

The watchdog only terminates the process — the restart is the supervisor’s job. Without Restart=always (systemd) or restart: unless-stopped (Docker) the service stays down after a watchdog exit.

Setting up the restart policy

Tip

Installing from a package ships exactly this unit ready to go — hardening and Restart=always included. This section is for installing from the bare binary.

systemd

[Unit]
Description=DNS-Editor
After=network-online.target

[Service]
ExecStart=/usr/local/bin/dnseditor
EnvironmentFile=/etc/dnseditor/env
Restart=always
RestartSec=3
# optional: dampen restart storms
StartLimitIntervalSec=120
StartLimitBurst=10

[Install]
WantedBy=multi-user.target

Docker / Compose

services:
  dnseditor:
    image: <your-image>
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8422/readyz"]
      interval: 30s
      timeout: 5s
      retries: 3

Verification commands

curl -s http://127.0.0.1:8422/healthz
curl -s http://127.0.0.1:8422/readyz

Watchdog messages appear in the log as watchdog: self-check failed (one warning per failure) and watchdog: process unhealthy - exiting for supervisor restart right before the self-restart.

With a German system language they read Watchdog: Selbstpruefung fehlgeschlagen and Watchdog: Prozess ungesund - Beenden fuer den Neustart durch den Supervisor. Which language applies, and why the German texts avoid umlauts, is explained under installing from a package.

Organisation backup

With DNSE_BACKUP_DIR set, the service writes a nightly full logical export of the organisation (JSON Lines) into that directory — the restore points. Points older than 35 days are pruned automatically. Without the variable, backups are off and the admin tab does not appear.

In the admin panel under Backup, points can be inspected, created on demand with Back up now (e.g. before risky changes) and downloaded. A restore is requested there with the confirmation word RESTORE; a safety export of the current state is taken automatically right before.

The restore executes: the organisation is briefly halted (writes are rejected), the selected point replaces the data completely — secrets are re-encrypted with the current key, derived views are rebuilt from the events — and the organisation continues seamlessly afterwards. If a restore aborts, the organisation stays recognisably incomplete (writes fail with a clear message) instead of silently half-filled; a repeated request cleans up and starts over — the cooldown steps aside for that repair case. Points from a different schema state are rejected with a reason, never imported silently.

Update hint

The admin area shows the running version at the bottom left — and a hint as soon as the package server repo.techeve.de carries a newer version of the running edition (package dnseditor or dnseditor-ee in the stable suite). The check reads the APT index that is public anyway — a plain anonymous GET, at most every six hours; no data about the installation leaves it. If the server is unreachable — say, in air-gapped networks — the hint simply stays away.

DNSE_UPDATE_FEED overrides the index URL (for a different suite, say); off disables the check entirely.