Cluster operation

The Cloud variant runs as multiple identical nodes behind a load balancer. Three things must be shared for that.

1. Database

All nodes talk to the same database. For distributed operation that is YugabyteDB (PostgreSQL protocol, distributed storage):

DNSE_DB_DSN=postgres://yugabyte@yb:5433/yugabyte
DNSE_DB_YB=on

Without DNSE_DB_YB=on the connection is treated as ordinary PostgreSQL — the YugabyteDB-specific driver would go unused.

Schema migration runs at startup. It is meant to run once: when bringing up a fresh cluster, one node starts first and the others follow once it is healthy.

2. Encryption key

All nodes need the same DNSE_ENCRYPTION_KEY. Otherwise node 2 cannot read what node 1 wrote encrypted — provider credentials would be broken depending on which node handles the request.

3. Sessions

DNSE_VALKEY_ADDR=valkey:6379
DNSE_VALKEY_PASSWORD=…          # optional

Without Valkey, sessions would live in the memory of each process: a sign-in on node 1 would be unknown to node 2, and the user would be thrown out on every load-balancer switch. With Valkey a session is valid everywhere — for tenant and operator sign-ins — and a logout takes effect immediately on all nodes.

Valkey is deliberately not backed up: at worst, losing sessions means signing in again; the durable state lives in the database.

How long a sign-in stays valid and how often its validity is extended comes from the region configuration (session.ttl_hours, session.touch_interval_minutes) — see Region catalogue.

3.1 Across several regions: one Valkey cluster

With several regions the single instance becomes a cluster, and DNSE_VALKEY_ADDR names several addresses:

DNSE_VALKEY_ADDR=vk-eu:6379,vk-na:6379,vk-apac:6379
DNSE_REGION=eu-central          # decides where this node's sign-ins live

Every session belongs to the region it was created in. The token carries that visibly (eu-central~…), and the storage key gets the region as a hash tag:

dnse:session:{eu-central}:eu-central~9f3a…

Everything in curly braces determines the slot. All sessions of a region therefore sit together — and writes land at the owner by themselves, because the client knows the slot map. A logout in a foreign region thus takes effect immediately and everywhere; reads still stay local, because the client picks the nearest among the nodes serving a slot.

The hash tag binds a region to one slot — but which node holds that slot is decided by how the cluster was built, not by the tag.

A plain valkey-cli --cluster create splits slots evenly and knows nothing about regions. In a three-node setup the result was: eu-central on node 3, na and apac together on node 1, node 2 empty. That would be locality without location — everything together, just in the wrong place.

So after building, check which node holds a region’s slot:

valkey-cli -p 6379 cluster keyslot 'dnse:session:{eu-central}:x'

and move the slot deliberately onto the node in that region (cluster setslot … importing/migrating, migrate, then cluster setslot … node <target> on every node). Live sign-ins survive the move — they travel with it.

Two operational settings belong with this, because the cluster bus now runs across the WAN: tls-cluster yes (or a tunnel) — otherwise sessions travel between regions in the clear — and a cluster-node-timeout measured against the actual round-trip between regions and set generously. Otherwise every hiccup on the line triggers a failover.

Existing sign-ins without a region mark keep working unchanged; they keep their old key and expire on their own. No migration is needed.

Background jobs in the cluster

Drift reconciliation, certificate issuance, expiry reminders, backup exports and the migration check run as scheduled jobs. Any node can execute them; a lease ensures that only one works on a given job at a time.

Test cluster

For local experiments there is a ready-made four-node setup with YugabyteDB and Valkey: Test cluster.