Umbryx — Multi-node HA deployment¶
Two (or more) hosts running the same logical Umbryx dataset, with
automatic failover of both Postgres and Redis, behind a single floating
VIP. This is a genuinely different deployment mode from the regular
single-node docker-compose.yml — read docker-compose.ha.yml's header
comment first.
Built and load-bearing-tested live on a real 2-node deployment (2026-09-07/08): killed the primary Postgres node under real app traffic — zero hung requests, ~41s to full automatic recovery, all data and encrypted fields (cred secrets) intact and readable from either node afterward. Same for Redis: killed the master, Sentinel promoted the replica and both app nodes picked it up within ~9s.
Re-verified 2026-09-08 with GET /api/system/ha (System → HA Cluster in
the UI) watching live: docker kill on the primary's Postgres container,
~50s to a writable new primary (monitor marks unhealthy at +8s,
demote/promote sequence starts at +20s, demote_timeout expires and the
standby reaches wait_primary — already writable, VIP already pointed
at it — at +50s), app back to 200s throughout via the VIP the moment the
new primary was writable. Confirmed at the SQL level too:
pg_is_in_recovery() false and inet_server_addr() the surviving node,
queried through the VIP.
Known limitation: 2 nodes can detect a problem but not always act on it alone¶
Both the pg_auto_failover monitor and Redis Sentinel need a node they can still talk to, to decide anything. With exactly 2 nodes:
- The monitor (pg_auto_failover) lives on one node only (node A here). If node A's whole host dies (not just its Postgres), the survivor has no monitor to report to — no automatic promotion, however healthy it is. Verified failover in this doc kills a container, not the host, which is the scenario that actually works automatically.
- Sentinel needs a majority of known sentinels to actually execute a failover, not just detect a problem. With 2 sentinels total that majority is 2 — if the sentinel co-located with the dead master goes down with it, the lone survivor can't reach quorum alone.
Both are structural, not implementation bugs — a genuine fix needs a 3rd
node (even a small one) purely for the monitor / a 3rd sentinel. Adding
node C later is additive: give it its own .env.ha (no monitor
profile — that only ever runs once), point its pgaf-node and sentinel
at the existing monitor/master IPs, and it just joins.
Prerequisites (every HA host)¶
sudo sysctl -w net.ipv4.ip_nonlocal_bind=1 # HAProxy/keepalived bind the
# VIP before VRRP assigns it
Pick a VIP address on the same subnet that is not already assigned to
any host (ping -c1 <candidate> should time out). Pick free ports for
PG_VIP_PORT/APP_VIP_PORT/PG_MONITOR_PORT/PG_NODE_PORT — ss -tlnp
on each host first; on a shared machine, 5432/80/6379 etc. are
often already taken by unrelated services, which is exactly why every
port in docker-compose.ha.yml is a variable, not a default that
happens to work.
Bootstrap order (first time only — NOT idempotent)¶
pg_auto_failover node registration is a one-shot handshake with the monitor. Do this in order, on one node at a time.
1. Node A — monitor, then this node's own Postgres:
cp .env.ha.node-a.example .env.ha
# edit OWN_IP/PEER_IP/VIP/ports for your environment
docker compose -f docker-compose.ha.yml --env-file .env.ha up -d pgaf-monitor
# wait ~10s, then:
docker compose -f docker-compose.ha.yml --env-file .env.ha up -d pgaf-node
docker exec -u postgres umbryx_pgaf_monitor \
pg_autoctl show state --pgdata /var/lib/postgresql/16/main
single (it's the only node so far, running as
primary).
1b. Node A — create the read-only monitoring role on the MONITOR
(optional — only needed for GET /api/system/ha / System → HA Cluster in
the UI; the monitor is a separate small Postgres instance from the app's
own database, so this is a second, distinct role creation from step 2):
docker exec -u postgres umbryx_pgaf_monitor psql -p 5433 -U postgres -d pg_auto_failover -c \
"CREATE ROLE umbryx_monitor WITH LOGIN PASSWORD 'change_me_ha_monitor_password';
GRANT CONNECT ON DATABASE pg_auto_failover TO umbryx_monitor;
GRANT USAGE ON SCHEMA pgautofailover TO umbryx_monitor;
GRANT SELECT ON pgautofailover.node TO umbryx_monitor;"
docker exec -u postgres umbryx_pgaf_monitor sh -c \
"echo 'host pg_auto_failover umbryx_monitor 172.16.0.0/12 scram-sha-256' >> /var/lib/postgresql/16/main/pg_hba.conf && pg_ctl reload -D /var/lib/postgresql/16/main"
172.16.0.0/12, not just <YOUR_SUBNET>/24 — the app makes this
connection directly from its own container (bridge network), not through
HAProxy (host network) like the actual DB traffic does, and self-connects
from a bridge-networked container back to the same host hit a Docker NAT
hairpin quirk in testing: the server sees the container's bridge-internal
source IP (e.g. 172.20.0.7), not the host's real LAN address — and that
internal subnet differs per host/compose project (observed 172.20.0.0/16
on one node, 172.18.0.0/16 on the other). 172.16.0.0/12 covers
Docker's whole default bridge allocation range plus the real LAN subnet
in one rule; only a role that already has the password can do anything
with it, and this role can only SELECT one table.
Then in every app node's .env:
HA_MONITOR_HOST=<node-a-ip>
HA_MONITOR_PORT=5433
HA_MONITOR_USER=umbryx_monitor
HA_MONITOR_PASSWORD=change_me_ha_monitor_password
SELECT one table — it has no path to anything
resembling --auth trust's blast radius from finding 1 above.
2. Add the app's DB role/database on node A (pg_auto_failover provisions Postgres itself but doesn't know about the app's own role — do this once, against whichever node is primary):
docker exec -u postgres umbryx_pgaf_node psql -p 5434 -U postgres -c \
"CREATE ROLE umbryx WITH LOGIN PASSWORD 'umbryx_secret' CREATEDB;"
docker exec -u postgres umbryx_pgaf_node psql -p 5434 -U postgres -c \
"CREATE DATABASE umbryx OWNER umbryx ENCODING 'UTF8' LC_COLLATE 'C.utf8' LC_CTYPE 'C.utf8' TEMPLATE template0;"
C.utf8 because the pg_auto_failover image is a minimal Ubuntu base —
locale -a inside the container to confirm what's available if this
differs.) If migrating an existing single-node deployment's data, pg_dump
it from the old db container and restore it here instead of creating an
empty database.
3. Node A — allow the app role to connect from the LAN (pg_auto_failover manages HBA rules for its own replication/monitor users automatically; the app's own role needs its own rule, once per node as it joins):
docker exec -u postgres umbryx_pgaf_node sh -c \
"echo 'host umbryx umbryx <YOUR_SUBNET>/24 scram-sha-256' >> /var/lib/postgresql/16/main/pg_hba.conf && pg_ctl reload -D /var/lib/postgresql/16/main"
4. Node A — HAProxy + keepalived:
ip addr show <VRRP_INTERFACE> should show the VIP on node A (it's the
only node up, so it wins by default).
5. Node A — Redis + Sentinel (this node seeds as primary — its
.env.ha has REDIS_REPLICAOF empty):
6. Node B — join everything:
cp .env.ha.node-b.example .env.ha
# edit for your environment — MONITOR_IP/REDIS_SEED_MASTER_IP point at node A
docker compose -f docker-compose.ha.yml --env-file .env.ha up -d pgaf-node
init → wait_standby → catchingup → secondary on its own (pg_basebackup
runs automatically):
docker exec -u postgres umbryx_pgaf_monitor \
pg_autoctl show state --pgdata /var/lib/postgresql/16/main
7. Verify the VIP is reachable and routes to the real primary, from either host:
psql -h <VIP> -p <PG_VIP_PORT> -U umbryx -d umbryx -c "SELECT pg_is_in_recovery();"
# should print 'f' (not in recovery = this is the writable primary)
Point the app at the cluster¶
In each node's app .env (the regular docker-compose.yml, not
.env.ha):
DB_HOST=<VIP>
DB_PORT=<PG_VIP_PORT>
# require, not the default `prefer`: this connection now crosses the LAN
# to a separate host instead of staying container-to-container on one —
# `prefer` silently falls back to plaintext if anything strips the TLS
# offer in transit, `require` refuses to. Confirmed working against the
# `host ... scram-sha-256` HBA rule from step 3 (Postgres's `host` HBA
# entries match both SSL and non-SSL connection attempts; the client
# picking `require` is what forces this one onto SSL).
DB_SSLMODE=require
REDIS_SENTINEL_HOSTS=<node-a-ip>:26379,<node-b-ip>:26379
REDIS_SENTINEL_MASTER=umbryx-master
REDIS_SENTINEL_PASSWORD=
ENCRYPTION_KEY and JWT_SECRET must be byte-for-byte identical on
every app node. They're normally generated fresh per install — if
you're converting an existing single-node deployment, copy the first
node's values to every other node's .env rather than regenerating.
Skipping this doesn't fail loudly: everything appears to work until
someone tries to read a credential secret, a confidential note, or
sensitive loot written by the "other" node, which then throws instead of
decrypting.
Then on every app node:
Restricting direct node access — VIP-only for the team¶
By default, each node's own nginx is still directly reachable on its own
IP (docker-compose.yml's ports: "${PORT}:80"), same as a single-node
install. On an HA deployment that means someone could bypass the VIP —
and therefore bypass failover — by hitting a node's real address
straight, which defeats the point of the whole setup: the team should
only ever reach the platform through the HAProxy VIP.
This is closed entirely inside the nginx container config — no host-level
firewall (iptables) rules — so it stays as portable and easy to tear
down as the rest of this layer. nginx/access-control.conf is bind-mounted
into every nginx container and included in the server{} block; it
ships empty (a no-op — a regular single-node install is unaffected by its
mere presence). For HA, fill it in on every app node with the real IP
of every HA node plus 127.0.0.1:
# nginx/access-control.conf
allow <NODE_A_REAL_IP>; # node A's real IP
allow <NODE_B_REAL_IP>; # node B's real IP
allow 127.0.0.1; # host-local admin curl/health checks
deny all;
One allow line per node — this is still everything HAProxy itself
needs, since it proxies to nginx using each node's real IP, not the
VIP. Apply with docker compose restart nginx (no rebuild — the file is
bind-mounted). Verified live: HAProxy's cross-node /health checks and
the umbryx_app backends stay UP throughout, the VIP keeps answering
normally, and direct requests to a node's own address time out/403 for
any source not listed.
(An earlier attempt at this used host-level iptables
DOCKER-USER/conntrack rules instead — reverted in favor of this
approach specifically to keep the HA layer entirely container-scoped,
matching the rest of it. If you do need host-level filtering for some
other reason, remember DOCKER-USER return traffic needs an explicit
ESTABLISHED,RELATED accept before any source-IP allowlist rule — the
reply leg's source is the container's bridge IP, not the peer's real IP,
so a naive -s <peer> -j ACCEPT / -j DROP pair hangs every connection
on the return leg.)
Recovering a node after it's lost (not just its container stopped)¶
pg_autoctl's own pid/state file lives under /tmp/pg_autoctl/... in
the container's writable layer, not in the PGDATA volume — so a
plain docker stop / docker start of the same container is fine (the
writable layer survives), but docker kill followed by recreating the
container (docker rm + docker run, or anything that replaces rather
than restarts the container — e.g. docker compose up -d after an
image change) discards it. Confirmed live 2026-09-08: killing a node's
Postgres container and recreating it (rather than stop/start) had it
try to register as brand new, colliding with its still-live monitor
registration with duplicate key value violates unique constraint
"node_nodehost_nodeport_key" below. If instead you just restart the
same container (docker start, or a crash-restart under
--restart unless-stopped without removal) and it FATALs with
pg_autoctl is already running with pid 1, that means the pid file
did survive but is now stale (pid 1 in the new process's own
namespace looks "already running" to a naive check) — the fix there is
the volume-recreate below too, since there's no supported way to just
delete the one stale file without knowing the image's internal layout.
If a node's pg_auto_failover container is removed (not just stopped)
while the monitor still has it registered, re-creating it hits
duplicate key value violates unique constraint "node_nodehost_nodeport_key"
— the monitor doesn't know the old registration is gone. Drop it first,
from any other node, then recreate fresh (same as first-time join):
docker run --rm --network host umbryx/pgautofailover:16 drop node \
--monitor postgres://autoctl_node@<MONITOR_IP>:<PG_MONITOR_PORT>/pg_auto_failover?sslmode=require \
--hostname <DEAD_NODE_IP> --pgport <PG_NODE_PORT> --force
docker volume rm pgaf_node_data # on the node being recovered
docker volume create pgaf_node_data
docker compose -f docker-compose.ha.yml --env-file .env.ha up -d pgaf-node
# then repeat the HBA rule step (step 3 above) — it's a fresh volume
If a Sentinel container is removed while its sentinel_data volume
still has a stale PID file from an unclean shutdown, it FATALs on start
with "pg_autoctl is already running with pid 1"-style errors (or the
Redis equivalent). docker volume rm sentinel_data && docker volume
create sentinel_data before recreating the container — Sentinel rejoins
and re-learns cluster state from its peer within seconds either way.
Operational notes¶
demote_timeout(pg_auto_failover default ~30s, hence the observed ~41s end-to-end recovery) is tunable — see pg_auto_failover's own docs if a shorter window matters more than avoiding a premature promotion on a flaky network.- The app's Postgres connections need TCP keepalives to recover cleanly
from a mid-failover connection that never gets an RST — already the
default in
backend/app/database.py; don't remove it if hand-rolling a different HA layer that also uses a TCP-passthrough LB.