The estate
The estate is three nodes — the machines that run its services — two in a house and one rented at the edge of the internet, and a private mesh network that joins them to the desktop and the phone. Everything below describes how the pieces relate. Nothing here says where anything lives. The two nodes at home have a page each: the R930 and the G11CD.
Network · what joins them
The internet
The public network everyone is on. Two parts of the estate face it: the far end of the tunnel at Cloudflare and the edge node. Nothing in the house does.
connected to
- carries Edge node
- carries Household network · Wi-Fi mesh router
Service · what the machines offer
Public ingress · Cloudflare Tunnel
The public front door. Cloudflare terminates visitors' connections and hands them down a tunnel the house dialled out; the house has no port open to the internet.
connected to
- provided by Tunnel connector · cloudflared
- serves Public tier
System software · what a machine runs
Tunnel connector · cloudflared
The house's end of the tunnel: a small program on the primary node that keeps an outbound connection to Cloudflare open and passes public requests to the reverse proxy.
connected to
- provides Public ingress · Cloudflare Tunnel
Service · what the machines offer
Game ingress · via the edge
The only other way in: the game's port on the edge node, carried across the private mesh to the game server at home. The edge holds nothing and can be thrown away.
connected to
- provided by Edge node
- serves Game hosting
Node · a place software runs
Edge node
The third node, and the only one that faces the internet: a small rented virtual machine in a data centre, built entirely from code, joined to the private mesh and allowed to reach exactly one port at home. It holds no state and no credential, so anything suspicious is answered by destroying it and building it again.
connected to
- sits on Private mesh · Tailscale
- sits on The internet
- provides Game ingress · via the edge
Interface · a way in
Public tier
This site: the pages anyone on the internet can read, served from the house through the tunnel.
connected to
- served by Public ingress · Cloudflare Tunnel
- served by HTTPS routing
Interface · a way in
Private tier
The other half of the same application, reachable only over the mesh: content administration and the management interface.
connected to
- served by HTTPS routing
- reached through Private mesh · Tailscale
Service · what software offers
Game hosting
The Minecraft world kept for friends, hosted in the house and reached through the edge; players are invited by name rather than the world being open.
connected to
- served by Game ingress · via the edge
Network · what joins them
Household network · Wi-Fi mesh router
The ordinary home network the machines, the controller and the desktop share with the household's devices. It forwards nothing in from the internet.
connected to
- carries Primary node
- carries Auxiliary node
- carries Out-of-band controller · iDRAC
- carries The desktop
- sits on The internet
Machine · a computer
Primary node
The main machine, and where nearly everything runs: a four-socket Dell PowerEdge R930 under Proxmox, every service in its own container on a bridge the host filters. Its own page has the hardware in detail.
connected to
- sits on Household network · Wi-Fi mesh router
- sits on Private mesh · Tailscale
Machine · a computer
Auxiliary node
The second machine: an ASUS ROG G11CD, retired as a desktop and rebuilt under Proxmox. It keeps the third copy of every backup, watches the primary node and pages the phone if it goes quiet, and drives the living-room television. Its own page has the hardware in detail.
connected to
- sits on Household network · Wi-Fi mesh router
- sits on Private mesh · Tailscale
Machine · a computer
Out-of-band controller · iDRAC
A separate small computer inside the primary node that runs whether the server is on or off: power control, a remote console and the hardware inventory, reachable from the household network only.
connected to
Machine · a computer
The desktop
The operator's own computer, where deploys and credential rotations start. Not a server: nothing persistent runs on it, by rule.
connected to
- sits on Household network · Wi-Fi mesh router
- sits on Private mesh · Tailscale
Network · what joins them
Private mesh · Tailscale
A WireGuard mesh with a key per device: the two machines at home, the desktop, the phone and the edge. Every management surface is reachable across it and nowhere else.
connected to
- carries Primary node
- carries Auxiliary node
- carries The desktop
- carries The phone
- carries Edge node
- reaches Private tier
Network · what joins them
Container bridge · internal, filtered
The internal network the containers sit on. The host filters it: a container reaches the reverse proxy and the internet, and never another container.
connected to
- carries Reverse proxy container
Node · a place software runs
Reverse proxy container
The container that runs the reverse proxy: the one place every web request, public or private, passes through on its way to a service.
connected to
System software · what a machine runs
Reverse proxy · Caddy
Caddy: it terminates TLS for every name with certificates it renews itself, routes the public tier in from the tunnel and the private tier in from the mesh.
connected to
- provides HTTPS routing
Service · what the machines offer
HTTPS routing
What the proxy offers: one HTTPS front for every name, public and private, with the private names answering only to requests that arrived over the mesh.
connected to
- provided by Reverse proxy · Caddy
- serves Public tier
- serves Private tier
Machine · a computer
The phone
The pager. It subscribes to the alert relay over the mesh and receives the watchdogs' pages, including the one that says the primary node has gone quiet.
connected to
- sits on Private mesh · Tailscale
The primary node
A Dell PowerEdge R930 running Proxmox, with every service in its own unprivileged container on a bridge the host filters: a container reaches the reverse proxy and the internet, and nothing else, so a compromise of one service is a compromise of one container. The primary node has a page of its own, with its hardware and the seven services it runs.
The auxiliary node
A retired desktop rebuilt as a second Proxmox host, so that the copy of every backup that is not on the primary node, and the watch that notices when that node goes quiet, live on a machine that is not it. The auxiliary node has a page of its own, with its hardware and its services.
The edge node
A small rented machine in a data centre fronts the one public service, holding no state and no credential and permitted to reach exactly one port at home. It is rebuilt from code rather than repaired. The edge node has a page of its own.
The mesh
The two nodes, the desktop and the phone talk over a private mesh with per-device keys. Management surfaces are reachable across it, and from inside the house; never from the internet. Nothing at home is forwarded to a service; the one exception the router allows is the mesh's own encrypted endpoint, and it does not answer anyone it does not already know. Public pages, this one included, leave through an outbound-only tunnel to Cloudflare, and the tunnel is told exactly which names it may carry.
Backups
Nightly, every container is backed up three times: locally, to the auxiliary node, and encrypted to object storage off-site. A restore is rehearsed monthly by a machine, into a throwaway container, and reported. Backups that are never restored are a hope, not a plan.
Principle · a rule we hold
Three copies, one elsewhere
The rule the backups answer to: three copies of everything, on two kinds of storage, one of them off the property. A copy that exists only in this building is not a backup.
connected to
- upheld by The nightly backup
Principle · a rule we hold
Watched in both directions
Each machine watches the other and pages the phone when its neighbour goes quiet, so the one that fails is never the one expected to report it.
connected to
- upheld by The watchdog on the primary node
- upheld by The watchdog on the auxiliary node
Process · runs on a schedule
The nightly backup
Runs every night and writes all three copies: the local archives, the copies on the auxiliary node and the encrypted archives off-site. Nothing here is copied by hand.
connected to
- reads or writes Local archives · on the primary node
- reads or writes Backup copies · on the auxiliary node
- reads or writes Encrypted archives · off-site
- upholds Three copies, one elsewhere
Process · runs on a schedule
The weekly verify
Reads every stored copy once a week and checks it against its own checksums, so a copy that has quietly rotted is found before it is needed rather than during a restore.
connected to
- reads or writes Backup copies · on the auxiliary node
Process · runs on a schedule
The monthly restore drill
Once a month a copy is restored into a throwaway container, reported on and destroyed - proof the copies can be read back, not merely that they were written.
connected to
- reads or writes Backup copies · on the auxiliary node
- reads or writes Encrypted archives · off-site
Artifact · a stored copy
Local archives · on the primary node
The first copy: a full archive of every container written each night to the primary node’s own mirrored storage. The quickest to restore from, and the one that dies with the machine.
connected to
- read or written by The nightly backup
- held by Primary node
Artifact · a stored copy
Backup copies · on the auxiliary node
The second copy, on the other machine: a deduplicating backup server that keeps its own history and can restore a container by itself if the primary node is gone.
connected to
- read or written by The nightly backup
- held by Auxiliary node
- read or written by The monthly restore drill
- read or written by The weekly verify
Artifact · a stored copy
Encrypted archives · off-site
The third copy, and the only one not in the building. It is encrypted before it leaves, so the storage provider holds ciphertext and never the key.
connected to
- read or written by The nightly backup
- held by Off-site object storage
- read or written by The monthly restore drill
Machine · a computer
Primary node
The main machine, and where nearly everything runs: a four-socket Dell PowerEdge R930 under Proxmox, every service in its own container on a bridge the host filters. Its own page has the hardware in detail.
connected to
- attached to The UPS
- attached to Out-of-band controller · iDRAC
- holds Local archives · on the primary node
Machine · a computer
Auxiliary node
The second machine: an ASUS ROG G11CD, retired as a desktop and rebuilt under Proxmox. It keeps the third copy of every backup, watches the primary node and pages the phone if it goes quiet, and drives the living-room television. Its own page has the hardware in detail.
connected to
Node · a place software runs
Off-site object storage
Where the off-site copy lives: object storage rented by the month, reached over the internet by the nightly job and by nothing else on the estate.
connected to
Equipment · power or control
The UPS
Battery behind the main server. It rides out the short cuts, and on a long one gives the machine time to shut down in order rather than drop.
connected to
- attached to Primary node
Machine · a computer
Out-of-band controller · iDRAC
A separate small computer inside the primary node that runs whether the server is on or off: power control, a remote console and the hardware inventory, reachable from the household network only.
connected to
- attached to Primary node
Process · runs on a schedule
The watchdog on the primary node
A check on the primary node that reaches for the second every minute and pages after three failures in a row, so an ordinary reboot is not an alarm.
connected to
- upholds Watched in both directions
- sends to The alert relay
Process · runs on a schedule
The watchdog on the auxiliary node
The same check the other way round, and the one that matters most: it is what reports the main server being dead, from a machine that is not it.
connected to
- upholds Watched in both directions
- sends to The alert relay
Application · software that runs
The alert relay
The notification service the watchdogs publish to. It runs on both machines, so an alert still goes out when the failure is the machine that usually sends them.
connected to
- provides Alert delivery
- receives from The watchdog on the primary node
- receives from The watchdog on the auxiliary node
Service · what software offers
Alert delivery
The push to the phone. It carries the alert out of the house over the internet, so it arrives whether or not anyone is home to see a screen.
connected to
- provided by The alert relay
- serves The phone
Machine · a computer
The phone
The pager. It subscribes to the alert relay over the mesh and receives the watchdogs' pages, including the one that says the primary node has gone quiet.
connected to
- served by Alert delivery
Alerting
Each node pages the phone if the other goes quiet, and the alert path has its own fallback so "the server is down" can still arrive when the server is down. The relay is self-hosted, alerts are plain text, and test messages never go to the live topics, because a fake page destroys trust in the real ones.
Documentation
Every design starts as a written proposal with the rejected alternatives and their reasons, is approved before it is built, and is updated to as-built the same day. A commit that changes the infrastructure without touching the documentation is refused by a hook. The documentation is the source of truth; this site is a curated view of it.
Read next
Decisions is what was chosen and what lost; Lessons is what it cost to learn.