Lessons

Each of these cost at least an hour, and each is written down in the estate's own documentation as a trap with its date. They are the part of a homelab worth reading, because none of them is about this estate: they are about the difference between a system reporting success and a system working.

The common thread is in the first group. A check that looks identical whether or not anything happened is not a check — and the second and third groups are the two ways that bites: a failure that reports success, and an instrument that is itself the fault.

Checks that were not checks

Each of these passed. Each proved a different claim from the one it was believed to prove.

A saved credential is not a working credential

A provider's "verify my token" endpoint returned an error for a token that worked perfectly, and an entire afternoon of diagnosis was built on that false signal. Test a credential by calling an operation it needs to perform. The verify endpoint tests whether the token can read itself, which is a different question.

A rotation is proven by the old credential failing

A leaked password was rotated, every path was checked, and six of six checks passed, because the vault was read before the new value had been saved into it, so the old value came back and the "rotation" re-applied the exposed one. Every check tested whether the store and the consumer agreed, and they agreed perfectly when nothing had changed. Now a rotation is proven by the superseded value being refused, never by the new one being accepted.

Confirming a control exists is not confirming it does anything

A router offered a custom DNS setting, saved it, displayed it back, and was believed for a day. Then a probe suggested it never forwarded anywhere. Then the probe itself turned out to be the problem: the resolver's rate limiter was dropping the probe's burst of queries with no answer and no log line, and the router fell back to its secondary and succeeded. The same figure was corrected three times in one evening. When a number keeps needing correction, suspect the instrument before the arithmetic.

A hook attached to a container is not a hook that runs for it

Every container carried the same start-up hook, and the hook's own list of which containers it applied to had not been extended for the two newest. One ran for a day without the protection everyone believed it had; the other was caught on its first. Attaching is not enabling; when a script gates on a list, adding a container means editing the list.

Additive configuration lies when read back

A package-update policy appended to the distribution's default instead of replacing it, so the whole archive quietly became eligible for automatic upgrades while every setting read back exactly as written. Only the tool's own debug output showed what it would do. Verify what a tool will do by asking it to name a package, not by reading its configuration.

Failures that reported success

A component broke and nothing reported it, because the report was downstream of the break.

A validator piped into anything hides its failure

A configuration change was validated with its output piped into a filter that kept only the last line, the pipe returned the filter's success, and the service was restarted on a broken file. Three minutes of outage on a change meant to reduce exposure. Gate a restart on the validator's own exit status, and never on a pipe.

Hooks and schedulers do not inherit the shell

A backup hook ran with no home directory and a path missing the usual tools, so the off-site upload failed on its first scheduled run while the local backup reported success. Running the hook by hand worked, which is exactly what hid it. Test scheduled jobs with an empty environment, not from a terminal.

Firewall rules can lose a race at boot and fail silently

At boot, two services installed their own firewall chains at the same moment, one of them exited with an error the network layer logged as a warning, and the boot reported success with three rules missing. Found by an actual reboot, not by reading configuration. Every rule now waits for the lock, and the count of rules is verified rather than the exit code.

A per-command lock does not make a sequence atomic

A script that deletes rules and re-adds them ran from six places at once during a boot, once for the bridge and once for each container, and the runs interleaved, leaving denies ahead of allows on a bridge that is first-match-wins and stateless. The alerting relay served errors: the alerting system was down. The fix was a lock around the whole rebuild, and the check became counting the rules in order.

Line endings, again

An archive exported from a subtree of a repository never sees the attributes file at the root, so the platform default silently converted every page to Windows line endings while the stored versions were clean. The only reason it was noticed was a deploy check that compares the hash of what the server serves with the hash of the stored file, not the working copy.

When the instrument is the fault

The measurement was wrong, or measured a system that had changed underneath it. Suspect the instrument before the arithmetic.

A gauge reading zero is not a path being absent

A subscriber count read zero on a relay that was delivering fine, because the phone's connection is suspended by battery saving and replayed on reconnect. The zero was read as "the client never connects here" and a working path was removed on the strength of it. Before concluding a path is dead, send traffic through it.

Turning a cache off can change the architecture

Switching off the router's DNS cache did not merely clear a cache: it changed which address every device on the house is handed, and removed the router from the DNS path entirely. Twenty-nine probes were fired at the old path before anyone asked what the system now did. After changing a setting, re-establish what the system does before measuring whether the old path still behaves.

The fault that was not the router

Weeks of intermittent packet loss looked like a router problem until the loss was measured at every hop from inside the house outward: zero at the router, heavy at the provider's first hop, and the cable modem's own event log full of upstream errors. The technician confirmed it. The lesson is not about cable modems: measure at each boundary before blaming the nearest box.

The choices these lessons shaped are in Decisions.