Your monitoring says backups are healthy. It is telling the truth. It is also answering a much smaller question than the one you think you asked.
Three separate claims hide inside “the backups are fine”, and they get progressively harder to prove:
- A job ran and exited zero.
- The archive it produced is internally consistent.
- The application can be brought back from it.
Almost every backup dashboard reports the first. Some tooling can be persuaded to check the second. Practically nothing checks the third, and the third is the only one that describes what happens after the disk dies.
The gap is not a failure of diligence. It persists because the defaults in the tools involved are quietly designed around the first claim, and because each layer’s success message sounds like it covers the layer above it.
The first claim: a job ended
Exit code zero means the backup process completed without an error it considered fatal. That is all it means.
It does not mean the data was in a usable state when it was captured. This is the part operators reliably underestimate, because it is invisible in every log: a backup of a running application can copy a database and a directory tree at two different instants and archive a combination that never existed.
Gitea’s documentation is unusually direct about this. “To ensure the consistency of the Gitea instance, it must be shutdown during backup.” Not should — must. Its own dump tool is documented as racy against a live instance, because the database can move while the repositories are being read. (Gitea backup and restore)
Almost nobody stops the service. The nightly job runs, exits zero, the dashboard goes green, and the archive contains a database row pointing at a repository state the filesystem copy does not have. Every part of that outcome looks like success.
The second claim: the archive is consistent
This is a genuinely stronger check, and it is where most careful operators stop — reasonably, because it feels like verification.
Then you read what the verification covers. From restic’s own documentation: “By default, the check command does not verify that the actual pack files on disk in the repository are unmodified, because doing so requires reading a copy of every pack file in the repository.”
The default validates the repository’s structural integrity — snapshots, trees, and the metadata tying them together. It does not read your data. --read-data does, and it is expensive enough that people run it rarely or never, which is a defensible engineering trade-off and an indefensible thing to forget. (Checking integrity and consistency)
So restic check passing means the repository is structurally sound. restic check --read-data passing means the bytes are the bytes you stored.
Neither means the bytes are useful.
That distinction is the entire argument. A structurally perfect archive of an inconsistent snapshot is a structurally perfect archive of garbage.
The third claim: the application comes back
Here the defaults stop being merely uninformative and start actively hiding failure.
pg_restore is the clearest case. Its documented behaviour on hitting an SQL error is: “The default is to continue and to display a count of errors at the end of the restoration.” It keeps going. It tells you at the end, in a summary line, after several thousand lines of output have scrolled past.
A restore that reports 31 errors and exits is a restore most people will call successful, because the prompt came back and the database is there. Some tables are missing constraints. A foreign key did not apply. The application starts, and the damage surfaces weeks later as a data integrity bug nobody connects to a restore.
The fix is one flag. --exit-on-error stops at the first failure. --single-transaction is better still: it wraps the whole restore so that “either all the commands complete successfully, or no changes are applied”, and it implies --exit-on-error. All-or-nothing is what you want from a restore, and it is not the default. (pg_restore)
Above the database, the failures are less about data and more about identity.
Compose assigns the project name from the directory containing the Compose file and uses it to isolate environments from each other. That is a sensible design and a nasty restore trap: recover into a directory with a different name and you have a different project, scoped to different named volumes. The stack starts perfectly. It starts perfectly empty.
And Gitea, once more, on what survives a move: “Repository Git Hooks should be regenerated if installation method is changed (eg. binary -> Docker), or if Gitea is installed to a different directory than the previous installation.”
Read that carefully. Hooks are per-repository files containing paths. Restore to a new location and they point at the old one. The web interface works. Authentication works. Clones work. Pushes fail — and pushes are the last thing anyone tests, because the first five things all worked.
Why the confusion is stable
Every layer here behaves correctly by its own standards. Restic is a backup program and its check verifies the backup repository. PostgreSQL’s permissive default exists because a restore that halts on the first error is genuinely worse in some workflows. Compose isolates projects because isolating projects is useful. Gitea’s hooks contain paths because hooks contain paths.
None of these is a bug. The failure is at the seams, and nothing owns the seams.
There is also an incentive problem worth naming. Backup success is easy to instrument and easy to display. Recovery success requires spare hardware, an afternoon, and a willingness to discover something unpleasant. One of these produces a green square on a dashboard this week. The other produces a document saying your recovery time is four hours and your runbook is wrong.
Only one of them gets built.
What to require instead
Replace “are the backups working” with three questions that have different answers.
When was the last time the archive’s contents were read, not just its structure? For restic that is the date of the last --read-data. If the answer is never, you are trusting metadata about data nobody has touched since writing it.
What does the restore do when it encounters an error? For PostgreSQL, if the answer is not --exit-on-error or --single-transaction, your restores are permitted to half-succeed.
When did an application last come back from the archive on a machine that had nothing else on it? Not restored over the original. Not restored beside a stale volume. A clean host, with a write operation as the final check.
The third question is the only one that tests the seams, and the only one that produces a recovery time.
The check worth adding this week
If you take one thing from this, make it the write check.
Almost every partial-restore failure described above passes a read test. The database is there, the pages render, the files are present. A write operation exercises ownership, hooks, configuration, secrets, and constraints at once, and it is the cheapest check with the best coverage.
Push a commit. Upload a file. Create a record.
If that works on a machine that was empty an hour ago, you have earned the green square.
Provenance: this article makes no measurement claims. Every behaviour described is quoted from current vendor documentation — restic, PostgreSQL, Docker, and Gitea — verified against those sources at the time of writing. No backup or restore was executed for this piece.