Testing a restore: a green job is not proof

The backup jobs run every night and they are all green. That proves the file was written. It does not prove the file can be read, it does not prove the machine inside boots, and it certainly does not prove the service on that machine runs. The gap between those shows up on the day you actually need it.

Three levels of testing

First: can it be read. A check that the backup is intact. It is cheap, it can be automated, and it at least catches silent corruption. On its own it still does not say "the machine boots".

Second: does it boot. Restore the backup under a new identity onto a test target, start it with the network disconnected, and look at the login screen. Most people never reach this level, although it costs about a coffee break and what it teaches is large.

Third: does it do its job. Does the application come up, is the data there, is the last hour missing. On critical machines this is the real question, because a machine that boots but boots empty rescues nobody.

The rule for testing without breaking anything

Always restore under a new identity, never over the running machine. And run the test with the network disconnected: two machines carrying the same identity and the same address will fight each other on the same network, and the price is paid not by your test copy but by the running original.

What people find when they finally test

What first-time testers find is strikingly similar from one setup to the next. The backup is three weeks old because the job stopped quietly and nobody looked. The machine boots but the application asks for a key that lived somewhere else. The restore needs more space than the target has. None of this is visible while the backup is being taken; all of it is visible while it is being restored.

Surprises at restore time

Identity conflict: that number is already in use. Storage name: the backup refers to a storage name that no longer exists on this machine. Bridge name: the network definition in the backup points to a bridge this server does not have, so the machine boots without a network.

None of these are faults, they are effects of moving. But met for the first time on the day of a real disaster, they are indistinguishable from faults.

Frequency and the calendar

A restore test done once a quarter is worth more than a flawless backup policy that was never exercised. Tie the test to a date; tests left for "when there is time" do not happen. Write down what you found, because if the next test finds the same thing, the problem is not the backup, it is the process.

The point is the rehearsal

On the day of a real loss you will not have time to learn any of this. The only thing to do that day is repeat work you have already done. The test is that day's rehearsal, and that is exactly what a rehearsal is worth.

What Atlas does

Atlas shows which backups exist, where they sit and when they were taken, so the question "do I have a backup and how old is it" is answered by looking rather than guessing. That is the first step of the testing above: making sure there is something to test. The restore itself is done from the Proxmox side screens; Atlas does not stand in the way, it makes visible what you have in hand.