The backup job stopped quietly: the costliest failure is not the loud one

The costliest failure mode of a backup arrangement is not the loud one. A loud error is seen and fixed the same day. A job that stops quietly stays stopped for weeks and is discovered on exactly the day it was needed.

Ways to stop quietly

The target filled up. A network share did not mount after a restart. A storage was renamed. The job points at a machine that no longer exists. A credential expired. All of these produce an error somewhere, and none of them produce an error in front of a human.

The notification trap

The sentence "if it fails I will get an email" carries two assumptions: that mail actually leaves this machine, and that someone reads it. In most setups neither is true. On top of that, a notification that only fires on failure is indistinguishable from a notification system that never worked: both are silent.

The only way to know a notification works is to have seen it work. On the day you set it up, cause a failure deliberately and watch the mail arrive. If you did not watch it arrive, that notification does not exist.

The right thing to look at: age

Instead of the status of the last job, look at the age of the newest backup. Age answers two questions at once: did the job run, and did it produce something. A job can run and produce nothing while still looking successful; age catches that, status does not.

A one minute check, once a month

Look at four things: how old the newest backup of each machine is, whether the number of kept backups matches your retention rule, how much room is left at the target, and whether any machine has no backup at all.

That last one is the most commonly found, and the reason is nearly always the same: that machine was created after the backup job was written.

Write the rule, not the list

A job that picks the machines to back up one by one is correct on the day it is written and slightly more wrong with every new machine. Where possible, define it as "all of them, except these". Then creating a new machine puts it inside the backup scope by itself, and the cost of forgetting drops to zero.

What Atlas does

Atlas shows the backups that exist together with their dates, so the age question above is answered by looking rather than guessing: when the newest backup of each machine was taken sits on one screen.

Atlas Watch watches the server itself, pending updates, the virtual machines and the age of the backups. It looks where this page advises looking: not at the last status of the job, but at the age of the newest backup. When the newest backup of a machine is older than the chosen number of days, it says so in the daily digest; the default threshold is three days, it can be changed from the panel, and if the check is raised to critical it is reported at once without waiting for the digest.

If a backup storage itself cannot be read, Watch does not go quiet, it says that it could not read it. Treating silence as health is the very failure this page describes, so a watchdog calling something clean when it could not read it was deliberately ruled out. The monthly check above is still worth keeping: the one case the check does not cover is a brand new machine with no backup at all.