When the watchman dies: why silence is not good news

You set up the alerts. Weeks have passed and not a single email has arrived.

There are two explanations. Either nothing really happened, or the alerting stopped working and you do not know it. From the outside the two look identical.

And the human mind tends to read silence as good news. That is exactly why this class of fault lives for months without being noticed.

This is a different problem from a check going quiet

Elsewhere in this wiki we wrote: when a check cannot read its data it must not go quiet, it must say that it could not read.

What we are describing here is one level above that. There, a check inside a working watchman went quiet. Here the watchman itself is gone. Nobody is left to speak, so nobody is left to say "I could not read it" either.

How a watchman dies quietly

The service stops. An update, a crash, running out of memory. The thing that would tell you is the thing that stopped.

The mail path breaks. A password changes, a provider blocks it, the address bounces. The watchman keeps working, writes its messages, and none of them reach you. This is the most deceptive one, because on the server side everything looks healthy.

Somebody turns it off and forgets. Notifications get switched off temporarily while investigating something. The problem gets solved. The notifications stay off.

The machine shuts down. The extreme case: nothing is broken, everything is simply silent.

What all four share is this: none of them produces a message. And if your way of noticing a fault is receiving a message, then faults that produce no message are invisible to you.

The fix has a different shape: not an alarm, a heartbeat

The direction has to be reversed.

An alarm says: speak when something is wrong. A heartbeat says: say "I am alive" at regular intervals even when nothing is wrong. Then have someone notice when the saying stops.

The difference matters. An alarm is a message about a problem; a heartbeat is a message about the messenger. They answer different questions and neither substitutes for the other.

Once there is a heartbeat, the meaning of silence changes. Silence now carries information: something that ought to be speaking is not.

Whoever notices must be outside the machine being watched

This is the part worth dwelling on. A machine cannot report its own death.

The place that judges the heartbeat has to be outside the server being watched: another machine, a phone, an outside point. A check running inside the server shuts down with the server and tells nobody anything.

Even the simplest homemade version is enough: let the server leave a mark somewhere every hour, and look at the age of that mark. No elaborate setup is needed; the only thing that is needed is that the judging happens somewhere else.

The heartbeat has a trap of its own

There is a subtle design mistake here, and most setups fall into it: attaching the heartbeat to the schedule of something the user can turn off.

Say it shares timing with the daily summary email. The user turns the summary off, because it felt like too much. The heartbeat stops with it. The outside party sees the signal stop and raises a "no news from your server" alarm while nothing at all is wrong.

You can guess the result: the user gets frightened for nothing once, and switches the alarm off the second time. A false alarm produced by your own safety mechanism destroys trust in that mechanism faster than having no mechanism at all.

The rule: the heartbeat must have its own schedule, riding on nothing that can be switched off.

The same shape, somewhere else

In the backup job article in this wiki we wrote: do not look at the job's status, look at the age of the newest backup.

The same shape turns up here: do not ask the alerting system whether it is working, look at the age of the last signal. A status is a claim, an age is a measurement.

And finally, the one thing to do on setup day: deliberately produce a fault and watch the email arrive. An untested alarm is not a mechanism, it is a hope.

What Atlas does

Atlas Watch sends a daily summary, reports critical situations immediately without waiting for the summary, and alongside those it emits an hourly "I am alive" signal.

What is worth telling is where that signal is not attached.

The signal rides neither on the daily summary's schedule nor on the alarm's. Both of those have their own settings and the user can turn either off. Had the signal depended on them, the moment the user switched off the summary the signal would have stopped too, and the far side would have said "no news from this server" while nothing was wrong. That is why the signal has its own schedule.

The second detail follows the same thinking: while the notification setting is off, the signal's schedule file is not written at all. So there is no in-between state of "enabled but not working"; either the signal exists or it does not.

The notification path itself is yours: you define your own mail server, and the mail leaves from your server. Sending a signal to an outside point is an optional extra, and the product does not depend on it to work.

The general lesson: a safety mechanism must not lean on the switchable parts of the thing it protects. If it does, it dies quietly along with it.