Why you do not want every alert: alarm or summary

The first instinct after a bad incident is always the same: turn on every alert. Six weeks later you are reading none of them, and that is not a discipline problem.

The value of an alarm is not in ringing, it is in being read. An alarm that rings every day teaches you to ignore it, and on the way out it takes the real one with it.

Three questions before adding an alarm

If this rings at three in the morning, would I get up? If the answer is no, this is not an alarm, it is a note. Notes belong somewhere else.

Can I do anything about it? A warning you cannot act on is not information, it is only anxiety. "Disk temperature is 41 degrees" is not an alarm.

How often will it ring when everything is fine? If the answer is more than rarely, the problem is not the alarm but the threshold. On a disk that sits at 82 percent all the time, an 80 percent threshold does not produce an alarm, it produces a habit.

The one distinction that fixes most setups

An alarm and a summary are two different things. An alarm is rare and it interrupts you: you have to put down what you are doing and look. A summary is regular and read calmly, next to your coffee.

Most of what people turn into alarms actually belongs in a summary. "There are this many updates", "that machine ran hot yesterday", "that backup has not been taken for three days": none of these get you out of bed, yet all of them are worth knowing.

The moment you make that distinction you win twice: the alarm list gets short and is taken seriously again, and the summary actually gets read because there is something in it.

The threshold is not a technical detail, it is the design itself

Instead of picking a threshold because it sounds reasonable, measure what normal looks like on your own machine. On a server whose disk always sits at 78 percent, 80 is wrong and 90 is right. On a machine whose load is always high, there should be no load alarm at all.

Setting a threshold once and moving on is not enough either. The machine changes, the work on it grows. Asking once a year "which warning actually taught me something" corrects the thresholds by itself.

The hardest part: not knowing is not the same as being fine

If a check goes silent when it cannot read its data, you read that silence as health. In reality nothing is known.

This is the most insidious failure in monitoring, because the screen is green. The right behaviour is not to say "nothing wrong" but to say "I could not read this". A watchman who tells you what he could not see is better than one who says nothing.

What Atlas does

Atlas Watch is not an alert stream, it is a daily summary. It puts the state that accumulated during the day in one place, in a form you can read calmly.

There are twelve checks in it: server reboot, root disk usage, memory, load, uptime, pending updates, security updates, updates that require a reboot, a major kernel jump, stopped critical machines, processor and memory per machine, and the age of the backup.

Every check can be turned off individually, each one can be given its own severity, and for most of them the threshold is yours. The thresholds are protected by a lower and upper bound, so a meaningless value is not accepted: backup age between one and sixty days, disk usage between fifty and ninety nine percent.

One of them deserves a separate mention, because it answers the hardest point above: when the backup information cannot be read, Watch does not go quiet, it says that it could not read it. Users read silence as health; the whole trust anchor of a watchman sits in that behaviour.