Kernel updates: why the most dangerous update is the quietest one

A kernel update installs, the installation finishes without trouble, and nothing looks changed. The system is still running the old kernel, because a new kernel only takes effect on a reboot.

And that is exactly where the danger hides.

Weeks come between cause and effect

Most package failures show up while you are watching. A service falls over, it errors, you notice the same day.

The kernel is not like that. The problem appears at the next boot, and that boot may not be a planned maintenance night. The power cuts, the machine restarts itself, and it does not come back.

At that moment nobody connects it to "the update I did last month". What you have is an unexplained failure that does not look like an update problem at all.

The general principle: what is new should not be the default until it is proven

This is the portable lesson of this article, and it does not apply only to kernels.

A new kernel can be installed, but the persistent default should remain the old kernel. The new one should be tried only as a "try this once on the next boot" choice.

Set up that way, the shape of the failure changes: if the machine hangs, it returns to the old one by itself on the following boot. Because a one-shot choice is, by definition, valid once.

What you gain is this: the name of the failure becomes "nothing happened", not "the machine is gone".

The second layer: what is installed should not quietly become the default

The moment a kernel is installed, most systems say "from now on I boot with the newest one". If you do not want that, you have to pin the running kernel before updating.

Pinning does not block the new kernel, it only stops it from becoming the default by itself. When you want to try it, you choose.

The third layer: who will notice the hang

For a fallback mechanism to work, something has to say "this machine did not come up". If there is a watchdog at the hardware level, it does that and restarts the machine.

If there is not, the fallback needs a manual reset. That does not make the mechanism useless, but it has to be known: the sentence "it falls back automatically" is only half true when there is no watchdog.

The rule that holds even without a product

After a kernel update, reboot deliberately and soon. While you are watching.

A reboot you scheduled yourself is a test. A reboot forced by a power cut three weeks later is an incident. The only difference between the two is whether you were there at the time.

What Atlas does

When Atlas performs an update it pins the running kernel by default, so the newly installed kernel does not become the boot default on its own. This behaviour is not compulsory: it is offered as an option you can switch off in the interface. The product does not decide here, it makes the safe side the default.

The verified-transition mechanism is built on the principle above too: the persistent default is always the old kernel, the new one is tried only with a one-shot next boot, and if it hangs the system falls back on its own. The type of boot manager is detected at run time, because it varies from machine to machine.

And two honest limits:

This mechanism runs in observe-only mode by default. That is, it says what it would do but does not touch the boot configuration. Actually intervening is a separate choice.

If there is no hardware watchdog, the mechanism does not quietly ignore it: it disables itself gracefully and says that a manual reset may be needed. Presenting a guarantee that does not exist is worse than giving no guarantee at all.