Persistent mounts: the single line that can stop a machine from booting

You mount a disk or a network store by hand and it works. You reboot the machine and it is gone.

The fix is obvious: add a line to the list of things mounted at boot. And that is exactly where the danger starts.

This file is not like the others

Most configuration files on the server, when broken, stop a service from working. When this file is broken, the machine may not boot.

The outcome is not "the service is down" but "the system dropped to a recovery shell". And at that point you cannot connect remotely; you need a keyboard and monitor or a physical console.

The asymmetry nobody notices

A broken network configuration is annoying: you lose remote access. But the machine still boots.

A broken mount line can stop the machine from booting at all.

And yet people are far more careful when editing the network file. The order of danger is the reverse of intuition, and it is intuition that is wrong.

Three rules

Validate the structure before writing. A line needs at least three fields: source, target and filesystem type. A line with fields missing gives trouble to every tool that reads the file.

Write atomically. A half-finished write is worse in this file than no write at all: what remains is a truncated file, and the machine tries to boot with it.

Do not let a missing disk hold the machine hostage. For network stores and removable disks, use the option that lets boot continue when the mount fails. A backup disk not being plugged in is not a reason for the server not to boot.

The verification trap: a check that crashes is not a pass

The most general lesson of this article is here, and it does not apply only to this file.

There is a standard tool for validating this file. When measured on a real machine, this came out: the tool crashes when it sees a line with fewer than three columns. That is, it blows up exactly when it sees the malformed shape it is supposed to catch.

A verification tool crashing does not mean the verification passed. The right answer is not "no problem" but "could not verify".

A process that does not make that distinction approves the most dangerous file at its weakest moment.

The second trap: not every complaint is an error

The same validation tool can say two different things, and mixing them up creates a new problem.

A format error means the file cannot be read. That is genuinely dangerous and the write should be rolled back.

A semantic complaint is different: things like "target unreachable at boot" or "unknown filesystem type". These do not corrupt the file and can be legitimate. Adding a line for a device not yet attached is a common and correct thing to do; the option that lets boot continue exists exactly for that.

A check that puts both in the same basket rejects a brand new, perfectly valid line. In other words, an over-strict gate makes the job impossible and pushes people to switch the gate off.

What Atlas does

In Atlas, every write to this file goes through a single door. Each line is structurally checked before writing, the write is atomic, and after writing the file is read back and verified; if the expected result does not appear, the previous state is restored.

Both traps above are handled deliberately. The external validation tool counts only as a secondary signal: if it crashes, the result is not "clean" but "could not verify". And format errors and semantic complaints are handled separately; only a format error causes a roll back, while semantic complaints are not swallowed, they are reported to the caller.

The need for that distinction was also found by measuring: in the first version both caused a roll back, and in that state even a brand new, entirely valid line was being rejected.

An honest note on history: the product used to write to this file from eight separate places, and none of them checked that what was written was valid; two of them overwrote the whole file non-atomically. In the same product, the network file, which cannot stop the machine from booting, was written atomically. So the more dangerous file was the less protected one. It was measured and brought behind one door.