I built a software RAID, rebooted, and the storage is gone: the array is not assembled at boot

You combined the disks, the array came up, the storage appeared, you wrote data to it. Everything worked.

Then you rebooted the machine and the storage is gone.

The disks are fine. The data is still there. What is missing is something else.

An array is not on the disks, it is in the assembly instruction

Presenting several disks as a single storage is not a property that sits on the disks. At every boot the system has to find those disks and put them back together.

There is a record that tells it how. Without that record the array may not be assembled. Even if it is, it may come up under a different name, which amounts to the same thing: your storage points at the old name and there is nothing there.

So the problem is not "the data is lost", it is "the route to the data was not built at boot". That sounds less frightening, but in the moment of panic the two feel identical.

Proxmox's own position

This deserves to be said honestly: the built-in and supported path on Proxmox is ZFS. Ask for disk mirroring during installation and that is what you get.

Classic software RAID works, but it is not the path the installer sets up for you. Which means that when you choose it, making sure the boot-time assembly steps were done correctly is on you.

There are still cases where it makes sense: you already have an array and you are migrating it, your controller layout is not what ZFS expects, or the machine's memory is tight for ZFS. Those are real reasons. Choosing it without one is making work for your future self.

The rule: an array that has not seen a reboot does not count as an array

This is the most practical sentence in this article.

After building the array, reboot once on purpose, while nothing depends on it. Does the storage come back, does it come back under the same name, are the contents visible.

We wrote the same thing in the alerting article in this wiki: an untested alarm is not a mechanism, it is a hope. The same holds for an array. An array that has not survived a reboot is an array you believe works.

The name-change trap

The second common trap: the array is assembled, but under a different name than last time.

Your storage definition points at the old name, so the storage is invisible again. This time the array is up, but nobody is looking at it.

The fix is to attach the storage by identity rather than by name. Names can change, identities do not.

What Atlas does

When Atlas builds an array it also writes the boot record, and it shows this warning on screen: if this step fails, the array may not be assembled at boot and the storage becomes invisible.

The warning was true. But for a while it was undermining itself.

The system's own tool was being called to produce the record. It was measured, and this is what came out: when there is no array at all, that tool exits with code zero and prints nothing. In other words it says "success" and hands you nothing.

The result: nothing was written, but the step was recorded as successful. The very thing the warning had just described was happening to the user, while the screen showed everything as fine.

The fix had two parts. The expected format was determined by measuring rather than by guessing, and if the output does not match that format the step counts as failed. Empty output is a failure too. And to avoid writing the same array twice, both the device path and the identity are checked.

The general lesson

The sentence that matters here is this: "the command succeeded" and "the work got done" are not the same thing.

An exit code tells you whether the tool ran. It does not tell you whether the outcome came about. If a step's entire purpose is to produce an effect, the thing to check is not the state of the tool but the effect itself.

There are two more articles from the same family in this wiki. In the orphaned disks article we separated an empty answer from a failure to get an answer. In the server name article there was the danger of putting a plausible-looking default in the place of the unknown. This is the third: putting an exit code in the place of an outcome that was never established.

All three are faces of the same fault: the program is stating something it did not verify.