How to arrange the disks: mirroring or distributed parity

The question is not which is faster. It is which failure you want to survive, and what the machine will be forced to do after that failure.

AtlasPVE ·

This entry answers

  • how to set up raid in proxmox
  • proxmox zfs mirror or raidz
  • how many disks does proxmox need
  • is raidz1 safe
  • proxmox after disk failure

The first serious decision at install time is how the disks will be arranged. On the internet this question is usually answered with a capacity calculation: which layout gives more space.

That is the wrong question. The right one is: which failure do you want to survive, and what will the machine be forced to do after that failure?

Two basic approaches

Mirroring. Two disks hold the same data. You lose half the capacity. When one disk dies, the remaining disk is already a complete copy.

Distributed parity. Data is spread across several disks with recovery information placed alongside it. The same disks yield more space. When one disk dies, the missing pieces are recalculated from the others.

On capacity the second wins by a wide margin. That is why most advice stops there. The real difference comes afterwards.

The part nobody mentions: the moment of repair

A disk died and you fitted a new one. What happens now?

With mirroring the repair is a plain copy. Data flows from the remaining disk to the new one. The other disks are not much affected by the work.

With distributed parity the repair means reading every remaining disk from end to end. That is, you perform the heaviest job of the whole setup at the exact moment one of your disks has already died. If your disks are the same age and came from the same batch, that is precisely when a second failure is most likely.

The sentence that follows: the risk is not the first failure, it is the second failure during repair. And repair is the work that most invites a second failure.

The matter of growth

The second practical difference shows up later.

A pool built from mirrors can be grown by adding two disks at a time. You start with two disks today and add two more next year.

In distributed parity groups the width is set at creation and changing it afterwards has traditionally been hard. So you make that decision once and live with it for a long time.

The small setup rule

When the disk count is low, mirroring is usually the better trade. The reason is not capacity, it is the lightness of the repair and the flexibility to grow.

Distributed parity starts earning its keep as the disk count rises: the difference between using six of eight disks and using four of them is serious, and at that scale the repair load spreads better too.

The test is simple: take into account the answer to "how long will I be waiting anxiously after a disk dies". In small setups, keeping that time short is worth more than a few extra terabytes.

And none of this is a backup

The sentence in this article that most needs repeating: disk layout is not a backup.

A file you delete by mistake is deleted from every disk at once. A corrupted database sits corrupted on every disk. The layout protects only against one disk dying; it does nothing against human error, software faults or ransomware.

The layout buys you time, not data. What buys you data is a backup.

What Atlas does

On the pool creation screen, Atlas lists the layout options together with the minimum number of disks each one needs and writes what each provides. So while choosing, you do not have to go and look up separately "what is this for and how many disks does it want".

The default is deliberately mirroring. The reason is the trade above: at the disk counts most setups have, mirroring sits on the safer side. A default is a recommendation, not a lock; you can pick another layout, but you will have picked it knowingly.

The screen also shows different suggestions per pool type: a store attached over the network and local disks do not have the same questions, so showing the same form would be wrong.

Sources

Proxmox's own documentation. In English, and it has the final word on this subject.

Related entries

How does this look inside Atlas?

Go to the product page