Why a two node cluster is the worst choice: the majority problem

You bought the second server and built a cluster. You feel safer: there are two machines now, and if one falls the other carries on.

It does not. A two node cluster is more fragile than a single node, and the day you learn this is usually a bad day.

Why

Cluster decisions are made by majority. If two of three nodes are up there is a majority and the cluster keeps working. If one of two nodes falls, the remaining one node is not a majority, so the cluster locks itself: machines keep running, but configuration cannot be changed, no new machine can be started, and most management work stops.

This is not a flaw, it is deliberate design. The alternative is far worse: when two nodes cannot see each other, both say "I am up, the other one fell" and start the same virtual machine in two places. The same disk is written from both sides and the data is damaged beyond recovery.

So the cluster protects you by locking you out. But with two nodes, that protection kicks in on half of all failures.

The right number is three

In a three node cluster, if one falls two remain, the majority holds and life goes on. That is why the number three appears everywhere in cluster documentation.

The third machine does not have to be powerful. It only casts a vote, it does not have to run any work. A small machine, even an old computer, does the job.

If you truly do not want a third machine, a small witness node that exists only to vote can be set up instead. It is not a full node and you cannot run machines on it, but it completes the majority arithmetic to three. If you are going to stay on two nodes, at least set this up.

If you are staying on two nodes, know this

When one node is taken out for maintenance, the remaining single node also becomes unmanageable. So even a planned update leaves you locked.

It is possible to say by hand "let a single node decide too", but that reopens exactly the double start risk described above. It is done temporarily, deliberately and briefly; it is not a permanent solution.

Building a cluster is not taking backups

These are the two things most often confused. A cluster is for availability: so that work continues when a machine falls. A backup is for recovery: so that you can come back after data is corrupted or deleted by mistake.

A cluster will not bring back a virtual machine you deleted by accident. The deletion spreads to all nodes immediately, because a cluster exists to make them see the same thing. Even with a cluster, you will still need backups.

What Atlas does

Atlas treats the cluster not as a concept but as a state: how many nodes there are, how many are voting, whether the majority holds right now.

The place this helps most is updating. Before a node is updated, Atlas looks at how many of the machines on it can move to another node and suggests you an order. The suggestion stays a suggestion: the migration command does not run by itself, you press the button.

One detail matters here in particular, and it goes straight to this article's subject. The command that asks for cluster status fails in the same way in two different situations: genuinely being a standalone machine, and being a member of a cluster whose cluster layer has collapsed. If those two are not told apart, the product says "no cluster, majority fine" at the exact moment the cluster has collapsed. That is, it produces the most reassuring sentence at the most dangerous moment.

Atlas tells them apart by looking at whether the configuration file exists: that file is written together with membership and stays in place even when the cluster layer collapses. The distinction lives in a single place, because it used to be written separately in three different spots and all three had the same bug.