Adding and removing a node in a cluster: the steps with no way back

Adding a machine to a cluster looks like a few clicks in the panel. In the background, though, it means a machine handing over its identity. And some of the steps have no way back.

The joining machine must be empty

Joining a cluster is not putting your name on a list. The joining machine's cluster configuration is replaced by the cluster's configuration.

The practical consequence: a server with machines on it cannot join the cluster while keeping those machines. The join expects the joining side's guest configuration to be empty.

The rule is clear: join with an empty node. If there is work on it, move that work first. The way to move it is to take a backup and restore it after joining. Do not join and hope; if the operation stops halfway you are left with neither the old state nor the new one.

Removal is one way

This is the part that needs real care. When you remove a node from a cluster, you cannot bring that node back with the same name and the same identity.

The reason: the remaining cluster keeps carrying the removed node's identity in its own state. A machine returning with the same identity creates a contradiction between what the cluster sees and what is real.

If you want to reuse the removed machine there is only one way: install it from scratch. This is not a convenience recommendation, it is the only safe path.

The machine is powered off before removal

The second rule: the node to be removed is powered off and stays off. Trying to remove a running node leads to that node coming back and disturbing the cluster.

The order is: move the work off it, power the machine off, then remove it from the cluster.

The hidden side effect of removal: the majority changes

This is the part nobody accounts for. When you remove a node, the number of votes the cluster expects has to change as well.

If you remove one node from a three node cluster, two nodes remain, and two nodes is exactly the fragile state: if one falls, the remaining single node cannot make a majority and the cluster locks itself.

So the operation "I am retiring a node" can move you, without your noticing, into the worst arrangement. When deciding to shrink, think from the start about whether you will drop to three.

Names are chosen at install time

A small but annoying detail: node names cannot easily be changed once they are in a cluster. The name you give during installation will stay with you for a long time.

Take a minute and choose a meaningful name. Three nodes all called "pve" make it hard, six months later, to know which machine you are looking at.

The order for shrinking

The order to follow when shrinking a cluster: move the work on it to other nodes, power the machine off, remove it from the cluster, verify the remaining nodes still hold a majority, and if you are going to reuse that machine, install it from scratch.

The fourth step is the one most often skipped, and when it is skipped the problem is not visible right away: everything keeps working, until a node falls.

What Atlas does

Let us state the limit plainly first: Atlas does not perform the adding or removing of nodes. That is the cluster's own job and it should stay that way.

What Atlas does is make the effect of those operations visible. When it reads cluster state it does not merely say "there is a cluster"; it shows separately how many votes were collected and how many votes are expected.

The value of that distinction shows up exactly in this article. After removing a node, you can see whether the expected vote count actually dropped. If the cluster is still expecting a node that no longer exists, the gap between votes collected and votes expected says so, and that is something to notice before another node falls.

The same information is used on the maintenance side: before a node is updated, whether the majority holds right now is calculated. Starting maintenance on a locked cluster does nothing but enlarge the problem.