I built a cluster but machines will not move: what is missing is storage

You built the cluster, the nodes see each other, they all sit side by side in the interface. Then you try to move a machine to another node and it does not work.

What is missing is not the cluster, it is storage.

What moves is not the machine, it is the memory

Moving a running virtual machine means copying the state in its memory to the other node. That happens over the network and it is fast.

But the machine's disk is needed too, and the disk is many times larger than the memory. If the disk sits only on the first node's own drive, the second node cannot reach it. The migration either never starts, or it has to copy the disk as well, which no longer takes seconds but minutes or even hours.

So the answer fits in one sentence: the disk has to be visible from both nodes.

Two ways to arrange that

Shared storage. The disk sits in a common place both nodes connect to: a network store, a storage server, or a distributed storage layer. At migration time the disk is not copied at all, because it is already there. This is the fastest and cleanest way.

Replication. The disk stays local but is copied to the other node at regular intervals. At migration time only what changed since the last copy is sent, which is why it is fast. The price is this: because the copy is taken on a schedule, you may lose the last few minutes. In a planned migration that is not a problem; in a sudden failure it can be.

The question that decides which suits you is this: if a node dies suddenly, how many minutes of loss do you accept? If the answer is "zero", you need shared storage. If the answer is "a few minutes is fine", replication is enough and far cheaper.

Containers do not move the way virtual machines do

This is where people building their first cluster are most often surprised. A virtual machine can be moved while running. A container cannot: it is shut down, moved, and started again.

So there is no such thing as uninterrupted migration for your containers, not even with a cluster. This is not a shortcoming but the nature of a container: a virtual machine carries its own kernel, a container uses the server's, and that kernel cannot travel.

The practical consequence: take this into account when deciding which work sits in a container and which in a virtual machine. If work that tolerates no interruption sits in a container, that work will stop on maintenance nights.

Set the order up front

On the night you update a node, the answers to these questions should already be ready: how many machines are on this node, how many can be moved, how many must be shut down, and which node the movable ones go to.

Working that out on the night itself is a bad idea. Write it once and keep it aside.

What Atlas does

Before a node is updated, Atlas lists the machines running on it and suggests you a migration order: which machine goes to which node. The suggestion is built to spread them across the online nodes in turn.

The container distinction is made explicitly here. For virtual machines the suggested method is migration while running; for containers it is the shut down, move, start chain. The product does not offer this as an option, it states that a container cannot be moved any other way.

Machines left to high availability management are marked too, because moving them by hand and leaving them to the manager's own decision are different things.

And the suggestion stays a suggestion: the migration command does not run by itself, you press the button.

An honest limit: today Atlas treats every running virtual machine as movable and does not check whether its disk is genuinely visible from the other node. So the product does not answer the storage question above on your behalf, you have to answer it once yourself. Read the suggestion knowing that.