Configuration went read-only: why the place it lives is different

You try to edit a file and permission is denied. You are root. There is space on the disk. You still cannot write.

Nothing is broken here. The place you are looking at is not a normal directory, and it is refusing writes on purpose.

The place configuration lives is not a directory

Proxmox keeps machine configurations, storage definitions, firewall rules and backup jobs in one place. It looks like a filesystem, it has folders and files, but behind it sits a small database, and that database is replicated to every node in the cluster.

The reason is simple: every server in a cluster has to see the same machine definition. If you are going to move a machine from one server to another, the far side must already know how that machine is defined. If the configuration were an ordinary file on one server's disk, the others would not know about it.

This design has three consequences, and all three turn up in daily life.

Consequence one: without a majority, writes are refused

If a server cannot see the majority of the cluster, writing shuts off. Reading keeps working, writing stops.

This is not a fault, it is a decision. If the network split in two and both halves could write, the same machine would end up with two different definitions. When the network came back nobody could say which one was right, and there would be no way to merge them. Instead the system chooses this: the side left in the minority stops writing.

How a majority is counted in a cluster, and why two-server setups are awkward, is covered in a separate article. The only thing to add here is that this is exactly what that majority rule feels like in practice: suddenly being unable to write.

Consequence two: the same structure runs even on a single server

Even without a cluster, this structure is running. A single server is a majority by itself, so normally nothing goes wrong.

But if the service providing that structure is not healthy, you will see the same symptom on a single server too. So "I have no cluster, this cannot happen to me" is not correct.

There is also a less known path: the database behind this structure sits on the local disk. When the local disk fills up, writes fail and it looks to you as though you cannot change settings. The symptom is on the configuration side, the cause is on the storage side.

Consequence three: whatever you write there goes to everyone

There is no such option as "let me change it only on this machine". What is written there reaches every node in the cluster.

The classic mistake made without knowing this is to make an experimental change on one node and assume it does not affect the others.

One more point: that place was designed for configuration, not for data. Small text files belong there; scripts, archives and backups do not. Do not get into the habit of leaving files there.

If you cannot write, check in order

First check the majority: can the server see the rest of the cluster? If it cannot, the real problem is in the network and the configuration file is innocent.

Then check the service: is the service that provides this structure running?

Then check the local disk: if it is full, the path above is in play.

And the last one, most often skipped: a node may have left the cluster. A network outage, a neighbour that got switched off, a misconfigured firewall rule. The symptom always looks the same, the cause is somewhere different every time.

What Atlas does

Atlas reads this structure directly as files, instead of running the system's query tool every time.

We measured why. Each call to that tool costs between one hundred and two hundred milliseconds of processor time and around one hundred megabytes of temporary memory. With the panel open there is a refresh every ten seconds, and each refresh needs eight to ten calls. In other words, the monitoring was producing a constant wobble on the very server it was monitoring.

Reading the same information from a file takes microseconds. The gap is thousands of times over, and that gap shows up on the user's machine as steadiness.

The second detail is more elegant: inside that structure there is a change counter that increments whenever the configuration changes. Atlas watches it. So the moment a setting is changed from the panel, the cache refreshes by itself; asking rarely and never showing stale information become possible at the same time.

This is another face of a principle written elsewhere in this wiki: the thing that watches should not be the thing that costs. The easiest mistake a monitoring tool can make is to slow down the system it watches, and thereby corrupt the very number it is measuring.