Logs: the only thing that grows without anyone deciding it should

Most of what fills your disk was added by you: virtual machines, backups, installation images. Each of them was a decision.

Logs are different. They grow because the system is running. Nobody ever said "let this file get bigger".

The growth is slow but unbounded

A log file might grow by a few kilobytes a day. That attracts no attention for months. A year later it is still small.

But it has no ceiling. And growth without a ceiling, however slow, becomes a problem if you wait long enough.

The multiplier nobody expects: failure accelerates writing

This is the real point of the article.

A healthy system writes little. If a check errors on every run, a line drops into the same file every run. For a job that runs every fifteen minutes that is ninety six lines a day, and it does not stop.

So the log grows fastest exactly when you are least able to look at it: while a failure is under way.

And the worst version of it

A problem appears, the logs accelerate, the disk fills. The full disk causes new problems. The new problems produce more logs.

From that point it becomes harder to find the original fault, because most of the errors on your screen are the result not of the first problem but of the full disk.

The rule: anything that writes must have a ceiling

"We will clean it up later" is not a plan. What is needed is a mechanical limit that runs without anyone having to remember.

This rule is not only for system logs: it holds for every file produced by the services you run, by scheduled jobs, and by collectors.

Two mistakes people make when setting rotation up

One: the rule breaking because of a file that does not exist. Some files only appear when the related feature is used. The rule has to be written so that it does not error when a file is absent; otherwise a feature you never use stops the rotation work entirely.

Two: the wrong rotation method. Renaming the file and sending the writing service a "reopen" signal is the common method, but it only works if there is a long-lived service. If the writer is a short-lived job that opens and closes each time, there is nobody to signal; in that case the correct method is to copy the contents and empty the file in place.

Both are silent failures: the rule is in place, the file keeps growing, and nobody notices that the rule is not working.

And the installed-machine question

If you write a rotation rule only in the install script, it never reaches machines installed before the rule existed. Those machines run for years without it.

The right way is to write the rule at every startup: that way older installations pick it up when they move to a new version.

What Atlas does

This whole article came out of Atlas's own measurement, so it needs to be told honestly.

Atlas runs as root on the customer's machine and writes to several log files: the watchman summary every fifteen minutes, the output of planned updates on every run, its own self-update record. It was measured and none of them were being rotated, because the installer wrote a rotation rule nowhere. On a live machine the watchman log had reached one hundred and eighty kilobytes in thirty two days and there was no mechanism to shrink it.

The number looks small and it was not a problem that day. What was a problem is that the growth was unbounded, plus the acceleration described above. The planned-update logs capture the entire output of the package manager on every run, which can reach megabytes per run.

Now the rule is written, and both traps above are handled deliberately: absent files are tolerated, and rotation uses the copy-then-empty method because the writers are short-lived processes run from a scheduled job.

And the rule is written at startup, not in the install script. The reason is exactly the question above: so that machines installed earlier also pick up the rule when they move to a new version.