A sudden No space left on device alert on a critical Linux host is often a storage problem, but the real cause is usually the journal.
The default systemd-journald configuration on modern distributions can quietly consume a large amount of disk space if retention rules are not managed. What looks like a single application failure often turns out to be a root partition that has been silently filled with logs.
Why this happens
On busy servers, the journal captures everything from service startups to warnings, errors and application churn. If the configuration leaves the system with no size or time limits, log volume can grow far beyond what is reasonable.
That turns a routine log burst into a full system problem. Suddenly there is no space left for updates, package state, or even normal service writes. The issue becomes operationally severe even though the root cause is not the application itself.
Set real limits
The first fix is to define what you are willing to keep. In /etc/systemd/journald.conf, the main controls are usually SystemMaxUse, RuntimeMaxUse and MaxRetentionSec.
A practical starting point is to cap the journal at a fixed size such as 1 GB or 2 GB on a server with limited disk, and to keep a shorter retention window for its rotating periods. This prevents the journal from becoming the largest consumer of the root filesystem without affecting the ability to troubleshoot.
[Journal]
Storage=auto
SystemMaxUse=1G
RuntimeMaxUse=256M
MaxRetentionSec=1month
After editing the file, restart the service:
sudo systemctl restart systemd-journaldOffload important logs
For production hosts, the journal should rarely be the only destination for logs. If the system is important enough to need monitoring, the logs should also be forwarded to a central logging platform or a dedicated log host.
This reduces the risk of a single noisy service filling the local disk. It also gives you better historical context when the system is under stress.
Watch for noisy services
Log bloat often appears because one application is generating repeated warnings or debug output. This is not always obvious at first. A service restart, a misconfigured polling loop or a noisy library can create a fast-growing journal without breaking the application process itself.
Check the journal for repeated warnings and then fix the source. If the application is writing too much, reduce verbosity or move the logs to a more appropriate logging path.
Do the boring maintenance
Log management is not glamorous, but it prevents the kind of outages that are expensive and avoidable. A host that has enough disk for a clean boot but no room for a log write is a host that has already lost the plot.
Set the limits, check the retention, and keep critical logs off the root filesystem where practical. That is how you prolong the life of a server without waiting for a failure to teach you the lesson the hard way.