PerlandFailure catalogue

The database ran out of disk space

Fataldatabase_disk_full

A MySQL or PostgreSQL server could not write its data because the disk is full, so it stopped or waits until space is freed.

What the log shows

Written example2 lines
2026-11-04T03:12:55.204118Z 9 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000014' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.
2026-11-04T03:22:55.211730Z 9 [ERROR] [MY-000035] [Server] Disk is full writing './binlog.000014' (OS errno 28 - No space left on device). Waiting for someone to free space... Retry in 60 secs. Message reprinted in 600 secs.

Your log will have other names, versions and times. Search a line of it from all failures.

What it means

A database writes to disk all the time: new rows, its transaction logs, temporary files for queries. When the disk (or the volume its data lives on) has no free space left, the operating system refuses the write with error 28, “No space left on device”.

What happens next depends on the database. MySQL often waits, repeating the message every minute until space is freed, and refuses the writes that need more room meanwhile. PostgreSQL ends the session that hit the full disk and may shut down when it cannot write its write-ahead log. Either way, what the server’s plugins or mods try to save is not stored while the disk is full.

Why it happens

  • The whole disk is full, often because of something else on the same machine: world backups, exports, log files, old container images.
  • The database grew. A plugin that logs every block change or chat line can fill gigabytes over time.
  • MySQL’s binary logs piled up. MySQL keeps them for a set time (30 days by default in 8.0 and later), and on a busy server they can take more room than the data itself.
  • The volume has a fixed size smaller than the data needs.

How to fix

  1. Find what fills the disk. On Linux, df -h shows how full each disk is and du -sh * in a folder shows what takes the room. On Docker, docker system df shows images, containers and volumes.
  2. Free space with what is safe to delete first: old exports and backup copies you have elsewhere, old log files, unused Docker images (docker image prune).
  3. Do not delete the database’s own files by hand to make room (data files, the redo or write-ahead log folders such as pg_wal). That corrupts the database.
  4. MySQL: once it has room, remove old binary logs from inside the database with PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;, and set binlog_expire_logs_seconds lower if you do not need them for long.
  5. If a plugin’s tables keep growing, look at its settings for a retention or purge option.
  6. Start or restart the database once there is room. MySQL that was waiting goes on by itself; check the server’s plugins reconnect.

In Perland

A server’s database sidecar is a MySQL or PostgreSQL container next to the server, with its data in a Docker volume on the same host as the servers. This issue appears in the sidecar’s diagnosis (the stack’s diagnosis with service mysql or postgres). Free disk space by deleting files the servers no longer need, such as old exports and world backups; if the host’s disk is full for other reasons, ask whoever runs the machine.

References

Details the diagnosis reports

Perland's server diagnosis returns this failure as the code database_disk_full with these parameters, each present only when the log names it.

ParameterMeaning
databaseWhich database printed it, `mysql` or `postgres`.

Log lines this page covers

  • No space left on device
  • Operating system error number 28
  • Disk is full writing
  • The table '…' is full