Engineer Spent 48 Hours Deleting Logs Before Discovering an Inode Exhaustion Bug
A software engineer spent nearly 48 hours troubleshooting a persistent 'No space left on device' error on a Python worker, only to find the root cause had nothing to do with available disk space. The system's inode table — which tracks file metadata and names rather than raw storage — had been fully exhausted by thousands of small JSON sidecar files, cache directories, and swap files. Standard disk-space checks using df -h repeatedly showed free gigabytes, masking the real problem and sending the investigation in the wrong direction for most of the two-day period. A cleanup script the engineer wrote actually worsened the situation by copying files before deleting them, consuming even more inodes in the process. The issue was only identified around hour 20 when the engineer ran df -i, a separate command that reports inode usage rather than block usage.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in