That night
Eleven thirty, about to sleep. My phone buzzed: site response timeout.
Probably network jitter. Looked at it and put the phone down. Half past midnight it buzzed again — disk space exhausted.
Got up, opened the laptop, made instant noodles.
Debugging
Disk full. First guess: logs were never cleaned. Logged in — the log directory was 200MB. Not it.
Kept digging. A cache directory held hundreds of thousands of small files. Every deploy had written there and nothing ever cleaned it up.
Cleaning up
rm -rf on three hundred thousand files does not finish in a second. It took over an hour. During that hour the site was limping.
Afterwards I added a cron job to clean files older than seven days, and a monitor that warns at 80% disk usage instead of 95%.
What I actually took away
After years of ops you develop a strange illusion that systems do not break. They break all the time. You just do not see it because someone — usually you — gets up at night.
The noodles went cold. I ate them anyway.
Comments