Disk Full on a Linux Web Server: How fail2ban Logs Silently Filled the Drive

A Mumbai healthcare business running a public-facing web server on Debian called with a straightforward complaint: the site was down. No warning. No gradual slowdown. Just down.

Root cause: disk at 100%. The culprit was fail2ban.

What happened

fail2ban is supposed to protect your server. On an internet-facing server it works hard — scanning logs, banning IPs, writing ban records. On a busy server getting hit by the usual background noise of bots and scanners, fail2ban generates a lot of log data.

The /var/log/fail2ban.log file had been growing for months. Log rotation was configured — but the rotation policy had not been tuned for the actual volume fail2ban was producing. The compressed archives were accumulating. The uncompressed active log was several gigabytes. Combined with application logs and system logs, the root partition ran out of space.

When a Linux server’s disk hits 100%, things stop working. Apache cannot write to its log file. MySQL cannot write transactions. PHP cannot write sessions. The web server stops serving requests. From the outside it looks like the site is down. From the inside it looks like every service is broken at once.

How to check if fail2ban logs are your problem

Check disk usage first:

df -h

If the root partition or /var is at or near 100%, find what is taking the space:

du -sh /var/log/* | sort -rh | head -20

If fail2ban.log or the fail2ban log archives are near the top of that list, you have the same problem.

Check the size of the active fail2ban log:

ls -lh /var/log/fail2ban.log*

Check how many ban events are being generated:

grep -c "Ban" /var/log/fail2ban.log

On a busy internet-facing server, that number can be in the tens of thousands per day.

The fix

1. Free up space immediately

If the disk is at 100% and services are down, the first priority is creating enough space to get things running again:

# Compress the active fail2ban log manually
gzip /var/log/fail2ban.log

# Restart fail2ban so it creates a fresh log file
systemctl restart fail2ban

# Check how much space you recovered
df -h

Then restart the services that stopped:

systemctl restart apache2
systemctl restart mysql

2. Fix log rotation for fail2ban

On Debian, fail2ban log rotation is handled by logrotate. Check the existing config:

cat /etc/logrotate.d/fail2ban

A sensible configuration for a busy internet-facing server:

/var/log/fail2ban.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    postrotate
        fail2ban-client flushlogs 1>/dev/null
    endscript
}

rotate 7 keeps 7 days of compressed archives. daily rotates every day instead of weekly. Adjust based on your server’s ban volume.

3. Monitor disk usage going forward

The real fix is knowing about disk growth before it becomes an incident. Add a Nagios or cron-based disk check that alerts when any partition exceeds 80%:

# Add to crontab — alerts when disk exceeds 80%
0 * * * * df -h | awk 'NR>1 {gsub(/%/,""); if ($5 > 80) print $0}' | mail -s "Disk warning: $(hostname)" arun@avservices.in

An alert at 80% gives you time to act. An alert at 100% means you are already in an incident.

Why this happens on internet-facing servers specifically

A server sitting behind a firewall with limited external exposure generates modest fail2ban activity. An internet-facing web server — running a public website, an API, a mail server — gets hit constantly. Bots, scanners, credential stuffers, vulnerability probes. fail2ban is doing its job, banning thousands of IPs. But every ban is a log line. On a default log rotation config designed for a quieter server, the logs outgrow the rotation schedule.

The Hufort.in server was handling normal healthcare web traffic plus the background noise every public server attracts. fail2ban had banned over 40,000 IPs in the months since the server was set up. Nobody had checked the log size. Nobody had tuned the rotation policy for the actual volume being generated.

This is not a fail2ban problem. fail2ban was working correctly. It is a monitoring gap — nobody was watching the disk, and the log rotation config had never been validated against real usage.

AV Services manages Linux servers for businesses in Mumbai on monthly retainer. Disk monitoring, log rotation, fail2ban configuration, and backup verification are part of every managed server. If your server is unmonitored, the free Linux server audit is the place to start.