The Saturday Night Call — What Happens When Nobody Is Watching Your Server

It was 11pm on a Saturday when my phone rang.

“The call came at 11pm on a Saturday. The disk was full, the web server had stopped, and nobody had been watching.”

— AV Services — incident log, Mumbai e-commerce client

A Mumbai e-commerce founder. His website was down. Orders were failing. He had no idea for how long ? his team only noticed when a customer complained on Instagram.

The server had been running unattended for two years. No monitoring. No patching. No one whose actual job it was to watch it.

What Two Years of Neglect Looks Like

• Kernel unpatched for over 700 days
• Disk at 94% capacity
• Backup job silently failing for four months
• Three unused open ports, one with a known CVE
• No monitoring. No alerts. No one to call.

How It Happens

The server gets set up by a developer or a freelancer. It works. The business moves on. The person who set it up moves on too. Nobody replaces them. The business grows. The server quietly accumulates risk ? unpatched vulnerabilities, filling disks, failing backup jobs that nobody checks. Until something breaks.

The Cost Nobody Calculates in Advance

That Saturday night cost my client more than six months of what he would have paid me on retainer. Lost orders, emergency recovery work, customer trust, and his own time spent managing a crisis instead of his business.

What Proactive Management Actually Covers

• Security patches applied and tested
• Disk, memory and CPU monitored with alerts
• Backup jobs verified ? not just scheduled, actually confirmed
• Failed services detected and restarted before they cause downtime
• Monthly written report so you know exactly where things stand

If Your Server Is Running Unattended Right Now

I offer a free 30-minute Linux server audit. Read-only access only ? I make no changes. You receive a written report within five business days telling you exactly what the state of your server is.

No obligation. No disruption. Just clarity.

If your business is based in Mumbai, AV Services provides Linux server management in Mumbai on a fixed monthly retainer ? with on-site response within four hours and 24/7 remote availability.

Email: arun@avservices.in
WhatsApp: wa.me/919220560056

What the Recovery Actually Involved

The immediate problem — website down — was the disk at 94%. The web server had stopped accepting requests because it could not write access logs. Fix that first, get the site back up, then deal with everything else.

Finding what was consuming the disk:

du -sh /* 2>/dev/null | sort -rh | head -20

Two years of unrotated application logs — 38GB in /var/log alone. The application had been logging every request, every error, every debug line, with no rotation configured. The web server logs had not been touched since the developer left. A secondary culprit: the backup job that had been failing for four months had been writing partial files to a local directory before failing — 11GB of incomplete backup fragments nobody knew existed.

Cleanup, in order:

# Clear old logs safely — check before deleting
ls -lh /var/log/*.log | sort -k5 -rh | head -20

# Truncate rather than delete active log files
truncate -s 0 /var/log/nginx/access.log
truncate -s 0 /var/log/nginx/error.log

# Remove the incomplete backup fragments
rm -rf /var/backup/incomplete/

# Confirm disk freed
df -h /

Disk went from 94% to 61% in 20 minutes. The web server restarted cleanly. Orders started processing again at 12:34am — 94 minutes after the call came in.

---

What We Fixed the Following Week

The Saturday night fix was triage. The real work happened over the following week once the client was on retainer. This is what two years of unmanaged drift looked like, worked through systematically.

Kernel patching. The server was on Ubuntu 20.04 with a kernel that had not been updated in 714 days. Running unpatched kernels is not just a security risk — it means missing bug fixes, driver updates, and performance improvements. The patch cycle took one maintenance window and a reboot. Downtime: 4 minutes.

Log rotation. Configured logrotate for every application writing to disk — nginx, the e-commerce application, MySQL slow query log, PHP-FPM. Each set to daily rotation, 30 days retention, compressed. The 38GB that had accumulated would not happen again.

Backup. The broken backup job was writing to a local directory that was itself on the same disk it was backing up — which would not have helped in a disk failure scenario anyway. Replaced with rsync to a remote destination, with exit code checking, email alerts on failure, and a weekly restore test to a temporary location.

Open ports. Three ports were open that served no current purpose — remnants of services the developer had installed and abandoned. Closed. The one with a known CVE was for a version of a monitoring agent that had not been updated since 2022. Removed entirely.

Monitoring. Set up basic alerting: disk usage alert at 75%, CPU alert sustained above 85% for 5 minutes, memory alert above 90%, and a service check on nginx and MySQL every 60 seconds. The Saturday night incident would have triggered a disk alert 3 weeks earlier if monitoring had been in place.

---

Two Years Since That Saturday

The client has been on retainer since. In two years, there has been one incident — a hardware failure on the hosting provider’s side, resolved within the provider’s SLA window. No data was lost because the backup was current and the restore had been tested the previous month.

The disk has never gone above 71%. The kernel is patched monthly. The backup restore test runs on the first Monday of every month and the result goes into the health report.

The Saturday night call cost him more than six months of retainer fees — in lost orders, emergency work, and his own time. The retainer costs Rs.15,000 a month for one server. The maths is not complicated.

If tonight is your Saturday night call, the fastest next step is Linux Data Recovery — no cure, no pay.

←What Does AV Stand For in AV Services? (It Is Not Audiovisual)The Backup That Was Never There→