The Backup That Was Never There

AV Services · avservices.in · Linux Infrastructure, Mumbai · Since 1999

“The backup job ran every night for 97 days. Every morning the log said it completed. There were zero valid backups.”

— AV Services — free audit finding, Mumbai professional services firm

The call came in on a Tuesday morning.

A jewellery manufacturer in Mumbai — 500-odd employees, a factory running full shifts — had lost their Linux server. Not a crash. Not a hack. Someone had accidentally formatted the drive running their ERP. Transactions, inventory, production records. Gone.

The first thing they asked was: what about the backup?

The backup job had been running every night for months. Green status. No alerts. Nobody had ever done a restore test — why would they, the job was completing successfully. What they didn’t know was that the destination folder had a permissions error. The job ran. It wrote nothing. For months.

So there was no backup. There had never been a backup. Just a cron job completing silently into a void.

---

This is the part that stays with me about that job.

The MD wasn’t angry about the formatted drive. Accidents happen. He was angry about the backup — because someone had told him, at some point, that backups were running. And they were. The job was running. The files just weren’t there.

That distinction matters. A backup job running and a backup working are two completely different things. Most businesses have one. Almost none have the other.

---

The recovery took multiple site visits. Data forensics on the formatted volume, partial reconstruction, full ERP restoration. It worked. The cost was Rs.33,000 — which, honestly, was underpriced for what it involved. The correct rate for that job is Rs.75,000 to Rs.1,50,000.

A monthly retainer at Rs.15,000 would have caught the permissions error in the first audit. The restore test we run every month would have flagged it on day one. The incident wouldn’t have happened.

The retainer is now in place.

---

If you run a business on a Linux server — ERP, billing, inventory, anything — there’s one question worth asking your IT person today:

When did we last actually restore from the backup?

Not “is the backup job running.” Not “when did the backup last complete.” When did someone pull a file, or a database, or a full volume, from the backup and confirm it works.

If the answer is “I’m not sure” or “we haven’t” — you don’t have a backup. You have a backup job.

Those are not the same thing.

---

What the permissions error actually looked like

The backup destination was an NFS mount — a network share from a NAS device on the same LAN. The mount point was /mnt/backup. The backup script ran as root. The NFS share had been configured with root_squash enabled — a common NFS security setting that maps root on the client to an anonymous user on the server. That anonymous user had no write permission to the backup directory.

So every night, the backup script ran as root, tried to write to /mnt/backup, got a permission denied error, and exited. The exit code was non-zero. But the script had no error handling — it did not check the exit code, did not log to a separate file, and did not send an alert. The cron log showed the job fired. That was all anyone checked.

Catching this during an audit takes 30 seconds:

# Check who owns the backup destination
ls -la /mnt/backup/

# Try writing to it as the backup user
sudo -u backupuser touch /mnt/backup/test_write && echo "Write OK" || echo "Write FAILED"

# Check NFS mount options
mount | grep backup

If the write test fails, you have found the problem before it becomes an incident. The jewellery manufacturer had no one running this check. The NFS mount had been configured once and never verified.

---

How to run a monthly restore test on a Linux server

A restore test does not have to be a full production restore. For a MySQL database backup, the minimum test is: decompress the backup file, load it into a temporary database, and confirm the tables exist and contain data. The whole process takes under 10 minutes.

# Pick the most recent backup
BACKUP_FILE=$(ls -t /backup/mysql/db-*.sql.gz | head -1)
echo "Testing: $BACKUP_FILE"

# Check the file is not empty
ls -lh "$BACKUP_FILE"

# Decompress and check the SQL header
zcat "$BACKUP_FILE" | head -20

# Create a temporary database for the restore test
mysql -u root -e "CREATE DATABASE IF NOT EXISTS restore_test;"

# Load the backup into the test database
zcat "$BACKUP_FILE" | mysql -u root restore_test

# Confirm tables loaded
mysql -u root restore_test -e "SHOW TABLES;" | wc -l

# Clean up
mysql -u root -e "DROP DATABASE restore_test;"
echo "Restore test complete"

The SHOW TABLES line count tells you whether the restore produced anything. If it returns 0, the backup is empty or corrupt. If it returns the expected number of tables, the backup is usable. This is the test. It takes a directory listing, a decompress, a database load, and one query.

For the jewellery manufacturer, we now run this test on the first Monday of every month. The result goes into the monthly health report — date tested, backup file tested, table count returned, pass or fail. Three lines. Unambiguous. If it ever fails, we know before the business does.

---

The three backup failures I see most often

After 25 years of auditing Linux servers in Mumbai and across India, the backup failures I find are almost always one of three things.

The destination is full or unreachable. The backup job runs, finds it cannot write, and exits silently. The destination might be a full disk, an unmounted NFS share, an S3 bucket with expired credentials, or a USB drive that was unplugged. The job looks fine from the cron log. The backup does not exist.

The backup format changed. A software update changed the output format of the backup utility — mysqldump, pg_dump, borgbackup. The new output is incompatible with the restore procedure. Files are produced, sizes look plausible, but the restore fails. This is what CVE-2026-31431 exposed at scale — the same pattern, at the kernel level.

The restore procedure was never documented. The backup runs correctly and produces valid files. But when a restore is needed, nobody knows the exact procedure — which flags were used, which database user has the required permissions, whether the restore needs to happen on a stopped service or a running one. The backup exists. The restore fails because the process was never tested end to end.

All three are caught by a monthly restore test. None of them are caught by checking whether the cron job ran.

---

AV Services manages Linux servers on monthly retainer for businesses in Mumbai and across India. Every retainer includes monthly backup restore testing. If you want someone to check whether your backup actually works, start with a free 30-minute audit call — no access needed, no commitment.

If this story sounds familiar because it’s happening to you right now, not as a cautionary tale, start here: Linux Data Recovery — no cure, no pay.

←The Saturday Night Call — What Happens When Nobody Is Watching Your ServerHow to Check If a Linux Cron Job Actually Ran — Not Just That It Was Scheduled→