The Alert That Never Fired
By Arun Valecha, AV Services · Linux Infrastructure Expert since 1999
“The failure-path alert had never been able to fire. Not during a real failure, not ever.”
— AV Services — infrastructure check finding, existing retainer client, Mumbai
A retainer client’s backup system looked healthy. Nightly borg backups completed on schedule, every night, for months. If anything went wrong, a Telegram alert was supposed to fire immediately.
It never would have. Not once.
We found this during a routine infrastructure check, not during an actual failure. The backup script sent its alert by calling out to api.telegram.org. That call needs DNS to resolve. This particular box had never had a DNS resolver configured, not since it was set up. Static network config, no nameserver line, nothing managing it. Every alert call had been failing silently since day one.
Here’s the part that made it worse. The script used set -e, a common bash safety habit that kills a script the moment any command fails. When the Telegram call failed, set -e killed the entire script right there, before it could reach the fallback logic that was supposed to try email instead. So the failure-path alert, the one that exists specifically to catch a bad night, had also never been able to fire. Not during a real failure, not ever.
The backups themselves were fine. They connect to the backup destination by IP, not hostname, so the missing DNS resolver never touched them. That’s the trap. Everything that mattered every day worked. The one thing that mattered on a bad day was dead from the start, and nothing about daily operation would ever have shown it.
What we fixed
- Added DNS resolvers directly to the network config, not just relying on a service that might not be running
- Rewrote the alert script so email and Telegram run independently, neither one able to block or kill the other
- Verified it end to end with a live manual run: confirmed clean exit, confirmed the Telegram message actually arrived
Why this matters beyond one client
A backup that “completes successfully” every night tells you the backup ran. It doesn’t tell you the alerting around it works. Those are two separate systems, and testing one doesn’t test the other. If your alerting has never actually fired, you don’t know if it works. You only know it hasn’t been needed yet.
We test alert paths on every retainer client now, not just backup jobs. An alert you’ve never seen fire is a hypothesis, not a safeguard.