Free Online Toolbox for developers

Silent failures: How to know when a cron job stops running

Most monitoring is built around things that answer. A web server responds to a request, a database accepts a connection, an API returns JSON. Scheduled jobs are different. A nightly backup, a cleanup script, an invoice export or a queue worker started by a timer does its work and exits. Nobody is waiting for its reply, so when it stops running, nothing complains. The failure is silence, and silence is very hard to alert on.

This article looks at why scheduled jobs fail quietly, why the usual fixes only cover part of the problem, and how heartbeat monitoring closes the gap with a single line of code.

How scheduled jobs fail without anyone noticing

A cron job can stop doing its work in more ways than most developers expect:

  • The scheduler is not running. The cron daemon was disabled during a migration, a container was rebuilt without it, or a systemd timer was never enabled after a reboot.
  • The environment changed. Cron runs with a minimal PATH and none of your shell profile. A script that works perfectly in your terminal fails under cron because it cannot find node, python3 or a credentials file.
  • The job runs but fails. A remote disk is full, an API token expired, or a database password was rotated. The script exits with an error that is written to a log nobody reads.
  • The job never finishes. A process hangs on a network call without a timeout, and every following run piles up behind it or is skipped by a lock file.
  • The machine is gone. The server was decommissioned or replaced, and the crontab did not come with it.

In every case, the visible symptoms appear much later: a restore that finds no recent backup, a customer asking why their report did not arrive, a disk that filled up because old files were never cleaned.

Why logs and email are not enough

Cron’s traditional answer is the MAILTO setting: any output from a job is emailed to an address. In practice this rarely holds up. Many servers have no working mail transfer agent, messages land in spam, and scripts that produce output on every run train people to ignore the emails. More importantly, MAILTO only fires when the job runs. If the scheduler is down or the server is gone, there is no output and no email.

Log aggregation has the same weakness. A log pipeline can tell you about errors in jobs that ran. It cannot easily tell you about a job that should have run at 03:00 and did not, because the absence of a log line is not an event.

Heartbeats: Turning silence into an alert

Heartbeat monitoring, also called a dead man’s switch, flips the direction of the check. Instead of a monitor asking the job whether it is alive, the job reports to the monitor after every successful run. The monitor knows the schedule, and if the report does not arrive within the expected interval plus a grace period, it raises an alert.

Because the alert is based on a missing signal, it covers every failure mode in the list above. A stopped scheduler, a broken environment, a crashed script, a hung process and a deleted server all produce the same result: no heartbeat, so you get an alert.

Adding a heartbeat to a job

The integration is usually one HTTP request at the end of the job. In a crontab entry, you chain a curl call to the heartbeat address after your script with the shell’s “and” operator, so the request is only made when the script finishes successfully. A few details make this reliable:

  • Because of the “and” chaining, the heartbeat is only sent after an exit status of 0, so a failing job is reported the same way as a missing one.
  • Give curl a short timeout and a couple of retries, and tell it to treat HTTP errors as failures. A slow network then neither hangs the job nor hides a problem.
  • Your script must return a non-zero exit code when something goes wrong. In Bash, enabling the errexit, nounset and pipefail options at the top of the script is a good start.

The same pattern works elsewhere. In a systemd service, an ExecStartPost line that requests the heartbeat address only runs after the main command succeeds. In GitHub Actions, add a final step that calls the URL. On Windows, end a Task Scheduler script with a PowerShell Invoke-WebRequest call. In application code, call the URL at the end of the worker function, after the database commit.

Choosing the grace period

The grace period is how long the monitor waits after the expected time before it alerts. Set it too short and normal variation in runtime will cause false alarms. Set it too long and you lose the benefit of knowing quickly.

A practical rule is to measure the job’s usual duration and add a comfortable margin. A nightly export that normally takes 10 minutes might get a grace period of one hour. A job that runs every five minutes might get a few minutes of slack. If the job can legitimately take much longer on certain days, such as month-end reports, give it a grace period that covers the slow days, or give that job a separate monitor.

Tools for the Job

You can build a basic heartbeat receiver yourself with a small web endpoint and a scheduled check, but then the receiver needs monitoring too, and it should not run on the same infrastructure as the jobs it watches. Hosted services avoid that loop. FastPing, for example, gives each job a unique URL that accepts a plain GET, POST or HEAD request with no API key, lets you set a grace period of up to 24 hours on top of the interval, and sends a recovery alert when the next successful call arrives. Its guide to cron job monitoring includes setup examples for Linux cron, systemd timers, Windows Task Scheduler and GitHub Actions. The free plan covers three monitors, enough to protect the backups and exports you care about most.

Start with the jobs that hurt most

You do not need to instrument every scheduled task on day one. Start with the jobs whose failure you would discover too late: database backups, data exports that customers rely on, certificate renewals and anything that deletes old data. Add a heartbeat to each one, stop one job on purpose to see the alert arrive, and you will have replaced a silent risk with a notification you can act on the same day.




Suggested Reads