Cron job not running? How to find out and get alerted

Cron gives you no error when a job fails and nothing at all when it never starts. Here is how to find the cause, and how to get an alert the next time it happens.

Updated

Short answer

Check the job as the user that owns it with crontab -l, use absolute paths, and look at the job log. To get alerted, add a heartbeat ping that runs only when the job succeeds, so a missing ping means the job failed or never ran.

Try a heartbeat monitor free

Two ways a cron job fails

A website that goes down returns an error to whoever requests it. A cron job has nobody requesting it. If the nightly backup stops, nothing complains. Often the first sign is a missing backup on the day you need it.

It helps to separate the two cases, because the fixes differ:

  • It never ran. The schedule is gone, the server was off, or cron could not start the command.
  • It ran and failed. The script exited with an error, or finished without doing its job.

Logs can explain the second case after the fact. Neither case sends you anything.

Find the cause

Most missing runs come from the first three rows.

CauseHow to checkFix
Different environmentThe command works in your shell but not from cronUse absolute paths, or set PATH at the top of the crontab
Wrong user or bad entryRun crontab -l as the user that should own the jobFix the schedule syntax and make the script executable
Server was downCompare the schedule with the uptime or reboot timeCron skips missed runs. Use a systemd timer with Persistent=true
Overlapping runsTwo copies of the job in ps at onceWrap the command in flock -n
Schedule removedThe entry is missing after a deploy or container rebuildKeep the crontab in version control and deploy it with the app
Time zoneThe job runs at the wrong hour, or twice around DST changesCron uses the server time zone, often UTC. Schedule in UTC

A crontab that avoids the first, fourth and logging problems looks like this:

crontabShell
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# m h dom mon dow command
0 2 * * * flock -n /var/lock/backup.lock /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

If you are unsure what a schedule means, paste it into the cron expression generator to see its next run times.

Get alerted with a heartbeat

A heartbeat monitor (also called a dead man's switch) waits for your job to check in. You tell it how often the job runs. The job sends a request to a unique URL when it finishes. If the request does not arrive on time, you get an alert.

The monitor does not need to know why the job went quiet. A job that never started, crashed halfway, or ran on a server that was off all look the same: no ping.

Ping only when the job succeeds

Send the ping after the job, joined with &&. The shell runs the second command only if the first exits with status 0, so a failed run sends nothing.

crontab: run at 02:00, ping on successShell
0 2 * * * /opt/scripts/backup.sh && curl -fsS -m 10 --retry 3 https://heartbeat.uptimeio.com/heartbeat/YOUR_HEARTBEAT_TOKEN > /dev/null
FlagWhy
-fTreat an HTTP error as a failure
-sSNo progress output, but still print errors
-m 10Give up after 10 seconds instead of hanging
--retry 3Retry a brief network failure so one lost request is not reported as a missed run

If the ping is the last line of a script, add set -euo pipefail at the top. Without it, the script keeps going after a failed step and still sends the ping.

backup.shShell
#!/bin/bash
set -euo pipefail

pg_dump mydb | gzip > /backups/mydb.sql.gz

curl -fsS -m 10 --retry 3 https://heartbeat.uptimeio.com/heartbeat/YOUR_HEARTBEAT_TOKEN > /dev/null

With systemd, put the ping in ExecStartPost. In a Type=oneshot service it runs only after ExecStart succeeds. Persistent=true on the timer also runs a job that was missed while the machine was off.

/etc/systemd/system/backup.service and backup.timerINI
# backup.service
[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
ExecStartPost=/usr/bin/curl -fsS -m 10 --retry 3 https://heartbeat.uptimeio.com/heartbeat/YOUR_HEARTBEAT_TOKEN

# backup.timer
[Unit]
Description=Run the nightly backup

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now backup.timer.

Pick a grace period

A heartbeat monitor has two settings. The interval is how often the job runs. The grace period is how late a ping can be before you are alerted. Because the ping is sent when the job ends, the grace period has to cover how long the job takes.

Base it on your slowest normal run. A backup that takes 10 to 20 minutes needs about 30. Too short and a slow night pages you; too long and a real failure waits.

Set it up in UptimeIO

  1. Create a monitor and choose the Heartbeat type.
  2. Set the interval and grace period.
  3. Copy the heartbeat URL and add it to your job as shown above.
  4. Choose where alerts go: email, Slack, Telegram, Pushover or a webhook.

Heartbeat monitors are on every plan, including Free. A few details worth knowing:

  • Intervals: 1 minute to 24 hours, with a grace period of 1 second to 24 hours. The shortest interval is 5 minutes on Free and 1 minute on Pro and Scale.
  • Requests: GET or POST, no authentication. A POST can carry a small JSON object, such as how long the job took.
  • Incidents: an incident opens when a ping is late, and resolves after 3 on-time pings in a row.

If you get an alert but the job did run, check that the ping comes after the job and that the URL is correct. The -fsS flags make curl print the error in your job log.

Questions

Why is my cron job not running?
Usually because cron runs with a minimal environment, the crontab belongs to a different user, the server was off at the scheduled time, or two runs overlapped. Check crontab -l as the right user, use absolute paths, and read the job log.
How do I get an alert when a cron job fails?
Add a request to a heartbeat URL after the job, joined with &&, so it is sent only on success. The heartbeat monitor alerts you when the request does not arrive, which also covers jobs that never started.
What grace period should I use?
Enough to cover your slowest normal run, since the ping is sent when the job ends. A job that takes 10 to 20 minutes needs about 30.
Does cron run a job it missed while the server was off?
No. Cron skips missed runs. A systemd timer with Persistent=true runs the job when the machine comes back, and a heartbeat monitor tells you the run was missed.
Can I monitor cron jobs on the Free plan?
Yes. Heartbeat monitors are on every plan. The shortest interval is 5 minutes on Free and 1 minute on Pro and Scale.

Related

Start monitoring in a few minutes

Start monitoring free

Free plan · 50 monitors · No credit card required