AI agent stopped running? How to find out and get alerted

A scheduled agent can stop for days before anyone notices the missing summaries or the untriaged queue. Here is how to find out why, and how to hear about it the next time.

Updated

Short answer

Check that the scheduler fired, then read the run log for model or API errors, rate limits, and timeouts. To get alerted, wrap the agent so it pings a heartbeat URL on success and a fail URL on error. A missing ping means the run never started or never finished.

Try a heartbeat monitor free

Three ways a scheduled agent stops

An agent on a schedule is a cron job with more ways to fail. It depends on the scheduler, on one or more model APIs, on tools it calls, and on input data. When one of those breaks, nothing usually tells you.

  • It never started. The scheduler did not fire, the server or container was gone, or the workflow was turned off.
  • It started and hung. A model call or tool call never returned, and nothing set a timeout.
  • It ran and failed. An API error, rate limit, or expired key ended the run, or it finished with nothing useful.

Error handlers and log alerts only cover the last case, and only when the process gets far enough to run them.

Find the cause

CauseHow to checkFix
Scheduler did not fireNo log line at all for the expected timeCheck the cron entry, workflow trigger, or scheduled workflow is still enabled and uses the time zone you expect
Process killed or timed outLog stops mid-run; exit code 137 or a platform timeoutRaise the memory or time limit, or split the work into smaller runs
Model or API error401, 403, or 5xx responses in the logRotate the API key, check billing on the provider account, retry 5xx with backoff
Rate limited429 responses, often on the largest runsLower concurrency, add backoff, or spread runs out
Hung on a callRun started but never logged an endSet a timeout on every model and tool call, and on the whole run
Ran but produced nothingRun ended normally with zero items or an empty resultCheck the input source, and treat an empty result as a failure in code

Two changes prevent most silent runs: a hard timeout on the whole run, for example timeout 30m ./run-agent.sh, and a non-zero exit when the result is empty. Both turn a quiet problem into an exit code you can report.

Get alerted with a heartbeat

A heartbeat monitor waits for each run to check in. You set how often the agent runs and how late a run may be. The agent requests a unique URL when it finishes. If the request does not arrive in time, you get an alert.

This covers all three cases. A run that never started and a run that hung both miss the ping. A run that failed can report it straight away through the fail URL, with its exit code.

run-agent-monitored.shShell
#!/bin/bash
BASE=https://heartbeat.uptimeio.com/heartbeat/YOUR_HEARTBEAT_TOKEN

curl -fsS -m 10 "$BASE/start" > /dev/null
timeout 30m ./run-agent.sh
code=$?

if [ "$code" -eq 0 ]; then
  curl -fsS -m 10 --retry 3 "$BASE?exit_code=0" > /dev/null
else
  curl -fsS -m 10 --retry 3 "$BASE/fail?exit_code=$code" > /dev/null
fi
CallWhat it does
/startMarks the start of a run so the duration is recorded. It never alerts and does not reset the timer
Heartbeat URLMarks a successful run and resets the timer
/failOpens an incident right away, with the exit code

Report a bad result, not only a crash

The heartbeat service does not read your agent's output. Your code has to decide what a bad run looks like. Common checks:

  • Empty output: zero tickets triaged, an empty summary, no rows written.
  • Too few items: far fewer processed than a normal run.
  • Too many errors: more than a set share of model calls failed or were retried.

Exit non-zero when a check fails and the wrapper sends it to /fail. To keep a record of each run, send a POST ping with a small JSON metadata object, such as items processed and tokens used.

ping with metadataShell
curl -fsS -m 10 -X POST "https://heartbeat.uptimeio.com/heartbeat/YOUR_HEARTBEAT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "metadata": { "items": 182, "tokens": 41000 } }'

Set it up in UptimeIO

  1. Create a monitor and choose the Heartbeat type.
  2. Set the interval to match the agent schedule, and a grace period that covers the slowest run.
  3. Copy the heartbeat URL into the wrapper above.
  4. Choose where alerts go: email, Slack, Telegram, Pushover or a webhook.
  • Plans: heartbeat monitors, including /start and /fail, are on every plan. The shortest interval is 5 minutes on Free and 1 minute on Pro and Scale.
  • Run history: each event in Recent Heartbeats shows its exit code and duration. A reported failure shows as Failed.
  • Incidents: a missed run opens an incident after the interval plus grace period; it resolves after 3 on-time pings in a row.
  • Secrets: anyone with the URL can send pings, so keep it out of prompts and logs the agent can repeat.

Questions

Why did my AI agent stop running?
Usually the scheduler stopped firing it, the process was killed or timed out, or a model API returned errors or rate limits. Check that a run started at the expected time, then read its log for 4xx, 5xx, or 429 responses.
How do I get alerted when a scheduled agent fails?
Have the agent ping a heartbeat URL when a run succeeds and call the fail URL when it fails. The monitor alerts on the failure report, and also when a ping does not arrive on time.
How do I catch an agent that hangs?
Put a timeout on the whole run, for example with the timeout command. Even without one, a hung run never sends its final ping, so the heartbeat monitor alerts once the grace period has passed.
Can the monitor check whether the agent output is correct?
No. Check the result in your own code, exit non-zero when it is wrong, and report that through the fail URL.
Can I monitor AI agents on the Free plan?
Yes. Heartbeat monitors are on every plan. The shortest interval is 5 minutes on Free and 1 minute on Pro and Scale.

Related

Start monitoring in a few minutes

Start monitoring free

Free plan · 50 monitors · No credit card required