Skip to main content

Job alerts

Every scheduled job runs from the unified job launcher, which replaced the CronJobs. Job alerts read Postgres, not kube-state-metrics. Use this page to triage an alert, find what a job depends on and run read-only checks.

Rules live in base/telemetry/ as ConfigMaps labeled grafana_alert: "1". homelab-deployment's Grafana alert sidecar loads them into the Fantasy folder. They query run_log and launcher_heartbeat through the FantasyFootballDB datasource and route to the ntfy contact point. They read columns (status, job_name, end_reason, attempt, chain_root_id, finished_at), never keys inside run_log.trigger.

Which alert fired?​

Schedule fields, not comments that assume UTC, determine local run times. The generated schedule inventory lists every job, command and timezone.

AlertFires whenFirst check
Job launcher heartbeat stalelauncher_heartbeat.at is older than 5m, or there is no rowHeartbeat stale
Job launcher dead-letter queue not emptyqueue_depths->>'job_runs.dlq' is above 0DLQ not empty
Job launcher cannot read its queuesThe heartbeat is fresh but job_runs or its DLQ depth is null, for 2mQueue depth unknown
Job launcher job_runs backlogjob_runs depth stays above 0 for 15mBacklog
Launcher run failed (critical / warning jobs)A job at that alert level had a run end failure or timeout on its last attempt, or skipped for a reason that needs a person, in the last 10m. One series per jobRun failed
Job launcher chain ran past max_durationA chain root ended timeout with end_reason = 'max_duration' in the last 10mChain past max_duration
Rumor action incomplete or failed processingA failure or partial_failure was recorded in the last 26hApplication outcome

What covers a missed or hung run?​

There is no per-job missed-schedule rule.

SituationWhat you see
The launcher is downThe heartbeat alert. Once it's back, a slot still inside its catch_up window fires late
A scheduled slot can't start inside its catch_up window because the pod cap is fullskipped (expired), reported by the run-failed rule
An event run finds its job busy or can't start in timeNo alert for a job with a schedule or chain step, by design: the next event or the scheduled slot covers it. A job with on_event and neither alerts, since nothing else would run it. When the busy run is itself an event run, it waits instead of skipping overlap, and ends skipped (expired) only if it can't start within its debounce window
The launcher was down for longer than a slot's windowThe slot never fires and leaves no run_log row. Compare run_log with the schedule inventory after an outage
A run hangsIt ends timeout at its deadline, reported by the run-failed rule on the last attempt
A run succeeds but did nothing usefulNo alert. Check the job's logs and application outcomes

Launcher alerts​

Launcher heartbeat stale​

The launcher writes the row every 15s on its own thread, so a stale row means the process is down or stalled, or the database is unreachable from it. Nothing scheduled or on-demand starts meanwhile; catch-up windows cover short outages.

kubectl get deploy,pod -l app=fantasy-job-launcher -n default
kubectl logs deploy/fantasy-job-launcher -n default --tail=100

A launcher that refuses to start logs Job catalog is not usable; fix the catalog or the env it names.

Launcher DLQ not empty​

job_runs.dlq holds run commands the broker dead-lettered, such as a message that waited past the 6h TTL while the launcher was down. Inspect without consuming:

kubectl exec -n default deploy/fantasy-broker -c rabbitmq -- \
rabbitmqctl list_queues -p / name messages_ready messages_unacknowledged

Decide with the owner whether each run is still wanted before replaying or purging; don't purge to silence the alert.

Launcher queue depth unknown​

The heartbeat is arriving but the launcher couldn't read a queue's depth: job_runs or job_runs.dlq is missing (the broker definitions didn't load) or the broker is unreachable from the launcher. It can't consume either way. Check rabbitmqctl list_queues as above and the launcher log for Launcher queue depths unavailable.

Launcher job runs backlog​

job_runs has had messages for 15 minutes, though the launcher takes them promptly. The heartbeat thread is alive but the consumer is stalled or blocked. Check the launcher log and run_log rows stuck in queued.

Launcher run failed​

These rules are generated from base/launcher/catalog.yaml, because run_log doesn't record a job's alert level: the critical and warning rules each list the job names and max_attempts at that level, and an empty catalog matches nothing. Jobs with alert: none never alert. After editing the catalog, run python3 scripts/generate-launcher-alerts.py and commit the result; CI fails on a stale rule.

Each rule returns one series per job, so the page names the job, and carries a severity label (critical or warning) to route on. A failure or timeout alerts only on a job's last attempt (attempt >= max_attempts), or when a chain step's root has already ended, since the watcher can no longer retry it. Skips with end_reason paused, chain_step, max_duration, chain_ended or fan_out_ended are excluded: a chain timeout alerts once, on the chain rule, and a cancelled fan-out's children were stopped on purpose. A fan-out parent's failure (fan_out_failed) is excluded too, since the failed child's own row alerts. Event-triggered runs (trigger_type = 'event') also drop overlap and expired skips: an event is a speed-up over the job's schedule, so one that finds the job busy or can't start in time is routine, and the next event or scheduled slot covers it. Schedule and chain skips keep alerting. To inspect a run, query run_log by job_name and finished_at, then see summary.log_tail.

kubectl get jobs -n default -l fantasy.job=<job> --sort-by=.metadata.creationTimestamp
kubectl logs -n default -l fantasy.job=<job> --tail=200

A skipped run with end_reason overlap means the previous run, or its retry, was still running at the next slot. expired means the run couldn't start inside its schedule's catch_up window, a chain step couldn't start within 30 minutes of its latest time, or an event run couldn't start within its debounce window (event runs' expired doesn't alert). dependency_failed means a chain step's on: success parent didn't succeed; fix that parent first, since the skip is only its consequence.

What each job depends on​

JobCheck first
sync-leagueFirst tuesday-weekly step. ESPN cookies (espn-session-secrets espn_s2/swid), league configuration, Postgres. If it fails, metrics-extractor and sync-actual-scores are skipped (dependency_failed) and the rest of the chain still runs.
metrics-extractorRuns after sync-league succeeds. External play-by-play source and Postgres. It refreshes historical metrics; it doesn't replace the projection/VORP refresh.
sync-actual-scoresRuns after metrics-extractor succeeds, and on its own Friday at 10:00 to reconcile. ESPN cookies (espn-session-secrets espn_s2/swid) and ESPN_LEAGUE_ID, the public NFL scoreboard, Postgres, and Redis for read-cache invalidation. See score sync failures.
espn-ownership-syncRuns daily at 17:45, and in the chain once sync-league finishes. ESPN cookies (espn-session-secrets espn_s2/swid) and Postgres. It writes the espn_player_ownership snapshot for the upcoming week; a failure doesn't block the refresh, the lineup or waivers, but leaves a day missing from the history.
projections-refreshRuns daily at 18:00 and in the chain once sync-actual-scores and espn-ownership-sync finish. Sleeper skill-position projections, ESPN K/D-ST projections (needs espn-session-secrets espn_s2/swid), then VORP for each horizon week. Check season/week/team configuration and Postgres credentials. It supplies target-week VORP for lineup and waiver decisions and later-week VORP for trade scoring. Its stages are child runs in run_log. See the refresh horizon.
lineup-optimizerCritical. Runs once the refresh finishes, or at 19:15 at the latest. Roster/weekly-stat rows, target-week VORP and prior-week fallback, Postgres, core API, trigger API key and downstream lineup processing. Zero swaps is success: its dashboard log panel distinguishes Lineup already optimal from Published N lineup swap(s). Missing input fails the run after core-pipeline#178.
waiver-triggerCritical. Runs once lineup finishes, or at 20:30 at the latest. Exact-week bench/free-agent pools from the refresh, Postgres, core API and trigger API key, downstream RabbitMQ/browser-worker claims. It has no prior-week fallback; empty pools fail the run after core-pipeline#181.
rumor-action-triggerRuns daily at 21:45, in the chain once waivers finish, and within minutes of each digest on rumor.signal_created (one run per 30-minute window). The slot, the chain step and a later event wait for an event run still in flight, which is safe: the second pass finds nothing new past the watermark. A manual run started while any run is in flight returns 409. Rumor sync and today's 11:00 digest, rumor-signals rows, team/season config, Postgres, Redis, core API and downstream action processing. A successful run isn't proof of complete application processing; see application outcome.
espn-session-healthcheckESPN reachability and espn-session-secrets (espn_s2/swid), league configuration, Postgres execution logging. Cookies require human refresh when expired.
rapid-status-syncerPlayer-status source, Postgres, Redis and RabbitMQ credentials/connectivity. Thu/Sun/Mon are the only scheduled days.
rumor-digest-botRumor inputs in Postgres, database/execution-log credentials and the Groq/Gemini LLM fallback credentials. No ESPN cookies or browser required. It runs at 11:00 and emits rumor.signal_created for each stored signal through the amqp bundle, which starts rumor-action-trigger. Signals that need fresh VORP wait for the 21:45 run.
rumor-syncerExternal rumor sources, ingestion logs, Postgres and Redis. Ingestion feeds the digest/action path.
draft-listener, draft-connectorCritical. Live draft services the draft API starts (POST /v1/draft/start) and cancels (/v1/draft/stop); a cancel never alerts. A failure or deadline timeout means the live listener or room connector died mid-draft: read GET /v1/draft/logs, then restart with /v1/draft/start. The listener needs Postgres, Redis and the core API; the connector needs espn-session-secrets and Redis.
practice-draft-listener, practice-draft-connectorNo alert. Practice-room services. The watcher retires both as timeout/unhealthy once the connector's draft:{league_id}:health heartbeat (1-hour TTL) has lapsed and the run is over 10 minutes old, which replaced the stale-draft sweeper.
trade-candidatesDaily at 07:00, Wednesday/Friday at 12:30. Core API startup migrations must create the candidate tables first. Needs league/team/season configuration, week resolution, trade context and refreshed projection/VORP inputs, Postgres and ESPN session secrets. An unresolved week or unavailable context fails the run. Check the stored run's freshness before the scan; see rollout and verification. One attempt.
trade-scanWednesday/Friday at 13:00. ESPN stored session/cookies, Groq/Gemini LLM credentials, league/team configuration, Postgres claims and core trade routes. One attempt: inspect trade_scan logs and execution_log claims before any recovery; see outgoing trade scan.
trade-submitNo schedule: core's trade approve route starts one run per approved offer, with its offer_key, parented to the scan that staged it. It runs trade_builder.js --offer-key behind the espn-session group and sends the offer to ESPN. Needs ESPN stored session/cookies, league/team configuration, Postgres execution_log claims and core GET /v1/trade/offers. One attempt: an ESPN outcome the builder couldn't confirm is never retried, so check the team's pending trades on ESPN and the offer's execution_log row before sending it again. An expired skip means the run waited behind the group past its queue window and the offer was never sent.
trade-inboxEvery two hours, 08:00 through 22:00. ESPN stored session/cookies, league/team configuration, Postgres inbox/execution logging, and core POST /v1/trade/score. No LLM keys. A successful run can still contain unavailable or stale scores; inspect refresh logs and score outcomes. One attempt.

Score sync failures​

Each sync-actual-scores run (the tuesday-weekly chain step and the Friday 10:00 reconcile) reprocesses every completed week. A week ESPN hasn't finished fails the run with a reason such as ESPN professional statistics are not official or NFL games are missing or not final. Weeks before it stay committed.

  • Wait for the next scheduled run. Friday catches a Tuesday miss. On Tuesday the failed step doesn't hold up the chain: the projection refresh, lineup, waivers and rumor actions still run.
  • Don't force a week with --week; it applies the same completeness checks.
  • A success that logs no Persisted ... week=N for the just-finished week means ESPN hadn't rolled its scoring period yet.
Alert ownership

core-pipeline#140 owns application outcomes for the shared rumor-action service (CLI and HTTP). Its rumor-action-trigger-failure rule now lives in fantasy-infra, and homelab-deployment retains rumor-digest-bot-failure.

Launcher alerts complement these signals and don't validate partially processed actions.

Launcher chain past max duration​

The scheduler ended a chain run because it outlasted the chain's max_duration, and skipped its unstarted steps. Look at the chain's run_log rows (chain_root_id = the root's id) for the step that was slow or never started.

Read-only live checks​

# Scheduler state, recent launches and any pod still running.
kubectl logs deploy/fantasy-job-launcher -n default --tail=100
kubectl get jobs -n default -l fantasy.job --sort-by=.metadata.creationTimestamp
kubectl logs -n default -l fantasy.job=<job> --tail=200

GET /v1/jobs lists the catalog with each job's latest run and any pause. GET /v1/launcher/health reports the heartbeat and queue depths. GET /v1/runs/<run id> shows one run. Launched k8s Jobs are removed after one day, so use run_log and Loki ({namespace="default", pod=~"<job>-.*"}) for anything older.

After a merge​

  1. Let Argo CD apply the PR.
  2. Check the launcher log for Launcher started with <n> jobs. A changed catalog rolls the launcher through its hashed ConfigMap.
  3. Verify read-only that Grafana lists the launcher rules, and none of the UIDs in delete-orphaned-alert-rules.yaml.
  4. Compare run_log with the schedule inventory over the next scheduled slots.

Testing and adding coverage​

Removing a rule's file does not remove the rule: Grafana keeps a provisioned rule until a deleteRules entry names its UID. List the UID in base/telemetry/delete-orphaned-alert-rules.yaml in the same PR that drops the rule, or it keeps firing on no data.

./tests/launcher_alerts/run.sh loads the committed SQL into a disposable postgres:16 and asserts fire and no-fire cases for every rule, including the generated ones rendered from a fixture catalog. python3 -m unittest tests.test_generate_launcher_alerts checks the committed rules match the catalog.

To add a job:

  1. Add it to base/launcher/catalog.yaml.
  2. Run python3 scripts/generate-launcher-alerts.py and python3 scripts/generate-schedules.py, and commit the result. CI fails on a stale rule or inventory.
  3. Add its row to What each job depends on.