Skip to main content

ADR 0001: Unified job launcher

Status: Accepted (2026-10-26). Supersedes the draft on branch infra-76-rabbitmq-launcher-adr.

Decision

Every job gets one entry point. Schedules, named chains, events, the web UI, the CLI and MCP tools all send the same run command to the job_runs queue. One launcher in fantasy-core-pipeline turns each command into a k8s Job (or an inline run for tiny jobs) and records it in run_log. All 16 CronJobs go away. The KEDA work-item worker stays as it is.

Why change?​

  • No on-demand runs. A run starts only from a CronJob, kubectl create job --from=cronjob/... or the draft endpoints. Lineup, trade and waiver scans can't be run from the UI.
  • Three ways to create Jobs. CronJobs, manual kubectl and src/api/draft.py each create Jobs, and alerting only sees the CronJob-owned ones.
  • Clock offsets instead of dependencies. The Tuesday sequence (16:00 → 21:45) is spaced by time (schedule order); nothing stops lineup running on stale projections or waits for a slow step.
  • Per-CronJob alerting. Every new CronJob needs alert-regex, dashboard and fixture edits (CronJob alerts).

What does it look like?​

The scheduler, consumer, watcher and inline runner are one process: python -m src.launcher in the core image, one replica, Recreate strategy. (Amended by core-pipeline#354: two replicas with RollingUpdate. They elect a leader over the fantasy-job-launcher Lease, and only the leader acts.)

What do the words mean?​

TermMeaning
JobA catalog definition, such as lineup-optimizer
RunOne execution of a job; one run_log row
k8s JobThe Kubernetes object a pod-executor run creates
ChainA named workflow of job steps with dependencies
EventA fact (rumor.signal_created) that can start runs

Where are jobs defined?​

In base/launcher/catalog.yaml, published as the ConfigMap fantasy-job-catalog and validated by core's pydantic schema (python -m src.launcher.catalog --check). Infra CI runs that schema and --check from the core image the production launcher is pinned to, so a catalog can't merge ahead of a launcher that accepts it.

version: 1
global: {max_concurrent_pods: 2, max_deadline: 2h, start_timeout: 10m, manual_queue_ttl: 30m}
profiles:
core: {image: CORE_IMAGE, secrets: [db, espn, redis, trigger-api]}
worker: {image: WORKER_IMAGE, serviceAccount: fantasy-browser-worker, volumes: [espn-session-state]}
defaults: {executor: pod, overlap: forbid, max_attempts: 1, deadline: 1h, alert: warning}
jobs:
lineup-optimizer:
profile: core
command: [python, -m, src.optimization.lineup_solver]
params: {week: int?, dry_run: bool?} # allowlist for on-demand runs
max_attempts: 3
alert: critical
acts: true # UI confirms before running
rumor-action-trigger:
profile: core
command: [python, -m, src.services.rumor_action_service]
schedules: [{cron: "45 21 * * *", tz: America/Chicago, catch_up: 1h}]
on_event: [{key: rumor.signal_created, debounce: 30m}]
chains:
tuesday-weekly:
schedule: {cron: "0 16 * * 2", tz: America/Chicago, catch_up: 2h, seasons: [regular, playoffs]}
max_duration: 8h
steps:
sync-league: {}
metrics-extractor: {after: [{step: sync-league, on: success}]}
lineup-optimizer: {after: [{step: projections-refresh, on: finish}], latest: "19:15"}
FieldRule
overlapforbid refuses a second concurrent run (409 or skipped); queue waits. Under either mode, a schedule, chain or event run waits when only event runs hold the job (see What happens when an event run holds a job?)
max_attemptsLauncher-owned retries; k8s backoffLimit is always 0. trade-scan, trade-inbox, trade-submit, trade-candidates and draft-prep stay at 1
catch_upA slot missed while the launcher was down still fires if inside this window
seasonsSlots only fire in the listed season windows
executorpod (default) or inline for tiny, frequent jobs once measured
after.onsuccess stops the chain on failure; finish runs either way. Data steps use success, acting steps (lineup, waivers) use finish
latestA chain step starts by this time even if its parent is still running
groupNames a top-level groups.<name> with a limit: at most that many runs of the group's jobs run at once, across jobs. A run behind a full group waits (never skipped or 409) until its expiry. trade-scan, trade-inbox and trade-submit share espn-session (limit 1)

Reserved for v2 (parsed, rejected until implemented): class: service, fan_out, approval.

What travels on RabbitMQ?​

Exchange / queueCarriesConsumer
job_runs (+ .dlq, 6h TTL)Commands: "run this job"Launcher only
job_events (topic)Facts: "this happened"Launcher, and any future subscriber
lineup_updates, waiver_actions, trade_*…Work itemsKEDA worker (unchanged)

Run command: {run_id, job, params, trigger, requested_by, dedupe_key, parent_run_id, chain_root_id, event_id, downstream, scheduled_for, attempt, traceparent}.

Triggerdedupe_key
schedulejob@slot
chainjob@chain_root
eventjob@key@debounce_window
manual, cli, mcpfresh UUID

The API and scheduler write the queued run_log row and an outbox row in one transaction and then publish, reusing core's _accept_durable and activity_runner.py, so the queue and the table can't drift apart.

Injury alerts stay direct

sync_players → waiver_actions → the waiver sniper is the fast pickup path. It is a work item, not an event, and does not go through the launcher.

How does a run end?​

What stops a run from hanging?​

Every stage has a hard limit, and every limit ends in a run_log status. Overlap and cap locks are just rows in queued or running, so any final status frees them.

StageLimitEnforced byEnds as
Waiting in job_runs6h message TTLRabbitMQ policyDLQ alert
Queued (cap full)manual 30m; schedule = catch_up; chain = latest + 30mconsumerskipped (expired)
Pod Pending10mwatcherfailure (start_timeout), k8s Job deleted
Runningper-job deadline, capped at 2hactiveDeadlineSeconds + watchertimeout
Lost trackdeadline + 10m, checked every minutewatchertimeout (orphaned)
Inline runsame deadline; SIGTERM, SIGKILL after 30sinline runnertimeout
Whole chainmax_durationschedulerremaining steps skipped
Finished k8s JobsttlSecondsAfterFinished 1 dayKubernetesremoved

How does the KEDA worker share the pod cap?​

max_concurrent_pods counts the launcher's running pod rows plus the unfinished k8s Jobs matching global.counted_k8s_jobs. That is scaledjob.keda.sh/name=fantasy-browser-worker-scaler, the label KEDA sets on every Job it creates. Worker rows in run_log have no executor, so before this the cap couldn't see up to 3 worker pods.

  • A launcher run that finds the cap full waits, as it does for its own pods. Workers exit 120s after their queues drain, so a waiver burst delays launcher runs by minutes, well inside every expiry window.
  • On draft day the warm worker holds one slot all day, so the launcher runs one pod at a time.
  • The count is one-way. KEDA can't see launcher pods, so the worst case stays 2 launcher + 3 worker pods, bounded by maxReplicaCount: 3.
  • If the k8s API errors, the lookup counts 0 instead of holding every run.

Full replacement was evaluated and rejected for now.

  • The launcher would have to re-implement queue-depth scaling across 8 work queues, and the cluster doesn't scrape RabbitMQ.
  • Idle exit already lives in the worker, and the draft-day warm worker is just minReplicaCount: 1.
  • KEDA stays for ollama regardless, so removing the ScaledJob saves no overhead.
  • The launcher is one replica, so the draft-pick and waiver-sniper paths would gain a single point of failure.

Revisit after launcher HA.

What happens when an event run holds a job?​

A schedule, chain or event run that finds only event runs holding its job waits for them, under forbid too. Every other overlap follows the job's overlap mode. Decided in core#387.

  • An event is a speed-up over the job's schedule. A slot or chain step that skipped behind one would alert, and could end its chain root failure, over work done moments early.
  • An event run behind another waits too, so a signal from the next debounce window isn't left for the next slot.
  • A manual, CLI or MCP run keeps forbid's 409. A schedule behind a schedule still skips.
  • The run's own expires_at bounds the wait: catch_up, a chain step's latest + 30m, or the event's debounce window. So an event job's deadline plus the watcher's 10m orphan grace should stay under its shortest wait, or a dead event pod can make the slot expire.

This replaced a per-job overlap: queue opt-in. That opt-in also turned manual 409s into queued waits and made schedule-on-schedule overlaps wait.

How do pods report back?​

The launcher creates the run_log row and passes RUN_ID, RUN_TRIGGER and RUN_PARENT_ID. The outermost run_scope in the pod adopts RUN_ID and clears it from the environment, so subprocesses and nested scopes become child rows. trigger_from_env() prefers RUN_TRIGGER over the CronJob-name heuristic. The worker's src/db/run_log.js follows the same rules.

New run_log fields: statuses queued|skipped|cancelled|timeout; dedupe_key (unique), scheduled_for, attempt, requested_by, chain_root_id, event_id, end_reason. New tables: job_overrides (pause/snooze) and launcher_heartbeat (including job_runs depths, since the cluster doesn't scrape RabbitMQ).

How is it monitored?​

Grafana SQL alerts on run_log replace every kube_cronjob_* rule and route to the existing ntfy contact point:

  • launcher heartbeat older than 5 minutes;
  • failure, timeout or skipped (excluding paused) on jobs with alert: warning|critical;
  • job_runs.dlq depth above 0;
  • a chain run past its max_duration.

How do we get there?​

Never open an infra PR that needs new core or worker code until that image is live. Each cutover PR removes its CronJobs and adds the matching catalog entries, alert edits and dashboard edits together.

StepMoves
Foundationcore run_log, catalog, launcher, scheduler, events, API; infra Deployment, queues, empty catalog, alerts
Batch Arumor-syncer, rapid-status-syncer, stale-draft sweeper, ESPN session check, rumor digest
Batch Bthe tuesday-weekly chain, cut over midweek
Batch Ctrade-candidates ×2, scores reconcile, trade-scan, trade-inbox; delete remaining CronJob alerts
Afterweb UI Jobs page, rumor and trade-offer events, CLI/MCP, right-sizing

v2 (designed, not built): draft Jobs under the launcher (before the August draft), approval steps in chains, fan-out steps, KEDA worker under the launcher (decided: shared pod cap only), concurrency groups, per-run resource accounting, launcher HA, Helm packaging.

What else was considered?​

OptionWhy not
Keep CronJobs, add a run-now endpointLeaves three ways to create Jobs and per-CronJob alerting
Airflow, PrefectScheduler, UI and metadata DB cost 0.5–3 Gi on a single node already at 81% memory; Node bots fit poorly
Argo WorkflowsClosest fit (k8s-native DAGs, UI), but needs Argo Events and an EventBus to start from RabbitMQ. Revisit at v2 fan-out; after: mirrors Argo's dependencies:
Postgres as the run queueFewer parts, but RabbitMQ keeps commands and events on the same shared bus other services can subscribe to
Replace KEDAKEDA stays for ollama regardless; replacing the ScaledJob saves no overhead and risks the draft-pick path. Re-evaluated for v2: the worker shares the pod cap instead (above)

Consequences​

  • The launcher is a single point of failure for scheduled work. Catch-up windows cover restarts, and the heartbeat alert covers outages. (Amended by core-pipeline#354: a standby replica takes the Lease in about 17s after the leader dies, inside the 2-minute slot grace, and fails the dead leader's inline runs so they can't block the next slot.)
  • It adds one small always-on pod (50m / 192Mi, sized to hold an inline run). The two frequent syncers run inline, which removes about 110 pod starts a day.
  • scripts/generate-schedules.py reads the catalog and generates the schedule table and chain diagrams.