ADR 0001: Unified job launcher
Status: Accepted (2026-10-26). Supersedes the draft on branch infra-76-rabbitmq-launcher-adr.
Every job gets one entry point. Schedules, named chains, events, the web UI, the CLI and MCP tools all send the same run command to the job_runs queue. One launcher in fantasy-core-pipeline turns each command into a k8s Job (or an inline run for tiny jobs) and records it in run_log. All 16 CronJobs go away. The KEDA work-item worker stays as it is.
Why change?
- No on-demand runs. A run starts only from a CronJob,
kubectl create job --from=cronjob/...or the draft endpoints. Lineup, trade and waiver scans can't be run from the UI. - Three ways to create Jobs. CronJobs, manual kubectl and
src/api/draft.pyeach create Jobs, and alerting only sees the CronJob-owned ones. - Clock offsets instead of dependencies. The Tuesday sequence (16:00 → 21:45) is spaced by time (schedule order); nothing stops lineup running on stale projections or waits for a slow step.
- Per-CronJob alerting. Every new CronJob needs alert-regex, dashboard and fixture edits (CronJob alerts).
What does it look like?
The scheduler, consumer, watcher and inline runner are one process: python -m src.launcher in the core image, one replica, Recreate strategy. (Amended by core-pipeline#354: two replicas with RollingUpdate. They elect a leader over the fantasy-job-launcher Lease, and only the leader acts.)
What do the words mean?
| Term | Meaning |
|---|---|
| Job | A catalog definition, such as lineup-optimizer |
| Run | One execution of a job; one run_log row |
| k8s Job | The Kubernetes object a pod-executor run creates |
| Chain | A named workflow of job steps with dependencies |
| Event | A fact (rumor.signal_created) that can start runs |
Where are jobs defined?
In base/launcher/catalog.yaml, published as the ConfigMap fantasy-job-catalog and validated by core's pydantic schema (python -m src.launcher.catalog --check). Infra CI runs that schema and --check from the core image the production launcher is pinned to, so a catalog can't merge ahead of a launcher that accepts it.
version: 1
global: {max_concurrent_pods: 2, max_deadline: 2h, start_timeout: 10m, manual_queue_ttl: 30m}
profiles:
core: {image: CORE_IMAGE, secrets: [db, espn, redis, trigger-api]}
worker: {image: WORKER_IMAGE, serviceAccount: fantasy-browser-worker, volumes: [espn-session-state]}
defaults: {executor: pod, overlap: forbid, max_attempts: 1, deadline: 1h, alert: warning}
jobs:
lineup-optimizer:
profile: core
command: [python, -m, src.optimization.lineup_solver]
params: {week: int?, dry_run: bool?} # allowlist for on-demand runs
max_attempts: 3
alert: critical
acts: true # UI confirms before running
rumor-action-trigger:
profile: core
command: [python, -m, src.services.rumor_action_service]
schedules: [{cron: "45 21 * * *", tz: America/Chicago, catch_up: 1h}]
on_event: [{key: rumor.signal_created, debounce: 30m}]
chains:
tuesday-weekly:
schedule: {cron: "0 16 * * 2", tz: America/Chicago, catch_up: 2h, seasons: [regular, playoffs]}
max_duration: 8h
steps:
sync-league: {}
metrics-extractor: {after: [{step: sync-league, on: success}]}
lineup-optimizer: {after: [{step: projections-refresh, on: finish}], latest: "19:15"}
| Field | Rule |
|---|---|
overlap | forbid refuses a second concurrent run (409 or skipped); queue waits. Under either mode, a schedule, chain or event run waits when only event runs hold the job (see What happens when an event run holds a job?) |
max_attempts | Launcher-owned retries; k8s backoffLimit is always 0. trade-scan, trade-inbox, trade-submit, trade-candidates and draft-prep stay at 1 |
catch_up | A slot missed while the launcher was down still fires if inside this window |
seasons | Slots only fire in the listed season windows |
executor | pod (default) or inline for tiny, frequent jobs once measured |
after.on | success stops the chain on failure; finish runs either way. Data steps use success, acting steps (lineup, waivers) use finish |
latest | A chain step starts by this time even if its parent is still running |
group | Names a top-level groups.<name> with a limit: at most that many runs of the group's jobs run at once, across jobs. A run behind a full group waits (never skipped or 409) until its expiry. trade-scan, trade-inbox and trade-submit share espn-session (limit 1) |
Reserved for v2 (parsed, rejected until implemented): class: service, fan_out, approval.
What travels on RabbitMQ?
| Exchange / queue | Carries | Consumer |
|---|---|---|
job_runs (+ .dlq, 6h TTL) | Commands: "run this job" | Launcher only |
job_events (topic) | Facts: "this happened" | Launcher, and any future subscriber |
lineup_updates, waiver_actions, trade_*… | Work items | KEDA worker (unchanged) |
Run command: {run_id, job, params, trigger, requested_by, dedupe_key, parent_run_id, chain_root_id, event_id, downstream, scheduled_for, attempt, traceparent}.
| Trigger | dedupe_key |
|---|---|
schedule | job@slot |
chain | job@chain_root |
event | job@key@debounce_window |
manual, cli, mcp | fresh UUID |
The API and scheduler write the queued run_log row and an outbox row in one transaction and then publish, reusing core's _accept_durable and activity_runner.py, so the queue and the table can't drift apart.
sync_players → waiver_actions → the waiver sniper is the fast pickup path. It is a work item, not an event, and does not go through the launcher.
How does a run end?
What stops a run from hanging?
Every stage has a hard limit, and every limit ends in a run_log status. Overlap and cap locks are just rows in queued or running, so any final status frees them.
| Stage | Limit | Enforced by | Ends as |
|---|---|---|---|
Waiting in job_runs | 6h message TTL | RabbitMQ policy | DLQ alert |
| Queued (cap full) | manual 30m; schedule = catch_up; chain = latest + 30m | consumer | skipped (expired) |
| Pod Pending | 10m | watcher | failure (start_timeout), k8s Job deleted |
| Running | per-job deadline, capped at 2h | activeDeadlineSeconds + watcher | timeout |
| Lost track | deadline + 10m, checked every minute | watcher | timeout (orphaned) |
| Inline run | same deadline; SIGTERM, SIGKILL after 30s | inline runner | timeout |
| Whole chain | max_duration | scheduler | remaining steps skipped |
| Finished k8s Jobs | ttlSecondsAfterFinished 1 day | Kubernetes | removed |
How does the KEDA worker share the pod cap?
max_concurrent_pods counts the launcher's running pod rows plus the unfinished k8s Jobs matching global.counted_k8s_jobs. That is scaledjob.keda.sh/name=fantasy-browser-worker-scaler, the label KEDA sets on every Job it creates. Worker rows in run_log have no executor, so before this the cap couldn't see up to 3 worker pods.
- A launcher run that finds the cap full waits, as it does for its own pods. Workers exit 120s after their queues drain, so a waiver burst delays launcher runs by minutes, well inside every expiry window.
- On draft day the warm worker holds one slot all day, so the launcher runs one pod at a time.
- The count is one-way. KEDA can't see launcher pods, so the worst case stays 2 launcher + 3 worker pods, bounded by
maxReplicaCount: 3. - If the k8s API errors, the lookup counts 0 instead of holding every run.
Full replacement was evaluated and rejected for now.
- The launcher would have to re-implement queue-depth scaling across 8 work queues, and the cluster doesn't scrape RabbitMQ.
- Idle exit already lives in the worker, and the draft-day warm worker is just
minReplicaCount: 1. - KEDA stays for ollama regardless, so removing the ScaledJob saves no overhead.
- The launcher is one replica, so the draft-pick and waiver-sniper paths would gain a single point of failure.
Revisit after launcher HA.
What happens when an event run holds a job?
A schedule, chain or event run that finds only event runs holding its job waits for them, under forbid too. Every other overlap follows the job's overlap mode. Decided in core#387.
- An event is a speed-up over the job's schedule. A slot or chain step that skipped behind one would alert, and could end its chain root
failure, over work done moments early. - An event run behind another waits too, so a signal from the next debounce window isn't left for the next slot.
- A manual, CLI or MCP run keeps
forbid's 409. A schedule behind a schedule still skips. - The run's own
expires_atbounds the wait:catch_up, a chain step'slatest+ 30m, or the event's debounce window. So an event job'sdeadlineplus the watcher's 10m orphan grace should stay under its shortest wait, or a dead event pod can make the slot expire.
This replaced a per-job overlap: queue opt-in. That opt-in also turned manual 409s into queued waits and made schedule-on-schedule overlaps wait.
How do pods report back?
The launcher creates the run_log row and passes RUN_ID, RUN_TRIGGER and RUN_PARENT_ID. The outermost run_scope in the pod adopts RUN_ID and clears it from the environment, so subprocesses and nested scopes become child rows. trigger_from_env() prefers RUN_TRIGGER over the CronJob-name heuristic. The worker's src/db/run_log.js follows the same rules.
New run_log fields: statuses queued|skipped|cancelled|timeout; dedupe_key (unique), scheduled_for, attempt, requested_by, chain_root_id, event_id, end_reason. New tables: job_overrides (pause/snooze) and launcher_heartbeat (including job_runs depths, since the cluster doesn't scrape RabbitMQ).
How is it monitored?
Grafana SQL alerts on run_log replace every kube_cronjob_* rule and route to the existing ntfy contact point:
- launcher heartbeat older than 5 minutes;
failure,timeoutorskipped(excluding paused) on jobs withalert: warning|critical;job_runs.dlqdepth above 0;- a chain run past its
max_duration.
How do we get there?
Never open an infra PR that needs new core or worker code until that image is live. Each cutover PR removes its CronJobs and adds the matching catalog entries, alert edits and dashboard edits together.
| Step | Moves |
|---|---|
| Foundation | core run_log, catalog, launcher, scheduler, events, API; infra Deployment, queues, empty catalog, alerts |
| Batch A | rumor-syncer, rapid-status-syncer, stale-draft sweeper, ESPN session check, rumor digest |
| Batch B | the tuesday-weekly chain, cut over midweek |
| Batch C | trade-candidates ×2, scores reconcile, trade-scan, trade-inbox; delete remaining CronJob alerts |
| After | web UI Jobs page, rumor and trade-offer events, CLI/MCP, right-sizing |
v2 (designed, not built): draft Jobs under the launcher (before the August draft), approval steps in chains, fan-out steps, KEDA worker under the launcher (decided: shared pod cap only), concurrency groups, per-run resource accounting, launcher HA, Helm packaging.
What else was considered?
| Option | Why not |
|---|---|
| Keep CronJobs, add a run-now endpoint | Leaves three ways to create Jobs and per-CronJob alerting |
| Airflow, Prefect | Scheduler, UI and metadata DB cost 0.5–3 Gi on a single node already at 81% memory; Node bots fit poorly |
| Argo Workflows | Closest fit (k8s-native DAGs, UI), but needs Argo Events and an EventBus to start from RabbitMQ. Revisit at v2 fan-out; after: mirrors Argo's dependencies: |
| Postgres as the run queue | Fewer parts, but RabbitMQ keeps commands and events on the same shared bus other services can subscribe to |
| Replace KEDA | KEDA stays for ollama regardless; replacing the ScaledJob saves no overhead and risks the draft-pick path. Re-evaluated for v2: the worker shares the pod cap instead (above) |
Consequences
- The launcher is a single point of failure for scheduled work. Catch-up windows cover restarts, and the heartbeat alert covers outages. (Amended by core-pipeline#354: a standby replica takes the Lease in about 17s after the leader dies, inside the 2-minute slot grace, and fails the dead leader's inline runs so they can't block the next slot.)
- It adds one small always-on pod (50m / 192Mi, sized to hold an inline run). The two frequent syncers run inline, which removes about 110 pod starts a day.
scripts/generate-schedules.pyreads the catalog and generates the schedule table and chain diagrams.