Queue publishing
Core publishes to seven RabbitMQ queues. Six are consumed by fantasy-browser-worker: the API sends all six, and sync_players also publishes straight to waiver_actions. The seventh, job_runs, carries run commands to core's own job launcher. Every message is durable JSON on the default exchange, with the queue name as the routing key. Core also publishes facts to the job_events topic exchange, described in How do events start runs?.
A 202 or queued status means the message left core, not that ESPN changed. The worker confirms results into execution_log, and the activity runner verifies them. See Activity submissions.
How does a message get from a request to the worker?
Trigger queues go through the outbox. The API saves the action and the message in one transaction, publishes immediately, and leaves retries to the activity runner. A saved message is never lost if the broker is down.
The trade queues skip the outbox. They claim the offer in execution_log first, then publish, and reconcile by hand if the result is uncertain.
What queues are there?
| Queue | Path | Source | Idempotency key |
|---|---|---|---|
lineup_updates | Outbox | POST /v1/trigger/lineup | lineup-<team>-<hash> |
waiver_actions | Outbox | POST /v1/trigger/waiver | waiver-<team>-<hash> |
draft_targets | Outbox | POST /v1/trigger/draft-pick | <league>:<pick_number>:<espn_id> |
trade_submissions | Direct from API | POST /v1/trade/offers/{offer_key}/approve | offer_key |
trade_responses | Direct from API | POST /v1/trade/incoming/{id}/decision and .../decision/retry | decision_key |
trade_negotiations | Direct from API | POST /v1/trade/incoming/{id}/counter | Publication key in outcome_details |
waiver_actions (alerts) | Ingestor | sync_players | None |
job_runs | Run outbox | launcher.publish.publish_run (Jobs API, scheduler) | run_log.dedupe_key |
The outboxes publish with a passive queue declaration, so lineup_updates, waiver_actions, draft_targets and job_runs must already exist on the broker. The trade queues are declared durable on publish.
POST /v1/trigger/rumor-action has no queue of its own. It evaluates rumor signals and sends the resulting moves through the lineup and waiver triggers.
What do the messages contain?
Every API message carries a nullable traceparent field, and the same value goes in the AMQP headers, so a worker span can join the API's trace. It is separate from tracking_id, which is the human-facing UUID. A caller running inside a recorded run (run_log_service.run_scope) also sends an X-Run-Id header through trigger_client, and the API links the new action to that run for the audit log. Lineup, waiver claim and draft player IDs are ESPN IDs; those trigger routes reject a Sleeper ID with a 422. Counter staging and injury alerts use Sleeper IDs.
See the queue contract explorer for schemas, required fields and placeholder examples, or download the AsyncAPI 3.0 spec.
waiver_actions carries both API claims (ESPN player IDs) and ingestor injury alerts (Sleeper player IDs). Injury alerts have event_type: "INJURY_STATUS_CHANGE" and no team_id, tracking_id or trace context. A consumer must not treat them as claims.
Lineup and waiver triggers use ESPN player IDs. trade_negotiations uses internal Sleeper IDs because the worker stages the counter from core's player mapping. Core strips decision_inputs, swap kind, and request-only after_tracking_id before publishing.
Draft targets expire while waiting in the outbox. The route returns 409 for a filled or expired pick and 503 on delivery failure; retry with the same key.
Direct publisher: injury alerts
Player sync publishes injury alerts directly when a rostered player's status changes to Questionable, Doubtful or Out. It checks that the queue exists before publishing.
How do I update the contract?
Edit docs/asyncapi.yaml alongside publisher changes. The contract tests validate real publisher output, reject undocumented fields, and compare channels with the shared queue constants.
DB_PASSWORD=test pytest tests/test_asyncapi_docs.py
npx @asyncapi/cli validate docs/asyncapi.yaml
How does idempotency work?
A repeated request returns status: "duplicate" with the original tracking_id instead of publishing again. The guard is a unique index on activity_actions(league_id, kind, idempotency_key), where kind is the queue name.
| Rule | Detail |
|---|---|
| Season and week suffix | If decision_inputs has season and week, they are appended to the key, so a week-old key can't swallow next week's action. |
| Reuse window | After a key's action has finished (not queued, delivered or running) and the TTL has passed, the key is released and can be used again. |
| Pending work | An action still queued, delivered or running keeps its key, however old. |
| No key | The request always publishes. The web UI's manual moves omit it. |
| Queue | Key release TTL | How the caller builds the key |
|---|---|---|
lineup_updates | 30 min | SHA-256 of the Job name, team, season and week, and the sorted swaps. The Job name comes from the CronJob's downward API, so each new Job publishes but pod retries within one Job don't. |
waiver_actions | 7 days | SHA-256 of team, season, week, add and drop, plus the IR player if chained. No Job name: a claim stays queued on ESPN until waivers process, so any re-run that week must dedupe. If season or week is missing, no key is sent. |
draft_targets | 15 min | <league_id>:<pick_number>:<espn_id>. |
Callers retry connection errors, timeouts and 5xx responses, six attempts with a 5 s delay that doubles to 30 s. They never retry a 4xx.
How does a claim wait for an IR move?
An add-only waiver claim can be chained behind an IR move that makes room for it.
| Rule | Detail |
|---|---|
| Caller order | Post the IR move to /v1/trigger/lineup first. Then post the claim with after_tracking_id set to the returned tracking_id. |
| Requirements | The claim must omit drop_player_id. The parent must exist, belong to the same team, contain an ir entry and not be failed. |
| Hold length | ACTIVITY_DEPENDENCY_HOLD_SECONDS, default 3600. |
| Failure | A cancelled claim is marked skipped and its key is released, so a fresh IR move can queue a new claim. |
| Transient parent failure | The claim stays held. |
| Unreadable parent | The route returns 503 and queues nothing. |
How do run commands reach the launcher?
src/launcher/publish.publish_run validates the job and its params against the catalog, then writes the queued run_log row and a job_run_outbox row in one transaction and publishes a RunCommand to job_runs. A failed publish is retried by activity_runner.drain_job_runs, which both the API's background tick and the launcher loop call.
| Trigger | dedupe_key |
|---|---|
schedule | job@slot |
chain | job@chain_root |
event | job@key@debounce_window |
manual, cli, mcp | A fresh UUID (the run id) |
| A fan-out child (any trigger) | job@parent_run#param=value |
A second request with a used key returns the existing run as duplicate, because run_log.dedupe_key is unique. The launcher acks each command once it has checked the run is still queued at the command's current attempt and added it to its pending list, not when the run starts, so a run waiting for the pod cap never trips RabbitMQ's 30-minute consumer_timeout. After a restart it rebuilds that list from queued run_log rows. The initial attempt past expires_at ends as skipped (expired); retries of an attempt that already started keep their retry budget. A run refused by overlap: forbid ends as skipped (overlap), except that a schedule, chain or event run waits when every run holding the job is an event run. An event is a speed-up over the job's schedule, so a slot or chain step that skipped behind one would alert, and could fail its chain, for work done moments early. A manual, CLI or MCP run still gets the 409, and a waiting run still expires at its expires_at.
A job with catalog fan_out: {param, values} fans out each run that leaves param unset. The launcher claims that run with executor fan_out and starts nothing for it. The scheduler then publishes one child command per value, with the parent's params plus param=value, the parent's trigger and chain root, and parent_run_id = the parent. Children are ordinary commands: they wait for the pod cap like any run, and neither their parent nor their siblings count as an overlap. A child whose parent has ended (cancelled) is skipped (fan_out_ended). The parent ends success once every child succeeds, or failure (fan_out_failed) once none is pending and any did not.
Jobs that share a resource (trade-scan and trade-inbox both drive the stored ESPN browser session) name a catalog concurrency group: top-level groups: {<name>: {limit: N}} and group: <name> on each job. A run whose group already has limit launcher runs of its jobs running waits, under forbid too, since another job holds the resource; its expires_at still bounds the wait. A fan-out parent takes no slot, its children each do. KEDA browser worker pods aren't counted yet: they join groups once the worker runs under the launcher.
The pod gets RUN_ID, RUN_TRIGGER, RUN_PARENT_ID and JOB_NAME=<job>@<dedupe_key>. Its outermost run_scope adopts the row and keeps the catalog job name.
How do events start runs?
An event is a fact, such as rumor.signal_created, that can start runs. A job subscribes through catalog on_event, with a debounce.
| Field | Meaning |
|---|---|
event_id | A fresh UUID. The run it starts stores it in run_log.event_id |
key | The fact, as <domain>.<fact> in lowercase. It is also the routing key, and never a wildcard |
subject | What the fact is about, such as player:4046 |
payload | A JSON object for logs and other subscribers. Runs don't receive it, so a job with required params can't subscribe |
emitted_by_run_id | The emitting run_log row, or null outside a run |
at | When it happened, in ISO 8601 with a timezone |
Publishing from core. Call src.launcher.events.emit_event(key, subject, payload) after your own work has committed, so the run it starts can read that work. It returns the event_id, or None if the broker didn't take the event. A broker failure is logged and never raised, because a retry of the job wouldn't emit again and every subscriber has a schedule to fall back on. A bad key or an unserializable payload raises ValueError.
Publishing from the worker. Publish the same six fields as JSON to the job_events exchange, with key as the routing key, persistent: true and contentType: application/json, on a confirm channel. Check the exchange exists without changing it (checkExchange), and send the W3C traceparent as a message header, not in the body. Take emitted_by_run_id from the run the bot adopted: RUN_ID is only in the environment until it is adopted.
rumor.signal_created. The worker's rumor digest emits one per rumor_signals row, after that run's rows commit, with subject player:<player_id> and payload {"signal_id": <rumor_signals.id>}. sync_rumors doesn't emit it: raw_signals rows mean nothing to rumor_action_service until the digest has interpreted them.
The launcher owns job_events.launcher, a durable queue it declares at startup with no arguments, bound with #. Its depth is in the heartbeat and GET /v1/launcher/health. An event published before that queue first exists is lost, because nothing is bound to the exchange. Another subscriber binds its own queue.
How does debounce work?
The first event in a window starts the run right away, and later events in that window start nothing. Windows are fixed, aligned to the Unix epoch, and picked from the event's own at, so a redelivered event is a no-op. An event is judged by its own at: once it is older than the debounce when the launcher handles it, it is skipped as expired, and a run it did start expires debounce after at.
| Case | Outcome |
|---|---|
| Key no job subscribes to | Acked and ignored |
| Several jobs subscribe to the key | Each gets its own run, on its own debounce |
| Job is paused | One skipped (paused) run per window, carrying the event_id |
| Another event run of the job is running | Waits for it, even under overlap: forbid |
A non-event run of the job is running under overlap: forbid | skipped (overlap), as for any other trigger |
Run still waiting after debounce | skipped (expired), because a later event has its own window |
| Malformed message | Logged and dropped |
| Launcher was down, events waiting in the queue | Processed on restart. An event older than its debounce is recorded as skipped (expired) and starts nothing, one row per old window. Newer ones start a run as usual |
Event can't be recorded, for example run_log unavailable | Returned to the queue and retried for 15 minutes, holding up the events behind it, then dropped with an ERROR log |
An event that lands after its window's run has read its data starts nothing. It waits for the next event in a later window or for the job's schedule, so a subscribed job should keep one. An emitter that sends a batch should commit the whole batch before the first emit_event.
Related
- API reference: which endpoints publish and which need an API key.
- Activity submissions: reading an action's delivery and verification state.
- Incoming trade inbox API: the trade message contracts and recovery.
- ESPN reverse engineering notes: how the worker acts on these messages.