Skip to main content

Architecture

The system decides roster moves in one service and carries them out against ESPN in another. Between them sit a RabbitMQ bus and Postgres, which records each action and its outcome. Use this page to find which component owns a step. Use End-to-end flows to follow one action through the system, and Job launcher for how jobs start.

The short version
  • Core decides and publishes. fantasy-core-pipeline is the only service that publishes ESPN actions to RabbitMQ.
  • The worker acts and verifies. fantasy-browser-worker consumes queues, writes to ESPN, then re-reads ESPN to confirm the change.
  • A person approves every trade. No trade offer or response reaches ESPN without a human approving the exact terms. See the Safety model.

What are the components?​

ComponentRepoRole
Web UIfantasy-web-uiReact console for lineups, waivers, trades, the inbox, activity and draft day. nginx serves it and proxies /v1/* to core.
oauth2-proxyfantasy-infraSigns you in before any UI page or API call. The API itself doesn't check user identity. See Authentication.
Core APIfantasy-core-pipelineFastAPI control plane. It reads data for the UI, scores trades, records actions and publishes them to RabbitMQ. See the API reference.
Ingestion and solversfantasy-core-pipelinePython modules the job launcher runs as Kubernetes Jobs. They sync players, projections and scores, compute VORP, and pick lineups and waiver claims. See Decision pipelines.
Job launcherfantasy-core-pipelineOne process, python -m src.launcher, that turns schedules, chains, events and on-demand requests into Kubernetes Jobs and records each in run_log. The catalog of jobs lives in fantasy-infra. See Job launcher.
Browser workerfantasy-browser-workerNode.js bots that act on ESPN through its REST API, a Playwright browser or the draft-room WebSocket. See Browser worker.
LLM chainfantasy-browser-workerGroq, then Gemini, then a self-hosted Ollama. The chain writes trade pitches and rumor digests, and drafts trade packages only when the scored candidate set is unavailable. LLM output never reaches ESPN without human approval.
Infrastructurefantasy-infraKustomize manifests for every service and store, the job catalog, dashboards and alerts.

Where is state kept?​

StoreHoldsMain writers
Postgres (TimescaleDB)Players, projections, weekly_stats VORP, league_rosters, draft_picks, rumor signalsCore ingestion
Postgres: activity tablesactivity_actions, activity_outbox, activity_events: each queued action, its pending message and its timelineCore API, worker
Postgres: run tablesrun_log (one row per job run), job_overrides (pauses), launcher_heartbeatLauncher, jobs
Postgres: audit tablesbot_decisions (why core published) and execution_log (what the worker did, trade inbox rows, trade claims)Core API, worker
RedisDraft-day pick stream, draft board, per-pick pause flags, draft-sync healthDraft sync, draft listener, core API
RabbitMQDurable work queues, one per action type, plus job_runs (run commands) and the job_events topicCore publishes, worker consumes. The launcher consumes job_runs and job_events.

Grafana reads Postgres through a read-only grafana_reader role, never the read-write bot_admin role. Table definitions are in the Data model.

Which queues connect core and the worker?​

QueuePublished byConsumed byFlow
lineup_updatesPOST /v1/trigger/lineupLineup botLineup
waiver_actionsPOST /v1/trigger/waiver; player sync injury alertsWaiver sniperWaivers
draft_targetsPOST /v1/trigger/draft-pickDraft clickerDraft day
trade_submissionsPOST /v1/trade/offers/{offer_key}/approveTrade builderOutgoing trades
trade_responsesPOST /v1/trade/incoming/{id}/decisionTrade responseIncoming trades
trade_negotiationsPOST /v1/trade/incoming/{id}/counterTrade negotiationIncoming trades
trade_evaluations, trade_proposals, llm_requestsNo scheduled producer, or the worker itselfTrade bot, async LLMNot part of a scheduled flow
job_runsJob API, launcher schedulerLauncherJob launcher

job_events is a topic exchange for facts such as rumor.signal_created; the launcher subscribes. Injury alerts from sync_players stay direct work items on waiver_actions and skip the launcher.

The broker pre-declares lineup_updates, waiver_actions and draft_targets with dead-letter queues. Field-level contracts are in Queue publishing and Message contracts.

Player ids come from two namespaces

players.player_id is Sleeper's id. ESPN ids are a separate, numerically overlapping namespace, crosswalked through players.espn_id. Every message bound for ESPN carries ESPN ids, and core rejects a Sleeper id with a 422. The trade evaluation and negotiation queues are the exception: they carry core's internal ids.

How does a queued message become a running bot?​

The worker is not a long-running service. KEDA watches each queue's length and starts one Kubernetes Job per waiting message, up to three at once. Each Job runs src/index.js, takes one message at a time and exits after 120 seconds idle. On draft day an overlay keeps one worker warm.

Details, including acknowledgement and retry rules, are in How bots run.

What starts a job?​

The launcher starts every job. The Tuesday jobs form the tuesday-weekly chain: ingest, score, refresh VORP, then optimize the lineup and waivers. Each step waits on its predecessor, or starts at its latest time.

TriggerExample
Schedulerumor-syncer every 30 minutes
Chain steplineup-optimizer after projections-refresh
Eventrumor.signal_created starts rumor-action-trigger
YouThe Jobs page, python -m src.cli jobs run, or an MCP tool

The generated schedule inventory lists every job, schedule and chain. Job launcher defines the terms.

How does code reach the cluster?​

StepDetail
BuildEach service repo's CI pushes an image to GitHub Container Registry.
Image UpdaterWatches GHCR and commits each new digest to the images: list in overlays/production/kustomization.yaml, as chore(images): update … [argocd-image-updater].
Argo CDSyncs overlays/production automatically on every push. There's no manual kubectl apply.
Draft JobsThe draft API asks the job launcher to start the draft listener and room connector, which are service jobs in the job catalog. They run from the same pinned CORE_IMAGE and WORKER_IMAGE as every other launched job.
SecretsExternal Secrets materializes ESPN cookies, database passwords and API keys from OpenBao. No credential is committed.
Draft dayThe overlays/draft-day overlay adds CPU and memory headroom and a warm worker.

A config change, such as flipping AUTO_PUBLISH_IR_MOVES, is a reviewed PR to fantasy-infra. See Unattended IR publication.

How do I see what happened?​

SignalWhereUse it to
TracesOpenTelemetry to Tempo. Queue messages carry a traceparent, so a worker span joins the API's trace.Follow one action across core and the worker.
LogsLoki. Worker lines carry trackingId and trace_id.Read a bot run. See Telemetry.
DashboardsGrafana, provisioned from fantasy-infra ConfigMaps: bot decisions, execution log, draft pipeline, job runs, tracesWatch the system at a glance.
AlertsGrafana alert rules on run_log and the launcher heartbeat, plus rumor-action outcomesGet told when a job fails or the launcher stops. See Job alerts.
ActivityGET /v1/activity/actions and the UI's Activity pageCheck whether an action was delivered and verified. See Activity submissions.

tracking_id is the human-facing id for one action. Look it up in Activity, then search Loki for it.

Known fragility​

RiskWhy it matters
ESPN's protocol is reverse-engineeredESPN can change endpoints, selectors or the draft WebSocket without notice. Bots fail closed, but a change stops that action until someone updates the worker. See ESPN reverse-engineering notes.
ESPN session cookies expireEvery write needs valid espn_s2 and SWID cookies. The Monday session check gives early warning.
Write routes trust the network/v1/trigger/* and the trade inbox writes need X-API-Key. Other routes, including outgoing trade approval, rely on oauth2-proxy and the cluster boundary.
RabbitMQ user provisioningThe broker image doesn't create users from environment variables once definitions are loaded. An initContainer renders the user into the definitions on every boot.
Kubernetes calls from the APIThe draft endpoints call the Kubernetes API directly. The UI wraps its calls in timeouts because a degraded control plane may never respond.