Run a backtest
Run a standalone replay against a development database to compare historical decision results. Read Backtesting before interpreting its scores.
Install the repo's dependencies and configure DB_HOST, DB_NAME, DB_USER and DB_PASSWORD for a reachable development PostgreSQL database with schema creation privileges. Each run wipes backtest_<season>; run in a standalone process because its temporary connection-pool swap must not overlap API requests or another replay.
Run the replay
-
Open a terminal in the repo root.
-
Run the season replay:
python -m src.backtesting.season_replay --season 2024 -
Verify the JSON summary and weekly CSV in
backtest_output/2024/. -
Verify the appended summary through the backtest results API or the development database's
backtest_resultstable.
The command downloads historical data and writes reports without publishing actions to ESPN. The original connection pool is restored after the isolated replay.
Tune the market weight
ESPN_MARKET_WEIGHT is chosen by replaying seasons with FantasyPros weekly ECR standing in for ESPN start%, over the rolling base projection:
- Check the stand-in still agrees with ESPN for a week that has a snapshot. Dump that week's
espn_player_ownership(player_id,position,percent_started) to CSV, then runpython -m src.backtesting.market_stand_in --season 2026 --week 10 --espn-csv espn.csv. A position under 0.8 Spearman is a poor stand-in. - Run the sweep against a throwaway database (it wipes
backtest_<season>once per weight and appends tobacktest_results):python -m src.backtesting.market_weight_sweep --seasons 2023 2024 2025. - Read the printed table: waiver gain and lineup points per season and pooled, each against weight 0 in its group.
Troubleshoot the run
| Symptom or need | Action |
|---|---|
| Cannot connect or create schema | Check development DB settings and schema privileges |
| Earlier replay data disappeared | Re-running wipes that season's isolated schema; inspect persistent reports/summaries |
| Need a shorter replay or custom output | Call run_season_replay with weeks, rounds or out_dir; these are not CLI flags |
| Need offline regression checks | Run DB_PASSWORD=test pytest tests/ instead of a historical replay |
| Unsure what a score measures | Read comparison definitions |