Skip to main content

Application outcomes and trade jobs

Use this page when the rumor-action outcome alert fires, or when you need the schedule and safety settings of trade-scan and trade-inbox. For launcher run alerts, see Job alerts.

Rumor-action application outcome​

Rumor action incomplete or failed processing (rumor-action-trigger-failure) complements the launcher's run-failed alert. The launcher runs python -m src.services.rumor_action_service daily at 21:45, as the last step of Tuesday's tuesday-weekly chain, and after each digest on rumor.signal_created, not the HTTP endpoint. Both paths record one execution_log row with transaction type rumor_action_trigger and bot rumor_action_service.

PropertyValue
Rule filebase/telemetry/rumor-action-outcome-alert.yaml
EvaluationEvery 5m
FiresImmediately when any failure or partial_failure was recorded in the last 26h
DashboardFantasy → Fantasy Execution Log Overview → Rumor Action Outcomes (panel 6)
Why it doesn't select only the latest run

Until core-pipeline#192 fixes retry and watermark semantics, an empty retry could otherwise erase evidence of lost work. A recovered retry stays visible until the original failure ages out. This is an observation window, not proof that the error remains unresolved.

What counts as healthy​

HealthyAlerts
No new signals after a successful fetchMissing identity mappings for proposed actions
Low-confidence or non-actionable signalsIncomplete accounting
Superseded signalsEvaluation failures
No beneficial movePublish failures
Already-queued actions

Signals and actions have separate counts, since one signal may produce multiple actions. “Accepted” means queued by the core API, not executed in ESPN.

Investigate​

  1. Open panel 6 and check reason and the signal reasons.
  2. Check the proposed, accepted, duplicate, missing and failed counts.
  3. Check the watermark range and error_message.
  4. Use job_name for pod logs.
  5. Look for rumor_action_outcome in Loki if PostgreSQL logging failed.
SituationWhat happens
Empty queryReturns zero.
Database query errorUses Grafana's Error state.
Run couldn't persist its outcomeThis SQL rule can't detect it. Inspect the log fallback and Kubernetes alerts.
Missing runsCovered by #118, not by a zero-action threshold.
Don't blindly rerun

#192 is still responsible for retry safety and watermark recovery. Investigate unresolved ESPN identities or input data before retrying.

The app signal also covers HTTP invocations, and its failure count is per invocation, not per launcher run.

Provisioning ownership and rollout​

  1. Deploy core-pipeline's additive outcome_details JSONB migration and service changes. Migrations run at API startup, so confirm the column exists before expecting job outcome writes or opening the new dashboard panel.
  2. Remove homelab-deployment's old rumor-action-trigger-failure.yaml from its alerting ConfigMap generator and deployed ConfigMap. Don't provision deleteRules: the existing Grafana rule is retained until fantasy-infra takes ownership.
  3. Deploy this fantasy-infra rule, retaining the old UID/group, folder and provisioning filename.
  4. Verify the Grafana rule shows the 26h SQL, 5m cadence and panel 6 link, and that the new table displays structured outcomes. No live Jobs need to be triggered.
One owner per rule

Never leave two ConfigMaps supplying the same file or rule.

Run ./tests/rumor_action_alert/run.sh (Python/PyYAML and Docker). It tests the committed SQL on isolated PostgreSQL 16 for:

  • no data and a healthy no-op
  • total and partial failure, and failure then success
  • unrelated rows and legacy rows
  • exact window boundaries

The launcher's own rules are tested in ./tests/launcher_alerts/run.sh.

Outgoing trade scan​

trade-scan runs in the worker image and stages proposals for separate human approval only.

SettingValue
ScheduleWednesday and Friday 13:00 America/Chicago
AttemptsOne per league/team/Chicago calendar date, shared with manual runs
Process bound / run deadline15 minutes / 16 minutes
Late startUp to 30 minutes (catch_up), counting time queued behind the pod cap
RetriesNone: max_attempts stays 1
Profileworker-scan: the worker secrets and config, plus the LLM keys
ImageWORKER_IMAGE, the digest the ScaledJob runs
ProducersNo trade_proposals producer/scaler pair, on purpose
  • Off-season, duplicate and no-target outcomes are successful no-ops.
  • Failures have no automatic retry. Inspect trade_scan logs and execution_log claims before recovery.
  • Only scheduled runs take the daily claim (RUN_TRIGGER=schedule). A manual run skips it, which is why the catalog marks the job acts: true, so the UI confirms first.
  • Pause this job after the configured ESPN league trade deadline, with PUT /v1/jobs/trade-scan/pause. Paused slots end skipped (paused) and don't alert. DELETE the same path to resume.
  • See the worker README's outgoing scan section for read-only checks, claim retention and failure recovery.

Incoming trade inbox​

trade-inbox runs node src/bots/trade_inbox.js --refresh. It only reads ESPN, persists incoming offers in Postgres, and publishes trade.offer_received to job_events for each new offer.

SettingValue
ScheduleEight times daily, 08:00 through 22:00 America/Chicago
Weekly runs56 inbox runs, plus two refreshes from the Wednesday/Friday 13:00 trade scan (between the noon and 14:00 inbox runs)
PersistenceThe worker CLI defaults to --check (no database writes). The scheduled command selects persistence explicitly.
BoundsTen-minute process, eleven-minute run deadline, thirty-minute late-start allowance (catch_up)
Concurrencyoverlap: forbid and no automatic retry (max_attempts stays 1). Manual or delayed runs aren't globally serialized with the trade scan.
Core URLWorker default http://fantasy-core-pipeline.default.svc.cluster.local:8000, matching trade-scan
Score timeoutDefaults to 20000 ms. TRADE_SCORE_TIMEOUT_MS stays unset.
EventsAMQP_URL from the amqp bundle (rabbitmq-credentials), for trade.offer_received. No job subscribes to it yet.

Each active-season refresh does the following:

  • reads pending transactions once
  • optionally reads history for vanished offers
  • makes at most ten sequential core score calls, and reuses unchanged successful scores

Core can read ESPN for cold scoring context. Offseason runs are successful no-ops.

No responses are enabled

TRADE_RESPONSE_ENABLED and both shadow thresholds stay unset, and unset DRY_RUN remains true. No accept, decline or counter is enabled by this rollout.

The launcher's trade jobs carry no TRADE_DRY_RUN or TRADE_SUBMIT_ENABLED either, because no catalog profile or bundle sets them, so scans keep dry-filling. Live trade submission is armed only on the queue worker (the fantasy-browser-worker-scaler ScaledJob) and still needs a matching human approval.

Verify​

  1. After an Argo CD sync, wait for the next slot, then inspect the run: kubectl logs -n default -l fantasy.job=trade-inbox --tail=200, and its run_log row with job_name = 'trade-inbox'.
  2. Confirm the image, command, deadline and Refreshed outcome (including scoreCalls/exits).
  3. A failed run or a skipped (expired) slot reports through the launcher run-failed alert. The overnight gap is intentional.
Don't create test traffic

Don't start a manual refresh or send test queue messages for verification. Core inbox API endpoints aren't a prerequisite for this job.