Operations
Use read-only evidence to identify the failing layer before retrying a workload. A completed Kubernetes Job proves process completion; it does not prove an ESPN action was applied.
| Detail runbook | Use it for |
|---|---|
| Job alerts | Launcher alert semantics and per-job dependencies |
| Schedule order and refresh horizon | Tuesday job ordering and the daily projection/VORP refresh |
| Application outcomes and trade jobs | Rumor-action outcome alert, trade scan and trade inbox |
| RabbitMQ rollout | Durable topology, storage migration gate and queue provisioning |
| Unattended IR publication | Shared OFF flag, rollout verification and rollback limitations |
| Schedules | Complete generated inventory |
Investigate a failed job
-
Find the run. The newest
run_logrow for the job has its status,end_reason, attempt andsummary.log_tail; older failures may be irrelevant. While its k8s Job still exists (one day), inspect it:kubectl get jobs -n default -l fantasy.job=<job> --sort-by=.metadata.creationTimestampkubectl describe job <k8s-job-name> -n defaultkubectl logs -n default -l fantasy.job=<job> --tail=200 -
Inspect events if the container never started (the run ends
failurewithstart_timeout). Check image pulls, missing Secrets, capacity, deadline, API/store reachability and the job's dependencies. -
For a run that never started, read its
end_reason:expired,overlap,pausedordependency_failed, or no row at all after a launcher outage. Anexpiredrun usually waited on a full pod cap, which active KEDA worker Jobs also fill (on draft day the warm worker always holds one slot). See what covers a missed or hung run. -
Fix the dependency and observe the next scheduled run. Confirm the run's success and application outcomes in Grafana/Loki.
-
To run a job on demand,
POST /v1/jobs/<job>/runswith the trigger API key. It queues through the same launcher, so the overlap and pod-cap rules apply.
Do not start a replacement run blindly. Check execution records, queue state and bot retry semantics first; rumor-action retries and outgoing trade claims require particular care. A zero-action success can be a healthy no-op.
Recover an expired ESPN session
- Check the latest
espn-session-healthcheckrun (kubectl logs -n default -l fantasy.job=espn-session-healthcheck --tail=200) and application authentication errors. Separate ESPN/network outages from rejected or expired authentication. - Have the operator refresh the ESPN login using the worker's private login workflow. Update the cookie pair and, when used, browser storage state together through OpenBao; never attach them to an issue.
- Verify ExternalSecret synchronization. Workers prefer stored browser state when present; a stale state file can matter even after the cookie pair changes.
- Refresh affected consumers under the operator's maintenance plan and observe the next healthcheck or normal authenticated read. Existing environment variables and active Jobs retain their prior values; subPath-mounted state requires a new pod.
Verify authentication before resuming dependent publishers. Do not test a login by submitting a lineup, waiver or trade action.
Recover a stuck queue
-
Inspect scaler conditions, worker Jobs and broker readiness:
kubectl get scaledjob fantasy-browser-worker-scaler -n default -o yamlkubectl get pods -n default -l app=fantasy-browser-workerkubectl get jobs -n default --sort-by=.metadata.creationTimestampkubectl logs deploy/fantasy-broker -n default -c render-definitionskubectl exec -n default deploy/fantasy-broker -c rabbitmq -- \rabbitmqctl list_queues -p / name messages_ready messages_unacknowledged consumers -
Use counts, consumer status and sanitized logs to choose the next check:
Evidence Check Ready messages, no workers KEDA trigger queue existence, authentication, scaler events and scheduling capacity Unacknowledged messages Active worker logs, ESPN session and downstream timeouts; a delivery may still be in flight DLQ messages Rejection reason and execution evidence; repair input before considering replay Broker NotReady Definitions init container, credential consistency, PVC binding and resource limits -
Repair the dependency and observe ready/unacknowledged counts decrease with normal successful executions. Do not purge queues, reset storage, synthesize messages or replay DLQs as a diagnostic.
For a broker replacement or first persistent-storage rollout, use the storage approval gate. Preserve the PVC on rollback once it holds messages.
Roll back a deploy
- Identify the failing reconciled Git revision and running image IDs using Deployment. Record the known-good digests and whether database migrations or queued actions occurred.
- For an image regression, coordinate pausing Image Updater for the affected image in the owning Application configuration. Otherwise its tracking of
latestcan overwrite a reverted digest. Preserve the other Application annotations. - Prepare a reviewed Git revert or digest-pinning change in
overlays/production. Check both overlay renders and runtime Job image replacements. Keep compatible schema changes and persistent-store mounts; an image rollback does not reverse migrations. - Merge the recovery change and let Argo CD reconcile it. Its self-heal can undo direct cluster edits; changing live state alone is not a durable rollback.
- Verify synced revision, ready replicas, pod image IDs and newly created Job templates. Inspect older active Jobs separately and validate normal application reads/outcomes before resuming publishers.
- Restore updater tracking only when the published
latestimage is safe again, then verify the next write-back.
Rollback does not cancel queued actions or undo ESPN moves. Keep AUTO_PUBLISH_IR_MOVES off unless separately approved; follow the IR rollback checks for active publisher pods.