Recover migration worker and fix stale provider retry reconciliation
Read-only investigation on git-prk-server of the reporting migration worker repeatedly emitting failureCode provider_deferral. Identify the actual failing operation from deployed code, runtime state and logs; distinguish provider outage from coordinator deferral/reconciliation failure. Findings: migration service/container running. Controller emitted provider-paused every ten minutes through 2026-10-24 18:17 UTC, unexpected at 18:27:57, then provider_deferral every minute from 18:28:57 onward; no subsequent analyzer calls. Analyzer /health returns 200 and authenticated /provider-state returns 200 with expired retryAt 2026-10-24 18:27:48 UTC for report b6525b9d-5466-4e00-6baf-39c3c1e634d6. Workflow readiness returns 200, release workflow_v0.1.7. Confirmed on Ithaca: coordinator logs show the worker's normal defer-provider request returning 409 in ~10ms, with successful database reads. Read-only SQL shows report is Delivered/Processing, attempt count 1, expired leaseUntil 2026-10-24 18:37:48 UTC, no decision. Saved retry lease and DB lease fingerprints differ; source hashes match. Thus worker reclaimed the report around 18:27:48 but failed before a new analyzer call, leaving the previous retry state. Runtime image code confirms reconciliation runs before every claim, while state clears only upon entry to analyzer runProviderAnalysis; stale 409 fails the entire batch indefinitely. 270 migration reports pending, one Processing, 357 Applied at inspection. Earlier coordinator logs rotated (3x10MB), so original unexpected error cannot be recovered there. Original provider cause unavailable because analyzer collapses errors to provider_unavailable. Safe recovery recommendation: stop migration controller, preserve and remove stale provider-retry.json, restart migration analyzer to clear its in-memory copy, then resume controller and verify analysis/queue progress. Do not change DB lease or weaken coordinator validation. Durable fix needs explicit acknowledgement/retirement of reconciled retry state across claim/pre-analysis failures. User authorized recovery and durable fix. Recovery performed: stopped migration service/controller, stopped analyzer, renamed stale retry file to provider-retry.stale-20261025T094018Z.json, restarted analyzer and service; service active. No DB state modified. Verified live controller accepted the formerly stuck report at 2026-10-25 09:41:22 UTC and a second report at 09:42:11, then began a third analysis. Durable fix implemented in clean worktree /home/mathias/Projects/worktrees/player-reporting-provider-recovery, branch fix/retire-reconciled-provider-retries, based on worker origin/main 5c0a8d4. Authenticated correlated acknowledgement persists deferred:true before another claim, retaining pause/report priority and skipping only already-reconciled deferrals. Actual stale/unacknowledged 409s still fail closed; no coordinator/DB change. Regression reproduces original failure on baseline, passes with fix and reload after evidence-fetch failure. HTTP tests cover authentication, invalid/stale state, idempotence, new-failure isolation, private persistence and retained pause; controller tests cover failed acknowledgement preventing new claims. Final npm test passes: TypeScript build, 45 Node tests and 20 Python tests; git diff --check clean. Manual diff review confirms stale conflicts remain protected and acknowledgement precedes all subsequent claims. Red-green regression was observed against the unchanged baseline. Final live check at 09:45:39 UTC shows six accepted migration proposals since recovery, one completed batch and seventh report analyzing, no new provider_deferral logs. Committed as 838d47b and squash-merged PR #12 into main as 3a14f11c84203a73068659b4e58715fc2ff4d517 at user's request. PR CI passed; main CI tests and default worker automatic deployment passed. Paused migration with analyzer idle, deployed the identical revision using the instance-aware deployment helper under its normal lock, verified deployment receipts and exact running/stable image IDs for both instances. Both analyzer health endpoints return 200; authenticated stale acknowledgement returns 409 with retry state unchanged. Resumed migration service. Final verification: migration controller prk-reporting-migration-controller-run-a43887c0a27c uses the receipt's new controller image and accepted report c9a606e7-8d92-95bf-d0c4-3ab8c6738601 at 2026-10-25 09:50:22 UTC. Migration systemd service active. Local worktree clean. Recovery, regression fix, CI, deployment and resumed processing verified. No DB changes or secret/auth modifications. Actual provider exhaustion was not induced; provider-retry edge cases are verified by local regression/HTTP tests, while live verification covers deployed image identity, health, stale-state rejection and accepted processing.
Checklist
4 of 4 complete- Recover live migration worker and verify accepted reports resume
- Implement durable correlated deferral acknowledgement with restart/pre-analysis regression coverage
- Run worker validation and review changes; record deployment boundary
- Deploy squash-merged revision to default and migration workers and verify resumed live processing
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Operations
- Task ID
tsk_y87c29rjey- Updated