Pause and retry reporting after Codex provider failures
Implement user-requested fixed 10-minute provider retries before bulk migration. No reset prediction/time parsing and no claim that invented provider responses verify real Codex behavior. Test our own control-flow, persistence and lease accounting; actual exhausted-account behavior remains unverified. Scope: engines coordinator and player-reporting-worker. Treat provider-call/model-execution failures as temporarily unavailable without guessing quota wording; non-provider schema/source/budget failures retain bounded attempts. Persist an isolated analyzer/account cooldown for 600 seconds across restarts; stop further claims while paused and prioritize interrupted report when retrying. Provider outages must not consume report attempts. Reuse leases/revisions/readiness and dedicated writable auth storage. Add idempotent lease/source-correlated deferral with protection against stale requests, accepted decisions and concurrent reclaims; avoid DDL if existing state suffices. Require advertised deferral capability before upgraded workers claim. Successful continuous batches should progress without daily pauses; idle normally on empty work, wait 600 seconds on provider errors. Preserve explicit-ID and per-batch bounds. Implemented on fix/retry-reporting-provider-outages in /home/mathias/Projects/engines (base origin/develop 890cd67bf) and /home/mathias/Projects/worktrees/player-reporting-queue-lanes (base origin/main 2d6948a). Committed and pushed under explicit PR-opening authorization: engines c7a690e82 (fix: Preserve triage attempts during provider outages) and worker 26de1e0 (fix: Retry provider outages after ten minutes). Both worktrees clean. Open, cross-linked PRs: engines #2829 — https://git.project-rk.com/prk/engines/pulls/2829 (fix/retry-reporting-provider-outages → develop); worker #10 — https://git.project-rk.com/prk/player-reporting-worker/pulls/10 (fix/retry-reporting-provider-outages → main). No production rollout or extra backlog runs. Production migration runtime remains unscheduled and unchanged. Engines merge-tree against fetched develop 683af0873 is conflict-free; merged result not executed. Backend: POST reports/{id}/defer-provider checks worker auth, report/lease/source, delivered state and absence of accepted decisions under existing SQL triage lock. Releases lease, returns Pending and subtracts only current claim attempt. Retained token makes replay idempotent; next claim replaces it. Expired but unreclaimed leases can be refunded; stale or accepted ones conflict. No DDL or Forgejo writes. Health advertises providerDeferral=true; upgraded workers/deployment preflight require it before claims. Worker: SDK prompt/model execution/deadline failures map to provider_unavailable without parsing provider error text/reset times; agent-level immediate retry disabled. Analyzer atomically persists provider-retry.json mode0600 alongside its own auth.json, with fixed600000ms retryAt and only interrupted report/lease/source metadata. Authenticated /provider-state exposes recovery data, health429 during pause (503 while busy). Controllers recover deferral before new claims, preserve lane/explicit-ID/batch limits, prioritize interrupted report and wait/retry indefinitely. Earlier successful proposals remain counted and are not resubmitted. Non-provider validation/code/output-budget failures retain bounded attempts. Transient deferral transport/5xx failures pause before claims; ownership conflicts stop for attention. Malformed durable state fails closed without leaking contents. Auth/deployment readiness accepts a valid paused429 service rather than rolling back or erasing pause. Continuous mode: deploy/run.sh triage-continuous runs successful batches back-to-back and polls empty work every60s, with fixed10-minute provider pause. Default restricted SSH router is not widened. Documentation requires updating the trusted migration wrapper/service to permit this mode, Type=simple and TimeoutStartSec=0 for long runs; stop continuous service before auth/deploy lock acquisition. One-shot runs also retain their original batch limit across provider waits; external process timeout can stop them, persisted recovery survives. Validation: PlayerPortal .NET suite245 passed/3 existing skips, including realSQL concurrent idempotent refund and reclaim safety. npm test39 Node +19 Python passed, including our own pause persistence, restart, selection, fixed delay, batch accounting, stale/ambiguous deferral and no-claim boundaries. These are checks of our defined internal contract, not invented Codex response simulations or proof of upstream quota behavior. Nominal isolated SQL/Forgejo/coordinator lane acceptance passed; latest controller Docker image built and image-build tests pass. Backend static feature review found no actionable findings (worker sibling outside tool scope, manually inspected). Shell syntax and both git diff checks pass. Logs /tmp/reporting-retry-dotnet.log, /tmp/reporting-retry-tests-final.log, /tmp/reporting-retry-stack.log, /tmp/reporting-retry-image.log. Dedicated acceptance stack cleaned up; no production calls/changes and no live account drained. Real exhausted-account behavior explicitly remains unverified as requested. Rollout: coordinator-only release first (providerDeferral=true), then worker update; no new portal/schema migration. Update the server's trusted migration wrapper/manual service for continuous mode only when deploying. Do not downgrade to old pause-ignoring workers with a saved pending pause.
Checklist
3 of 3 complete- Implement and verify idempotent lease/source-correlated deferral with neutral attempt accounting, stale/concurrent request protection and coordinator capability negotiation.
- Implement persistent isolated 600-second provider pause and interrupted-report retry, preserving selection/batch limits and keeping non-provider failures bounded.
- Run relevant backend/worker validation and document rollout/continuous operation plus explicit limitation that no live Codex usage exhaustion was tested.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Bug
- Task ID
tsk_agjvehgzdd- Updated