Prevent reporting workflow stalls from emptying the known-issues catalogue
Fix the verified production failure where publication confirmations refresh roughly every eight minutes while the portal expires them after five, causing recurring empty General and Spiritech catalogues. Keep fail-closed audience/publication/privacy guarantees. Prioritize publication freshness independently of slow lifecycle work, bound and project resolution processing so it does not materialize large unused report context or monopolize the workflow, add last-success/phase-duration signals and fail readiness when confirmations are stale, and render a temporary-unavailable state rather than a genuine zero result when all published snapshots have expired. Raise the reporting coordinator memory limit as headroom, not as the only fix. Add production-scale regression coverage for long/failed resolution phases, freshness health, bounded work/fairness and memory-sensitive query shape. Implemented and pushed as commit 4a3fef5cb on feature/player-issue-tracker; existing pull request #2940 updated: https://git.project-rk.com/prk/engines/pulls/2940. Production verification completed read-only on Ithaca. Portal portal_v0.2.0 and coordinator workflow_v0.2.0 are both healthy, restart count 0, kernel OOM false, and both images carry merged revision e9544b6ded552f8a9b4e7af1a64a4dd1bc814990 (PR #2940 merged at 2026-11-01T17:45:44Z). Portal health returns ready. Coordinator health returns ready/schemaVerified/databaseReady/publicationFresh true with phase duration/failure data. Startup health returned expected 503 until the first successful publication, then became healthy. Migration 014 is recorded once. At 18:02:53Z, General had 571 published/visible rows and Spiritech 174, with a single shared confirmation epoch and age 37 seconds. No pending player revisions. Nine read-only health samples over 160 seconds observed publication epochs advancing every ~30 seconds, ages 27–47 seconds, publication durations 23.9–26.8 seconds, no publication failures, and resolution durations normally ~0.02–0.03 seconds with one 1.48-second sample. Final coordinator memory was 372 MiB/1 GiB (36.3%), CPU 1.87%, restart count 0, healthy. The prior deployment sampled ~459–548 MiB under 512 MiB and had an OOM. Retained coordinator logs since deployment contained no workflow phase failure, work-budget timeout, tick failure or OutOfMemoryException. Portal recent logs contained no matching 5xx/unhandled exception. Database and health evidence prove publication cadence is now well inside both the four-minute readiness limit and five-minute portal gate. Catalogue visibility was verified through authoritative snapshot gates/counts and coordinator freshness. User subsequently confirmed the authenticated production UI is working and looks correct. No production mutation, restart or manual refresh was performed. Behavior: publication is first each tick with a single confirmation epoch. Post-publication phases share a configurable 120-second budget and rotate start phase after exhaustion. Delivery/executor cancellation discards pending tracked mutations, reloads only durable claims, clears/shortens leases and reconciles with CancellationToken.None. Publication report reads are projected; resolution selects at most 20 actionable rows, excludes same-cycle terminal suppressions, revalidates mutable canonical state and routing revision per report, and defers temporary holds. Health readiness requires publication age under four minutes and reports phase durations/failures. Container limit raised 512 MiB to 1 GiB and startup grace to 180 seconds. Portal behavior: ownership-scoped stale detection returns API 503 code publication_stale and an explicit temporary-unavailable server state. Live 503s hide previously rendered rows; successful retry restores them without document reload. Five-minute fail-closed visibility remains intact. Validation: PlayerPortal suite with disposable SQL passed 332 tests, 3 unrelated trade-DB tests skipped. Portal and coordinator warnings-as-errors builds passed. 15 Node UI tests and 53 deployment tests passed. SQL regressions cover report projection without TriageContextJson, schema/readiness, concurrency and bounded resolution. Runtime regressions cover publication-before-slow-resolution, post-timeout phase rotation, cancellation lease cleanup, batch fairness, manual closures, routing epochs and canonical hold changes. Chrome fixture verified live stale 503 hides rows, explicit message fits at 390px, and retry restores rows without reload. Working tree clean. Existing production incident evidence remains recorded in the investigation task; fix not deployed or production-verified.
Checklist
3 of 3 complete- Keep publication confirmations refreshed within the portal freshness window even when resolution work is slow or fails.
- Bound/project resolution processing, add freshness/phase observability and stale-catalogue UX, and increase coordinator memory headroom.
- Run SQL/unit/browser/deployment validation and update PR #2940 without claiming deployment.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Feature
- Task ID
tsk_gzxhwj5m6c- Updated