Investigate intermittent empty player known-issues catalogue before release
Investigate the old known-issues page intermittently rendering filters but no issue rows. Trace pre-change reader visibility/freshness/hash gates and coordinator publication scheduling, retries, remote pagination and health signals. Prioritize intermittent causes over permanent filters/configuration. Reproduce relevant mechanisms with controlled tests where practical. Distinguish code-proven causes from incident root cause requiring production timestamps/logs/database confirmation. Preserve fail-closed publication/audience/evidence protections. No production mutation, release or code fix authorized by this investigation request. Working tree remains clean; PR #2940 has not been changed. Verified mechanism: the old reader and new tracker both require ConfirmedAt strictly newer than now minus five minutes, with future timestamps excluded. A temporary probe compiled the original reader from the PR parent and used identical synthetic published records with controls: 3 visible initially and at 4m59.999s, 0 at exactly 5m and at 6m, then 3 again after changing only confirmation timestamps. Probe: /tmp/prk-catalogue-probe. Freshness checks must not simply be removed because they protect withdrawn/private publications. Read-only production evidence on Ithaca, 2026-11-01: deployed coordinator workflow_v0.1.9, revision d42dd4cbea614894a93daced108d5a02ec12cece. Inspected that revision's worker/publication/resolution code, not merely the pending PR. It runs delivery/staff/decisions/publication/resolution serially under one tick lease, configured PollSeconds=30. General confirmation timestamps observed 16:01:48.945Z and 16:04:46.406Z, approximately 177 seconds apart. At 16:04:34Z, 571 General and 174 Spiritech snapshots passed database visibility gates before content/hash validation; the catalogue was not empty during the checks. Runtime health remains green on schema/DB connectivity even when publication processing fails. Concrete operational fault: retained coordinator logs contain resolution-phase OutOfMemoryException at 16:04:11.561Z while Docker health remained healthy, with no coordinator restart or kernel OOM kill. Coordinator memory limit is 512 MiB; docker stats sampled 461.5 MiB (90.14%). SQL aggregates found 970 reports with 170,561,088 bytes (~163 MiB) of TriageContextJson, largest 233,060 bytes, plus ~8.9 MB report bodies. Deployed publication/resolution syncs materialize full report entities, including this context they do not need, making them a concrete memory-pressure investigation target. Source proof does not establish an allocation stack or memory leak. The first loose log filter counted ordinary SQL CommandTimeout metadata; it was corrected. A broader retained sample contained 135,823 lines covering 14:56–16:10Z with one resolution OOM and no observed publication-phase failure, not a full six-hour history. Incident timing supplied: user last saw the blank catalogue approximately 30 minutes before 2026-11-01T16:16:20Z, so investigate around 15:46Z with a 15:36–15:56Z buffer. Retained coordinator logs should cover that window. There is still no explicit per-audience successful-refresh history/availability signal, so correlate SQL publication-write activity and error/request logs without treating inferred write bursts as exact confirmation timestamps. Live incident correlation now proves a >5-minute production refresh gap and the exact empty/recovery mechanism. At 2026-11-01T16:19:15.403Z, General latest ConfirmedAt was 16:12:44.096Z (391 seconds old) and Spiritech 16:12:51.785Z (384 seconds old); both had VisibleGate=0 while the user reported the page empty. Coordinator and portal were healthy, coordinator had no restart/kernel OOM kill, memory was 459.4 MiB/512 MiB, and the preceding ten minutes had no logged workflow phase failure. At 16:23:57.011Z after the user reported recovery, General had 571 visible rows confirmed at 16:21:04.451Z and Spiritech 174 confirmed at 16:21:14.005Z. Thus recovery occurred solely through the next publication refresh. The General publication completion gap was 16:12:44.096 to 16:21:04.451 (~8m20s); the five-minute fail-closed threshold produced an approximately 3m20s blank interval beginning around 16:17:44Z. The prior sampled General completion gap was 16:04:46.406 to 16:12:44.096 (~7m58s), showing repetition. PollSeconds=30 is not the effective publication refresh period because the serial tick runs delivery/staff/decisions/publication/resolution under one lease and the next timer wait follows completion. The deployed resolution phase scans all linked reports and makes per-report DB/Forgejo calls; it previously threw OutOfMemoryException at 16:04:11.562Z. Source and production evidence therefore identify serial workflow duration/memory pressure as the root operational cause of intermittent empty catalogues, even when no error occurs during a particular gap. The exact per-phase duration/allocation split remains uninstrumented. Release recommendation was to block portal/coordinator release until publication freshness cannot lapse during normal processing. Fix should decouple or prioritize refresh so it completes comfortably within five minutes, bound/project resolution and publication queries to avoid loading unused large report context, and expose per-phase duration plus last successful publication timestamps in readiness/metrics. The portal should return an explicit temporary-unavailable state when all otherwise-published snapshots are stale, rather than rendering a genuine zero-results state; do not remove the fail-closed privacy/publication gate. The authorized fix was subsequently implemented as commit 4a3fef5cb and pushed to PR #2940: https://git.project-rk.com/prk/engines/pulls/2940. It prioritizes publication, bounds/projects lifecycle work, adds stale readiness and portal UX, raises memory headroom, and includes the recommended runtime, SQL and browser regressions. No production deployment or live verification of the fix has occurred. SSH endpoint selection: configured git-prk-server was only the Git/analyzer host; Ithaca required the documented root@ithaca.project-rk.com user, so normal SSH used the existing configured agent without changing SSH/auth configuration.
Checklist
1 of 1 complete- Identify and reproduce intermittent catalogue-emptying mechanisms and separate verified behavior from unconfirmed production causes.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Research
- Task ID
tsk_brm8ahf2zn- Updated