Analyze post-hot-loop PFE memory and CPU capture

Analyze verified 2026-10-05 primary-PFE capture from image 0792bb89c4662faf2448eabaabea8e9e7924583f after deployed PR #2574/#2575 (capture tsk_bdca42mmww). Compare with post-navigation 2026-10-04 capture/investigation tsk_5j446c4ges and pre-navigation 2026-10-03 evidence. New capture at ~11h uptime: RSS ~16.3 GiB before collection, heap dump 18,081,497,088 bytes, 60-second dotnet-sampled-thread-time + gc-verbose nettrace 91,195,826 bytes. Quantify managed live/free composition, glibc arenas/free/live retention, Bepu/navigation ownership and playfield/entity counts; compare allocation rate, GC cadence and hot stacks to verify wall/dynel/fight/corpse improvements. Explicitly separate true kernel CPU attribution from managed wall-clock samples and trace overhead. Read-only offline analysis; no production experiments or repository edits. Verified findings Deployment identity correction - Captured image 0792bb89c4662faf2448eabaabea8e9e7924583f contains merged hot-loop PR #2574/f80f2847e but does NOT contain deadline-queue commit f25adfe7c; Gitea PR #2575 remains open. The capture metadata's initial #2575 assumption is wrong and this capture cannot validate corpse due queues. Capture task tsk_bdca42mmww was corrected. Memory at ~11h uptime - RSS before collection ~16.3 GiB, essentially the same as the ~16.3 GiB seen at ~6h in the 2026-10-04 capture despite nearly double uptime. This is evidence of a high plateau over this interval, not proof of a permanent bound. - Managed GC allocated 6,696,425,224 bytes / committed 6,734,430,208 bytes (6.27 GiB). Full heap walk 6,687,397,677 bytes includes 477,214,936 Free. SOS live analysis finds 78,886,970 objects / 5,541,209,188 bytes (~5.16 GiB live). - A striking ~1.06m InternalEntityEntry / PlayerCharacterStat/snapshot cohort is dead, not rooted: only 32 InternalEntityEntry objects are live, while 1,058,183 entries / 101.6 MB plus ~130 MB dictionary capacity are dead and collectible on the next full GC. Representative gcroot has zero roots. No gen2 occurred during the 60-second trace, explaining transient managed retention; this is not evidence of a rooted EF context leak. - Live world is larger than 2026-10-04 in some dimensions: 18,861 NPCs vs 16,252; StatPair 11.61m/928 MB vs 9.95m/796 MB; wearable items 535.5k vs 413.5k. There are fewer current playfields: 38 locator nodes / 38 Bepu pools vs 46. Managed growth is consistent with a larger entity/stat set plus uncollected gen2 garbage, not disposed-playfield accumulation. Native allocator comparison - ptmalloc: 197 secondary heaps / 115 arenas / 9.806 GiB mapped. Only 1.301 GiB is in-use-marked; 8.505 GiB is free but retained (3.415 GiB top chunks, 5.090 GiB interior). - 2026-10-04 at ~6h: 217 heaps / 134 arenas / 10.531 GiB, 9.142 GiB free. Thus at ~11h the newer process has 19 fewer arenas, ~0.725 GiB less mapped native memory and ~0.637 GiB less free retention. It did not continue upward over this interval, although 8.5 GiB retained remains the dominant excess. - Managed Thread objects fell 112 -> 50 and worker-local queues 89 -> 39, consistent with reduced hot-loop work/thread-pool demand and fewer arenas; ThreadPool.SetMinThreads(1000,1000) remains configured, so causality is not solely the setting. - 38 current Bepu pools own 1.025 GiB vs 46/1.112 GiB before. Navigation remains clean: 34 active sessions, zero detached validated node pools or tile arrays; 33 owned arrays / 35,652 tiles / 243,163,184 bytes. Allocation and GC comparison (61.8-second trace) - Estimated allocation fell from 136.9 MiB/s to 93.2 MiB/s: ~31.9% reduction. - Collections fell 1,029 -> 711 and 16.63/s -> 11.51/s: ~30.8% reduction. New split: 599 gen0, 112 gen1, no gen2. - GC event duration fell 7.170s -> 2.378s (~66.8%); p95 15.44 -> 7.83ms (~49%); max 33.40 -> 14.37ms (~57%). gc-verbose adds suspension/stack-walk overhead, so compare directionally rather than treating these as uninstrumented pauses. - The prior dominant sources are absent from top-100 stacks/allocation leaves: DynelCollisionPfSystem, WallCollisionPfSystem and FightTally reconciliation. This directly validates the deployed #2574 changes. Current CPU/allocation leaders - Managed wall-clock profile still is not a kernel CPU profile. Container point sample was ~542% vs ~582% previously, but point samples are too noisy to claim a percentage improvement. - Physics is now the clearest active CPU path: PhysicsManagementPfSystem.OnTick 5.27% inclusive / 0.82% exclusive of all wall samples; Bepu Tree.GetOverlaps 3.22% exclusive. PlayfieldCell.HeartbeatNpc is 6.69% inclusive / 1.23% exclusive. Indoor doors remain ~1.56%; brute-force ACG RayHitsMesh falls 1.20 -> 0.69% exclusive. - Allocation leader is now Character[] (~939 MB estimated). Top leaves are List(IEnumerable) ~602 MB, CorpseManagementPfSystem.OnTick ~579 MB, string concat ~535 MB, HeartbeatNpc ~408 MB, stat entry arrays ~322 MB and inventory snapshots ~248 MB. - Corpse OnTick attribution is from the deployed direct-enumeration version, not #2575. Mass expiry/removal still calls player snapshot broadcasts and workload differed; do not compare its 579 MB directly to the previous 241 MB as a regression. Merge/deploy #2575 before evaluating this path. - The next general hot-loop target after #2575 is cell/subscriber snapshot materialization: PlayfieldCell.SubscribersInVicinity returns copied lists and Character[]/List construction now dominates allocation. Any fix must preserve lock/stable-view semantics under parallel cell ticks—no raw mutable enumeration. Conclusion/priorities 1. PR #2574 produced a material verified reduction: ~32% allocation, ~31% minor-GC frequency, ~67% GC event time; its named hotspots disappear. 2. Memory did not grow between comparable 6h and 11h RSS observations; native arena retention remains ~8.5 GiB and is still the largest footprint issue (tsk_pwwgzyv3kz). 3. Merge/deploy PR #2575, then capture again to validate corpse deadline queues; this capture predates it. 4. Next CPU work should use collect-linux/perf and focus Bepu overlap/Physics OnTick. Next allocation work should design an immutable/versioned subscriber publication instead of per-read List/array copies. Evidence under /home/mathias/Downloads/pfe-memory-20261005/: heap-summary.txt, live-dead-summary.txt, trace-summary.txt, CPU top reports, ef-entry-root.txt and analysis/{arena-totals,pool-totals,navigation-inventory,managed-diff-vs-20261004}.txt. No repository or production changes.