Analyze October 5 evening PFE capture after lifecycle and physics changes
Analyze primary Ithaca PFE capture tsk_sg8xnx2bxa, local /home/mathias/Downloads/pfe-memory-20261005-evening/. Image b126bf974bf267794ce0af956092868c9595c865 contains f25adfe7c (PR #2575 lifecycle deadline queues) and 8254444bd (PR #2586 physics/cell snapshots), verified by git ancestry. Compare with earlier October 5 analysis tsk_hyt5pej7t7/capture tsk_bdca42mmww and October 4 baseline tsk_5j446c4ges. New process started 2026-10-05T18:02:37.658900274Z, capture approximately four hours later; baseline approximately eleven hours. Read-only offline analysis; no production intervention or repository edits. Acceptance: quantify capture uptime/load/RSS, managed committed/live/dead/free populations with deep reachability, native arena mapped/in-use/free retention and validated navigation/Bepu ownership, sampled allocation/GC rates and top managed wall-clock paths. Correlate changes with deployed source; account explicitly for unequal uptime/world counts. Do not equate managed wall-clock percentages to CPU-core shares. Preserve evidence outside repository and report verified findings separately from hypotheses. Verified findings - Capture 2026-10-05T22:03:04Z, uptime 4h00m26.341s, image b126bf974; RSS 14,310,276 kB / 13.647 GiB, swap zero. Earlier image 0792bb8 at ~11h had ~16.3 GiB RSS. New primary locator contains 39 current playfields (sum of _countPerLock raw array), versus 38 earlier. Live dump objects: 15,639 NPCs versus earlier raw 18,861; 47 live PlayerCharacters (72 raw, not an authenticated online count), 39 live corpses, 964 live terminals. World/uptime differ; raw deltas are not causal or steady-state guarantees. - Managed allocated 6,849,399,432 bytes, committed 6,876,254,208 / 6.404 GiB; full walk 6,840,193,505. Live 5,540,647,905 / 5.160 GiB, essentially unchanged from 5,541,209,188 earlier. Free holes 1,102,002,184 / 1.026 GiB; dead summary 1,299,545,600 includes Free, so actual dead object data 197,543,416 / 0.184 GiB. EF entries 501 raw, 67 live, 434 dead: no million-entry cohort this time. Representative StatPair gcroot verified reachability; no raw-count-based leak conclusion. - glibc secondary arenas: 109 arenas / 166 heaps, mapped 7,482,839,040 / 6.969 GiB; in-use-marked 1,529,480,736 / 1.424 GiB; free retained 5,953,108,752 / 5.544 GiB (top 2,453,608,544, interior 3,499,500,208). system_mem matches mappings; only 912 unclassified bytes. Earlier mapped 9.806 / in use 1.301 / free 8.505 GiB: ~2.84 GiB less mapped and ~2.96 GiB less free retention, but new uptime is much shorter. 111 mostly empty heaps retain 4,099,445,536 free bytes. These totals exclude main arena and are mapped, not directly resident, bytes. - 39 Bepu pools own 1,198,784,512 / 1.116 GiB of native blocks versus 38 / 1.025 GiB earlier. Representative pool roots through PhysicsManagementPfSystem; pool PowerPool layout checked with SOS before adapting previous parser. 36 navigation handles are all active state=4, with 36 sessions/meshes/queries. Secondary-arena scan validates 31 large and 28 small node pools, none detached; 36 owned tile arrays / 35,328 tiles / 278,631,248 bytes, none detached. Coverage is validated secondary-arena allocations, not exhaustive main-arena inventory. Representative navigation root reaches NavigationMesh owner. - Same nominal 60s dotnet-sampled-thread-time,gc-verbose providers; actual duration 61.200s versus 61.771s. New sampled allocation 6,455,531,944 bytes / 100.596 MiB/s vs 93.2 (+7.9%); GCs 760 / 12.418 per second vs 711 / 11.51 (+7.9%). 629 gen0, 131 gen1, no gen2; event duration 2828.260ms vs 2378.1, p95 8.432ms vs 7.825, max 15.283 vs 14.368. These instrumented event times are not uninstrumented pause metrics. - Physics OnTick inclusive wall samples 5.27% -> 0.85%, Bepu Tree.GetOverlaps exclusive 3.22% -> 0.03%; deployed SetPosition equality guard skips ApplyDescription for unchanged poses. Character[] estimated sampled bytes 938,598,704 -> 92,480,408 (~90% reduction, ~90% rate reduction). Subscribers getter and CorpseManagement OnTick absent from top allocation leaves; previous corpse leaf 578,669,984 bytes. Named dynel collision, wall collision and FightTally hotspots remain absent from top reports. Top-100 absence means below ranking threshold, not zero work. These findings support PR #2575/#2586, but not an exact CPU-core improvement. - Remaining allocation owners, inclusive and overlapping: TemporaryItemPfSystem ~616,664,720 bytes (9.61 MiB/s); ACG FindPath ~401,989,528 (6.26 MiB/s); NpcEquipment.HandleSpawnMonsters ~449,689,632 (7.01 MiB/s); suppression ~180,900,864. Exact stacks prove temporary-item ticks call ContainerPage.GetContents()->Where().ToList() on all five player inventory pages; main KVP[] path ~463 MB. Selector concat paths total ~507 MB; RandomFrequencyGate.GetDebugText inclusive ~292 MB. Deployed Selector/Sequence call GetDebugText before BrainContext.WriteLog checks debug, so debug-disabled NPCs still format strings. Gate text reads/formats timer state. Suppression gas and StatPair trickle still contain capturing delegates. NPC equipment repeatedly creates event/function LINQ views and equipment arrays; trace exposes both direct cell and Character.OnCharacterTick callers, requiring ownership/semantic audit before consolidation. Conclusion: physics broadphase and proportional cell/corpse copying are materially reduced, but total allocation/GC is ~8% higher in this particular workload. Native retained free space remains the largest excess-memory category; no navigation orphan evidence in validated scan. Prioritize guard-before-format behavior-tree diagnostics, concurrency-safe/versioned inventory views or due scheduling, NPC equipment/heartbeat iterator and closure removal, then allocation-bound ACG scratch reuse. No implementation or production intervention performed. Validation and evidence - Archive and inner dump/trace checksums passed; capture task completed with archived health checks and successful cleanup. Later independent SSH health recheck unavailable due agent refusal/timeouts. - Deep SOS reachability, native arena parser, pool/nav inventory, trace summaries/top stacks and custom allocation call-path attribution completed. The stock script final summarizer required offline DOTNET_ROLL_FORWARD=Major because net8 tool ran in SDK10; rerun passed. Old-format baseline makes some compare_capture.py fields n/a; comparisons above use original baseline outputs, not guessed values. - Evidence at /home/mathias/Downloads/pfe-memory-20261005-evening/analysis/: heap-summary.txt, live-dead-summary.txt, glibc-arena-summary.txt, native-ownership.txt, verified-roots.txt, current-world.txt, locator-counts.txt, trace-summary.txt, cpu-{inclusive,exclusive}-100.txt, allocation-paths.txt. Repository remains clean.
Checklist
2 of 2 complete- Verify deep managed/native ownership and world-size evidence; compare memory to earlier baseline.
- Compare matched-provider allocation/GC and managed wall-clock stacks; correlate deployed fixes and summarize limitations.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Research
- Task ID
tsk_dx96mv5x98- Updated