Investigate production PFE memory growth using Ithaca heap dump

Analyze the verified 2026-10-03 Ithaca PFE heap dump at /home/mathias/Downloads/pfe-memory-20261003/ithaca-playfield-20261003T1645Z.heap.dmp (~24 GiB process RSS, ~27-hour uptime). Capture task tsk_wn8ck8pxzj. Identify dominant managed allocations, retained object graphs and root chains; distinguish reachable retention, dead allocations, heap fragmentation/reservation and native memory where dump permits. Correlate evidence with deployed commit d39d35493c334ebd4eee320bde8ff5193016fbe6 and prior disposal investigations. Read-only investigation; do not modify concurrent client work or restart production. Acceptance: quantified leading memory consumers and evidence-backed cause(s), explicit unknowns, and concrete next fix/validation recommendations. Verified initial dump findings - .NET 8 workstation GC: allocated heap 3,869,116,952 bytes; committed heap 3,922,305,024 bytes (3.653 GiB). Object walk 48,648,782 objects / 3,860,364,152 bytes including 67,982,376 bytes free. Managed heap cannot explain ~24 GiB RSS. - ELF-native ptmalloc heap inspection identifies 358 glibc secondary heaps / 139 distinct arenas, totaling 21,553,782,784 bytes (20.074 GiB). Independent sum of each malloc_state.system_mem matches the heap_info size sum exactly. Chunk walk classifies 17.720 GiB as free, 2.353 GiB in-use-marked, only 3,504 bytes unclassified plus arena headers. Free chunks include 4.251 GiB in top chunks and 13.469 GiB interior holes. Fastbin/tcache chunks may remain in-use-marked, so free total is conservative. These are captured allocator virtual regions, not per-region RSS measurements. - 39 Bepu BufferPool objects own 825,622,528 bytes (0.769 GiB) of native blocks, calculated from SOS-confirmed PowerPool layout, BlockCount and BlockSize. They do not explain the excess. 39 registered local PF dictionary nodes; 29 normal PF plus 14 mission PF objects (not all proven reachable). - Dominant managed objects include 6.07 million StatPair objects (~486 MB), 6.08 million stat dictionary nodes (~243 MB), arrays and normal cached game data. No claim yet of reachable managed leak. - Deployed Program.cs:116 sets ThreadPool.SetMinThreads(1000,1000), a plausible arena/churn amplifier, not a proven allocation source; min threads is not evidence of 1,000 actual threads. Current production container shows no MALLOC_*, GLIBC_TUNABLES, or GC overrides in Docker configuration. - Evidence/scripts/raw SOS logs saved outside repository at /home/mathias/Downloads/pfe-memory-20261003/analysis/. No production or repository changes made. Confirmed offline root cause Deployed PathfindingPfSystem.Cleanup uses dtwFree(NavmeshQuery) at line 534 and dtwFree(Navmesh) at line 539. The exact RecastSharp 0.1.3 native library proves dtwFree delegates directly to dtFree/raw free and skips both C++ destructors. dtFreeNavMeshQuery and dtFreeNavMesh are the destructor-aware APIs that release child pools, tile tables and owned tile blobs. Current metrics report logical free operations, not release of those children. Library identity and verification - Local NuGet libRecastWrapper.so SHA256 399ef1ebf0c571e191747bc6acb64726d4740b633f8d04146fb2fc37c127ee61 matches the production binary. Local Pyroscope 0.13.0 binary also matches production; profiler presence is not proof it caused this growth. - GDB DWARF type layouts from that exact library and SOS field offsets drive an offline allocator inventory. There are 43 PathfindingPfSystem objects with 37 nonzero native mesh/query pointers (including potentially unreachable managed objects, so this conservatively overcounts owned buffers). - 1,497 validated dtMeshTile arrays: 37 have allocated dtNavMesh parents and current managed owners; 1,460 have NO allocated dtNavMesh parent or managed pathfinding owner. Validate every populated tile's native magic/version, flags, pointer and blob size against allocated chunks. Detached tile arrays and 85,373 tile blobs total 929,421,120 bytes. - Also identify 1,377 detached 2,048-node pools and 1,266 detached 64-node pools with validated child allocation sizes. Combined conservative detached Detour inventory: 97,405 allocated chunks / 1,018,199,520 bytes (~971 MiB). Excludes unclassified lookup tables/open lists and other native allocations. - Detached buffers occur in 311/358 secondary heaps across 120 arenas. Those heaps contain 16.212 GiB of free regions. 11.650 GiB of free regions directly border detached Detour allocations. Sixteen heaps have no in-use-marked chunks except validated orphan Detour buffers, surrounding 0.743 GiB free. Example: a single leaked 48-byte node-pool chunk at 0x7f04cadee850 is the only allocated chunk in a ~46 MiB secondary heap. These are structural fragmentation relationships, not a promise that fixing cleanup immediately releases every byte of RSS. - Local ctypes reproduction invokes the exact binary and tracks its allocator callbacks: 20 empty mesh/query create/init/dispose cycles with deployed generic cleanup leak 240 allocations / 3,789,600 bytes; destructor-aware cleanup leaks zero allocations / zero bytes. This reproduction does not create tiles, so real zone loads add leaked tile storage. - Related deployed failure paths: outdoor AddTile failure continues without freeing unowned tile data; ACG initialization/tile failures return zero handles without releasing partially allocated mesh/query/tile state. They require a cleanup audit; this dump does not quantify their individual incidence. Scope decision and next actions User explicitly prefers offline causal analysis over disruptive live tests. No production experiments, restart, trim calls or settings changes. Investigation and minimal reproduction run locally outside the repository; remote access limited to read-only binary checksums/config inspection. Prior arena-limit suggestion is secondary, not the proposed first fix. Repair destructor-aware ownership through a portable native wrapper/package API, and make partial-init/AddTile failure paths release untransferred buffers. Validate repeated normal/failure/mission teardown locally with native allocation accounting. Managed GC tuning cannot fix leaked native children. Remaining uncertainty: exact historical allocation mix and how much of the mapped free memory would be returned automatically after fixing leaks. Final root-chain review passed: SOS gcroot -nostacks for representative current PathfindingPfSystem 0x7f0c8f669888 reports five unique managed roots through the playfield/PfSystemManager systems list and engine infrastructure; native field values match validated current dtNavMesh and query allocations. Investigation acceptance criteria met: quantified footprint, concrete deployed cleanup bug verified against identical native library and reproduced offline, orphan-buffer fragmentation relationship, remaining uncertainty and repair/validation recommendation. Research complete; implementation is not performed and requires separate follow-up scope. Evidence files: detour-inventory.txt, fragmentation-owners.txt, detour-cleanup-repro.txt, detour-layouts.txt, nav-system-layout.txt and nav-system-root.txt under the existing local analysis directory.