Rank primary Ithaca PFE true on-CPU costs
Analyze process-scoped 60-second perf cpu-clock capture tsk_sccd4kxghg on image b126bf974 (PR #2575/#2586), primary host PID 1646325 / container 4e68b8f07a9d. Evidence /home/mathias/Downloads/pfe-cpu-20261005-2244/. Linux5.15 lacks collect-linux user_events support; standalone matching perf binary extracted without package install or config change. Collected 99Hz software CPU-clock samples with frame-pointer callchains, concurrent 1s process CPU counters and a 65-second .NET JIT/loader EventPipe trace with rundown for address-to-method correlation. Acceptance: quantify mean/range actual CPU cores, validate sample counts/loss/coverage and user/kernel/JIT/native categories, resolve managed samples with contemporaneous method ranges, distinguish exclusive leaves from overlapping inclusive ownership, trace leading costs to deployed code, give focused CPU reduction priorities. State unwind/inlining/symbol limitations and non-simultaneous world-load counts from earlier heap (39 PFs/15639 live NPCs). Read-only offline analysis; no implementation/restart/deployment. Verified CPU findings - Actual process counters over62.077s bracketing the profile:165.05 user +8.18 system CPU seconds, average2.791 cores (2.659 user,0.132 system); 1s range2.327–3.865 cores. Perf event-count estimate164.09 CPU seconds over60s (~2.735 cores) agrees directionally. CPU percentages below are equal-frequency on-CPU sample shares, not wall-clock thread-time. Counter window brackets rather than exactly equals perf window; frequency adaptation and instrumentation mean approximate attribution, not exact saved cores. - 16,245 kernel cpu-clock samples spanning59.995s; perf reports zero lost. 12,716 leaves resolve to managed methods (78.28% of all samples;95.42% of JIT leaves),610 unresolved JIT leaves (3.76%),2,343 native leaves (14.42%),576 kernel leaves (3.55%). 15,896 samples have>=10 frames;61 only one frame.124,213 contemporaneous method ranges from JIT/rundown, exported using TraceLogOptions.KeepAllEvents=true; ordinary ETLX conversion drops bookkeeping method events and initially yielded only49 ranges, corrected before ranking. Range matching is address-based and does not reconstruct inlinees or historical tier lifetimes; frame-pointer native transition/inlining limitations apply. Native CLR1837 leaves/11.31% remain unsymbolized and cannot all be called GC. .NET BGC thread369 samples/2.27% is identifiable but not total GC CPU. Full native symbol enrichment blocked by later intermittent SSH failures; collection remains valid. Exclusive leaf ranking - Stat ConcurrentDictionary<StatIds,...>.TryGetValue11.32%, HeartbeatNpc7.44%, Indoor.HandleDoorStates5.93%, StatBlock.GetStat5.59%, stat dictionary enumerator.MoveNext4.49%, Physics.OnTick4.35%, HeartbeatCell3.20%. CLR native unresolved11.31%. Bepu Tree.GetOverlaps only0.215%; raycast0.357%; ACG RayHitsMesh0.289%. Allocation-heavy temporary-item/debug paths are not the largest resolved CPU leaves in this round. Inclusive ownership (overlapping, do not sum) - Heartbeat83.95%, PfSystemManager.OnTick71.97%, HeartbeatCell39.10%, HeartbeatNpc33.99%, OutdoorCell.OnTick23.16%, Indoor.HandleDoorStates19.49%, StatBlock.GetStat18.66%, IndoorCell.OnTick17.87%, HeartbeatPlayer11.19%, Selector.Execute8.80%, Physics.OnTick8.32%, stat Values5.36%. A one-off/background Leaderboard.CalculateHighScores query consumes4.845% inclusive; do not treat that automatically as persistent heartbeat work. Source-backed CPU priorities 1. Indoor door proximity is clearest first target. AOEmu.PlayfieldEngine/Systems/Playfield/Cells/IndoorCellManagementPfSystem.cs:HandleDoorStates snapshots all current character references once, then scans every character for every unlocked door each tick, checking both endpoints. The far-away branch still reads CharacterId for HashSet.Contains. Character.CharacterId getter calls GetStat(character_id), so nested loops induce stat dictionary lookup/custom FinalValue work. Among stat dictionary leaves,1139 samples/7.011% of ALL CPU have HandleDoorStates as first application owner; StatBlock.GetStat leaves add522/3.213% on that same path, distinct from door exclusive5.934%. Cache immutable per-tick ID/position values under correct ownership and reuse room/cell/spatial topology to consider nearby candidates rather than doors×all characters. Preserve two-sided doors, lock/open/close state, teleports and proximity teardown; do not trade the scan for stale memberships. 2. Dirty-stat propagation/heartbeat: NPC HeartbeatNpc scans npc.Stats.Values every active tick to find Changed, yet broadcasts only health/HasKnuBotData and clears Changed on all stats.537 stat-enumerator leaves/3.306% of total CPU belong to this NPC scan;191/1.176% belong to player CollectChangedStatBroadcast. Reuse existing StatPair Changed/dirty abstractions for explicit mutation-driven publication, preserving changed clearing and concurrent-owner semantics; avoid blindly replacing ConcurrentDictionary with unsafe mutable storage. NPC heartbeat itself7.44% exclusive can include inlined work, so no single-loop causal claim for all of it. 3. Remaining Physics.OnTick8.32% inclusive/4.35% exclusive is now the cheap full player/NPC scan + ID/position lookup, not the removed broadphase storm. Deployed equality guard skips unchanged ApplyDescription; Tree.GetOverlaps0.215% exclusive. Movement-only dirty publication/coalescing may remove the remaining scan, but requires safe owner/generation handoff with add/remove/teleport/disposal races. 4. Allocation follow-ups (debug guard-before-format, inventory snapshots, NPC equipment iterators) still sensible, but CPU ranking places door/stat scans first. ACG/pathfinding is not the leading CPU path in this sample. Highscores is a workload confound worth separating in the next comparable capture. Evidence and validation - /home/mathias/Downloads/pfe-cpu-20261005-2244/analysis/cpu-ranking.txt includes resolved exclusive/inclusive rankings and caller attribution; method-ranges.jsonl, resolved-samples.jsonl, jit-symbols source/log and rank-cpu.py make offline derivation reproducible. Native raw report has zero losses. Archive and inner checksums verified in capture task. Python syntax validation passed; engines repository remains clean. No production restart, kernel/runtime setting change, installation, debugger or implementation. - Capture image b126bf974, uptime~4h44m at22:46:50Z; earlier heap world counts39PFs/15639 live NPCs at22:03Z are reference context, not simultaneous activity/online counts. This is the first true on-CPU baseline and cannot be quantitatively compared to earlier wall-clock profile percentages.
Checklist
1 of 1 complete- Validate CPU counters, kernel sample quality and contemporaneous managed symbol resolution; rank exclusive/inclusive costs and deployed-source causes.
- Area
- Project Rubi-Ka
- Project
- engines
- Kind
- Research
- Task ID
tsk_nka9m9gvx8- Updated