Deploy PFE Prometheus and Grafana configuration to Skyshard2

Deploy the production-only observability configuration from PR #2596 branch feature/pfe-performance-observability to ssh mathias@skyshard2, replacing the existing Prometheus/Grafana configuration while preserving established OIDC, retained data and unrelated Loki/Tempo/Pyroscope services. User explicitly authorizes this deployment; Ergo engine rollout is separate and handled by user. Preflight 2026-10-06T18:18Z: host prk-skyshard-02, 196GiB disk available,44GiB RAM available, no swap. Docker requires passwordless sudo. Existing compose project grafana-stack at /root/Grafana-Stack: Grafana/Prometheus/Loki/Pyroscope up~7h, Tempo crash-looping. Grafana public https://stats.project-rk.com redirects to Keycloak realm ProjectRubika client GrafanaOAuthConnection. OIDC is stored in persistent grafana.db sso_setting (generic_oauth enabled, PKCE, strict role mapping) rather than compose env; preserve grafana-stack_grafana_data. Existing Prometheus/Grafana/Loki/Tempo/Pyroscope named volumes must be preserved. Existing Grafana/Prometheus/Loki ports bind all interfaces; new files should keep public routing compatibility and not break existing OIDC or dependent engine OTLP/Pyroscope endpoints. Scope: inventory actual metrics routes/ports and target reachability; create timestamped verified backup of compose/configs and relevant persistent-data metadata (plus Grafana DB/OIDC and Prometheus snapshot if practical without disruptive downtime); merge PFE dashboards/rules/provisioning/targets into existing stack rather than deleting Tempo/Pyroscope/Loki; pin images; controlled recreate only changed services; validate OIDC redirect, datasource health, targets/rules/dashboard provisioning and retained history. Keep PFE target failures expected until matching engine telemetry is deployed, but configure currently reachable legacy /metrics endpoints correctly. No engine deployment, no PFE restart, no production data deletion, no OAuth secret exposure. Deployment milestone 2026-10-06T18:32Z - Real targets verified from Skyshard2: Ithaca PFEs10.144.92.85:9090-9093, Ergo PFEs public9090-9091, Ithaca node9100, MSSQL4000, Ithaca Market9094, Ergo Market9092. All12 configured targets currently healthy. PFEs remain legacy-only until engine deployment. - Backup /home/mathias/observability-backups/20261006T182855Z: checksummed original config tar, online-safe Grafana SQLite backup containing OIDC, redacted-sensitive container inspect protected0600, and17GiB Prometheus rollback snapshot using hardlinked immutable blocks plus copied mutable WAL/head. Existing named volumes retained. - Merged into existing grafana-stack project rather than replacing data services. Preserved current Grafana13.2.3, Prometheus3.15.0, Loki3.7.8 and Pyroscope image digests/volumes; added node-exporter1.9.1. Removed crash-looping Tempo container; Tempo data volume preserved. Removed dead PFE trace-export env in merged engine branch for future deploy. No Loki/Pyroscope data deletion. - Prometheus uses15s scrapes/30s rules,15d existing retention, same data volume/history; jobs: six PFEs, local+Ithaca node exporters, MSSQL, two Markets and Prometheus. TSDB retained historical pf metrics (6h count evidence), corruption count zero; existing37k-head series increased as extra Ergo targets added. Both PFE rule groups/21rules healthy with ~2ms evaluation. All targets healthy. - Grafana persistent OIDC sso_setting remains generic_oauth enabled for Keycloak ProjectRubika/GrafanaOAuthConnection, user confirms login works. Existing6dashboards/datasources retained; pfe-prometheus/pfe-loki inserted and five PFE dashboards provisioned in Grafana13 unified resource storage. Temporary probe file removed and resource action=2 deleted tombstone; five real dashboards action=1. Provisioning confusion was due checking obsolete legacy dashboard table, not rejected dashboards. Added missing empty provisioning dirs to silence startup errors; anonymous auth remains off through existing OIDC deployment. - Tempo not needed per user; service removed. Its volume is explicitly preserved for later decision. Ergo post-deploy verification - Both targets switched to revision f6a8db478273a9f30c38ebe9bcab40917706ee5c and expose new metrics with distinct roles playfield1/pfe-1 and playfield2/pfe-2. Deployed server_tag is title-case Ergo; Grafana discovers values dynamically and production queries/rules work. Snapshot ages ~6-8s, registration rejects0, overdue instances0. Runtime/process/cgroup/PSI/navigation collectors available; allocator collector unavailable as intentionally disabled. - Recording rules populated. Initial5m outdoor baseline: pfe-1 CPU~0.16cores, attainment~0.91, overrun~3.1%, p99~87.6ms, allocations~5.9MiB/s, GC pause~0.54%; pfe-2~0.09cores, attainment~1.01, overrun~0.18%, p99~3.9ms, allocations~1.2MiB/s, GC pause~0.09%. Session queues zero and no overdue instances. Workload differs; values are observations, not pre/post regression claims. - Loki receives new deployment_environment=Ergo, engine_name and engine_role structured metadata. Verified SlowTick info record for pfe-1:71ms total,69.6ms OutdoorCellManagement,~1.3ms Physics,~3ms overlapping process GC pause with sampled character/async/nav/physics waits. This validates actionable slow-tick transport and attribution. Dashboard LogQL can filter engine_role as structured metadata even though it is not an indexed label. - User confirms public Keycloak OIDC works after cutover. Grafana13 unified dashboard resources show five real PFE dashboards action=1; temporary probe action=2 deleted tombstone and file absent. Earlier legacy dashboard-table query was invalid for Grafana13 unified storage; dashboards were never rejected. - All Skyshard services healthy: Grafana, Prometheus, Loki, Pyroscope and node exporter. Tempo removed; preserved Tempo volume remains. Existing history and six legacy dashboards retained.