Centralize engine error capture in MSSQL with admin Razor Pages

Outcome Capture errors from Playfield, Login, Chat, Market, PlayerPortal and other deployed server processes into existing MSSQL storage, with administrator and developer inspection in PRK.PlayerPortal. C# backend and server-rendered Razor Pages; no new error API, storage platform or dependency on Sentry. Implemented shared capture, Site DB persistence and admin Razor Pages; local acceptance validation passes. Production deployment is not authorized or performed. Workspace Implementation worktree /home/mathias/Projects/worktrees/engines-error-logging; feature/centralized-error-logging based on origin/develop 05d2bed25. Original checkout and unrelated navigation changes remain untouched. Implementation takes place only in this isolated worktree; no commits or deployment authorized. Repository evidence and reuse - Engines already use Microsoft.Extensions.Logging; shared ILoggerProvider is the proposed capture boundary. Preserve existing console/OpenTelemetry outputs. - Playfield LoggingHelper sends basic messages through ILogger but rich gameplay context only through Sentry. Migrate useful context into bounded structured logging scopes rather than serializing mutable entity graphs or raw packets. Audit Sentry-only captures and stringified exceptions in AOEmu.Net. - Reuse shared library boundaries, pooled context-factory/data access patterns and numbered SQL migrations. PlayerPortal already has direct per-server database access, server selection and Keycloak administrator claims; its broader staff visibility helper is not sufficient authorization for error pages. - Existing Bus.PublishWorker declares non-durable queues, lacks publisher confirms, catches publish failures without returning success, and ACKs consumer exceptions. Do not use it unchanged as reliable error transport or put logging payloads through its recursive logging paths. Agreed capture direction One shared logging provider captures Warning/Error/Critical, including messages without exceptions. Snapshot bounded safe metadata (UTC timestamp, engine, server/environment, process instance, release, category/EventId, message template/parameters, exception chain/stack, trace and allowlisted gameplay/request scope). No SQL/network I/O on logging caller threads. Each process uses a bounded background queue and batched writes with isolated database resources, capped retries/backoff and shutdown drain. Prefer this over a new broker collector initially; benchmark aggregate multi-process load before committing to topology. All engine and portal logs go into the existing Site DB. Engines require an explicit Site DB connection and restricted log-writing permissions, independent of their game database connection. Coordinate schema deployment with the Site DB rather than assuming the game-database migrator targets it. Volume and failure policy Group repeated signatures using stable category/template/exception/stack identity rather than formatted dynamic messages. Keep occurrence counts, first/last seen and bounded representative details; independent incidents must not disappear under storm control. Memory, field sizes, concurrency, retention and retry budgets must be explicit. Pipeline failures must use an independent fallback and metrics, never feed recursively into the provider. Queue overflow, prolonged DB outage and forced termination have observable, documented loss semantics. User accepts best-effort durability: bounded in-memory buffering only, no persistent disk spool or replay. Prefer engine availability over lossless capture; dropped counts must be observable. Small bounded retries must not inflate counts on ambiguous commits. Retain only 10 days of aggregate data, using time-bucketed counts so recurring signatures do not preserve old counts or samples indefinitely; delete expired buckets in bounded batches. Portal behavior Administrator or developer selects server and engine from retained-source dropdowns, then filters a paginated issue list by time/severity, and opens counts and representative exception/context details. Direct C# database queries and Razor HTML; no new API or SPA. Enforce authorization on every handler and prevent cross-server disclosure; encode all untrusted log content. Warning/Error/Critical entries are grouped with counts and bounded representative details; individual occurrences are not retained. Retention is 10 days. Acceptance criteria - All agreed deployed processes capture Warning/Error/Critical through a common provider, including exceptionless errors; important existing helper context and direct-capture paths are not silently lost or double-counted. - Callers do not wait for SQL/network work; buffering/writes/aggregation are bounded and measured under representative multi-engine load and repeated-error storms. - DB outage, retry ambiguity, overflow, poison payloads, startup/shutdown and forced crashes follow agreed observable durability/loss behavior without recursive logging or harming gameplay/request availability. - Existing Site DB schema has indexed, bounded admin queries and chunked 10-day retention cleanup for counts and representative samples. Retry ambiguity cannot silently double-count successfully persisted batches. - Administrator/developer-only Razor Pages expose useful grouped errors/details while excluding unauthorized roles, secrets and unsafe raw payloads. - Focused unit/integration/load tests, affected builds and authenticated browser authorization/filter/detail checks pass before completion. Confirmed decisions - Best-effort durability; bounded memory and explicit loss reporting, no durable disk buffering. - Capture Warning, Error and Critical with grouped counts and representative details, not every occurrence. - Centralize all sources in the existing Site DB; retain aggregate counts and samples for 10 days. No unresolved product questions. Defaults are 512 pending signature/hour groups, 512 groups/batch, two-second coalescing, two write attempts with one-second backoff, three-second SQL deadlines and shutdown drain. Counts coalesce before queueing. Retention uses hourly buckets and bounded 2,000-row chunks, with up to 100 chunks/ten seconds per minute. Verified implementation - Shared provider/snapshot/SqlClient writer live in PRK.Common/AOEmu.Common/Logging. Registered in Playfield, Login, Chat, Market, ClientWebEngine, PlayerPortal, legacy Portal and WebAPI. Existing outputs remain unchanged. Helper gameplay scopes and network exception objects are preserved; Sentry-only leaderboard failures now use ILogger via LoggingHelper. Exception messages/arbitrary values are deliberately excluded; safe context, types and bounded stacks remain. Legacy literal messages still require caller-side secret discipline and structured templates to avoid group fragmentation. - Site DB migration: PRK.PlayerPortal/Data/Migrations/0001-SiteErrorLogs.sql. Transactional batch receipts prevent retry double-counting; restricted writer/read/maintenance roles, indexed hourly aggregates and chunked retention are included. Portal runs budgeted cleanup. The migration is copied to build/publish output but never automatically applied to a game or production DB. - Administrator/developer-only /Admin/Errors list/detail Razor Pages provide source/severity/hour filters and pagination. No new production API. Shared field/table styling is reused. Invalid filters return 400; missing details 404; unavailable schema/connectivity gives safe 503 pages without a second navigation DB failure. - Docker validation (host has no dotnet): 21 focused tests pass in Release with TreatWarningsAsErrors, including real MSSQL concurrency, receipts, rollback/poison data, retention, restricted permissions, overflow, retries, context/redaction, independent console filters, six-provider and six-process mixed-severity storms. Six independent processes persisted 60,000 calls with no drops; capture took approximately 436–484 ms per 10,000-call worker, and startup/capture/write/shutdown completed in about 727 ms overall on this local fixture. These are isolated fixture measurements, not production gameplay latency or allocation guarantees. - Existing logging/network regression selection: 27 passed. Primary three-engine build passes with zero warnings/errors. Market, ClientWebEngine, PlayerPortal, legacy Portal and WebAPI builds pass; legacy Portal/WebAPI retain existing unrelated warnings. New capture/page unit tests are added to CI; workflow YAML parses and git diff --check passes. Remote CI/full solution suite were not run. - Real Chrome acceptance of compiled Razor pages with synthetic cookies and disposable MSSQL verified anonymous challenges, administrator and developer access plus player denials for list/details, grouped counts, filters, pagination, desktop/mobile layout and safe 503 outage pages. Literal script markup was rendered as text; raw HTML assertions confirmed encoding. Production Keycloak login was not tested because no credentials were supplied; existing claims/auth integration was reused unchanged. Follow-up validation after enabling developer access: 21 focused Release tests pass with warnings as errors; direct synthetic-cookie HTTP checks return 200 for developer and 403 for player. Server and engine are rendered as database-backed dropdowns (ERGO/ITHACA and server-specific engine options in the fixture), selected values persist, and filtered pagination returns the expected 50+5 rows. The user explicitly requested no additional Chrome run due current external browser-test slowness. - Disposable databases, browser container and temporary credential files are cleaned up. No commits, deployment or existing game/site DB migration performed. Rollout requirements Explicitly apply the numbered SQL migration to Site DB, provision restricted users and configure SITE_DB for each engine plus portal reader/maintenance permissions. Setup, limits, retention and test commands are documented in PRK.Common/AOEmu.Common/Logging/README.md and linked from PRK.PlayerPortal/README.md. Production identity-provider/deployment smoke checks remain rollout checks, not claimed local verification. No client/tool logging rollout is included. Pull request - #2568 — feature: Add centralized engine error logging - https://git.project-rk.com/prk/engines/pulls/2568 - feature/centralized-error-logging -> develop