Add Murmur worker observability and diagnostics

Add safe operational insight into ephemeral Murmur GPU workers without requiring interactive SSH or weakening the worker security group. Initial scope: periodic worker usage reports and central Prometheus exposition. - Workers send bounded, authenticated reports to the control plane over HTTPS, approximately every 30 seconds. Reporting must remain independent of model loading and transcription and must not affect job success or automatic deletion. - Proposed first-pass measurements: GPU utilization/VRAM, CPU/RAM, readiness/busy state, model-load time, transcription counts and processing time. Missing GPU telemetry must be distinguishable from zero utilization. - Prometheus scrapes the control plane, never ephemeral workers. Preserve the deny-by-default worker firewall. - Authenticate report origin and bind reports to active workers; reject malformed/oversized reports. Do not include tokens, audio URLs, prompts, transcript content or job IDs in metrics. - Retain only the latest worker snapshot in the control plane, expose report freshness, and expire stale/retired-worker measurements. Prometheus owns historical retention. Keep metric labels low-cardinality. - Prometheus will scrape over public HTTPS with a dedicated 32+ character bearer token, separate from job authentication and worker reporting credentials. Enable reporting explicitly in configuration after provisioning the scrape credential; do not alter live production configuration automatically. - Provision each instance with a reporting token bound to worker ID and provider instance ID. Derive it from a fresh control-plane-only signing key, not the shared transcription token or scrape token. The existing clean-start/drain requirement makes this key safely process-local. - Latest usage measurements expire after three reporting intervals; preserve freshness/receipt metadata for non-retiring workers. Counters are per worker process and best-effort, not billing/audit totals. Worker logs and provider/image-pull timings remain deferred. Acceptance criteria for the initial slice: - Reports continue during startup/model loading and transcription, at the configured interval with bounded request timeouts and no unbounded retry queue. - The authenticated report endpoint validates worker identity and schema, and stale/deleted-worker reports cannot keep usage measurements alive indefinitely. - Prometheus exposition has valid names/types/units, explicit freshness, and no sensitive or unbounded job-derived labels. - Tests cover sampling failures, reporting failures, invalid authentication/payloads, freshness/expiry, and exposition. Relevant Rust/Python validation passes. - Configuration and deployment documentation explain HTTPS reporting, scrape access and retention; no production networking is weakened implicitly. Verified initial-slice outcome: - Implemented opt-in 30-second HTTPS reporting, instance-scoped credentials, bounded ingestion, inventory validation, report ordering, per-process counters, freshness and stale/retired-series cleanup. The worker starts reporting before model loading and continues during transcription without affecting job outcomes or cleanup. - Added dedicated bearer-authenticated /metrics, narrow public Caddy routes, configuration examples and Prometheus/retention/rotation documentation. At the user's request, a random scrape credential was provisioned locally in /home/mathias/.config/murmur/prometheus-scrape-token (mode 0600), transferred without disclosure into the production server's root-owned mode-0600 /etc/murmur/runtime.env, and verified to match. Production configuration on feature/worker-usage-metrics now has observability.enabled = true and passes validate-config. The last verified running release remains murmur_v0.1.7 with reporting disabled. Commit bdea017 is pushed on feature/worker-usage-metrics and PR https://git.kaukus.no/Kaukus/murmur/pulls/12 is open against main. The working tree is clean. No services were restarted or deployed; the user explicitly owns deployment. - Added scope: activate reporting in production and verify public HTTPS metrics access. The user requested a PR-only handoff and will deploy. Awaiting that rollout and production verification; enabling repository configuration alone does not activate the old running version. The host environment checksum differs from the current runtime record as expected after adding the credential; normal release/configure must reconcile it. - Validation passed: Rust formatting/check/strict Clippy and all 80 Rust tests, including all disposable-PostgreSQL tests; 25 worker tests with Python 3.12 plus Ruff and Pyright; 23 deployment tests including a real pinned-Caddy routing regression; Compose rendering and diff whitespace checks. - A temporary harness using the actual modules verified Python Reporter -> certificate-validated local HTTPS -> production Caddy routes -> Rust ingestion -> Prometheus parser (18 metric families), including auth separation, body limits, missing GPU telemetry, retirement cleanup and unchanged API/ready restrictions. This caught and fixed the Caddy matcher needing two identity segments. - Worker validation used the pinned API/test dependencies in an isolated Python environment, not a full frozen WhisperX/GPU environment. Real-GPU sampling and production HTTPS deployment remain unverified. Independent static reviews were partial and are not a blanket approval. Later diagnostics scope, not part of the initial metrics slice: - Surface provisioning, image-pull, readiness and teardown timings centrally. - Collect bounded, access-controlled and redacted worker logs without exposing worker tokens, source URLs or transcript content. - Expose sufficient model/profile/runtime details to diagnose compatibility failures; define log retention before collecting logs.