Build tagged streaming-engine releases and isolated Gitea runner

Implement tagged CI/CD releases for streaming-engines, reusing relevant Project-RK release patterns in /home/mathias/Projects/engines/deployment and release-services.yml. Registry-only components are released by publishing images; cloud-router and cloud-dashboard also require controlled server deployment. Provision a new isolated Gitea runner on 5.9.138.8 without interfering with existing workloads. Acceptance criteria: - Explicit validated release-tag/component mapping and immutable image references. - All agreed deployable engines have working build/publish paths, including native/private dependency prerequisites. - Router/dashboard server rollout preserves persistent data and infrastructure, verifies health, serializes deployments, and documents retry/rollback constraints. - Dedicated repository-scoped runner with isolated execution and appropriately scoped credentials; existing workloads remain untouched. - Release tooling tests and configuration validation pass; live publication/deployment gates are recorded accurately. Confirmed decisions: individual component tags only (no grouped cloud tag); publish both versioned images and latest for existing box/server consumers; cloud-router and cloud-dashboard deploy independently to 204.168.249.25. Existing registry convention is cr.kaukus.no/kaukus/<service>. Initial linux/amd64 coverage matches existing images. Runner registration and dedicated deployment credentials are configured. Repository refreshed by clean fast-forward to merged backup commit 3f77533; work branch feature/tagged-engine-releases created, no source changes yet. Merged backup design uses Litestream with off-host Scaleway replication, health checks and a weekly isolated restore check. Integrate with the existing /home/deploy/router host stack rather than replacing its host-specific Compose/settings. Runner-host discovery succeeded as mathias with passwordless sudo: Ubuntu 24.04, 62 GiB RAM, 20 logical CPUs, Docker 29.2.1. Host already runs production GPU and Teams workers. Runner must not expose their Docker daemon/data to CI jobs. User verified/trusted the deployment host key; discovery now succeeds. Live Compose project router uses /home/deploy/router/docker-compose.yml and .env. cloud-router, cloud-dashboard and Litestream are healthy; weekly restore timer is installed. Rollouts must preserve that host-specific configuration and independently update only the tagged application service. Implementation and verified setup milestone: - Changes on feature/tagged-engine-releases add .gitea/workflows/release.yaml and deployment/{release.py,deploy.py,install-host.py,README.md,tests/,runner/}; README links release runbook. User authorized publication through /open-pr. Committed as 9212796 (feature: Add tagged engine releases and isolated runner), pushed with upstream, working tree clean. No release tag or application rollout performed. - Eleven independent image component prefixes: base, gst-build, gst-runtime, cascade, ruststreamer, local-streamer, local-panel, kaukus-provisioner, dhcp-server, cloud-router, cloud-dashboard. Stable component_vX.Y.Z tags publish X.Y.Z and latest. Initial linux/amd64 matches existing Dockerfiles/NDI. No grouped tag; local-provisioning-web is a local test tool without Dockerfile. Existing shared bases remain latest (not fully reproducible builds). - Release helper validates manifests/tags, reuses matching published version, refuses changed identities and superseded latest, promotes latest only after cloud rollout (immediately after publication for registry-only components). Existing private dependency secret remains in use. User configured STREAMING_REGISTRY_USERNAME and STREAMING_REGISTRY_PASSWORD; secret names/presence verified, values not read. - Cloud helper installed under /usr/local/libexec/streaming-deploy on 204.168.249.25. Dedicated streaming-deploy user has a forced SSH command and sudo permission only for helper with no arguments. No app Compose/config/volume/container replacement. Preflight as the restricted user passed strict public HTML + /health=ok checks and remote Litestream sync. Extra sudo arguments, unknown component and oversized input were rejected. SSH authentication using the generated CI key has not been acceptance-tested. - Rollouts pin digest overrides separately from existing host Compose, lock the whole cloud stack, update only selected service with --no-deps, check image identity/health/public endpoints, and sync backups before/after router. Per-component applying/failed journals block all cloud rollouts until explicit operator reconciliation. No automatic old-binary rollback after startup migrations. Local-only reconciliation accepts only candidate or predecessor; explicit rollback requires schema compatibility. - Dedicated repository-scoped runner streaming-release-gpu (Gitea runner ID50, v5.2.0 checksum pinned) is online at 5.9.138.8, sole label streaming-release, capacity1. Systemd streaming-build uses rootless Docker as streaming-build with separate data/socket and aggregate 24GiB/no-swap/8CPU cap; streaming-runner uses separate account with 2GiB/2CPU cap. Neither has production Docker group or sudo. Rootless socket uses --group=0 to map to build-user group; cgroupfs avoids absent systemd-user cgroup. Runner registration is private to orchestration account. Cache disabled, metrics localhost9108. Host production GPU and Teams containers remained running. - Real gitea-runner exec isolation smoke passed tooling, production Docker socket denial, runner-registration denial, nested BuildKit build+run. This was local workflow execution on remote runner, not a Gitea-dispatched release. Rootless daemon exposes its own socket directly because nonroot runner cannot chown proxy socket; tested boundary remains OS user separation, not proxy filtering or VM isolation. Storage shares host filesystem; free-space admission guard50GiB is not a disk quota. - 21 Python tests pass. Both reviewed rollout defects were reproduced red (failed newer candidate allowing old binary retry; SPA HTML accepted at /health), fixed and covered green; explicit reconciliation candidate/predecessor boundaries also tested. Python/YAML parsing and git diff whitespace pass. Local systemd verification cannot resolve Ubuntu-specific executables; remote services run on target. Full remote Actions checkout/artifact/private-dependency/registry publication and actual cloud rollout still unverified. Remaining blockers/decisions: - Registry publishing secrets are present (user supplied); authentication/push access remains unverified until the real Actions acceptance run. - User confirmed release rights for anyone who can push to main. Main currently has no branch protections; repository code-write teams are Owners and Developers, with no direct collaborators. Created and verified *_v* tag protection allowing those two teams (Reviewer has only code-read and is excluded). Keep the protection aligned if future main policies/permissions change. - Dedicated deployment Ed25519 key was generated and its public half installed; private/public files are preserved outside the repository at /home/mathias/.local/state/streaming-release-bootstrap/deploy-key{,.pub} (0600 private; parent0700). No secret values in chat/repo. STREAMING_DEPLOY_SSH_KEY and STREAMING_DEPLOY_KNOWN_HOSTS are now configured and their presence verified. Known-hosts uses the user's explicitly trusted ED25519 key, not an unverified scan. Key authentication and registry pull access on cloud host still need live release validation. - Draft PR #128 opened against main: https://git.kaukus.no/Kaukus/streaming-engines/pulls/128 (head feature/tagged-engine-releases, commit 9212796). Complete task only after release acceptance gates pass. Publication validation: all 21 Python tests rerun successfully, Python AST/YAML parsing and complete branch whitespace checks passed. PR is explicitly draft pending end-to-end release acceptance; no claim of remote CI success or application deployment. User follow-up implemented and pushed to existing draft PR128 as 72d11de (ci: Install recording test codecs and verify runtime). CI Test job now installs GStreamer tools/base/good/bad/ugly/libav plus FFmpeg, preflights the source/converter/encoder/parser/muxer factories and ffprobe, then runs the unchanged full workspace test suite. No test skips, replacement codecs, application-code changes or dependency manifests added. Verified: the exact recording/EOS regression passes locally; cargo fmt --all -- --check, cargo check --workspace and cargo test --workspace all pass, along with21 release tests and branch whitespace checks. Clean Debian bookworm-slim reproduced missing x264enc; the exact YAML setup step made every plugin check pass and produced an H264/AAC MP4 validated by ffprobe. Evidence logs: /tmp/streaming-pr128-workspace-tests.log, /tmp/streaming-pr128-workspace-check.log, /tmp/streaming-pr128-ci-runtime-smoke.log. Working tree clean at72d11de; PR summary/testing updated. Remote reruns3906/3907 are queued/in_progress, not yet confirmed green. PR128 merged as4cf679b378bf9b42eba0621eb12ce2443fd5cb83. User now authorizes initial router/dashboard tags and real production releases. PR CI3906 concluded success, including Check/Test. Main refreshed cleanly; no existing component release tags (only legacy next). Use initial cloud-router_v0.1.0 and cloud-dashboard_v0.1.0 matching workspace0.1.0, both on4cf679b, router first then dashboard. Production helper preflight including Litestream sync and public health passed before tagging. Published annotated cloud-router_v0.1.0 at4cf679b; Actions3909 ran on streaming-release-gpu and failed before any build/deployment. Checkout, the original21 tooling tests, private-key agent setup, registry login, and BuildKit startup worked. Failure: Harbor reports an absent version as `unknown: artifact kaukus/cloud-router:0.1.0 not found`, which inspect_image did not recognize. No image publication or application replacement occurred; dashboard tag deliberately not created. Reproduced the exact authenticated Harbor error locally using the existing Docker client's configured auth read-only (no credentials copied to CI or exposed). On dedicated fix/recognize-harbor-missing-release-images, patch adds exact requested-artifact missing-response recognition and two regression tests. User approved committing/publishing the fix; committed and pushed as dd9d29a (fix: Recognize missing Harbor release images). Working tree clean; all23 tests rerun successfully. New missing-Harbor test failed red before the fix; all23 tests pass after. Real Harbor read-only checks using the patched helper now classify absent0.1.0 correctly and successfully inspect existing latest digest5fa39a3c49ca4cb2cac53e6ab1fdf4b6db28f6466a98b2158d8bffeae46ffcf5. Whitespace checks pass. Fix PR #129 opened against main: https://git.kaukus.no/Kaukus/streaming-engines/pulls/129. User confirms fix merged; PR129 merge verified at d89ad8ea85f03569dcabb3b6efabd708b817b3c8. Clean main fast-forwarded. Production preflight passed again. Annotated cloud-router_v0.1.1 created at that exact merged commit and pushed. Actions3915 built and published the image successfully (5min fresh Rust build), then authenticated via the dedicated CI SSH key and reached the restricted cloud helper. Digest: sha256:2fd99cea18ab3822734553754bbbb7e9d27e403eabd69bfeada326c277529951. Artifact upload also succeeded. Workflow failed at the host's initial docker pull with `authorization failed: no basic auth credentials`; root Docker registry login is missing on204.168.249.25. Latest promotion was skipped; production app, dashboard, Litestream and NATS containers remain unchanged and healthy. Failure occurred before image overrides/attempt state or container replacement, so no reconciliation is needed. User confirms root registry login configured. Actions3915 rerun requested successfully (attempt2) on unchanged cloud-router_v0.1.1, reusing the published immutable version. Router attempt2 completed successfully in37s: https://git.kaukus.no/Kaukus/streaming-engines/actions/runs/3915. Existing version was reused without rebuilding; dedicated SSH authentication, root registry pull, backup sync, digest-pinned rollout, exact public health, latest promotion and manifest artifact upload all passed. Host current manifest records cloud-router0.1.1 at d89ad8e and digest2fd99cea18ab3822734553754bbbb7e9d27e403eabd69bfeada326c277529951. Never move tags. After router verification, annotated cloud-dashboard_v0.1.0 was pushed at the same d89ad8ea85f03569dcabb3b6efabd708b817b3c8 commit. Dashboard build/publish/rollout/latest/artifact upload succeeded in49s: https://git.kaukus.no/Kaukus/streaming-engines/actions/runs/3916. Host current manifest records cloud-dashboard0.1.0 and digest2be65064cabfe1a15eb1be5866df371b51231e5d6666c5006649660e315d0346. Release log: /tmp/streaming-router-release-3915.log; root-only host failure log /var/lib/streaming-deploy/command-failure.log contains the exact pull denial. Never move cloud-router_v0.1.0. Final independent read-only verification passed: both running containers are healthy, pinned to their recorded digests and expected source revision; each registry version tag and latest resolves to its corresponding host digest. Router start09:23:25Z remained unchanged during dashboard rollout09:24:40Z. Litestream and NATS retained their original starts, confirming no infrastructure recreation. Final production preflight passed including public HTML, exact /health=ok and live Litestream remote sync. Working tree clean on main. Successful release logs are /tmp/streaming-router-release-3915-attempt2.log and /tmp/streaming-dashboard-release-3916.log. Core system acceptance is now verified end-to-end for both cloud services and first failed-run retry/immutable-image reuse. Registry-only/native/base components retain tested shared release selection/build paths but were not individually released in this acceptance run; no claim of eleven live image builds, rollback execution, browser-authenticated UI acceptance, VM isolation, disk quotas or byte-reproducible shared-base builds.