Set up Murmur CI/CD and safe image deployments

Set up automated validation, image publication and safe production deployment for Murmur, reusing the existing streaming-engines release model where appropriate and comparing Project-RK engine deployment conventions. Verified starting point: - Kaukus/murmur main has no CI/release workflows. Rust control-plane and Python 3.12 GPU worker Dockerfiles exist. - Production is managed by Kaukus/Ansible under /opt/murmur: PostgreSQL, control-plane and Caddy. The control image is private Harbor (cr.kaukus.no/kaukus/murmur-control-plane); ephemeral workers pull a public Scaleway image by digest. - Streaming releases use independent component tags, immutable OCI-labeled artifacts, registry build caches, a rootless isolated release runner, bounded manifests and forced-command SSH deployments. - The local Project-RK engine checkout was stale. Fetched origin/develop at 8f0e3bc991c314f5ce55272f0adc0c19a2a1bf8f without changing its working tree; the remote main branch no longer exists. Project-RK replaced Kamal in #2748 and removed obsolete Kamal files in #2766. Current .github/workflows/release-services.yml builds cached digest-pinned images and uses host-local Python/Docker Compose rollout. Develop auto-deploys Ergo; protected engines_v*/portal_v*/workflow_v* tags release independent Ithaca scopes. The host serializes deployments, rejects superseded releases, records rollout stages, verifies exact readiness identity, drains without SIGKILL and applies game migrations once before staged startup. - Murmur startup applies PostgreSQL migrations and requeues running jobs. Scheduler provisioning runs in background tasks. A release must not interrupt transcription/provisioning or assume automatic binary rollback is schema-safe. - Ansible currently owns both image pins and worker_image in config.toml. Its normal convergence must not overwrite a CI-selected release; release-state ownership must be resolved before deployment. Approved implementation design (user requested implementation): Release model and CI: - Run CI on pull requests and main. Production releases use protected control-plane_vX.Y.Z and worker_vX.Y.Z tags on approved main commits; main itself does not deploy production. - Keep independently versioned linux/amd64 images in the existing private Harbor control repository and public Scaleway worker repository. Enforce immutable version tags, verify OCI source/revision/component/version and platform, and reuse matching published versions on reruns. Deploy digests, never latest. Retain release artifacts and the selected control/worker pair; reject stale, conflicting and incompatible promotions under the host lock. - Each tag validates its exact checked-out revision before publication/deployment. Gates include Rust format/locked check/test/strict Clippy, worker Python 3.12 frozen dependency sync/strict typing/lint/tests, deployment-tooling tests, Compose rendering and PostgreSQL migration/upgrade integration tests. Ordinary CI uses mocked workers and no GPU/cloud/deploy credentials. - Use registry BuildKit caches and the worker's existing independent model stages. Pin remaining model revisions and image bases, keep secrets out of layers/cache, and do not pull the huge GPU image onto the small control host. The manifest records both selected artifacts plus schema/protocol compatibility; coordinated breaking changes require an explicit tested pair promotion, not an incompatible intermediate rollout. Runner and deployment boundary: - Follow the current engines repository directly instead of building a separate deployment platform: tag-triggered workflow in the application repository, immutable image build, versioned deployment-directory upload over pinned SSH and host-local Python/Compose rollout. Keep release authority restricted through protected tags/branches and repository secrets. Reuse existing runner operations rather than shipping a runner installer in Murmur. - Use separate scoped registry publishers and a dedicated forced-command SSH key with verified known_hosts. Application deployment/bootstrap tooling must come from the Murmur release system rather than the Ansible application role. CI submits only a bounded strict release manifest to fixed repositories/project/services; it cannot upload executable deployment code, choose host paths, run arbitrary commands, access app secrets or use operator rollback arguments. - PostgreSQL, Scaleway and worker credentials remain on the host; Kaukus OAuth client credentials never enter this release system. Protect release tags and release-secret access; no deployment secrets on pull-request jobs. Release authority remains a trusted-code boundary, not protection against malicious authorized maintainers. Application lifecycle prerequisite: - Add a durable maintenance/dispatch gate and protected host-local status/drain/resume control. Entering drain must be serialized with job claiming and provisioning, not implemented as a racy database count. Continue accepting durable queued submissions and serving status while draining; briefly unavailable API calls during replacement use existing Kaukus retries. - Track and await transcription, provisioning, deletion and reconciliation operations. Drain stops new dispatch/provisioning, waits for results to commit and provider operations to settle, and verifies database/provider consistency. Persist provisioning intent/provider IDs as early as allocation occurs and reconcile tagged server orphans; current detached-IP/volume cleanup alone does not cover this requirement. Reuse the existing worker-lifecycle hardening work instead of a second cleanup path. - Version one uses a single scheduler, no blue/green control planes. Drain and retire remaining workers before replacement so no old-image cohort survives promotion. Both control and worker releases use the same safe control-plane restart path because worker_image is currently captured at startup; do not add hot reload just for deployment. - Bound the helper's drain wait, never escalate to SIGKILL, and leave unresolved work alive/maintenance state explicit on timeout. SIGTERM must use the same tracked shutdown path. Startup remains paused while a deployment is unresolved; do not blindly requeue uncertain in-flight work after an interrupted rollout. - Preserve public /health compatibility; add meaningful DB/schema/runtime readiness and a protected release identity/status response. Worker readiness includes revision, protocol and existing runtime-profile identity. Routine checks do not require provisioning a GPU. Migration, backups and rollout: - Refactor migration execution into an explicit migrate-only mode with no OAuth discovery, scheduler, recovery or cloud calls. Normal serve checks schema compatibility rather than applying migrations implicitly. Use transactional SQLx migrations and a migration lock; prefer expand/contract schema changes. Database upgrades, Caddy upgrades and destructive migrations are separate operations, not automatic image-release side effects. - Add encrypted off-host PostgreSQL backups: consistent daily logical backups plus a pre-migration snapshot, verified upload/checksum and a tested restore into an isolated database. Recommended baseline retention is seven daily and four weekly backups plus release snapshots; the selected storage is the murmur-backup bucket in fr-par; key custody and off-host restore remain activation gates. Daily logical backups imply up to a 24-hour RPO between releases; tighter recovery requires a deliberate WAL/PITR decision. Keep secrets, source URLs and transcripts out of release artifacts and shared logs. - Rollout: validate manifest/current pair and pull/verify the control artifact before disrupting work; acquire/persist the deployment attempt; enter durable drain; wait and retire workers; stop only control-plane without forced kill; create/verify the final pre-migration backup; run candidate migrate-only if needed; atomically apply image pins/release settings; start candidate with dispatch paused; verify exact image/revision, readiness, HTTPS and auth/network boundaries; validate a worker candidate if applicable; record the new release and resume scheduling only after all required gates pass. PostgreSQL/Caddy/volumes stay running. - Worker promotion includes a bounded one-L4 fixture validation using normal provision/transcribe/delete helpers under maintenance, not cloud credentials in CI or a parallel provisioning implementation. Verify nb/nn/da readiness/output contracts and cleanup of server/IP/volume/DB records. Persist validation resource IDs and enforce an explicit runtime/cost cap. L4 capacity shortage is a retryable validation blocker, not permission to use unvalidated hardware; ordinary control-only releases allocate no GPU. Application/server ownership correction requested by the user: - Ansible is for server management only: OS packages, Docker, accounts, SSH/firewall and baseline host maintenance. It must not own Murmur application Compose/configuration/secrets, image selection, application rollout or migrations. - Match the current engines deployment model: the Murmur repository owns versioned deployment bundles and Compose/configuration; CI builds/publishes digest-pinned images, delivers the approved release bundle and invokes the host-local Python/Compose rollout. Secrets are root-owned host-local configuration, supplied directly by the user, not an Ansible Vault requirement. Preserve the existing persistent database/certificate volumes and release safety gates while removing application Ansible ownership. - Cleanup completed after the user rejected the overengineered first implementation. Production tooling now matches the engines shape: deployment/release.py, deployment/deploy.py and deployment/upload.sh, plus repository-owned Compose/Caddy/config and one test module. Removed the relay, separate automation repository, custom runner/receiver/installer/maintenance framework and every application-management change/worktree in Ansible. The existing unrelated Ansible checkout remains untouched. - Every Compose invocation, Ansible handler and application/config restart must use the same selected release and the same host lock/drain protocol; remove broad pull/up behavior from routine application rollouts. Planned application-affecting maintenance uses this path too. Retain Caddy/PostgreSQL volume definitions and existing source-IP/JWT boundaries. - Store applying/ready/healthy/failed/reconciled state atomically with bounded diagnostics, previous pins and backup identity. A disconnect/interrupted rollout remains auditable and blocks new releases until an operator reconciles actual image/schema/provider state. Before disruption, abort can explicitly restore normal dispatch on the unchanged release after checking state. After migration or uncertain startup, no automatic binary or database rollback. Explicit schema-compatible rollback reuses the full drain/identity/backup workflow; database restore is separate and coordinated with Kaukus to avoid replay/loss. Implementation sequence and acceptance: 1. CI/publication tooling and immutable manifest tests. 2. Application lifecycle, revision readiness, explicit migrations and backup/restore prerequisites. 3. Engines-style versioned release-bundle delivery and host-local Python/Compose execution; remove application deployment/configuration ownership from Ansible. 4. Fake-worker/PostgreSQL container rehearsal with fault injection, concurrency and interrupted provisioning/rollout recovery; verify Ansible idempotence preserves selected digests. 5. One controlled initial migration from the legacy manual deployment: pause upstream new submissions, establish an empty/stable legacy queue and no provider operations/resources, record existing digests, take/restore-verify a backup and install the lifecycle-capable release. The legacy binary cannot be assumed to support the new drain protocol. 6. Configure scoped credentials/protection, execute a real control-only release with zero GPU allocation, then one explicitly bounded worker release/transcription/cleanup rehearsal. Verify the final Ansible rerun changes nothing and recovery instructions match observed behavior. Deployment documentation belongs in deployment/README.md with project-native setup/release/status/retry/drain/reconcile/rollback/restore instructions. Keep the planning deliverable in chat; no standalone document was requested. Activation decisions/dependencies: - Confirm the recommended protected-tag production model and default one-worker L4 validation budget. Project-RK's branch auto-deployment targets Ergo, not production Ithaca. - The user selected https://murmur-backup.s3.fr-par.scw.cloud and will supply credentials themselves. Ansible group_vars/murmur.yml now sets RESTIC_REPOSITORY via murmur_backup_repository=s3:https://s3.fr-par.scw.cloud/murmur-backup (bucket root) and murmur_backup_region=fr-par. Public destination settings are merged separately from murmur_backup_environment so a credentials-only Vault/extra-vars dictionary cannot discard or redirect them. Credentials and the independent Restic encryption password remain unset; no bucket access or initialization was attempted. The user has now clarified that these must be host-local application configuration, not Ansible Vault; move the public bucket settings and credential setup instructions out of Ansible as part of the ownership correction. Targeted local Ansible renders pass for credentials-only and stale-destination secret dictionaries, preserving mode 0600 and disabled activation; site syntax, role lint and YAML lint pass. Runner placement, retention/RPO approval, encryption-key custody and live storage/restore verification remain outstanding. - Use a clean Ansible worktree based on current origin/main for implementation; the existing checkout has unrelated untracked host_vars/gpu.yml and must remain untouched. - Provision repository-scoped runner registration, scoped publishers, dedicated deployment/pull credentials, verified SSH host keys and tag protections before activation. - The cleaned offline implementation is ready on Murmur feature/add-safe-cicd-deployments. There is no Murmur Ansible worktree or application integration. The existing Ansible checkout and unrelated host_vars/gpu.yml remain untouched. All Murmur changes are uncommitted; no remote production changes have been made. - Implemented: one protected grouped murmur_vX.Y.Z release; exact-source tests; parallel cached immutable control/worker image builds; one strict manifest; six-file versioned deployment upload and host-local Python/Compose rollout. The repository owns Compose/Caddy/config; root-owned /etc/murmur files contain runtime, S3 and validation inputs supplied directly by the user. Deploy.py handles practical drain, explicit migration, backup/isolated restore, health, bounded worker validation, retry/reconcile/rollback and timer installation. Application code retains the previously verified pause/lease/allocation/cleanup protections. - Cleaned-system validation passed: sh deployment/check.sh runs 22 deployment/release tests, 65 Rust tests including real disposable-PostgreSQL lifecycle tests, 12 worker tests, fmt/check/strict Clippy/Pyright/Ruff and Compose rendering. All release workflow jobs parse in checksum-verified Gitea Runner 5.2 dry-run; static Caddy validation passes. Earlier current-image Docker smoke covered migration, JWT/idempotency, graceful stop/restart and isolated PG restore. This is local evidence only, not live registry/SSH/S3/GPU activation. - Allocation safety: unknown create outcomes retain capacity through visibility grace AND fresh successful empty provider inventory, even if an older provider ID remains. A cleaned profile identity is cleared atomically before fallback; incomplete cleanup forbids fallback. Backoff retains the reservation until it expires. Real-PostgreSQL regressions cover unknown and mixed fallback state, surviving orphans, stale callbacks, validation backend termination, result commit before drain and inventory outages that must not block normal dispatch. - Runner follow-up implemented on ci/use-kaukus-runner: all Murmur CI/release jobs now request `murmur`; deployment/Dockerfile.runner defines the dedicated image. On 5.9.138.8, `/opt/kaukus-ci/Dockerfile.murmur` is installed, `custom-runner-murmur:latest` is built (image sha256:d20e0c27e9ca97c5f83f62037cd7d2b5e4ba2aab9b563aa3cb61304513fae44f, ~734 MB), and the existing kaukus-runner registration now includes `murmur:docker://custom-runner-murmur:latest` while preserving its three existing labels. No active job was interrupted; runner restart count is zero. Tool/Docker-socket smoke passed. A full `deployment/check.sh` run inside the host job image on a Gitea-style user-defined Docker network passed with no leaked PostgreSQL test containers; the initial default-bridge manual attempt failed only because Docker's default bridge has no container-name DNS. Temporary acceptance source/archive/logs and candidate tag were removed. Host has ~1.5 TB free, 64 GB RAM and 20 CPUs; shared runner capacity remains one, so matrix builds serialize. Local full checks also pass. - First tagged release evidence: run 4012 failed before the worker build because CR credentials were mistakenly used for Scaleway. Scaleway login expects a valid worker registry publisher credential; the workstation's previously validated `rg.fr-par.scw.cloud` Docker credential was copied directly into the two Gitea worker registry secrets without displaying it. Rerun 4013 then succeeded for prepare, control-plane and worker builds; worker registry login passed and the cold worker build completed in about 16 minutes, seeding the managed cache. Deployment failed safely before migrations/container changes because none of the required root-owned host files exist: `/etc/murmur/runtime.env`, `backup.json`, `restic-password`, or `validation-fixtures.json`. There is no attempt/current release state. The uploaded bundle is retained at `/srv/murmur-deploy/releases/8-1`; traceback confirms missing runtime.env. This is an unmet documented first-adoption prerequisite, not a worker build failure. - Host preparation after run 4013: Restic 0.16.4 is installed. `/etc/murmur` exists root:root 0700. Existing `/opt/murmur/.env` was copied without exposure to `/etc/murmur/runtime.env` root:root 0600. Root-only mode-0600 backup.json and a generated 64-character Restic password exist. The user supplied both AWS values; the fixed murmur-backup repository/region and 512 MiB budget validate. The Restic repository initialized successfully and authenticated snapshots access reports zero snapshots. The generated password exists only on the host and must be escrowed independently. - User explicitly removed deployment-time transcription fixtures/GPU validation as unnecessary for the seven-day delivery model. Work on refactor/remove-deploy-worker-validation removes `verify-worker`, fixture configuration/mounts/receipts/commands and validation-only database state while retaining `/ready` revision/protocol/profile checks on every normally provisioned worker. `/etc/murmur/validation-fixtures.json` is now `[]` and will be unused after the new release. Local full validation passes: 22 deployment tests, 64 Rust tests and 12 worker tests. Run 4024 / murmur_v0.1.1 built both images quickly (worker cache hit: ~12 seconds) and entered bootstrap. Backup upload plus isolated restore succeeded (snapshot 17155ca73beb4ff1aed2e81852ef1d3d01fdc1c961d530ca3f9f20100df8ef46), migration applied, then explicit provider recovery failed because the first Scaleway list-servers request sent `page_token=`; the live v2alpha1 API rejects the empty continuation token with HTTP 400. Legacy control was already verified idle and stopped cleanly. To restore availability, the 0.1.1 control plane is running healthy with provisioning temporarily disabled; HTTPS health is 204 and submissions are accepted durably but not dispatched. PR #6 omits page_token on the first request and adds it only for a non-empty returned continuation token; focused test and strict Clippy pass. - Final activation: murmur_v0.1.2 run 4036 completed successfully in about four minutes. Control build took about one minute and the cached worker build took 12 seconds. Before deploy, the emergency control plane was confirmed idle (zero queued/running jobs and workers) and stopped cleanly. The release completed backup, isolated restore, migration, provider reconciliation and normal startup. Production current.json records murmur_v0.1.2 at d4d1ff5210db5029f47622c43add8aa6092cda0f; attempt state is healthy, HTTPS health is 204, the control plane is running and unpaused, scheduler heartbeat is healthy, Scaleway provisioning is enabled with max_workers=1, worker rows are zero, and backup snapshot ccf0be5699fd4b8474c050b8effa123eada53528c457fab3de68895bc7162543 is recorded. The daily backup timer is enabled/active and temporary emergency override files were removed. Deployment-time audio fixture/GPU validation is absent by design; normal demand-driven workers retain exact readiness/profile checks. - Worker-source cache test: murmur_v0.1.3 changed only `worker/murmur_worker/main.py`. All dependency, Norwegian model, Danish model, alignment, NLTK and model-generation steps reported `CACHED`; the source `COPY murmur_worker` rebuilt in 1.3 seconds. Registry image push reused existing blobs (`pushing layers` completed in 1.2 seconds; manifest in 2.1 seconds; cache export in 2.8 seconds), so the models were not rebuilt or re-uploaded. However the worker job took about three minutes because each release creates and deletes a new Docker-container Buildx builder: materializing the cached final stage downloaded/extracted many large cached layers, with `/opt/torch` taking about 185 seconds. A persistent named Buildx builder on the capacity-one Kaukus runner would avoid most of this repeated cache hydration after one warm-up. - Persistent cache follow-up: `murmur-release-builder` is running on 5.9.138.8 with repository-owned BuildKit GC policy. Cheap local/cache-mount records expire after 48 hours; reusable records keep up to 30 days; max cache is 150 GB with a 200 GB host free-space target and 40 GB reserve. First registry-cache hydration took 215.7 seconds and left 31.53 GB cache. A second build after a temporary worker Python edit completed in 1.0 second: all dependency/model/asset/final-copy inputs were local cache hits and only `COPY murmur_worker` rebuilt (0.0s). Temporary source/logs were removed; builder remains running with restart count zero. Full repository validation passes. PR #8 switches release jobs to reattach to this builder and stops deleting it at job cleanup; registry buildcache remains fallback. - Live persistent-cache release: murmur_v0.1.4 run 4060 completed successfully in about four minutes. The worker job took 16 seconds total; the BuildKit portion ran from roughly 12:00:00 to 12:00:11. Every dependency/model/asset stage, `/opt/torch` copy and `COPY murmur_worker` reported `CACHED`; unlike 0.1.3 there was no multi-gigabyte cache hydration. Existing layer push took 1.9 seconds, manifest 1.8 seconds and registry cache export 3.0 seconds. Persistent cache is 43.97 GB, builder is running with zero restarts, host has ~1.47 TB free, below the 150 GB cache cap and above the 200 GB free-space target. Production accepted murmur_v0.1.4 at c30efd6855b8bba517629be7ff3ddc4de25aecfa; attempt is healthy, HTTPS returns 204 and scheduler is healthy/unpaused. Full GPU image build was not attempted below the specified 100-GiB storage headroom; local free space is about 50 GiB. No real Gitea release, remote registry push, forced-command SSH acceptance, encrypted off-host restore, live Ansible idempotence or L4 validation has run. Feature reviews were patch-scoped/partial and could not access the external Ansible worktree; that integration was manually syntax/lint/render validated only. Do not infer live activation or approval from local tests. Acceptance criteria: - Agreed CI/release triggers and control/worker promotion behavior are documented in project-native deployment documentation. - Relevant automated checks gate publication/deployment; immutable retry and invalid/stale manifest cases are covered by tests. - CI can deploy only approved Murmur artifacts through a narrowly scoped command; PostgreSQL/Caddy, secrets and unrelated services remain untouched. - Deployment cannot duplicate active work or leak interrupted provisioning resources; migration failures leave auditable state and block unsafe follow-up. - Ansible server-baseline convergence never deploys/reconfigures Murmur or overwrites its selected release; application deployment and credentials are independent of Ansible. - A bounded real CI build/publication/deployment rehearsal verifies image identity, HTTPS health and no unexpected GPU allocation. Record any unverified activation dependencies explicitly. Pull requests: - #2: https://git.kaukus.no/Kaukus/murmur/pulls/2 (merged) - #3: https://git.kaukus.no/Kaukus/murmur/pulls/3 (merged) - #4: https://git.kaukus.no/Kaukus/murmur/pulls/4 (merged) - #5: https://git.kaukus.no/Kaukus/murmur/pulls/5 (merged) - #6: https://git.kaukus.no/Kaukus/murmur/pulls/6 (merged) - #7: https://git.kaukus.no/Kaukus/murmur/pulls/7 (merged; worker-source cache verification) - #8: https://git.kaukus.no/Kaukus/murmur/pulls/8 (persistent bounded BuildKit cache)