Build Prism autoscaled video processing

Goal Design Prism, a standalone Rust video-processing service with demand-based provisioning and retirement of GPU processors on Scaleway using the proven Murmur lifecycle. User selected Prism as the project name; use prism for the repository/service and prism-worker for the processor image. Implementation and production-readiness work authorized. User approves hosting Prism on the same physical server as Murmur with isolated services/database/credentials, live time-boxed Scaleway acceptance using at most two Prism GPU workers and Prism-scoped storage/resources, and ZITADEL authentication. Kaukus/cloud-router/Murmur integrations may be implemented but MUST NOT be deployed or switched into live routing. Existing application/streaming integrations and old processors remain active until separately approved cutover. User explicitly requested extracting Murmur's scaleway.rs into a shared crate before duplicating provider lifecycle logic. Extract scaleway-workers under Murmur's workspace, preserve Murmur behavior via an adapter, and depend on it from Prism. User explicitly prioritizes durability over process exit codes: FFmpeg can return misleading errors or exit successfully with a valid but truncated output. Map the whole existing pipeline, identify safe simplifications and make independently verified media completeness a publication requirement. Verified starting points - moeteadm/app/models/video_job.rb provides a PostgreSQL heavy/light queue with SKIP LOCKED claims, 120-second leases, priorities and retries. - config/deploy.production.yml declares fixed heavy CPU (concurrency 1), heavy NVENC GPU (concurrency 4) and light orchestration workers. - VideoQueue::Worker shutdown currently terminates active handlers and releases work; this is not an idle-drain mechanism. Lease renewal/finalization is not ownership-fenced, so safe autoscaling requires addressing stale attempts before relying on crash recovery. - FFmpeg handlers instantiate Rails status updaters that upload outputs and trigger domain side effects. A standalone processor must return artifacts/analysis while Rails retains those effects. - ../murmur contains a Rust control plane with durable jobs, authenticated idempotent submission, bounded Scaleway worker provisioning, idle retirement, resource reconciliation and pause/drain lifecycle. Its sample defaults are min_workers=0, max_workers=2, idle_seconds=1800; these are not yet selected for video. Agreed direction Build a standalone video-processing service in Rust, with a Rust control plane and Rust processor agent invoking FFmpeg/ffprobe. Prefer moving media-specific processing and orchestration out of Rails; retain application-specific authorization and domain effects in Kaukus. The fixed CPU/GPU servers will be retired: this is replacement capacity, not permanent burst capacity alongside existing hosts. Cold allocation taking around 10 minutes is acceptable; the user reports Murmur allocation is usually much faster. No instant-allocation requirement or warm-worker requirement. Proposed minimum is zero; exact idle cooldown and provisioning deadline remain to be chosen. Design constraints Confirmed: eliminate intermediate audio upload/download handoffs. The existing S3 WAV boundaries were a workaround for Sidekiq scaling and main-application deployments, later improved by the custom queue; they are not required media-processing boundaries. Keep channel analysis, temporary audio and chunk extraction local to an independent video worker, reusing one local source. Only durable final artifacts required by consumers (delivery video and audio chunks for Murmur) cross object storage. A Kaukus deploy must not restart, cancel or reset video work. Video-service deployment must separately support drain/version-compatible completion of existing workers. Worker-loss recovery may recompute uncommitted local work; do not reintroduce intermediate S3 handoffs merely to preserve every FFmpeg step. Reuse applicable Murmur infrastructure lifecycle through a deliberate shared boundary rather than copying its control plane wholesale. Use typed processing operations, attempt-scoped outputs, fenced completion and short-lived scoped storage access. GPU profiles/concurrency must be benchmarked for FFmpeg NVENC and disk/CPU needs rather than inherited from Whisper workloads. Distinguish allocation/readiness time from job execution and storage-access expiry. Preserve application behavior while moving media computation, validation and processing-stage dependencies into Rust. Standalone implementation, shared crate reuse, approved Prism infrastructure/identity setup and live GPU/storage validation are authorized. Integration code changes in consumer/producer services are now authorized; deployment of those integrations, production callback-routing changes and retirement of existing processors are not authorized. Settled policies - Maximum two GPU workers initially; no CPU worker pool or CPU comparison is needed. User has already benchmarked GPU superiority. Determine tested GPU profile and per-worker concurrency, not whether to use GPUs. - Never automatically publish incomplete output. Failures and validation evidence must be easy to discover and diagnose. - Completeness guarantee is faithful processing of received media: detect FFmpeg stopping early on malformed data even with exit zero. Producer capture/segment inventories are explicitly not a priority or prerequisite for this release; do not expand scope into guaranteeing that a producer captured an entire meeting. - Retain originals for 30 days after validated output is committed; failed processing does not trigger deletion and pending references must remain safe. - Full processing parity is required. Implementation of Kaukus/cloud-router/Murmur integrations is authorized; their deployment and live-routing/cutover changes are explicitly prohibited for now. - User proposes producer-to-video-service submission and video-service-to-Kaukus completion callbacks so Kaukus is not a recording-ingestion middleman. Recommended design is a callback/event-driven integration with durable delivery and idempotent consumers; service result/status retrieval remains available for reconciliation. Kaukus still originates its own editing/manual-upload requests when appropriate. Current delivery scope User authorized proceeding with isolated Prism deployment preparation and bounded GPU/S3 qualification, explicitly including CI-driven deployment like Murmur. Build off-host; preserve Murmur service and immutable release files. Implement CI validation and digest-pinned release/image tooling, isolated configuration/database/credentials, controlled shared-host ingress, encrypted backup/restore verification and safe lifecycle/recovery gates. Do not deploy producer/consumer integrations or switch live processing. Do not create release tags or publish implementation commits without explicit permission. Qualify production-image/native compatibility before enabling GPU allocation. User now explicitly requests moving Caddy out of both application Compose projects into an independently managed host edge proxy, so Murmur and Prism can deploy without managing each other's ingress. The user confirms Friday-to-Monday downtime is acceptable because Murmur will be idle for three days. Host proxy migration and necessary Murmur deployment adaptation are authorized. The user subsequently explicitly approved pushing both Murmur and Ansible configurations and aligning Ansible while preserving unrelated work. No service-integration cutover, GPU allocation or release tagging is authorized by this source-publication step. Preserve certificate state, separate proxy-facing networks from PostgreSQL, verify existing authentication/source restrictions, keep a tested rollback path and restore normal Murmur availability. The original Ansible checkout contains unrelated untracked host_vars/gpu.yml on feature/reconcile-murmur-provider-orphans; it and that checkout remain untouched. Approved Ansible work was isolated in /home/mathias/work/ansible-edge on refactor/externalize-control-host-edge, based on origin/main rather than inheriting the original feature commit. Current Murmur is revision e2538be, zero running jobs/workers, active accepted runtime /srv/murmur-deploy/releases/36-1, with certificate volumes murmur_caddy-data/config. Live preflight inventory succeeded; the final systemctl is-active check returned expected exit 4 because no native Caddy service exists, not an SSH/access problem. Decisions needed Complete integration contracts, qualify GPU profile/concurrency and source-format behavior, and finalize operator diagnostics/retention/rollout acceptance. Same-host Prism/Murmur control planes, zero warm capacity, two-GPU initial cap, future replacement of old processors, approximately 10-minute cold-allocation tolerance and no deployment of integrations are settled. Authorized real recording https://vod.kaukus.no/vod/municipality-skien/video_edits/4875/9658.mp4 User has no known cuts/mutes or silent-truncation source sample. Use deterministic synthetic fault/cut/mute cases alongside this real received-media sample. Read-only HEAD verified HTTP 200, Content-Length 747369466, video/mp4 and Accept-Ranges bytes; the multipart ETag is not a content checksum. No full download or real processing has yet run. Production access and identity - User bound prism.kaukus.tv to 151.115.147.97; DNS resolution was verified to that address. - First read-only SSH attempt as the documented root account, using the configured agent and normal host verification, failed with “Too many authentication failures.” No alternative identity/user, credential search, agent/SSH configuration change or host-key bypass was attempted. User later corrected the required SSH account to mathias. remote_python to 151.115.147.97 now authenticates as mathias (UID 1002); non-interactive sudo succeeds. No alternate key/agent method or SSH configuration change was used. Read-only inventory confirms murmur-control-1, 2 CPUs, about 2 GiB total RAM/1.3 GiB available and 13.3 GB free disk. Build images off-host and bound Prism memory/disk usage. Murmur Caddy owns ports 80/443 and mounts an immutable versioned Caddyfile from /srv/murmur-deploy/releases/36-1; isolated Prism ingress needs a controlled plan rather than editing that release in place. No host service was changed. - User supplied ZITADEL project 398311851844173829 and client credentials in ~/prism_key. The credential file was consumed only in memory, restricted to 0600, and values/tokens were neither printed nor persisted. A client-credentials request to login.kaukus.no succeeded and returned an RS256 JWT; subsequent live signature/issuer/audience/subject verification by the shared Rust verifier and Prism's identity-check confirmed issuer https://login.kaukus.no, subject 398311983545319429 and audience [398311851844173829]. Issuer token lifetime is about 31 days, so Prism must independently enforce the agreed one-hour maximum accepted token age and refresh outgoing tokens early. Shared verification and Prism's subject-to-client mapping have now passed using the supplied credentials, without displaying or persisting credentials or access tokens. - Authentication refactor is implemented locally on Murmur branch refactor/extract-service-auth from origin/main. Its JWT/JWKS verifier was extracted into crates/service-auth with an explicit subject allowlist and verified-subject return, while Murmur retains a thin single-subject adapter and its existing configuration/API behavior. Shared OAuth client-credentials handling has bounded secret-safe responses, no redirects, encoded Basic credentials, serialized early-refresh caching and explicit invalidation. Prism now uses this crate for mandatory ZITADEL API authentication and outgoing callback JWTs, with no production static-token fallback. Acceptance receipts are explicit/scoped/payload-checked/idempotent and have no application publication side effects. Auth-check/identity-check operator commands are available. Shared auth is now pinned in Prism to merged Murmur commit 1b21822cfb1b1da4ba6905452272aa0a5ef07b8e from PR #14. The sibling path dependency and named Docker context are removed. The pin, lockfile and docs update was committed/pushed as 3bff857 (build: Pin service authentication to merged revision) on feature/build-prism; working tree clean. The merged crate was verified identical to the reviewed crate, and all 33 Prism tests, formatting, checks and strict Clippy passed again against the merged Git dependency. The user explicitly authorized committing/pushing Prism and publishing the shared-auth PR; no deployment or release tagging is authorized by that publication step. Latest verified readiness milestone - Prism: 33 tests pass including the full disposable PostgreSQL/native/HTTP suites; strict Clippy and all-target checking pass. Murmur workspace: all 87 tests pass, including all 14 database lifecycle regressions; strict Clippy passes. Final local build, diff whitespace checks and Compose configuration validation pass. The supplied secret was checked in memory against repository files and is absent; no credential or JWT was displayed or stored in source. The temporary PostgreSQL container was removed. Authorized real media and derived local-only probes remain in the private acceptance cache for subsequent GPU tests. No live GPU/S3/host operations or integration deployment ran. - The authorized 747369466-byte recording was downloaded read-only once to private ~/.cache/prism-acceptance/9658-source.mp4; SHA256 06cd0653ad30def8b8dadeb86cd87c2bc748e4a566bc5db0d0a0e4cd9611174a. Source is H.264 1280x720 about 30 fps, stereo 48-kHz AAC, 3951.973229 seconds. A full strict native scan exited 0 without decode errors: video 118446 frames, first 0.1/end 3951.466666; audio 185101 frames/189543424 samples, first 0.078/end 3952.051229. Received video includes 95 one-frame presentation gaps, not processing loss. Audio presentation time differs from nominal PCM length by about 3.15 seconds. - This exposed an invalid equal-A/V-end assumption. Validation now models independent stream starts/durations, retains global-relative offsets, checks video frame count and timestamps plus audio coverage/sample counts, and does not merely loosen a cross-stream threshold. Synthetic regressions cover unequal tails, audio truncation masked by longer video, timestamp drift and nonzero chunk offsets. - Audio is normalized once locally to 16-kHz mono FLAC using its received presentation clock. Exact input decoded-frame/sample counts must match the independent scan before accepting resampler boundary padding. Full real-source normalization passed with all input evidence consumed, producing 63231572 samples over 3951.97325 seconds. No intermediate audio is uploaded. Chunk offsets preserve the stream's original relative start. This avoids rejecting or truncating healthy drifted inputs and improves subtitle alignment. - A local 90-second stream-copy excerpt of the real input passed whole-render and deterministic cut/mute semantic checks through the Rust engine, retaining 2701 frames whole and 2430 edited. Software encoding was only a local semantic probe, not a CPU/GPU comparison or GPU acceptance. - Producer/consumer integration code in Kaukus/cloud-router remains to implement; these repositories have not been changed or deployed in this phase. Their current callback path was inspected: router uploaded notifications are spawned HTTP deliveries with attempt records, not a durable submission outbox. The Prism handoff must persist intent before asynchronous submission and must not acknowledge custody merely on notification success. Ruby already has a reusable OAuth::ClientCredentialsClient and the jwt gem; reuse these rather than duplicate OAuth plumbing. Temporary callback/receipt mode allows Prism qualification without publishing to live Kaukus. Live host access is now working after the user explicitly specified mathias rather than root; read-only sudo inventory completed. No alternative identity/authentication fallback was attempted. Existing Murmur containers remain unchanged. Shared-auth prerequisite was committed/pushed as 6ff42b9 on refactor/extract-service-auth and opened as Murmur PR #14 against main. Prism's initial standalone snapshot was committed/pushed as 88da253 on feature/build-prism, with auth pinned to that exact published commit. Both worktrees are clean; no published history was rewritten. Acceptance criteria - Extract shared Scaleway lifecycle crate and verify Murmur remains behavior-compatible, including uncertain allocation, cleanup, tags, profile fallback, readiness and telemetry. - Implement standalone Rust Prism control plane and worker with typed rendering/editing/audio operations, worker-local intermediates and strict media validation. - Verify durable idempotent submissions, fenced attempts, stage recovery, authenticated persistent callbacks and discoverable diagnostics. - Implement GPU autoscaling (zero idle, maximum two), safe drain and resource reconciliation using the shared crate. - Provide configuration, API examples, deployment instructions and deterministic tests/native FFmpeg regressions. GPU/provider acceptance requires explicitly authorized live validation; do not claim it from CPU fixtures. Kaukus and other-service integration remains pending. Pipeline map (source-inspected) 1. Capture upstream: ruststreamer/local-streamer record MP4 segments and bookmark metadata. Shared gst-recording validates segment headers/duration, skips unprobeable segments and concatenates with stream copy, using ignore_err. Local boxes enqueue the finalized file for durable tus upload; ruststreamer uploads MP4/bookmarks to S3. This is remux/concatenation, not the later delivery encode. Producer capture/shutdown and upload recovery were inspected only at relevant boundaries, not fully audited. 2. Rails ingestion: ruststreamer uploaded callback creates original VideoEdits for source channels and imports bookmarks.json; first file drives audio extraction. Cloud-router uploaded callback consumes tus capability URLs and inline bookmarks, with main driving transcription and hdmi normally video-only. Browser multipart uploads, legacy local-temp upload job and training uploads converge on full-video transcode. FTP CSV imports create meetings/folders/bookmarks and low-priority transcodes; one preferred source per folder drives transcription. Archive retranscode and transcription rebuild commands reuse these paths. 3. Delivery rendering: plain transcode produces H.264/AAC faststart MP4; manual edits calculate kept/muted intervals and render a trim/concat filtergraph. Downloads occur before processing, own S3 objects bypass VOD via presigning, HLS is materialized by FFmpeg remux, FTP via curl. Video uploads use multipart as needed and verify object byte length. Application callbacks set URLs/status, broadcast, adjust bookmarks or promote training replacements. Streamer source cleanup is scheduled 30 days later. 4. Audio: mono input goes straight to chunk extraction. Stereo fans out left/right 48-kHz WAV extraction, uploads both, downloads them to mix into summed.wav, uploads/downloads the mix, then volumedetect selects left-only if peak is below -40 dB, otherwise stereo downmix. Temp WAVs are deleted. Audio chunks are 30-minute 16-kHz mono AAC/M4A files. Each chunk job currently downloads the entire video separately. 5. Transcription: upload chunk, submit idempotently to Murmur, durably poll at 15 seconds up to the configured 12-hour poll count, require every chunk completed, offset and combine WebVTT, persist transcription and delete chunk audio. Manual rebuild destroys existing chunks/transcription before starting again. Video and subtitle outcomes should remain independent. Material durability findings - FfmpegHandler output-duration gate accepts >=95%, skips unknown/<=10-second expected duration and skips failed output probes. It checks container duration only, not per-stream media coverage or full decode. FTP pre-probing deliberately returns unknown duration. HLS materialization accepts exit zero plus nonempty output only. - Manual render light handler calls complete! and starts transcription immediately after FfmpegService.process merely enqueues heavy processing; it can therefore transcribe the unedited source. Heavy edit success never establishes its own completed transition. Preserve intended behavior, not this ordering. - Queue leases/finalization and stable S3 output keys are not attempt-fenced; stale workers can renew/finalize or overwrite a newer attempt. Bookmark remapping mutates existing positions and is not safe to replay blindly. - Application effects, next-stage enqueueing and job completion are not one durable transaction. Several audio kickoff errors are swallowed. Duration-zero chunk planning returns normally. Stereo coordination uses a transaction without a row lock or a unique downstream-stage key. Destructive transcription restart and fixed chunk keys allow old/new runs to interfere. - Upstream may already have skipped corrupt segments before the service sees its input. A valid final MP4 alone cannot establish that an entire meeting was captured. User explicitly limits this release's guarantee to not truncating the received media; upstream capture completeness and producer segment-inventory changes are outside initial scope. Implementation contract (verification required) - Two principal typed media operations: render(source, cuts/mutes, output profile) and prepare_transcription_audio(validated media), with materialize/probe/validate/upload as shared internal stages. Full transcode is render with an unchanged timeline. Keep media-local work together, reuse one download/probe and eliminate distributed WAV artifacts; perform stereo analysis directly in one local filtergraph, then generate/upload final chunks. Keep durable checkpoints at validated delivery video and final audio chunk manifests, not every disposable analysis file. Do not build a general-purpose distributed DAG engine. - Freeze immutable source identity/size/checksum or object version, processing spec/profile and source-coordinate timeline. For HLS, snapshot a finite playlist and explicitly acquire every required segment/init/key with bounded retries, no silent skips; preserve discontinuities/ranges. Validate local inputs before rendering. Do not infer intended media time from meeting wall time. Received-source completeness is the baseline; producer capture manifests are outside the initial scope. Preserve enough local source stream/timeline/decode evidence to detect early termination or dropped intervals during conversion, even when FFmpeg reports success. - Render to attempt-local files. Validate exit/signal/timeout plus required streams/codec/layout, strict full decode, per-stream start/end/continuity and expected retained interval coverage, using frame/timebase/AAC-padding tolerances rather than a 5% allowance. Duration/full decode are necessary but not sufficient. Probe failure is failure, never skipped. Analysis-only stages require complete scan and valid measurements, not a nonexistent output-file probe. Preserve unknown/rejected outputs for diagnosis; no automatic partial publishing or source deletion. - Use stage-specific bounded retries/backoff; keep valid rendered/uploaded artifacts when upload or Rails application fails. Persist stage outcomes and pending delivery in the control database. Lease renewal, artifact commit and completion require a matching attempt generation. Upload under immutable attempt-scoped keys; commit only verified manifests in one DB transaction. S3 upload and DB completion are not atomic: reconcile uncertain uploads and clean abandoned artifacts after a safety interval. Checksum support/read-back must be verified on Scaleway; multipart ETag is not a content digest. - Recording producers such as cloud-router should submit directly to the video service, rather than routing ingestion through Kaukus. Producer submission needs durable intent/retry plus a stable idempotency key. The control plane commits validated artifacts and a pending completion event in one database transaction, then delivers authenticated callbacks to configured consumers with persistent retries. Results remain retrievable and are applied idempotently by run/generation; Kaukus downtime must not rerun processing or lose completed recordings. Preserve source/recording/meeting/output identities and bookmarks through the event contract, handle out-of-order delivery, and acknowledge upstream custody only after durable acquisition. Kaukus-originated edits/uploads still submit directly when appropriate. Integration implementation is pending. Keep source-coordinate bookmarks and apply returned timeline mapping once. Start Murmur on committed processed-media chunks only; persist all offsets/IDs, resume failed work instead of deleting a whole run, retain previous successful subtitles while replacement is pending. Video ready and subtitles ready are distinct states. Verified native reproduction / limitations Local FFmpeg n9.0.2, temporary localhost HTTP fixture: generated 12 seconds of 10-fps HLS with three 4-second segments. Complete input: exit 0, readable 12-second video, 120 decoded frames. Missing tail: exit 0, readable 8-second video, 80 frames. Missing middle: exit 0, reported video duration still 12 seconds (container 12.096 seconds, same as complete), but only 80 decoded frames. FFmpeg logged skipped segments. This confirms exit status and container duration alone miss media loss. The fixture and media were temporary; no production requests or repository changes. This does not reproduce a particular reported production incident or verify deployed FFmpeg behavior. Attempt to exercise current Ruby validator in isolation could not run because ruby is not installed/on PATH; Rails specs were not run. Threshold/fail-open/manual-completion findings are source-verified, not runtime Rails tests. The earlier media investigation did not change production. Current implementation milestone - Extracted scaleway-workers into ../murmur/crates/scaleway-workers as a workspace library. Shared provider owns profile allocation/fallback, uncertain allocation handling, namespace-scoped resource ownership, endpoint discovery, server/IP/volume cleanup and reconciliation. Applications supply AllocationStore and WorkerRuntime adapters; no Whisper, FFmpeg or application database schema in the library. - Murmur now delegates through a small adapter that retains Whisper profile/readiness/revision checks and allocation-scoped usage reporting. Added reusable shell-free Docker cloud-init construction with root-only environment files. Updated Docker build context and workspace CI checks. - Validation passed: cargo fmt --all --check; locked offline workspace all-target check; strict Clippy; all 82 Rust tests including disposable PostgreSQL lifecycle regressions; 22 Python deployment tests. One Docker ingress test was skipped. Provider HTTP tests are local mocks, not live Scaleway allocation. Docker images and Python Whisper worker suite were not rebuilt/retested in this extraction. - Murmur extraction is committed as dda9f7e (refactor: Extract shared Scaleway worker lifecycle) and pushed on refactor/extract-scaleway-workers with upstream tracking. The branch was moved onto updated origin/main (e2538be) after verifying its tree matched the original metrics branch head, so its complete comparison contains only the extraction commit and no inherited metrics commit. Working tree clean. Formatting, locked workspace all-target check, strict Clippy and standalone shared-crate tests were rerun successfully before publication; full Rust/PostgreSQL and deployment validation results above remain applicable because the starting trees matched. No deployment. Prism has a standalone implementation in /home/mathias/work/prism on feature/build-prism. Origin is git@git.kaukus.no:Kaukus/prism.git; the remote was verified empty and attached without creating an empty initial commit. Implemented modules cover typed render/audio jobs, PostgreSQL leases and attempt fencing, immutable artifact grants, worker-local processing, strict native media validation, checkpoint replay, persistent callback delivery, producer/consumer read permissions and the shared GPU lifecycle. Verified first standalone implementation milestone: all 27 tests pass through scripts/check.sh, including disposable PostgreSQL, native FFmpeg and real HTTP protocol fixtures. Formatting, locked all-target check, strict Clippy and a normal binary build pass. Native coverage includes misleading exit-zero/full-duration outputs, one-frame tail loss, missing HLS segments, nonzero source timestamps, cuts/mutes, phase cancellation, audio-only and video-without-audio. The pipeline fixture verifies one source download, no intermediate WAV uploads, local validated-render reuse after a failed upload attempt, idempotent claim/checkpoint replay and a completion response lost after storage committed. S3 fixtures independently verify signatures/checksums, escaped keys, multipart transfer beyond 32 MiB including a small final part, length/digest checks, persisted bucket identity and cleanup. Authorization/lifecycle cases cover consumer read grants, worker/readiness purpose separation, stale-attempt fences, bounded retries, partial audio manifests, callback outages/missing configuration, migration leadership and old worker images finishing owned work without taking new jobs; attempt image/revision history survives instance removal. CLI migrate/status/resume/pause/drain and validate-config were exercised on a fresh disposable database, as were real binary HTTP startup, authorization and SIGTERM shutdown. Placeholder and reused-root credentials are rejected. Example TOML, image digests and Compose configuration parse. Dockerfile/build instructions, configuration examples, README and deployment notes are present; Docker images themselves were not built. Prism was subsequently committed/pushed as noted in the latest publication milestone. Temporary PostgreSQL containers and native smoke media/config fixtures were removed; existing Kaukus/Murmur worktrees and running development services are unchanged. Public MinIO images were unavailable anonymously; no credentials or authentication overrides were attempted. Actual Scaleway GPU/S3, production-image native compatibility and FTP-server acceptance remain unverified. Kaukus/cloud-router/Murmur integrations remain pending; no live deployment. This is a working initial standalone pass, not full-parity/production sign-off. Remaining release work includes external-rendition/live/SAMPLE-AES HLS requirements and source-inventory compatibility; live GPU, S3, production-image FFmpeg and FTP acceptance; producer enforcement of original-source availability/30-day retention; stable/refreshed chunk reads for Murmur (15-minute download grants must not be enqueued as durable transcription sources); consumer acknowledgements/cleanup of committed audio; and durable rejected-output quarantine. Failed media files currently remain only on the worker until retirement, while fingerprints, inspection summaries and bounded diagnostics persist. Authentication has since been replaced by ZITADEL JWT verification and OAuth callback credentials; see the latest readiness milestone. The broader task stays in progress. Pull request - Murmur #13: https://git.kaukus.no/Kaukus/murmur/pulls/13 — refactor: Extract shared Scaleway worker lifecycle. Head refactor/extract-scaleway-workers; base main. Merged by the user on 2026-11-05. Prism now depends on merged Murmur revision 7062d3c933d34f7b555c9a5c9ec8b8a8a06e7985 through a pinned Git dependency, not a sibling path or copied provider code. The broader Prism implementation remains in progress. - Murmur #14: https://git.kaukus.no/Kaukus/murmur/pulls/14 — refactor: Extract shared service authentication. Head refactor/extract-service-auth; base main. Commit 6ff42b9fc8b6c722e0080b01dbec082d57802b1f. Merged by the user as 1b21822cfb1b1da4ba6905452272aa0a5ef07b8e; only the auth extraction/new shared client was in the comparison. Prism publication - Initial commit 88da253db2a3100329bb6203b4ed7cb63059cdac (feature: Build durable GPU video processing), pushed with upstream on feature/build-prism to git@git.kaukus.no:Kaukus/prism.git. Remote was empty, so this is a root commit on the feature branch; no main branch or Prism PR was created. - Both shared crates now use exact published Git pins. Prism's full 33-test suite, strict Clippy, formatting and all-target checks passed again after replacing the auth path dependency. Supplied-secret and private-tracking scans passed on exact staged content. Shared-auth standalone 12 tests/check/Clippy reran successfully; full 87-test workspace/database validation from the implementation phase still applies to the unchanged extraction. - Nothing deployed or release-tagged; no live routing changes. Integrations and live GPU/S3 acceptance remain outstanding. CI and production-image qualification milestone (published for review) - Added .gitea/workflows/ci.yaml and a shared dependency-identity composite action. Uses the existing murmur runner label (actual Prism eligibility not yet verified), branch pushes/manual dispatch only, no automatic fork PR execution. Repository/runner-level fork approval remains required; editable YAML is not a trust boundary. Dependency key must be a dedicated read-only Murmur deploy key, with pinned Git host keys; Rust checks remove signing keys after fetch. Ordinary checks have no registry/deployment credentials. - Added scripts/check_images.py and 12 Python helper regressions. It resolves the configured IdentityAgent rather than silently falling back to SSH_AUTH_SOCK, requires pinned known-hosts entries, builds both runtime images, runs the 10 native media regressions in the shared production runtime stage, checks non-root configuration loading and anonymous worker-volume ownership/NVENC encoder availability, and removes only its own test images/containers/volumes. Native checks have no external network, read-only rootfs and explicit CPU/memory/PID limits. - Initial real BuildKit fetch failed because the default forwarded agent differed from ~/.ssh/config's configured IdentityAgent. No SSH configuration/key/authentication bypass was used. Forwarding the already configured agent explicitly succeeded. Both runtime targets built and all 10 native media tests passed with actual Debian FFmpeg 5.1.9. This is software semantic/native-image qualification, not live GPU/NVENC acceptance. Non-root config validation and worker writable-volume/encoder checks passed. The full 33 Rust tests, strict Clippy/check/formatting and 12 helper tests pass locally. Actionlint passes with only its unsupported Gitea fully qualified action-URL syntax diagnostic filtered; Gitea run 4369 attempt 2 now passed both the check and images jobs on the existing global kaukus-runner (murmur label). - Manual publish_images is false by default. When explicitly requested, CI logs into isolated job-local Docker credentials and publishes already-tested images to dedicated Prism repositories. A clean checkout/full matching revision, positive CI ordering, digest resolution for both images and anonymous worker inspection are required before emitting a qualification-only artifact. Wrong/missing digests, private worker images and wrong/dirty source emit no manifest. Ordinary PR/branch checks never publish; nothing deploys. Actual CI publication has now passed for both registries in run 4372; exact digests, linux/amd64 platform and source/component/protocol labels were independently verified, including anonymous worker inspection. The v3 artifact uploader reports success, but REST listing/retrieval remains unverified (see qualification publication milestone). - docs/deployment.md records exact secrets/worker-repository variable, trust model and remaining rollout requirements. User explicitly approved pushing this milestone. CI source committed as e1b47a5d9c878c6bb86ba77f03cd7e5cf297452a (ci: Add runtime image qualification and publishing) and pushed on feature/build-prism. Created/pushed main at the previously published baseline 3bff8576062f5ad0d19348bba956c1a83be1c7b0, so feature/build-prism compared against main contains only the CI commit. Both refs were published atomically; working tree clean. No PR was opened and no default-branch/protection settings were changed. Re-ran the 12 helper tests, scoped actionlint validation, whitespace and exact staged private-content scans before publication; all passed. All six Prism CI secrets and PRISM_WORKER_REPOSITORY are configured and verified as repository metadata. Runner eligibility is verified by the successful real CI run. Registry logins succeeded with the supplied control publisher and new isolated worker publisher. The original non-publishing credential-verification rerun was followed by the explicitly approved publishing run 4372, which passed and published both qualified images. CI artifact REST readback remains a separate limitation, recorded below. No production release tags or deployment workflow were added. Future deployment still needs protected main/tag policy, isolated runtime credentials/database, encrypted backup/restore checks and readiness/revision/scheduler gates. The host now has an independent edge proxy; application/Ansible source alignment and publication remain required as described in the edge migration milestone. - A requested review-tool run was unable to target Prism because the harness snapshot remained moeteadm; it did not review the Prism patch or provide an approval. No unrelated repository source changes were made. Murmur runtime and the production host remain untouched in this phase. CI credential provisioning verified - User authorized CI credential setup, then supplied robot_deploy's cr.kaukus.no password in /home/mathias/work/moeteadm/harbor_pass.txr. Used that supplied push/pull account (not an administrator) for PRISM_CONTROL_REGISTRY_USERNAME/PASSWORD; login verified with an isolated temporary Docker configuration. No existing credential was modified/rotated. An earlier stored Harbor administrative login was rejected; it was not used further. - Created dedicated Ed25519 dependency deploy key ID 1 on Kaukus/murmur, confirmed read_only=true. Installed its private key and existing independently verified git.kaukus.no host entries as PRISM_DEPENDENCY_SSH_KEY and PRISM_GIT_KNOWN_HOSTS. No local SSH/agent configuration was changed. Real CI dependency fetch succeeded with the new identity. - Created dedicated Scaleway Prism project 875bfbe6-7b2c-4e76-b384-416d9e13dfd2 in Kaukus organization af1a29c3-6bd1-4924-a2a1-74ec946451ed. Public namespace prism-875bfbe6, ID ee8f607d-ed70-4d9d-803b-7fad2c3f34f1, is ready and empty. Set PRISM_WORKER_REPOSITORY=rg.fr-par.scw.cloud/prism-875bfbe6/worker. The generic name prism was globally taken, so a project-qualified namespace was created instead; no existing namespace was changed. - New IAM application e3924aa4-9f52-4def-ba57-d63124f51e6b (prism-ci-registry-publisher), policy 1913f5e9-96cf-4ae7-bde0-d841534a9c9e. Read back and confirmed exactly ContainerRegistryFullAccess on the new Prism project; no Instance, Object Storage, IAM or Murmur-project grant. Installed PRISM_WORKER_REGISTRY_USERNAME/PASSWORD and verified registry login. New worker API key expires 2027-02-04T11:25:52Z; rotate before expiry. - Secret backups in Kaukus-Server: Prism CI control registry publisher (n4cmjqh72hpgbi7rhaj7rrs2ey), Prism CI dependency reader (ql3hw6wkldgflvo34rx6lgdvse), Prism CI Scaleway registry publisher (cxpxiv3s5aqsiumwynmvfen4zy). Normalized built-in fields and verified recoverable stored values without printing secrets. Temporary generated-key and registry-auth directories were removed. Verified the control backup exactly matched the supplied file, then moved that untracked mode-0600 plaintext handoff outside Git to /home/mathias/.cache/prism-ci-handoff-c4t5p0av/harbor_pass.txr (private mode-0700 directory). - Initial Gitea run 4369 attempt 1 failed before credentials were provisioned. Explicitly reran the same known non-publishing source with new credentials: attempt 2 passed check and images. https://git.kaukus.no/Kaukus/prism/actions/runs/4369 . No image publication, release tags, GPU allocation or deployment occurred during credential setup. No repository source edits/commits were needed; Prism working tree remains clean. Qualification image publication verified - User explicitly authorized step 1: manual CI qualification image publication only. Dispatched ci.yaml on feature/build-prism at e1b47a5d9c878c6bb86ba77f03cd7e5cf297452a with publish_images=true. Run 4372, workflow sequence 2, attempt 1: https://git.kaukus.no/Kaukus/prism/actions/runs/4372 . Both check and images jobs succeeded, including actual publisher authentication/pushes, native runtime checks, anonymous worker inspection and credential cleanup. - Tag qualification-2-1-e1b47a5d9c87. Control: cr.kaukus.no/kaukus/prism-control-plane@sha256:0c9b4b9e4da0c5ded536e6be583fbed501636ff50f4a418f011215f5454606ab . Worker: rg.fr-par.scw.cloud/prism-875bfbe6/worker@sha256:11d944d495560c2ffefe566acb272d882f3409e436910090d733f1b9043461fd . Registry inspection independently confirmed both exact digest references, linux/amd64, matching e1b47a5 source revision, component titles, Prism source URL and protocol 1 labels. Worker inspection succeeded anonymously using an empty local Docker credential configuration before control login. Temporary local registry credentials were removed. - Artifact prism-qualification-4372-1: upload-artifact v3 reports one file finalized/uploaded successfully (333 compressed bytes). However, Gitea's run/repository REST artifact listings return empty, so the original CI manifest was NOT retrieved. Observations match the documented older/v3 artifact REST visibility limitation; no server upgrade/authentication bypass attempted. Do not claim artifact readback passed. Before production deployment, verify download-artifact v3 readback or another supported durable retrieval path rather than relying on the current REST list endpoint. - Saved an explicitly reconstructed verification record at /home/mathias/.cache/prism-acceptance/qualification-4372-1.verified.json, containing verified digest references and provenance/limitations, not represented as a downloaded CI artifact. No release tags, source edits/commits, GPU instances, service deployments or live-routing changes. Production readiness and GPU/S3 acceptance remain outstanding. Independent host edge migration verified - User selected a standalone host edge proxy outside both application Compose projects and explicitly allowed Friday-to-Monday production outages. Live configuration is /srv/kaukus-edge. Its desired source is now versioned in Ansible roles/host_edge/files on the published infrastructure branch; /home/mathias/work/kaukus-edge is a compatibility symlink to /home/mathias/work/ansible-edge/roles/host_edge/files, not a duplicated source tree. Dedicated Compose project kaukus-edge and enabled/active kaukus-edge.service now own public TCP 80/443. Pinned Caddy image is unchanged. The unit depends only on Docker/network, includes PartOf=docker.service, and has no application lifecycle dependency. The proxy has read-only configuration/root filesystem, 128 MiB memory/PID limits and bounded log storage. No Prism route/backend or certificate was enabled. - Host-owned external certificate volumes kaukus-edge-data/config were copied from stopped legacy Murmur volumes; old volumes remain intact and mode-0600 tar archives are in root-only /var/lib/kaukus-edge. New site configuration is byte-for-byte the old live route except the upstream alias. Existing TLS certificate SHA256 a0dd7332c25e8c16bafdfe2b7152266c45eb8945c140091dbba8b4b8ffa30d1d is unchanged. No historical Murmur release file was modified. - Separate external bridges: kaukus-edge-murmur has only kaukus-edge-caddy-1 and murmur-control-plane-1 (alias murmur-control-plane). kaukus-edge-prism has only Caddy, reserved for future Prism. PostgreSQL remains only on murmur_default; no database ports were published. Legacy murmur-caddy-1 is retained stopped with restart policy no, so it cannot reclaim listeners after Docker/host restart. - Migration used the existing Murmur deployment/backup lock, idle/queued-work gates, dispatch pause, certificate copy and live TLS/authentication checks. A real rollback exposed that Murmur pause clears drain confirmation; resume must await the next scheduler heartbeat. Fixed the host-owned migration tool to explicitly wait for a verified healthy drain, added success/timeout regressions, then successfully exercised rollback and reactivation. Unknown subprocess timeouts fail closed rather than auto-rerun. - Live independence tests passed: restarting Caddy did not change the Murmur container; stopping Murmur left Caddy serving the same trusted TLS certificate and a 502 for the unavailable backend; restarting Murmur used its unchanged e2538be image and left the proxy untouched. Murmur is now resumed, scheduler healthy, zero jobs/workers. PostgreSQL was never restarted or recreated. External HTTPS checks independently passed /health=204, /metrics=401, unauthenticated /v1=403, unchanged HSTS/nosniff and spoofed-source-header rejection. Host reboot/Docker-daemon restart were not performed; boot enablement, systemd configuration and dependency properties were verified. Root-only acceptance record: /var/lib/kaukus-edge/acceptance.json; migration.json and failure.log retain the audit/recovery state. - Murmur adaptation was committed/pushed as 1e8f8f0223c4bb20d086d5a3ed0d809c9f2f7a71 (refactor: Leave ingress ownership to the host) on refactor/externalize-edge-proxy, based on origin/main (1b21822). Removed project Caddy service/volumes, release Caddyfile membership/upload and Caddy manipulation from preflight/rollout/reconcile. Runtime overrides attach only control-plane to the host-managed external network, including when targeting a historical bundle. Preflight validates the external bridge before registry/database operations. Moved ingress configuration and real Caddy route tests into the neutral edge source instead of duplicating them. 25 Murmur deployment tests, 9 neutral edge/migration tests, Compose rendering, Caddy config validation, Python syntax and whitespace checks pass. Rust/application code was unchanged; Rust/Whisper suites were not rerun in this deployment-only patch. - REQUIRED FOLLOW-UP before another Murmur release/convergence: merge the two published branches. The previously accepted release's --configure/--rollback/--reconcile still manage Caddy and must not be invoked after this migration. Its daily database-backup-only command can continue. The legacy Ansible Murmur role has now been retired in the infrastructure branch; do not converge an older branch that still owns the application/proxy. No new live convergence, GPU allocation, integration rollout, application image deployment or release tag occurred during source publication. Published Murmur/Ansible source alignment - Explicit user approval: push both configurations, leaving original host_vars/gpu.yml untouched. Murmur commit 1e8f8f0223c4bb20d086d5a3ed0d809c9f2f7a71 on refactor/externalize-edge-proxy; Ansible commit 0c5319180c6caa8f9711e6d5d3cdf5c137458f39 (refactor: Manage control-host ingress independently) on refactor/externalize-control-host-edge. Remote refs verified. Each comparison against origin/main contains exactly its one new commit; no unrelated provider-orphan feature commit was inherited. Both implementation worktrees are clean; original Ansible checkout remains on its prior feature branch with only its original untracked gpu.yml. After updating/authenticating the Pi Gitea adapter, opened Murmur PR #15 https://git.kaukus.no/Kaukus/murmur/pulls/15 (refactor/externalize-edge-proxy -> main, title “refactor: Leave ingress ownership to the host”) and Ansible PR #5 https://git.kaukus.no/Kaukus/Ansible/pulls/5 (refactor/externalize-control-host-edge -> main, title “refactor: Manage control-host ingress independently”). Both were opened non-draft and mergeable, then merged by the user. Murmur main merge revision 53113dc (PR #15, merged 2026-11-09T09:18:01Z). Ansible main merge revision 029b3ac (PR #5, merged 2026-11-09T09:17:19Z; includes 0c53191). Remote main refs were fetched and verified. The published external-proxy deployment and infrastructure source alignment milestone is complete. - Ansible desired source is roles/host_edge/files (canonical Compose, Caddy root/site, systemd unit, manual migration/recovery tool and real ingress/migration tests). Retired the old application-deploying murmur role, stale image/scaling/auth variables and secret templates; this prevents full convergence from rolling the CI-managed application back to an old Ansible image. host_edge inventory parent preserves the existing murmur host selector and inherits the unchanged firewall/monitoring catalog. site.yml invokes only the independent infrastructure role for that host. - New role refuses a legacy proxy that is running or can auto-restart; does not auto-migrate, stop, redeploy or restart any application. External network management explicitly uses appends=true, preserving current application endpoints/aliases; certificate volumes use recreate=never. Runtime configuration is validated before edge-only restart, and unknown enabled sites fail closed instead of being deleted. Documentation/recovery/test updates do not notify a restart; readiness depends only on the proxy, not backend application health. Dry-run skips actual activation. Role derives network/volume/image identities from its sole canonical Compose file rather than duplicate constants. - Final verification passed: 25 Murmur deployment tests, 4 Ansible role-ownership/safety regressions, 9 real Caddy/migration regressions, yaml lint, production-profile ansible-lint with no warnings/failures, site playbook syntax, inventory edge/firewall inheritance and absence of stale application variables, Compose rendering and whitespace. Local Ansible commands needed LC_ALL/LANG=C.UTF-8 to avoid the harness locale initialization problem; no host locale was changed. Exact staged scans found no supplied secrets, private keys or private tracking metadata. Rust/Whisper suites were not rerun because no Rust/worker code changed. The Ansible role itself was not applied during the source-publication step; earlier manual host migration and live restart/rollback evidence remain applicable. Current blocker (2026-11-09) - After Prism and both ingress changes were merged, started the next production-readiness phase by attempting the required Ansible check/diff for the merged host_edge role before convergence. Local Prism main is e1b47a5 and clean; merged Ansible origin/main 029b3ac has the same tree as the clean ansible-edge worktree. Inventory resolves murmur-control-1 to ansible@151.115.147.97 and the configured 1Password agent socket is present. - The initial Ansible fact-gathering SSH connection failed with `Too many authentication failures` before any task or host mutation ran. Per remote-access policy, no retry, alternate identity/user, SSH option/config change, agent manipulation or raw SSH fallback was attempted. User configured SSH identity selection for the `ansible` destination user using the dedicated 1Password-backed public-key identity and confirmed access now works. The authentication blocker is resolved; the user explicitly asked to resume the next steps. Rerun Ansible check/diff before applying, then verify the live independent proxy and Murmur state. Prism release/deployment implementation has not begun in this phase, and no Prism service, route, storage or GPU resource was changed. Host-edge convergence outcome and new tooling blocker - After the user fixed Ansible identity selection, merged Ansible host_edge check/diff succeeded. It proposed only canonical Compose formatting plus documentation; only the Compose change notified an edge-only restart. Applied successfully: 20 tasks ok, 3 changed, no failures. The role validated Compose/Caddy/systemd before restarting the standalone proxy and verified independent Compose ownership. No Murmur/PostgreSQL deployment task exists in the role. - Independent external HTTPS verification after convergence passed: https://murmur.kaukus.tv/health=204, /metrics=401 and unauthenticated /v1 probe=403. This confirms the restarted edge is serving the preserved route and restrictions. - The final read-only host inventory/status check through remote_python did not connect because its validator now rejects the newly added SSH `Match` directive (`SSH Match directives are unsupported; configuration predicates must not execute`). Per remote-access policy, no retry, raw SSH fallback, alternate identity, validator bypass or SSH configuration change was attempted. User chose to update remote_python rather than remove the working SSH Match configuration. Implemented and committed e9f5e15 on personal Pi main (`fix: Support safe SSH match blocks`), currently one local commit ahead of origin/main and not pushed. The runner now permits only static `Match host`, `originalhost`, `user`, `localuser`, and standalone `all` predicates; it still rejects exec, localnetwork, canonical/final/tagged/command, unknown, negated, expanded or malformed predicates before SSH runs. OpenSSH remains responsible for first-value resolution; non-allowlisted authentication/config options are still not copied to execution. - Verification: 19 Python tests pass including the full isolated sshd/agent suite; 5 Pi extension tests pass; direct standalone runner connected through the user's actual Match-based config to 151.115.147.97 and returned hostname murmur-control-1. Empty-agent fallback and unknown-host-key rejection remain covered. Commit e9f5e15 created directly on main per explicit user instruction. User reloaded/resumed Pi and requested continuation. Verify the registered remote_python tool with a read-only call before further host work, then begin Prism's isolated production release/deployment implementation. Keep autoscaling, producer/consumer integrations and Prism Caddy route disabled until their separate acceptance gates. Ansible convergence itself completed successfully and Murmur public health remains good. Prism deployment implementation is now underway on feature/add-production-deployment, with host preparation on Ansible branch feature/prepare-prism-deployment. Source is now committed and pushed, but nothing is deployed in this phase yet. Prism commit f639d87f6af0512c9c8d8f9fcb95f8cc0e311fc5 (`feature: Add protected production deployment`) is on feature/add-production-deployment. Ansible commit d881135e3bfba80eb839e10f3a2761d5fcd30f93 (`feature: Prepare Prism deployment host`) is on feature/prepare-prism-deployment. Both remote refs and clean worktrees were verified; no PRs opened yet. Production deployment implementation milestone (uncommitted) - Added scheduler lifecycle migration/state: fresh scheduler startup invalidates prior heartbeat/drain evidence; status reports revision/protocol/health/drain/counts; resume requires a current scheduler heartbeat, complete drain and no workers/running jobs. Added embedded migration manifest for pre-drain rollback compatibility checks. Full Rust suite now has 37 tests and passes. - Added protected prism_v release workflow with every external action pinned to a full commit. Tag push tests exact source, builds each image exactly once, takes the signed digest from Buildx metadata (tag races fail), anonymously verifies worker digest, and sends only a signed immutable release record to the host. Manual dispatch does not rebuild or reread mutable registry tags; it signs a one-time bootstrap/activation/fix-forward operation referencing the host's accepted immutable release. - Release upload account is unprivileged. Root-owned entrypoint snapshots regular files with O_NOFOLLOW/O_NONBLOCK into root-only storage, verifies signature/hash/workflow ordering, records exact image/config identity durably, rejects conflicting tag reuse, makes identical acceptance idempotent across workflow ordering, and consumes each operation once before execution with fsynced receipts. Root-owned Compose/deployer/helper files are not uploaded. Signed fix-forward can replace only a newer accepted non-bootstrap candidate; unresolved bootstrap cannot switch manifests. - Added acceptance-only safety: exact bind/public URL, issuer/audience/Prism subject/permissions, source origins, storage endpoint/region/bucket/env names, self receipt callback, max one worker and disabled Scaleway are deployment-enforced. No Prism Caddy route exists. Compose publishes no host ports; only control plane receives future external edge attachment and PostgreSQL stays private. Resource limits/log limits are set. - Added encrypted Restic backup/restore checks, exact repository identity recording/revalidation, single-line password policy, explicit bootstrap initialization state, attempt-labeled database volume ownership/revalidation, safe surviving-candidate stop before retry migration, root-only credential re-snapshot recovery, schema-safe rollback refusal and daily timer design. Deployment tests cover bootstrap paused-by-default, later activation, interruption fail-paused, retries/reconcile, fix-forward, rollback ordering and host isolation. - Validation currently passes: Rust fmt/check/strict Clippy and 37 Rust tests; 45 deployment/helper tests; Compose rendering; actionlint; whitespace; changed-file private-key/tracking scans. Multiple parallel feature reviews found and drove corrections for registry/tag substitution, uploaded root code, TOCTOU, action pinning, operation replay, bootstrap ownership, scheduler staleness, Restic ambiguity and recovery paths. Latest reported issues were corrected and the full suite rerun. - Added Ansible prism_deploy role on feature/prepare-prism-deployment: creates upload-only system account, exclusive public key, root-owned engine/Compose/signers, narrow verifier-only sudoers, non-overwriting secret placeholders and native prerequisites; it starts no Prism service, database, route or GPU. YAML lint, syntax, production-profile ansible-lint, 18 existing edge tests and 5 new role tests pass. Canonical Prism deployer files were recopied after final edits. - Configured PRISM_DEPLOY_SSH_KEY, PRISM_RELEASE_SIGNING_KEY and PRISM_DEPLOY_KNOWN_HOSTS in Gitea and committed only distinct public keys to the uncommitted Ansible tree. 1Password CLI timed out while locked before any item/secret was created; partial state was checked and orphan public keys removed before regeneration. The initial Gitea-only deployment/signing identities were rotated before use. New private keys were created, stored and read-back verified in the Kaukus-Server vault, then installed as PRISM_DEPLOY_SSH_KEY and PRISM_RELEASE_SIGNING_KEY. 1Password items: Prism CI deployment upload key (rol66iickvarrka5hloikafpwu) and Prism protected release signing key (rmc6qdhzyte5hvxgcnrvnjqc5y). Matching distinct public keys were committed/pushed to the Ansible branch as 4d85aed (`chore: Rotate Prism deployment identities`). Temporary private files were deleted automatically. Role tests and private-key scans passed; no private material entered Git. - No host prism-deploy role convergence, runtime secrets, artifact/backup buckets, release tag, Prism containers/database, Caddy route, GPU worker, application integration or processor cutover occurred. Prism PR #2 https://git.kaukus.no/Kaukus/prism/pulls/2 and Ansible PR #6 https://git.kaukus.no/Kaukus/Ansible/pulls/6 were opened non-draft, verified mergeable and merged by explicit user request without deleting branches. Remote main revisions verified: Prism b4c4a1f (`feature: Add protected production deployment`), Ansible b689c96 (`feature: Prepare Prism deployment host`, containing d881135 and key rotation 4d85aed). Architecture correction requested by user before host preparation: follow Murmur's current deployment trust model for Prism and defer a shared hardened deployer redesign across services. Keep application-specific deploy.py/release.py/Compose canonical in Prism and upload/execute the reviewed release bundle through the dedicated deploy account with sudo, accepting that the deployment key/release source has root authority. Ansible should own only account/key, directories, packages, sudo policy and non-overwriting host credential placeholders—not copies of Prism deployment logic or Compose. Remove the separate release-signing operation model and unused signing key from active CI/Ansible configuration; the password-manager item may remain as an unused archived credential record. No host prism_deploy convergence has occurred, so correction is source-only. Corrective source implementation is complete but uncommitted on Prism branch refactor/align-deployment-with-murmur and Ansible branch refactor/limit-ansible-to-prism-host-prep. Prism again follows Murmur's current model: deploy.py/release.py/Compose/config/manifest are uploaded into a numeric root-owned release directory and that uploaded deployer executes with sudo. The release workflow builds or reuses matching identity-labeled version tags, assembles digest-pinned release.json and passes bootstrap/activation/backup/fix-forward inputs to the bundle deployer. Scheduler lifecycle, acceptance-only safety, backup/restore, volume ownership, rollback and fix-forward safeguards remain. Ansible now contains only host preparation: dedicated account and exclusive upload public key, root-owned directories, Restic/Skopeo, non-overwriting credential placeholders and fixed install/find/tar/Python sudo commands scoped to numeric Prism release paths. All copied Prism deployer/Compose/verifier files and the signing public key were removed. The unused PRISM_RELEASE_SIGNING_KEY Gitea secret was deleted; its password-manager item remains unused and may be archived later. PRISM_DEPLOY_SSH_KEY remains backed up and active. Validation after correction passes: Prism fmt/check/strict Clippy, 37 deployment tests and 37 Rust tests, Compose rendering and actionlint; Ansible YAML lint, syntax, production ansible-lint, sudoers parse and 4 role tests. No host role was applied and no Prism runtime resource changed. Corrective branches were committed, pushed, opened and merged by explicit user request. Prism PR #3 https://git.kaukus.no/Kaukus/prism/pulls/3 merged as main revision 32b5422 (`refactor: Align deployment with Murmur`). Ansible PR #7 https://git.kaukus.no/Kaukus/Ansible/pulls/7 merged as main revision ffb61ef (`refactor: Limit Ansible to Prism host preparation`). Remote main refs verified. Validation before merge passed: Prism 37 deployment tests, 37 Rust tests, strict Clippy, Compose and actionlint; Ansible YAML/syntax/production lint, sudoers parse and 4 role tests. Acceptance control-plane deployment completed. Applied merged Ansible prism_deploy role: dedicated account/key, root-owned directories, Restic/Skopeo, credential placeholders and scoped sudo. Created private versioned `prism-artifacts` and `prism-backup` buckets in dedicated Prism project. Created separate runtime/backup IAM applications with bucket-read plus object read/write/delete permissions; credentials, database/worker secrets and Restic password are mode-0600 on host and backed up in Kaukus-Server. Runtime artifact credentials passed a real put/head/get/delete round trip; backup credentials initialized Restic and bootstrap uploaded, checksum-verified and isolated-restore-tested a PostgreSQL snapshot. Released protected tag prism_v0.1.0 at revision 32b5422. CI run 4416 built and published control digest 5c5763c2815c54f0be287f26188b0189b3944f7c96d9e351ff24f52cf774e0d6 and public worker digest d458b1a85b3c27752c29f4e37384d877675e7e59dd3f93d8253fe41d89e7b879; initial deployment correctly stopped before bootstrap after exposing a sudo rule that required trailing flags. Fixed/apply sudo rule and merged Ansible PR #8 (fix/allow-initial-prism-release). Bootstrap dispatch run 4417 reached database ownership and exposed the installed Restic/S3 missing repository response (exit 1 with exact missing-key diagnostic). Applied narrow compatibility patch to recorded deployer, retried the recorded candidate successfully, and merged Prism PR #4 (fix/detect-missing-restic-repository) for future releases. Live state verified: Prism control plane and PostgreSQL running with no host ports; only control plane joins kaukus-edge-prism; PostgreSQL stays on prism_default. Revision 32b5422, protocol 1, scheduler healthy, paused=true, drained=true, zero jobs/workers/callbacks. Backup timer active. Added/merged Prism Caddy site through Ansible; HTTPS /health returns 200 with HSTS/nosniff. Live ZITADEL identity-check verifies Prism subject 398311983545319429, and external OAuth-authenticated GET /v1/jobs returns 200 empty. Autoscaling remains disabled, no GPU resources allocated, no producer/consumer integrations deployed, and no processor cutover occurred. Deployment is intentionally paused for acceptance. Real GPU acceptance started by explicit user request. Created dedicated fr-par-2 Prism worker security group 8fbf0cb0-fefc-4bc5-b8e5-2b5234d481dc (stateful, inbound default drop, outbound accept) with only TCP 8080 from 151.115.147.97/32. Created and backed up provider IAM application b9e6254c-efa6-40b1-9921-3f5e73998952 with Instances/BlockStorage/IPAM access in the dedicated Prism project; secret installed root-only. Enabled exactly one L4 profile through merged Prism PR #5 and successfully deployed/activated prism_v0.2.0 at revision 27efe2883866f17d80fae6d8b420a03bfaf6014b (CI run 4422 release, 4423 activation). No Kaukus/router integrations enabled. Submitted real Skien source as job c7172267-8d99-4b01-a091-57b921d725c1 with exact source SHA256, balanced full render, transcription-audio preparation and self acceptance callback. Live evidence: initial allocation attempts reported L4 out of stock and were cleaned/retried; provider server 070e7c6e-ded4-4142-a046-6d84da8f8cd5 was eventually allocated but remained stopped and was retired/cleaned; a subsequent worker attempt entered provisioning. Job remained queued during observed transitions; no output was published. Monitoring remote_python session then failed with SSH transport timeout (`server 151.115.147.97 not responding`, exit 255) while the service/provider work may still be running. Per remote-access policy, no automatic rerun, raw SSH fallback, identity/config change or alternate access attempt was made. User reports normal SSH connectivity should be restored and requested continuation. Inspect existing Prism job, worker rows and provider inventory read-only before any retry or resubmission. Do not assume the timed-out monitor changed remote work. Remaining gates are completion/cleanup of this live GPU acceptance, independent artifact validation, source-format parity/retention/quarantine, and integration implementation/cutover approval.