Replace Kaukus Swiftner transcription with Murmur
Replace the production Kaukus chunked-transcription path with Murmur in one irreversible hard cutover, preserving safe retries and subtitle output but no Swiftner runtime path or rollback implementation. Scope: - Add idempotent Murmur job creation keyed by the stable Kaukus audio-chunk identity before routing production retries to Murmur. - Build a reusable Kaukus Murmur API client around the existing ZITADEL OAuth client, with bounded timeouts, one safe token-refresh retry after 401, typed errors and no token/media-URL leakage. - Add explicit Murmur job/result/error state to audio chunks rather than overloading Swiftner IDs or remote transcription URLs. - Submit extracted public audio chunks to Murmur in `nb`, `nn`, or `da`, poll queued/running/terminal state through the existing durable light-lane job, persist word-aligned results and Murmur-generated canonical WebVTT, and reuse the existing offset-aware final subtitle combiner. - Replace municipality `swiftner_org_id` gating with provider-neutral transcription enablement and remove Swiftner-specific user-facing copy. - Use a planned weekend hard cutover when Kaukus has no active transcription work. Preflight must prove there are no extracting/submitted chunks or queued/running transcription jobs before writes are switched. - Do not build provider coexistence, dual-write, replication, runtime fallback, or rollback code. Murmur becomes the sole path in one maintenance window. - Remove Swiftner client code, environment variables, organization IDs, chunk IDs/URLs and user-facing configuration in this branch. Production issues after cutover are fixed forward. Behavior decisions to preserve: - Murmur is the sole provider for new work after cutover. - Source audio is deleted only after a terminal result has been safely persisted/combined; retryable failures retain enough state for recovery. - A failed chunk must not silently produce a partial successful meeting transcript. - Stable idempotency prevents duplicate GPU cost when Kaukus retries an uncertain POST. - Unsupported municipality languages fail configuration/eligibility explicitly rather than being mislabeled as Bokmål. Acceptance criteria: - Repeated submission of one audio chunk produces one Murmur job and one billable transcription. - Successful multi-chunk nb/nn/da jobs produce correct combined WebVTT with offsets and word timing in Kaukus, with canonical per-chunk WebVTT generated and validated by the Murmur control plane. - OAuth expiry, 401 refresh, network timeout, queued/running polling, retryable/non-retryable Murmur failures, job restart and duplicate delivery are covered. - The cutover preflight refuses to proceed unless all existing Swiftner transcription work is terminal and the relevant Kaukus queues are empty; the deployed application contains no Swiftner submission or polling path. - The admin/UI eligibility model is provider-neutral and translated in all required locales. - A staging/production rehearsal uses max one worker, validates result quality and cleanup, and supports disabling new submissions without losing state. - Swiftner code, credentials and obsolete schema are removed by this branch and the empty-queue migration succeeds in the cutover rehearsal. Current state: - Murmur idempotency, Rust-owned canonical WebVTT, and Danish model support are committed and pushed to `main` at `a3f7a16`. - The Kaukus replacement consumes Murmur WebVTT and supports nb/nn/da at `b49c3e828`; PR #4230 is updated. - The live Skien (`nb`) and Aabenraa (`da`) matrix passed on L4 and corrected P100 profiles. Every accepted job completed in one attempt with the expected language-specific model, canonical WebVTT, stable idempotent replay and 409 conflict behavior. - L4 (`float16/pyannote/cuda/cuda`) became ready in 196.8s; nb took 4.3s (115 words/18 cues) and da took 2.2s (127 words/12 cues). P100 initially exposed a Silero regression that truncated Danish at 23s; the profile was fixed to `float32/pyannote/cpu/cpu` and became ready in 286.9s, then nb took 14.2s (115 words/18 cues) and da took 9.5s (127 words/12 cues). - Both corrected workers deleted themselves after 60 idle seconds. Final production state uses the corrected image digests with provisioning disabled, no queued/running jobs, no worker rows or provider servers, temporary media/routes/tokens removed, HTTPS health 204, and an idempotent Ansible rerun with zero changes. - A deeper Edda run passed against the complete 6,220-second Aabenraa recording as four Kaukus-style chunks on one bounded L4 worker. All chunks succeeded in one attempt in 48.9s, 47.7s, 48.9s and 21.4s after startup; combined output has 15,851 text words, 15,865 aligned words, 1,175 cues, timing through 6,215s and a 1.3% repeated five-gram ratio. Idempotency and conflict checks passed for every chunk, cleanup completed, and a VLC-ready MKV with embedded Danish subtitles was produced locally. - Prompt support is pushed as Murmur `0941906`, Ansible `49e78a8`, and Kaukus `08a1c0e03`. Kaukus reuses active member/party vocabulary, snapshots each chunk prompt before submission, and Murmur persists but does not expose it in job responses. - Full prompted A/B on identical L4 hardware selected Edda turbo over large: turbo processing was 178.7s with 215 uppercase characters, 1,287 punctuation marks and 97.44% baseline word-sequence similarity; large took 388.2s with 97 uppercase characters, 80 punctuation marks and 96.02% similarity. Prompting helps casing/punctuation but does not reliably case all proper names or places. - Worker Docker stages now cache Norwegian ASR, Danish ASR, and fixed alignment/VAD assets independently; after one cold build the large variant took 181s and switching back to turbo took 1.8s. Comparison transcripts/VTT and a three-subtitle-track VLC review MKV are stored locally. - Human review selected prompted `openai/whisper-large-v3` for Danish while `NbAiLab/nb-whisper-large` remains selected for Bokmål/Nynorsk. On the full L4 run, prompting raised exact-case known-person matches from 5 to 19 (including `Philip Tietje`) with 97.29% word-sequence similarity; prompted processing took 409.1s versus 313.2s unprompted. - Production worker configuration is L4-only. P100 batch size 4 cannot run 30-minute prompted Whisper chunks; batch size 1 succeeds but takes 487.8s per chunk, so it is not enabled. L4 shortages leave jobs durable while provisioning retries with backoff. - Murmur `c37705f` and Ansible `82c003d` are pushed and deployed in provisioning-disabled configuration. Kaukus `6c9be7664` addresses review findings by keeping polling errors inside the durable 12-hour reschedule loop and handing uploaded chunks to durable idempotent submission retries; focused transcription coverage passes 94 examples. - Murmur `5c4cf26` and Ansible `9266db9` implement and deploy reading-speed-aware subtitle segmentation. On the full prompted Whisper meeting, cues below Netflix's 5/6-second minimum fell from 91 to 0, cues below 1.5s from 184 to 2, over-84-character cues from 440 to 0, over-42-character lines to 0, and over-20-CPS cues from 267 to 93. Median output is 3.6s/55 characters; p90 is 5.0s/76 characters/19.3 CPS. The VLC review file defaults to the improved cue track. - Murmur `0b4639c` and Ansible `1c9f0a5` add and deploy deterministic prompt-glossary normalization before WebVTT generation. Compact-equivalent variants (`Åben Rå`/`Åben Råd`) and one-edit name variants are rewritten to authoritative prompt terms (`Aabenraa`, `Philip Tietje`) while timings and unrelated wording remain unchanged. Rust checks, 43 tests and strict Clippy pass; the VLC review defaults to the corrected glossary/cue track. - PR #4230 was merged and deployed through the empty-pipeline hard cutover after the production backup was taken. - Historical Ulstein pause probing uses the exact 1:41:33–1:53:38 interval with three minutes of padding. Raw L4 output after restoring Whisper's `<|nospeech|>` token contains zero segments and zero words in the detected quiet interval (clip seconds 172.43–920.22); speech ends at 171.370 and resumes at 920.700. The prior control-plane failure was unrelated to the pause: forced alignment assigned `kommunedirektør` a 14.504-second word duration after the pause, which exceeded the seven-second cue bound. Cue rendering now caps display time while preserving source word timing and successfully renders the complete captured result with no cue overlapping the pause. Source inspection also found that WhisperX 3.8.6's batched generation does not request or apply `no_speech_prob` despite exposing `no_speech_threshold`; Murmur now restores Faster Whisper's standard probability/log-probability gate in batched inference. The fixed live result is byte-for-byte identical to the baseline, including zero words in the pause. Murmur `2633b3a` and Ansible `ad000bc` are deployed; temporary media and capture files are removed. - The production Kaukus hard cutover is deployed and healthy. Migration `20261014130000` is applied, Murmur columns exist, the runtime sees `MURMUR_SECRET`, 209 municipalities are transcription-enabled, and active chunk/transcription counts remain zero. Murmur provisioning is enabled with `min_workers=0`; the Caddy API allowlist now includes the observed Kaukus container egress address as well as the host address. A production Rails client probe obtained OAuth credentials and reached Murmur, receiving the expected authenticated `404 not_found` for a synthetic job ID. Both services are healthy and no workers/provider servers were created by the probe. Final demand-driven production configuration is enabled with `min_workers=0`, `max_workers=1`, the verified Kaukus container egress allowlist, and Ansible commit `be77742` pushed to `main`.
Checklist
10 of 10 complete- Add idempotent Murmur job creation and contract tests
- Implement Kaukus Murmur API client and audio-chunk state migration
- Generate and validate canonical WebVTT in Murmur; consume it in Kaukus for nb, nn, and da
- Replace Swiftner-specific municipality gating and user-facing configuration
- Rehearse the empty-queue weekend hard cutover, backup, smoke test, and bounded worker enablement
- Remove all Swiftner code, secrets, configuration and obsolete schema in the replacement branch
- Add bounded transcription prompts and compare full Aabenraa Edda output using authoritative city/person names
- Implement readable subtitle cue segmentation, timing and line breaks against measured full-meeting output
- Normalize compact-equivalent ASR name/place variants from authoritative transcription prompts
- Verify long-pause hallucination behavior on the historical Ulstein failure clip
- Area
- Kaukus
- Project
- Deploy Murmur to production
- Kind
- Feature
- Priority
- High
- Task ID
tsk_p7ufs6pfcc- Updated