Investigate failed transcripts for meetings 21462 and 18131
Investigate why the Danish meeting 21462 and Bokmål meeting 18131 did not produce acceptable transcripts. Trace Kaukus audio-chunk extraction/submission/polling state, Murmur jobs/attempts/results, worker/provider lifecycle and persisted combined subtitle output. Establish whether each failure is operational, model/output quality, media/input, or pipeline-state related before changing or reprocessing production data. Verified findings: - IDs are meetings. Danish 21462 maps to active main edit 9709 (3 chunks); Nynorsk meeting 18131 maps to active politician edit 9701 (9 chunks). - DK chunk 2 and NB chunk 0 were recorded failed in Kaukus with `invalid_worker_result: transcript contains invalid word timing: word timings are out of order`; polling stopped after the first terminal chunk failure. DK chunk 1 also later failed in Murmur but remained submitted in Kaukus. The other 9 Murmur jobs succeeded in one attempt and already contain complete WebVTT, so recreating all audio chunks would duplicate substantial GPU work. - Source media is valid mono AAC/16 kHz: NB failed chunk 1800.064s/14.5MB; DK chunks 1800.064s/16.7MB and 973.701s/8.9MB. - Direct replay of exactly the three failed requests on the same current worker (same URLs/languages/prompts) produced coherent output: 10,278 words total, zero missing timestamps, zero within-segment word regressions, zero cross-segment overlaps and low repeated-five-gram rates (0.08–0.29%). Root-only raw diagnostics are temporarily retained under `/var/lib/murmur-deploy/diagnostics/failed-transcripts-20261029`. - Requeuing the same three existing Murmur job IDs under current code reproduced all three failures on attempt 2; no new job IDs or provider resources were created. Control logs show the failure is classified non-retryable. Several other production jobs also show the same error, so this is systemic rather than bad source media. - Root cause is control-plane glossary/WebVTT interaction: glossary replacement can collapse three recognized words into one authoritative term, clear the extra words' text, and extend the kept word's timing. WebVTT validation previously validated the now-empty words before skipping their text, creating an artificial end/start regression. PR #10 moves the empty-text skip before timing validation and adds raw three-token glossary plus WebVTT regressions; focused tests and strict Clippy pass. Recovery and final verification: - The fix is deployed in production revision 1db532e0fe1577f4ab6df342bab839c319cb9a83. Requeuing the three existing failed jobs was initially blocked only by L4 out-of-stock; no new job IDs were created. To avoid waiting/cost, the successful raw responses captured from the same worker were passed through the deployed fixed Rust glossary/WebVTT logic offline, producing complete canonical results (NB 332 segments/30,924-byte VTT; DK 317 segments/39,003-byte VTT and 151 segments/20,661-byte VTT). Scheduling was paused/drained, those results were stored on the same Murmur job IDs at attempts=2, and scheduling resumed with zero workers/queued jobs/provider resources. - Kaukus reused all existing Murmur IDs and successful chunk results. Failed chunks were reset to submitted, transcriptions to processing, and one durable poll job was enqueued per edit. Poll jobs 40702 and 40703 each completed in one attempt. Video edit 9701 / meeting 18131 completed 9/9 chunks, 203,128-byte WebVTT and 2,371 cues. Video edit 9709 / meeting 21462 completed 3/3 chunks, 95,184-byte WebVTT and 1,089 cues. Progress is 100 and `Transcription complete` for both. - Combined VTT checks pass: zero start/end regressions, zero cue overlaps, coherent first/last speech, low repeated-five-gram ratios (0.67% NB/NN, 0.75% DK), timing through 14,843.742s of 14,850.19s and 4,563.02s of 4,573.64s, and sensible text across chunk boundaries. Meeting 18131 is configured/transcribed as Nynorsk (`nn`), despite the initial report calling it NB. - Source files are valid mono AAC/16 kHz. Murmur DB has zero queued/running jobs and workers; Scaleway inventory has no managed servers, IPs or volumes. Temporary raw/canonical diagnostic files were removed from control host and workstation. - A systemic recovery weakness remains for project code: Kaukus stops polling after the first failed chunk, leaving other terminal Murmur jobs as submitted. This investigation recovered production safely without duplicate jobs; a separate follow-up can make out-of-order glossary timing retryable and/or let Kaukus persist all terminal chunk results before failing the combined transcription. - Human review found the first Danish cue misrecognized `byrådsmødet` as `bygårdsmødet`. The same Murmur result contained the correct `byrådsmøde` later at 03:05, so this was an opening ASR lexical error rather than timing/combination corruption. The erroneous term existed once semantically but in four Murmur result representations (plain text, segment text, aligned word and WebVTT) and once in the combined Kaukus VTT. All five persisted representations were transactionally corrected to `byrådsmødet`; Murmur now verifies text starts `Velkommen til byrådsmødet`, no old spelling remains, and Kaukus first cue is 00:00.543–00:04.265 `Velkommen til byrådsmødet. Vi starter`. A future prevention follow-up should add authoritative Danish council-meeting vocabulary (and the known `bygårdsmødet` variant) rather than relying on the current prompt containing only `Aabenraa`. Acceptance criteria: - Identify every affected chunk and its Kaukus/Murmur terminal state, attempts and error/result metadata. - Correlate production logs and database timestamps with worker provisioning and cleanup where available. - Inspect transcript/VTT output and source media around the reported failure rather than relying only on status codes. - Document concrete root cause and safe recovery/reprocessing action for each meeting. - Verify no duplicate billable jobs or leaked worker resources are created during investigation.
Checklist
4 of 4 complete- Trace Kaukus video edit/chunk/job state and persisted output for 21462 and 18131.
- Trace corresponding Murmur jobs, attempts, worker/provider lifecycle and relevant production logs.
- Inspect source media and transcript/VTT failure regions for each language.
- Define and verify safe recovery/reprocessing without duplicate cost or leaked resources.
- Area
- Kaukus
- Project
- Deploy Murmur to production
- Kind
- Bug
- Priority
- High
- Task ID
tsk_x7b8h6pdpd- Updated