Fix ruststreamer shutdown crashes and recover interrupted uploads

Goal Make ruststreamer pipeline shutdown reliable. If the native crash cannot be clearly isolated, ensure container restart automatically completes interrupted recording processing and S3 upload; restarting stopped live streams is outside scope. Reported incident 2026-10-01: stopping stream 18642 with transcode=true logged final segment metadata for kviteseid-kommunestyresalen-salen, reported that source fully stopped, then logged final metadata for kviteseid-kommunestyresalen-talerstolen. Container exited with code 139, OOMKilled=false, without further application logs. Exit 139 is consistent with SIGSEGV, but the responsible native component is not confirmed. Container restart currently does not resume upload. Repository findings / investigation targets - ruststreamer/src/pass_through/mod.rs: stop_source ignores returned stop errors; timeout messages claim leaking while the SourcePipeline wrapper is still dropped and the shutdown worker may remain active. - ruststreamer/src/pass_through/source/pipeline.rs: duplicated stop paths ignore state-wait results before reporting fully stopped; metadata is saved before callbacks quiesce, without explicit recording EOS drain. - ruststreamer/src/stream_manager.rs: transcode/upload is scheduled only after teardown, so a teardown crash loses upload intent. - ruststreamer/src/transcoder.rs: stream directories are selected by name prefix; individual source failures are logged without failing the overall job. These are hypotheses and concrete weaknesses, not an established segfault diagnosis. Proposed approach 1. Reproduce repeated multi-source stop, segment rollover, rebuild, and unavailable-input scenarios using production GStreamer/plugin versions; collect native cores and all-thread backtraces with source-specific lifecycle logs. 2. Consolidate shutdown lifecycle: verify state transitions, propagate failures, coordinate recording finalization and metadata/callback completion, retain safe ownership during timeouts, and avoid false success. 3. Decision point: fix the demonstrated native cause; if diagnosis remains inconclusive, implement durable recovery. Persist transcode/upload intent before teardown; resume pending work on startup with bounded retries and durable completion only after every required source output succeeds. 4. Unify stop-triggered, manual, and recovered processing through one job runner. Use exact recording directory identities, prevent duplicate concurrent work, and report partial failures accurately. Reuse / constraints Reuse gst-recording segment validation, concatenation, bookmark reconstruction, and S3 retry helpers. Follow local-streamer/src/commands.rs boot recovery and atomic completion-marker patterns without importing its tus-specific upload queue. Keep ruststreamer job orchestration local; extract shared helpers only for actual reuse. No major architecture rewrite or new dependencies without approval. Preserve recordings on failure and transcode=false behavior. Recovery requires persistent recording/job storage across container restart. Acceptance criteria - Repeated stop/rebuild validation does not crash or falsely report shutdown success; stop errors and timeouts remain observable. - Clean shutdown retains final playable segments and bookmark metadata. - A native cause is demonstrated and fixed, or the unresolved diagnosis and recovery fallback are documented and verified. - If fallback recovery is implemented, termination during teardown, transcode, or upload resumes pending processing after restart without concurrent duplicate jobs or premature completion. - Partial source/upload failures do not report overall success; corrupt trailing segments, missing credentials, similarly named streams, duplicate requests, and transcode=false are handled safely. - Affected Rust format/check/test/clippy validation passes; container restart testing confirms persistent-storage recovery when implemented. Outstanding inputs Obtain deployed image digest/GStreamer versions, container recording mounts, and any core dump. Confirm whether recovery covers only explicitly recorded stop/manual-transcode jobs or also older orphan recordings. Default proposal: explicit pending jobs, with deliberate legacy recovery rather than uploading every directory automatically. Planning only: leave open; no implementation has started. Planning findings (2026-10-02, repository HEAD 3ea3cec) Current execution and failure boundaries - POST /streams/stop calls StreamManager::stop_stream_with_transcode synchronously. The stream is removed from the manager, PassThrough::end_stream tears down each source, the stopped callback is sent, and only then is a detached blocking transcode/upload task created. A process death anywhere in teardown therefore leaves no durable upload intent. - PassThrough::stop_source removes SourcePipeline ownership before waiting, accepts Ok(Err(_)) as success, and drops the wrapper after a 5-second timeout while the worker owns only a gst::Pipeline clone. SourcePipeline::stop and stop_async both ignore gst::Element::state wait results and log “fully stopped” after only set_state(Null) succeeded. - SourcePipeline carries a BusWatchGuard, probes, BookmarkHandler, and optional flow monitor. Teardown has no stopping flag, so bus errors can still enqueue rebuilds. With audio enhancement enabled, gst-audio-enhance also has a detached worker holding appsrc; it is another native-lifetime boundary to include in reproduction, not a proven cause. - BookmarkHandler saves the final sidecar before EOS and spawns unjoined writes at format-location. splitmuxsink requires EOS to finalize the last MP4 and emits splitmuxsink-fragment-closed only after fragment closure; metadata duration probing can currently race file finalization. - Recording sessions are written as ./data/{stream_name}{source_name}/{timestamp}. Transcoder scans every directory whose name starts with the stream name, so similarly named streams can be mixed. Rebuilds create more timestamp sessions, but PassThrough does not retain exact session identities after a source is replaced. - Transcoder logs a per-source failure and still returns Ok; no recordings also returns Ok([]). gst-recording::run_transcode_and_upload likewise treats missing credentials as a successful local transcode with zero uploads. The uploaded callback and success metric can therefore report incomplete work. - Manual transcode uses the same prefix scan and can run while a stream is still writing. Stop, manual, and recovery need one exact-target runner; manual processing of an active recording must be rejected or explicitly quiesced, not raced. - Ruststreamer has no processing tests. gst-recording only tests bookmark serialization/reconstruction. local-streamer provides reusable examples for serial boot recovery, exact path layouts, idempotent queue checks, and atomic completion markers. Implementation constraints and chosen fallback shape - Keep gst-recording APIs used by local-streamer compatible. Add explicit-session transcode/finalization helpers and retain the existing input-root wrapper for current callers. - Have each ruststreamer pipeline expose and retain a generated recording run/source/session identity and path. Persist a small session manifest containing exact stream/source names; do not infer ownership from path prefixes. A processing job snapshots ordered exact session paths grouped by required source. - For transcode=true stop, atomically persist the job before starting teardown. If intent persistence fails, return an error without claiming a successful stop. transcode=false creates no job and startup recovery must not auto-upload those recordings. - Proposed ruststreamer-local job store: versioned JSON intent/state under the configured recording data root, written through same-directory temp file + sync + rename, plus an atomic completion marker only after every required source artifact has uploaded. No new database or dependency is required. Interrupted Running/AwaitingShutdown jobs become recoverable at startup because no live pipelines are restored. - Use one serialized background job runner for stop, manual, and startup recovery. Deduplicate by exact ordered target set, guard queued/running job IDs in memory, persist bounded attempt counts and per-source completion, and treat missing credentials, zero valid segments, any required source failure, or any required upload failure as non-completion. S3 object PUT retries may repeat after a crash but remain idempotent at the stable object key. - Preserve raw recordings and failed/interrupted outputs. Write concatenated output to a temporary path and rename only after ffmpeg succeeds. Filter bookmark reconstruction to the exact validated segments so a corrupt trailing segment cannot skew the timeline. - Default legacy policy is explicit jobs/manifests only. Existing orphan directories without manifests are left untouched and logged/documented; adding an opt-in legacy importer is a separate deliberate decision. - Startup recovery belongs after GStreamer/S3/callback initialization and before serving API traffic. It resumes processing jobs only; stopped live streams are never restarted. - The data root holding recordings, manifests, jobs, final outputs, and completion markers must be one persistent mount. ruststreamer/Dockerfile currently declares none, and no ruststreamer deployment manifest exists in this repository, so the runtime mount path must be confirmed and documented before operational acceptance. Diagnostic and verification requirements - The repository does not contain the deployed image digest, exact native package versions, core dump, or ruststreamer deployment/mount configuration. The Debian trixie base and apt packages are not digest/version pinned. Capture image digest, gst-launch/gst-inspect/ffmpeg versions, loaded plugin versions, core_pattern/ulimit, and an all-thread native backtrace before attributing the SIGSEGV. - Add lifecycle logs keyed by stream, source, recording session, and processing job around EOS request, fragment-closed, callback/metadata drain, Paused/Null request and confirmed state, timeout, worker completion, retry, per-source upload, and durable completion. Do not log secrets. - Reproduction matrix must cover multi-source stop with and without audio enhancement, stop during segment rollover, unavailable input, source rebuild/update, repeated start/stop, and forced termination during teardown/transcode/upload. Run it in the production image/plugin set with the persistent data mount. - Automated coverage must include exact-name selection for similarly named streams, duplicate job submission, interrupted Running recovery, bounded/terminal retries, partial source failure, missing credentials, corrupt trailing segments with correct bookmarks, transcode=false exclusion, and completion/callback only after all required uploads. Resolved production inputs (2026-10-02) - Incident image ID: sha256:0923ebb243dacab73a45b49ffe9e424256beaab9efac0646be3327b425b5340b. - Immutable repository digest: cr.kaukus.no/kaukus/ruststreamer@sha256:7aefdb1fda709461430871a80e6781e72e51d94db6650cc40aa415bca7181859. - Image creation time: 2026-08-19T10:00:55.233181792+02:00; amd64; working directory /app; command /app/ruststreamer. - Recording data is already backed by a persistent Docker volume and remains available when the service restarts. Recovery job state and completion markers must be stored under that same mounted recording data root. Do not require a new volume or move the container data path without coordinating the deployment mount. - Remaining diagnostic inputs are the exact runtime GStreamer/plugin/ffmpeg versions and any core dump/all-thread backtrace. Pull request - #122: https://git.kaukus.no/Kaukus/streaming-engines/pulls/122 - #124: https://git.kaukus.no/Kaukus/streaming-engines/pulls/124 — Recover source rebuilds and prevent poisoned image caches Oct 8 source-recovery regression - Incident 18616: GStreamer reported text/plain in the cached pause-image input for both source pipelines; the actual incident cache bytes have not been inspected. A rebuild removed one source, then recording finalization timed out; early-return processing discarded the other queued rebuild and left the removed source with no retry. - Follow-up branch fix/recover-ruststreamer-source-rebuilds retains failed rebuild requests across monitor ticks, processes every requested source independently, waits for pending shutdown workers, and rebuilds after completed shutdown failures while retaining finalization errors for processing safety. Missing sources remain eligible for recreation after failed stop/start attempts. - Added regression tests for independent retained retries, shutdown ownership, and retained finalization failures. Targeted tests and cargo check pass. Full suite currently requires locally missing GStreamer avenc_aac; strict clippy reports a pre-existing while_let_loop lint in source/pipeline.rs. Production RTMP recovery and invalid pause-image handling remain unverified; this fix does not repair invalid downloaded image content. - Pause-image investigation: the exact URL currently returns HTTP 200 image/jpeg, 1,522,236 bytes, valid 1920x1080 JPEG; local GStreamer typefinding and decoding succeed. Current image SHA-256 is 33f890de3a08509a18c278c27406d75d14e737b774cd1ad4a3acc4fe5036ecc5. URL hash matches the incident cache filename exactly. - Confirmed downloader weakness in crates/gst-overlay/src/download.rs: curl lacks HTTP-error rejection, writes directly to the final cache filename, and cached content is never validated. A controlled HTTP 502 text response exits curl successfully and saves a .jpg; sufficiently long plain text saved as .jpg reproduces the exact decodebin/typefind error pair from the incident. Confirmed by the user inspecting the incident container cache: the .jpg contains a plain-text proxy error reporting failure to send the upstream S3 GetObject request. The origin image is valid; a transient upstream request failure poisoned the cache. The specific upstream failure and HTTP response status remain unknown. Do not preserve or log the error body's signed URL. Downloader follow-up implemented on fix/recover-ruststreamer-source-rebuilds: curl rejects HTTP errors and restricts redirects to HTTP(S); transient HTTP/network failures have three bounded attempts with 1s/2s backoff, 5s connect and 15s request timeouts. Downloaded and cached images are fully decoded with the existing image dependency (20 MiB file, 8192-pixel side, 128 MiB decode-allocation limits). Invalid cache entries are removed; unique temporary downloads are atomically renamed only after validation and cleaned on failures. Per-entry OS file locks serialize concurrent callers across processes; lock files deliberately persist. Download errors/logs exclude response bodies and raw URLs; logo-resolution logging also omits remote URLs. No new dependencies or blackout fallback introduced. - Validation: 12 deterministic gst-overlay local-server tests cover transient and permanent HTTP errors, poisoned/valid caches, HTTP-200 text/truncated images, interrupted transfers, concurrent publication, timeout/size/dimension limits, special-character URLs, and URL rejection. Ruststreamer uses valid-cache and invalid-URL compatibility tests instead of external-network tests. Workspace all-target check, formatting, diff checks, and workspace tests pass with the known avenc_aac-dependent integration test explicitly skipped. gst-overlay strict clippy passes; ruststreamer clippy passes only when its pre-existing while_let_loop lint is allowed. Production-image RTMP/rebuild validation remains outstanding; live service was already recovered by container recreation. Oct 21 stop/upload investigation (investigation only; no fix authorized) - Stream 20181, source agder-produksjon-produksjon, session 2c485d83-308d-4474-9f33-f790574ac535, processing job 307dfa40-6d11-4374-9c9d-110f3507f003: recording EOS requested, segment_00803.mp4 closure logged, final metadata save logged, then stop rejected with duplicated Playing -> Paused pending errors. User reports having to restart it; what was restarted is not yet confirmed. - Current source/pipeline.rs classifies any intermediate Paused confirmation error as recording finalization failure, retains it even if subsequent Null and metadata operations succeed, and stream_manager.rs marks the processing job Failed instead of enqueueing it. This explains upload suppression without an S3 failure. Actual recording integrity is not established by fragment-closed/save-start logs alone. - Current processing recovery explicitly skips Failed jobs. Existing shutdown_finalization_failure_does_not_poison_future_processing test passed locally and verifies this terminal behavior. Manual processing can create a fresh job because matching excludes Failed jobs; restarting the service alone would not recover this terminal job under current code. - Deployed revision remains unresolved: supplied logs say source pipeline fully stopped despite failure and show shutdown completing in ~0.49 seconds; current source only emits fully stopped when its local failure is empty and uses a 5-second state wait. Identical duplicated errors may be retained earlier rebuild failures, but the excerpt is insufficient to establish their origin. - Next evidence: deployed image/revision, full earlier source rebuild/shutdown logs, persisted job.json for this job, clarification of restart action, and ffprobe/sidecar validation of final recording. Potential future fix direction is distinguishing confirmed recording safety and final Null teardown from failure of an intermediate Paused transition, without hiding real EOS/metadata/Null failures. No repository edits made during the Oct 21 investigation; native pause failure was not reproduced then. Oct 29 follow-up (implemented locally) New logs for stream 21025 show both recording fragments closing after input flow stops, followed by a five-second Paused timeout during source removal. Those errors are retained until the later transcode=true stop and suppress processing even if flow subsequently recovers. Native local GStreamer regression reproduces the exact Playing with Paused pending failure under blocked audio/video input, even though final Null succeeds. On fix/finalize-recordings-without-pause, removed only the intermediate Paused request/confirmation from stop_owned; recording EOS/fragment, final Null and metadata failure checks remain unchanged. Extended the existing recording integration test to exercise the real stop_owned path with running, stalled and demonstrably recovered input, confirming Null, readable MP4s and sidecar existence. Added rejected-EOS and incomplete-EOS/fragment barrier tests. The final regression fails with the old pause code and passes with the fix. All 37 ruststreamer and 10 gst-recording tests pass; cargo fmt, cargo check --workspace, strict affected-crate all-target Clippy and diff checks pass. Only source/pipeline.rs changed; no dependencies or deployment. Committed as 7c8f7eb (fix: Avoid pause timeouts during recording shutdown) and pushed with upstream tracking; worktree clean. No production-image RTMP/S3 acceptance was performed. Existing failed recordings require deliberate integrity validation/manual reprocessing; historical finalization failures are not automatically reclassified. Broader native-crash and production recovery acceptance remains outstanding. Pull request #144: https://git.kaukus.no/Kaukus/streaming-engines/pulls/144 — fix: Avoid pause timeouts during recording shutdown. Head fix/finalize-recordings-without-pause; base main. Open and ready for review. Full base comparison contains only this session's shutdown fix and regressions in source/pipeline.rs. Status unchanged by publication.