Investigate Rust service compilation and release build times
Investigate Gitea Actions run 4057 and comparable releases to explain compilation costs for larger Rust services. Read-only investigation; no production changes or implementation authorized. Inspect job/BuildKit logs, release workflow and Docker cache lifetimes, Cargo profiles, dependency/features and service structure. Deliver evidence-backed prioritized recommendations with measurement/validation plan; distinguish observed timings from unbenchmarked hypotheses and disclose unavailable runner or compiler profiling data. Acceptance: quantify the target run's critical path, verify cache behavior against source and related runs, identify reusable improvements and relevant risks, and provide actionable phased recommendations. Verified evidence: run 4057 attempt 2 (local-streamer v0.1.1) totals 353s; build/publish step 335s, Cargo 228.7s (467 compile notices, 475 downloads), builder-layer materialization ~34s, runtime-layer materialization ~25s, export/cache/publish remainder. Registry layer import and cached NDI/WebRTC plugin build work, but changed-source Cargo layer rebuilds cold. release.yaml creates docker-container release-builder then removes it without --keep-state; registry mode=max does not make mount contents part of regular cached layers. Read-only SSH inspection of 5.9.138.8 confirms zero rootless containers/volumes and no retained release builder. Host i5-13500 (20 threads), ~64GiB RAM, NVMe RAID1; build cgroup CPU quota currently unlimited via override (repository unit still says 800%), 24GiB memory limit, ~17.7GiB historical peak, zero memory/OOM limit events. Historical CPU throttling exists but cannot be attributed to a particular run. No remote changes or builds performed. Comparison logs: 3941 ruststreamer recompiles 493 crates in 166.2s despite registry cache; 4074 cloud-router recompiles 241 in 142.5s. Initial 3927/3928 builds were much slower (1073.6s/1006.7s Cargo respectively), with no registry cache; differing host conditions prevent assigning the entire speedup to caching. CI 4084 separate Check/Test containers redownload the same workspace dependencies; check Cargo 56.17s and test Cargo 85s plus ~31s test execution. Runner has a cache service enabled, but ci.yaml never restores/saves Cargo caches. Dependency-only experiment in a /tmp git archive, not working tree: optional S3 dependencies in gst-recording reduce local-streamer's target-specific normal/build package-version graph from 467 to 392; selecting image png/jpeg/webp/gif reduces it further to 355 (112 packages / 24%). Cargo tree --offline --locked succeeds for all variants. Source inspection confirms local-streamer calls transcode-only then tus queue, while unconditional shared S3 pulls aws-config/S3/SSO/STS. This is graph validation only, not a compiled or runtime-tested patch; source feature gates and ruststreamer's explicit S3 opt-in are still required. image defaults currently pull rav1e/ravif/EXR/TIFF; ruststreamer's direct image dependency also needs alignment. Root Cargo release profile enables full debug info globally; line-tables-only is a benchmark candidate, preserving debugging requirements. Investigation complete; recommendations prepared for delivery: first preserve same-name BuildKit state with docker buildx rm --keep-state on the existing rootless release daemon (or reuse a persistent dedicated builder), keep credential cleanup and trusted-release-only cache boundary, and set a bounded GC/disk policy. Registry cache remains a layer fallback, not mount backup; cargo-chef layer-based dependency builds are the portable fallback already demonstrated by cascade, not a necessary new first step. Next make gst-recording S3 opt-in for ruststreamer, align required image codecs across direct dependencies, cache Cargo registry/git/target in Gitea-compatible CI and colocate serial Check/Test to share artifacts, and preinstall recording test runtime in a pinned Rust-specific runner image. Benchmark line-tables-only debug info, faster linker, incremental release or sccache only after cache correctness and --timings identify remaining costs; preserve DSP optimization and core-dump needs. Pin toolchains/base identities intentionally; retain original runtime isolation and release serialization. Do not attribute old/new duration differences to cache or predict a wall-time gain from graph counts. Acceptance for follow-up implementation: compare same source/toolchain/base/profile cold, identical repeat, executable-only edit, shared-crate edit, version-only edit and lockfile-change builds; bypass immutable-image short-circuit and do not publish benchmark images. Repeat measurements and retain Cargo HTML --timings, actual rustc/linker versions, downloads/compiled/fresh units, wall time, cgroup CPU throttling/memory and cache sizes. Warm unchanged deps should be Fresh with zero crate downloads; private dependency access, version embedding, tus recovery and ruststreamer S3 behavior must remain verified. No source edits, host configuration changes, expensive host benchmarks, image publication, commits or deployments were performed. Temporary graph changes exist only under /tmp/streaming-compile-investigation/graph-experiment; working tree remains clean.
Checklist
2 of 2 complete- Quantify target and comparable runs, including compilation, caching and non-compilation overhead
- Verify dependency, profile and Docker/workflow causes and deliver prioritized recommendations with validation plan
- Area
- Streaming
- Project
- streaming-engines
- Kind
- Research
- Task ID
tsk_b839t8acn6- Updated