Replace JSON capture storage with incremental binary chunks and parallel loading
Replace new capture output with a versioned custom single-file binary container: independently encoded/compressed/checksummed event chunks, committed snapshot indexes and crash-safe commit selection. Background batched writes should release payload memory promptly; rolling --frames 10000 retains only small disk references instead of all serialized frame data in application RAM. Incremental snapshots must append only new payload/index data, preserve readers during updates, tolerate interrupted writes, evict old intervals and periodically compact expired disk chunks via atomic replacement. Bound producer queue bytes, batching and parallel decode work. Reuse shared Event types, semantic validation, rolling/first modes and viewer model; support only the current binary format. Optimize real file loading with index-guided bounded parallel reads/decode, no JSON intermediary or whole-file duplicate buffers. Benchmark representative existing capture load/save/RSS/size and validate round-trip parity, corrupt/truncated files, snapshot fallback, compaction, active readers, late readbacks, 10000-interval memory behavior, first/rolling semantics and real Vulkan captures. Document format/version, resource limits and measured limitations. No commits. Scope update: user explicitly does not need old captures. Remove the repack command/full-trace conversion helper, JSON trace reader/version compatibility and buffering stream fallback; keep only indexed binary storage/loading. Retain JSON inspection output and Vulkan/config serialization. Replace old-format compatibility tests with binary validation/round-trip coverage and update documentation/benchmark tools. Implemented binary container v1 with immutable native-bincode Event chunks, independent LZ4 compression/CRC32, two CRC-protected snapshot roots and indexed commits. Disk-backed capture stores only RecordRefs; background batches process up to 64 events with a 1 MiB output buffer and 128 MiB heap-estimated queue admission. Indexes publish about once/second; append-only payloads are released promptly. Old disk history compacts by raw-copying retained chunks into an atomic replacement inode; existing readers remain valid. Semantic validation protects the binary decoder. File loading uses indexed pread and up to four decode threads, with no text/stream fallback or conversion API. The benchmark is load-only. Loader remains eager. JSON remains only for configuration and inspection output, and the trace crate has no serde_json dependency. Measured release baseline inspect (legacy binary from clean HEAD): trace7 438.77 MiB / 2.125 s / 688.7 MiB peak RSS; trace8 2789.35 MiB / 13.207 s / 4220.9 MiB RSS. Repacked files: 12.02 MiB and 72.67 MiB; new inspect ~0.39 s / 652.7 MiB and ~2.317 s / 4118 MiB. Large binary load-only benchmark 2.696 s single-thread versus 0.879 s four-thread; binary save phase 1.284 s. Storage-only 10000-frame synthetic stress retained 2,622,409,586 logical bytes at 8 MiB peak RSS, with 18,209,077 bytes on disk; highly repetitive data, not a whole-app/GPU memory benchmark. Evidence under target/storage-bench/ (ignored build artifacts). Independent static review found two medium correctness issues; deterministic tests reproduced both red. Fixed first-N exhaustion by checking before retention/disk admission, publishing the accepted prefix with a resource warning, then stopping. Fixed live-reader root/length race by capturing roots before sampling append-only inode length. Both regressions now pass. Unit tests cover native/JSON parity, single/parallel ordering, torn roots/index fallback, corrupted payload rejection, hostile lengths, compaction/remapping/old-reader safety, active snapshots, late/pending readbacks, rolling/first retention, queue reservations and 10000-frame reference-only memory. Finished native GPU tests and README plus docs/trace-format.md. Current container v2 indexes contain only record references. Import-only incomplete-source state was removed with repacking; incomplete-tail status now comes from current-file root fallback. Only the current container and event versions are accepted. Final cached release inspect measurements: trace7 0.354 s / 649.8 MiB RSS and trace8 2.062 s / 4118.6 MiB; binary sizes 12,598,976 and 76,202,607 bytes. Final large load-only phase 2.442 s single-thread versus 0.855 s four-thread. Full save phase measured 1.3–1.6 s. Final explicit 10000-interval storage stress: 2,622,409,586 logical bytes, 18,209,118 physical bytes, 8 MiB RSS; ignored by default CI but explicitly run successfully. Baseline/new inspect metadata outputs match exactly on both real user captures. Final evidence: target/storage-bench/final-measurements.json and phase *.out files. Native AMD/Wayland/Vulkan validation with DISPLAY unset and synchronization validation enabled passed final binary headless large inventories (six executions, 6002-operation upload regions and ten repeated passes), rolling 300 intervals -> 180–299/120 GPU samples, first -> 0–119/121 samples including startup, --frames 7 -> 293–299/seven samples. Concurrent live CLI readers successfully observed sliding starts 23,167,311 while writing; timed SIGTERM kept a readable 120-interval checkpoint with expected no-End warning and no other trace warnings. Clean exits report no drops. No reported validation errors/synchronization hazards. Native large repacked-trace overview and binary draw-detail render/readback/shutdown passed; both PNGs visually inspected with real timings/counters, 576 draws and resolved viewport/state. Final GPU evidence: target/storage-bench/binary-final-gpu-gn26_c97/. No PRK game launch, authentication, installation edits, calibrated overhead/cold-disk benchmark, 8 GiB disk-fill or power-loss durability test. Loader remains eager, 8 GiB retained logical budget remains; these limitations are documented. No approval is claimed from the earlier static review; its two verified findings were corrected and tested. Final validation passed: 79 workspace tests, explicit ignored-by-default release storage stress, strict workspace/all-target Clippy without lint exceptions, formatting/check, debug and release builds, and git diff --check. Changes remain uncommitted; original user captures were not modified. Binary-only cleanup verified: removed CLI repack and full-trace conversion helper; removed JSON reader, Trace::read, whole-file stream buffering, earlier-version acceptance and obsolete compatibility tests/docs/copy. Renamed MAX_LINE_BYTES to MAX_RECORD_BYTES. Native tests replace JSON round-trip coverage. Final cargo fmt/check/build (debug+release), strict Clippy and whitespace checks pass; 78 workspace tests plus explicitly run release 10000-interval stress pass. Fresh v2 real AMD GPU capture retained six heavy workload executions with 6002-operation regions. Native Wayland rolling 150-frame run retained 30–149 with 120 GPU samples; first retained 0–119 with 121 including startup; timed SIGTERM retained a readable 378–497 window with 120 samples and expected no-End warning. Clean runs had no dropped events/metadata warnings; no reported validation errors or synchronization hazards. Old JSON and binary v1 files are explicitly rejected; repack is an unknown command. Load-only benchmark succeeds on fresh v2 input. Fresh native draw-detail rendering/readback/shutdown passed and screenshot was visually inspected. Evidence: target/storage-bench/binary-only-ozfo704f/. Earlier benchmark/native evidence predates this intentional format break and is historical, not current-format acceptance evidence.
Checklist
4 of 4 complete- Implement versioned binary chunks, safe incremental snapshot commits and bounded parallel current-format loading with corruption/boundary tests; no legacy compatibility.
- Integrate disk-backed rolling/first capture, byte-bounded async batching, incremental indexes and compaction without retaining payload windows in RAM; verify retention and interrupted saves.
- Benchmark load/save/size/memory against existing JSON captures, verify 10000-interval storage and real Vulkan rolling/first/termination behavior, run Rust validation and document results/limitations.
- Remove repacking and obsolete JSON/stream trace readers and compatibility-only state; verify binary-only rejection, current-format round trips, native capture/loading and docs.
- Area
- Personal tooling
- Project
- Wayrender
- Kind
- Feature
- Task ID
tsk_z46cyk49ja- Updated