Productionize and rehearse the Rust voting cutover
Outcome Final release phase only: after cleanup, correctness, privacy, reporting and migration work is complete, make the Rust service deployable, reachable, recoverable and observable in every production process, then execute a measured hard-cutover rehearsal with an explicit rollback boundary. Sequencing decision Do not start deployment or cutover implementation early. Keep this task open until the preceding Rust integration cleanup and product/data correctness gates are complete; deployment-specific changes land last so they target the settled architecture. Confirmed blockers - config/deploy.yml declares KAUKUS_DATABASE_URL and KAUKUS_SIGNING_SECRET, but the checked-in Norway and Poland Kamal secret loaders neither fetch nor export them. No checked-in step provisions the dedicated Rust database. - Rust is supervised only in the web container and binds loopback. Sidekiq workers use the same loopback KAUKUS_API_URL, so BigscreenBroadcastJob—and future background reports—cannot reach Rust in production. - The configured backup accessory and restore tool cover only the Rails database. Rust voting state has no backup, restore or cross-database consistency procedure. - Rust exposes liveness only; the proxy checks Rails only. There is no database readiness endpoint, dependency health integration, metrics, alerts or stuck-run operational view. - Gitea CI runs Rails/JS/CSS but no Cargo fmt/check/Clippy/tests, real Rust PostgreSQL round trip, Docker build or real-service browser workflow. Real-Rust system examples skip unless invoked manually. - The fixed 10-connection Rust pool, 3-second Rails timeout, 500 ms Sidekiq poll loop and 1,000-voter limit have no representative concurrency/capacity certification. Acceptance criteria - Norway and Poland provision dedicated Rust databases and shared signing secrets through verified secret loading; every web/worker process that queries Rust uses a documented reachable topology with least exposure. - Readiness verifies database/migrations and is part of container deployment health. Graceful shutdown, pool/timeouts and migration startup are tested under failure and rolling/restart conditions. - Rust database backup and restore are automated and rehearsed together with Rails/Redis dependencies at a defined RPO/RTO; restore validation proves current and historical voting state and source mappings. - Metrics/alerts cover availability, request latency/status, pool saturation/timeouts, migration failure, service restarts, reconciliation failure, polling failures and unexpectedly old open runs, with bounded labels and no voter/choice data. - Required CI gates run Rust formatting, strict Clippy/check/docs, unit and real-PostgreSQL tests, Rails-Rust round trips, relevant authenticated browser flows, assets, translations, RuboCop and production-image build. - Capacity tests cover 1,000 voters, burst vote/change traffic, concurrent meetings, history/report reads, worker polling and restart/recovery; thresholds and tuned limits are recorded. - A maintenance-window runbook freezes old writers, verifies no active meeting/run, backs up, imports and reconciles history, deploys one authority, smoke-tests all audiences and defines go/no-go. Rollback is explicitly limited before new Rust-only votes unless a verified reverse/recovery procedure exists; after that boundary the plan is fix-forward or a documented incident procedure.
Checklist
0 of 4 complete- Fix production topology, Rust database provisioning and secret delivery so web and worker processes can reach one authoritative service in both deployments.
- Add dependency readiness, automated Rust database backup/restore, metrics, alerts and operational stuck-run/reconciliation visibility.
- Enforce Rust, real-PostgreSQL, Rails-Rust, browser and production-image CI gates; run representative capacity/failure tests and tune limits.
- Rehearse maintenance-window import/deploy/smoke/go-no-go and the pre/post-first-Rust-vote rollback boundary on production-shaped infrastructure.
- Area
- Kaukus
- Kind
- Operations
- Priority
- High
- Task ID
tsk_usau5c4fhe- Updated