Retain encrypted box configuration and support recoverable re-provisioning
Outcome: cloud-router retains box provisioning configuration for inspection and re-provisioning without requiring operators to re-enter all settings. Recover existing deployed-box settings and credentials over an encrypted channel. A leaked database or database backup must not expose retained secrets. Verified repository findings: - cloud-router/src/provisioning.rs persists a plaintext serialized BoxConfig in boxes.pending_config before dispatch. cloud-router/src/db/boxes.rs clears it on Applied. - cloud-dashboard/src/views/BoxProvisionView.vue only restores the room ID, not Bosch/upload/extra environment settings. - streaming-protocol already provides HPKE using X25519/HKDF-SHA256/ChaCha20-Poly1305 for sealed assignments. Reuse existing crypto rather than add dependencies. - Deployed agents write streamer environment and NATS credentials to configured local files; panel tokens are generated locally and preserved during re-provisioning. - Setup credentials are shared with broad provisioning subject access. Recovery must not export secrets to an arbitrary caller-provided key. Confirmed deployment constraints: boxes cannot be accessed over SSH or receive manually installed files; customers block outbound traffic except allowlisted hosts. cr.kaukus.no is allowed, router HTTP API is not. User approved embedding the router recipient public key in the provisioner binary through trusted image builds. Recovery must use only the existing NATS connection, with no HTTP bootstrap, runtime key override or manually installed box files. Only the public key may enter image builds; the private key stays on the router and in its separate recovery copy. Confirmed scope: - Permanently retain only the last successfully applied configuration per box; no version history or historical restore UI. Keep this independently of in-flight desired configuration, distinguish desired/applied/failed assignments and correlate acknowledgements to command IDs. A failed or pending change must not replace the last applied configuration. - Make retained configuration available through the dashboard and API using the existing boxes.manage ability, including API-key authentication. - Support partial configuration edits and re-provisioning: an operator can change only one entry, for example the Bosch IP/URL, while retaining all omitted settings and credentials. Distinguish omission (preserve) from explicit clearing/removal; never require re-entry of unchanged passwords or upload keys. No upstream-service independence requirement. - Keep a dedicated decryption key in a read-only router secret file outside SQLite, data volumes and database backups; retain an independently secured recovery copy. Use versioned ciphertext/key identifiers and bind encrypted payloads to purpose and box/snapshot identity. - Recover deployed configuration without restarting services or changing passwords, preserving Bosch credentials, upload keys, panel token and other secret environment values. Mint fresh room NATS credentials where appropriate. - Redact secrets in normal API/UI responses, support retaining existing secrets on edit, and re-provision using stored configuration. - Migrate any pending plaintext safely. Existing backups/WAL/free pages may retain pre-migration secrets; encryption is not retroactive. - Required operational follow-through: after credential recovery/encryption is verified, ensure the new backup source no longer contains recoverable legacy plaintext in SQLite free pages/WAL/sidecars, produce a fresh independent backup, restore it in isolation and verify retained configuration can be decrypted with the separately held key. Only after that verification, delete all old pre-encryption backups and their dependent snapshot/LTX history, including retained object versions where applicable. Verify the new backup remains independently restorable after cleanup and record any retention/versioning constraints that prevent complete removal. Do not delete old backups before a verified replacement exists. - Roll out router key/storage support before agent recovery; unavailable/older agents remain explicitly unrecovered. No live access/deployment authorized or performed. Acceptance criteria: - The last successfully applied configuration is retained permanently and available through the dashboard/API with boxes.manage API-key access; no history feature is required. - A single-entry edit such as Bosch IP/URL can be re-provisioned while preserving every omitted setting and secret. Explicit removal is supported and failed/stale acknowledgements cannot overwrite the last applied configuration. - Existing boxes can securely backfill their configuration; older/offline boxes have clear recovery status. - Mixed-version rollout must preserve the actual panel/kiosk token: initial assignments omit panel_api_token so old agents cannot ignore a router-invented value that is later retained as applied. Applied records without a recovered panel token remain explicitly incomplete/recovery-eligible; backup verification must reject incomplete records. Cover legacy initial provisioning, agent upgrade, actual token recovery and subsequent re-provisioning. - No retained or pending provisioning secret is written plaintext to SQLite or leaked in normal API responses/logging. - Database-only restore/leak does not reveal configuration secrets; recovery with the separately held key is verified. - A fresh post-encryption backup is created and restore/decryption verified before old plaintext-bearing backups/history are deleted. Post-cleanup restore is verified; any storage retention constraints or remaining historical plaintext copies are reported explicitly. - Relevant crypto, migration, acknowledgement ordering, recovery, secret-preserving edit and re-provisioning regressions pass. Implementation and verified local state: - Implemented on feature/retain-encrypted-box-configuration; committed as 4a9bd62 (feature: Retain encrypted box configuration) and pushed with upstream tracking. No new dependencies, production access, key installation by the assistant, deployment or backup deletion. - Dedicated external HPKE key file with generation and offline backup-decryption verification commands; encrypted pending/last-applied storage, legacy pending migration, command-correlated promotion and optimistic edit checks. Failed first attempts remain separately editable without being labelled applied. - Updated agent exports managed env/NATS credentials only to the router public key embedded in its binary through trusted image builds, serialized with assignment application. Recovery uses only the existing NATS connection; no HTTP bootstrap, runtime key override or box file installation exists. Router backfills on announce. Persistent reported deployment state prevents boxes missing from restored inventory from being provisioned from defaults before recovery. - Redacted boxes.manage configuration endpoint and dashboard partial edits preserve omitted Bosch/upload/custom credentials and panel tokens. Failed accepted edits retain the applied patch baseline so unchanged-form retries do not drop requested changes. - Latest local validation: cargo fmt --all and cargo check --workspace pass; 110 cloud-router, 10 provisioner and 47 protocol tests pass. Both separately invoked opt-in NATS integration scenarios pass serially against a disposable JetStream broker, covering actual boxes.manage-only API-key auth, recovery, encrypted single-field assignment, refusal before recovery, failed-initial correction, stale revisions and a simulated legacy-agent provisioning/upgrade/token-recovery/re-provisioning sequence. Five dashboard tests/production build and 28 release/deployment-tooling tests pass. Clippy for affected crates completes with seven existing warnings (large Axum error variants and old nested conditionals); strict -D warnings is not clean. - Embedded-key validation: compiled local binaries with and without a temporary fixture public key; --check-embedded-key rejects missing compiled keys even if runtime env supplies one, and a compiled key cannot be changed by a runtime override. Router --print-box-configuration-public-key prints only the public half and leaves the existing key file unchanged. The unfinished HTTP endpoint/client/TLS fixtures were removed and Cargo dependencies/lock restored to their previous state. - Production Dockerfile requires KAUKUS_ROUTER_CONFIG_PUBLIC_KEY and runs the compiled binary's --check-embedded-key. Local build.sh and tagged release tooling forward the public build argument; Gitea reads its same-named repository Actions variable. Public-key image-label checks reject mismatched immutable image reuse/promotion. No actual Docker image build/publish or Gitea workflow execution was performed. - Follow-up static review found backup verification could create an empty DB on a mistyped path. Fixed it to open existing DBs read-only without migrations; both a missing-path CLI regression and restored-database read-only/decryption regressions pass. - Chrome acceptance against a stateful mocked API confirms prefill, saved/blank secret fields, boxes.manage-only access, URL-only payloads, failed-application retry and persistence after refresh. Browser evidence is frontend-only, not live device/crypto validation. Fixture server logs additionally contain no municipality requests. Temporary browser fixture and NATS containers were stopped. - Patch-scoped feature review identified failed-retry baseline and pre-recovery overwrite races; both were corrected with regressions. Review was static and does not replace the executed checks above. - Compatibility follow-up committed as 05e0e34 (fix: Preserve applied panel tokens across agent upgrades) and pushed to the existing PR #130 branch: initial assignments omit panel_api_token for every agent version, so only the agent's actually applied token can later be recovered and retained. A shared completeness predicate keeps missing/empty tokens recovery-eligible, exposes configuration.complete=false in the API/dashboard and rejects incomplete records during backup verification. The NATS mixed-version regression failed on the original invented-token assignment, then passed after the fix through legacy initial apply, upgrade, actual token recovery and a Bosch-only re-provision preserving that token. Additional default tests cover redacted completeness and backup rejection of unknown/empty tokens. This exercises simulated agent protocol behavior, not deployed legacy binaries. Follow-up tests/check/build pass. The commit is pushed and the working tree is clean; no live deployment or backup change has been performed. Remaining operational acceptance: - User reports the real private key is installed on the router and the Gitea repository Actions variable KAUKUS_ROUTER_CONFIG_PUBLIC_KEY is now set to its public key. Neither installation nor the configured value/matching image identity has been independently verified; the separate private-key recovery copy remains unverified. Do not put private JSON/secret_b64 in CI or image builds. Confirm the embedded image public-key label matches the installed router key during release/pilot validation. - Publish a new immutable provisioner version with the matching embedded key, deploy compatible router/agent/dashboard images through the existing release/fetch mechanisms, and verify NATS-only recovery/re-provisioning on a pilot and then deployed boxes. No new outbound destination or box SSH/manual files are required. Rotation requires another provisioner image; automatic key rotation remains out of scope. Code/build plumbing is locally validated, but real key selection, image publication and fleet activation are still outstanding. - After verified encryption and legacy source/WAL cleanup, create a new independent backup generation, verify isolated integrity and credential decryption, then delete old plaintext-bearing backup/history/object versions and verify restore again. These production steps are still unchecked; do not mark the task done before they pass. Repository rollout instructions are in docs/box-configuration.md and linked from the backup runbook. Pull request: - #130: https://git.kaukus.no/Kaukus/streaming-engines/pulls/130 - Head: feature/retain-encrypted-box-configuration; base: main. Working tree was clean after commit/push. PR opening does not complete the outstanding production rollout and backup-replacement acceptance.
Checklist
6 of 9 complete- Confirm last-applied-only retention, boxes.manage dashboard/API-key access and single-entry re-provisioning preserving omitted credentials.
- Implement and verify encrypted retained configuration, pending-data migration and command-correlated applied state.
- Implement and verify NATS-only recovery using a build-time embedded public key, image-build validation and immutable key identity checks, without box SSH/files or runtime key overrides.
- Implement and verify permanent last-applied configuration inspection through boxes.manage dashboard/API-key access and partial edits preserving omitted settings/secrets, including a Bosch-IP-only re-provision.
- Pass relevant Rust/frontend validation and encrypted backup/restore regressions; document key custody, rollout and legacy plaintext limitations.
- Configure the public build variable from the installed router key, publish/deploy matching images, and verify NATS-only recovery and a single-field re-provision on a pilot and expected deployed boxes; verify separate private-key recovery copy.
- After verifying credential encryption/recovery and removing legacy plaintext from the backup source, create a fresh independent backup and verify isolated restore plus decryption using the separately held key.
- Only after replacement-backup verification, delete old pre-encryption backups and snapshot/LTX/object-version history; verify post-cleanup restore and report any retention constraints or remaining plaintext copies.
- Fix and verify legacy-agent initial provisioning โ upgrade โ actual panel-token backfill โ re-provisioning, with no invented token retained as applied and incomplete records rejected by backup verification.
- Area
- Streaming
- Project
- streaming-engines
- Kind
- Feature
- Task ID
tsk_defq8cjhc4- Updated