Add remotely configurable OS update fetching pause and URL

Investigate dashboard-configurable OS update base URL (for example https://os.kaukus.no), applied by kaukus-provisioner. Existing Fedora CoreOS boxes must migrate remotely without reflashing or user installation work. Municipality networks whitelist outbound HTTPS as well as inbound traffic: OS update metadata, payloads and layered RPM dependencies must not require Fedora or other unapproved destinations. Immediate user priority: no OS updates for now; stop deployed boxes attempting to contact Fedora. User asks whether os.kaukus.no can initially be a dummy endpoint. Recommend explicitly disabling Zincati OS updates on existing units through an upgraded agent, optionally retaining os.kaukus.no as the future source. A dummy graph alone cannot stop layered RPM repository access or unrelated upstream jobs. Inventory pending deployments and other Fedora-facing jobs/repo configuration; do not silently reboot, clear staged deployments, or disable Kaukus application/agent updates. Full upstream mirror/proxy is deferred, not required for the initial pause. Confirm whether the initial acceptance boundary is update-related Fedora attempts or all Fedora-facing host traffic before implementation. Implementation authorized. User accepts restarting streamer and panel when saving configuration and will save only while boxes are inactive. Reuse the complete retained configuration save/acknowledgement/recovery workflow rather than add parallel OS commands. Add an OS update URL (default https://os.kaukus.no) and fetching toggle. Upgraded agents default to paused on first run without reflashing; preserve explicit subsequent operator settings. Capability-gate settings so older agents cannot silently ignore OS policy. Preserve application/agent updates, host data and maintenance-window strategy. Implement locally on feature/configurable-os-update-fetching; no deployment or reboot performed. Full mirror/proxy and public dummy endpoint hosting remain deferred until hosting is available. Runtime-starvation review follow-up committed as 46eea27 (fix: Keep OS policy work off async workers) and pushed to the existing PR #146 branch after user authorization. Full assignment host work (including policy application, host file writes and streamer/panel restarts), full heartbeat collection and OS startup initialization now run via spawn_blocking. OS inspection uses try_lock and returns unknown/in-progress while a policy operation holds the mutex, so heartbeats do not queue behind long transitions. Async configuration/state coordination and status publication remain on Tokio tasks. The red regression reproduced a one-second async-worker stall before offloading; after the fix, three tests run the real policy application/operation lock with a real one-second sleep subprocess and confirm busy inspection, loopback TCP ping/pong and Tokio timers complete before it finishes on current-thread and one-/two-worker runtimes. All 32 provisioner tests, cargo check --workspace, formatting and diff checks pass. Affected strict Clippy passes with only existing collapsible_if/items_after_test_module lints allowed. Patch-scoped independent static review completed with no actionable findings; it did not execute tests. This follow-up does not validate real FCOS/systemd or NATS protocol integration. Follow-up commit is published; no deployment performed. Focused review follow-up committed locally as 09d2c23 (fix: Preserve healthy OS policies and reconcile mismatches); push to PR #146 failed because the configured SSH agent could not sign (communication with agent failed). Branch is ahead by one commit with a clean working tree. Awaiting user restoration of agent access; no authentication fallback or automatic retry attempted. Implementation: fresh assignment IDs carrying an identical successfully persisted healthy policy skip the OS transition, including with staged deployments or normal active package work. Health reuses config/guard/service inspection without checking rpm deployment state; failed/incomplete commands, transition latches, remembered errors or unhealthy service state still require a real retry. Healthy no-op command IDs are journalled as completed so later safety pauses cannot be cleared by replay. Agent regression reproduced the staged-deployment failure before the fix and passes after it, with additional active-package/error-state/replay coverage. Dashboard now detects reported-versus-retained policy mismatch (including actual fetching drift), warns clearly, enables an unchanged explicit retry, forces the reviewed OS policy into the patch, and names reconciliation values in confirmation. Synthetic browser flow verifies enabled OS application followed by full-assignment failure retains disabled router policy, then restores it via no-edit retry using the original applied revision and no unrelated credential fields; warning clears and state remains settled after reload. Bosch-only patches still omit OS settings and dirty drafts survive polling. No host-application ordering change or new transaction machinery. Validation for this follow-up: 35 provisioner tests and 28 dashboard tests pass; cargo check --workspace, dashboard production build, formatting/diff checks and affected strict Clippy with only known pre-existing collapsible_if/items_after_test_module allowed pass. Chrome desktop/mobile synthetic partial-failure reconciliation passes; real systemd/FCOS and NATS application integration remain untested for this follow-up. Static patch review found the need to journal healthy no-op command IDs; this was corrected with a regression. Follow-up commit exists locally; push is blocked by SSH agent access. No deployment performed. Implementation acceptance: dashboard can retain/change OS URL and disable/enable fetching through existing configuration workflow; new agent defaults safely to disabled; legacy agents are gated; pause stops Zincati and identified automatic Fedora-facing RPM jobs without affecting Kaukus updates, rebooting, clearing staged deployments or replacing unrelated settings. Surface observed policy, application errors and pending deployment warnings honestly. Focused protocol/router/agent/UI tests and relevant Rust/dashboard checks pass; real-browser acceptance verifies controls and save payloads. Live appliance network-level verification remains an explicit rollout gate. Initial verified findings - Both repositories had clean working trees. kaukus-streamer config.bu sets Zincati periodic maintenance but no custom Cincinnati endpoint. Current files and locally available all-ref history searches found no earlier os.kaukus.no/Cincinnati/mirror implementation; this does not exclude deleted or unfetched remote history. - config.bu layers avahi, btop, pipewire, wireplumber and pipewire-alsa using rpm-ostree. Layer requests persist across updates/rebases. OS-only mirroring is insufficient: RPM repositories and dependencies must also resolve exclusively through approved Kaukus destinations. - Existing kaukus-provisioner-update.service pulls the static agent binary from cr.kaukus.no and replaces/restarts it on change; dashboard and agent already support triggering this. This is the intended no-reflash bootstrap, subject to verifying actual deployed units have the unit, working registry credentials and control connectivity. - Reuse cloud-dashboard BoxProvisionView System/settings patterns, cloud-router box_configuration revision/pending/applied persistence, existing NATS transport and host agent. Do not put an OS URL into extra_env: that config only writes container environment. Host OS settings need separately applied/recovered state and capability gating; legacy agents must never be reported as successfully migrated merely because unknown fields deserialize. - Zincati supports an administrator drop-in [cincinnati] base_url in /etc/zincati/config.d; graph is GET /v1/graph with stream/basearch parameters. Metadata discovery is separate from payload download. Official Fedora major-changes docs describe F42 OCI transition and later existing-node migration; do not assume every deployed box is OCI or every current main-branch feature exists on installed versions. - Current upstream Zincati OCI source matches the current graph node using the full payload reference; blindly rewriting all Quay references to a mirror can make the booted node disappear from the graph. Graph generation/mirror selection must retain a valid source-node match and valid upstream transition edges. Current main also uses OCI-only release parsing, so do not choose an old OSTree design without inventory. Recommended direction and validation gates - One operator-facing HTTPS OS update base URL, with server-owned graph, OS payload storage/registry, and versioned RPM repositories. Kaukus infrastructure fetches upstream content; boxes must not follow redirects, mirrorlists, registry authentication realms, blob redirects or fallback paths to unapproved hosts. Retain upstream signature verification and digest integrity; trust is not disabled for migration. - User confirms using official upstream content via mirroring or a proxy with URL rewriting; do not introduce a custom OS distribution/bootc. Evaluate a restricted caching reverse proxy first, with persistent caching/prefetch where needed, against explicit mirroring. It must cover graph, version-compatible OS payload transports and RPM repos, handle upstream authentication/redirects server-side, and never expose unapproved destinations to boxes. Rewrite only routing references, not signed payload bytes; preserve signature verification, valid transition edges and exact current-node matching. Do not build an unrestricted forward proxy. Consider baked-in packages separately later only if requested. - Stage rollout: read-only inventory of representative deployed units (OS/tool versions, deployment origin/transport/ref/digest, requested RPMs/repos, Zincati settings/pending deployment, update service availability); build a compatible origin/graph/payload/RPM profile; upgrade agent through existing registry; capability-gate and persist host policy; preflight all required destinations/artifacts; serialize migration against Zincati/rpm-ostree; apply atomically with recoverable prior configuration; stage OS update and reboot only in an approved window. Preserve /var data, credentials, network and application configuration. - UI must distinguish desired URL, actual applied source, current/staged OS versions, pending reboot, unsupported agent and errors. Saving a URL must not imply OS migration or trigger immediate reboot. Enforce operator authorization, HTTPS URL validation and trusted control messages; no silent Fedora fallback. - Preserve a known-good local boot deployment. OS rollback is distinct from reverting persistent host configuration; /var is shared and config changes need explicit recovery. Do not promise unattended rollback for a nonbooting/disconnected unit without testing that mechanism. - Pilot acceptance: on a representative old installation with Fedora/Quay/mirror egress denied, update agent, change source, download/stage/apply OS and RPM updates, reboot, reconnect to NATS and verify streaming/audio; repeat a subsequent update. Capture DNS/network evidence to prove no unapproved requests, and test unavailable server, interrupted staging, bad signatures and recovery. No box access, mutation, build, tests, deployment or reboot has been performed in this investigation. Sources: https://coreos.github.io/zincati/usage/configuration/ ; https://coreos.github.io/zincati/development/cincinnati/protocol/ ; https://coreos.github.io/zincati/usage/auto-updates/ ; https://docs.fedoraproject.org/en-US/fedora-coreos/major-changes/ ; https://docs.fedoraproject.org/en-US/fedora-coreos/os-extensions/ ; https://coreos.github.io/rpm-ostree/administrator-handbook/ ; current upstream src/cincinnati/mod.rs and src/rpm_ostree/{mod.rs,cli_deploy.rs}. Local implementation outcome - Branch feature/configurable-os-update-fetching, committed as 483bfaa (feature: Add configurable OS update fetching) and pushed. Added optional retained BoxConfig OS policy, HTTPS-origin validation, redacted API view, configuration recovery and per-box observed policy/error/staged-deployment inventory (migration 28). Dashboard Configuration has URL/fetching controls, legacy-agent gating, change confirmation and explicit unchanged-policy retry after a safety error. User accepts streamer/panel restarts on save. - New agent pauses by default before NATS connection on first upgraded start. Manages dedicated Zincati/drop-in state without replacing maintenance windows. Blocks appliance RPM layering, rpm-ostree automatic/Count Me and DNF/DNF5 cache/automatic services/timers, leaving Kaukus application/agent updates untouched. Running package transactions and staged deployments are not cancelled/cleared; enabling refuses pending deployments. - Failure handling attempts emergency stops independently of persistence. Durable pause and whole-transition latches prevent resuming after failed/interrupted changes. Per-command journal rejects automatic replay of failed/interrupted OS commands and makes successful OS application idempotent. Startup never discards a valid saved origin/policy in favor of defaults. Legacy capability validation covers normal and pending-key-approval resend paths; OS-bearing envelope v2 is also rejected by existing v1 agents if downgrade races delivery. - cloud-dashboard/Caddyfile now includes os.kaukus.no empty-graph placeholder, paused health, no upstream proxy/redirect and payload 404. No public DNS/service deployment performed. Complete payload/RPM mirror remains deferred; enabling is only safe with trusted no-update placeholder until proper mirroring is implemented. Documentation explicitly explains that changing the graph URL alone does not reroute payloads/repositories. Verified local validation - cargo fmt --all -- --check and cargo check --workspace pass. Targeted Rust suites: 126 router + 29 agent + 51 protocol tests pass, plus both isolated real-NATS integration scenarios (including off/on/off retention, auth, legacy rejection and replacement-key pending-policy rejection). - 26 dashboard tests and production build pass. Chrome synthetic-fixture acceptance verifies default disabled controls, URL/toggle change confirmation, enable/disable payloads, reload retention, Bosch-only omission, draft polling, legacy/staged/unknown/offline/pending/failure states, unchanged-policy retry and desktop/mobile usability. Browser tool could not independently compare revision timing/authenticated responses; a separate authenticated fixture GET confirmed full public-key equality for all seven captured saves. Real-NATS integration verifies revision handling. No real hardware exercised. - Strict affected-package Clippy fails only on verified pre-existing collapsible_if, items_after_test_module and result_large_err lints; passes with precisely those lints allowed. git diff --check passes. - Caddy config adapts/validates. Exact adapted placeholder handler passes an isolated network-disabled Caddy runtime test for empty graph with stream/basearch parameters, JSON/no-store headers, paused health and payload 404. Runtime harness used plaintext with automatic HTTPS disabled only in temporary test config; public DNS/TLS issuance remains unverified. - Static reviews found failure/interruption/resend edge cases; all reported candidates were addressed with regression tests. Follow-up static review was partial (some reviewers did not open the untracked host module); do not describe it as whole-feature approval. Last two findings (enable interruption window and automatic replay clearing pause) were addressed after that review with whole-transition latch/command journal and passing tests, not a further independent review. Outstanding rollout No releases, production changes, box access or reboot performed. Deploy router/dashboard first (migration 28), arrange os.kaukus.no DNS/Caddy/TLS, then release/update the provisioner through the existing registry path, starting with a pilot. Verify real installed unit/tool versions, pause across reboot, Count Me/RPM guard behavior, staged deployment handling, control reconnection and audio/streaming. Capture outbound traffic with Fedora/Quay destinations denied before claiming deployed appliances are quiet. Fresh-install activity before the upgraded agent starts is explicitly not covered. Full mirror/proxy source/signature/registry behavior remains separate deferred work. Pull request - #146: https://git.kaukus.no/Kaukus/streaming-engines/pulls/146 - Title: feature: Add configurable OS update fetching - Head feature/configurable-os-update-fetching; base main. Open, not draft. Local validation is recorded above; public endpoint and real-appliance rollout remain outstanding.