Make Murmur worker teardown reliable
Fix and verify worker cleanup after idle completion, provisioning failure, control-plane restart, and provider deletion errors. Evidence: the first production smoke job succeeded, but its ready RENDER-S worker remained running for several minutes despite `idle_seconds = 60` and had to be deleted manually. Earlier failed provisioning attempts also left transient database provisioning rows until backoff cleanup. Verified root cause for the idle deletion failure: the v2alpha1 `stop-and-delete` endpoint rejects the combination of `delete_all_ips=true` and `delete_all_volumes=true` with HTTP 400. A live disposable lifecycle check confirms `delete_all_ips=true` alone successfully stops the server, deletes its root volume/IP and removes the resource. The fix is released in Murmur `d4ec040`. Live release validation now confirms normal idle deletion: the P100 provider/database records disappeared 64 seconds after completion and the L4 records disappeared after 61 seconds with `idle_seconds=60`; no attached GPU resources remain. Startup/orphan reconciliation, interrupted provisioning cleanup, independent provider/database retries and observability remain outstanding. Current restart evidence: deployment interrupted provisioning for Scaleway server `b44a7f8b-3f7c-4477-b53d-9277e9af8e3e`, leaving it stopped and untracked by the database. Its cloud-init still references the superseded worker image. This confirms startup reconciliation must discover and remove tagged provider orphans whose provider ID was never persisted. Block-storage incident: production teardown deleted servers and IPs but retained every SBS root volume because the v2alpha1 `stop-and-delete` request only set `delete_all_ips`. Inventory found 77 detached `murmur-worker-root` volumes plus one disposable IAM-check root; all 78 detached Murmur volumes were deleted through the Block API and verification left only the control server's in-use root. The fix now explicitly selects IP, volume and private-NIC behavior, captures attached SBS IDs, and retries Block API deletion for any volume not removed with the server. Live disposable checks pass both paths: a running server's full `stop-and-delete` removed server/IP/SBS, while a stopped server used selector-bearing DELETE plus explicit Block deletion; both left no resources. Murmur `1954ccf` and Ansible `6f88d9b` are pushed and deployed; health is 204 and provisioning remains demand-driven with a one-worker cap. IPAM/billing incident: billing showed €38.83 for 7,766 Zonal Flexible IPv4 minutes and €24.75 for 190,323 SBS GB-hours. Inventory found 65 detached zonal IPv4s (54 created during October 20 retry activity, seven during earlier October tests and four from August) plus the 78 detached SBS roots already identified. All 65 detached IPs and all 78 detached volumes were deleted. Current inventory contains only the control server, its attached IPv4 and its in-use root volume; there are no LBs, gateways, private networks, snapshots or managed databases. A post-cleanup out-of-stock retry produced no new detached IP. Defense-in-depth implementation is ready: worker servers, inline zonal IPv4s and SBS roots receive `murmur-worker`/`managed-by-murmur`/profile tags. The scheduler reconciles every 15 minutes and deletes only tagged IPv4s with no attached resource and tagged available volumes with no references after a 15-minute grace period; API failures are logged and retried without blocking job dispatch. The dedicated project-scoped IAM rule now includes `IPAMFullAccess`, and its own credential successfully lists tagged detached IPs. A disposable DEV1-S check verified the provider persists all tags on both inline resources and that full cleanup leaves no server, IP or volume. Rust formatting, 48 tests and strict Clippy pass. Murmur `0c6eacb` and Ansible `3cdedd7` are pushed and deployed with control digest `sha256:525c1123e8bfafe00c371923a617db7017da17cbbf0212773b068f1ba3363795`; startup reconciliation completed successfully under the dedicated identity, the first post-deploy out-of-stock retry left no detached IP/volume, health is 204, and the final Ansible rerun had zero changes. Acceptance criteria: - A completed worker is deleted within the configured idle window plus one scheduler interval. - Provider deletion and database-row removal are retryable and reconciled independently; one failure cannot leave an untracked billed server or a stale ready worker indefinitely. - Tagged provider resources are reconciled at startup, including interrupted provisioning. - Metrics/logging expose worker age, provider ID, deletion attempts/failures and orphan reconciliation. - Reconciliation also inventories detached IPAM IPv4s and SBS volumes for the Murmur project so partial provider allocations cannot accrue unnoticed. - Tests cover successful idle deletion, provider failure, database failure, process restart and idempotent recovery. - A bounded live test confirms both provider resource and database row disappear.
Checklist
0 of 0 completeNo checklist items.
- Area
- Kaukus
- Project
- Deploy Murmur to production
- Kind
- Bug
- Priority
- High
- Task ID
tsk_z53s7ap4k3- Updated