Add capacity-aware compatible GPU worker profiles
Make Murmur provisioning resilient to L4 shortages without launching incompatible or insecure workers. Evidence from the first production smoke test: - L4-1-24G was out of stock in every listed zone. - RENDER-S was available but uses Tesla P100 (CUDA capability 6.0). - The locked worker image rejected CTranslate2 float16 on P100, and its PyTorch build supports only CUDA capability 7.0+, so Pyannote VAD/alignment failed. - A disposable runtime override succeeded by using CTranslate2 CUDA float32 with Silero VAD and CPU alignment. Acceptance criteria: - Configuration represents ordered per-zone worker profiles with server type, zone-specific security group, compute type, VAD backend and alignment device. - Provisioning checks capacity/profile compatibility and falls back only to tested profiles. - The normal L4 profile remains float16 GPU; any P100 profile is implemented in the worker image, benchmarked, and explicitly cost-bounded, or RENDER-S is rejected as unsupported. - Worker readiness reports safe compatibility diagnostics without leaking secrets. - The dedicated Scaleway IAM policy is reduced from temporary project-scoped full Instance/Block Storage access to the exact verified permissions. - Tests cover zone/security-group selection, stock failures, profile fallback and incompatible hardware. - A live bounded smoke test passes on every enabled profile and cleans up all resources. Implementation decision: - Profiles are an explicit ordered allowlist, not arbitrary server discovery. Each profile binds one zone, server type, security group, image, compute type, VAD backend, alignment device, batch size and cost intent. - Initial profiles are L4-1-24G (float16 with GPU VAD/alignment) and RENDER-S/P100 (float32 ASR with pinned Silero VAD and CPU alignment). Multi-GPU types remain excluded until one worker can use each GPU; Blackwell remains excluded until its OS/driver path is validated. - The Silero source that passed the disposable smoke test will be pinned by immutable revision rather than downloaded from a moving branch at worker startup. Verified implementation milestone: - Rust configuration/provisioning now accepts an ordered profile allowlist with per-profile zone, type, security group, image, compute/VAD/alignment settings and batch size. Only explicit `503 resource is out of stock` errors advance to the next profile; all other failures stop. - Cloud-init passes the selected runtime profile without exposing the worker token, readiness reports the active runtime, and the control plane rejects a worker whose runtime does not match its selected profile. - The worker supports float16/float32, Pyannote/Silero and CUDA/CPU alignment with fail-closed configuration validation. Silero revision `1e261b036686cd0017d500ee96acd1c4ba572a9d` is checksum-pinned and baked into the image; an offline no-network container load passes. - Full Rust checks/19 tests/strict Clippy and Python strict typing/lint/6 tests pass. The 21.6 GB local worker image builds successfully. - Test artifacts are published as control digest `sha256:5c22022c82e48e6d5adaa5fc1058e34da8993b2e52e51e06d7c4ba2f0bd2a031` and anonymously pullable worker digest `sha256:7f8610113cd35fc1626f8ef09bd4f705b372cfd77652c4ee123ead3fdb69a329`; these are test artifacts, not committed releases. - Ansible renders the ordered `l4-par2` then `p100-par2` profiles correctly and remains disabled pending the release/live matrix. - The dedicated IAM policy is reduced from `InstancesFullAccess` to 19 explicit Instance permissions plus project-scoped `BlockStorageFullAccess`. A live DEV1-S lifecycle verified create, cloud-init write, start, read and stop/delete with full IP/resource cleanup. - Live API verification found and fixed the worker teardown request: `stop-and-delete` accepts `delete_all_ips=true` but rejects the prior simultaneous `delete_all_volumes=true`; a regression test pins the working body. - Release source is committed and pushed as Murmur `d4ec040`; revision-labeled images are control digest `sha256:fa9ab0592ce4f83195fd872c14ebebeb877c79c0a78ef2dc94f1efcecaf6d2ab` and anonymously pullable worker digest `sha256:ef49cd5f9485a7bf11147448b628f28219896e9092a9aee850dc468796100b98`. Ansible `5bc9d5c` pins both, renders the ordered profiles, safely restarts bind-mounted runtime configuration, and is pushed to main with provisioning disabled by default. Verified live profile matrix: - Released P100 profile: RENDER-S provisioned as `p100-par2`, reported exact `float32/silero/cpu` readiness, started in 271.8 seconds, transcribed the 90-second fixture in 35.2 seconds with 7 segments/96 words, and automatically deleted provider resources plus its database row 64 seconds after completion. - Released L4 profile: L4-1-24G allocated despite the catalog `shortage` status, reported exact `float16/pyannote/cuda` readiness, started in 176.4 seconds, transcribed the same fixture in 3.8 seconds with 9 segments/101 words, and automatically deleted provider resources plus its database row 61 seconds after completion. - Both jobs succeeded in one attempt. Final production configuration has provisioning disabled, no GPU resources or worker rows remain, temporary media/token/result files were removed from both hosts, the public smoke route is removed, HTTPS health is 204, and a final Ansible deployment rerun made zero changes. - Cleanup validation exposed and corrected bind-mounted configuration restart handling; the role now uses flushed handlers so Caddy/control reload even when Compose changes another service.
Checklist
6 of 6 complete- Implement ordered per-zone worker profiles and capacity-only fallback in the control plane
- Implement configurable compute, VAD, alignment and batch settings in the worker
- Pin and bake Silero VAD assets for the P100 profile
- Update Ansible profile configuration and zone-specific security groups
- Verify unit/static/container tests and bounded live profile cleanup
- Reduce the dedicated Scaleway policy to verified permissions
- Area
- Kaukus
- Project
- Deploy Murmur to production
- Kind
- Feature
- Priority
- High
- Task ID
tsk_bhvnuxmn33- Updated