Prioritize Teams and production video over opportunistic CI CPU use
On 5.9.138.8, replace streaming build's fixed CPU quota with persistent low CPU weight so CI can use idle CPU while yielding to production. Teams is highest priority, video processing next, CI last. Inspect and preserve container/deployment persistence, apply live without restarting production, retain CI memory protection, and verify effective cgroup settings and production health. CPU weights provide proportional scheduling, not hard real-time guarantees; do not promise protection from every shared-resource failure. Current findings: all relevant workloads are sibling cgroups in system.slice; streaming-build.service has CPUQuota=800% temporarily overridden to unlimited; Teams compose /opt/kaukus-teams/compose.yml currently caps CPU at 14 and memory at 12 GiB; video container has no CPU quota. CI has MemoryMax=24G, no swap; runner has MemoryMax=2G and CPUQuota=200%. Acceptance: CI unlimited when idle and lower weighted than production; Teams highest weighted and not CPU quota-limited; configuration survives reboot and Teams recreation; no service interruption; validate actual cgroups, configuration, and health. Avoid artificial production saturation without approval. Applied and verified: persistent systemd set-property overrides set streaming-build CPUQuota= (unlimited), CPUWeight=1; runner CPUWeight=1 (existing 2-CPU orchestration quota retained). Video remains unlimited, effective cpu.weight=100. Teams /opt/kaukus-teams/compose.yml removes cpus:14 and sets cpu_shares:26214, effective cpu.weight=1394 on this host. Existing Teams container updated live with shares and --cpu-quota -1; verified cpu.max=max 100000, no restarts, original start time retained. Docker --cpus 0 was a no-op; NanoCpus remains 14e9 in existing HostConfig but persisted CpuQuota=-1 overrides the effective quota; future Compose recreation has no cpus setting. Compose validation passes, backup /opt/kaukus-teams/compose.yml.before-cpu-priority-20261022T121724Z. Both production containers running; no container healthcheck configured for Teams. CI memory 24 GiB/no swap and runner 2 GiB unchanged. Three-second build sample had no new quota throttling, but it was not fully parallel at that point. No production saturation or end-to-end streaming test performed; proportional CPU priority verified through live cgroup settings, not a latency guarantee. GPU, network, disk and application behavior are not protected by CPU weights. Follow-up requested: apply the same idle-capacity/low-priority CPU policy to Docker Compose CI in /opt/kaukus-ci. Both gitea-runner and pi-review-runner launch sibling job containers through the host Docker socket, so daemon-only CPU shares are insufficient. Preserve existing auth mounts and runner settings, persist job container resource options, and validate actual job-container cgroups without triggering real CI. Avoid interrupting active jobs; inspect runner activity before restarting/recreating runner daemons. Follow-up findings/blocker: both Compose runners use rootful host Docker socket; job containers are sibling cgroups. gitea-runner currently has CI task 4688 active (started 2026-10-22T12:25:04Z); pi-review-runner appears idle. gitea-runner currently mounts no config file despite /opt/kaukus-ci/config.yaml existing, so persistent job options require adding CONFIG_FILE and config bind mount and recreating that runner. pi-review config already mounted. User explicitly authorized interrupting the current CI job and recreating the runners. Apply and validate without restarting production. Follow-up implemented: /etc/systemd/system/system-ci.slice has CPUWeight=1 with unlimited CPU. /opt/kaukus-ci/docker-compose.yml sets both runners cgroup_parent=system-ci.slice and cpu_shares=2; gitea-runner now mounts config.yaml and sets CONFIG_FILE=/config.yaml. Both runner YAML container.options add --cgroup-parent=system-ci.slice --cpu-shares=2; existing pi auth volume option preserved. Files backed up with .before-cpu-priority-20261022T122941Z. Compose validation passes. Both runners recreated with user permission and confirmed registered. Live runner cgroups have weight=1/unlimited. Local act_runner exec smoke workflow using the same saved job resource options completed successfully, and the actual spawned container was observed in system-ci.slice with weight=1/unlimited. Smoke container cleaned up. This tests container option behavior, not a server-dispatched CI job; no real CI was triggered. Teams and video retained original start times, weights 1394/100 and unlimited CPU. Remaining boundary explicitly communicated: this is a default priority policy, not hard confinement. Workflow-specific container.options can replace defaults; service containers or extra containers launched through the rootful Docker socket need the same cgroup-parent option and may otherwise bypass priority. Full enforcement for arbitrary Docker children would require a dedicated CI daemon/resource domain or changes to workflow/container creation. No production saturation or end-to-end streaming test run.
Checklist
0 of 0 completeNo checklist items.
- Area
- Streaming
- Project
- streaming-engines
- Kind
- Operations
- Priority
- High
- Task ID
tsk_nm9v8j52je- Updated