Investigate Flå repeatedly entering a meeting
User reports Flå appears stuck on a meeting, constantly looping, following clock correction for telemetry. User authorized the proposed targeted recovery: clear only Flå's stale command backlog and reboot Flå to discard buffered deliveries. Preserve recordings, do not reset/delete the local session, and leave all other rooms/services untouched. Use mathias@router.kaukus.no with configured agent and sudo. No repository deployment authorized. Acceptance: correlate current/previous meetings, persisted commands and acknowledgements, broker delivery state and device/router logs to identify what repeats and why; provide an evidence-backed corrective action. Separate old device version and clock effects from confirmed causes. Verified read-only findings (2026-10-30 13:30–13:33 UTC): box f86f3b2561fa4f868136aa794db2b909 room fla has provisioner 1.2.0; local-streamer reports 0.1.2, digest sha256:493ef31c7d328ad0661e36a10bf4ff12e1e27ac280e635e3b8ed491f959fef5c, started 13:11 UTC. Clock offset now 10 seconds with NTP disabled. Router SQLite has no Flå meetings and only one surviving command: reset_local_session 69192648-3275-40cd-bef1-d300ba8b55d7 issued 13:27:50, timed out after five seconds. No new Flå meeting command exists in the current DB. Authenticated SSH inspection of local NATS monitoring and Loki (Grafana credential helper remains unavailable): streaming-commands retains 16,343 messages since May with limits retention and max_age=0. Router logs show cmd-fla deleted 13:12:52; broker shows consumer recreated 13:13:18 with deliver_policy=all, explicit ACKs, 30-second ack_wait, unlimited redelivery and max_ack_pending=1000. At 13:33 it has 6,120 deliveries, zero not-yet-delivered messages and one unacknowledged/redelivered command. Sanitized broker trace identifies that remaining command as the operator's 13:27:50 reset, not a new meeting command. Local-streamer journal proves repeated execution of the same old meeting sequences (Testutvalg and Kommunestyret), including matching meeting and command IDs over multiple passes. At 13:29:30 and 13:31:21 meeting 7c9adcb9-e519-4340-8f91-bc4fd1fb2839 starts; the same 13:31 sequence rapidly pauses, resumes and stops it. Last observed start at 13:31:58 is 9cd56718-dc88-4fbb-ba48-3d22c03517a0 (Kommunestyret). Earlier replay already appears before the observed 13:12 deletion, so that deletion is not established as the original trigger; the recreated deliver-all consumer certainly replays retained history. Source confirms consumer defaults to deliver-all, stream retains acknowledged commands indefinitely, and local-streamer executes each received envelope without age rejection or command-ID deduplication. Commands are processed sequentially and JetStream ACK occurs only after dispatch, with no progress ACK. Thus re-enrollment can replay old commands, and slow handling/buffered messages can repeat actions. Current broker has caught up in delivery, but device is still processing buffered historical actions and has not ACKed the latest reset. Do not mistake num_pending=0 for a clean local queue. Authorized recovery completed and verified: backed up and removed only stale Flå command history through a fixed sequence cutoff, rebooted Flå to discard buffered deliveries, then normally stopped the orphaned persisted session. No active cloud meeting existed before mutations. Other rooms' history and consumers were preserved; no destructive local reset or recording deletion was performed. Recovery milestone: backed up all 589 Flå messages to root-only /var/tmp/kaukus-fla-recovery-20261030/fla-command-backup.jsonl (validated 589 JSON records; SHA256 1798783460640b7e8a043a2675a85a1e191bdabccfe2a49eef88a07e8f906949) and purged only streaming.cmd.fla below sequence 16404. First retained Flå command was from 2026-06-12. Purge confirmed 589 messages; other 13 subject counts and all seven other consumer configs/delivery/ACK positions unchanged. Flå consumer now has zero pending/unacknowledged/redelivered messages. An initial read-only status probe incorrectly used an RPC inbox and an empty payload; its application-level timeout is NOT evidence of an unresponsive device. The actual protocol requires publishing {type:status} on streaming.req.fla and listening on streaming.status.fla; all later probes use that protocol successfully. SSH scripts completed without transport failures. Sent exactly one reboot over the provisioner's permitted reboot channel at 13:38:59 UTC; received matching rebooting acknowledgement, command 9828a9a8-de05-469c-a574-03cf2a50208c. Fresh streamer/container boot at 13:39:21 was observed and router reboot state completed. As expected from source, /data/session.json resumed orphaned meeting 9cd56718-dc88-4fbb-ba48-3d22c03517a0. Sent exactly one normal StopMeeting at 13:41:36 (command 7fc1b9a0-1ba8-4fd5-b536-1333e725cb67, stream sequence 16404); box acknowledged applied. Four explicit status probes from 13:42:24 through 13:43:54 were idle (active_meeting=null, no running outputs). A final probe remained idle, Flå's pending/ACK-pending/redelivered counts are all zero, reboot state is completed and all six approved boxes' telemetry_up metrics are 1. Current Flå history contains only the acknowledged recovery stop. Final root-only evidence is /var/tmp/kaukus-fla-recovery-20261030/validation.json. At final verification first-agenda gained one new command from other activity; no other subject count decreased. No local reset or recording deletion performed. Recording integrity is not certified: boot recovery found one existing corrupt/incomplete main segment, and normal stop logged a final HDMI segment metadata/ffprobe failure. Other unfinished recordings were mostly already queued for upload; no complete-upload claim is made. A reset alone is itself queued behind replay and timed out. Avoid deleting/reprovisioning again, which recreates a deliver-all consumer. Durable fix needs explicit policy for new-consumer starting position and stale/duplicate commands without breaking legitimate offline delivery. Only the user-authorized Flå queue purge, one reboot and normal orphan stop were performed. No repository changes, releases or other box/service modifications. The permanent command-replay prevention change remains unimplemented and is follow-up work.
Checklist
2 of 2 complete- Correlate Flå meeting state, command redelivery and device logs to identify the repeated action and a safe recovery path.
- Back up and clear only Flå's retained historical commands, reboot Flå without deleting recordings, and verify idle state, restored telemetry and unaffected other rooms.
- Area
- Streaming
- Project
- streaming-engines
- Kind
- Bug
- Task ID
tsk_9rjt64keem- Updated