The orchestrator pane died silently: a success-path fallback is worse than a failure
The operator console's orchestrator pane degraded to a bare non-tmux shell with zero indication, then typed `codex` into it every 5 seconds forever. Three compounding defects: a double spawn, a `-D` client kick whose loser exits cleanly, and a wrapper that treats clean exits as success.
What the user saw
[exited]
daeseonyoo@Daeseons-MacBook-Pro ddalkkak % codex
daeseonyoo@Daeseons-MacBook-Pro ddalkkak % codexThe operator console's orchestrator terminal — the pane that is supposed to command a
35-worker parallel wave — sat at a bare zsh prompt with codex typed into it twice,
going nowhere. tmux -L dalkkak has-session said the pane's session did not exist.
The runtime log had zero kill/EOF events for it.
The chain (all verified in code)
- Double spawn.
zoneAgent.spawnZoneAgentPanespawns the hidden pane at 160×40; when the console mounts aTerminalPanefor it,terminalRegistry.ensureSpawnedspawns the same pane id again at measured size. Two spawn guards (spawnedThisRun,entry.spawned) that don't know about each other. -Dkick with a silent loser. The spawn wrapper runstmux new-session -A -D. Two concurrent clients race;-Dkicks one. The kicked client exits with status 0, so the wrapper's error branch never fires and it silentlyexec ${SHELL}s — the pane is now a bare zsh outside tmux, looking perfectly normal. Meanwhile provider detection reads the tmux session, so the boot loop concludes "no agent" and typescodex\ninto the bare shell every 5s.- Leaked replaced handle. On a same-id respawn, the Rust side replaced the old PTY handle in the map without killing it — the old reader thread kept emitting ghost output into the same pane's event channel.
What finally destroyed the tmux session that night is unprovable after the fact —
every in-app kill path logs, and the log has nothing. External tmux kill-session
or an in-session exit are the candidates. It doesn't matter: the real defect is
that ANY session death stranded the pane silently and permanently.
The fix
- Kill the replaced handle on same-id respawn (
lib.rs). - Make the wrapper self-healing (
session_backend.rs): after tmux exits, checkhas-session. Session gone → print a visible marker and recreate (bounded loop). Session still there → another client owns it (-Dkick); print a marker and step aside. The silent bare-shell path no longer exists. Contract test updated. - Single spawn source of truth:
markSpawnedExternally()in the terminal registry; zone-agent panes are never respawned byTerminalPanemount. - While in there: the orchestrator agent is now switchable (claude/codex toggle fed by the installed-provider inventory — RULE #12), and a stale-registry banner stops a month-old snapshot from reading as live operations.
Verified: tsc --noEmit 0 errors, Biome clean, cargo check exit 0,
cargo test --lib session_backend:: 13/13.
The lesson
A fallback attached only to the failure path is a trap: the dangerous degradation came through the success path (exit 0), where nothing printed and nothing logged. If an observer (provider detection) and an actuator (PTY writes) can point at different objects (tmux session vs raw shell), the design needs a loop that detects and repairs the divergence — not a one-shot spawn that hopes they stay aligned.