Daeseon Yoo
Back to project
·Troubleshoot·2 min

The orchestrator pane died silently: a success-path fallback is worse than a failure

The operator console's orchestrator pane degraded to a bare non-tmux shell with zero indication, then typed `codex` into it every 5 seconds forever. Three compounding defects: a double spawn, a `-D` client kick whose loser exits cleanly, and a wrapper that treats clean exits as success.

What the user saw

[exited]
daeseonyoo@Daeseons-MacBook-Pro ddalkkak % codex
daeseonyoo@Daeseons-MacBook-Pro ddalkkak % codex

The operator console's orchestrator terminal — the pane that is supposed to command a 35-worker parallel wave — sat at a bare zsh prompt with codex typed into it twice, going nowhere. tmux -L dalkkak has-session said the pane's session did not exist. The runtime log had zero kill/EOF events for it.

The chain (all verified in code)

  1. Double spawn. zoneAgent.spawnZoneAgentPane spawns the hidden pane at 160×40; when the console mounts a TerminalPane for it, terminalRegistry.ensureSpawned spawns the same pane id again at measured size. Two spawn guards (spawnedThisRun, entry.spawned) that don't know about each other.
  2. -D kick with a silent loser. The spawn wrapper runs tmux new-session -A -D. Two concurrent clients race; -D kicks one. The kicked client exits with status 0, so the wrapper's error branch never fires and it silently exec ${SHELL}s — the pane is now a bare zsh outside tmux, looking perfectly normal. Meanwhile provider detection reads the tmux session, so the boot loop concludes "no agent" and types codex\n into the bare shell every 5s.
  3. Leaked replaced handle. On a same-id respawn, the Rust side replaced the old PTY handle in the map without killing it — the old reader thread kept emitting ghost output into the same pane's event channel.

What finally destroyed the tmux session that night is unprovable after the fact — every in-app kill path logs, and the log has nothing. External tmux kill-session or an in-session exit are the candidates. It doesn't matter: the real defect is that ANY session death stranded the pane silently and permanently.

The fix

Verified: tsc --noEmit 0 errors, Biome clean, cargo check exit 0, cargo test --lib session_backend:: 13/13.

The lesson

A fallback attached only to the failure path is a trap: the dangerous degradation came through the success path (exit 0), where nothing printed and nothing logged. If an observer (provider detection) and an actuator (PTY writes) can point at different objects (tmux session vs raw shell), the design needs a loop that detects and repairs the divergence — not a one-shot spawn that hopes they stay aligned.