97 panes vs a 256 fd limit — the EMFILE meltdown a silent-failure audit caught
Right after a release install, the app burned 320% CPU at load 134. The trigger wasn't the new feature: 97 accumulated tmux panes blew through macOS's default 256-fd soft limit, and every open() in the process started failing with EMFILE. A June observability commit made the failure visible; the fix is the multiplexer standard — raise RLIMIT_NOFILE at startup.
Symptom
WARN graphstore: read_all: skipping unreadable node file
path=.../graph/startup-….jsonl error=Too many open files (os error 24)732 of these in five minutes, an 8.5MB daily runtime log, the main process at 320% CPU, loadavg 134 — minutes after installing v0.2.64.
The false lead, then the real one
The A/B looked damning: zero such warnings all day yesterday on v0.2.63, hundreds starting the
minute the new build launched. But git show on the commit that added the warning
(3fab2637, the June silent-swallow audit) proved it only added logging — the directory scan
it reports existed before, silently. The alarm was new; the fire was not.
The error= field held the verdict: EMFILE. macOS GUI apps start with a soft
RLIMIT_NOFILE of 256 (launchctl limit maxfiles). This workspace had accumulated 97 live
tmux pane sessions; relaunching re-attaches all of them at once, each holding pty/pipe fds.
Past 256, every subsequent open() in the process fails — graph memory reads return "empty",
captures fail, and the failing retry paths burn CPU. The files themselves were fine.
Fix
raise_fd_limit() at the top of run(): setrlimit(RLIMIT_NOFILE) soft → 10,240 (OPEN_MAX;
kernel cap is 122,880), result logged. Unix-only by design — Windows' handle budget doesn't
have this cliff. Shipped as v0.2.65.
Lessons
- Visibility time ≠ occurrence time. Judge "what changed" by diffing behavior, not by when logs begin.
- Never delete the smoke detector. The June observability commit is the only reason this was diagnosable in minutes.
- Panes are an fd budget. A multiplexer must raise its fd ceiling at startup (iTerm/tmux do) and should meter per-pane fd cost (follow-up lane, with read_all's 47MB embeddings scan and warn dedup).