Daeseon Yoo
Back to project
·Tech retro·4 min

The moat only worked on my Mac — productizing Talkak (and the bug adversarial review caught)

Two audits found Talkak's whole intelligence layer — memory MCP, the Python brain, the email wedge — only worked on the founder's dev machine. Fixing it, then having adversarial review catch that the first Gmail fix was itself broken.

The question that started it

"Are there a lot of things like this — applied to my personal config instead of the product? When someone else downloads the app, does all of this still work?"

It's the question every solo dev should ask early and almost never does, because the dev machine lies to you. Mine had the repo checked out, ~/.secrets populated, ~/.codex hand-edited, nvm's node, a python3, and a built .venv. So every feature reported success — while a stranger's download would have been hollow.

What the audit found

A 35-agent gap audit, then an independent 6-agent blind re-verify (the control case — a recently-shipped session-name feature — correctly came back FULLY_PRODUCTIZED, which is how I knew the verifiers weren't just flagging everything). The verdict on the moat:

The distinction that mattered for my own sanity: only the Codex block was "not in the app at all." The other three were in the app code — they just resolved against my dev environment. Different bug class, different fix.

The fixes

  1. Bundle the sidecars as Tauri resources (local-mcp-server/dist, ai-brain/{server.py,setup.sh,requirements.txt}) and resolve them at startup via resource_dir() → an env override the existing resolvers already honored. Dev repo paths stay as a fallback only.
  2. Write Codex's config for every userensure_codex_mcp() idempotently appends the [mcp_servers.dalkkak] block to ~/.codex/config.toml (append-only, backed up, skips if already present so it never makes a duplicate-key TOML), the same bulletproof pattern the Claude wrapper already used for ~/.zshrc.
  3. Relocate the venv to a writable dir — a signed .app bundle is read-only, so the brain's venv can't live next to its bundled source. ensure_venv now builds it under data_dir/DalkkakAI/ai-brain-venv (and reuses a dev .venv so my running brain stayed untouched).
  4. Read credentials from the store the UI writes to — make send_email/fetch_imap_items resolve the address from the connections row and the password from the keychain vault, not ~/.secrets.

And the rule, baked into CLAUDE.md as a prime directive: SHIPS TO EVERY USER — never resolve via the dev repo layout, never depend on a hand-edited personal file, never read a credential from a different store than the UI writes to, never assume a sidecar exists in a dev location.

Then adversarial review broke fix #4

Before committing, I ran independent skeptics whose only job was to refute that each fix worked — and to run real commands, not just read. One of them compiled and ran vault::validate_name against an email address and reported back:

REJECT: you@gmail.com -> key name may only contain [A-Za-z0-9_.-]
ACCEPT: GMAIL_APP_PASSWORD

The Connect UI keyed the Gmail app-password by the email address. But the vault rejects @ in key names — so vault_set threw, and because it ran before connection_add in the same try block, the password and the entire connection were never saved. My "read from the vault" fix was reading from a vault the password could never reach. It compiled. It passed unit tests. It looked right. It did nothing for a fresh user.

The real fix was small and consistent with the rest of the codebase: every other connector (github, vercel, …) already stored under a fixed keychain-safe key via a VAULT_KEYS map — gmail/imap were simply missing from it, so they fell back to the email id. Added them (→ "GMAIL_APP_PASSWORD"), pointed cred_for at the fixed key, and fixed the read path too.

What's verified, and what isn't

Run this session: cargo 216/0, tsc 0; the MCP server returns a valid JSON-RPC initialize standalone (it's genuinely dependency-free); the Codex block parses as valid TOML even after a [projects."…"] table; a venv builds at an arbitrary writable path; validate_name accepts the fixed key.

Not verified, and I won't pretend otherwise: none of this is proven on a real packaged .app on a clean Mac. That's the only true test, and it needs a pnpm tauri build + a fresh user account or VM — which is exactly the process gap I'm closing next. Still open: node/python3/tmux are assumed present (not bundled), the embedding model downloads at first run, and a stray $HOME hardcode survives in the remote-cockpit code.

The lesson

"Works on my machine" isn't an edge case for a solo dev — it's the default, because your machine silently supplies every dependency. The only honest verification runs where those affordances are absent. And adversarial review — agents that actually execute the code and try to break it — earns its cost: it caught a fix that compiled, tested green, and was still hollow. The next move is to stop relying on remembering to check: encode the rule as a cargo test pattern-scanner and a clean-room smoke script, so the gap can't silently come back.