Keeping Python the right way: ONNX instead of PyTorch, and a brain that runs as a background service

Where the morning left it, and where I kept going wrong

By midday the Python brain worked: it ranked and classified well, on-device, no API. But it had two real holes — it broke on the founder's own machine (a venv bootstrap failure), and it only resolved a dev repo path so it wouldn't run for any shipped user. My instinct was to rip Python out and do it in Rust's existing fastembed engine: in-process, ships in the binary, no separate process, no packaging problem. I kept pushing that, turn after turn.

The founder kept saying: keep Python — fix the install. He was right, and the reasons that came out of pushing on it are the actual content of the day.

"Why does Rust not have a separate install?" — because it compiles in

The honest distinction I should have led with: Rust compiles into the .app (51MB binary), so a user installs nothing. Python is interpreted, so it needs an interpreter + libraries present on the user's machine at runtime. That's the whole asymmetry. And it's exactly why most "AI is Python" SaaS have no problem: they run Python on a server that starts once and serves forever — the desktop, on-device case is the rare one where Python's runtime weight bites.

"What's the 950MB, and why did you bring it all?" — PyTorch, and I wasn't watching size

Measured, not guessed: the venv was ~950MB, of which PyTorch was 441MB plus the scientific stack (transformers, scipy, sympy, sklearn) that sentence-transformers drags in. We don't train models; we only run one forward pass + cosine. PyTorch is a training framework — we used ~1% of it. I'd reached for the easy default library without thinking about distribution size. (I'd also earlier said "2-2.5GB" — the real measured number was 950MB. Overstated again.)

The fix that keeps Python AND makes it shippable: ONNX

fastembed (Qdrant's library) does embeddings via ONNX Runtime — no PyTorch — and supports the exact same model we were using (paraphrase-multilingual-MiniLM-L12-v2). Swapped it in:

• venv 950MB → 219MB (no torch); model 458MB → 220MB (the ONNX export is smaller)
• same vectors: KO "환불 처리" → EN "refund request" cosine 0.529 both ways (was 0.53 on PyTorch)

The irony I owe the founder: the "light Python" path (ONNX) is literally what Rust fastembed already does. So keeping Python doesn't cost much weight once you drop PyTorch — and you keep the Python ML ecosystem for later. His call held up.

Two more holes the founder named

The install break. On his machine the system python3 resolved to a 3.14 build whose ensurepip failed, so the venv bootstrap died and the brain silently fell to keyword. setup.sh now probes candidate pythons (3.12/3.11/…) and picks the first whose venv+pip actually works — never blindly trusts python3.

The silent fallback. When the brain is down the app falls back to keyword — but it did so with no log and no signal ("병신 두뇌로 도는 거 아님?" — exactly). Added a BRAIN_UP transition log (only on up↔down) and a 🧠 / ⚠️ keyword badge in the top bar, so the dumb mode is never invisible. Not a hard error — the brain has a brief warm-up, and erroring every query during it would feel broken.

The one that mattered most: a background service, not a child of the app

The brain was a child process of the app — spawned on launch, killed on quit. So the model reloaded (~5s) on every launch and died whenever the app closed. The founder: "당연히 계속 트는 거 아니냐, 앱이랑 상관없이?" Right — real on-device AI (Ollama, Docker) runs as a persistent background service.

Moved it to a launchd LaunchAgent (RunAtLoad + KeepAlive): auto-starts at login, auto-restarts on crash, outlives the app, loads the model once. ai_brain_start is now idempotent — if the agent is already loaded with the same plist it leaves it warm (no per-launch restart); it only reloads when the plist changes. Verified live: quit the app → the brain (pid 9358) stays up; kill -9 it → launchd restarts it. And the per-launch 5s — which I'd implied Rust avoids "for free" — is really just the restart-every-launch pattern; a long-lived process (Rust in-app, or Python as a daemon) pays it once.

What's actually true now

• Python kept. Brain is ONNX-light (219MB), observable, and runs as a warm background service.
• Python is the main for semantic ranking + classification (4 sites) — NOT for everything: generation stays BYO Claude, and a Rust fastembed engine still serves memory Q&A / conflict detection (a real duplication to consolidate later).
• Still open for shipping to other users: bundle the brain + venv + model with the app (now feasible at 219MB) instead of resolving a dev repo path.

The lesson

The founder's pushback drove every right call here — keep Python, question the 950MB, run it as a service. My job on a day like this is to follow the goal (a shippable on-device Python brain), not re-litigate the language each turn because the local-minimum looked lighter in Rust. And: measure before you quote a number. I quoted 26 sites, then 2.5GB — both wrong, both caught. The fix isn't to talk more confidently; it's to run du and git diff --numstat first.