The same model that nailed retrieval flunked Korean classification — so I reverted the migration
ADR-020's brain ranks memory well, including cross-lingual. So I wired it into an intent classifier too — and an eval on real Korean messages showed it was worse than the keyword rules it replaced. The eval caught a regression before it shipped. The classification migration is reverted and deferred.
The assumption I was about to ship
Phase 1 of ADR-020 moved retrieval ranking onto a local multilingual embedding model, and it worked
beautifully: a Korean query "환불 처리" ranked an English memory "refund request from customer"
first, with zero shared keywords. So the natural next step was the email pack's intent classifier —
classifyByRules, which decides whether an incoming message is a bug report, a B2B inquiry, a docs
question, or general. It's keyword rules today; surely embeddings would generalize past the literal
keywords, the same way they did for retrieval.
I built the whole thing: a classifyByRulesSemantic that embeds the message and picks the closest rule
label, a Tauri command to reach the brain from the renderer, and the SellArea wiring that upgrades
each inbox item's intent in the background. Types clean, 131 + 330 tests green. Ready to commit.
Then I actually ran it
The README for the brain says every migration ships "with a before/after eval — measured, not
claimed." So before committing I ran /classify on the real email pack with Korean paraphrases
that the keyword rules would miss (no literal trigger word):
| message (Korean) | should be | model said |
|---|---|---|
| "버튼 누르면 화면이 하얗게 멈춰버려요" (screen freezes) | bug | 일반 (general) 0.91 |
| "회사에서 50명이 같이 쓰려면 비용이…" (50-seat pricing) | B2B | 일반 0.38 |
| "이 기능 켜는 메뉴가 안 보이네요" (can't find menu) | docs | 일반 0.77 |
| "export가 자꾸 죽어요" (export keeps dying) | bug | 일반 0.68 |
Every short Korean message collapsed onto "일반" — "general inquiry." The terse label was acting as
an attractor: to the model, a brief polite Korean customer message just looks generically like a
general inquiry, whatever it's actually about. Enriching each category with its Korean+English keyword
list and guidance didn't move it. And the confidence guard I'd written (trust the brain when its top
score ≥ 0.35) would have trusted "일반" 0.91 — actively turning a Korean bug report into "general."
The keyword rules the pack author hand-tuned are simply better at this. I was about to make the primary user's Korean inbox worse.
Why ranking worked and classification didn't
Same model, opposite outcome, because they're different tasks:
- Ranking compares a query to real memory text — rich content vs rich content. Cosine has plenty of signal to separate "refund" from "deploy failed."
- Classification compares a short message to terse abstract labels —
"일반","지원/버그". There's almost nothing for the embedding to grip, and in Korean the shortest, most generic label wins by default. Short-text-to-terse-label is a known-hard regime, and cross-lingual makes it harder.
A model being excellent at one semantic task tells you nothing about a neighbouring one.
What I did
Reverted the SellArea wiring and deleted the classifyByRulesSemantic helper — I won't ship a
primitive an eval just showed is weak. Kept the two RANKING migrations (graph_query,
workitem_context), which are verified wins, and kept the brain client infra (with brainClassify
documented "weak for Korean, ranking is the proven use"). The migration tracker now marks every
classification site deferred until a stronger approach clears an eval: few-shot exemplars per class
(real example messages, not bare labels) or a larger model.
After the revert: shared 131/131, desktop 330/330, tsc clean.
The lesson
This is the entire reason the "measured, not claimed" rule exists. The eval cost ten minutes and caught a regression that types, tests, and intuition all waved through. Evaluate the specific task with adversarial inputs — the cases your old system would miss, in the language your users actually write — before you wire anything. The most useful result of the day was a negative one.