유대선
← 프로젝트로
·기술 회고·2 분

The first eval measured the enforcer: a 20-minute scoreboard found three flaws code review missed

After a day of the honesty verifier falsely accusing an honest agent, we built eval v1 as read-back lenses over nodes the app already keeps: a Lab judgment-trend chart and a dk-verify scoreboard. The scoreboard's first run over 491 real receipts immediately surfaced three structural flaws — permanent 'failed' verdicts, an unverifiable Java stack, and a week-old misjudgment pattern.

The founder's frustration peaked after dk-verify's false accusations: "I want real eval." The answer wasn't a metrics platform — it was the house pattern applied to measurement: read-back lenses over memory the app already keeps. No new store, no new capture.

① Lab trend — is my judgment actually improving?

Every sealed Lab rep already persists a lab_scorecard artifact (a Korean scorecard string: overall/100, coverage, catch rate, precision, tone, prediction accuracy). The lens parses those back into numbers and draws the series with a last-5-vs-before delta. The Lab's whole thesis — structured friction trains judgment — finally has a measurable curve instead of a feeling.

② dk-verify scoreboard — watch the watchman

Every re-run verdict is an immutable verification_receipt node. First run over the real graph (491 receipts, 10 days): verified 78%, unsafe 11%, failed 9%, unrunnable 2%. Three env failures (exit 127) branded as lies. Twenty-seven claims whose bad verdict was later overturned. Twelve refusals that only passed after the agent swapped commands — the over-blocking cost of the old denylist, now a number instead of an anecdote.

The eval's first 20 minutes found three flaws a week of code review hadn't:

  1. "Failed" is forever. Verdict nodes dedupe on claim+command hash, so a claim re-verified with the SAME command can never be redeemed in the ledger — it's structurally biased against the agent.
  2. Java projects can't verify. Every ./gradlew attempt in history was unrunnable, and the new allowlist doesn't include it — a whole stack has no way to emit test claims.
  3. The cwd-drift misjudgment wasn't new. exit-2 grep failures matching yesterday's bug signature recur from June 23 — an eval would have caught the pattern a week earlier.

Lesson

An eval is a discovery device, not a dashboard. And the first thing worth measuring is the enforcer itself — a verifier nobody scores drifts into a cop nobody trusts.