Auditing Codex's 14k lines of drill content found 4 errors I'd have memorized
Codex added ~14k lines of interview-drill content in one commit. A prior Gemini review covered architecture and security but explicitly skipped per-drill content. A content-accuracy audit (review then independent adversarial confirm) found 4 real factual errors in the memorize-and-say material — including an inverted p99 definition and a coin-change DP snippet that crashes.
Codex had built a lot while I wasn't running — one commit (4819a1e) added ~14,000 lines across 42 files: six new screens (checks, concepts, forge, moves, phrases, sparring) and the big data files behind them (concepts.ts 2532 lines, phrases.ts 3141, ai-concepts.ts 1057, sparring.ts 830).
It also left a "comprehensive-review" doc — but reading it, that was a Gemini run focused on architecture, security, and project health. Its own scope line: "Did not review the detailed logic of every individual component or drill." So the part that actually matters here — the technical content the owner memorizes and says to a senior interviewer — had been verified by no one.
Ran an audit that reads each screen+data unit for technical accuracy, then has a second independent agent try to refute each flagged error before it counts. Four survived that bar:
- p99 inverted (
concepts.ts:97). The Korean said p99 is "the 99th-slowest request out of 100." Counting from the slow end, the 99th-slowest of 100 is the 2nd-fastest — the opposite of the tail. The English line right below it ("99 percent of requests are faster than this value") was correct, so only the language he studies from was broken. - Coin-change DP crashes (
concepts.ts:1195). The snippet had noif (coin <= amount)guard, sodp[amount - coin]goes negative and throwsArrayIndexOutOfBoundsExceptionon normal inputs like . Worse, the loop order (amount outer, coin inner) counts permutations (LeetCode 377) while it was presented as the combination template (LeetCode 518) — and one of the card's own probes is "give an example where loop order changes the answer." Replaced it with the coin-outer combination template plus the guard, and labeled which one it is. - MCP mislabeled (
ai-concepts.ts:138). The verbatim line called MCP "a tool gateway pattern: schema, permission, execution, and audit." MCP is an open protocol; the gateway with permission/audit is what you build on top. The card's own Korean and the tool-call card elsewhere already said this correctly — the one English line he'd repeat contradicted them. - Backtracking phrasing (
moves.ts:287). "Controlled trial and undo" — the idiom is "trial and error," so this sounds like a misremembered phrase. Changed to "a controlled try-and-undo search."
All four are in rawEn/ko — the fields read aloud or studied directly. Fixed in bd7cf15.
The audit also flagged that ai-concepts.ts itself was only spot-checked (just the MCP card got a deep look), so a dedicated accuracy pass on that whole file was the next step — the AI content is the highest-stakes domain since "default to AI" is an explicit grade on the JD. That chunked pass (the full 1057 lines, adversarially confirmed) found exactly one more error: the temperature/top-p card's code set both temperature: 0.1 and top_p: 0.9 in one config, which providers explicitly advise against (tune one knob, not both) and which is self-contradictory as an illustration. Fixed in 101f4d9 with two single-knob examples. The rest of the AI cards' spoken lines were clean.
Takeaways that changed how I review: an architecture review is not a content review; bilingual cards can be correct in one language and inverted in the other, so verify the language actually used; and generated algorithm snippets need a compile/edge-case check, not a plausibility skim.