유대선
프로젝트로
·트러블슈팅·1

Re-ran the rate-limited verify — 22 wrong model answers, corrected

The deep-study layer shipped generate-only because the verify stage was rate-limited. Re-running the fact-check found 22 model answers across 21 cards that were technically wrong — about 3.5% of the generated answers — and corrected them, so the answers are now verified, not just generated.

The deep-study answers had shipped unverified — the verify stage of that workflow was rate-limited and the pipeline fell back to the generated draft. So the first follow-up was to actually run the fact-check, and it earned its keep: 22 of roughly 600 model answers (across 21 cards) were technically wrong or misleading enough that a senior would catch them.

The catches were real, not nitpicks. Availability claimed 99.9% allows about 43.8 minutes of downtime in a 30-day month — it's 43.2; 43.8 is the average calendar month, and the answer didn't state which window. Idempotency told you to write the Redis key "in the same transaction as the charge," which is impossible across two systems — the key and result have to live in the same relational database as the charge, with Redis only as a fast first-line dedup. Sharding called a range-allocation ID service "strict global ordering" when it's only loose. Schema-registry said backward compatibility lets you remove any field directly, when only defaulted or optional fields can go. Kubernetes-deployment had the maxUnavailable and maxSurge surge semantics muddled.

Applied the 22 corrected answers back into the cards (13 in concepts, 8 in AI) by rebuilding each touched card's answer array, tsc clean.

The lesson worth keeping: a rate-limited verify is not a skip — re-run it when the limit clears. And about 3.5% of generated technical answers were genuinely wrong, which is exactly the error rate that makes the verify step non-optional for anything the user memorizes and repeats in an interview.