A live architecture map is worthless the moment it's confidently wrong
Building an always-live, agent-driven architecture map forced a hard product realization: the goal isn't maximal freshness or coverage — it's calibrated confidence. Every claim must carry its provenance, staleness must be visible, and the system should promote claims up the trust ladder rather than hide uncertainty.
We're building an architecture map for DalkkakAI: point it at a repo, get a C4-style zoomable view (Container → Component → File → Symbol), click anything to open it in Cursor at the exact line, and — the actual product bet — have it stay live as agents (Claude, Codex) write code through the harness.
The drawing turned out to be the easy part. The hard part is the question a user asks in the first ten seconds: can I trust this? For this product that isn't a nice-to-have. If the map is confidently wrong even once, nobody believes any of it again — including the 99%-accurate parts. So the real engineering problem is trust calibration, and going real-time makes it sharper, not easier.
Three kinds of claims, three accuracy profiles
The trap is presenting everything with the same visual confidence. A map actually makes three very different kinds of claims:
| Claim | Accuracy | Verifiable by user? | Real-time cost |
|---|---|---|---|
| Parsed fact — files, symbols, line numbers, explicit imports, git churn/coupling | ~99%, deterministic | yes | ~10 ms (cheap) |
| Derived — the dependency graph | sound but incomplete | partially | cheap-ish |
| Inference — AI subsystem grouping, AI summaries, risk flags | probabilistic, can be confidently wrong | dev: yes / non-dev: no | expensive |
Parsed facts come from tree-sitter + a per-language import resolver; they're as accurate as the source. The dependency graph is sound but incomplete: every edge we draw is real, but we resolve only explicit imports — dynamic imports, DI, reflection, and module-level languages (Swift, Go) have no file-level import edges at all. So the honest framing is "a drawn edge is real; the absence of an edge does not mean no dependency." Never imply completeness.
Inference is the dangerous tier. It's the most human-readable, so it's the most trusted — and a non-developer literally cannot sanity-check it. An AI summary that's subtly wrong, read by a founder who can't verify it, poisons trust in the entire tool. One confident wrong claim is more expensive than ten honest "I don't know"s.
That's why source badges (📄 author's doc-comment = fact vs ✦ AI-generated = inference) aren't cosmetic. They're the trust architecture. Measured on two real repos: doc-comment coverage was 79% of files on the well-documented codebase and 20% on the other — the rest filled by AI and marked as such, never blended in.
Real-time adds a second failure mode: mixed freshness
Batch generation can be wrong, but at least it's uniformly wrong. Real-time introduces a subtler failure: the cheap layer and the expensive layer update at different speeds, so fact and inference drift out of sync.
Incremental parsing is fast — re-parsing a single changed file measured ~10 ms (1 of 179 files), versus a full cold parse. But AI re-summarization is expensive and can't run per-edit. So the moment an agent edits a file, its structure is fresh while its description still describes the old code. A freshly-moved file shown with an authoritative-looking but stale summary is the single most trust-destroying state the product can be in — worse than showing nothing.
The resolution is not "make AI fast." It's:
- Make staleness visible. If a file changed but its inference hasn't caught up, show "✦ (may be outdated)" — degrade gracefully, never present stale-as-fresh.
- Map intentional states, not keystrokes. Debounce, and prefer save/commit boundaries over every edit, so the map reflects states the author meant to create.
- Invalidate dependents, not just the edited file. Editing B can change what A's import resolves to; local re-parse is locally correct but the graph needs wider invalidation.
The thin-trigger rule
A related temptation is to bake the work into the trigger — run the whole pipeline inside the editor hook. That's wrong on three axes: it lags the agent's editing, it makes the hook a fragile single point of failure, and it drags expensive AI into the hot path. The hook should be a thin signal ("file dirty") that enqueues; a debounced background worker does the tiered work; the view subscribes and patches. Cheap-deterministic work can run often; expensive inference stays lazy, cached, and out of the hot path — and never blocks editing.
The part that surprised me: real-time matters least for the user who needs trust most
A developer mid-task benefits from live updates and can catch an inference error — low stakes. A founder checks in weekly for "what changed, what's risky," doesn't watch a live stream, and cannot verify — so for them an inference error is catastrophic. Freshness helps the audience that least needs the trust guarantees; trust helps the audience that most needs it. The implication: don't over-invest in sub-second liveness. Save/commit-granularity freshness is plenty. Spend the effort on calibrated confidence instead.
The principle
The product's job isn't maximal freshness or maximal coverage. It's calibrated confidence: every claim carries its provenance (fact / incomplete-derived / inference), staleness is visible, and the system is built to promote claims up the trust ladder — an AI guess becomes a declared fact once a human or agent commits it to the architecture manifest — rather than to hide uncertainty behind a confident-looking diagram.
Concretely, that "promotion" is an enforcement layer: a declared .dalkkak/arch.json that the engine reads in preference to AI guesses, with a measured coverage / drift number instead of a fake "99% accurate" badge. On the first seeded repo that read as 92% declared, 12 drift (files added since seeding) — a defensible trust meter you can actually move, not a marketing number.
A diagram that admits what it doesn't know is the only kind anyone trusts twice.