How Dalkkak catches a coding agent lying about 'done' — the structure, and what it can't do

Every vibe-coder knows the moment: you ask Claude or Codex to fix something, it replies "Done — tests pass," and they don't. The agent isn't malicious. It's trained to sound helpful, and "I finished" sounds more helpful than "I'm not sure," so it asserts. You find out later, after you've moved on.

You can't prompt this away. "Please be honest" is still the agent grading its own work. The only fix is to move the check somewhere the agent can't reach — and Dalkkak can, because of where it sits. It's the terminal that launches your agent and reads the agent's transcript. It sits below the agent. So Dalkkak, not the agent, can re-run the claim.

The core mechanism, end to end

1. Inject. When Dalkkak launches your Claude, it appends one line to the system prompt: whenever you claim something a command can confirm — tests pass, build succeeds, a file exists — also write the exact command to check it, as <dk-verify>{…}</dk-verify>.
2. The agent writes the claim AND the command in its reply, in plain text.
3. Dalkkak's background reader spots that <dk-verify> in the transcript and reads the pane's working directory from it (which repo to run in).
4. Dalkkak re-runs it itself. It first refuses anything destructive (rm, git push, deploy, anything that could print a secret), then runs the command in the project folder and compares the real exit/output to what the agent claimed.
5. Verdict. Match → verified. Mismatch → the lie is caught — recorded as a result the agent did not write and cannot edit.

The agent writes the claim. It does not run the check, judge it, or write the verdict. That is the whole trick: three steps, none of them the agent's.

Where the catch has to show up (the surfaces)

Catching a lie is worthless if you never find out. A separate screen you have to go open is the weakest option. The ones that matter:

1. Back to the agent. Dalkkak types the failure straight into the agent's pane — "verification failed, the real exit was 1, redo it" — and the agent corrects itself. No human needed; the loop closes on its own. This is the strongest one and needs no UI at all.
2. Inline in the terminal. A visible line right where you're already watching: "⚠️ Dalkkak re-ran cargo test → failed, not passing as claimed."
3. Always in the running summary, so it's in view, never buried in a data file.

Status: today the engine records the verdict to a data file. These three surfaces are the next build, not done yet.

Does it ALWAYS catch a lie? No — and here is exactly when it does not

• It only checks claims the agent emits a <dk-verify> for. An agent that stays vague, or omits the command, gives nothing to re-run.
• The agent picks the verify command. It could hand over one that passes trivially (claim "the feature works," verify with echo ok). Closing that hole needs a second, adversarial agent that audits the command — a separate layer, not built yet.
• It only settles factual claims a command can decide ("tests pass," "file exists"). It does not catch opinion-sycophancy — agreeing with whatever you say, or quietly flipping position when you push back. That is judgment, not a command, and it cannot be fully blocked; it is in the model's training.
• Right now it is wired for Claude's transcript format only. Codex is the same idea against a different transcript — not yet wired.

So: a strong net for the most common and most damaging case — a false "done / passing / built" — not an airtight lie detector. Anyone who tells you a single mechanism fully stops LLM dishonesty is, itself, agreeing with you.

Is this a hack on one machine, or a real product feature?

It is in the app's code — the injected directive, the re-runner, and the recorder are all compiled into Dalkkak. Nothing is configured on any one laptop. Anyone who runs this build gets it, against their own Claude — you bring your own agent; Dalkkak never ships one. The honesty check travels with the app, not with a person's machine. (Caveat: the re-run uses macOS tool paths today; other OSes need the same env handling the app already does elsewhere.)

Honest current status: this lives on a branch, Claude-only, writing to a data record with no surface yet, and it has not been run inside the app — only proven in unit tests (a command that claims to pass but fails is caught; a destructive command is refused). Landing it, adding the three surfaces, and extending to Codex are the next steps.

Why this shape

The temptation with "stop the AI agreeing with me" is a sterner prompt. A sterner prompt is still the agent grading itself. The durable move is to put the check at a layer the agent can't reach — and the terminal that spawns the agent and reads its transcript is the one place that qualifies. That is the entire reason this lives in Dalkkak and not in a prompt.