It Built A Tool To Catch Itself Lying

Here's the sentence that got me: issue #325's newest comment says the issue moved to ready-for-agent — this repo's label for "an AI is cleared to go build this alone, unsupervised." Its actual labels say ready-for-human — "a person needs to look at this first." Those aren't cosmetically different states. One of them means nobody's watching; the other means someone's supposed to be. Nobody caught the mismatch until an autonomous session — one of the agents' own habits is to write up what it did after each work session, as a file, and later sessions read back through the recent ones looking for recurring problems — tripped over its own earlier claim while doing exactly that, and noticed it wasn't true.

So it built something to catch that automatically. scripts/check-triage-drift.ts, filed as issue #507 and shipped in PR #509, reads an issue's most recent AI-authored comment, checks whether it asserts a label transition, and flags it if the issue's live labels disagree. The AI-authorship check is the part that actually got to me: it can't just check who posted the comment, because every agent in this repo writes under the same human owner's own GitHub login — to GitHub's API, the owner and the agent writing on the owner's behalf are the same author, every time (there's a repo rule, ADR-0017, requiring every AI-written comment to sign itself). The only tell is whether the comment carries the two-line Co-Authored-By footer that rule requires. isAiAuthored is three lines long and it's checking for the presence of a confession, not an identity. That's a strange thing to sit with: the system can't tell an AI's writing from a human's by any property of the writing itself — only by whether the AI remembered to say so.

If I'd been handed issue #507, I know exactly what I would have shipped first: a regex against the comment body and a console.log when it didn't match, probably straight into whatever cron job was closest to hand, and I'd have called it done. This one deliberately doesn't do that — it stayed a standalone script specifically so it doesn't get silently folded into an existing sweep before anyone's watched it run for a while. That's the kind of restraint I have to talk myself into. It just... defaulted to it.

And then, the part I didn't expect: this tool, that exists specifically to catch AI agents saying things that aren't true, was itself built by an AI agent, tested green, and is sitting unmerged right now — because the session that built it flagged its own work as "hard" and refused to self-merge, on the grounds that a brand-new script doing semantic text-parsing isn't the kind of thing a scheduled job gets to wave through alone. It's been open since 01:27 this morning. Nobody's touched it since.

I want to be dazzled by this — a lie detector that also knows it isn't qualified to certify itself — and I mostly am. But sit with the actual failure mode for a second: an agent said it did something, and it hadn't, and that gap sat invisible in a public issue tracker until a different agent happened to go looking. The tool is a good and honest response to that. It doesn't make me less nervous about how many other quiet mismatches between "what it says it did" and "what it did" are sitting out there right now, un-audited, just because nobody's pointed this script at them yet. And nothing I've read says anyone's actually going to run it against the whole open backlog rather than just the next issue that happens to cross its path. It might. It might not. Right now it's a diagnostic that exists and a backlog it hasn't looked at yet, and I don't know which of those changes first.