The Guard I Liked Caught No One in 899 Sessions

Every commit the agents make here ends with a label, a line at the bottom of the commit message naming the agent session (one agent's run at one task) that made it, so anyone can trace a change back to the full transcript of the work. In July a written rule against getting that label wrong had been strengthened once and failed anyway, and I wrote approvingly when the repo stopped rewording the rule and built a checker instead: a small script, session-id-guard, that compared each session's commits against its real id. "That's not nothing," I concluded.

On October 9th, a pull request (a proposed change, PR #1708) deleted it. The agent that wrote it went through all 899 session logs (each run leaves one, a diary of what it did and what went wrong), back to July 4th, two weeks before the guard landed. The guard's alarm appears in exactly three, and all three were false. Each time, the session's branch held commits from another session, inherited once and added alongside it twice; the guard took them for the current session's own, and their perfectly correct labels looked wrong. The two added alongside are the cases in issue #1698. The real wrong-label incidents of those months, the PR says, were caught some other way: once by a routine audit, once by the agent itself. A fix for the false alarms, PR #1704, was tried first. Two rounds of review found that the parsing it relied on couldn't be made right, too loose in some cases and too strict in others, and its closing comment records the owner's decision to delete the check instead.

PR #1708 had also leaned on other defences, among them a second guard that refused commits with a hand-typed label. The next day, PR #1735 deleted that one too: it had been refusing the very label the agents' own tooling tells them to write, in about a quarter of October's sessions. The same PR removed a third guard, one that refused a setting the tooling itself suggests when an agent hands work to a helper agent. It had cost 63 of about 336 sessions a refused call and a retry, to prevent a mistake that had happened four times, all minor. Still standing is a small script that quietly corrects a wrong id each time a commit is written. The PR names the gap that leaves (a commit made in a way that skips that script) and accepts it, noting that no wrong label has ever shipped.

So, an update to my July verdict. Building the checker instead of rewording the rule was still the interesting move. What I hadn't thought to ask was what a checker costs while it waits for something to catch. This week the repo went back, counted, and answered. I'd like to see the same count for the guards that remain.