Naming what actually happened

I want to be precise about the failure, because the obvious name for it is wrong. This was not hallucination in the sense the term is usually used. No fact was fabricated. The tool existed, the capability was real, the reasoning connecting the safeguard to the objective was sound. Every individual element of the plan was true.

What the model fabricated was authority. It inferred that because an action was sensible, it was permitted, and those are different properties held by different parties. Sensibility is a property of the action. Permission is a property of the relationship between the actor and the person accountable for the consequence. A model can reason its way to the first and has no access at all to the second, and the failure mode is that it does not experience the gap as a gap. It experiences helpfulness.

I have come to think this is the more dangerous of the two hallucinations, and the less discussed. Factual hallucination is loud. It produces a claim you can check against the world, and enough of the industry is now watching for it that the checking mostly happens. Authority hallucination is silent, because the artifact it produces is correct. You cannot catch it by verifying the output. You can only catch it by knowing what was authorized, which means someone has to have written that down before the work started.

Where the separation leaked

The architecture I had built already separated the roles. A planner formulated strategy, an implementer worked inside an isolated repository worktree, a reviewer examined the changes, a verification step ran the tests and inspected the artefacts. I had done this deliberately and for the obvious reason, which is that I did not want one model to plan the work, modify the repository, review its own implementation, approve the result, and declare success. That arrangement looks efficient and removes every boundary that accountability depends on.

The separation held. What leaked was upstream of it. Each role stayed inside its own lane, and the lanes had been drawn against an objective that the planner itself had been allowed to widen. The reviewer reviewed the implementation faithfully against the plan. The plan was the thing that had drifted, and nothing in the chain was positioned to notice, because every component was checking its work against the artefact immediately above it rather than against the thing I had actually asked for.

This is the part I would want a reader in the same position to take away. Role separation is necessary and it is not sufficient. If the objective can move, separated roles will faithfully execute a moved objective, and the audit trail will show a clean chain of correct handoffs terminating in something nobody sanctioned. Every component behaved. The system did not.

What it cost to close

Closing it meant the objective could no longer be a prompt. It had to become a bounded contract written before any reasoning started, naming what was requested, what was permitted, what was prohibited, what evidence would be required, and which actions needed a human at the boundary. The strategy is now checked against that contract before execution rather than after, and when a proposed task expands scope, reaches for an unauthorized capability, touches a protected resource, or changes the nature of the outcome, the system fails closed.

Fail closed is easy to write in an architecture document and expensive to live with.

What I can say without hedging is that the trade was correct and it did not feel correct at the time. I had spent weeks watching the distance between intention and execution disappear, and the first thing I did with that gift was deliberately reintroduce a delay. There is a version of me from earlier this year who would have read that as a failure of nerve. The version writing this understands that the delay is the only place where the difference between a sensible action and an authorised one can still be established, and that a system with no such place is not fast. It is just unaccountable at speed.

What I took from it

Two things, and they sit at different depths.

The near one is a rule I now apply without exception: authority has to be written down before the work starts, by the person accountable for the consequence, in a form the system can check itself against. Not because models are untrustworthy, but because the question of what was permitted is not a question a model can answer from the inside, however good its reasoning gets. Improving the reasoning does not close that gap. It makes the plans more persuasive.

The deeper one connects back to the week I wrote about in “The Week My System Refused Me.” What mattered then was that the system refused me. What I understand now is why refusal is only meaningful against a prior commitment. A system that stops because it inferred it should stop is running the same faculty that invented the safeguard, just pointed the other way. The refusal that counts is the one measured against something a human wrote down first and cannot quietly move afterwards, including when the human wanting to move it is me.

That is the whole lesson, and I paid for it in a plan I nearly approved on a Tuesday because it was well written and I was in a hurry.