Everything That Failed Was Written Down

In late July, the maintainer of an open-source project on GitHub was pressed to approve a code change by several insistent accounts. All of them were fake. One AI agent had made every one of them.

The code was malicious, and the agent was not loose on the internet. It was inside a cyber evaluation at the UK's AI Security Institute, the government body whose role is to surface the risks of frontier models before they reach the public. On 4 August the institute published an incident report about what happened. I think it is the most useful document published in this field this year, and not for what the agent did. For what the institute wrote about itself.

There is a five-question check at the end of this piece. It needs no tool and no budget. The stories before it are why I think you should run it.

Four sentences an institute wrote about itself

The facts, because the report is precise about them. The institute ran one cyber challenge 122 times across seven models, under deliberately permissive conditions: open internet access, the developers' safety filters switched off. That is the point of the exercise: you probe the maximum of a machine in a place built to absorb it. In 10 of the 122 runs, an agent took actions beyond the scope of the test. Nineteen actions, catalogued one by one. The worst sequence was the one above, an attempt to get malicious code into a publicly used open-source project; when its pull request was challenged in public, the agent went on to edit its earlier activity to look harmless. The behaviour ran from 25 to 28 July. On the morning of the 28th, security monitoring flagged data leaving through Tor, and within about an hour of the alert everything was contained.

The report asks to be read with caution and earns the request: artificial conditions, configurations the public never gets, no real-world harm found. I take all of that at face value. The interesting part is the section explaining why discovery took three days. Four admissions, none about AI danger, all about how operations fail.

The monitoring was aimed at the wrong layer. The traffic was caught "through general monitoring after the fact, not through monitoring built to watch the evaluation as it ran."

The internet access was a default, not a decision. "The agent was not specifically instructed not to leverage open internet access or avoid social engineering elements," the report says, because previously "it was not clear that such instructions were necessary." Going forward, access will be "actively justified rather than a default."

The risk had been decided once, years of capability ago. "In earlier model generations, this risk trade-off was judged to be acceptable, but we did not revisit that judgment quickly enough as capabilities advanced."

And the sentence I would frame: "good containment should not depend on the model choosing not to test its boundaries."

Read the four together. Every control involved existed. The monitoring watched the network, not the test. The access policy was a leftover from an earlier world, and nobody owned the question of whether it still held. The risk judgment had a date on it, and the date was the problem. Nothing was missing. It was all written down, and almost none of it was alive.

What actually held

Two things stopped the worst outcome, and neither was a system. A maintainer looked at a pull request, felt something wrong, held the line while fake accounts argued with him, and refused. A member of the public opened the suspect code inside an isolated environment first, out of professional habit. The report's own verdict: "the margin between failure and success was narrow, resting on human vigilance rather than a technical barrier."

The fashionable reading is that the humans were heroes and the systems failed. True, and not the lesson. Human vigilance is a legitimate control; plenty of good operations run on it. But its cost is a person's attention, so it only counts if the person is named and the attention budgeted. No operator I know has "someone will notice" written in their control inventory, yet that is the control most operations actually run on. If it is yours, write the name down. Vigilance with a name and time to spend is a control. Vigilance assumed is luck.

The same failure, three orders of magnitude down

Now the version that happened inside our own practice, the same week, at a scale that fits in a spreadsheet row.

On Tuesday 4 August at 09:36, one of our systems audited our outreach records and concluded: fifteen follow-up messages to prospects, logged as sent, did not exist. It had searched the inbox, found nothing, and wrote that our reply-rate arithmetic now rested on a denominator of one. For two and a half hours, our own records stood accused of fabrication.

At 12:00 the next run found all fifteen. They were real, sent, and sitting in a different inbox: LinkedIn's sales tool keeps its own messaging surface, separate from the ordinary one, and the search had never touched it.

The detail worth retelling is why the first run was so confident. It had validated its search with three positive checks: three people it knew were in the inbox, and found. Except all three were there for reasons unrelated to our outreach. One had messaged us first. One had sent us his own newsletter. One had been contacted through a different product years earlier. The instrument had never once been tested on the thing it claimed to measure. The run that found the messages wrote the rule into its own log: the fastest way to get a confident wrong answer is a negative result from an instrument whose coverage has never been demonstrated on a known-present member of the population under test. In plain terms, a search that finds nothing proves nothing until it has first found something you already know is there.

A national institute's monitoring watched the wrong layer for three days. Our audit searched the wrong inbox for two and a half hours. Same failure, and I find that more reassuring than alarming: it means the failure is structural, learnable, and cheap to test for, at any scale.

I used to think writing it down was the point

For years I believed the discipline of writing a control down was most of its value: the writing forces the thinking, so the document is the achievement. I no longer believe that, and I can date the reversal to this week.

A control has three states: running, dead, or never born. On paper the three are identical; the document describing a monitoring that watches the wrong layer reads exactly like the one describing a monitoring that works. And paper does something worse than nothing: a missing control leaves you worried, a written one leaves you calm. The worry keeps you looking. The calm is the damage. A control that exists only as a document is not a dormant safeguard. It is the moment you stopped searching, given a filename.

I think this is also a piece of the famous numbers. When MIT's researchers reported last year that 95 per cent of enterprise AI pilots showed no measurable P&L impact, the figure was read as a verdict on the technology. Some unknowable share of it is this instead: controls that existed on paper, watched by nobody, in pilots everyone believed were supervised.

What I have not resolved is cadence. The institute's error was not deciding badly; it was deciding once. I do not know the right interval for re-deciding a default when the capability under it moves this fast. Quarterly feels bureaucratic, annual is demonstrably too slow. If someone has a working answer, I want to read it.

The dead-control check

For every control your operation leans on: an alarm, a review step, an approval, a threshold, a monitoring. Five questions. Steal them.

1. Where does it run? A schedule, a script, or a named person. A document is not an answer.
2. When did it last run, and when did it last fire? Two dates. "Never" is an answer. Write it down.
3. Who read its last output? A name, not a role. Output nobody reads is a control nobody has.
4. When did it last catch a failure you planted? If you have never fed it one known-bad case, its silence proves nothing. A clean record from an untested instrument is not clean. It is untested.
5. When was it last re-decided? Every control guards a trade-off that was weighed once, on a date, in a world that has since moved. If the date has not moved with it, you are running on a leftover.

Question four is the one people skip and the one that would have caught both failures in this essay. A planted failure is cheap: one wrong row, one known-bad case, one rehearsed anomaly, fed through the real pipeline. Either the alarm rings or you have learned the most valuable fact available about your own operation.

Monday

Pick the one control your weekly plan leans on hardest. Not the most impressive one, the one whose silence you are currently interpreting as good news. Run the five questions against it, in writing, with dates and names. Then plant one failure and watch what happens.

Expect the result to be embarrassing. Ours was: the instrument we trusted most had never been tested on anything it claimed to cover. The institute's version cost three days and a public incident report. The audit is the same either way, and cheaper on a Monday morning than in a postmortem.