The approve button wears out
This week Anthropic made auto mode the default in Claude Code. The tool that turned “approve every command” into a workflow now approves most commands itself.
Their numbers explain why. Developers approved 97% of permission prompts. In a study of 1,053 developers, humans caught 13.6% of dangerous commands. Anthropic’s classifier caught 89%. And the humans got worse as they went: people blocked about 17% of bad commands early in a session and about 5% after fifty prompts. The machine held steady the whole time.
Some people read that as an indictment of developers. It is not. It is a measurement of a design.
People are calibrated, not careless
While in graduate school at the Institute of Design in Chicago, we read a 1995 paper in which researchers tested alarms of varying reliability. When an alarm was right 25% of the time, people responded to it about 25% of the time. Right 75% of the time, about 75%. The finding is called probability matching, and it should unsettle anyone building a review tool. Bliss, Gilson, and Deaton, 1995
It means ignoring alerts is not a discipline problem. People calibrate their attention to what alerts are worth. If most of your prompts are noise, users will treat your prompts as noise, and no amount of training fixes it. The complacency research is blunt on this point: experts do it too, and practice does not help. Parasuraman and Manzey, 2010
Hospitals learned this the hard way. Between 72% and 99% of clinical alarms are false or non-actionable. Clinicians tune them out, sometimes with fatal consequences. Refinery operators learned it at Milford Haven, facing an alarm every two to three seconds while the plant failed. Every industry that asks humans to approve a firehose gets the same curve. Clinical alarm-fatigue review · Milford Haven investigation
So does every app. Windows Vista asked users to confirm everything and taught people to click Allow without reading. The button did not fail. It wore out. Motiee et al., 2010
What actually works
The industries that beat this did not ask humans to try harder. They rationed. Alarm standards for refineries target roughly one alarm per operator per ten minutes, then force the plant to fit the budget. Airbus builds cockpits that are dark by default: nothing lights up until something deviates. Google kills any static-analysis check that developers ignore more than 10% of the time, on the theory that a check nobody acts on is a false positive no matter how correct it is. EEMUA 191 · Airbus’s dark-cockpit philosophy · Static analysis at Google
The pattern under all of it: attention is a budget. Spend it only where a human judgment beats a machine one.
That is what Anthropic actually did. They did not remove the human. They moved the human from per-prompt gate to supervisor of a measured filter, and reserved interruptions for commands its classifier considers risky or uncertain.
Why I care
I build Stet, a document editor where every change, human or AI, arrives as a proposal you review. Which means I am building the thing this research warns about. A big agent session can produce a hundred proposals. Present those as a hundred equal decisions and I have built the ICU monitor. I learned that by using Stet myself for a few months.
In the latest Stet release, I took one small step toward fixing it. Stet now grades every incoming change as significant, normal, or trivial using deterministic rules that run locally on your Mac. Significant changes lead the summary, so you are more likely to review the consequential edits while your attention is still fresh.
It is not perfect, and there is much more to build. But it makes attention part of the design instead of pretending the approve button never wears out.