I proofread my own writing badly. I know this because I have sent emails with a typo in the subject line after reading them three times. The words are so familiar that my eyes supply what I meant instead of what is on the screen. A colleague with no stake in the sentence sees the error in two seconds.
That small, slightly embarrassing fact is the reason I do not trust the phrase "a human will review it." Review is a skill with failure rates, and the failure rates depend on conditions that can be designed. Most policy language treats a person in the loop as a safeguard by definition. I want to look at what the evidence says about that, then show some arithmetic about what real oversight costs, and finish with what I think a serious requirement would contain.
What the law asks for
Article 14 of the EU AI Act is one of the more specific attempts to describe oversight. High-risk systems must be designed so that natural persons can effectively oversee them while in use, with appropriate human-machine interface tools. The people overseeing must be able to understand the system's capacities and limits and to monitor its operation, to remain aware of automation bias, which the text defines as the tendency to automatically rely or over-rely on the output, to interpret the output correctly, to decide not to use the system or to override or reverse its output, and to intervene or stop it, for example with a stop button.
That is a thoughtful list. Notice what it leaves out: nothing in it says how well the overseers must perform, how many decisions they can review, or how that would be measured. It describes capabilities the person should have, not results that should be demonstrated.
What the evidence says about people
Ben Green surveyed 41 policies that require human oversight of government algorithms and found two flaws. People cannot reliably do the job the policies expect of them, and because of that, the policies end up approving the use of faulty or controversial algorithms without fixing the underlying problems, which gives a false sense of security and lets vendors and agencies avoid accountability. His alternative is institutional: agencies should justify why an algorithm belongs in a decision, show with empirical evidence that any proposed oversight works, and put those justifications through public review before adoption.
The Center for Security and Emerging Technology reached a related conclusion in November 2024. Kahn, Probasco and Kinoshita studied automation bias through Tesla Autopilot incidents, aviation accidents involving Boeing and Airbus aircraft, and Army and Navy air defence incidents. They sort the contributing factors into three levels: the user, the technical design, and the organisation. Their conclusion is that human-in-the-loop oversight cannot prevent all accidents or errors, and that the better path is to calibrate technical and human fail-safes together, supported by training, design and process.
The older human factors literature says the same thing. Parasuraman and Manzey's 2010 review in Human Factors brought together research on complacency and bias in human use of automation. Lisanne Bainbridge's 1983 paper "Ironies of Automation" made an uncomfortable point that I find easy to remember: when you automate most of a task, you leave people with the job of monitoring a system that usually works, which is something humans are bad at, and you let the skills they would need in a failure fade from disuse.
None of this says oversight is useless. It says that oversight is a performance, and performance depends on how it is set up.
The technical cousin: oversight as a scarce budget
AI control research approaches the same problem from the other direction. Greenblatt, Shlegeris, Sachan and Roger studied a setting with three resources: a powerful but untrusted model (GPT-4), a weaker trusted model (GPT-3.5), and a limited amount of trusted human labour. The task was to solve a sequence of programming problems without ever submitting a solution containing a backdoor, meaning a logic error the test cases would not catch. They tested protocols such as trusted editing, where the weaker model rates how suspicious the stronger model's code looks and rewrites the suspicious parts, and untrusted monitoring, where a second instance of the strong model judges the first, with measures to stop the two colluding. Their abstract says the protocols improve substantially on simple baselines.
What I take from it is a change of mindset. Human review is not a free layer you add at the end. It is a budget, and the design question is where to spend it. A protocol that sends the most suspicious outputs to the human is using the budget better than one that checks everything lightly or nothing at all.
The arithmetic of catching a problem
Here is a calculation I find clarifying. Suppose a system produces outputs, a fraction p of which are bad in a way that matters. A reviewer samples outputs at random, and when they see a bad one, they catch it with probability d. To be 95% confident of catching at least one problem if the failure rate is p, you need n samples where n is the natural log of 0.05 divided by the natural log of one minus p times d.
I ran the numbers. Rows are the true failure rate. Columns are how often the reviewer notices a bad output when they see one.
| Failure rate | Perfect reviewer | Catches 70% | Catches 50% | Catches 30% |
|---|---|---|---|---|
| 5% | 59 | 85 | 119 | 199 |
| 2% | 149 | 213 | 299 | 498 |
| 1% | 299 | 427 | 598 | 998 |
| 0.5% | 598 | 855 | 1,197 | 1,996 |
The numbers are the number of randomly sampled outputs needed to have a 95% chance of seeing at least one failure. A reviewer who catches half of what they see needs about twice the sample of a perfect one, and one who catches 30% needs more than three times. For a failure that happens in one output in a hundred, a team with a third-rate catch rate would need about a thousand reviewed outputs just to have a good chance of noticing it once.
Three things follow. The cost of oversight depends heavily on the catch rate, so catch rate is worth measuring. The rare failures, which are often the serious ones, are the most expensive to find. And "someone checks a few" is not a number. If a team reviews 20 outputs a week and the true failure rate is 1%, they will most weeks see nothing at all and conclude that things are fine.
How to find out what your catch rate is
A reviewer's catch rate can be estimated, and the method is simple. Seed the review queue with known bad outputs, without telling the reviewer which ones, and record how many they catch. Quality assurance in many fields works this way, as I understand it, and the principle is plain: you cannot manage what you have not measured, and a reviewer's self-assessment is the least reliable measure available.
That leads to several design choices I would want in any serious oversight arrangement:
- A stated sampling plan that gives the sample size, how items are chosen, and the failure rate it is meant to detect.
- A measured catch rate from seeded tests, reported with its uncertainty.
- A time budget per item that is realistic, since a reviewer given ten seconds per output is performing a different task from one given two minutes.
- Real authority to stop or override, with no penalty for doing so, because the Article 14 power to reverse a decision means little if exercising it gets you into trouble.
- Rotation and fresh eyes, to fight the familiarity problem I started with.
- Stratification, so that the sample covers different contexts and not just the most common ones.
What this has to do with my research
My work on brand governance uses an idea I call Intelligent Human Oversight: someone who is not the person operating the tool should periodically sample its output against a written standard. The arithmetic above tells me how to make that concrete. If a brand's AI-generated customer messages drift off its stated values in one out of fifty, a reviewer checking ten a month will mostly see nothing, and the organisation will read the silence as safety. The same maths applies to a hospital algorithm and to a chatbot.
Where the table is too tidy
The calculation assumes failures are independent and spread evenly, and real failures cluster. They tend to show up in particular contexts, such as a certain type of customer, a certain time of day, or after a model update, so a random sample can miss a concentrated problem. The table should be read as a floor on how much review is needed, not an estimate of what is sufficient. It also treats a failure as something visible on inspection, when some failures can only be seen by comparing the output against information the reviewer does not have in front of them. And seeding known failures only measures the reviewer's ability to catch failures that look like the seeded ones.
I would also be cautious about pushing Green's argument too far. He is sceptical of oversight requirements as a regulatory tool, and he argues for empirical proof that oversight works. That argument can be read as support for removing human review. I read it the other way: it is a case for making oversight measurable so that it earns the trust it is currently given for free.
What I would ask regulators and organisations to say
Instead of "a human reviews the output," the claim should read more like this: "A reviewer independent of the operating team samples N outputs per period, chosen by this method, to detect failures at rate p with 95% confidence. Their measured catch rate on seeded failures is d. They have the authority to halt the system and have done so on these occasions." That is a sentence one can check. It also reveals the cost, which is useful to everyone involved, because an honest oversight plan is more expensive than a job title.
If you've measured how well human reviewers catch AI errors, I'd like to compare notes and hear where my arithmetic is too tidy. Get in touch →
References
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775-779.
European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14. Official Journal of the European Union.
Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681. arXiv:2109.05067.
Greenblatt, R., Shlegeris, B., Sachan, K., & Roger, F. (2024). AI control: Improving safety despite intentional subversion. arXiv:2312.06942.
Kahn, L., Probasco, E., & Kinoshita, R. (2024, November). AI safety and automation bias: The downside of human-in-the-loop. Center for Security and Emerging Technology, Georgetown University.
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381-410.