The stories we hear about flying are about crashes. The reason flying became safe is mostly the stories we do not hear, the near-misses that pilots, controllers and mechanics reported before anything happened, and that someone read, grouped and acted on. I find this one of the most underrated ideas in safety. A system that learns only from disasters is a system that has agreed to pay full price for every lesson.
AI does not have anything like that yet. What it has is closer to a clipping service. In this post I want to look at what a near-miss system for AI would need, what exists, and how I would test a first design. I will use some chatbot incidents I have written about before as test data, but the argument is about system design, not about those companies.
What the aviation system actually does
The Aviation Safety Reporting System began in 1976. The FAA funds it and NASA administers it, which already tells you something: the body that collects the reports is not the body that enforces the rules. It is voluntary and confidential. Pilots, controllers, flight attendants, maintenance technicians and others can report a situation where they think safety was compromised. Identifying information is stripped from each report before it enters the database. The FAA offers limited immunity from enforcement action as an incentive to report, provided the report is filed within 10 days, and the FAA cannot seek, and NASA cannot release to it, information that would identify the reporter.
The FAA's own description is candid about the limits. Reports are generally not investigated, so accuracy is not verified, and because not every event gets reported, the data is better for identifying hazards than for measuring trends or distributions.
Three design features are worth copying. A neutral third party holds the data. The reporter is protected. And the system is built to find hazards, not to count incidents.
What AI has instead
The OECD proposed definitions in May 2024 that draw a line aviation people would recognise. An AI incident is an event in which the development, use or malfunction of an AI system directly or indirectly leads to harm, such as injury to health, disruption of critical infrastructure, violations of rights, or harm to property, communities or the environment. An AI hazard is an event that could plausibly lead to an AI incident. The hazard category is the near-miss space.
The best-known database is the AI Incident Database, which Sean McGregor described at AAAI 2021 with an explicit aim: to prevent repeated real-world failures by cataloguing incidents so that developers can learn from earlier ones. The Stanford AI Index 2025 reports that 233 AI-related incidents were recorded in 2024, a record, up 56.4% from 2023.
I would be careful with that number. These are publicly reported incidents. A rise could reflect more failures, more deployment, more press attention or better collection, and the database cannot tell those apart. It is a count of what became public, not a measure of what happened.
What the law now asks for
Regulation has started to add duties. Under Article 73 of the EU AI Act, providers of high-risk AI systems must report serious incidents to the market surveillance authorities of the Member State where it occurred. The general deadline is immediately once a causal link or reasonable likelihood is established and no later than 15 days after becoming aware; for a widespread infringement it is two days, and for a death, 10 days. Deployers have reporting duties in some cases too.
California's SB 53, signed in September 2025, requires frontier developers to report critical safety incidents to the state's Office of Emergency Services within 15 days, and within 24 hours if there is an imminent risk of death or serious injury. One law firm summary describes a critical safety incident as one involving a frontier model causing harm or loss of control. The law also protects whistleblowers.
My reading is that both regimes concentrate on the developer or provider and on serious events. That is sensible for catastrophic risk. It leaves a gap in the middle: organisations that deploy models, whose systems misbehave in ordinary, embarrassing, sometimes costly ways that fall below "serious," and which have no channel for reporting a near-miss and no protection for doing so. That is a reading of the texts as summarised, not a legal opinion, and anyone relying on it should check the statutes.
Our current near-miss system is social media
Three chatbot cases I have written about elsewhere show the pattern. In 2024, a British Columbia tribunal held Air Canada responsible for wrong information its website chatbot gave a customer about bereavement fares, in a decision known as Moffatt v. Air Canada, and rejected the airline's attempt to treat the chatbot as something separate from the company. In January 2024, a customer posted screenshots of DPD's delivery chatbot swearing and criticising the company, and DPD disabled the AI element. In December 2023, a user got a car dealership's chatbot to agree to sell a vehicle for a dollar.
What do these have in common? Each reached the company from outside. A customer complained, or someone posted a screenshot. I do not know when any of them first happened internally, because that information was not made public. That absence is the point. If the first time an organisation learns that its system can be steered into nonsense is when a stranger posts it, the organisation has no sensing mechanism, and the public record contains almost nothing about how long the problem existed before anyone noticed.
A first design for a deployer near-miss report
Here is the minimum I would ask for. Each report would capture:
- The system and context: what the AI was doing, for whom, with what human oversight.
- What happened, in plain language.
- Whether harm occurred, was avoided, or was only possible. Hazard versus incident, in the OECD sense.
- How it was detected: customer, employee, automated monitor, outside party.
- Time from first occurrence to detection, if known, and to containment.
- What was done to contain it.
- Contributing factors, chosen from a short list: specification, data, model update, vendor change, missing oversight, user manipulation.
- Whether it has happened before in this system or in a similar one.
- Whether the reporter wants the report shared, and in what form.
The nine fields are meant to be answerable by a small organisation in ten minutes. If a form takes an hour, nobody will fill it in.
How I would test it
A taxonomy is a hypothesis, so I would test it. Take ten public incidents and try to fill in every field from public information. The fields I can complete are the ones public reporting already supports. The fields I cannot complete are the interesting result, because they show what a reporting system would have to collect that the press does not.
I did a rough version of this with the three cases above. Fields 1, 2 and 3 can be filled from the public record in each case. Field 6 can be filled only for DPD, which disabled the AI element. Field 4 is answered the same way each time: customers or the public. Fields 5, 7 and 8 mostly cannot be filled at all. We do not know how long the Air Canada chatbot had been giving the wrong answer, what changed in DPD's system before the behaviour appeared, or whether either had seen similar events earlier. That is a sample of three, chosen because I already knew them, so it demonstrates the method and proves nothing. A proper test would use ten or more incidents selected by a stated rule, with two people coding them independently and agreement reported.
The part that is hard
The reason aviation's system works is not only the form. Pilots have an employer-independent regulator that offers immunity and a culture, built over decades, in which reporting is routine. AI deployers have neither. If a company reports a near-miss, what protection does it get? A report that says "our chatbot agreed to a $1 sale" could be read by a plaintiff's lawyer. I do not know how to solve that, and I suspect the answer involves legislation that grants some protection to good-faith hazard reports, as the FAA does, plus a neutral holder of the data.
There are other problems. Small organisations carry a larger relative burden. A voluntary system suffers from selection bias, as the FAA itself says about trend measurement. And a taxonomy that is too rigid can miss new kinds of failure, so it needs a free-text field and a review cycle.
The link to my own work
In my research on brand governance I call the gap between what an organisation says it stands for and what its AI systems actually produce Brand Drift. Drift is a hazard in the OECD's sense long before it is an incident. The output has already diverged, nobody has been harmed yet, and the only question is whether anyone is looking. A near-miss system for deployers would capture exactly that period.
If you work on AI incident reporting, I'd like to know which of my nine fields wouldn't survive contact with a real reporting system. Get in touch →
References
Clifford Chance. (2025, October). SB 53: California sets standards for AI transparency.
European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 73. Official Journal of the European Union.
Federal Aviation Administration. (n.d.). Aviation Safety Reporting System (ASRS). Aviation Safety Information Analysis and Sharing (ASIAS).
McGregor, S. (2021). Preventing repeated real world AI failures by cataloging incidents: The AI Incident Database. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17). arXiv:2011.08512.
Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia).
OECD. (2024, May). Defining AI incidents and related terms. OECD.AI Policy Observatory.
Stanford Institute for Human-Centered AI. (2025). The AI Index 2025 Annual Report, Chapter 3: Responsible AI.
The Guardian. (2024, January). DPD AI chatbot swears, calls itself "useless" and criticises delivery firm.