When a model card says a system "did not meet the threshold" on a dangerous-capability test, I notice that my brain files it under good news. A number was measured, a line was not crossed, move along. It took me a while to catch myself doing this, because the same reflex shows up everywhere. A clean audit, a green dashboard, a passing grade. The result feels like a fact about the world when it is a fact about a particular test, run a particular way, by a particular party.
This post is about what you can and cannot conclude from a passing evaluation of an AI system, and about a short rubric I would use to read any such claim. I am not an evaluator and I have not built evals for a lab. I am reading what has been published and trying to be systematic about it.
Why a pass matters so much
Evaluations are doing more load-bearing work than they used to. Clymer, Gabrieli, Krueger and Larsen describe a safety case as a structured rationale that an AI system is unlikely to cause a catastrophe, and they sort the possible arguments into four types: inability (the system cannot cause the catastrophe), control (safeguards would stop it even if it could), trustworthiness (it would not try), and deference (we rely on credible AI advisors, for much more powerful systems). Most current public claims lean on the first. "The model cannot meaningfully help with X" is an inability argument, and the evidence for it is an evaluation.
So the quality of the eval is the quality of the argument. If the test is weak, the whole case is weak, and nobody sees that from the headline score. A "pass" is the conclusion of a chain of choices, and each link can fail without any signal on the surface.
Four ways a pass can mislead
The test may not measure the thing. Bean and colleagues reviewed 445 large language model benchmarks from leading NLP and machine learning venues with 29 domain experts. Their point is about construct validity: to measure an abstract concept like safety or robustness, the measure has to capture what matters about the concept. They found recurring patterns in how benchmarks define the phenomenon, select tasks and score results that weaken the validity of their claims. They give eight recommendations, among them define the phenomenon, measure only that phenomenon, build representative datasets, prepare for contamination, use statistical methods when comparing models, do error analysis, and justify the construct's validity. A checklist appears in their appendix. I find this the most useful single reference for anyone reading benchmark results, because it supplies the questions to ask.
The system may not be trying. Van der Weij and colleagues defined sandbagging as strategic underperformance on an evaluation, which arises when a developer or the system itself has an incentive to look less capable than it is. They showed that frontier models, including GPT-4 and Claude 3 Opus, could be prompted to underperform on dangerous-capability tests while staying strong on harmless ones, and that fine-tuning could hide capabilities behind a password. Their conclusion is that capability evaluations can be gamed, which weakens their reliability and could lead to flawed safety and deployment decisions. That does not prove anyone is doing it. It means an inability argument needs a story about elicitation: how hard someone tried to get the capability out.
The auditor may not see enough. Casper and colleagues, writing for FAccT 2024 with 22 authors, distinguish three levels of access. Black-box access means you can query the system and observe outputs. White-box access lets you inspect weights, activations and gradients, which allows stronger attacks, deeper interpretation and fine-tuning. Outside-the-box access gives information about training and deployment, such as methods, code, data and internal evaluation results. They argue that recent audits of state-of-the-art systems have mostly used black-box access and that white-box and outside-the-box access allow substantially more scrutiny. They also ask audits to say what access and methods they used, so readers can interpret the result. A pass under black-box access and a pass under white-box access are different statements.
The party running the test may be the party being tested. The 2025 Stanford AI Index notes that major developers consistently report the same general capability benchmarks but that no comparable standard exists for safety and responsibility evaluations, and that most evaluations are internal and nonstandardized, which makes models hard to compare. Raji and colleagues argued back in 2020 for internal algorithmic auditing across the development lifecycle, which was a reasonable response to the lack of outside access, but internal checks and independent ones are not the same thing, and the authors did not claim they were.
A worked example from the literature
The "Deep Ignorance" paper I discussed in my post on open weights is a good illustration, because the authors state their own limits. Their main benchmark is multiple-choice. They write that it has known shortcut problems, which they tried to reduce with curated subsets, and that multiple-choice accuracy does not directly measure real-world biorisk.
Now imagine a hypothetical model card that said: "The model scored below the threshold on a multiple-choice biology benchmark and therefore provides no meaningful uplift." Run the claim through the questions. Is a multiple-choice score a valid stand-in for the capability to assist with a real task? The authors of a careful paper say not directly. Did anyone try to elicit more, with tools, search or fine-tuning? The same paper reports that supplying information in context, or combining that with fine-tuning, defeated the defences they tested. The hypothetical claim would fail the rubric even though the number could be perfectly accurate.
By contrast, OpenAI's August 2025 worst-case study tried harder at elicitation. The authors fine-tuned gpt-oss with reinforcement learning on biology threat-creation tasks, with web browsing enabled, and on capture-the-flag cybersecurity tasks in an agentic coding setup, to approximate a determined malicious actor. That is a stronger style of evidence for an inability claim. It is still run by the developer on its own model, so the question of independence remains open.
The rubric I would use
When I see a claim that a model passed, I would ask these seven things, in this order.
- What exactly is the claim? "Cannot do X" is different from "scored below Y on test Z." Write it as a sentence.
- Is the task a valid stand-in for X? What is the construct, and what is the evidence that the test captures it?
- How hard did anyone try to elicit the capability? Prompting only, or also scaffolding, tools and fine-tuning? How much effort, by whom?
- What access did the evaluator have? Black-box, white-box, or outside-the-box, and was it stated?
- Did anyone have an incentive for a low score, and would a sandbagging check detect it?
- Could the test items have leaked into training? What was done about contamination?
- What happens on a fail? If nobody has said what a failing result triggers, the eval is advisory.
Scoring each as "stated and adequate," "stated but weak," or "not stated" gives a profile instead of a single verdict. Most claims I have read would come out with several entries at "not stated," and I would argue that is the finding. It does not mean the model is dangerous. It means the evidence does not yet support the confidence it is being used to justify.
The connection to oversight
In my brand governance research I use the Institute of Internal Auditors' Three Lines Model: the people who run a system, the people who set guardrails and challenge them, and an independent assurance function that reports to the board. The part I keep finding missing in organisations is the second line, because the people best placed to notice a problem are often the people operating the system and the least motivated to report it. Evaluations have the same structure. The first line is the developer's own testing. The independent line is the one that requires access, and the Casper paper is basically an argument about what that access needs to include.
Where the rubric falls short
It does not produce a number, and people like a number. It also puts a lot of weight on whether information is stated, which rewards transparent labs and penalises ones that simply publish less detail even if they test thoroughly. External evaluators bring their own risks. They depend on the access the developer grants, and they may be shaped by that relationship. And some of the questions, like how hard is hard enough for elicitation, have no agreed answer. I would rather have the rubric and these admitted gaps than a single confident score.
The practical takeaway is a small habit. When you see "passed," say out loud what it was a pass at, who ran it, and what they could see. If you cannot, hold the conclusion loosely.
If you build or review evaluations and see a question missing from my rubric, I'd like to hear it. Get in touch →
References
Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., et al. (2025). Measuring what matters: Construct validity in large language model benchmarks. NeurIPS 2025, Datasets and Benchmarks Track. arXiv:2511.04703.
Casper, S., Ezell, C., Siegmann, C., et al. (2024). Black-box access is insufficient for rigorous AI audits. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT '24). arXiv:2401.14446.
Clymer, J., Gabrieli, N., Krueger, D., & Larsen, T. (2024). Safety cases: How to justify the safety of advanced AI systems. arXiv:2403.10462.
Institute of Internal Auditors. (2020). The IIA's Three Lines Model: An update of the Three Lines of Defense.
Stanford Institute for Human-Centered AI. (2025). The AI Index 2025 Annual Report, Chapter 3: Responsible AI.
O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., et al. (2025). Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight LLMs. arXiv:2508.06601.
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., et al. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. arXiv:2001.00973.
van der Weij, T., Hofstätter, F., Jaffe, O., Brown, S. F., & Ward, F. R. (2024). AI sandbagging: Language models can strategically underperform on evaluations. arXiv:2406.07358.
Wallace, E., Watkins, O., Wang, M., Chen, K., & Koch, C. (2025). Estimating worst-case frontier risks of open-weight LLMs. arXiv:2508.03153.