For a while I used the phrase "open-source model" in meetings the way people say "Kleenex" for any tissue. Llama, Mistral, whatever had downloadable files. Then I sat down to write a short note about model choice for a founder, tried to be exact about what she would be allowed to do with the model, and realised I could not be. I had been using one phrase for at least three different things, and the differences were not cosmetic. They decide what a regulator will ask of the developer, what a researcher can inspect, and what happens when somebody with bad intentions gets hold of the file.
This post is my attempt to untangle that, and then to say what I would actually do with a release decision if it landed on my desk. I am an outsider to the labs that make these calls, so read the second half as a way of thinking, not as insider knowledge.
Two words doing different jobs
The Open Source Initiative published its Open Source AI Definition 1.0 in October 2024. It borrows the logic of software freedoms: a system counts only if you can use it for any purpose, study it, modify it and share it. To make those freedoms real, it asks for three things in the form you would need to modify the system. Sufficiently detailed information about the training data, the complete code used to train and run it, and the parameters, meaning the weights, under approved terms.
Most of what gets called open source in the AI press does not meet that bar, because the weights are public and the training data and training code are not. That is what "open-weight" describes. The paper I rely on most for this topic, the 2024 position paper by Kapoor, Bommasani and 23 co-authors, defines open foundation models simply as models with broadly available weights. That definition is useful precisely because it is narrow. It lets you talk about a property that matters for safety, which is that anyone can download the thing and run it on their own hardware.
The distinction has legal weight. Article 53(2) of the EU AI Act exempts providers of certain general-purpose models from some documentation obligations if the model is released under a free and open-source licence and its parameters, architecture information and usage information are public. Two things stay in place even for those providers: a copyright compliance policy and a published summary of training content. And the exemption does not apply at all to models with systemic risk. So a label that sounds like a marketing choice can change which duties a developer carries. I would not want to be the person who got that wrong because I was loose with a word.
What openness buys
Kapoor and colleagues list five benefits: it spreads out who gets to decide what acceptable model behaviour is, it supports innovation, it accelerates science, it enables transparency, and it reduces monoculture and market concentration. They are also careful about limits. Weights alone do not allow every kind of scientific work, since for many questions you need the training data too.
The benefit I care about most is local language. Atlas-Chat is a good example to look at. A team that includes researchers at MBZUAI released models of 2B, 9B and 27B parameters adapted for Moroccan Arabic, known as Darija, along with DarijaMMLU, an evaluation suite they built for it. Their abstract reports that the 9B model improves on a 13B model by 13% on that suite and that the family beats Llama, Jais and AceGPT, and it says all of their resources are publicly accessible. I want to be careful here. I am reading an abstract, I do not know the licensing terms of every component, and one dialect is an anecdote, not evidence for a general rule. But it shows the shape of the argument. A language variety with modest commercial pull gets served when researchers can build on shared infrastructure, and a blanket restriction on weights would fall hardest on exactly that kind of work.
What openness costs
The same paper names the properties that make open weights hard to govern: broader access, greater customizability, local inference, the inability to rescind access, and weak monitoring. Once a file is out, no one can recall it, and no one can see how it is being used.
Customizability is where the safety problem gets sharp. Qi and colleagues showed in 2023 that fine-tuning GPT-3.5 Turbo on just 10 adversarially designed examples jailbroke its guardrails, at a cost under $0.20 through the fine-tuning API. They also found that fine-tuning on ordinary, benign data can weaken safety alignment, to a lesser degree. Read that alongside the open-weight point and the implication is plain: refusal behaviour is something a downstream user can often remove, and nobody is standing between that user and the weights.
So what are the options? Two lines of research try to make safeguards survive tampering. Tamirisa and colleagues proposed TAR, a method meant to keep safeguards in place after hundreds of fine-tuning steps; their default attack was 500 steps, and they report robustness to 1,000. They are open about the cost. Retain accuracy drops by about 10.6% on average, the robustness varies under some other attack types, and they say tamper-resistance can eventually be circumvented and should supplement other efforts. A 2025 paper from EleutherAI, the UK AI Security Institute and Oxford, "Deep Ignorance," takes a different route: filter dual-use biology text out of the pretraining data so the model never learns it. Their filtered models resisted adversarial fine-tuning for up to 10,000 steps and about 300 million tokens, with no observed loss on unrelated capabilities. They also report what it cannot do. A model that never learned the material can still use it if someone supplies it in the prompt, for example through search tools, and none of the defences they tested resisted an attack combining fine-tuning with that kind of retrieval. Their evaluation is multiple-choice, which they note is not a direct measure of real-world biorisk, and they only tested biology.
I find that honesty more persuasive than a clean result would be. It tells me that "downstream safeguards" are partial by construction.
Ask about marginal risk, not about risk
This is where I think the debate gets better. Kapoor and colleagues propose asking how much additional risk a release adds compared to what is already available, through closed models or through ordinary tools like web search. Their framework has six steps: identify the threat, assess existing risk without open models, assess existing defences, look for evidence of marginal risk, ask how easy it is to defend against new risks, and write down your uncertainty and assumptions. They find marginal risk low in some cases, such as automated vulnerability detection in cybersecurity, and high in others, such as non-consensual intimate imagery, where defences are hard to put in place. They also say that for several misuse vectors the research is simply not good enough yet to say.
Policy documents landed in a similar place. In July 2024 the US NTIA concluded that current evidence was not sufficient to show that restricting open-weight dual-use models is warranted, or that restrictions will never be warranted. Its recommendation was to monitor a portfolio of risks and prepare to act: collect evidence, evaluate it, and be ready to restrict access if the evidence calls for it. The International AI Safety Report, published in January 2025 under Yoshua Bengio's chairmanship, describes a broad emerging consensus that openness should be assessed by marginal risk, and it is plain about the downside, that once weights are public there is no rolling back all copies or making sure each one gets safety updates.
There is also a developer's own attempt to put numbers on the worst case. In August 2025, OpenAI researchers published a study in which they fine-tuned gpt-oss to be as capable as possible in biology and cybersecurity, using reinforcement learning on threat-creation tasks and capture-the-flag challenges, to simulate a malicious actor. Their conclusion was that the fine-tuned model underperformed OpenAI o3, which is itself below the company's "High" capability level in both areas, and that gpt-oss may modestly increase biological capabilities among open-weight models without pushing the state of the art. That is useful, and it is also the developer grading its own homework. I would want someone else to replicate it.
How I would make the release decision
If I had to design a decision rule, I would not start from "open or closed." I would start from a short list of questions that need an answer before anything ships, and I would write the answers down before the model exists, not after.
First, name the specific threat and the baseline. Not "misuse" but one concrete capability, and what a determined person can already do without this model. Second, run a worst-case test on the weights themselves, adversarial fine-tuning included, and have it done or at least reproduced by someone with no stake in the release. Third, where the threat is tied to a domain, consider filtering at the data stage, because the evidence so far suggests it holds up better than patching behaviour afterwards, with the in-context caveat above. Fourth, decide in advance what each outcome triggers. If the worst-case model stays within the margin of what closed models already provide, release with documentation. If it sits near the line, stage the release, perhaps with the largest variant held back. If it clearly exceeds the baseline, withhold the weights and consider structured API access. Fifth, plan monitoring that does not depend on recalling the file: track derivative models and reported misuse so the next decision has evidence.
The hard part, and I will say it plainly, is the thresholds. I do not have defensible numbers for "within the margin," and I would distrust anyone who handed me one without showing how they derived it. What the rule does is force the argument to happen early, in the open, with a stated reason for each outcome.
Where I would push back on my own framing
Three things bother me. Marginal risk compares against what exists today, and what exists keeps moving, so a release that adds little now can look different once several such releases stack up. The framework is also better at discrete threats than at diffuse harms that build up slowly. And a rule that trusts the developer to run the worst-case test is only as good as the developer's incentives, which is why independent replication keeps coming up as the missing piece.
I also notice that I have been comfortable treating "local-language access" as a clear benefit, when the same fine-tuning that adapts a model to a dialect is the kind that Qi and colleagues found can wear down safety behaviour. I do not know what that means for any particular regional model. It is a reason to test those models for safety in their own language, which is a question I will come back to.
If you work on open-weight release decisions, or have replicated worst-case fine-tuning evaluations, I'd like to hear where my framing is wrong. Get in touch →
References
Bengio, Y., et al. (2025). International AI Safety Report. arXiv:2501.17805.
European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 53. Official Journal of the European Union.
Kapoor, S., Bommasani, R., Klyman, K., et al. (2024). On the societal impact of open foundation models. Position paper, ICML 2024. arXiv:2403.07918.
National Telecommunications and Information Administration. (2024, July 30). Dual-use foundation models with widely available model weights. U.S. Department of Commerce.
O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., et al. (2025). Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight LLMs. arXiv:2508.06601.
Open Source Initiative. (2024). The Open Source AI Definition 1.0.
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., & Henderson, P. (2023). Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv:2310.03693.
Shang, G., Abdine, H., Khoubrane, Y., et al. (2024). Atlas-Chat: Adapting large language models for low-resource Moroccan Arabic dialect. arXiv:2409.17912.
Tamirisa, R., Bharathi, B., Phan, L., et al. (2025). Tamper-resistant safeguards for open-weight LLMs. ICLR 2025. arXiv:2408.00761.
Wallace, E., Watkins, O., Wang, M., Chen, K., & Koch, C. (2025). Estimating worst-case frontier risks of open-weight LLMs. arXiv:2508.03153.