Source-linked AI summary
Open Technical Problems in Open-Weight AI Model Risk Management
Stephen Casper, Kyle O'Brien, Shayne Longpre, Elizabeth Seger, Kevin Klyman, Rishi Bommasani, Aniruddha Nrusimha, Ilia Shumailov, Sören Mindermann, Steven Basart, Frank Rudzicz, Kellin Pelrine, Avijit Ghosh, Andrew Strait, Robert Kirk, Dan Hendrycks, Peter Henderson, Zico Kolter, Geoffrey Irving, Yarin Gal, Yoshua Bengio, Dylan Hadfield-Menell
TL;DR
Open-weight models are becoming more capable and widely used, but their modifiability, decentralized use, and irreversible spread leave important safety gaps. The paper organizes 16 open technical challenges across the model lifecycle and finds that reporting on several safeguards remains sparse. It argues that open research, methods, evaluations, and reporting—not only open weights—are needed to build the field.
Problem
Open-weight models have distinctive risk-management challenges because they can be modified, used without centralized oversight, and distributed irreversibly, while safety tooling specific to them remains limited.
Method
The paper synthesizes open technical problems and evaluates how documentation from prominent open-weight developers reports five safeguard categories across the model lifecycle.
Results
Documentation is sparse for several safeguards: data curation is reported by 6 of 10 analyzed models, tamper-resistant training receives no mention, tampering evaluations appear in one model, and staged deployment and provenance/forensics are absent.
Takeaways & Limitations
Building open-weight model risk management requires technical safeguards together with openness about research, methods, evaluations, and reporting.
Takeaways & Limitations
Current tamper-resistance remains limited: second-party red-teaming finds state-of-the-art defenses withstand only around several hundred adversarial fine-tuning steps.
Abstract
from arXiv · showhide
Frontier AI models with openly available weights are steadily becoming more powerful and widely adopted. However, compared to proprietary models, open-weight models pose different opportunities and challenges for effective risk management. For example, they allow for more open research and testing. However, managing their risks is also challenging because they can be modified arbitrarily, used without oversight, and spread irreversibly. Currently, there is limited research on safety tooling specific to open-weight models. Addressing these gaps will be key to both realizing their benefits and mitigating their harms. In this paper, we present 16 open technical challenges for open-weight model safety involving training data, training algorithms, evaluations, deployment, and ecosystem monitoring. We conclude by discussing the nascent state of the field, emphasizing that openness about research, methods, and evaluations -- not just weights -- will be key to building a rigorous science of open-weight model risk management.
1 Introduction
Open-weight models are rapidly approaching frontier capabilities while retaining distinctive safety challenges because they can be modified, used, and distributed without the controls available for closed models. The paper organizes five technical safeguard areas spanning tampering resistance, evaluation, deployment, and ecosystem monitoring.
- Motivation: Open-weight models may reach critical capability thresholds soon, with frontier systems estimated to lag proprietary models by only 6–12 months.Their capabilities and prominence have grown rapidly across language, image, and video models.
- Motivation: Open-weight models support more open research and testing but cannot reliably rely on proprietary developers’ training interventions, classifiers, and usage enforcement.They can be used, tampered with, and distributed without limitations.
- Technical objectives: Training-data curation methods aim to prevent harmful capabilities and have made open-weight models resist few-shot learning of harmful behaviors.These methods address tampering risks through properties established during training.
- Technical objectives: Tamper-resistant training and unlearning algorithms seek to add defenses against malicious fine-tuning and other model modifications.The objective targets harmful tampering after model release.
- Technical objectives: Model-tampering evaluations are needed to assess real-world risks from downstream modifications to open-weight models.The evaluations focus on risks that ordinary pre-release or untampered assessments may miss.
- Technical objectives: Staged deployment enables partial-access experimentation so developers can monitor unexpected uses and revise safeguards before full release.The approach preserves an opportunity to change release plans before weights become openly available.
- Technical objectives: Model provenance and forensics strategies provide tools for studying real-world uses and impacts across the diffuse open-weight ecosystem.These tools are intended for developers, academics, and other stakeholders.
2 Why is open-weight model risk management challenging?
Open-weight models create risk-management challenges because safeguards can be disabled or removed, models cannot be centrally controlled, and releases spread through complex decentralized supply chains. These conditions make their real-world use and impact difficult to track.
- Control and tampering: Open-weight models enable broad adaptation and independent research, but users can disable external safety tools and downstream modifications can remove safeguards or add harmful capabilities.Fine-tuning, steering, editing, pruning, and related interventions create risks beyond ordinary jailbreaks.
- Irreversibility: Open-weight models can spread quickly and irreversibly, limiting developers’ ability to patch hazardous releases or withdraw them from distribution.Unlike centrally served models, released weights can continue circulating after a developer changes course.
- Decentralized control: Open-weight models generally cannot be centrally monitored or moderated through developer-controlled APIs, customer checks, misuse monitoring, and usage enforcement.This removes several controls commonly available for closed-weight deployments.
- Supply chains: Open-weight supply chains involve many stages of modification and redistribution, often across jurisdictions, making accountability and risk tracing more complex.The supply chain is less centralized and coordinated than those typically associated with closed models.
- Ecosystem visibility: Because usage is decentralized, researchers may struggle to understand models’ spread, uses, and impacts, complicating risk studies, cost-benefit analysis, and intervention design.The difficulty concerns ecosystem-level visibility rather than only individual model behavior.
3 The Toolkit
The paper defines a toolkit of technical safeguards with distinct implications for open-weight models and deliberately excludes broader technical, nontechnical, and governance strategies. Its taxonomy spans the model lifecycle from training data to post-deployment monitoring.
- Taxonomy: The taxonomy organizes safeguards into training data curation, training, evaluation, deployment, and post-deployment monitoring.These categories correspond to stages of the model lifecycle.
- Scope: The paper focuses on technical safeguards applicable to open-weight releases that cannot be trivially disabled.It defines a technical safeguard as a machine-learning-based risk-management technique.
- Scope: Out-of-scope technical tools include general safety fine-tuning, safety scaffolding and content moderation, and rigorous black-box evaluations without distinct open-weight implications.These methods are excluded from the paper’s open-weight-specific focus.
- Scope: The paper also excludes data provenance and forensics, fine-tuning API monitoring, acceptable-use enforcement, transparency and documentation, know-your-customer methods, and incident-reporting infrastructure from its focal toolkit.These exclusions include both technical methods and organizational practices.
- Scope: Strategies for accelerating beneficial open-weight development and governance approaches with distinct ecosystem implications are outside the paper’s scope.The paper’s primary goal is building the technical science of open-weight model risk management.
4 Open Technical Problems
The paper presents open technical problems across the model lifecycle, beginning with resistance to harmful fine-tuning and other tampering. It then addresses evaluation and later-stage safeguards through deployment and ecosystem monitoring.
- Lifecycle challenges: The first three challenges concern open-weight models’ resistance to harmful fine-tuning and other forms of tampering.They cover training-data curation, tamper-resistant training algorithms, and model-tampering evaluation.
- Lifecycle challenges: The paper examines training-data curation, training algorithms, and evaluation before turning to deployment and post-deployment ecosystem monitoring.This ordering follows the model lifecycle from pretraining through later-stage monitoring.
4.1 Training Data Curation
Training data curation aims to prevent models from learning harmful capabilities, but its effectiveness varies across harms and depends on implementation details, scale, and the relationship between data and emergent capabilities.
- Motivation: Training data curation is a key intervention for improving model safety off-the-shelf and under tampering.The rationale is to minimize exposure to unsafe data because models acquire much of their core knowledge during pretraining.
- Existing work: Filtering harmful content is widely used in dataset design, but blocking risky sources remains a common, nonstandardized practice.Open datasets have increasingly released their curation pipelines, while recent work studies whether curation can prevent unsafe capabilities.
- Effectiveness across harm categories: Filtering can reduce models’ knowledge of biorisk-related topics, but one implementation was ineffective on gpt-oss.These findings suggest effectiveness may depend on implementation details, while comparisons are hindered by missing information about filtering and compute.
- Scaling data curation: Internet-scale curation must balance cost, filtering errors, dataset quality, multilingual coverage, moderation bias, contextual harmfulness, efficiency, precision, and recall.A multi-stage approach used less than 1% of subsequent training compute but sacrificed efficiency for precision by filtering many benign documents.
- Open questions: The relationship between model architecture, training-data content, and emergent capabilities remains unclear, raising questions about how unsafe capabilities arise from training data.Recent work examines influence functions, out-of-context reasoning, coreset analysis, and domain-aware scaling.
4.2 Tamper-Resistant Training and Unlearning Algorithms
Tamper-resistant training and unlearning seek to preserve safety when downstream users modify open-weight models, but current defenses remain vulnerable and belief-editing approaches have substantial unresolved side effects.
- Motivation: Post-training safeguards against downstream tampering are a core strategy for mitigating risks from malicious or negligent use of open-weight models.This work complements training-data interventions by targeting harmful modifications after training.
- Tamper-resistant unlearning: Current safety fine-tuning and machine-unlearning methods can be undone within dozens of adversarial fine-tuning steps.Second-party red-teaming places state-of-the-art tamper resistance at only around several hundred adversarial fine-tuning steps, despite some defenses reporting thousands or tens of thousands.
- Tamper-resistant unlearning: Prior tamper-resistant unlearning approaches include pruning, meta-learning, specialized objectives, and training under tampering.The persistent struggles of these methods motivate reassessing current approaches.
- Belief revision: Belief editing may complement refusal training and unlearning, but current approaches face weak generalization, limited scalability, intervention interference, and ripple effects.The paper identifies benchmarks for surgical knowledge revision, improved scalability, reduced side effects, and realistic risk-mitigation use cases as opportunities.
- Retrieval resistance: Models can provide harmful information retrieved from context even when they do not internally know that information.A biothreat-ignorant language model answered biothreat-related questions when supplied with relevant textbook or scientific-paper content.
4.3 Model Tampering Evaluations
Model tampering evaluations are needed to assess practical open-weight risks, yet current infrastructure lacks standardized, comprehensive coverage across attack types, configurations, metrics, and the scale of the ecosystem.
- Why tampering evaluations matter: Evaluations under fine-tuning, steering, editing, pruning, and other tampering threats are necessary to assess practical risks from open-weight models.Open models may be modified with unsafe data or backdoors after pretraining, so untampered evaluations are incomplete.
- Current gaps: Many frontier open-weight model assessments omit tampering evaluations, and standardized procedures have not yet been established.The paper notes that gpt-oss was, to its knowledge, the only frontier open-weight model with reported pre-release adversarial fine-tuning evaluations.
- Evaluation infrastructure: Existing toolkits do not fully address elicitation sensitivity, hyperparameter sensitivity, diverse attacks, or multiple tampering metrics.The resulting patchwork evaluations can be difficult to compare and trust.
- Threat coverage: Current tamper-resistance research focuses mainly on fine-tuning, leaving pruning, low-rank modifications, and backdoor insertion insufficiently studied alongside defenses.More rigorous evaluation requires considering the full tampering toolkit and how simple modifications alter capabilities and risks.
- Systematic evaluation: Exhaustively testing attack configurations is computationally prohibitive because fine-tuning attacks vary across learning rates, steps, algorithms, and datasets.Standardized assessment of many attack and defense configurations could reveal patterns useful for future safety approaches.
- Scaling evaluations: Ecosystem-scale evaluation must balance efficiency with thoroughness across many architectures, continuously arriving models, and costly tampering attacks.Model provenance techniques may help support infrastructure for evaluating the large open-weight ecosystem.
4.4 Staged Deployment Strategies
Staged deployment offers intermediate access strategies for monitoring risks before full open release, but split, hardware-locked, and encrypted deployments face security, efficiency, practicality, and scope constraints.
- Staged deployment: Gradually increasing access before full release helps developers monitor unexpected uses and adjust safeguards and deployment strategies.Deployment strategies form a spectrum between fully closed and fully open, allowing different tradeoffs between access and centralized control.
- Scope: Some intermediate strategies, including beta testing, fall outside this paper’s open-weight technical-problem scope.The paper focuses on technical strategies on the openness spectrum that can support staged deployment.
- Split deployment: Split deployment divides a model between client and server devices, but private layers must remain secure against reconstruction and distillation attacks.Prior split-learning research often targets running large models on small devices or protecting user inputs from developers.
- Split deployment: Communication overhead makes split deployment less competitive, especially for autoregressive and diffusion models requiring client-server communication at every iteration.Future methods could reduce shared information, message counts, or communication delay.
- Hardware locking: Hardware locking links a model to specific secure hardware and creates a hardware-to-model chain of trust.Deploying such infrastructure in a changing open-weight ecosystem presents design and deployment challenges.
- Scope and practicality: Split inference, hardware locking, and homomorphic encryption are treated as staged-deployment intermediates rather than traditional open-weight releases.Users cannot independently run the full model in these deployments; homomorphic encryption may be practical only in niche applications for frontier models.
4.5 Model Provenance and Forensics
Model provenance and forensics help stakeholders identify open-weight models, reconstruct their spread, and verify properties of their training origins. The paper highlights unresolved challenges in making these tools durable, scalable, and practical across the ecosystem.
- Model provenance methods help answer what model is being used and what modifications it has undergone since release.They support study of the spread and real-world uses of open-weight models.
- Model provenance and forensics comprise watermarking, model heritage inference, and proof of training.These methods address identification, model genealogy, and verification of training properties, respectively.
- Current watermarks can become undetectable after quantization, fine-tuning, model merging, or pruning, and standardized durability benchmarks are lacking.The open problem is to improve durability against common modifications without introducing side effects.
- Ecosystem-wide heritage inference is not yet tractable because current approaches require millions of pairwise model comparisons and must accommodate thousands of new uploads daily.The challenge extends beyond computational scaling to other limitations in comparing models.
- Proof-of-training methods face computationally prohibitive overhead when generating zero-knowledge proofs for training runs with trillions of datapoints and billions of parameters.Practical adoption also requires integrating the proofs into the broader AI ecosystem.
5 What techniques are prominent open-weight developers reporting on?
The analysis examines how prominent open-weight developers document technical safety safeguards across language, multimodal, image, and video models. Reporting is strongest for data curation, while tamper resistance, tampering evaluations, staged deployment, and provenance receive little or no coverage.
- The study selected ten widely adopted 2025 language or multimodal models and five image or video models associated with documented deepfake use.The sample spans multiple organizations, jurisdictions, architectures, and modalities, although it is not exhaustive.
- Table 2 qualitatively classifies each model’s safeguard reporting as no mention, 1-3 sentences, a paragraph, or a dedicated section or paper.The table documents disclosure trends rather than evaluating safeguard substance or implying that no mention means no implementation.
- 6 out of 10 language or multimodal models report at least brief data-curation safeguards, ranging from 1-3 sentences to dedicated sections.Some paragraph-level accounts focus on post-training or mid-training safety fine-tuning rather than pre-training data filtering.
- Tamper-resistant training algorithms receive no mention, tampering evaluations appear in only one model, and staged deployment and provenance or forensics are absent from the examined technical reports.The single tampering-evaluation report is a dedicated adversarial fine-tuning assessment for gpt-oss.
- Among five image and video models, only FLUX and Wan2.x reports mention safety-focused data curation.The broader pattern is similarly sparse reporting across the image and video sample.
- The reporting gap suggests safeguards may be underimplemented, undocumented, or supported by methods that remain underdeveloped.The analysis presents these as alternative explanations rather than determining which explanation applies.
6 Discussion
Open-weight models offer broad access, research, and decentralization while creating distinct risk-management challenges. The paper argues that addressing these challenges requires technical safeguards alongside holistic monitoring and greater openness in research, evaluations, methods, and reporting.
- Open-weight model capabilities are consistently estimated to lag frontier proprietary models by only 6-12 months, while releases support innovation, safety research, flexible adoption, and broader access.Their benefits coexist with risks arising from how they are deployed and adapted.
- The paper focuses on technical tools with distinct implications for open-weight models, without claiming these tools or problems are the most important parts of a holistic risk strategy.The authors emphasize that ecosystem-wide monitoring and mitigation will also be crucial.
- The effectiveness of safeguards and the counterfactual risk of open-weight models relative to closed-weight models remain unclear.The paper therefore emphasizes gathering more information through additional research and ecosystem impact analysis.
- Some safeguards will resist intervention only to a limited degree, while incentives for private actors to develop robust frontier open-weight safeguards are currently limited.These conditions make technical-intervention research comparatively high-risk for researchers.
- Building open-weight risk-management science requires open collaboration, research, evaluations, risk-methodology reporting, and standardized documentation—not only openly available weights.The paper frames this as a collective effort needed to produce a collective good.