PaperlayerView exact revision on arXiv

Source-linked AI summary

gpt-oss-120b & gpt-oss-20b Model Card

OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, Shengjia Zhao

arXiv:2508.10925v1cs.CLcs.AI

TL;DR

Open-weight models create safety challenges because released weights can be modified to bypass refusals or optimize for harm. This paper introduces agent-oriented reasoning models with a structured chat format and evaluates their capabilities and safety. The models remain below tested High capability thresholds, including after adversarial fine-tuning, while generally matching the performance of existing open models in biological evaluations.

  • Problem

    Released open models can be fine-tuned to bypass safety refusals or directly optimize for harm, creating a distinct safety profile from proprietary models.

  • Method

    The paper introduces two open-weight reasoning models designed for agentic workflows and uses a harmony chat format to delineate message boundaries and roles.

  • Results

    The default gpt-oss-120b model did not reach High capability thresholds in three tracked categories, and adversarial fine-tuning did not reach High capability in Biological and Chemical Risk or Cyber risk.

  • Takeaways & Limitations

    For most biological evaluations, existing open models already came near to matching adversarially fine-tuned gpt-oss-120b performance, so the release did not significantly advance that frontier.

  • Takeaways & Limitations

    The evaluation results likely represent lower bounds on capability because additional scaffolding or improved capability elicitation could increase observed performance.

Abstract

from arXiv · show

We present gpt-oss-120b and gpt-oss-20b, two open-weight reasoning models that push the frontier of accuracy and inference cost. The models use an efficient mixture-of-expert transformer architecture and are trained using large-scale distillation and reinforcement learning. We optimize the models to have strong agentic capabilities (deep research browsing, python tool use, and support for developer-provided functions), all while using a rendered chat format that enables clear instruction following and role delineation. Both models achieve strong results on benchmarks ranging from mathematics, coding, and safety. We release the model weights, inference implementations, tool environments, and tokenizers under an Apache 2.0 license to enable broad use and further research.

1 Introduction

OpenAI introduces two open-weight reasoning models designed for agentic workflows, while examining their safety profile and frontier impact. The evaluations found no High capability under the tested thresholds, including after adversarial fine-tuning, and no significant advance in open biological capabilities.

  • Model introduction: gpt-oss-120b and gpt-oss-20b are open-weight reasoning models available under the Apache 2.0 license and usage policy.They are designed for instruction following, web search, Python execution, adjustable reasoning effort, customization, full CoT, and Structured Outputs.
  • Safety considerations: Open models have a different risk profile because released weights can be fine-tuned to bypass refusals or optimize directly for harm.OpenAI notes that additional mitigations or access revocation may not remain available after release.
  • Safety considerations: Developers and enterprises may need extra safeguards because gpt-oss models will be incorporated into systems maintained by varied stakeholders.The model card distinguishes model-level behavior from system-level protections provided by API products.
  • Capability safety evaluations: The default gpt-oss-120b model did not reach High capability thresholds in Biological and Chemical, Cyber, or AI Self-Improvement categories.These categories are part of OpenAI’s Tracked Categories in its Preparedness Framework.
  • Capability safety evaluations: Even robust adversarial fine-tuning did not bring gpt-oss-120b to High capability in Biological and Chemical Risk or Cyber risk.The testing simulated potential attacker actions and was reviewed by OpenAI’s Safety Advisory Group.
  • Capability safety evaluations: For most biological evaluations, existing open models came near to matching gpt-oss-120b’s adversarially fine-tuned performance.The authors therefore found no significant advance in the frontier of biological capabilities in open foundation models.

2 Model architecture, data, training and evaluations

gpt-oss-120b and gpt-oss-20b are mixture-of-experts reasoning models with quantized weights, customizable reasoning levels, and a harmony chat format for instruction following and tool use. Across reasoning, coding, tool-use, and health evaluations, the models are competitive with leading closed models, while longer reasoning generally improves accuracy at higher latency and cost.

  • Model architecture: The models use autoregressive mixture-of-experts transformers with 116.8B and 20.9B total parameters, respectively.gpt-oss-120b has 5.1B active parameters per token, while gpt-oss-20b has 3.6B active parameters.
  • Model architecture: MXFP4 quantization reduces MoE weights to 4.25 bits per parameter, allowing gpt-oss-120b to fit on one 80GB GPU and gpt-oss-20b to run with 16GB memory.
  • Evaluations: gpt-oss-120b surpasses OpenAI o3-mini and approaches o4-mini on main reasoning, coding, and tool-use evaluations, while gpt-oss-20b remains competitive despite being 6 times smaller.The evaluations include canonical reasoning tasks, Codeforces, SWE-Bench Verified, and τ-Bench.
  • Data and training: The models are trained on trillions of text-only tokens focused on STEM, coding, and general knowledge, then post-trained with chain-of-thought reinforcement learning for reasoning and tool use.The training data has a June 2024 knowledge cutoff.
  • Data and training: The harmony chat format separates message roles and visibility channels, enabling instruction hierarchy, chain-of-thought, tool calls, and user-facing answers within agentic workflows.The role hierarchy is System > Developer > User > Assistant > Tool.
  • Evaluations: Increasing reasoning effort produces smooth test-time scaling, with longer chain-of-thought generally yielding higher accuracy but greater response latency and cost.Users are recommended to balance model size and reasoning level for their use case.
  • Evaluations: At high reasoning effort, gpt-oss-120b nearly matches OpenAI o3 on HealthBench and HealthBench Hard and significantly outperforms several other frontier models.

3 Safety testing and mitigation approach

The safety approach combines deliberative alignment, default adherence to safety policies, and evaluations of both standard and adversarially fine-tuned models. The reported testing found no High capability threshold reached in the tracked categories and no significant advance over existing open models in biological capability.

  • Mitigation approach: Deliberative alignment trains the models to refuse illicit advice, resist jailbreaks, and follow the instruction hierarchy.
  • Mitigation approach: Open-weight release creates a distinct risk profile because attackers may fine-tune models to bypass refusals or optimize directly for harm.
  • Safety testing: gpt-oss-120b did not reach indicative High capability thresholds for Biological and Chemical capability, Cyber capability, or AI Self-Improvement.
  • Safety testing: Adversarial fine-tuning with OpenAI’s training stack still did not bring gpt-oss-120b to High capability in Biological and Chemical Risk or Cyber risk.The adversarially fine-tuned versions were created for internal testing and were not released.
  • Safety testing: On most biological evaluations, another existing open-weight model scored at or near gpt-oss-120b, making a significant frontier advance unlikely.
  • Mitigation approach: Developers and enterprises may need extra safeguards because downstream systems must reproduce protections provided by OpenAI-served products.The model card distinguishes model-level behavior from system-level protections.

4 Default Safety Performance: Observed Challenges and Evaluations

The models are evaluated across disallowed-content, jailbreak, instruction-hierarchy, and hallucination settings, revealing strengths alongside specific robustness and factuality gaps.

  • Disallowed Content: Production Benchmarks are more challenging and representative than the saturated Standard Disallowed Content Evaluation, so their scores are expected to be lower.The newer set uses highly multi-turn, less straightforward conversations and is intended to measure continuing safety progress.
  • Disallowed Content: Both models generally perform within 1-2 points of OpenAI o4-mini on the Standard Disallowed Content Evaluation and tend to significantly outperform it on Production Benchmarks.gpt-oss-20b underperforms o4-mini on the illicit/violent Production Benchmarks category but still outperforms GPT-4o.
  • Jailbreaks: Both models generally perform similarly to OpenAI o4-mini on the StrongReject jailbreak evaluation.The evaluation inserts known jailbreaks into prompts across several harm categories and grades outputs with not_unsafe.
  • Instruction Hierarchy: The models are less strong than OpenAI o4-mini at preventing users from overriding system messages, although developers can fine-tune them for greater robustness to encountered jailbreaks.This creates a practical difference between jailbreak robustness and protection against instruction override.
  • Bias: The models perform at about parity with OpenAI o4-mini on the BBQ evaluation.The supplied passage reports the overall comparison without specifying a numerical score.

5 Preparedness Framework

The Preparedness Framework evaluates whether open-weight models could reach high-risk capability levels after adversarial fine-tuning. For gpt-oss-120b, the reviewed testing found no High capability in Biological and Chemical Risk or Cyber risk.

  • Framework: The Preparedness Framework tracks and prepares for frontier capabilities that create new risks of severe harm.It includes safeguards intended to sufficiently minimize risk for highly capable models.
  • Safety Training: The models use harmful-data filtering, deliberative alignment, and instruction hierarchy training to refuse unsafe prompts and defend against prompt injections.CBRN-related harmful data were filtered during pre-training, while safety behaviors were taught during post-training.
  • Adversarial Training: The evaluation created adversarially fine-tuned gpt-oss-120b versions for Biological and Chemical and Cyber capability categories.The simulated adversary had strong post-training infrastructure, machine-learning knowledge, harmful in-domain data, and a large compute budget.
  • Adversarial Training: The adversarial training combined helpful-only reinforcement learning with domain-specific capability maximization using web browsing, expert biological data, or cybersecurity capture-the-flag environments.The biological model was trained end-to-end for web browsing, while the cyber model used cybersecurity capture-the-flag challenge environments.
  • Findings: Even with robust fine-tuning using OpenAI’s training stack, gpt-oss-120b did not reach High capability in Biological and Chemical Risk or Cyber risk.OpenAI’s Safety Advisory Group reviewed the testing and reached this conclusion.
  • External Review: External reviewers submitted 22 recommendations, of which OpenAI acted on 11, including 9 of 12 items labeled high urgency.Implemented changes included new analyses, clarified assumptions, stronger scaffolding tests, and improved reporting.

5.2 Capability findings

gpt-oss-120b shows notable biological knowledge but remains below high capability thresholds in complex protocol debugging and visually dependent laboratory contexts. Across biosecurity, cyber, and AI self-improvement evaluations, the results do not indicate a major advance over current open models.

  • Biological and Chemical: gpt-oss-120b shows notable strength on textual biological knowledge and harm scenarios but not on complex protocol debugging or visually dependent laboratory tasks.Its text-only architecture limits applicability in visually dependent laboratory contexts.
  • Biological and Chemical: All tested models outperformed the average human baseline of 40% on the virology troubleshooting benchmark, while OpenAI o3 scored highest.The evaluation used 350 held-out virology troubleshooting questions and controlled browsing access.
  • Biological and Chemical: All models underperformed the consensus expert baseline of 54% and median expert baseline of 42% on open-ended protocol troubleshooting.OpenAI o3 remained the highest-performing model on this benchmark.
  • Biological and Chemical: All models underperformed the 80th-percentile human score of 36.4% on TroubleshootingBench, where OpenAI o3 ranked highest.TroubleshootingBench targets tacit, hands-on knowledge from uncontaminated expert-written procedures.
  • Biological and Chemical: Qwen 3 Thinking and Kimi K2 reached a level competitive with adversarially fine-tuned gpt-oss-120b on biosecurity evaluations.No single open model consistently outperforms the others in this domain.
  • AI Self-Improvement: OpenAI o3 and o4-mini remained highest across AI self-improvement benchmarks, while gpt-oss models showed no improved performance on relevant software engineering and AI research tasks.On one evaluation, OpenAI o4-mini was one percentage point above OpenAI o3, and gpt-oss models were two percentage points below OpenAI o4-mini.

6 Appendix 1

The harmony format represents system, developer, and user instructions in the model input, then supports reasoning and tool calls in the model response.

  • Input format: The harmony input format specifies a low-reasoning system message, a developer-provided weather function, and a user request for San Francisco weather.The example illustrates role delineation and function availability in the rendered chat format.
  • Response format: The harmony response format shows chain-of-thought followed by a model tool call.The example demonstrates how reasoning and tool use are represented in the response.

7 Appendix 2

Appendix 2 describes recommendations received on adversarial testing methodology and the corresponding responses.

  • Appendix 2: The appendix covers recommendations on adversarial testing methodology and how the authors responded to them.It frames the appendix as a record of methodological feedback and revisions.

1. Clarifying Threat Model and Risk Categorization

The authors clarified assumptions about low-resource actors and retained ProtocolQA as a required part of the preparedness assessment.

  • Threat model: The authors added clarifying language on low-resource actors’ compute, machine-learning expertise, and data-access assumptions.Future cost estimates were flagged for follow-up.
  • Risk categorization: The authors clarified preparedness criteria, retained ProtocolQA as required, and reran OpenAI o3 with a blocklist for consistency.The paper text was edited accordingly.

2. Strengthening Evaluation Completeness and Reliability

The evaluation was strengthened through additional robustness checks, improved benchmark comparability, clearer expert baselines, and explicit analysis of refusal behavior.

  • ProtocolQA troubleshooting results were checked for refusals, additional protocol-debugging training data, and contamination using a new related evaluation.
  • Inference-time scaling plots were added for bio and cyber evaluations to show performance across trial counts.
  • Text-only Multimodal Virology Troubleshooting results improved comparability, while VCT was run on the final 322-question dataset with human baseline comparisons.
  • Expert profiles and baseline calculations were specified more clearly in the evaluation reports.
  • Refusal-based failures were separated from other failure modes, with pre- and post-naughtification rates reported.

3. Improving Evaluation Setup

The evaluation setup considered enhanced scaffolding, aligned training data, hyperparameter verification, and policy decisions about benchmarking and refusal comparisons.

  • Higher-quality agent scaffolding for measurements: Internal “Best of K” scaffolding was tested in cyber evaluations, while broader best-of-N application was recommended but only partially conducted.Full reruns were expected to provide limited additional gains.
  • Improving Evaluation Setup: Analogous RL datasets were tested during training to assess whether alignment with ProtocolQA produced harmful uplift.
  • Improving Evaluation Setup: Fine-tuning hyperparameters were aligned with internal researchers’ recommended settings and changed when necessary for maximum performance.
  • Omit ProtocolQA from preparedness thresholds: ProtocolQA was retained because it provided a unique safety signal despite imperfect real-world troubleshooting coverage, while broader preparedness-criteria changes were out of scope.
  • Closed vs. open model refusal comparison: Closed-model combined performance was recommended to substitute non-refusal responses and treat refusals as zero.
  • Closed vs. open model refusal comparison: Past testing found that closed models generally did not refuse on benign-proxy tasks, except Gryphon, limiting the signal about closing gaps on real malicious tasks.

8 Contributors

The contributors are listed alphabetically by surname, spanning the named authors from Sandhini Agarwal through Shengjia Zhao.

  • 8 Contributors: Contributor names are specified as alphabetically ordered by surname.
  • 8 Contributors: The contributor list begins with Sandhini Agarwal and continues through Johann Raso, Hongyu Ren, and Kimmy Richardson.
  • 8 Contributors: The list concludes with Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao.
Loading 2508.10925v1…