PaperlayerView exact revision on arXiv

Source-linked AI summary

Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

NVIDIA, :, Aastha Jhunjhunwala, Aayushi Ashokkumar Srivastava, Abhibha Gupta, Abhinav Garg, Abhinav Khattar, Achsah Wildish, Adam Rajfer, Adi Renduchintala, Adil Asif, Adriana Flores Miranda, Aileen Zaman, Aishwarya Chhabra, Akanksha Shukla, Akhiad Bercovich, Alejandra Rico, Aleksa Gordic, Aleksander Ficek, Alex Gronskiy, Alex Steiner, Alexander Bukharin, Alexandre Milesi, Ali Roshan Ghias, Ali Shehper, Ali Taghibakhshi, Alice Gatti, Alisa Liu, Alok Kumar, Alok Tripathy, Amala Sanjay Deshmukh, Aman Miryala, Aman Shanbhag, Ameya Sunil Mahabaleshwarkar, Ameya Velingker, Amir Klein, Amir Samani, Amit Zuker, Amnon Geifman, Anahita Bhiwandiwalla, Ananth Subramaniam, Andre Manoel, Andrea Santilli, Andrew Tao, Andrii Skliar, Anil Thomas, Anish Mahishi, Anna Shors, Anna Warno, Anuj Doshi, Anusha Kamath, Anwith Kiran, Ao Tang, Arham Mehta, Arnab Dhara, Arnav Komaragiri, Arnav Kundu, Arti Jain, Arun Rao, Arun Venkatesan, Arushi Goel, Asha Anoosheh, Ashish Vaswani, Ashwath Aithal, Asif Ahamed, Asma Kuriparambil Thekkumpate, Aswin Visva, Atefeh Sohrabizadeh, Avinash Vem, Ayush Dattagupta, Benedikt Schifferer, Benjamin Chislett, Bernard Nguyen, Besmira Nushi, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Branislav Kisacanin, Brendan Johnson, Brian Dang, Brian Yu, Bryan Catanzaro, Byung-Kwan Lee, Carlo del Mundo, Chankyu Lee, Chantal Hwang, Charles Wang, Charlie Truong, Chen Cui, Cheng Wang, Cheng-Ping Hsieh, Chenhan Yu, Chenjie Luo, Chintan Patel, Chris Alexiuk, Chris Johnson, Chris Wing, Christian Munley, Coco Li, Collin McCarthy, Cynthia Gao, Dan Su, Dan Zhao, Dane Corneil, Danial Mohseni Taheri, Daniel Afrimi, Daniel Fatade, Daniel Lo, Daniel Machlab, Daniel Serebrenik, Daniele Molinari, Daniil Sorokin, Daria Gitman, Daria Levy, David Mosallanezhad, David Yu, Deepak Narayanan, Dev Churiwala, Devin O'Kelly, Dhruv Nathawani, Di Allan Wu, Dima Rekesh, Dina Yared, Dipendra Misra, Divyanshu Kakwani, Dmitry Belenko, Dominic Manno, Dongyang Fan, Dor Tsur, Duncan Riach, Dustin VanStee, Dylan Lu, Edgar Minasyan, Ehsan Hosseini-Asl, Eileen Long, Elena Lantz, Ellie Evans, Elliott Ning, Elvis Chen, Emanuel Scoullos, Eric Chung, Eric Harper, Eric Pham-Hung, Eric Tramel, Eric Yang, Erik Arakelyan, Evelina Bakhturina, Ewa Dobrowolska, Fadi Saady, Farzan Memarian, Fay Wang, Fei Jia, Felipe Soares, Felipe Vieira Frujeri, Feng Chen, Fengguang Lin, Ferenc Galko, Frankie Siino, Frederic Lafrance, Fuxiao Liu, Gantavya Bhatt, Garvit Kulshreshtha, George Armstrong, Giulio Lovisotto, Gomathy Venkata Krishnan, Gosia Lazuka, Grace Lam, Greg Heinrich, Greg Pauloski, Grzegorz Chlebus, Guilin Liu, Guo Chen, Guoming Zhang, Guyue Huang, Haifeng Qian, Hanrong Ye, Hao Gao, Hao Zhang, Haoqin Tu, Haran Kumar Shiv Kumar, Haribhau Hud, Harris Nover, Harrison Saturley-Hall, Harshil Kotamreddy, Hemil Desai, Henry Estela, Hexin Wang, Himanshu Singh, Hui Wang, Hung-Yueh Chiang, Igor Gitman, Ilia Karmanov, Ilya Loshchilov, Ingo Guehring, Itay Neeman, Ivan Moshkov, Jacky Kwok, Jakub Slowikowski, James Shen, Jane Polak Scowcroft, Jared Casper, Jatin Mitra, Jeff Willette, Jenna Diamond, Jenny Chen, Jesse Oliver, Jessica Medina, Jiacheng Xu, Jialin Song, Jian Hu, Jian Tu, Jian Zhang, Jiantao Jiao, Jiaqi Zeng, Jie-Fang Zhang, Jimmy Zhang, Jin Xu, Jingyu Xin, Jinhang Choi, Joey Conway, Joey Guman, John Kamalu, John Liu, John Roberts, Johnny Greco, Jonah Alben, Jonas Yang, Jonathan Cohen, Jonathan Li, Jonathan Raiman, Joseph Jennings, Juhi Mittal, Julia Gusak, Julien Veron Vialard, June Yang, Junkeun Yi, Jupinder Parmar, Ka Shun Shum, Kaiwen Shi, Kajal Jain, Kanishk Singla, Karan Sapra, Kari Briski, Kateryna Chumachenko, Katherine Huang, Katie Lyons, Keith Wyss, Keshav Santhanam, Keval Morabia, Kevin Shih, Kezhi Kong, Khanh Nguyen, Khushi Bhardwaj, Kiran Praveen, Konstantinos Krommydas, Krishna Puvvada, Krzysztof Pawelec, Kushal Shah, Li Ding, Lily Lee, Linda Liu, Linglin Jing, Luis Vega, Maarten Van Segbroeck, Mahan Fathi, Mahdi Nazemi, Maka Dong, Makesh Narsimhan Sreedhar, Maor Ashkenazi, Marc Cuevas, Marcin Chochowski, Marco Pavone, Markus Kliegl, Marta Stepniewska-Dziubinska, Martina Gonzalez Vilas, Martyna Patelka, Matan Rusanovsky, Matthieu Le, Matty Machczynski, Matvei Novikov, Mauricio Ferrato, Meenakshi Sharma, Mehrzad Samadi, Melissa Corpuz, Melissa Saelzer, Meng Xin, Mengru Wang, Mengxi Wu, Meriem Boubdir, Michael Boone, Michael Callahan, Michael Elad, Michael Lightstone, Michal Bien, Michal Zawalski, Miguel Martinez, Mikail Khona, Mike Chrzanowski, Mike Ranzinger, Mike Wojcikiewicz, Min-Hung Chen, Ming Kang, Mingjie Liu, Mingyuan Ma, Misha Nekorystnov, MJ Mikulski, Mohammad Qazim Bhat, Mohammad Shoeybi, Morgan Purcell, Mostofa Patwary, Nabin Mulepati, Nancy Agarwal, Narimane Hennouni, Nathan Sala, Nave Assaf, Nayeon Lee, Neal Vaidya, Negar Habibi, Netanel Haber, Nick Gupta, Nick Reamaroon, Nickson Quak, Nidhi Bhatia, Nik Spirin, Niket Agarwal, Nikolai Ludwig, Nima Tajbakhsh, Nithin Sunny, Oleksii Hrinchuk, Oleksii Kuchaiev, Olivier Delalleau, Omer Ullman Argov, Pablo Ribalta, Padmavathy Subramanian, Parth Mannan, Pavlo Molchanov, Pawel Morkisz, Peter Jin, Petr Lapukhov, Petro Junior Milan, Philip Monk, Pinky Xu, Piotr Januszewski, Piotr Laszkiewicz, Po-Han Huang, Praateek Mahajan, Pradeep Thalasta, Prakash Gurumurthy, Pranav Prashant Thombre, Prasoon Varshney, Pratyusha Maiti, Preeti Chadha, Pritam Biswas, Pritam Gundecha, Przemek Strzelczyk, Puhui Meng, Pulkit Kumar, Rabeeh Karimi Mahabadi, Rachel Oberman, Rajpreet Thethy, Ramani Akkati, Rameshwar Shivbhakta, Ran Zilberstein, Randy Gelhausen, Rasoul Shafipour, Raviraj Joshi, Ray Macalisang, Rayen Tian, Reka Kovacs, Renjie Pi, Resha Shetty, Ria C, Rima Shahbazyan, Rishabh Garg, Rita Fernandes Neves, Ritika Borkar, Ritu Gala, Riya Talwar, Riyad Islam, Robert Clark, Robert Kirby, Roger Waleffe, Rohan Joshi, Rohan Pillai, Rohit Jena, Rohit Watve, Roi Garnett, Roi Koren, Rongguang Ye, Russell J. Hewett, Ryan Prenger, Ryan Stewart, Ryota Egashira, Sadegh Mahdavi, Sai Nishanth Padala, Salika Dave, Sandhya Rani Narravula, Sandip Bhaskar, Sangkug Lym, Sanjeev Satheesh, Sarah Amirslani, Saran Vikas Murari, Saurabh Mishra, Saurabh Srivastava, Saurav Muralidharan, Sean Narenthiran, Sean Wang, Seokju Cho, Seong Hee Lee, Seonjin Na, Seonmyeong Bak, Seph Mard, Serge Panev, Sergei Kolchenko, Sergio Perez, Seth Henneman, Seth Poulos, Shahar Mor, Shantanu Acharya, Shaokun Zhang, Shaona Ghosh, Sharath Turuvekere Sreenivas, Shaun Kotek, Shaya Gharghabi, Sheng-Chieh Lin, Shengliang Xu, Shi Chen, Shihao Wang, Shilpa Amireddy, Shiqing Fan, Shiyang Chen, Shrimai Prabhumoye, Shriram Venugopal, Shuang Yu, Shubham Gandhi, Shubham Pachori, Shubham Toshniwal, Shuoyang Ding, Shwetha Krishnamurthy, Simeng Sun, Sivakumar Arayandi Thottakara, Somshubra Majumdar, Soumye Singhal, Sridurga Krithivasan, Sriharsha Niverty, Stephen Ge, Sugam Dipak Devare, Sukrit Rao, Sumeet Kumar Barua, Sungsoo Ha, Surya Nistala, Suseella Panguluri, Sweta Priyadarshi, Syeda Nahida Akter, Talor Abramovich, Tanay Varshney, Tandeep Singh Chandehoke, Teo Ene, Terry Kong, Thanh Do, Thinh Le, Tian Zhang, Tijmen Blankevoort, Tim Romanski, Tiyasa Mitra, Tom Balough, Tom Huang, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Traian Rebedea, Trenton Starkey, Tugrul Konuk, Tuomas Rintamaki, Twinkle Vashishth, Tyler Kniess, Tyler Poon, Udi Karpas, Ushnish De, Utkarsh Vaidya, Vahid Noroozi, Vanshil Atul Shah, Venkat Srinivasan, Venkata Karri, Venmugil Elango, Vibhor Agrawal, Vibhu Jawa, Victor Cui, Victoria Edmonds, Vijay Korthikanti, Vikas Mehta, Vikranth Jeyakumar, Vinay Rao, Vineeth Kalluru Srinivas, Vitaly Kurin, Vitaly Lavrukhin, Wahidur Rahman, Wanli Jiang, Wasi Uddin Ahmad, Wataru Ishihara, Wei Du, Wei Ping, Wei-Ming Chen, Weiheng Chai, Weijia Chen, Wendy Quan, Wenliang Dai, Wenyan Cong, William Zhang, Wojciech Prazuch, Xiaowei Ren, Xin Yu, Xiwen Yu, Yan Wang, Yang Chen, Yang Yu, Yangyi Chen, Yaniv Galron, Yash Jain, Yashaswi Karnati, Ye Yu, Yejin Choi, Yemi Ibrahim, Yev Meyer, Yi Dong, Yi-Fu Wu, Yian Zhang, Yifan Zhang, Ying Lin, Yiyang Zhou, Yoav Miron, Yogendra Pandey, Yonatan Geifman, Yonggan Fu, Yoshi Suhara, Youngeun Kwon, Yu-Cheng Chou, Yu-Chiang Frank Wang, Yuanguo Kuang, Yuekai Zhang, Yugi Guvvala, Yuki Huang, Yunheng Zou, Yunsheng Liu, Zhenming Wang, Zhiding Yu, Zhiqi Li, Zhiyu Cheng, Zhongbo Zhu, Zhuolin Yang, Zichao Yue, Zihan Liu, Zijie Yan, Zirui Liu, Zoe Faul, Zuhair Ahmed

arXiv:2610.38095v1cs.CLcs.AIcs.LG

TL;DR

Long-horizon agents need models that can reason across large contexts, use tools, and work in multimodal environments while remaining efficient to deploy. Nemotron 3.5 combines hybrid Mamba-Transformer models with distillation and specialized training; Super reaches 51.78% on SWE-bench Pro, exceeding Nemotron 3 Ultra with approximately 4.6× fewer active parameters.

  • Problem

    Long-horizon agents must write code, use tools, reason over large contexts, and operate across multimodal environments, making capability and inference efficiency important.

    Source: §1 · p. 1

  • Method

    Nemotron 3.5 combines hybrid Mamba-Transformer models with distillation and long-context training, while Super adds Indic and agentic mid-training and comprehensive post-training.

    Source: §6 · pp. 53–54

  • Results

    51.78% on SWE-bench Pro lets Super exceed Nemotron 3 Ultra with approximately 4.6× fewer active parameters, alongside reported gains in Indic benchmarks and test-time agent scaling.

    Source: §1 · pp. 2–3, §6 · p. 54

  • Takeaways & Limitations

    Super is developed as a multimodal agent, while Lightning follows a lighter text-only post-training pipeline for efficient deployment.

    Source: §6 · p. 54

    Evaluation scores may vary across runs because of stochastic sampling and, where applicable, judge variability.

    Source: §4.9 · p. 39

Abstract

from arXiv · show

We introduce Nemotron 3.5 Super and Nemotron 3.5 Lightning, open and efficient Mixture-of-Experts Hybrid Mamba-Transformer models designed for reasoning, long-context understanding, and agentic workloads. Super is a 120B-parameter, 12B-active multimodal model, while Lightning is a smaller 30B-parameter, 3B-active text model; both designed for efficient deployment. Both models combine large-scale distillation, long-context training, and native speculative decoding, while Super further incorporates Indic continued pretraining and agentic software-engineering mid-training. We develop a comprehensive post-training pipeline spanning supervised fine-tuning and reinforcement learning (RLVR and RLHF), complemented in Super by specialized teacher training and multi- teacher on-policy distillation (MOPD). This transforms Super from a text-only base model into a multimodal agent capable of reasoning over text, images, video, and interactive tool environments. We further explore test-time scaling through parallel agent execution and self-verification. Across training and deployment, we jointly optimize capability and efficiency. Indic continued pretraining improves Indian-language benchmarks by up to 9.82 percentage points, while agentic mid-training strengthens software-engineering capabilities and robustness across agent harnesses. The post-trained Super model achieves 51.78% on SWE-bench Pro, surpassing Nemotron 3 Ultra despite using approximately 4.6x fewer active parameters. Post-training MTP boosting improves speculative- decoding acceptance length by up to 21.8% under probabilistic draft sampling, while NVFP4 quantization achieves 99.62% median accuracy recovery relative to BF16. Inference optimizations enable up to 4,856 output tokens/s/GPU on GB200 while sustaining at least 50 tokens/s/user.

1. Introduction

Nemotron 3.5 Super and Lightning combine hybrid Mamba-Transformer MoE architectures with distillation, long-context training, and speculative decoding. Super adds Indic and agentic software-engineering training, comprehensive post-training, and inference optimizations.

1. Introduction

  • Super is a 120B-total, 12B-active multimodal model, while Lightning is a 30B-total, 3B-active text model; both use hybrid Mamba-Transformer architectures and target reasoning and long-running agentic workloads.Both models are distilled and trained for long context, with native speculative decoding using Multi-Token Prediction.Source: §1 · p. 1
  • Super consolidates two non-shared MTP heads into one recursively shared head, while Lightning initializes its head through architecture-aware projection fusion and pruning; both receive a 100-billion-token MTP warm-up.Super additionally uses hidden-state mixing to mitigate train–inference conditioning mismatch; both models are distilled for 1 trillion tokens.Source: §1 · p. 2
  • Up to 9.82 percentage points of gains across five Indic benchmarks accompany Super’s targeted continued pretraining; agentic mid-training improves DeepSWE pass@1 from 6.4% to 10.4% and Codex-harness SWE-bench Verified from 48.4% to 61.8%.Super also extends to a 1M-token context, then receives 65.4 billion tokens of agentic software-engineering mid-training at 256K context and a 9-billion-token recovery stage.Source: §1 · p. 2
  • Super’s post-training spans SFT, RLVR, RLHF, specialized teacher training, and MOPD; its two-stage SFT pipeline transforms the text-only base model into a vision-language model.The introduction also describes RL tasks involving competitive programming and finance-agent tool use.Source: §1 · p. 3
  • 51.78% on SWE-bench Pro lets Super exceed Nemotron 3 Ultra with approximately 4.6× fewer active parameters; parallel agents and self-verification raise Terminal-Bench 2.1 from 62.7% to 71.9% and DeepSWE from 24.6% to 34.5%.The comparisons and test-time scaling results are reported in the introduction.Source: §1 · p. 3
  • 4,856 output tokens/s/GPU on GB200 is achieved on a 20-turn agentic proxy workload while sustaining at least 50 tokens/s/user; NVFP4 delivers 2.43× BF16 throughput under evaluated interactive serving conditions.The paper also reports MTP acceptance-length gains and quantization accuracy recovery.Source: §1 · pp. 2–3

Checkpoints

  • Checkpoint listings include Super and Lightning base, BF16, and NVFP4 variants, alongside standalone speculative-drafter variants.The listed drafters include LiLiCorr for Super and DFlash and DSpark for Lightning.Source: Checkpoints · p. 3

2. Pretraining

This section describes the training stages for Nemotron 3.5 Super and Lightning, including large-scale distillation and long-context training, with additional Indic, agentic, and context-recovery stages for Super. It also covers Super-V3.5-VL post-training, in which specialized teacher models developed from a shared trunk are merged into a unified student through MoPD.

2.1. Multi-Token Prediction (MTP)

  • MTP warmup exposed Super’s drafting head to mixed hidden states from earlier steps, addressing the train–inference conditioning mismatch that degrades acceptance at deeper draft positions.The 100B-token warmup freezes the backbone and trains only the MTP head; the mixing strategy improves robustness without computationally intensive multi-step test-time training.Source: §2.1 · p. 4

2.2. Distillation

  • Lightning distillation used top-128 logits from Super 3 Base over 1 trillion Ultra 3 Phase 2 data tokens, while Super used Ultra 3 Base with Skew Reverse KL divergence.Lightning combined forward KL and cross-entropy losses at weights 0.9 and 0.1; Super used Skew Reverse KL with α=0.1.Source: §2.2 · p. 5

2.3. Indic Continued Pretraining

  • The Indic corpus contains approximately 2.83 trillion generated tokens across 22 Indian languages and 34 language-script configurations, including native-script and romanized data for twelve priority languages.The twelve priority languages contribute approximately 2.50T tokens, including 224.6B romanized tokens; ten additional lower-resource languages contribute approximately 327B native-script tokens. The broader corpus is slated for public release.Source: §2.3 · pp. 5, 9
  • Translation cost fell from 0.835 to 0.335 GPU-hours per million English input tokens, about a 60% reduction, using concurrent scheduling, speculative decoding, FP8 weights, and FlashAttention.The throughput gain applies to translation only; quality checks and rewriting add further cost.Source: §2.3 · p. 6
  • Indic continued pretraining improved Super across all five Indic benchmarks by 4.98–9.82 percentage points, without separating synthetic-text and natural-text contributions.The later mid-training checkpoint largely retained the Indic benchmark gains, and the Indic-CPT model showed no regressions in tested English benchmark categories.Source: §2.3 · pp. 7–8, Table 2 · p. 8

2.4. Long Context

  • After Super mid-training at 256K context, RULER-1M fell from 79.9 to 33.5, then recovered to 81.0 after 9B tokens of long-context recovery training.The broader long-context phase used 21B tokens for Lightning and 35B for Super; Super’s recovery stage followed its mid-training run.Source: §2.4 · p. 9

2.5. Mid-training

  • Mid-training raised DeepSWE pass@1 from 6.4 to 10.4; across five other harnesses, it improved SWE-bench Verified by 8.6 points on average and Terminal-Bench 2.1 by 6.8 points.Under OpenHands, SWE-bench Verified, Multilingual, and Pro stayed within run-to-run variation of about one point; DeepSWE is averaged over four runs.Source: Table 5 · p. 10, §2.5 · p. 10

2.6. Base Model Evaluations

  • Base-model evaluations cover general knowledge, mathematical reasoning, code generation, commonsense reasoning, reading comprehension, and multilingual capability.Detailed evaluation settings and results are summarized in Table 6 for Super 120B-A12B Base and Lightning 30B-A3B Base.Source: §2.6 · pp. 11–12, Table 6 · p. 11
  • The pipeline specializes teacher models through SFT and RL, then merges them into a unified student with Multi-Teacher On-Policy Distillation (MoPD).Teachers are developed from a shared trunk.Source: Figure 4 · p. 13

3. Post-Training

This section describes Super’s multimodal post-training pipeline and Lightning’s lighter text-only pipeline, covering supervised fine-tuning, reinforcement learning, specialized teachers, distillation, and MTP boosting across agentic, reasoning, vision, and alignment capabilities. It also reports inference-time improvements, benchmark protocols and decontamination, and reasoning-efficiency controls, including the token-use and accuracy trade-offs of reasoning modes.

3.1. Supervised Fine Tuning

  • 14,169 agentic-vision trajectories containing 81,366 assistant actions were retained for SFT after filtering for successful, reference-matching, grounded tool use.Super is trained in two SFT stages with vision, reasoning, chat, long-context, safety, and agentic data; Lightning uses one text-only SFT stage. The agentic corpus teaches image operations and video agents to gather visual evidence and interpret tool observations.Source: §3.1 · pp. 13–14

3.2. Reinforcement Learning; 3.3.1. Agentic Teacher; 3.3.2. Reasoning teacher

  • Router replay and FP32 LM-head computation reduce train–inference KL error by more than an order of magnitude in the joint RLVR and RLHF pipeline.The agentic teacher trains end-to-end on executable terminal, software-engineering, and search tasks; the reasoning teacher uses a 25,600-example science, mathematics, and competitive-coding curriculum, with the selected checkpoint taken after 80 updates.Source: §3.2 · p. 16, §3.3.1 · p. 17, §3.3.2 · p. 17

3.4.1. Video reasoning and tracking teacher

  • 74.90 on Video-MME under the 256-frame reasoning protocol, 19.10 on Video-MME-v2 under its corresponding protocol, 52.59 on GOT-10k, and 60.00 on Molmo2-VideoTrack were achieved by the selected step-60 video teacher.The teacher is used during MOPD; its video-QA and object-tracking training uses deterministic, verifiable rewards.Source: §3.4.1 · pp. 17–18

3.4.2. Vision tool use teacher; 3.4.3. Vision reasoning RLVR

  • Vision-tool-use training targets difficult questions requiring image operations, while vision-reasoning RLVR trains chart, document, table, OCR, geometry, and visual reasoning with outcome-correctness rewards.The tool-use teacher also receives an auxiliary reward against duplicate or malformed tool calls; vision-reasoning data is organized into two stages of 16,090 and 13,848 examples.Source: §3.4.2 · p. 18, §3.4.3 · p. 18

3.5. Alignment teacher

  • The alignment teacher is trained primarily with RLHF, using rubric-level mean scores, fixed Gemma 4 31B-it comparisons, refreshed preferences, and abstention training.A reasoning-length-penalized reward model supports token-efficient training, and the policy training adds updated identity and capabilities data and emphasis on language mixing.Source: §3.5 · p. 19

3.6. MOPD

  • Super’s MOPD distills specialized teachers into the RL student using token-local reverse-KL updates on student-generated responses; Lightning omits MOPD.The two stages shift from agentic and reasoning domains toward chat quality and vision, which improved more slowly; stage 2 retires the agentic and reasoning teachers and uses a proximal-teacher target for stability.Source: §3.6 · pp. 19–20

3.7. MTP Boosting

  • MTP boosting freezes the backbone and updates only MTP heads, using hidden-state mixing and backbone-teacher distillation to improve alignment, especially under non-greedy sampling.Super also receives a short MTP warm start from its SFT checkpoint; Lightning skips that warm start and uses only the post-training boost.Source: §3.7 · p. 20

3.8.1. Evaluation Setup

  • Evaluation spans agentic, reasoning and knowledge, multimodal, conversational and instruction-following, long-context, and multilingual benchmarks, with Super and baselines evaluated under shared settings.The paper describes its evaluation suite and reproducibility resources, and states that models use the same evaluation settings, including agentic resources, inputs, prompts, repeats, and metric implementations.Source: §3.8.1 · pp. 21–22

3.7. MTP Boosting

  • At T=1, MTP boosting raises average acceptance length from 3.78 to 4.61 accepted tokens, with the largest per-position gains at deeper draft positions.The boosted head improves acceptance length in every SPEED-Bench category.Source: §3.7 · p. 21, Table 9 · p. 22, Figure 6 · p. 22

3.9. Reasoning Efficiency and Control

  • Medium-effort reasoning uses around 2.2X less tokens than regular mode, at the cost of approximately 9% lower accuracy.Introduced during SFT using traces generated by Kimi K3’s low-effort mode and later optimized during RL, medium-effort training covers about 4% of RL prompts across math, STEM, and coding, with length-based reward adjustments. Figure 7 compares SWE-bench Pro accuracy against GPU-hour cost.Source: §3.9 · p. 26, Figure 7 · p. 27

3.10. Agentic Coding Across Harnesses

  • Harness rankings differ across SWE-bench Pro and Terminal-Bench 2.1: Claude Code scores highest on the former at 49.45%, while Codex leads the latter at 61.80%.The results underscore the importance of reporting the harness and its configuration alongside model scores.Source: §3.10 · p. 26, Figure 8 · p. 27

3.11. Test-Time Scaling through Parallel Agents and Self-Verification

  • Selection raises observed accuracy on six benchmarks: Terminal-Bench 2.1 accuracy 62.7%→71.9%, DeepSWE 24.6%→34.5%, and BrowseComp 71.5%→79.8%.SWE-bench Verified, Pro, and Multilingual improve by 1.1, 2.4, and 1.9 percentage points, respectively; BrowseComp selection recovers approximately 70% of the pass@1-to-oracle-best-of-three gap without changing model parameters.Source: §3.11 · p. 28, Figure 9 · p. 29

3.12. Benchmark Decontamination

  • The decontamination pipeline removed 20,671 samples (~0.09%), matching 1,808 benchmark questions (6.0%) across 22 of 30 benchmarks; around 3% of judged candidate pairs were severe contamination.Its pipeline uses normalization, verbatim and semantic similarity to retrieve candidates, and an LLM judge to verify contamination; on synthetic variants, the judge identified 87.5% of contaminated Terminal-Bench samples and 79.1% of GPQA Diamond samples, flagging one distractor.Source: §3.12 · pp. 29–30

4. Quantization

The section examines quantization approaches for balancing accuracy, verbosity, and inference efficiency across models and speculative-drafting components. It presents pipeline and deployment studies of weight, cache, and module precision, reporting accuracy recovery, verbosity changes, throughput effects, and tradeoffs under stated evaluation conditions.

4.1. Recipe Design

  • 99.62% median accuracy recovery across 36 benchmarks was achieved by the released aggressive PTQ recipe followed by 200 iterations of SFT-based QAD.Across 36 benchmarks with token measurements, mean per-benchmark verbosity increased by 0.44%.Source: §4.1 · pp. 31–32

4.2. Post-Training Quantization

  • The MSE calibration algorithm provides superior accuracy recovery to Max and MSE 4-over-6, so it is used for PTQ and AutoQuant recipes.Calibration uses 1,024 image–text samples from the post-training SFT data, with images passed through the vision encoder.Source: §4.2 · p. 33

4.3. Sensitivity-Driven Mixed Precision

  • AutoQuant’s sensitivity profile places nearly half of NVFP4 sensitivity in the lm_head, while MoE experts are comparatively robust.Assignments concentrate NVFP4 on parameter-dominant expert weights and reserve FP8 or BF16 for sensitive attention and Mamba projections and the lm_head.Source: §4.3 · p. 34

4.4. Quantization-Aware Distillation (QAD)

  • Among the evaluated final candidates, the 4.82-BPE recipe’s 200-iteration checkpoint provided the strongest accuracy–verbosity tradeoff, with accuracy recovery almost matching median BF16 accuracy and verbosity.The principal QAD benefits largely saturate by 300 iterations.Source: §4.4 · p. 35

4.5. KV Cache Quantization Evaluations

  • 99.36% median accuracy recovery with NVFP4 KV cache is comparable to 99.52% with the released FP8 KV cache across the same 35 benchmarks.NVFP4 cache payload is ideally 43.75% smaller than FP8; the released checkpoint defaults to FP8 KV cache for runtime-efficiency and software-readiness considerations.Source: §4.5 · pp. 35–36

4.6. Attention and Projection Precision

  • MoE expert precision strongly affects accuracy and verbosity, while attention and Mamba projection quantization have distinct, more limited effects.Expert quantization can cause token-distribution drift and runaway loops; attention projection has little accuracy impact in most cases, though per-tensor FP8 can impair long-context disambiguation. FP4 Mamba output-projection weights and FP4 lm_head weights can inflate generation length, so the lm_head should remain FP8 or higher.Source: §4.6 · pp. 36–37

4.7. MTP Quantization

  • Quantizing MTP experts reduces acceptance length from 4.612 to 4.592 at draft length 7 against the same NVFP4 backbone.Latency projections estimate reductions of 47% for the MTP layer and 49% or 71% for the lm_head at FP8 or W4A16, respectively; the target lm_head can remain BF16 while drafting uses a lower-precision lm_head.Source: §4.7 · pp. 37–38
  • The MTP uses the backbone’s KV-cache precision, which empirical validation found does not degrade acceptance lengths; its target lm_head uses FP8 for both verification and drafting.Released checkpoints quantize the boosted MTP separately and splice it into the quantized backbone, requiring an exact match in MTP tensor names, shapes, and dtypes; co-quantization is identified as the preferred future approach.Source: §4.7 · p. 38

4.8. Parallel Drafter Quantization

  • 11% throughput improvement on DGX Spark followed NVFP4 quantization of Super’s LiLiCorr linear layers with BF16 activations; quantizing the LM head added another 5%.Acceptance lengths were not meaningfully sacrificed; the experiments target memory-bound scenarios where weight quantization reduces drafter-parameter loading overhead.Source: §4.8 · p. 39

4.9. Lightning 3.5 Evaluation and Results

  • 99.24% Median Accuracy Recovery and −3.63% Ratio of Totals Verbosity change are reported for Lightning’s W4A16 NVFP4 checkpoint versus BF16.The evaluation serves both checkpoints with FP8 KV cache and FP32 Mamba SSM cache; results may vary across runs because of stochastic sampling and judge variability.Source: §4.9 · pp. 39–40, Table 22 · p. 42

4.10. Super 3.5 VL Evaluation and Results

  • Super’s Terminal-Bench verbosity change was +15.7% in mean tokens across all rollouts, while the mean over agent-completed rollouts changed +0.2%.About 4.5% of non-completing runaway trajectories, at roughly 700k tokens each, drove the all-rollout mean; excluding them, the Ratio of Totals Verbosity change was 2.99%.Source: §4.10 · p. 41

5. Inference

This section examines inference methods for Nemotron 3.5 Super, spanning prefix-aware serving and platform-specific kernel and cache optimizations for agentic contexts, speculative decoding and parallel-drafter approaches, and deployment results. It also presents a two-stage LiLiCorr drafter with a calibration objective and evaluates draft acceptance on SPEED-Bench within the drafter’s training window.

5.1. Efficiency

  • The efficiency tuning objective selects the highest aggregate output throughput among serving points sustaining at least 50 tokens/s/user on a 20-turn, 2,048-input/1,024-output-token workload.Shared prompts and prior conversation are eligible for prefix reuse; Figure 10 compares max-throughput at 50 tps/user and minimum-latency performance on H100 and GB200.Source: §5 · p. 44, §5.1 · p. 44

5.2. Performance across deployment platforms

  • Nemotron 3.5 Super provides higher performance than a comparable-size Qwen model in the usable interactive-serving region, with points below 50 output tokens/s/user excluded.Its fewer attention layers and Mamba-2 sequence mixing make per-request memory grow more slowly with context, allowing more requests to share each weight read.Source: Figure 11 · p. 45, §5.2 · p. 45

5.3. Prefix reuse and speculative decoding

  • An additional Mamba cache checkpoint near the speculative-decoding replay boundary lets vLLM replay only the final 16 tokens in a 16-token-match example, rather than an entire Mamba block.The scheduler and cache manager select a valid checkpoint, trading one extra Mamba page for less recomputation and enabling prefix caching to complement speculative decoding.Source: §5.3 · p. 46

5.4. Kernel control at the target operating point

  • 4.7–6.4% higher output throughput and 6.4–12.5% higher per-user generation throughput: ReplaySSM improved matched GB200 serving configurations.The comparison used a 16-step ReplaySSM buffer; checkpointing amortizes state writes and reduces memory traffic while shifting work to Tensor Cores.Source: §5.4 · p. 47, Table 24 · p. 48

5.5. Reducing the cost of Mamba checkpoints

  • Internal Mamba checkpointing extracts recurrent state and convolution windows at a desired cache boundary during the prompt’s single model forward, avoiding an extra forward pass.The Mamba scan materializes intermediate states at logical chunk boundaries, which can be aligned to the cache boundary.Source: §5.5 · p. 48

5.6. Disaggregated serving on NVIDIA Vera Rubin

  • 10.3% more output tokens/s per GPU and 61% higher median output speed per user: disaggregated serving outperformed aggregated serving while median TTFT remained below the 5 s target.On 11 NVIDIA VR200 GPUs, median TTFT rose from 0.49 s to 1.12 s; separating prefill and decode lets decode GPUs run larger batches.Source: Table 25 · p. 49, §5.6 · p. 49

5.7. Speculative Decoding with Parallel Drafters

  • Parallel drafters target low-concurrency, memory-bandwidth-bound decoding by proposing all draft tokens in one forward pass, unlike recursive MTP’s pass-per-token drafting.DSpark adds a sequential Markov correction to improve cross-slot coherence; LiLiCorr instead uses a single-pass correlation head to score compatibility across slots.Source: §5.7 · pp. 49–50

5.7.1. Training LiLiCorr for Super3.5

  • LiLiCorr jointly processes candidate tokens across draft slots with a small Transformer correlation head, then commits candidates sequentially without further network evaluation.Its drafter body is distilled from the target’s full next-token distribution, supervising both likely tokens and distribution shape for greedy and sampled verification.Source: §5.7.1 · p. 50
  • The calibration term uniquely matches each slot’s candidate distribution to the target’s renormalized probabilities, unlike the distractor penalty, which has many minimizers.This objective matters for probabilistic draft sampling, which draws each token from the head’s per-slot distribution at temperature 1.Source: §5.7.1 · pp. 51–52
  • Training uses two stages on 6.32M prompts: the first streams from a post-SFT checkpoint, and the second adapts to the released post-RL checkpoint.After filtering, stage one retains 5.67M conversations and 22.6B tokens, while stage two retains 11.01M of 18.2M generated samples and 35.6B tokens; their training windows are 4,096 and 8,192 tokens, respectively.Source: §5.7.1 · p. 52, Table 26 · p. 52

Evaluation

  • Evaluation uses 880 qualitative SPEED-Bench prompts across 11 categories, reporting per-sample acceptance length averaged over requests and three seeds.The drafter uses draft length 7 and concurrency 64; temperature 0 is greedy decoding, while temperature 1 with top-p 0.95 uses probabilistic draft sampling, and NVFP4 Super performs verification.Source: Evaluation · p. 53
  • Because drafter attention is trained only up to 8,192 tokens, evaluation uses sliding-window attention over the latest 8,192 positions to avoid extrapolation to longer query-key distances.Nearly all SPEED-Bench qualitative samples fit within the window, making the reported results effectively those of full attention.Source: Evaluation · p. 53

6. Conclusion

Nemotron 3.5 Super and Lightning pair capability with inference efficiency through distillation, long-context training, and native MTP; Super adds Indic and agentic training and multimodal post-training. The conclusion reports benchmark gains across language, software engineering, speculative decoding, and serving, while the long-translation evaluation covers book translations into six languages.

6. Conclusion

  • Super combines Indic continued pretraining and agentic software-engineering mid-training with a post-training pipeline that makes it a multimodal agent; Lightning uses a lighter text-only pipeline.The conclusion frames their development as jointly optimizing capability and inference efficiency and notes release of checkpoints, datasets, and recipes.Source: §6 · p. 54
  • 51.78% on SWE-bench Pro, outperforming Nemotron 3 Ultra despite approximately 4.6× fewer active parameters.The conclusion also reports test-time scaling improving Terminal-Bench 2.1 from 62.7% to 71.9%.Source: §6 · p. 54
  • MTP boosting improves speculative-decoding acceptance length by 10.1% under greedy decoding and 21.8% under probabilistic draft sampling.Table 28 reports average acceptance length on SPEED-Bench at draft length 7, with greedy and probabilistic sampling settings.Source: §6 · p. 54, Table 28 · p. 54
  • The conclusion reports 99.52% median accuracy recovery relative to BF16 for NVFP4 quantization and 1.9× serving throughput versus Qwen3.8 Flash-Next on GB200.The throughput comparison is for the evaluated interactive agentic workload.Source: §6 · p. 54

A. Long-Translation

  • Figure 12 evaluates book-level translation across 100 PG19 test books, producing 600 translations into German, Spanish, French, Italian, Japanese, and Chinese.Translations are scored with the SEGALE pipeline.Source: Figure 12 · p. 66

B. Ablation study of post-training quantization

The passage examines runaway-loop termination failures in truncated rollouts and reports that they are sharply reduced after MOPD in the tau3-banking example.

B.1. Runaway loop: the termination failure in truncated rollouts

  • Runaway loops are termination failures in which an agent fails to stop before reaching the context wall.The passage distinguishes these from ordinary long reasoning, which often reflects code execution, tool use, and reading results.Source: §B.1 · p. 68
  • After MOPD, tau3-banking runaway-loop failures fell from 19.7% (153/776 rollouts) to one captured failure, while truncation fell from 20.9% to 0.3%.Output tokens per rollout also decreased from 35,173 before MOPD to 13,669 after it.Source: §B.1 · p. 68
  • Truncation measures rollouts with at least one model call hitting an output limit; paired with output length per rollout, it serves as an early-warning indicator.The passage says post-training RL removed the non-terminating tail, not baseline verbosity.Source: §B.1 · p. 68

B.2. FP4 weight quantization swings the stop-versus-continue margin

The section links FP4 expert-weight quantization to shifts in stopping behavior and output verbosity, while reporting that MOPD strengthens termination under quantization. Ablations distinguish expert weights from activation and output-token generation as contributors to runaway loops.

B.2. FP4 weight quantization swings the stop-versus-continue margin

  • In tau3-banking, promoting routed/shared experts from NVFP4 to FP8 reduced runaway from 28.2% to 17.9%, below the BF16 reference's 19.7%.Output verbosity decreased by 21.5% at 2.7σ.Source: §B.2 · p. 68
  • Changing expert activations to BF16 moved runaway by approximately zero, while verbosity fell on all five focused benchmarks, ruling out the activation path as the termination-failure carrier.The five benchmarks were mmlongbench-doc, scicode, tau3-banking, tau2-AA, and AA-LCR.Source: §B.2 · pp. 68–69
  • Returning lm_head to BF16 did not recover termination, ruling out final output-token generation as the source of runaway loops.The section describes the stopping decision as riding on a weak signal, with routed-expert contribution relatively weaker in the late band when runaway occurs.Source: §B.2 · p. 69
  • FP4 expert GEMM output error was estimated at 7.5–22.9% versus BF16, exceeding 10% in 39 of 40 MoE layers and averaging 15%.The authors hypothesize expert-weight quantization as the culprit, while noting that MOPD recovers verbosity without runaway regression in FP4 expert weights.Source: §B.2 · p. 69
  • Changing the Mamba output projection to W4A16 increased verbosity by 18.5% at 2.4σ in tau3-banking, without affecting runaway rate or knowledge abstention.Across post-MOPD checkpoints, W4A16 was more verbose on all six benchmark tests than a recipe keeping the output projection at W8A8.Source: §B.2 · p. 69
  • Under MOPD, only 1 of 776 rollouts looped; with FP4 expert weights, runaway failures were reported in 3 cases, and median stop probability on runaway rollouts reached 0.729.Pre-MOPD runaway rollouts had median peak stop probability 0.043 and never closed the reasoning block; MOPD closed it in 6 of 10.Source: §B.2 · p. 69

B.3. The damage of long-context retrieval on attention projections

The section finds that quantization damage in long-context retrieval depends on whether candidates must be distinguished by position, and on candidate counts and context length. It attributes the vulnerability to limited attention-score resolution and reports precision changes that recover some accuracy.

B.3. The damage of long-context retrieval on attention projections

  • Quantization costs 2.65 to 3.6 pp on disambiguation and nothing measurable elsewhere; the same effect appears on Nemotron 3.5 Lightning.The section focuses on Super 3.5 VL to examine the mechanism.Source: §B.3 · p. 70
  • Across 12 contexts, 4,730–7,981 keys compete for only 35–80 distinct e4m3 score levels, concentrating 60–120 keys per level and limiting position-based distinction.The account describes attention-score dilution, while logits entering softmax are not cast to e4m3.Source: §B.3 · p. 71, Table 34 · p. 71
  • On MRCR-128k, q/k/v FP8 retains 95.3% and 95.5% below 32k, versus 91.4% and 92.1% above 32k; q/k/v MXFP8 recovers above-32k accuracy to 95.3% and 94.3%.The damage is present at every length but is exacerbated at 32k+; Table 35 reports recovery against BF16 by served context over 1,470 tasks.Source: §B.3 · p. 71, Table 35 · p. 72
  • MRCR-128k BF16 accuracy drops from 0.800 to 0.378 as near-identical candidate count rises from two to eight, with candidate count the dominant factor above 32k.Table 36 stratifies accuracy by candidate count and served context over 1,470 tasks.Source: §B.3 · p. 71, Table 36 · p. 72
  • For Lightning, moving the KV cache from FP8 to BF16 recovered +1.43 pp on MRCR-128k, while moving projections from FP8 to BF16 recovered +2.97 pp.On AA-LCR, all-FP8 to NVFP4 projections cost 2.50 pp; the study recommends FP8 q/k/v projection at per-block scale granularity.Source: §B.3 · p. 72, Table 37 · p. 73
  • The output projection is described as structurally exposed yet measurably tolerant in per-tensor FP8, while the AutoQuant search keeps q/k/v in BF16 on three of eight attention layers.The search leaves the output projection FP8 on every layer at the 4.85 bit-per-parameter budget.Source: §B.3 · p. 73

B.4. Mamba output projection: FP4 weights, and the sensitivity is on generation length

This section separates the Mamba output projection's effect on generation length from task accuracy. It reports consistent verbosity inflation under FP4 weight quantization and explains why the projection's activation structure argues against W4A4.

B.4. Mamba output projection: FP4 weights, and the sensitivity is on generation length

  • The Mamba output projection receives near 16 surviving values per FP4 block, unlike expert fc2, whose squared-ReLU input is 90–99% exactly zero.The Mamba projection's normalized SiLU-gated input never has exactly zero values, and 27.6% of elements fall below the FP4 amaxblock/24 rounding limit.Source: §B.4 · p. 73
  • The tensor analysis argues against taking the Mamba output projection to W4A4; rotation or higher activation precision can mitigate the flush.The W4A16 versus W8A8 comparison uses high activation precision to isolate the effect of weight quantization.Source: §B.4 · p. 73
  • All six benchmarks showed verbosity inflation under W4A16 versus W8A8 for the Mamba output projection, while task accuracy was unchanged.OSWorld and SciCode had the clearest inflation, at 1.27× and 1.16×; Table 38 measures output generation length per rollout.Source: §B.4 · p. 73

B.5. The unembedding: FP8 weights, and the sensitivity is on termination

This section locates quantization sensitivity in the unembedding's stop-versus-continue decision, despite close similarity of the quantized stop row. It rejects rare-token structure as the explanation and reports recovery from a two-pass FP4 residual correction.

B.5. The unembedding: FP8 weights, and the sensitivity is on termination

  • NVFP4 unembedding flips 2.45% of stop-versus-continue decisions despite the quantized </think> row retaining cosine similarity 0.9961 to the original.The decision is evaluated as margin = h · (v_stop − v_cont); the competing rows are nearly orthogonal, so small rotations can perturb their difference.Source: §B.5 · pp. 73–74
  • Across group sizes from 1 to 64 elements, pooled error was identical to four decimal places within each format: 0.0265 in FP8 and 0.0952 in FP4.The row-amax spread was 2.84 binades, or 2.12 at p99.9, against roughly 17.8 usable in FP8.Source: §B.5 · p. 74
  • The authors reject rare-token distribution as the source of damage: quantization error is uniformly distributed across the vocabulary, and flushing shows no vocabulary structure.The lowest-norm decile's relative error is no worse than the highest, and NVFP4 flushes 6.8% of elements across 66.9% of blocks.Source: §B.5 · p. 74
  • A two-pass FP4 residual correction lowered KL to 0.00117 versus per-tensor FP8's 0.00134 and raised top-1 agreement with BF16 to 0.9951.The authors say reducing lm_head quantization error remains open to exploration, while suggesting its numerical sensitivity is confined in FP8 precision.Source: §B.5 · p. 74

B.6. A precision recommendation with limitations

The section reports that quantization’s effect on termination behavior depends on the training checkpoint, although several structural measurements remain nearly invariant. It qualifies the resulting precision recommendation with limits on attribution, measurement, and benchmark scope.

B.6. A precision recommendation with limitations

  • Across pre- and post-RL checkpoints, the same quantization recipe aggravated termination failure before RL and reduced it afterward, while structural measures stayed nearly invariant.The stop-versus-continue geometry varied within 1% and routed-expert contribution within 0.007; the authors conclude RL changed hidden-state direction, not the lm_head or expert contribution structure.Source: §B.6 · p. 75
  • The weight-versus-activation attribution is inferred from a bound; the study did not test W16A4.Source: §B.6 · p. 75
  • Expert GEMM output error and benchmark verbosity are separate, sign-consistent measurements, but no specific displacement is shown to cause extra tokens.Source: §B.6 · p. 75
  • Attention score-path dilution was measured on clean weights, while every ablation configuration, including BF16, used an FP8 KV cache.Thus projection deltas are measured on top of an already-diluted score.Source: §B.6 · p. 75
  • The Super VL MRCR-128k ablation uses a full quantized configuration and does not compare against BF16 KV precision.The Lightning study identifies the KV cache as the damage carrier, but the Super VL ablation does not test that contrast.Source: §B.6 · p. 75
  • MRCR-128k is the suite’s only disambiguation instrument, limiting that result to one benchmark and the reported candidate-count split.Three repetitions measure the decoding-noise floor but do not support further per-stratum claims.Source: §B.6 · p. 75
  • Unembedding tests use a proxy input that isolates quantization effects, so their absolute rates are not the model’s true output on those tokens.Source: §B.6 · p. 75
  • Within-stage BF16-versus-FP4 contrasts use fixed token windows, but cross-stage comparisons change both weights and window, leaving the root cause unresolved.Source: §B.6 · p. 75

C. Benchmark Decontamination Details

The decontamination-run configuration shares benchmark-side resources across runs and processes training data in CPU shards, using remote services for embeddings and judge decisions.

C.1. Configuration

  • The question store, n-gram index, and embedding index are built once and shared across runs.Source: §C.1 · p. 75
  • Training data are processed in CPU-node shards, while embeddings and judge decisions come from remote endpoints.Source: §C.1 · p. 75
  • Table 40 lists the settings used across all SFT and RL data decontamination runs.Source: §C.1 · p. 75, Table 40 · p. 76

C.2. Benchmarks

The benchmark suite spans established and modern evaluations, and the reported matches concentrate in a few benchmarks. The judge compares training samples with benchmark questions using a labeled contamination rubric and a severity threshold for removal.

C.2. Benchmarks

  • The benchmark suite spans established standard and modern evaluations, with CoDeC contamination evaluations informing selection.Source: §C.2 · p. 76
  • A quarter of MATH-500 problems and about 15% of IFBench and KernelBench items matched; MMLU-Pro had 1,254 matched questions, while eight mostly recent agentic and expert benchmarks had none.Source: §C.2 · p. 76

C.3. Judge Prompt

  • The judge receives training_sample and benchmark_question as separate JSON strings and decides from those texts whether the sample gives an unfair advantage.It is asked to provide reasoning before its verdict and rate removal severity separately from the label.Source: §C.3 · p. 76
  • The rubric distinguishes direct copy, paraphrase, answer leakage, method leakage, style overlap, and no contamination, with severity and confidence scored from 0 to 4.A sample is removed if any pair receives a contaminating label with severity ≥3.Source: §C.3 · pp. 76–77, Table 41 · p. 77

C.4. Synthetic Evaluation

The synthetic evaluation tests contamination detection on transformed benchmark questions and hard same-domain negatives because real-data ground-truth labels are unavailable. Results show strong recovery for many transformations, with misses concentrated in aggressive changes that obscure the source question.

C.4. Synthetic Evaluation

  • Because real-data ground-truth contamination labels do not exist, the evaluation transforms benchmark questions with 13 augmentations at four strengths.The augmentations include surface edits, structural changes, and LLM-generated semantic changes such as paraphrase, translation, and abstraction.Source: §C.4 · p. 77
  • Transformed target questions are positives by construction, while hard negatives come from a different benchmark in the same domain.The LLM generating semantic augmentations differs from the judge, preventing the judge from recognizing its own rewrites.Source: §C.4 · p. 77
  • The judge recovered most synthetic positives with one false positive among 12,763 negatives; verbatim overlap recovered about a quarter and semantic overlap about half.Recall was lower on GPQA than Terminal-Bench because its short multiple-choice questions carry less distinctive text.Source: §C.4 · p. 77, Table 42 · p. 78
  • Copies, case changes, reordering, burial, paraphrases, and translations were recovered almost completely, while aggressive crops, heavy word dropout, and strong abstraction drove most misses.Every transformed question counted as positive regardless of how far it drifted, making recall conservative.Source: §C.4 · p. 77, Table 43 · p. 78

C.5. Notable Cases

The removed examples show that benchmark contamination can survive translation, reformulation, and embedding in other training data, beyond what string matching captures. All identified cases were kept out of the training mix.

C.5. Notable Cases

  • Translated and reformulated MATH-500 problems shared almost no n-grams with the benchmark but still taught the answer.Some asked for a quantity derived from the original answer.Source: §C.5 · p. 78
  • Benchmark prompts reappeared in RLHF collections in seven other languages, sometimes with changed numbers.The examples included Chinese, Japanese, Spanish, Polish, Russian, German, and French.Source: §C.5 · p. 78
  • Terminal-Bench task descriptions appeared nearly verbatim in agentic RL prompts with environment instructions and different file paths.Source: §C.5 · p. 78
  • Humanity’s Last Exam questions appeared in math and tool-integrated reasoning data as copies, paraphrases, or problems containing benchmark answers as intermediate steps.Source: §C.5 · p. 78
  • All identified contamination cases were prevented from entering the training mix.Source: §C.5 · p. 78

C.6. Limitations

The contamination checks have bounded recall and score only problem statements; their synthetic-data evaluation is a proxy for contamination in practice.

C.6. Limitations

  • Contaminated samples outside the judge’s candidate pairs or below the similarity gate are always kept.Source: §C.6 · p. 79
  • Scoring only problem statements misses contamination from shared answer sets or other artifacts.Source: §C.6 · p. 79
  • Synthetic augmentations approximate but do not exhaust the ways contamination arises in practice.Source: §C.6 · p. 79

Related papers

Loading 2610.38095v1…