PaperlayerView exact revision on arXiv

Source-linked AI summary

Intern-S1: A Scientific Multimodal Foundation Model

Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, Yu Cheng, Pei Chu, Tao Chu, Erfei Cui, Ganqu Cui, Long Cui, Ziyun Cui, Nianchen Deng, Ning Ding, Nanqing Dong, Peijie Dong, Shihan Dou, Sinan Du, Haodong Duan, Caihua Fan, Ben Gao, Changjiang Gao, Jianfei Gao, Songyang Gao, Yang Gao, Zhangwei Gao, Jiaye Ge, Qiming Ge, Lixin Gu, Yuzhe Gu, Aijia Guo, Qipeng Guo, Xu Guo, Conghui He, Junjun He, Yili Hong, Siyuan Hou, Caiyu Hu, Hanglei Hu, Jucheng Hu, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Xu Huang, Zixian Huang, Zhe Jiang, Lingkai Kong, Linyang Li, Peiji Li, Pengze Li, Shuaibin Li, Tianbin Li, Wei Li, Yuqiang Li, Dahua Lin, Junyao Lin, Tianyi Lin, Zhishan Lin, Hongwei Liu, Jiangning Liu, Jiyao Liu, Junnan Liu, Kai Liu, Kaiwen Liu, Kuikun Liu, Shichun Liu, Shudong Liu, Wei Liu, Xinyao Liu, Yuhong Liu, Zhan Liu, Yinquan Lu, Haijun Lv, Hongxia Lv, Huijie Lv, Qitan Lv, Ying Lv, Chengqi Lyu, Chenglong Ma, Jianpeng Ma, Ren Ma, Runmin Ma, Runyuan Ma, Xinzhu Ma, Yichuan Ma, Zihan Ma, Sixuan Mi, Junzhi Ning, Wenchang Ning, Xinle Pang, Jiahui Peng, Runyu Peng, Yu Qiao, Jiantao Qiu, Xiaoye Qu, Yuan Qu, Yuchen Ren, Fukai Shang, Wenqi Shao, Junhao Shen, Shuaike Shen, Chunfeng Song, Demin Song, Diping Song, Chenlin Su, Weijie Su, Weigao Sun, Yu Sun, Qian Tan, Cheng Tang, Huanze Tang, Kexian Tang, Shixiang Tang, Jian Tong, Aoran Wang, Bin Wang, Dong Wang, Lintao Wang, Rui Wang, Weiyun Wang, Wenhai Wang, Jiaqi Wang, Yi Wang, Ziyi Wang, Ling-I Wu, Wen Wu, Yue Wu, Zijian Wu, Linchen Xiao, Shuhao Xing, Chao Xu, Huihui Xu, Jun Xu, Ruiliang Xu, Wanghan Xu, GanLin Yang, Yuming Yang, Haochen Ye, Jin Ye, Shenglong Ye, Jia Yu, Jiashuo Yu, Jing Yu, Fei Yuan, Yuhang Zang, Bo Zhang, Chao Zhang, Chen Zhang, Hongjie Zhang, Jin Zhang, Qiaosheng Zhang, Qiuyinzhe Zhang, Songyang Zhang, Taolin Zhang, Wenlong Zhang, Wenwei Zhang, Yechen Zhang, Ziyang Zhang, Haiteng Zhao, Qian Zhao, Xiangyu Zhao, Xiangyu Zhao, Bowen Zhou, Dongzhan Zhou, Peiheng Zhou, Yuhao Zhou, Yunhua Zhou, Dongsheng Zhu, Lin Zhu, Yicheng Zou

arXiv:2508.15763v2cs.LGcs.CLcs.CV

TL;DR

Scientific foundation models remain weaker in challenging, low-resource professional domains than in popular tasks. Intern-S1 addresses this gap with a multimodal MoE model, scientific-data curation, and multi-task reinforcement learning, reporting strong general reasoning and superior scientific performance across evaluated benchmarks.

  • Problem

    Open-source foundation models have advanced rapidly on popular tasks but still lag in high-value scientific domains where multimodal understanding and rigorous reasoning are required.

  • Method

    Intern-S1 combines multimodal MoE modeling with scientific pre-training, InternBootCamp reinforcement learning, and Mixture-of-Rewards across more than 1000 tasks.

  • Results

    Intern-S1 achieves top-tier general reasoning among open-source models and state-of-the-art or competitive performance across scientific benchmarks against open- and closed-source systems.

  • Takeaways & Limitations

    Intern-S1 provides an open-source generalist–specialist foundation intended for scientific research and diverse science-focused applications.

  • Takeaways & Limitations

    Dynamic tokenization currently supports four modalities, with broader modality support deferred to future iterations.

Abstract

from arXiv · show

In recent years, a plethora of open-source foundation models have emerged, achieving remarkable progress in some widely attended fields, with performance being quite close to that of closed-source models. However, in high-value but more challenging scientific professional fields, either the fields still rely on expert models, or the progress of general foundation models lags significantly compared to those in popular areas, far from sufficient for transforming scientific research and leaving substantial gap between open-source models and closed-source models in these scientific domains. To mitigate this gap and explore a step further toward Artificial General Intelligence (AGI), we introduce Intern-S1, a specialized generalist equipped with general understanding and reasoning capabilities with expertise to analyze multiple science modal data. Intern-S1 is a multimodal Mixture-of-Experts (MoE) model with 28 billion activated parameters and 241 billion total parameters, continually pre-trained on 5T tokens, including over 2.5T tokens from scientific domains. In the post-training stage, Intern-S1 undergoes offline and then online reinforcement learning (RL) in InternBootCamp, where we propose Mixture-of-Rewards (MoR) to synergize the RL training on more than 1000 tasks simultaneously. Through integrated innovations in algorithms, data, and training systems, Intern-S1 achieved top-tier performance in online RL training. On comprehensive evaluation benchmarks, Intern-S1 demonstrates competitive performance on general reasoning tasks among open-source models and significantly outperforms open-source models in scientific domains, surpassing closed-source state-of-the-art models in professional tasks, such as molecular synthesis planning, reaction condition prediction, predicting thermodynamic stabilities for crystals. Our models are available at https://huggingface.co/internlm/Intern-S1.

1 INTRODUCTION

Intern-S1 targets the persistent gap between strong open-source performance on popular tasks and weaker scientific reasoning by combining multimodal modeling, scientific data, and scalable training. It reports top-tier general reasoning and strong scientific performance across evaluated benchmarks.

  • Scientific research requires multimodal understanding of low-resource data and long-horizon reasoning for tasks such as hypothesis validation and experimental design.
  • Open-source models have approached closed-source performance in popular domains, but their scientific capabilities remain substantially less developed.
  • Intern-S1 processes images, text, molecular structures, and time-series signals to address complex scientific tasks.
  • The training strategy combines scientific-data curation with offline and online reinforcement learning over more than 1000 tasks using Mixture-of-Rewards.
  • Intern-S1 achieves state-of-the-art performance among contemporary open-source models, remains competitive with leading closed-source systems, and performs strongly across scientific and general-reasoning benchmarks.

2 MODEL ARCHITECTURE

Intern-S1 combines specialized encoders and a dynamic tokenizer with an MoE language model to represent heterogeneous scientific modalities efficiently. The tokenizer adapts segmentation and embeddings to scientific structures, improving SMILES compression by up to 70%.

  • 2 MODEL ARCHITECTURE: The architecture combines an MoE language model with vision, time-series, and dynamic-tokenization components for heterogeneous scientific inputs.
  • 2.1 VISION ENCODER: Intern-S1 uses InternViT-6B, while Intern-S1-mini uses the smaller InternViT-300M encoder for efficiency.
  • 2.2 DYNAMIC TOKENIZER: Static tokenization is inefficient for scientific strings and can force shared characters across modalities into representations dominated by frequent natural-language usage.
  • 2.2 DYNAMIC TOKENIZER: The dynamic tokenizer detects modalities, applies modality-specific splitting and embeddings, and concatenates the resulting sequences for standard transformer processing.
  • 2.2 DYNAMIC TOKENIZER: Up to 70%: dynamic tokenization improves compression ratio over OpenAI’s GPT-OSS-120B, Deepseek-R1, and Qwen3 series on SMILES data.
  • 2.3 TIME-SERIES ENCODER: The time-series encoder processes continuous numerical signals with adaptive downsampling and transformer blocks to represent long scientific sequences efficiently.

3 INFRASTRUCTURE

Intern-S1’s infrastructure supports large-scale multimodal training and reinforcement learning through distributed execution, reduced-precision computation, workload balancing, and colocated rollout and training.

  • 3 INFRASTRUCTURE: The infrastructure is organized around next-token training and reinforcement-learning paradigms, with the latter using distributed expert execution.
  • 3.1 TRAINING INFRASTRUCTURE: FSDP distributes model parameters across GPUs for continued pre-training and supervised fine-tuning, while one-way expert parallelism avoids inter-expert communication during RL.
  • 3.1 TRAINING INFRASTRUCTURE: FP8 computation is used for training and inference to reduce memory-bandwidth pressure and increase throughput.
  • 3.1 TRAINING INFRASTRUCTURE: Grouped GEMM, fused loss computation, Flash Attention, and variable-length balancing address MoE overhead and workload imbalance at scale.
  • 3.2 RL INFRASTRUCTURE: Training and inference share devices through lightweight redistribution, while rollout uses EP8, CPU off-loading, continuous batching, and dynamic slot rebalancing.

4 CONTINUE PRE-TRAINING

Intern-S1 uses a four-stage training procedure with distinct data preparation and training strategies.

  • 3 CONTINUE PRE-TRAINING: The training procedure consists of four distinct stages, each serving a separate purpose through specific data preparation and training strategies.

4.1 SCIENTIFIC DATA

The scientific-data pipeline combines domain-focused recall and filtering with hybrid PDF and web parsing to build high-quality text and multimodal training data. It uses specialized processing for scientific structure, costly cases, and domain-specific quality differences.

  • Scientific data composition: Over 2.5T of the 5T continued-pretraining tokens are scientific data, with distributions adjusted across six scientific domains.The domains are Mathematics, Physics, Chemistry, Life Science, Earth Science, and Materials Science; filtering strictness differs because their natural distributions vary greatly.
  • Page-level PDF documents parsing: PDF parsing uses a low-cost first pass, routing pages with equations or symbolic markers to high-cost VLM processing before specialized post-processing and sample merging.The pipeline was designed because scientific PDFs contain extensive knowledge and parsing failures concentrate in equation- and symbol-heavy documents.
  • Page-level PDF documents parsing: Quality control removes about 20% of archived-library PDF tokens and preserves 50% of web-crawled PDF tokens after garble detection, deduplication, and education-level scoring.The reported preservation ratios show that cleaner PDF sources still require substantial filtering.
  • Domain-centric web data parsing: Domain-centric web parsing samples hundreds of pages per URL domain, then discards, rewrites, or retains the domain using aggregated LLM classifications and page-level filters.Grouping pages by domain captures recurring structural issues while limiting the cost of applying an LLM-based agent to every page.
  • Scientific data recall and filtering: Scientific-data recall uses domain taxonomies, LLM annotations, lightweight classifiers, and in-domain/OOD validation sets to filter web data and recover target-domain material.A strong LLM labels a subset for training fastText models and 1.5B-parameter classifiers, while validation sets support prompt evolution.
  • Multimodal scientific data pipeline: The multimodal pipeline preserves figures, equations, symbols, tables, and charts, aligns them with surrounding text, and produces instruction- and exam-style supervision.Image-text CPT combines interleaved and purely textual datasets; the stated configuration contains approximately 250B training tokens, including 30B scientific tokens.

4.2 TRAINING STRATEGY

Intern-S1’s training strategy combines multi-stage multimodal continuation pre-training with schedule choices for batch size, learning rate, starting checkpoint, and joint parameter updates. These choices target both training efficiency and performance across modalities.

  • 4.2.1 BATCH SIZE WARMUP: A batch-size warmup switches from a smaller batch to a larger one, preserving early performance while improving later training efficiency.Small batches outperform large batches during the first 700B tokens, whereas large batches provide higher training efficiency; the proposed schedule follows this trade-off.
  • 4.2.2 STARTING POINT CHOICE: The instruction model is an acceptable CPT starting point after SFT and RL, and is preferable when post-training substantially improves capabilities, although this effect is domain-specific.The instruction model only shows an advantage on the coding benchmark in the reported starting-point experiments.
  • 4.2.3 HYPER PARAMETERS: Under WSD scheduling, gradient noise and batch size must satisfy a relationship for reaching a specified training quality, motivating a gradually increasing batch size.The training system transitions the batch size from 66M to 132M tokens after evaluating efficiency and effectiveness.
  • 4.2.3 HYPER PARAMETERS: Learning-rate selection is formulated as a constrained optimization over the loss-learning-rate relationship under the WSD scheduler.The constraints are represented by ϕ(Ω), and solving the optimization determines the learning rate for the full pre-training process.
  • 4.2 TRAINING STRATEGY: Text CPT was stable without loss spikes, and the final training loss was predicted to be around 1.16 from the fitted scaling relationship.The reported control relied on data quality and hyperparameter configuration within the Xtuner framework.
  • 4.2.4 MULTI-MODAL TRAINING: Intern-S1 uses integrated multimodal continuation pre-training with all model parameters updated jointly, while visual tokens condition text prediction rather than being predicted.The multimodal process interleaves image-text, video-text, and textual data, and applies a left-to-right autoregressive objective.

5 POST-TRAINING

After multimodal continued pre-training, Intern-S1 enters a two-stage multimodal post-training process that mixes text and multimodal data. The mixture is designed to optimize both text and multimodal benchmark performance.

  • 5 POST-TRAINING: Two-stage multimodal post-training mixes text and multimodal data to optimize performance on both text-only and multimodal benchmarks.This stage follows multimodal continued pre-training.

5.1 OFFLINE REINFORCEMENT LEARNING

Offline reinforcement learning begins with curated instruction data spanning text and multimodal scientific capabilities. The pipeline filters, labels, balances, and validates data mixtures to support broad benchmark performance.

  • 5.1 OFFLINE REINFORCEMENT LEARNING: Offline RL starts from curated instruction responses selected through best-of-N sampling against accuracy, fluency, and safety criteria.The curated data covers multiple domains and is followed by mixture experiments to select the training composition.
  • 5.1.1 INSTRUCTION DATA CURATION: The text-data pipeline filters harmful or low-quality samples, labels categories and scenarios, and supplements underrepresented domains with reconstructed or synthetic data.Filtering removes repetitive, truncated, or hallucinated content before labeling and stratified sampling.
  • 5.1.1 INSTRUCTION DATA CURATION: The resulting candidate dataset spans agent, code, dialogue, instruction following, mathematics, reasoning, long text, safety, chemistry, life sciences, and physics.Extensive ablations are used to identify the best mixtures across these categories.
  • 5.1.1 INSTRUCTION DATA CURATION: Multimodal instruction data combines specialized vision-language capabilities with stricter reasoning-data quality controls and additional scientific-domain examples.The controls include rejected sampling, deduplication, length and format constraints, self-consistency checks, and programmatic verification when available.
  • 5.1.1 INSTRUCTION DATA CURATION: Mixture selection validates individual data components first, then jointly evaluates merged datasets to address inter-domain conflicts and tune ratios and hyperparameters.The compositional phase begins with heuristic domain ratios and refines the training configuration through evaluation.
  • 5.1.1 INSTRUCTION DATA CURATION: A 32K-token context length, random JPEG-compression augmentation, and a squared-loss objective support long-range, document-level, and multi-image reasoning.These choices are adopted during multimodal instruction-data training.

5.2 ONLINE REINFORCEMENT LEARNING

Intern-S1’s online RL trains more than 1,000 tasks through unified reward integration, while addressing MoE-specific instability, data quality, and entropy-control challenges. The resulting filtering and entropy strategies improve training progress and maintain rising correctness across evaluations.

  • Mixture-of-Rewards: MoR unifies diverse reward signals across more than 1,000 multimodal tasks, enabling simultaneous acquisition of domain-specialized and general-purpose capabilities.InternBootCamp supplies the large-scale task environment, while rewards combine signals from multiple task types.
  • Reward and data construction: The RL pipeline combines verifiable rewards, generative and rule-based correctness verification, and POLAR-7B preference rewards for open-ended scenarios.These components cover multimodal reasoning with ground-truth answers, instruction following, and human-preference alignment.
  • Policy Optimization: Directly adapting GRPO-style algorithms to large-scale MoE models causes severe instability because dynamic expert routing and FP8 quantization amplify inference–training numerical discrepancies.OREAL avoids token-level old/new-policy ratio clipping and thereby inherently avoids the MoE training-collapse problem described for GRPO-style methods.
  • Entropy control: KL-Cov entropy control maintains exploration during MoE RL, using effect-token ratio k = 0.2 and KL coefficient β = 0.01 while entropy remains approximately 0.2 and correctness continues rising across evaluation sets.The coefficient is increased because the cold-started Intern-S1 MoE model has relatively low initial entropy.

6 EVALUATION

Intern-S1 is evaluated across general and scientific reasoning benchmarks in text-only and multimodal settings. It leads open-source models broadly, with especially strong scientific results, while retaining some gaps on instruction-following and visually intensive tasks.

  • General reasoning: Intern-S1 leads open-source models across general and scientific evaluations, including the best overall result on MathVista at 81.5.It is the top open-source multimodal model across all eight general-reasoning tasks.
  • Scientific reasoning: On science-focused text-only benchmarks, Intern-S1 is best on SmolInstruct (51.0), ChemBench (83.4), and MatBench (75.0).Its MatBench score exceeds InternVL3-78B by 25.7 and Qwen2.5-VL-72B by 23.5.
  • Scientific reasoning: On science-focused multimodal benchmarks, Intern-S1 is best on 4/5 datasets, including SFE (44.3), MicroVQA (63.9), MSEarthMCQ (65.7), and XLRS-Bench (55.0).It ranks second on Physics at 44.0.
  • Intern-S1-mini: Intern-S1-mini sets open-source state-of-the-art results on text-only benchmarks: 74.8 on MMLU-Pro, 65.2 on GPQA, and 80.0 on AIME2025.These scores improve over the strongest open-source baselines by 0.9, 3.2, and 12.7 points, respectively.
  • Intern-S1-mini: Intern-S1-mini achieves the best open-source score on MMMU at 72.3, while trailing leading models on MathVista, MathVision, and MMStar.The visual-reasoning gaps indicate remaining headroom on visually intensive mathematics.
  • Intern-S1-mini: Across 17 benchmarks, Intern-S1-mini achieves the top open-source score on 11, matching or exceeding Qwen3-8B while remaining competitive with contemporary VLMs.The evaluation covers both general-purpose and scientific reasoning tasks.

7 CONCLUSION

The report introduces Intern-S1 and Intern-S1-mini as scientific multimodal models with strong general and scientific reasoning. Intern-S1 combines extensive scientific pre-training with InternBootCamp and Mixture-of-Rewards reinforcement learning.

  • 7 CONCLUSION: Intern-S1 combines over 2.5T scientific pre-training tokens with InternBootCamp and Mixture-of-Rewards reinforcement learning.The reported RL process reduces training cost by 10X versus publicly available baselines.
Loading 2508.15763v2…