PaperlayerView exact revision on arXiv

Source-linked AI summary

STEP3-VL-10B Technical Report

Ailin Huang, Chengyuan Yao, Chunrui Han, Fanqi Wan, Hangyu Guo, Haoran Lv, Hongyu Zhou, Jia Wang, Jian Zhou, Jianjian Sun, Jingcheng Hu, Kangheng Lin, Liang Zhao, Mitt Huang, Song Yuan, Wenwen Qu, Xiangfeng Wang, Yanlin Lai, Yingxiu Zhao, Yinmin Zhang, Yukang Shi, Yuyang Chen, Zejia Weng, Ziyang Meng, Ang Li, Aobo Kong, Bo Dong, Changyi Wan, David Wang, Di Qi, Dingming Li, En Yu, Guopeng Li, Haiquan Yin, Han Zhou, Hanshan Zhang, Haolong Yan, Hebin Zhou, Hongbo Peng, Jiaran Zhang, Jiashu Lv, Jiayi Fu, Jie Cheng, Jie Zhou, Jisheng Yin, Jingjing Xie, Jingwei Wu, Jun Zhang, Junfeng Liu, Kaijun Tan, Kaiwen Yan, Liangyu Chen, Lina Chen, Mingliang Li, Qian Zhao, Quan Sun, Shaoliang Pang, Shengjie Fan, Shijie Shang, Siyuan Zhang, Tianhao You, Wei Ji, Wuxun Xie, Xiaobo Yang, Xiaojie Hou, Xiaoran Jiao, Xiaoxiao Ren, Xiangwen Kong, Xin Huang, Xin Wu, Xing Chen, Xinran Wang, Xuelin Zhang, Yana Wei, Yang Li, Yanming Xu, Yeqing Shen, Yuang Peng, Yue Peng, Yu Zhou, Yusheng Li, Yuxiang Yang, Yuyang Zhang, Zhe Xie, Zhewei Huang, Zhenyi Lu, Zhimin Fan, Zihui Cheng, Daxin Jiang, Qi Han, Xiangyu Zhang, Yibo Zhu, Zheng Ge

arXiv:2601.09668v2cs.CV

TL;DR

Compact multimodal models have struggled to combine deployment efficiency with frontier-level reasoning and perception. STEP3-VL-10B addresses this through unified 1.2T-token pre-training, scaled reinforcement learning, and Parallel Coordinated Reasoning, achieving reported performance that rivals much larger models while remaining open-source. The paper also identifies missing perceptual reasoning traces and broader computational and physical-grounding bottlenecks as limitations and future challenges.

  • Problem

    Lightweight multimodal models under 10B parameters have traditionally struggled to advance sophisticated reasoning and perceptual capabilities within restricted parameter budgets.

  • Method

    STEP3-VL-10B combines fully unfrozen 1.2T-token multimodal pre-training with over 1k reinforcement-learning iterations and Parallel Coordinated Reasoning for scalable perceptual reasoning.

  • Results

    STEP3-VL-10B rivals or surpasses much larger open-weight and proprietary models across multimodal perception and reasoning benchmarks, including 94.43% on AIME2025.

  • Takeaways & Limitations

    The released model weights and training documentation provide a powerful, efficient, and reproducible open-source baseline.

  • Takeaways & Limitations

    The authors identify missing verbalized perceptual cognitive traces as a bottleneck for RL scaling, alongside unresolved computational-density and physical-grounding challenges.

Abstract

from arXiv · show

We present STEP3-VL-10B, a lightweight open-source foundation model designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. STEP3-VL-10B is realized through two strategic shifts: first, a unified, fully unfrozen pre-training strategy on 1.2T multimodal tokens that integrates a language-aligned Perception Encoder with a Qwen3-8B decoder to establish intrinsic vision-language synergy; and second, a scaled post-training pipeline featuring over 1k iterations of reinforcement learning. Crucially, we implement Parallel Coordinated Reasoning (PaCoRe) to scale test-time compute, allocating resources to scalable perceptual reasoning that explores and synthesizes diverse visual hypotheses. Consequently, despite its compact 10B footprint, STEP3-VL-10B rivals or surpasses models 10$\times$-20$\times$ larger (e.g., GLM-4.6V-106B, Qwen3-VL-235B) and top-tier proprietary flagships like Gemini 2.5 Pro and Seed-1.5-VL. Delivering best-in-class performance, it records 92.2% on MMBench and 80.11% on MMMU, while excelling in complex reasoning with 94.43% on AIME2025 and 75.95% on MathVision. We release the full model suite to provide the community with a powerful, efficient, and reproducible baseline.

1. Introduction

STEP3-VL-10B targets the gap between compact efficiency and frontier-level multimodal intelligence through unified pre-training, scaled reinforcement learning, and parallel perceptual reasoning. Despite its 10B footprint, it reports strong perception and reasoning results while competing with much larger models.

  • Lightweight models under 10B parameters have traditionally offered efficiency but struggled with sophisticated reasoning and perception under restricted budgets.
  • STEP3-VL-10B rivals or surpasses substantially larger open-weight and proprietary models while delivering strong visual perception, complex reasoning, and human-centric alignment.The cited larger comparisons include GLM-4.6V-106B, Qwen3-VL-Thinking-235B, Gemini-2.5-Pro, and Seed-1.5-VL.
  • 94.43% on AIME2025, 80.11% on MMMU, and 75.95% on MathVision demonstrate the model’s reported benchmark performance.
  • STEP3-VL-10B uses a single-stage, fully unfrozen strategy over a 1.2T-token multimodal corpus, jointly optimizing its Perception Encoder and Qwen3-8B decoder.The corpus targets reasoning and perception capabilities, including grounding, counting, OCR, and GUI interactions.
  • Parallel Coordinated Reasoning generates diverse perceptual hypotheses in parallel and synthesizes them through sequential cross-checking.This design externalizes implicit visual processes to scale perceptual reasoning at test time.
  • The authors release model weights and detailed training documentation as an open, reproducible baseline for the community.The paper frames this release as part of closing technical gaps in the open ecosystem.

2. Pre-train

The pre-training design combines a language-optimized visual encoder with Qwen3-8B and a broad multimodal corpus spanning knowledge, education, OCR, grounding, VQA, and GUI interaction. A fully unfrozen 1.2T-token training recipe prioritizes data quality and architectural synergy.

  • The model couples a 1.8B language-optimized Perception Encoder with Qwen3-8B through a projector that performs 16× spatial downsampling.The encoder is selected for pre-aligned linguistic features and the decoder for text generation and multimodal adaptation.
  • The multimodal pre-training dataset covers knowledge, education, OCR, grounding and counting, VQA, and GUI interaction domains.These domains are intended to support fine-grained perception, complex reasoning, and executable interface understanding.
  • The corpus includes 10M real-world images and 30M synthetic samples for OCR image-to-text training across diverse fonts, layouts, and orientations.
  • Approximately 400M grounding and counting samples support fine-grained perceptual understanding with box-, point-, and counting annotations.
  • Approximately 23M GUI samples span Android, iOS, Windows, Linux, and macOS across more than 200 applications.Grounding annotations are generated jointly with trajectory data for consistent action and perception supervision.
  • The model is trained in a single-stage, fully unfrozen procedure on 1.2T tokens over 370K iterations.Training uses a two-phase learning-rate schedule, with a higher-quality data mixture during the final 300B tokens.

3. Post-Train

Post-training combines multimodal supervised fine-tuning with PPO-based reinforcement learning, verifiable and preference-based rewards, and parallel coordinated reasoning. The pipeline progressively aligns text and visual reasoning while scaling sequential and parallel test-time compute.

  • The post-training pipeline applies PPO with GAE and scales inference compute from sequential reasoning to parallel coordinated reasoning.
  • Two-stage supervised fine-tuning shifts from a 9:1 text-to-multimodal mixture to a balanced 1:1 mixture for multimodal integration.Training uses a global batch size of 32 and sequence length of 128k.
  • The RL phase runs for 1,400 iterations while updating only the decoder and keeping the encoder frozen.
  • Verifiable rewards combine perception-based and model-assisted signals, using geometric correctness and reasoning consistency for tasks with ground truth.The framework covers mathematics, geometry, physics, scientific reasoning, perception, recognition, chart reasoning, puzzles, and grounding.
  • Non-verifiable tasks use pairwise generative preference modeling and heuristic constraints to guide human-centric alignment.GenRM evaluates reasoning before assigning a fine-grained scalar score to distinguish plausible responses.
  • Parallel coordinated reasoning explores diverse perceptual hypotheses in parallel and synthesizes them into a unified conclusion.The approach extends test-time compute beyond sequential generation.

4. Evaluations

STEP3-VL-10B is evaluated across more than 60 multimodal and text-centric benchmarks, spanning perception, reasoning, recognition, interaction, mathematics, code, and alignment. It establishes leading compact-model performance and remains competitive with substantially larger and proprietary systems, with PaCoRe enabling frontier-level results on several tasks.

  • Evaluation Setup: More than 60 benchmarks assess STEP3-VL-10B across multimodal and text-centric capability domains.The evaluation covers nine multimodal domains and six text-centric categories.
  • Compact-Model Comparison: STEP3-VL-10B ranks first in almost all capability domains among evaluated 7B–10B open-source models.Table 1 compares the model with representative compact open-source systems.
  • STEM and Multimodal Reasoning: 78.11%/64.08% on MMMU (Standard/Pro) and more than 10-point gains over MiMo-VL-RL-2508 and Qwen3-VL on MathVision demonstrate strong STEM and multimodal reasoning.The reported gains are attributed primarily to sufficient pre-training and scaled RL compute.
  • Recognition and General VQA: 92.05%/91.55% on MMBench (EN/CN) establishes the strongest recognition and general-VQA performance among models at the 10B scale.The authors attribute this result to large-scale, high-quality multimodal pre-training and scaling the 1.8B Perception Encoder.
  • GUI Grounding and Interaction: 92.61% on ScreenSpot-V2 and 59.02% on OSWorld-G lead GUI grounding and interaction evaluation.The approach trains on granular action trajectories, with RL and perception rewards further improving generalization in complex GUI environments.
  • Comparison with Larger Models: With PaCoRe, the model reaches 80.11% on MMMU, 75.95% on MathVision, 94.43% on AIME2025, and 92.14% on HMMT25, surpassing some frontier systems.PaCoRe synthesizes parallel reasoning rollouts and outperforms standard SeRe on several reasoning-heavy and perception-centric benchmarks.

5. Discussion

The discussion links architectural and optimization choices to sustained RLVR gains and emergent perceptual reasoning. It also identifies missing perceptual traces as a limitation that PaCoRe addresses through parallel hypothesis exploration and verification.

  • 5.1. Ablations and Design Insights: PE-lang achieves superior multimodal data efficiency and benchmark performance over DINOv3 because it is pre-aligned with language.DINOv3 excels in pure vision tasks but converges slowly in the multimodal setting due to the modality gap.
  • 5.1. Ablations and Design Insights: Muon improves selected tail-knowledge results, including +6.48% on SimpleVQA, but initialization mismatch and prolonged warmup reduce overall training efficiency versus AdamW.The authors therefore exclude Muon from the final architecture and defer deeper exploration.
  • 5.1. Ablations and Design Insights: Deepstack accelerates training convergence but does not produce meaningful downstream benchmark gains, so its computational overhead outweighs its marginal utility.The technique is excluded from the final model configuration.
  • 5.2. RL Dynamics, Performance, and Emergence: RLVR shows rapid reward and metric growth during the first 200 iterations followed by steady improvement, with reward approaching 0.8 without saturation.Average rollout length initially rises but eventually returns to its starting level, unlike monotonic sequential scaling in text-only RL.
  • 5.2. RL Dynamics, Performance, and Emergence: Reasoning tasks benefit from longer inference-time computation, whereas deterministic perception improves through entropy reduction and shorter, more confident rollouts.The discussion attributes this contrast to sequential scaling in reasoning and policy-refinement-driven length diminishment in perception.
  • 5.2. RL Dynamics, Performance, and Emergence: PaCoRe addresses missing verbalized perceptual traces by parallelizing visual hypotheses and sequentially cross-checking them, producing complex verification behavior during inference.Compared with vanilla SeRe, PaCoRe improves MathVision by +5.14%, DynaMath by +5.09%, CountQA by +4.6%, OCRBench by +2.25%, All-Angles-Bench by +7.50%, and SpatialViz-Bench by +6.52%.
  • 5.2. RL Dynamics, Performance, and Emergence: PaCoRe functions as a primitive multi-agent framework whose parallel proposals and sequential controller verification may later be internalized through self-distillation.The proposed direction aims to compress explicit coordinated reasoning into more efficient intrinsic responses.

6. Conclusion and Future Work

The conclusion presents STEP3-VL-10B as a compact model whose capabilities rival frontier systems while outlining unresolved bottlenecks in computational density and physical grounding. Future work therefore emphasizes broader RL scaling, reasoning-density optimization, and active physical-world modeling.

  • 6. Conclusion and Future Work: STEP3-VL-10B combines a 1.2T-token corpus with over 1k RL iterations to achieve perception, reasoning, and alignment capabilities that rival leading frontiers.The conclusion cautions that raw capability does not yet equal systemic maturity.
  • 6. Conclusion and Future Work: The roadmap identifies computational density and physical grounding as critical bottlenecks on the path toward comprehensive multimodal intelligence.These limitations motivate the proposed future-work directions.
  • 6. Conclusion and Future Work: The authors prioritize universal RL scaling across sequential depth and parallel width to uncover high-value perception and reasoning traces.This strategy shifts emphasis toward using compute directly to increase intelligence density.
  • 6. Conclusion and Future Work: Future reasoning-density work aims to compress parallel exploration into efficient sequentiality and eventually instinctive System 1-like responses.The stated goal is to retain benefits of parallel exploration while eliminating redundant over-thinking.
  • 6. Conclusion and Future Work: Bridging the reality gap requires moving beyond passive data consumption toward active physical grounding.The proposed direction extends semantic synthesis to video trajectories and sensorimotor action sequences for holistic physical world models.

7. Author List

This section lists the paper’s authors, core contributors, contributors, and sponsors, with authors ordered alphabetically by first name and project leaders marked by daggers.

  • 7. Author List: Authors are listed alphabetically by first name, and † identifies project leaders.The section separately enumerates core contributors, contributors, and sponsors.
  • 7. Author List: The core contributor list includes project leaders Jianjian Sun and Liang Zhao, each marked with †.The supplied author list identifies these individuals as core contributors and project leaders.
  • 7. Author List: The contributors section lists additional collaborators, while the sponsors section names Daxin Jiang, Qi Han, Xiangyu Zhang, Yibo Zhu, and Zheng Ge.These are presented as separate personnel categories in the author list.

A. More Results

The qualitative-results section illustrates STEP3-VL-10B’s fine-grained perception and structured reasoning, including sequential Morse-code decoding from a visual reference.

  • 7. More Results: Qualitative examples demonstrate fine-grained perception, structured logical reasoning, and multi-step integration of domain knowledge.The section presents these examples as evidence of cognitive systematicity.
  • Case 1: Visual Pattern Matching and Sequential Decoding: The Morse-code case converts each letter in “STEPFUN” into its corresponding code before concatenating the sequence.The displayed reasoning identifies the symbols for S, T, E, P, F, U, and N individually, then combines them in order.

Final Answer

The section presents several visual reasoning answers, including counting chapter files, decoding a monitor brand, and counting shortest graph edges, alongside PaCoRe input serialization.

  • Final Answer: 5 visible chapter .tex tabs remain after excluding the thesis.tex tab.The count is based on five truncated chapter tabs and one non-chapter thesis tab.
  • Final Answer: CK is the result of applying a Caesar cipher with a left shift of 5 to HP.H shifts to C and P shifts to K.
  • Final Answer: 4 edges have the shortest length because four listed edges carry weight 1.The edges are A→T, T→X, P→O, and W→X.
  • Final Answer: PaCoRe serializes the original problem and compact reference responses for synthesis, bypassing the template when the message set is empty.The template represents the problem as original_prompt and the message set as ref_responses.

C. Evaluation Details

This section introduces the evaluation setup and its corresponding prompts for the paper's benchmarks.

  • C. Evaluation Details: The evaluation setup includes benchmark-specific evaluation prompts.The section serves as an overview of the evaluation details developed in the following material.
  • C. Evaluation Details: The evaluation prompts are presented alongside the corresponding benchmark procedures.The stated purpose is to describe both the setup and the prompts used for evaluation.
  • C. Evaluation Details: The section connects benchmark evaluation with the prompt formats used to obtain model responses.Its scope covers evaluation setup and prompt specification.

C.1. Evaluation Details for Multimodal Benchmarks

The multimodal evaluation uses benchmark-specific official metrics or protocols and prompt formats tailored to answer types, visual inputs, and task domains.

  • C.1. Evaluation Details for Multimodal Benchmarks: Prompt templates use {question} for the textual problem and <image> for visual input, preserving original image positions when explicit positional semantics are present.Otherwise, images are placed before the question text.
  • C.1. Evaluation Details for Multimodal Benchmarks: Benchmark evaluation follows official metrics or protocols across MathVista, DynaMath, MathVerse, We-Math, MMBench, SimpleVQA, and other multimodal tasks.The passages repeatedly specify adoption of official metrics, protocols, or judgement procedures.
  • C.1. Evaluation Details for Multimodal Benchmarks: MathVista prompts distinguish floating-point, multiple-choice, integer, and Python-list answer formats.The templates request the final answer at the end in the corresponding format.
  • C.1. Evaluation Details for Multimodal Benchmarks: Several benchmarks constrain final responses to option letters or boxed answers, while task-specific prompts request integers, identified letters, or option numbers.Examples include LogicVista, MathVerse, VisuLogic, MMMU-Pro, and MathVision.
  • C.1. Evaluation Details for Multimodal Benchmarks: DynaMath reports worstcase accuracy, defined as the percentage of correctly answered seed questions across generated variations.Its prompts separately handle multiple-choice, floating-point, and other answers, with floating-point values rounded to three decimal places.

C.2. Evaluation Details for Text-Centric Benchmarks

Text-centric benchmarks use repeated evaluation on selected tasks, averaging independently evaluated runs to reduce metric variance and improve reliability.

  • C.2. Evaluation Details for Text-Centric Benchmarks: Selected text-centric benchmarks are evaluated repeatedly, with each sample run independently according to its benchmark-specific Repeat value.The repetition settings are listed for each selected benchmark.
  • C.2. Evaluation Details for Text-Centric Benchmarks: The final score is the average across all runs when a benchmark has Repeat = N.This procedure is intended to reduce metric variance and improve result reliability.
  • C.2. Evaluation Details for Text-Centric Benchmarks: AIME 2024, AIME 2025, HMMT25, CNMO2024, and BeyondAIME each use Repeat = 64.GPQA-Diamond and LiveCodeBench use Repeat = 16, while IFEval and IFBench use Repeat = 4.

C.3. Evaluation Details for Ablations

The ablations use controlled pre-trained checkpoints with matched token counts and few-shot evaluation, rather than the final checkpoint.

  • Ablation checkpoints are pre-trained with the same number of billions of tokens for fair, controlled comparisons.
  • The ablation results do not extend to the final checkpoint because of computational cost.
  • Evaluation uses a few-shot setup on the pre-trained checkpoints.
Loading 2601.09668v2…