PaperlayerView exact revision on arXiv

Source-linked AI summary

Innovator-VL: A Multimodal Large Language Model for Scientific Discovery

Zichen Wen, Boxue Yang, Shuang Chen, Yaojie Zhang, Yuhang Han, Junlong Ke, Cong Wang, Yicheng Fu, Jiawang Zhao, Jiangchao Yao, Xi Fang, Zhen Wang, Henxing Cai, Lin Yao, Zhifeng Gao, Yanhui Hong, Nang Yuan, Yixuan Li, Guojiang Zhao, Haoyi Tao, Nan Wang, Han Lyu, Guolin Ke, Ning Liao, Xiaoxing Wang, Kai Chen, Zhiyu Li, Feiyu Xiong, Sihan Hu, Kun Chen, Yanfeng Wang, Weinan E, Linfeng Zhang, Linfeng Zhang

arXiv:2601.19325v1cs.CVcs.AI

TL;DR

Scientific multimodal models face demanding reasoning challenges, while many approaches rely on large datasets or opaque pipelines. Innovator-VL uses a transparent, reproducible training recipe and achieves a 61.83% average across 37 benchmarks, surpassing similarly sized models.

  • Problem

    Open-source multimodal models still struggle on scientific tasks requiring precise, multi-step, cross-modal reasoning, an important milestone toward AGI and scientific general intelligence.

    Source: §1 · p. 3

  • Method

    Innovator-VL presents a fully transparent, end-to-end reproducible training recipe for scientific multimodal modeling to facilitate further community exploration.

    Source: §1 · p. 4

  • Results

    61.83% average score across 37 diverse benchmarks, Innovator-VL-8B-Thinking surpassed similar-scale models, with competitive performance across scientific, general-vision, and multimodal-reasoning tasks using fewer than five million curated samples.

    Source: §1 · p. 4, §6.4 · p. 13

  • Takeaways & Limitations

    The reproducible training recipe is presented as a basis for further community exploration of scientific multimodal large language models.

    Source: §1 · p. 4

    The high-quality scientific corpora used for training are not yet publicly available because data-compliance review is ongoing.

    Source: §4.1 · p. 7

Abstract

from arXiv · show

We present Innovator-VL, a scientific multimodal large language model designed to advance understanding and reasoning across diverse scientific domains while maintaining excellent performance on general vision tasks. Contrary to the trend of relying on massive domain-specific pretraining and opaque pipelines, our work demonstrates that principled training design and transparent methodology can yield strong scientific intelligence with substantially reduced data requirements. (i) First, we provide a fully transparent, end-to-end reproducible training pipeline, covering data collection, cleaning, preprocessing, supervised fine-tuning, reinforcement learning, and evaluation, along with detailed optimization recipes. This facilitates systematic extension by the community. (ii) Second, Innovator-VL exhibits remarkable data efficiency, achieving competitive performance on various scientific tasks using fewer than five million curated samples without large-scale pretraining. These results highlight that effective reasoning can be achieved through principled data selection rather than indiscriminate scaling. (iii) Third, Innovator-VL demonstrates strong generalization, achieving competitive performance on general vision, multimodal reasoning, and scientific benchmarks. This indicates that scientific alignment can be integrated into a unified model without compromising general-purpose capabilities. Our practices suggest that efficient, reproducible, and high-performing scientific multimodal models can be built even without large-scale data, providing a practical foundation for future research.

1 Introduction

Innovator-VL pairs transparent, reproducible training with data-efficient scientific multimodal reasoning. It reports competitive performance on scientific and general multimodal benchmarks without large-scale scientific-domain pretraining.

1 Introduction

  • Scientific multimodal reasoning remains difficult because it demands precise understanding, multi-step reasoning, and cross-modal domain knowledge; prior approaches often require costly specialized data, opaque pipelines, or trade-offs in general performance.Existing open-source models also struggle to match their general-benchmark performance on scientific tasks.Source: §1 · pp. 3–4
  • RICE-ViT encodes fine-grained visual regions, PatchMerger compacts visual patches, and Qwen3-8B-Base serves as the language decoder.The structured visual units align entities with scientific concepts and reduce the burden on downstream language reasoning.Source: §1 · p. 4
  • The pipeline uses LLaVA image-language data for alignment, reasoning-focused supervised fine-tuning, and discrepancy-driven GSPO reinforcement learning with structural and semantic rewards.The RL dataset and hierarchical reward system target long-horizon reasoning paths, while the model receives no scientific-text or scientific multimodal pretraining.Source: §1 · p. 4
  • Innovator-VL provides a fully transparent, end-to-end reproducible training recipe intended to support community exploration of scientific multimodal models.Source: §1 · p. 4
  • Fewer than five million curated scientific samples support competitive performance across a wide range of scientific tasks without large-scale scientific-domain pretraining.The model uses human-in-the-loop synthetic data pipelines for specialized domains, including in-the-wild OCSR, chemical reaction understanding, and microstructural characterization.Source: §1 · p. 4
  • Innovator-VL achieves competitive performance on general vision, multimodal reasoning, and scientific benchmarks among models of comparable size, without compromising general-purpose capabilities.Source: §1 · p. 4

2 Model Architecture

Innovator-VL uses a three-stage vision encoder–projector–language model architecture. Its components are chosen to support region-aware scientific vision, efficient visual-token processing, and multimodal reasoning.

2 Model Architecture

  • Innovator-VL uses a three-stage vision encoder–projector–language model architecture to bridge visual and textual modalities.The paper describes this as a widely adopted paradigm.Source: §2 · p. 5

2.1 Vision Encoder

  • RICE-ViT [Xie et al., 2025] combines holistic and localized visual cues through a Region Transformer and jointly optimized object and OCR region representations.This region-aware design is intended for scientific imagery with fine-grained structures, dense labels, and spatially localized patterns.Source: §2.1 · p. 5

2.2 Projector

  • PatchMerger [Bai et al., 2025a, Wang et al., 2024a] compresses dense visual tokens to reduce downstream computational complexity and memory usage.The learned merger is described as enabling more efficient cross-modal interaction without compromising visual representation richness.Source: §2.2 · pp. 5–6

2.3 Language Model

  • Qwen3-8B-Base [Yang et al., 2025a] serves as the language backbone for scientific reasoning, multimodal question answering, and descriptive generation.Its stated suitability stems from STEM, logical reasoning, and long-context capabilities, alongside open-source availability and a mature tooling ecosystem.Source: §2.3 · p. 6

3 Pre-training

Innovator-VL’s pretraining has two stages: projector-based language-image alignment followed by full-parameter, high-quality mid-training. The design uses broad multimodal data while retaining the pretrained language model and emphasizing visual understanding.

3 Pre-training

  • Mid-training uses full-parameter training across all modules and approximately 85 million image-text pairs: 65 million English and 20 million Chinese samples.The corpus is LLaVA-OneVision-1.5-Mid-Training [An et al., 2025].Source: §3 · pp. 6–7
  • The authors skip continued scientific-text pretraining because high-quality scientific data are scarce, while further training risks bias and overfitting and could compromise the visual foundation for scientific reasoning.They cite Qwen3-8B-Base’s extensive pretraining and substantial absorbed scientific knowledge as reasons to retain it.Source: §3 · pp. 6–7
  • Language-image alignment trains the projector on LLaVA-1.5 558k [Liu et al., 2024b] to align visual features with the LLM word-embedding space.Source: §3 · pp. 6–7
  • Feature-based concept-balanced sampling uses MetaCLIP embeddings and a 500,000-concept taxonomy to generate top-K pseudo-captions, promoting semantic diversity despite missing or weak annotations.Images and concepts are projected into a shared embedding space to retrieve nearest concepts for refined pseudo-captions.Source: §3 · p. 7

4 Post-training

Innovator-VL uses staged supervised fine-tuning and reinforcement learning to improve instruction following, multi-step reasoning, and scientific multimodal reasoning. The post-training design combines general and scientific instruction data with targeted reasoning data and a specialized RL pipeline.

4.1 Supervised Fine-tuning

  • Full-parameter visual SFT develops general instruction following, preliminary chain-of-thought and multi-step reasoning, and scientific understanding; its scientific corpora are not yet publicly released.The SFT data comprise general multimodal instruction, reasoning, and scientific-understanding categories; release is pending data-compliance review.Source: §4.1 · pp. 7–8

4.1.1 General Multimodal Instruction Data

  • The general multimodal SFT set is LLaVA-OneVision-1.5-Instruct [An et al., 2025], with approximately 22 million samples spanning six broad visual-task categories.The passage lists Caption, Chart & Table, Code & Math, General VQA, Grounding & Counting, and OCR.Source: §4.1.1 · p. 8

4.1.2 Chain-of-Thought and Multi-step Reasoning Data

  • Honey-Data-15M [Zhang et al., 2025b] supplies approximately 15 million chain-of-thought and multi-step reasoning samples, with explicit think tags removed to reduce templated noise.The dataset is used to build reasoning patterns for complex scientific tasks and subsequent RL.Source: §4.1.2 · p. 8

4.1.3 Scientific Understanding Data

  • Scientific SFT data target generalization across domains, using dedicated pipelines for in-the-wild OCSR, literature-based reaction understanding, and EM characterization.OCSR uses E-SMILES and iterative expert correction; reaction QA pairs and hard negatives are expert-reviewed; EM data combine segmentation labels with structured descriptions and quality control.Source: §4.1.3 · pp. 8–9

4.2.1 RL Data

  • Innovator-VL-8B-Instruct receives RL training on Innovator-VL-RL-172K, selected to elicit learnable reasoning and standardized into a unified stepwise format.Selection uses the Pass@N–Pass@1 gap and reward-based filtering to retain medium-difficulty instances; the corpus is approximately 172K samples, led by STEM & Code (56.4%) and general multimodal tasks (34.9%).Source: §4.2 · p. 9, §4.2.1 · pp. 9–10

4.2 Reinforcement Learning

  • RL is applied to Innovator-VL-8B-Instruct to strengthen multimodal reasoning and its ability to solve complex scientific problems.This follows the broader staged post-training approach of supervised fine-tuning and reinforcement learning.Source: §4 · p. 7, §4.2 · p. 9

4.2.2 Optimization Algorithm

  • GSPO aligns importance sampling and clipping with sequence-level rewards, addressing token-level update mismatch for long-context reasoning.The paper says this sequence-level objective supports robust convergence and greater efficiency when scaling RL to long-context tasks.Source: §4.2.2 · p. 10

4.2.3 Reward System

  • The reward system combines strict output-format scoring with hierarchical correctness checks using rule-based matching, symbolic math verification, and an LLM judge for semantic cases.The cascade rewards deterministic answers cheaply and quickly while using model-based judging for complex semantic matches.Source: §4.2.3 · p. 11

5 Infrastructure

Innovator-VL uses optimized distributed training for pre-training and SFT, offline data packing to reduce padding, and asynchronous reinforcement learning to overlap trajectory generation with learning.

5 Infrastructure

  • Pre-training and SFT use AIAK-Training-LLM†, a Baidu Cloud Megatron-LM [Shoeybi et al., 2019] optimization, with Transformer Engine, distributed optimizer parallelism, and selective activation recomputation.These strategies target mixed-precision speed and lower memory overhead.Source: §5 · p. 11
  • Offline parallel bucket-based packing groups short samples into contiguous sequences, reducing padding and improving context-window use, training throughput, and computational efficiency.The preprocessing step is designed to improve GPU utilization.Source: §5 · pp. 11–12
  • For RL, AReaL [Fu et al., 2025] decouples rollout from learning so inference workers generate trajectories while learner updates run in parallel, supporting near-100% GPU utilization and faster convergence.The framework uses a non-blocking producer-consumer system with a replay buffer.Source: §5 · p. 12

6 Evaluation

Innovator-VL is evaluated against similarly scaled multimodal models on 37 benchmarks spanning general vision, math and reasoning, and science. Innovator-VL-8B-Thinking leads overall and in reasoning and science, while Innovator-VL-8B-Instruct performs on par with Qwen3-VL-8B in general multimodal evaluation.

6.1 Comparison Models

6.2 Benchmarks

  • The evaluation suite covers general vision, math and reasoning, and science benchmarks, including AI2D, MathVision, ScienceQA, and RxnBench.The paper describes the selected benchmarks as primarily multimodal and spanning diverse scenarios and capability aspects.Source: §6.2 · p. 12

6.3 Implementation Details

  • Evaluations use lmms-eval with deterministic decoding for both Innovator-VL-8B-Instruct and Innovator-VL-8B-Thinking (temperature 0.0, top-p 1.0).Additional evaluation details are provided in Appendix B.Source: §6.3 · p. 12

6.4 Performance Comparison

  • 61.83% is Innovator-VL-8B-Thinking’s state-of-the-art average across 37 benchmarks, surpassing all compared models of similar scale.The benchmarks span general, mathematical, and scientific domains.Source: Table 2 · p. 13, §6.4 · p. 13
  • 1.4× to 2× higher accuracy-to-token ratio than MiMo-VL-7B-RL and 3.9× to 4.3× higher than Intern-S1-mini is achieved by Innovator-VL-8B-Thinking.Figure 5 also shows significantly shorter reasoning chains than the other models.Source: Figure 5 · p. 14

6.4.1 General Multimodal Understanding

  • 74.50% average score places Innovator-VL-8B-Instruct on par with Qwen3-VL-8B (74.71%); it also achieves the best results on AI2D and RealWorldQA.It demonstrates superior scene understanding on MME-RealWorld.Source: Table 2 · p. 13, §6.4.1 · p. 14

6.4.2 Multimodal Reasoning

  • 55.41% average score makes Innovator-VL-8B-Thinking the top model in Math & Reasoning, a 4.54% absolute improvement over Innovator-VL-8B-Instruct.The result demonstrates the benefit of reinforcement learning for complex reasoning.Source: Table 2 · p. 13, §6.4.2 · p. 14

6.4.3 Scientific Knowledge

  • 50.13% and 49.79% are Innovator-VL’s top two Science-category average scores, while its OpenRxn and MolParse scores exceed 57% and 64%, respectively, versus baselines below 17%.The reported advantage is especially pronounced on specialized chemistry tasks.Source: Table 2 · p. 13, §6.4.3 · p. 14

7 On Token Efficiency of Reasoning

Innovator-VL-8B-Thinking uses fewer reasoning tokens while achieving higher accuracy per token; the paper attributes this efficiency to reinforcement learning that optimizes correctness and concision.

  • Innovator-VL-8B-Thinking uses 62%–66% fewer tokens than Intern-S1-mini and 18%–48% fewer than MiMo-VL-7B-RL across three benchmarks (564 vs. 1,640 average tokens on WeMath).Source: §7 · p. 14
  • Its accuracy-to-token ratio is 1.4×–2× higher than MiMo-VL-7B-RL and 3.9×–4.3× higher than Intern-S1-mini across benchmarks, meaning each token yields more useful information.Source: §7 · pp. 14–15
  • RL optimizes correctness and concision, teaching the model to focus on critical reasoning steps and avoid redundant computation rather than relying on costly test-time scaling.The paper presents this efficiency as valuable for scientific applications and practical in resource-constrained research environments [Chen et al., 2025] and latency-sensitive applications [Xiong et al., 2025, Yang et al., 2025b, He et al., 2025].Source: §7 · p. 15

8 Conclusion and Future Works

Innovator-VL combines a transparent, reproducible training pipeline with strong scientific and general-vision capabilities, demonstrating data-efficient scientific reasoning and compatibility between scientific alignment and general-purpose abilities. Planned work extends the model toward scientific workflows, additional modalities, lighter variants, and external tools and knowledge bases.

  • Innovator-VL achieves strong performance across scientific domains while maintaining excellent general-vision capabilities through a transparent, reproducible training pipeline.Source: §8 · p. 15
  • Fewer than five million carefully curated samples support competitive scientific reasoning, challenging reliance on massive domain-specific pretraining.Source: §8 · p. 15
  • Scientific alignment and general-purpose capabilities coexist in a unified framework without mutual degradation.Source: §8 · p. 15
  • Principled reinforcement learning elicits accurate, concise reasoning, with superior token efficiency compared to peer models.Source: §8 · p. 15
  • Future work targets automated experimental design and cross-modal hypothesis generation, video, 3D molecular structures and time-series, lightweight compressed variants, and integration with scientific tools and knowledge bases.Source: §8 · p. 15

A More Results · A.1 Qualitative Case Study

Innovator-VL’s curated qualitative cases illustrate performance beyond aggregate benchmark scores: it answers correctly where baseline models fail under the same evaluation setting, offering non-exhaustive evidence of robustness and generalization.

A.1 Qualitative Case Study

  • In representative evaluation cases, Innovator-VL answers correctly while baseline models fail under the same setting, providing non-exhaustive qualitative evidence of robustness and generalization.The cases include the original questions, associated images or figures when applicable, and full model responses.Source: §A.1 · p. 21

A.1.1 General Visual Tasks … Qwen3-VL-8B

The examples span general visual questions, image-based reasoning, and molecule transcription. They include different model answers to triangle-counting and sparrow-whistling tasks, alongside E-SMILES outputs and formatting guidance.

Case 1; Case 2; Case 3; Case 4

  • The visual examples ask about handbag and clock colors and direction, while the bike-rider example identifies a blue backpack.The supplied passages provide no answers to the handbag, clock, or direction questions.Source: Case 1 · p. 21, Case 2 · p. 23, Case 3 · p. 24

Case 1; Innovator-VL-8B-Thinking

  • For the pictured girl, Innovator-VL-8B-Thinking counts five triangles: two in the bow, one in the skirt, and two in the feet.Its analysis says the face has no triangles and identifies the bow, skirt, and feet as the counted shapes.Source: Innovator-VL-8B-Thinking · p. 26

MiniCPM-V4.5 (8B)

  • MiniCPM-V4.5 counts two triangles in the picture, identifying one in the bow and one in the dress.It rules out triangles in the face, limbs, belt, and hair.Source: MiniCPM-V4.5 (8B) · p. 27

InternVL3.5-8B

  • The described figure contains 10 triangles: two in the bow, one each in the dress top and skirt, and two each in the arms, legs, and feet.Source: InternVL3.5-8B · p. 27

Innovator-VL-8B-Thinking; Step 3: Calculate the Total Number of Whistles

Case 2; Qwen3-VL-8B

  • Qwen3-VL-8B totals eight whistles, interpreting the sparrows as alternating right- and left-facing and counting those each can see ahead.Its per-sparrow counts are 4, 1, 2, 1, and 0.Source: Qwen3-VL-8B · p. 29

MiMo-VL-7B-RL

  • MiMo-VL-7B-RL also totals eight whistles, based on its stated left- and right-facing assignments for the five sparrows.Its per-sparrow counts are 2, 3, 2, 1, and 0.Source: MiMo-VL-7B-RL · pp. 29–30

Case 1; SMILES String; E-SMILES; Intern-S1-mini (9B)

  • The molecule transcription gives an E-SMILES string with an empty extension after <sep>, because no additional annotations are needed.The supplied format guidance describes extensions for atom, ring, or abstract-circle annotations; Intern-S1-mini instead gives a different SMILES and proposes atom-level groups.Source: Case 1 · pp. 30–31, SMILES String · p. 31, E-SMILES · p. 32

Intern-S1-mini (9B) … MiMo-VL-7B-RL

The supplied passages combine a compound-structure identification, reaction-scheme descriptions, scientific image-question prompts, and model answers to a White Sea tidal question. They provide specific chemistry and model responses but no selected answers for several other prompts.

Intern-S1-mini (9B)

  • For the compound image labeled q2, the passage gives the ESMILES string COC(=O)C1=C(I)C=CC2=C1OC(C(=O)O)C=C2.The text identifies this string as representing the compound shown in the image.Source: Intern-S1-mini (9B) · p. 33

Case 3

  • Scheme 4 describes four 2f derivatizations: deborylation (3, 54%), Suzuki arylation (4, 74%), oxidation to ketone (5, 54%), and allylic alcohol formation (6, 40%).The descriptions differ on some conditions and stereoselectivity: product 5 is reported with either >99:1 Z:E or >99:1 E:Z, and product 6 with either H2O2 or O2 oxidation.Source: Case 3 · pp. 34–35

Case 4

  • The passage lists alternatives in which phenylacetylene (Ph–C≡CH) and N-oxide imine 1 react under Cu catalysis with Et3N or pyridine, yielding either a four-membered-ring or bridged-ring compound 3.Source: Case 4 · p. 36

Case 5; Case 6; Case 7

  • The passages pose prompts about particle-area uniformity, defining non-uniformity as max/min diameter mean > 30%, sulfur-atom count from an MS/MS spectrum using isotope pattern, and RNA structures from a secondary-structure diagram.Source: Case 5 · p. 37, Case 6 · p. 38, Case 7 · p. 39

Case 8; Case 9; Case 10

  • The supplied prompts ask for the galaxy’s structure from an image, the type of a multiband light curve, and the approximate regions with highest and lowest global surface-temperature anomalies.Source: Case 8 · p. 40, Case 9 · p. 41, Case 10 · p. 42

Case 11; Innovator-VL-8B-Instruct; Qwen3-VL-8B; MiMo-VL-7B-RL

B Evaluation Details … Generation

This section catalogs evaluation benchmarks across general multimodal, mathematical, and scientific tasks, and documents task-specific prompts and output formats. These include constrained answers, stepwise solutions, and molecular predictions and generations.

B.1 Benchmark Details.

  • The evaluation suite includes chart-structure reasoning, visual text recognition, and broad multimodal perception and domain-specific reasoning benchmarks.AI2D uses Diagram Parse Graphs to support chart-structure reasoning; OCRBench covers 29 text-related visual datasets, while MMMU spans six core disciplines.Source: §B.1 · p. 46
  • Visual mathematics benchmarks test fine-grained understanding and compositional reasoning, with MathVerse also varying modalities to assess visual-information use.MathVista aggregates 6,141 examples from 28 existing and three newly created datasets; MathVerse uses fine-grained chain-of-thought evaluation.Source: §B.1 · p. 47
  • Specialized chemistry benchmarks assess molecular parsing, reaction reasoning, and expert-solution-path fidelity.MolParse includes 2,000 question–answer pairs for molecular QA and structure parsing; OpenRxn includes 2,000 reaction-based pairs, while SuperChem has 500 expert-curated problems.Source: §B.1 · p. 48

B.2 Evaluation Prompts

  • Evaluation prompts constrain standard responses to a single word or option letter, while MathVision prompts request stepwise analysis and a separately tagged final answer.The MathVision template places analysis within <think> tags and the final answer within <answer> tags.Source: §B.2 · pp. 49, 51
  • Evaluation prompts constrain responses to task-specific formats, including letter-only multiple choice, short exact-match or open-ended answers, and JSON answers with explanations.Some multiple-choice prompts also request step-by-step reasoning or allow option content as an answer.Source: §B.2 · pp. 52, 54

ClinTox

  • ClinTox prompts ask for a Yes-or-No clinical toxicity prediction for the input molecule.The template requires a strictly binary response.Source: ClinTox · p. 55

Side Effect

  • Side-effect prompts require a Yes-or-No prediction, while aqueous-solubility and lipophilicity prompts request numerical logSol and logP values, respectively.The logSol prompt asks the model to consider molecular weight, rotatable bonds, and aromatic proportion; the logP prompt asks it to assess hydrophobic and hydrophilic balance.Source: Side Effect · p. 55

Forward Synthesis

  • Chemistry prompts cover forward synthesis, retrosynthesis, SMILES-to-IUPAC and IUPAC-to-SMILES conversion, and molecular-formula prediction from SMILES.Outputs are constrained to product or reactant SMILES, an IUPAC name, or a molecular formula, without extra explanation.Source: Forward Synthesis · pp. 55–56

IUPAC to Molecular Formula

  • The IUPAC-to-molecular-formula prompt requests only the formula corresponding to the provided IUPAC name.It explicitly excludes explanation or other text.Source: IUPAC to Molecular Formula · p. 56

Captioning

  • The captioning prompt asks for a detailed description of a molecule’s structure and chemical classification.Source: Captioning · p. 56

Generation

  • Generation prompts include producing a molecule’s SMILES from a description and answering land-use image questions with or without land-use classification context.The land-use formats request either one option or multiple correct options, as applicable.Source: Generation · pp. 56–57

Related papers

Loading 2601.19325v1…