Source-linked AI summary
SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning
Kun Xiang, Terry Jingchen Zhang, Zirong Liu, Bokai Zhou, Yueling Tang, Junjie Yu, Jiacong Lu, Shangrui Huang, Heng Li, Likui Zhang, Kunkun Liu, Changzheng Zhang, Yangle Fang, Boqiang Guo, Hui-Ling Zhen, Dandan Tu, Yinya Huang, Xiaodan Liang
TL;DR
The paper asks whether multimodal models preserve physical reasoning when critical information moves from text into images, and whether RL gains reflect valid visual grounding. It introduces aligned modality-transfer variants and blind-training diagnostics, finding persistent visual fragility and non-visual sources of improvement. The results motivate evaluating both representation robustness and dependence on task-critical visual evidence.
Problem
Existing evaluations provide limited evidence about whether multimodal models preserve reasoning across equivalent representations or whether accuracy gains rely on valid visual evidence.
Method
SeePhys Pro uses four semantically aligned physics variants with progressive modality transfer, while PhysRL corpora and blind RL test training-time grounding.
Results
Performance decreases on average as information moves from language to diagrams, with visual variable grounding the largest bottleneck; blind RL can still improve unmasked validation accuracy.
Takeaways & Limitations
Multimodal reasoning evaluation should combine modality-transfer robustness with diagnostics that test dependence on task-critical visual evidence.
Takeaways & Limitations
The benchmark targets controlled diagnosis rather than maximum physics difficulty, and the paper does not validate a complete mitigation strategy.
Abstract
from arXiv · showhide
We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical information is progressively transferred from text to image. Unlike standard vision-essential benchmarks that evaluate a single input form, SeePhys Pro features four semantically aligned variants for each problem with progressively increasing visual elements. Our evaluation shows that current frontier models are far from representation-invariant reasoners: performance degrades on average as information moves from language to diagrams, with visual variable grounding as the most critical bottleneck. Motivated by this inference-time fragility, we further develop large training corpora for multimodal RLVR and use blind training as a diagnostic control, finding that RL with all training images masked can still improve performance on unmasked validation sets. To analyze this effect, text-deletion, image-mask-rate, and format-saturation controls suggest that such gains can arise from residual textual and distributional cues rather than valid visual evidence. Our results highlight the need to evaluate multimodal reasoning not only by final-answer accuracy, but also by robustness under modality transfer and by diagnostics that test whether improvements rely on task-critical visual evidence.
1 Introduction
SeePhys Pro studies whether multimodal models preserve physical reasoning when equivalent information shifts progressively from language to vision. Its benchmark and training-time controls reveal fragility under visual transfer and show that accuracy gains may not reflect valid visual grounding.
- SeePhys Pro provides four aligned variants that move task-critical information from text to structure, variables, and fully rendered images.The benchmark decomposes degradation into structural transfer, variable grounding, and full-rendering effects.
- Across evaluated MLLMs, accuracy generally drops as information transfers from language to vision, with visual variable grounding often producing the largest degradation.
- Blind RL with all training images masked can still improve accuracy on unmasked validation sets.Text-deletion, mask-rate, and format-saturation controls suggest residual language, templates, and dataset statistics may drive these gains.
- The paper argues that multimodal reasoning should be evaluated through modality robustness and grounding diagnostics, not final-answer accuracy alone.
2 Related Work
Prior multimodal benchmarks establish that scientific and visual reasoning remain difficult, but most evaluate fixed input forms. SeePhys Pro instead controls representation while preserving the underlying physics and uses blind training to test whether RL gains depend on visual evidence.
- Existing science and physics benchmarks show persistent difficulty in multimodal problem solving, while typically evaluating each problem in a fixed input form.
- Vision-grounding benchmarks expose modality gaps, but SeePhys Pro further separates structural grounding, variable grounding, and full-rendering effects in physics diagrams.Physics diagrams may define topology and variable-to-entity bindings rather than merely illustrate accompanying text.
- Outcome-only multimodal RLVR rewards can encourage shortcuts that do not depend on valid visual evidence.
- Blind training masks all training images as a negative control; continued gains on unmasked tests are therefore not fully attributable to improved visual grounding.
3 SEEPHYS PRO: A Fine-Grained Benchmark for Modality-Transfer
SeePhys Pro constructs controlled, semantically aligned physics variants and annotates them across domains, visual evidence types, and reasoning skills. Its metrics quantify accuracy, transfer costs, and consistency across representations.
- Four-level modality transfer: Each seed problem is converted into four aligned variants with unchanged physical semantics, queried quantity, variables, constraints, solution path, and answer.Only the carrier of information changes across levels, from text-only to increasingly visual representations.
- Benchmark data collection: The benchmark is built from heterogeneous physics sources, including textbooks, examinations, olympiad archives, and public datasets, using OCR and expert curation.The source pool contains over 5,000 PDF pages and is processed by engineering-trained annotators.
- Taxonomy and metadata: SEEPHYS PRO categorizes problems by physics domain, visual information type, and reasoning skill, including structure, variable labels, directions, graphs, and geometric relations.
- Diagnostic metrics: The benchmark reports Level-ℓ accuracy and signed modality-transfer gaps for structure, variable grounding, rendering, and total transfer.Positive gaps indicate degradation under a more visual representation, while negative gaps indicate higher accuracy on the visually richer variant.
- Diagnostic metrics: Four-way representation consistency measures the percentage of seed problems answered correctly across all four aligned representations.This separates absolute problem-solving ability from robustness to modality transfer.
4 Test-Time Modality Transfer
SEEPHYS PRO tests whether MLLMs preserve performance when identical physics problems become progressively more visual, using controlled four-level variants and aligned evaluation. Across models, accuracy falls substantially toward fully visual inputs, with variable grounding producing the largest staged degradation.
- Evaluation setup: The evaluation covers 10 closed-weight and 5 open-weight multimodal language models.The model set includes both frontier closed-weight systems and open-weight systems.
- Evaluation setup: The benchmark evaluates the same seed problems across four controlled modality-transfer levels, enabling direct measurement of representation sensitivity.Each level preserves the underlying problem while changing the information carrier; results are reported with accuracy, consistency, and signed transfer gaps.
- Evaluation setup: The test split contains 800 examples per level, with a 200-example testmini subset used for selected models under API-budget constraints.The standard reported results use test unless otherwise stated; marked models use testmini.
- Main results: 49.2% at Level 1 fell to 35.8% at Level 4, producing a 13.4-point average total modality-transfer gap.This average decline spans the evaluated models and measures the cost of moving task-critical information from language toward vision.
- Main results: ΔV = 7.4 points was the largest model-average staged gap, compared with ΔS = 3.0 for structural transfer and ΔR = 2.9 for final rendering.The result identifies visual variable grounding as the dominant bottleneck, including reading quantities and binding them to physical entities.
- Main results: Marginal accuracy can overestimate cross-representation stability: Claude-4.7-Opus reached 33.5% four-way consistency, while GPT-5.4 reached 32.6%.The total gap mixes underlying physics-solving ability with the ability to preserve solutions after information moves into vision.
5 Training-Time Diagnostic: Can RL Help Close the Modality Gap?
The paper evaluates whether RL closes modality-transfer gaps using accuracy gains and gap dynamics, comparing normal training with a blind-image control. Both regimes improve accuracy, but the reported evidence indicates gains do not reliably reflect improved visual grounding or gap closure.
- Diagnostic setup: The study evaluates RL by both validation-accuracy gains and changes in SEEPHYS PRO modality-transfer gaps.The diagnostic asks whether visually grounded training preferentially improves performance on more visual levels and reduces transfer failures.
- Diagnostic setup: PhysRL-38K and PhysRL-8K are source-matched, test-disjoint physics RL training corpora used for multimodal RLVR experiments.PhysRL-38K contains approximately 38K examples, while PhysRL-8K retains approximately 8K vision-necessary examples.
- Diagnostic setup: Normal RL preserves original training images, whereas blind RL replaces all training images with black images under matched splits, rewards, and other settings.The blind control makes training instances visually unsolvable while holding the main training conditions constant.
- Accuracy and gap dynamics: Both normal and blind RL improve all four SEEPHYS PRO levels for Qwen3-VL-4B.Level-1 accuracy rises from 9.9% to 18.3% under normal RL and 20.9% under blind RL; Level-4 rises from 6.4% to 10.8% and 13.0%, respectively.
- Accuracy and gap dynamics: The total transfer gap widens from 3.5 percentage points before training to 7.5 after normal RL and 7.9 after blind RL.The variable-grounding gap also lacks stable closure because normal and blind curves remain close and cross during training.
- Cross-benchmark controls: Blind RL also improves external math and physics benchmarks, indicating that accuracy gains can arise from non-visual adaptation rather than valid visual evidence.The strongest pattern appears on math benchmarks, while normal RL is clearly stronger on some visually demanding physics benchmarks such as PhyX and OlympiadBench.
6 Mechanism Analysis: What Does Blind RL Learn?
Blind RL can improve unmasked validation accuracy without valid visual evidence, primarily by exploiting residual language and distributed dataset regularities. Auxiliary controls weaken all-black-image and format-only explanations, framing the effect as non-visual gains from outcome-only rewards.
- Blind-training gains can arise from exploitable text, options, formula templates, answer ranges, and task-style regularities even when individual problems require images.The distinction is between instance-level vision necessity and distribution-level learnability.
- Complete text deletion reduces blind gains on PhysReason and SeePhys Pro to near zero, while gains on MathVerse and MMK12 fall sharply.For Qwen3-VL-4B, MathVerse drops from about 26.6 points at 25% deletion to nearly zero at 100%; MMK12 drops from about 19.0 points to below one point.
- Targeted deletion shows that physics shortcuts are distributed across weak textual cues and dataset regularities rather than explained by one span.Deleting numbers, options, or formulas can reduce gains, but gains persist under most single-category deletions.
- Mask-rate gains are early, positive, and non-monotonic, weakening the all-black-image-artifact explanation.The control varies training-image masking from 10% to 90%.
- After format reward reaches 90%, Qwen2.5-VL-7B still gains 21.2/19.1 points on MMK12 and 9.6/3.9 points on MathVerse under normal/blind RL.This weakens a purely format-compliance explanation for blind-training gains.
- SEEPHYS PRO diagnoses modality-transfer robustness but does not propose or validate a complete mitigation strategy for inconsistency or non-visual blind-RL gains.The paper identifies counterfactual pairs, black-image unanswerability, and process-level rewards as promising directions.
7 Conclusion
The paper concludes that SEEPHYS PRO exposes fragile modality transfer and that blind RL can improve accuracy without valid visual information. It therefore supports evaluating multimodal reasoning with both modality robustness and training-time grounding diagnostics, not final-answer accuracy alone.
- SEEPHYS PRO uses four aligned variants to diagnose reasoning as information moves from text to visual structure, visual variables, and fully rendered diagrams.
- Visual variable grounding emerges as a key bottleneck, while blind training can improve unmasked validation accuracy despite training images containing no valid visual information.
- The paper motivates combining inference-time modality robustness with training-time grounding diagnostics instead of relying only on final-answer accuracy.
A.3 Benchmark Embeddings
The embedding visualization offers a qualitative distributional snapshot: SEEPHYS PRO text-only and multimodal inputs appear more separated at Level 3 than Level 2, while MathVerse subsets appear broadly comparable.
- Figure 7 compares embeddings for SEEPHYS PRO Levels 2 and 3 with embeddings for MathVerse Vision Intensive and Vision Dominant subsets.Panels separate text and multimodal inputs for each benchmark setting.
- SEEPHYS PRO text-only and multimodal inputs appear more separated at Level 3 than at Level 2, whereas the MathVerse subsets appear broadly comparable.The figure is an illustrative distributional snapshot rather than a quantitative metric.
B.1 Physics Training Pool Validation
PhysRL-38K validation shows transferable physics reasoning signal: Qwen3-VL-4B improves on held-out SeePhys Pro and PhysReason validation sets while training reward accuracy also rises.
- PhysRL-38K is source-matched, test-disjoint, and larger than the derived PhysRL-8K pool.
- The cross-benchmark improvements indicate transferable physics reasoning signal rather than only SEEPHYS PRO-specific artifacts.
- PhysRL-38K training improves validation accuracy on SeePhys Pro Level 3/4 and PhysReason while training reward accuracy also increases.The validation uses held-out, unmasked sets and two sequence-length settings.
- Figure 9 reports post-format-saturation gains, with answer accuracy still increasing after format reward crosses 90%.The effect is especially highlighted in Qwen2.5-VL-7B math runs.
- Figure 10 plots accuracy and format reward across image mask rates for Qwen3-VL-4B on unmasked SeePhys Pro and PhysReason.Figure 5 summarizes peak gains within the first 200 updates.
- Figure 11 provides expanded separate cross-benchmark peak-gain plots corresponding to Figure 4.
C Error-Type Clustering
Across modality-transfer levels, frontier-model errors shift from primarily physics modeling and reasoning failures toward more structural and numerical figure-reading failures in visually richer settings.
- Text-level failures are dominated by physics modeling and reasoning errors.
- Visually richer settings introduce more structural figure-reading and numerical figure-reading errors.
- Figure 12 compares manually annotated error distributions for GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.7 across Levels 1–4.The center value n gives the number of analyzed errors for each model-level cell.
D Case Study Examples
The case studies illustrate errors involving visual geometry, motion constraints, numerical reading, equilibrium reasoning, reaction forces, and induced-charge calculations, alongside correctly grounded solutions.
- Visual geometry grounding: Models can misread geometric cues such as angles, radii, and arc marks.
- Motion and constraints: The tension-force case study reports an answer of 26.1 N for a Level 1 problem.
- Motion and constraints: Motion examples require modeling vertically constrained objects in the cart’s non-inertial frame, with friction forces canceling in the stated equations.
- Constraint and equilibrium: Constraint and equilibrium cases include oversimplified constraints, false equilibrium assumptions, and incorrect motion assumptions.
- Numerical reading: Models may misread numerical values in visual inputs, propagating errors into frequency and answer-option judgments.
- Induced charge: The induced-charge example computes q_C = -2.0 × 10^-7 C from the charge share across parallel capacitors.