PaperlayerView exact revision on arXiv

Source-linked AI summary

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

Kun Xiang, Terry Jingchen Zhang, Zirong Liu, Bokai Zhou, Yueling Tang, Junjie Yu, Jiacong Lu, Shangrui Huang, Heng Li, Likui Zhang, Kunkun Liu, Changzheng Zhang, Yangle Fang, Boqiang Guo, Hui-Ling Zhen, Dandan Tu, Yinya Huang, Xiaodan Liang

arXiv:2605.09266v2cs.AI

TL;DR

Single-format physics benchmarks offer limited evidence about whether models preserve reasoning as equivalent information shifts from text to diagrams. SeePhys Pro probes this transfer with aligned variants and blind-training controls; models remain fragile under modality transfer, while blind RL can improve unmasked validation accuracy despite masked training images.

  • Problem

    Single-format benchmarks provide limited evidence about whether models preserve reasoning when equivalent physics information shifts from text to diagrams, which can define the physical system.

    Source: §1 · p. 1

  • Method

    The paper builds four aligned modality-transfer variants per physics problem and uses multimodal RLVR corpora with all-image-masked blind training as a diagnostic control.

    Source: §1 · p. 2

  • Results

    Models remain fragile under modality transfer, especially when variables must be grounded from images, while blind RL can improve accuracy on unmasked validation sets despite masked training images.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The findings motivate evaluating multimodal reasoning for modality robustness and training-time grounding diagnostics, rather than relying only on final-answer accuracy.

    Source: §7 · p. 10

    The paper does not propose or validate a complete mitigation strategy for modality-transfer inconsistency or non-visual gains under blind RL.

    Source: §6 · p. 9

Abstract

from arXiv · show

We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical information is progressively transferred from text to image. Unlike standard vision-essential benchmarks that evaluate a single input form, SeePhys Pro features four semantically aligned variants for each problem with progressively increasing visual elements. Our evaluation shows that current frontier models are far from representation-invariant reasoners: performance degrades on average as information moves from language to diagrams, with visual variable grounding as the most critical bottleneck. Motivated by this inference-time fragility, we further develop large training corpora for multimodal RLVR and use blind training as a diagnostic control, finding that RL with all training images masked can still improve performance on unmasked validation sets. To analyze this effect, text-deletion, image-mask-rate, and format-saturation controls suggest that such gains can arise from residual textual and distributional cues rather than valid visual evidence. Our results highlight the need to evaluate multimodal reasoning not only by final-answer accuracy, but also by robustness under modality transfer and by diagnostics that test whether improvements rely on task-critical visual evidence.

1 Introduction

SEEPHYS PRO evaluates whether physics reasoning remains consistent as task-critical information shifts from text to images, and tests whether blind-training gains reflect visual evidence or nonvisual cues.

1 Introduction

  • Modality consistency asks whether equivalent information preserves the same reasoning behavior across forms; physics is a sharp test because diagrams can define the system itself.As information shifts into vision, models must ground and bind labeled variables to entities and relations, not merely perceive generic visual content.Source: §1 · p. 1
  • Across MLLMs, SEEPHYS PRO’s four aligned variants show performance dropping as task-critical information shifts from language to vision, often most when variables require image grounding.The variants progress from text-only to structure-in-image, structure-plus-variables-in-image, and fully rendered problem images.Source: §1 · p. 2
  • Text-deletion and mask-rate controls suggest blind-training gains arise from residual language, problem templates, and dataset-level regularities rather than effective visual evidence.These findings motivate evaluating whether accuracy gains depend on task-critical visual evidence or textual shortcuts.Source: §1 · p. 2
  • A blind-training control that masks all training images still improves accuracy on unmasked validation sets, but these gains do not reliably close modality-transfer gaps or establish visually grounded learning.Source: §1 · p. 2
  • SEEPHYS PRO metrics decompose performance into structure recognition, variable grounding, modality gap, and representation consistency.Source: §1 · p. 2

2 Related Work

This section situates SeePhys Pro against fixed-form multimodal physics benchmarks and visual-grounding tests, then motivates blind training as a diagnostic of whether RLVR gains depend on valid visual evidence.

Multimodal physics reasoning

  • ScienceQA [13], MMMU [33], OlympiadBench [8], SeePhys [28], PhyX [23], PhysReason [36], PhysicsArena [4], and QuantiPhy [21] typically evaluate problems in fixed input forms.These benchmarks show that multimodal scientific problem solving and diagram-based physical reasoning remain difficult.Source: §2 · p. 2
  • SeePhys Pro holds the underlying physics fixed while progressively transferring structure, variables, and the full problem statement from text into vision.Its four aligned levels range from text-only to a fully rendered visual input.Source: §2 · p. 2, Figure 1 · p. 3

Vision grounding ability in reasoning

  • Unlike MathVista [14], MathVerse, and CrossMath [29], SeePhys Pro tests diagrams that define the physical system, separating structural grounding, variable grounding, and full-rendering effects.The diagrams can specify system topology and bindings between variables and physical entities.Source: §2 · p. 2

Reinforcement learning with verifiable rewards

  • RLVR post-trains reasoning models [22] [32] [37] and has been explored for multimodal tasks [26] [25] [11] [17] [31], but outcome-only rewards may encourage shortcuts unrelated to valid visual evidence.Prior analyses suggest reasoning-style training can amplify ungrounded behavior or improve behaviors already latent in the base model [27] [10] [34].Source: §2 · p. 2
  • Blind training masks all training images; if RL still improves unmasked test performance, the gain is not fully attributable to better visual grounding.Source: §2 · p. 3

3 SEEPHYS PRO: A Fine-Grained Benchmark for Modality-Transfer

SeePhys Pro turns each physics problem into four aligned representations that progressively shift critical information from text to images while preserving its physical meaning and answer. Its annotations and transfer metrics are designed to distinguish failures in visual perception and grounding from downstream physics reasoning, and to measure robustness across representations.

Four-level modality transfer

  • Each seed becomes four semantically aligned variants, making SeePhys Pro a controlled probe of reasoning under modality transfer rather than a set of different questions.The physical system, queried quantity, variables, constraints, solution path, and gold answer stay fixed as information moves from text to visual structure, diagram labels, and finally a full handwritten image.Source: §3 · pp. 3–4

Benchmark data collection

  • The benchmark’s seed problems come from diverse physics sources and are curated, checked, and transformed into aligned four-level groups.Sources include public datasets, textbooks, qualifying and entrance examinations, olympiad archives, and school or university exams; 10 engineering-trained annotators curate the problems and redraw diagrams with separate structure and variable layers.Source: §3 · pp. 3–4

Taxonomy and metadata

  • SeePhys Pro annotates physics domains, visual information types, and reasoning skills to support analysis of structural perception, variable grounding, and downstream reasoning failures.Visual categories include structure/topology, variable labels, directions and vectors, graphs, geometric relations, and symbolic diagrams; reasoning metadata includes skills such as conservation-law reasoning and circuit reduction.Source: §3 · p. 4

Diagnostic metrics

  • Transfer gaps quantify accuracy changes between adjacent representation levels and from Level 1 to Level 4, while four-way consistency measures correctness across all four variants.ΔS = A1 − A2, ΔV = A2 − A3, ΔR = A3 − A4, and ΔT = A1 − A4; positive gaps indicate degradation under a more visual representation, while negative gaps indicate better accuracy on the visually richer variant.Source: §3 · p. 4

4 Test-Time Modality Transfer

Across controlled modality-transfer levels, model accuracy falls as information moves from text into images, with visual variable grounding the largest source of degradation. Even models’ strong marginal accuracy does not ensure stable performance across representations.

Main Results

  • Average accuracy falls from 49.2% at Level 1 to 35.8% at Level 4, a 13.4-point gap; GPT-5.4 and Claude-4.7-Opus also decline.GPT-5.4 drops from 67.4% to 53.0%, and Claude-4.7-Opus from 74.0% to 46.5%.Source: Table 3 · p. 5, §4 · p. 5

Evaluation Setup

  • The evaluation compares 15 MLLMs across four levels using the same seed problems, with each level split into 800 test and 200 testmini examples.Unless otherwise stated, results use the test set; models marked in Table 3 use testmini.Source: §4 · p. 5

Main Results

  • Variable grounding is the dominant bottleneck: moving variables and labels into images causes the largest stage gap, ∆V = 7.4 points.Moving only structure into images yields ∆S = 3.0 points, while final rendering adds ∆R = 2.9 points, with OCR, formula recognition, and layout understanding as additional burdens.Source: Table 3 · p. 5, §4 · p. 6
  • Claude-4.7-Opus has only 33.5% four-way consistency despite the highest Level-1 accuracy, while GPT-5.4 has 32.6% consistency.Among problems each model already solves at Level 1, Level-4 accuracy remains 64.8% for GPT-5.4 and 57.4% for Claude-4.7-Opus; no model matches the human group’s four-way consistency.Source: §4 · pp. 5–6

5 Training-Time Diagnostic: Can RL Help Close the Modality Gap?

RL increased accuracy on SEEPHYS PRO under both normal and blind training, but the modality-transfer gaps did not close. Across external benchmarks, blind training also improved accuracy, showing that gains alone do not establish improved visual grounding.

5 Training-Time Diagnostic: Can RL Help Close the Modality Gap?

  • The diagnostic tests whether RL gains preferentially benefit visually demanding levels and reduce transfer gaps such as ∆V and ∆T, as expected from visually grounded improvement.The section frames this as a training-time question following the inference-time accuracy losses observed when physics problems are represented more visually.Source: §5 · p. 6

5.1 Diagnostic Setup

  • PhysRL-38K is a source-matched, test-disjoint physics VQA corpus, and GPT-5-mini filtering yields PhysRL-8K vision-necessary examples.The corpora support scalable RL training without SEEPHYS PRO’s manual redrawing and four-level alignment; training also includes a vision-necessary math subset, ViRL39K-VN.Source: §5.1 · p. 6

5.2 SeePhys Pro Reveals Accuracy Gains without Gap Closure

  • Qwen3-VL-4B Level-1 accuracy rose from 9.9% to 18.3% (normal RL) and 20.9% (blind RL), while ∆T widened from 3.5 points to 7.5/7.9 after normal/blind RL.Both training conditions improved accuracy across all four levels, but the variable-grounding gap showed no stable closure.Source: Figure 3 · p. 7, §5.2 · p. 7, Figure 4 · p. 8

5.1 Diagnostic Setup

  • Blind RL replaces all training images with black images while keeping splits, rewards, and other settings matched; validation images remain unmasked.The diagnostic evaluates accuracy gains alongside modality-gap dynamics; positive gap closure means RL reduces a benchmark-defined failure mode.Source: §5.1 · pp. 6–7

5.3 Cross-Benchmark Negative Controls

  • Blind RL also improves external math and physics benchmarks, often recovering a large fraction of normal-RL gains on MathVerse and MMK12, while normal RL is stronger on some physics benchmarks.For Qwen3-VL-4B, normal RL is clearly stronger on PhyX and OlympiadBench; persistent blind gains show that accuracy gains alone do not demonstrate improved visual grounding.Source: Figure 4 · p. 8, §5.3 · p. 8

5.2 SeePhys Pro Reveals Accuracy Gains without Gap Closure

  • The close normal- and blind-RL trajectories suggest that gains mainly reflect general physics problem-solving improvements rather than more reliable visual grounding.Examples include physical-law selection, equation manipulation, numerical calculation, adaptation to common problem templates, and answer generation.Source: Figure 4 · p. 8

6 Mechanism Analysis: What Does Blind RL Learn?

Blind RL can improve despite image-dependent individual problems because datasets expose exploitable textual and structural regularities. The controls indicate that gains rely on these non-visual cues rather than valid visual evidence.

Text deletion

  • On MathVerse, Qwen3-VL-4B’s blind peak gain falls from about 26.6 points at 25% text deletion to nearly zero at 100%; on MMK12, it falls from about 19.0 points to below one.Complete text deletion also reduces blind gains on PhysReason and SeePhys Pro to near zero.Source: §6 · p. 8, Figure 5 · p. 9

Targeted deletion

  • Physics blind gains persist under most single-category deletions, indicating that shortcuts are distributed across weak textual cues and dataset regularities rather than tied to one field.Deleting numbers, options, or formulas can reduce gains; in math, answer options are a particularly visible shortcut source.Source: §6 · p. 9, Figure 5 · p. 9

Mask-rate and format controls

  • Mask-rate gains stayed positive and non-monotonic; after 90% format reward, Qwen2.5-VL-7B gained 21.2/19.1 points on MMK12 and 9.6/3.9 on MathVerse under normal/blind RL.These controls indicate that outcome-only RL exploits residual textual and structural regularities; final-answer rewards cannot distinguish visual evidence from priors or shortcuts, a credit-assignment failure [26] [25].Source: §6 · p. 9, Figure 5 · p. 9

Limitations

  • SeePhys Pro is designed for controlled modality-transfer diagnosis, not maximal physics difficulty, and the paper validates no complete mitigation strategy for modality inconsistency or non-visual blind-RL gains.Proposed directions include counterfactual image-text pairs, black-image unanswerability, and process-level rewards.Source: §6 · p. 9

7 Conclusion · A Additional Evaluation Results · A.1 Evaluation Protocol Details

SEEPHYS PRO tests whether multimodal models preserve physics reasoning as information shifts from text to visual structure, variables, and diagrams; the results show modality-transfer fragility and that blind training can improve validation accuracy without valid visual training information. The paper advocates assessing inference-time modality robustness and training-time grounding, alongside final-answer accuracy.

7 Conclusion

  • SEEPHYS PRO’s four aligned variants reveal fragile reasoning under modality transfer, with visual variable grounding as a key bottleneck.The variants progressively move information from text to visual structure, visual variables, and fully rendered diagrams.Source: §7 · p. 9
  • Blind training with PhysRL can improve validation accuracy even when training images contain no valid visual information.The authors use SEEPHYS PRO as a diagnostic target for the physics reasoning VQA training set.Source: §7 · p. 9
  • The authors advocate evaluating inference-time modality robustness and training-time grounding diagnostics, not just final-answer accuracy.This recommendation follows from modality-transfer fragility and gains under blind training.Source: §7 · pp. 9–10

A.1 Evaluation Protocol Details

  • All models receive answer-oriented prompts asking for internal-monologue reasoning in <thinking> tags and a final answer in \boxed{}.The Chinese prompt is the equivalent instruction; the English prompt specifies these tags directly.Source: §A.1 · p. 13
  • Closed-weight models use 32K context and temperature 0 when supported, while GPT-family models use temperature 1; open-weight models use 16K context.Open-weight decoding otherwise uses default greedy or deterministic settings when available.Source: §A.1 · p. 13
  • Answer equivalence is judged first by deterministic checks and then, for unresolved cases, by DeepSeek-V3.2, which does not rescore reasoning quality.Candidate answers are extracted from boxed spans and other answer markers, then checked across options, normalized strings, symbolic or numerical values, and units.Source: §A.1 · p. 13

A.2 Benchmark Construction Details

SeePhys Pro is constructed through source collection, curation, aligned problem transformation, and controlled sketching. The pipeline preserves each problem’s underlying physics and answer while progressively transferring information from text to diagrams.

Source collection and curation

  • Problems were gathered from more than 5,000 PDF pages across physics datasets, textbooks, exams, and olympiad collections, then curated and labeled by ten engineering-trained annotators.Curation included deduplication, filtering, standardization, and borderline-case review; labels cover discipline, field, domain, visual evidence, and reasoning skills.Source: §A.2 · p. 13

Transformation

  • Accepted seeds were transformed into aligned four-level variants through scenario reframing, variable substitution, objective shifts, and structural changes while preserving the physical system, target quantity, and gold answer.Source: §A.2 · p. 13

Sketching

  • For diagram variants, problems were hand-redrawn, with structure-only diagrams removing symbolic quantities and variable-layer diagrams adding labels and values; Level 4 jointly renders the statement, formulas, and diagram.The structure and variable layers support Levels 2 and 3, while Level 4 remains semantically aligned with the text-only Level 1 variant.Source: §A.2 · p. 13

A.3 Benchmark Embeddings · A.4 Discipline-Level Results · B Additional Training Diagnostic Results

The appendix offers a qualitative embedding snapshot, a diagnostic Level-3 breakdown across physics disciplines, and additional training-diagnostic visualizations. It cautions that the embedding view is not quantitative and that results for smaller disciplines require care.

A.3 Benchmark Embeddings

  • Figure 7 compares text and multimodal embeddings for SeePhys Pro Levels 2–3 and MathVerse’s Vision Intensive and Vision Dominant subsets.Its panels show inputs from both datasets, with the SeePhys Pro panels covering Levels 2 and 3.Source: Figure 7 · p. 14
  • SeePhys Pro’s text-only and multimodal inputs appear more separated at Level 3 than Level 2, while the two MathVerse subsets appear broadly comparable.The figure is an illustrative distributional snapshot, not a quantitative metric.Source: §A.3 · p. 14, Figure 7 · p. 14

A.4 Discipline-Level Results

  • Level 3 places both structural information and variables in the image, making its discipline breakdown a diagnostic of visually grounded physics reasoning.The results are intended primarily as a diagnostic breakdown, not a fine-grained leaderboard.Source: §A.4 · p. 14
  • Mechanics has 456 Level-3 examples and Electromagnetism has 389, while the remaining disciplines have substantially fewer samples.Performance on Thermodynamics, Optics, Waves/Acoustics, and Modern Physics should therefore be interpreted with caution.Source: §A.4 · p. 14

B Additional Training Diagnostic Results

  • Appendix summaries use raw exported validation records, although the main-paper claims use smoothed curves for readability.Unless stated otherwise, RL runs use 16 NVIDIA GH200 GPUs.Source: B Additional Training Diagnostic Results · p. 15
  • Additional visualizations cover PhysRL-38K validation, post-format-saturation gains, full image-mask-rate curves, and expanded cross-benchmark peak-gain plots.These are presented in Figures 8, 9, 10, and 11, respectively.Source: B Additional Training Diagnostic Results · p. 15

B.1 Physics Training Pool Validation

Training on PhysRL-38K improves held-out physics validation accuracy alongside training reward accuracy, and post-format-saturation answer accuracy can continue to rise.

B.1 Physics Training Pool Validation

  • PhysRL-38K is the larger source-matched physics training pool used to derive PhysRL-8K, and its validation gains indicate transferable physics reasoning signal rather than only SeePhys Pro-specific artifacts.Duplicate metric values at the same step are averaged, with light moving-average smoothing used only for visualization.Source: §B.1 · p. 15
  • Validation accuracy improves on SeePhys Pro Level 3/4 and PhysReason across both sequence-length settings, while training reward accuracy also increases.Qwen3-VL-4B is trained with GSPO on PhysRL-38K and evaluated on held-out, unmasked validation sets; the examples are instance-disjoint from benchmark test sets.Source: Figure 8 · p. 15, §B.1 · p. 15
  • After format reward crosses 90%, answer accuracy can still increase substantially, especially in Qwen2.5-VL-7B math runs, arguing against a purely format-compliance explanation for blind gains.Source: Figure 9 · p. 16

C Error-Type Clustering · D Case Study Examples

Across modality-transfer levels, text-level errors are dominated by physics modeling and reasoning, while visually richer settings add figure- and rendered-text-reading errors. The case studies illustrate modeling, geometry, constraint, numerical-reading, and force-reasoning failures.

C Error-Type Clustering

  • Text-level failures are dominated by physics modeling and reasoning, while visually richer levels add structural-figure, numerical-figure, and rendered-text reading errors.Figure 12 summarizes manually annotated error-type distributions for GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.7 across Levels 1–4.Source: C Error-Type Clustering · p. 18, Figure 12 · p. 18

D Case Study Examples

  • Case studies show models oversimplifying or misreading physical motion structure.The examples are selected independently for interpretability and are not restricted to Figure 12’s fixed model set.Source: Figure 13 · p. 19, D Case Study Examples · p. 19
  • Visual geometry cases show models misgrounding cues such as angles, radii, and arc marks.Source: Figure 14 · p. 20
  • Constraint and equilibrium cases include oversimplified constraints, false equilibrium assumptions, and incorrect motion assumptions.Source: Figure 15 · p. 21
  • Transformer examples show numerical misreading in visual inputs propagating into incorrect frequency and option judgments.Source: Figure 16 · p. 21
  • Reaction-force examples contrast correct force-balance reasoning with wrong reaction directions and moment-equation assumptions.Source: Figure 17 · p. 22
  • Induced-charge examples show numerical misreading disrupting otherwise correct physical grounding.Source: Figure 18 · p. 22

Related papers

Loading 2605.09266v2…