Source-linked AI summary
Multimodal Reasoning for Science: Technical Report and 1st Place Solution to the ICML 2025 SeePhys Challenge
Hao Liang, Ruitao Wu, Bohan Zeng, Junbo Niu, Wentao Zhang, Bin Dong
TL;DR
Multimodal reasoning remains challenging even as text-only reasoning advances, with state-of-the-art systems struggling to integrate visual information. The paper proposes caption-assisted reasoning to bridge visual inputs and textual reasoning, reports first place in the SeePhys challenge, and demonstrates generalization to MathVerse.
Problem
Even state-of-the-art multimodal systems struggle to extract and integrate visual information, leaving effective cross-modal reasoning a persistent challenge.
Source: §1 · p. 1Method
The framework uses generated or human-provided captions to connect visual inputs with structured textual reasoning, including caption-only reasoning for relatively low-information images.
Source: §1 · p. 1Results
The approach achieved 1st place in the ICML 2025 SeePhys challenge and demonstrated strong generalization on MathVerse.
Source: §6 · p. 6Takeaways & Limitations
The experiments report that high-quality captions consistently improve multimodal reasoning and that strong LLMs with caption-only reasoning can rival or outperform direct multimodal pipelines.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Multimodal reasoning remains a fundamental challenge in artificial intelligence. Despite substantial advances in text-based reasoning, even state-of-the-art models such as GPT-o3 struggle to maintain strong performance in multimodal scenarios. To address this gap, we introduce a caption-assisted reasoning framework that effectively bridges visual and textual modalities. Our approach achieved 1st place in the ICML 2025 AI for Math Workshop \& Challenge 2: SeePhys, highlighting its effectiveness and robustness. Furthermore, we validate its generalization on the MathVerse benchmark for geometric reasoning, demonstrating the versatility of our method. Our code is publicly available at https://github.com/OpenDCAI/SciReasoner.
1. Introduction
The introduction frames visual–textual integration as a persistent multimodal reasoning challenge despite stronger text-only reasoning. It presents caption-assisted reasoning as a bridge and reports first place in SeePhys and generalization to MathVerse.
1. Introduction
- Multimodal reasoning remains challenging: even state-of-the-art systems struggle to extract and integrate visual information, unlike their stronger text-only performance.The passage identifies effective cross-modal reasoning as an unresolved challenge.Source: §1 · p. 1
- Caption-assisted reasoning uses generated or human-provided captions to connect visual inputs with structured textual reasoning.The framework is intended to improve cross-modal alignment and support more robust inference.Source: §1 · p. 1
- The framework targets unstable perception-to-inference integration, limited cross-domain generalization, and reliance on costly fine-tuning or annotated chain-of-thought data.The introduction characterizes captioning as a semantically rich and interpretable bridge.Source: §1 · p. 1
- For relatively low-information images, concise captions can capture key visual details, and reasoning from captions alone can yield competitive results.The introduction presents this as a contribution of the framework.Source: §1 · p. 1
- The method achieved 1st place in the ICML 2025 SeePhys competition and showed generalization to MathVerse.The introduction also describes effectiveness across physics reasoning tasks.Source: §1 · pp. 1–2
2. Method
The method motivates captions as a concise way to represent simple visual structures, then tests captioning and reasoning enhancements for more complex, information-rich figures. Its final pipeline combines structured captions, image reintegration, format optimization, and critical review.
2.1. Motivation
- For simple diagrams and schematic systems, captions can express sparse visual relations concisely, filter distractions, and reduce redundant image tokens.The passage links this setting to clearer logical chains and more stable reasoning performance.Source: §2.1 · p. 2
2.2. Methods
- Captioning variants include unconstrained Default Captioning, entity-and-relation Grounding, and structured descriptions intended to improve consistency for downstream reasoning.Grounding is intended to reduce ambiguity in physics descriptions.Source: §2.2 · p. 2
- Adaptive Answer Routing selects image input for categories where it outperformed caption reasoning and caption-based reasoning for other categories.The reported image-selected categories include quantum mechanics, projectile motion, electromagnetic fields, charge distribution, circuit diagrams, spring force, and atomic physics.Source: §2.2 · pp. 2–3
- The method also tests rephrasing, Image Reintegration, Adaptive Answer Routing, standardized answer formats, and a second-model Critical Review.The evaluation proceeds from direct reasoning and rephrasing to structured captioning and further enhancements.Source: §2.2 · pp. 2–3
- The final reasoning pipeline combines Structured Captioning, Image Reintegration, Format Optimization, and Critical Review.The pipeline is identified as the integration of the most effective components.Source: §2.2 · p. 3, Figure 2 · p. 3
3. Experiments
Experiments evaluate captioning and reasoning strategies on SeePhys-mini, then test performance across its knowledge levels. Structured descriptions and staged refinements yield the strongest reported results, while the best configuration varies by level.
3.1. Dataset and Implementation Details
- Experiments use 200 randomly selected SeePhys questions; o3 judges API-generated answers with binary scoring that counts only fully correct answers.Source: §3.1 · p. 3
3.2. Performance of Foundational Methodologies
- Direct G2.5P reaches 58.0% accuracy, while Rephrasing peaks at 50.5%, below the direct multimodal baseline.Source: §3.2 · p. 3
3.3. The Impact of Captioning Quality
- Default captioning reaches 58.5% (o3 →o3), Grounding reaches 59.0% (G2.5P →o3), and Structured captioning reaches 61.5%.Source: §3.3 · pp. 3–4
3.4. Stepwise Enhancement Strategies
- Adding the image raises Grounding accuracy from 59.0% to 60.5%, whereas AAR lowers Grounding's 59.0% to 58.0% by sometimes replacing stronger caption answers.Source: §3.4 · p. 4
- Structured+Img+CR reaches the overall maximum 66.0% accuracy on SeePhys-mini.Source: §3.4 · p. 4
- Adding rules to the Structured method raises accuracy from 61.5% to 63.5%; combining Structured + Img reaches 65.5%, and adding Critical Review yields the overall maximum of 66.0%.Source: §3.4 · p. 4
3.5. Analysis of SeePhys Results
- At foundational levels, Structured+FO reaches 81.0% at UG, while Structured alone reaches 72.2% at High School.Source: §3.5 · p. 4
- Higher-level peaks vary: BO reaches 76.9% with Structured+FO or AAR, AO 58.7% with Grounding, SUG 68.4% with Default+AAR, Master’s 85.7% with G2.5P, and PhD 57.4% with Structured, AAR, Img, and FO or CR.Source: §3.5 · p. 5
4. Generalization to Other Domains
On MathVerse geometric problems, captioning improves performance across tested models, and caption-only reasoning can also be competitive. The reported results include caption-assisted systems exceeding GPT-o3's direct multimodal performance on Vision Intensive tasks.
- With Claude-Opus-4, GPT-o3 captioning raises accuracy from 60.2% without captioning to 85.5% with image input on Vision Intensive.Source: §4 · p. 6
- DeepSeek-R1 with GPT-o3 captions scores 68.2% on Vision Only in the text-only setting, surpassing GPT-o3 under the same setting.Source: §4 · p. 6
- On Vision Intensive, Qwen2.5-72B-Instruct scores 69.7% and Qwen2.5-7B-Instruct with Qwen2.5-VL-72B-Instruct captions scores 68.3%, above GPT-o3's unc captioned 64.5%.Source: §4 · p. 6
- Captioning consistently improves multimodal reasoning across tested models, while caption-only reasoning with advanced LLMs can rival or outperform direct multimodal reasoning.Source: §4 · p. 6
5. Future Work
The authors identify future work in extending reasoning strategies and scientific domains, adapting captions to task demands, and incorporating human feedback. These directions aim toward a general-purpose scientific reasoning system.
- The authors plan to integrate program-of-thought and tool-augmented reasoning to strengthen complex problem-solving beyond physics and mathematics.Source: §5 · p. 6
- They propose extending evaluation to chemistry, biology, and engineering and developing captions that adapt granularity and structure to task complexity and modality requirements.Source: §5 · p. 6
- They identify human-in-the-loop feedback and interactive reasoning as future directions toward a general-purpose scientific reasoning system.Source: §5 · p. 6
6. Conclusion
The conclusion presents caption-assisted reasoning as a bridge between visual and textual modalities, reporting first place in the ICML 2025 SeePhys challenge and generalization to MathVerse. The appendix points readers to prompt figures for seven pipeline components.
6. Conclusion
- The framework uses generated or human-provided captions to bridge visual and textual modalities, achieved first place in the ICML 2025 SeePhys challenge, and showed generalization on MathVerse.Source: §6 · p. 6
A.1. Prompt for Rephrasing
- The Rephrasing prompt is shown in Figure 3.Source: §A.1 · p. 8
A.2. Prompt for Default Captioning
- The Default Captioning prompt is shown in Figure 4.Source: §A.2 · p. 8
A.3. Prompt for Grounding
- The Grounding prompt is shown in Figure 5.Source: §A.3 · p. 8
A.4. Prompt for Structured Captioning
- The Structured Captioning prompt is shown in Figure 6.Source: §A.4 · p. 8
A.5. Prompt for Answer
- The Answer prompt is shown in Figure 7.Source: §A.5 · p. 8
A.6. Prompt for Format Optimization
- The Format Optimization prompt is shown in Figure 8.Source: §A.6 · p. 8
A.7. Prompt for Critical Review
- The Critical Review prompt is shown in Figure 9.Source: §A.7 · p. 8
B.1. Example of Rephrasing
The appendix identifies Figure 11 as the Rephrasing example.
B.1. Example of Rephrasing
- Figure 11 is identified as the Rephrasing example.Source: §B.1 · p. 8
B.2. Example of Default Captioning
The appendix points to Figures 12–14 for examples of Default Captioning, Grounding, and Structured Captioning.
B.2. Example of Default Captioning
- Figure 12 is identified as the Default Captioning example.Source: §B.2 · p. 8
B.3. Example of Grounding
- Figure 13 is identified as the Grounding example.Source: §B.3 · p. 8
B.4. Example of Structured Captioning
- Figure 14 is identified as the Structured Captioning example.Source: §B.4 · p. 8
B.5. Example of Critical Review
The supplied appendix passages identify prompts for the captioning and answer pipeline stages and refer readers to examples of Rephrasing, Default Captioning, Grounding, Structured Captioning, and Critical Review.
B.5. Example of Critical Review
- Figures 3–6 present prompts for Rephrasing, Default Captioning, Grounding, and Structured Captioning.Source: Figure 3 · p. 9, Figure 4 · p. 9, Figure 5 · p. 10
- Figures 7–10 present prompts for Answer, Format Optimization, Critical Review, and MathVerse captioning.Source: Figure 7 · p. 12, Figure 8 · p. 13, Figure 9 · p. 14
- Figures 11–15 are identified as examples of Rephrasing, Default Captioning, Grounding, Structured Captioning, and Critical Review.Source: §B.5 · p. 8, Figure 11 · p. 15, Figure 12 · p. 16