SCI-Reason: A Dataset with Chain-of-Thought Rationales for Complex Multimodal Reasoning in Academic Areas

1. Introduction

• Academic image reasoning requires cross-modal alignment of domain-specific semantics and structured visual logic, from pixel perception to knowledge-based hypothesis testing.Image-text consistency checks may require combining multiple subfigures to assess support for textual conclusions.Source: §1 · p. 1
• Existing benchmarks are limited by narrow mathematical focus, weak validation of reasoning chains, and single-image inputs that miss multi-document academic analysis.The cited examples include symbolic-math emphasis, susceptibility to keyword matching, and lack of mechanisms for verifying reasoning.Source: §1 · pp. 1–2
• SCI-Reason is introduced as an academic multimodal reasoning dataset built from high-confidence PubMed image-text pairs with MCTS-generated thought chains.Its construction uses a fine-grained knowledge-guided QA paradigm and a Validable Thought Chain annotation framework.Source: §1 · p. 2
• The dataset formalizes scientific image reasoning through five competencies: entity location, temporal, cross-subgraph role, causal mechanism, and methodological technical reasoning.The authors report that separate training- and test-set evaluations found it useful for assessing and improving model reasoning in academic scenarios.Source: §1 · p. 2
• The evaluation plan fine-tunes Qwen2-VL-7B on SCI-Reason, tests cross-domain generalization, and assesses eight advanced models while analyzing their errors.The authors frame the error analysis as a way to inform future research.Source: §1 · p. 2

2.1. Collection Guidelines

• SCI-Reason uses peer-reviewed academic images, expert-validated biomedical content, figures with 3–7 interconnected subfigures, four visualization types, and efficient chains of thought.Its stated design principles are source authenticity, structural complexity, multimodal diversity, and reasoning guidance.Source: §2.1 · p. 2
• MCTS is used to generate logically consistent reasoning chains, with stepwise verification and difficulty-aware sampling to counter a bias toward simpler reasoning.Backward-chaining verification prunes claims unsupported by image evidence.Source: §2.1 · pp. 2–3

2.2. Data Collection

• The data pipeline aligns PubMed figures with article descriptions and conclusions, then creates expert-checked fill-in-the-blank questions by masking domain-specific entities.The questions are organized into five reasoning types: entity location, temporal, cross-subgraph role, causal mechanism, and methodological technical reasoning.Source: §2.2 · pp. 3–4

2.3. CoT Generation

• For each question-image instance, MCTS seeks an optimal reasoning path composed of atomic visual observations or knowledge-based deductions leading to the answer.The framework aims to provide explicit reasoning guidance and enable granular evaluation of inference validity.Source: §2.3 · p. 4
• MCTS selects paths with depth-adaptive UCB, expands leaves with a vision-language model, and rewards both path fidelity and explanatory completeness.The search proceeds through selection, expansion, simulation, and backpropagation phases.Source: §2.3 · p. 4

2.4. Data Analysis

• 73.2% of figures contain statistical charts, 47.0% microscopic images, 10.5% conceptual schematics, and 11% temporal sequences or other composite visualizations.The image categories can overlap; the paper describes this mix as broad coverage of scientific visual representations.Source: §2.4 · p. 4
• 39.8% of questions involve multimodal temporal reasoning, 25.9% professional entity location, 13.2% cross-subgraph role, 11.7% causal mechanism, and 9.4% methodological technical reasoning.Microscopic images more often pair with entity-location questions, while statistical diagrams more often pair with causal-mechanism questions.Source: §2.4 · p. 4
• Answers are 41.4% descriptive text, 38.6% jargon, and 20.0% numerical, averaging 1.6 words.The paper characterizes these answers as precise and concise.Source: §2.4 · pp. 4–5

3.1. Evalution Protocols

• Evaluation combines exact-match accuracy, answer similarity measured by ANLS, and semantic similarity measured by WUPS.ANLS uses normalized Levenshtein distance; WUPS uses the WordNet hierarchy to assess approximate matching of technical terms.Source: §3.1 · p. 5

3.2. Experiment set up

• The experiments compare eight multimodal models using direct-answer and CoT-enhanced generation, analyze reasoning errors, and fine-tune Qwen2-VL on the training set.The study also assesses fine-tuned performance on public datasets; Figure 3 presents examples of five classified task types.Source: §3.2 · pp. 5–6, Figure 3 · p. 6

3.3. Experiment Result

• Claude-3.7-Sonnet achieved an accuracy score of 55.19, while GPT-4O achieved a WUPS score of 61.34 on the SCI-Reason test set.The results indicate substantial room for improvement on complex academic image reasoning.Source: Table 2 · p. 6, §3.3 · p. 6
• On the SCI-Reason test set, fine-tuned Qwen2-VL-7B outperformed the base model in accuracy (53.66% vs. 41.64%), ANLS (56.19% vs. 44.08%), and WUPS (60.79% vs. 49.50%).The comparison evaluates LoRA fine-tuning on the training set.Source: Table 3 · p. 7, §3.3 · p. 7
• On MMMU, fine-tuned Qwen2-VL-7B outperformed the base model in Business (45.6% vs. 42.3%), Science (38.7% vs. 36.0%), Health (51.3% vs. 50.0%), and Tech (36.9% vs. 35.7%).The authors report these results as evidence of generalization to another dataset.Source: Table 3 · p. 7, §3.3 · p. 7
• Knowledge-based Deficiency (37.1%) and Logical Inference Error (29.5%) were more common than Visual Perception Bias (15.2%) and Contextual Integration Deficit (18.2%).The analysis identifies reasoning and knowledge use, rather than visual perception, as the main challenges.Source: Figure 4 · p. 7, §3.3 · p. 7
• Yi showed no significant improvement after adding CoT, which the authors attribute to limitations in image understanding and expertise.The authors note that CoT did not compensate for these shortcomings in knowledge or comprehension.Source: §3.3 · p. 7

• Most existing research focuses on pure-text or structured-symbolic reasoning, and support for multimodal scenarios remains insufficient.Source: §4 · p. 8
• A-OKVQA [31] and Visual Genome [20] prioritize object recognition; SuperCLEVR [22] and GQA [18] use simplified geometry, while ScienceQA [24] and MMMU [41] retain shallow task designs.Source: §4 · p. 8
• Complex academic-image reasoning is important for image-text consistency detection and research integrity [10].Source: §4 · p. 8

• SCI-Reason combines knowledge-guided fill-in-the-blanks with MCTS-driven interpretable inference chains for academic multimodal reasoning.Source: §5 · p. 8
• 55.19% accuracy was the best result among eight evaluated foundation models on complex academic image reasoning.Source: §5 · p. 8
• Fine-tuned Qwen2-VL-7B reached 53% test accuracy and surpassed GPT-4O.Source: §5 · p. 8