Source-linked AI summary
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan
TL;DR
Science question benchmarks often lack answer explanations or are limited in modality, scale, and topic diversity, hindering study of interpretable multi-hop reasoning. The paper introduces ScienceQA, a 21,208-example multimodal benchmark with lectures and explanations, and trains models to generate them as chain-of-thought. CoT improves question answering in few-shot GPT-3 and fine-tuned UnifiedQA, while explanations also improve GPT-3 when supplied as inputs.
Problem
Existing science datasets often lack answer explanations or are restricted to text-only modality, small scales, and limited topic diversity.
Method
The paper constructs ScienceQA with multimodal questions, grounded lectures, and explanations, then trains language models to generate lectures and explanations as chain-of-thought.
Results
CoT improves few-shot GPT-3 by 1.20% and fine-tuned UnifiedQA by 3.99%, while feeding explanations into GPT-3 improves few-shot performance by 18.96%.
Takeaways & Limitations
The results show that explanations can support language-model question answering and help models learn from fewer data.
Takeaways & Limitations
GPT-3 still fails on challenging questions because it may not understand multimodal inputs, lack domain knowledge, or generate irrelevant, incorrect, or incomplete reasoning.
Abstract
from arXiv · showhide
When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.
1 Introduction
SCIENCEQA addresses gaps in science-question datasets by combining multimodal questions with grounded lectures and explanations. The paper trains models to generate these explanations as chain-of-thought and reports gains in few-shot and fine-tuned settings.
- The task targets reasoning transparency because models may produce correct decisions without reliably revealing whether they understand or can generalize.
- Existing science datasets often lack annotated explanations, while explanation-annotated alternatives are text-only, small-scale, or topic-limited.
- SCIENCEQA contains 21,208 multimodal science examples spanning natural, social, and language science, with lectures and explanations paired to answers.Lectures provide general external knowledge, while explanations give specific reasons for the correct answer.
- Models generate lectures and explanations alongside answers as chain-of-thought to mimic the multimodal, multi-hop reasoning process.
- 3.99% improvement was achieved by fine-tuned UnifiedQA with CoT, while few-shot GPT-3 reached 75.17%, a 1.20% improvement over GPT-3 without CoT.GPT-3-generated explanations met the gold standard in 65.2% of human evaluations, and supplying explanations as input improved few-shot GPT-3 by 18.96%.
2 Related Work
Prior science and visual-question datasets support multimodal reasoning but generally omit answer explanations or restrict them to text, smaller scales, or narrower topics. SCIENCEQA combines multimodal contexts with grounded explanations and broader scientific coverage.
- SCIENCEQA differs from conventional VQA datasets through multimodal contexts, diverse scientific topics, and answer annotations with lectures and explanations.
- Existing multimodal science datasets generally lack annotated explanations that reveal the reasoning steps behind answers.
- Other explanation-annotated datasets use supporting facts, entailment trees, explanation graphs, or reasoning chains but remain text-only, small-scale, or topic-limited.
- Learning-from-instruction research shows that task-level explanations can improve model performance, motivating explanation-based reasoning in science question answering.
3 Dataset
SCIENCEQA is a large multimodal science-question dataset spanning diverse subjects, contexts, and grade-level content, with lectures and explanations annotating most answers.
- Dataset overview: SCIENCEQA contains 21,208 multimodal multiple-choice science questions spanning natural, social, and language science.The dataset includes 26 topics, 127 categories, and 379 skills.
- Annotations: 83.9% of questions have lectures and 91.3% have explanations, providing general background knowledge and specific reasons for answers.The dataset combines questions, choices, multimodal contexts, answers, lectures, and explanations.
- Question analysis: Question lengths range from 3 to 141 words, averaging 12.11 words, with a flatter distribution than other VQA datasets.The broader length distribution highlights varied question formats.
- Context analysis: 66.11% of questions include at least one context type, with 7,803 unique image contexts and 4,651 unique text contexts.Contexts include diagrams, natural images, semantically rich text, or simple hints.
- Dataset comparisons: Compared with existing science datasets, SCIENCEQA is larger, spans all 12 grades, includes more images and input-source diversity, and covers three subjects.Most questions are annotated with textual lectures and explanations.
4 Baselines and Chain-of-Thought Models
The paper establishes heuristic, zero-shot, few-shot, and fine-tuning baselines, then adapts UnifiedQA and GPT-3 to generate answers with lectures and explanations as chain-of-thought outputs.
- Baselines: The evaluation establishes random-chance and human-performance heuristic baselines alongside UnifiedQA and GPT-3 zero-shot and few-shot baselines.The standard input format combines question text, context, and multiple-choice options.
- Baselines: Fine-tuning baselines use VQA models with textual and visual inputs, while UnifiedQA processes textual information and predicts answers.VQA models predict score distributions over answer choices through a linear classifier.
- Chain-of-thought formulation: A chain of thought decomposes multi-hop reasoning into intermediate steps and can appear as preceding thoughts or post-answer explanations.SCIENCEQA lectures and explanations serve as demonstrations of this reasoning process.
- UnifiedQA with CoT: UnifiedQA with CoT is fine-tuned to generate the answer followed by a lecture and explanation instead of only a short final answer.This modifies the model’s output format to produce a long explanatory sequence.
- GPT-3 with CoT: GPT-3 CoT prompting uses in-context examples and instructs the model to generate the answer, lecture, and explanation without parameter updates.The prompt includes instructions for both a one-shot training example and the test example.
5 Experiments
Experiments compare VQA and language-model baselines on ScienceQA, finding that chain-of-thought improves answer performance and supports more efficient learning, while substantial gaps and failure modes remain.
- Question Answering: 61.87% average accuracy made VisualBERT the strongest VQA baseline, although Patch-TRM led on natural and language science and higher-grade questions.VisualBERT exceeded Patch-TRM by 22.39% on social science, revealing substantial class-dependent variation.
- Question Answering: UnifiedQABASE reached 70.12% average accuracy after fine-tuning, while CoT improved it by +3.21% with explanations and +3.99% with lectures and explanations.The results associate generating answer rationales with improved language-model reasoning performance.
- Question Answering: Human performance exceeded every benchmark consistently, including a 20.07% image-context gap over the best model.The remaining gap motivates further multimodal scientific reasoning research.
- Generated Explanations: GPT-3 CoT explanations best matched human judgments, while UnifiedQA CoT produced the most similar explanations under automatic evaluation.Automatic metrics assessed BLEU, ROUGE-L, and semantic similarity; annotators rated relevance, correctness, and completeness.
- Analysis: QCM→ALE produced the highest average GPT-3 accuracy and smallest variance, peaked with two in-context examples, and then declined as examples increased.Generating lectures and explanations together was more stable than prompting with explanations alone.
- Analysis: Gold lectures and explanations increased GPT-3 accuracy by 18.96% over QCM→ALE, indicating substantial headroom from improving explanation generation.The upper-bound experiment supplied gold reasoning text in the test prompt.
- Analysis: Generating lectures and explanations before answers caused large accuracy decreases, partly because long outputs could terminate early or exhaust token limits.The reported ordering effect was measured on 1,000 test examples.
6 Discussion and Conclusion
The paper concludes that ScienceQA is a large multimodal science benchmark with explanation annotations and that chain-of-thought improves language-model question answering and explanation quality.
- Discussion and Conclusion: ScienceQA contains 21,208 multimodal science questions with lectures and explanations, supporting evaluation of multimodal reasoning and interpretable answers.The dataset spans science-curriculum questions and annotates most answers with corresponding reasoning text.
- Discussion and Conclusion: UnifiedQA CoT improved performance by 3.99%, while few-shot GPT-3 CoT reached 75.17% accuracy on ScienceQA.Human evaluation found that 65.2% of GPT-3 CoT explanations met the gold standard.
- Discussion and Conclusion: ScienceQA provides a basis for studying whether generated lectures and explanations improve language-model reasoning ability.The conclusion frames the benchmark and experiments jointly around explanation generation and question answering.
Checklist
The checklist records dataset licensing and release details, experimental reproducibility information, and the authors’ stated ethics and broader-impact disclosures.
- Checklist: The paper states that it discusses limitations, negative societal impacts, and broader impacts in the error-analysis and ethics materials.The checklist points to Section 5.4 and Appendices B.4 and B.5.
- Checklist: ScienceQA is released under the CC BY-NC-SA 4.0 license for non-commercial research purposes.The source data comes from IXL and the paper states the license of the released dataset.
- Checklist: The authors provide data examples, a visualizer, dataset and code availability, training details, error bars, and compute-resource information.They report four repetitions for GPT-3 CoT experiments and point to appendices for implementation details.
- Checklist: The authors report no personally identifiable information or offensive content in the collected data and mark consent and IRB questions as not applicable.Participant compensation and study instructions are documented in the appendices where applicable.
A.1 Data Collection
ScienceQA is assembled from open K–12 science resources and designed with broad topical, linguistic, choice-length, and grade-level diversity.
- Data Collection: Questions are sourced from IXL science problems aligned with California Common Core Content Standards and decomposed into structured components using heuristic rules.Extracted components include questions, hints, images, options, answers, lectures, and solutions.
- Data Collection: Manual cleaning removes invalid, faulty, single-choice, and duplicate questions, while answer options are shuffled to avoid positional patterns.When multiple answers applied, the construction retained one correct answer and reformatted lectures and explanations semi-automatically.
- Data Collection: ScienceQA covers diverse topics across natural, social, and language science, with subject-specific vocabulary distributions.Examples include force and magnet in natural science, capital and state in social science, and dictionary and page in language science.
- Data Collection: Choices range from two to five options, are usually at most five words long, but about 5% exceed 15 words.The long tail requires models to handle varied textual choice lengths.
- Data Collection: Question lengths have long-tail distributions across all subjects, with social science shortest on average and language science longest and more evenly distributed.These distributions indicate compositional diversity in the dataset.
- Data Collection: Most questions target grades 3–8, while around 10% come from grades 9–12 and approach U.S. college-admissions-test difficulty.The grade mix requires both broad scientific knowledge and complex reasoning.
B.1 Experimental Details
The experiments describe training, input-processing, captioning, compute, and GPT-3 settings, alongside the AMT test-set procedure.
- Fine-tuning baselines and UnifiedQA used the training set, validation set, and test set for training, development, and evaluation, respectively.
- VQA baselines used a maximum input length of 100 words or tokens, with batch sizes of 64 for VQA and 4 for UnifiedQA.
- Image captions were generated with tool2 using a maximum length of 16, four beams, and up to 512 output tokens.
- GPT-3 experiments used default zero temperature and penalties, top probability 1.0, online API access, and specified trial examples.
- The human test evaluation shuffled 4,241 questions into 425 batches, added five training examples per batch, and assigned each set to three AMT workers.
B.3 Human Evaluation of Generated Explanations
The human evaluation asked AMT workers to judge whether model-generated explanations were relevant, correct, and complete, using majority votes.
- AMT workers evaluated each model explanation for relevance, correctness, and completeness after seeing the question, context, choices, and answer.
- The instructions and test-question interface for AMT workers are shown in Figure 13.
- The explanation-rating interface for UnifiedQA (CoT) and GPT-3 (CoT) is shown in Figure 14.
- Explanation outputs were grouped into batches of 10, assigned to three workers, and evaluated with a $0.3 compensation per batch.
B.4 Case Study and Limitations
Case studies show that GPT-3 (CoT) can produce correct answers with reasonable reasoning chains, but explanations and answers still fail under weak visual information and difficult domain knowledge.
- GPT-3 (CoT) produced correct answers and reasonable explanations that followed human-like multi-hop reasoning in three examples.
- Correct answers sometimes accompanied irrelevant, incorrect, incomplete, or repetitive explanations rather than complete thought chains.
- Caption-based visual inputs lacked fine-grained semantics and often performed poorly for diagrams, producing wrong answers in two failure cases.
- GPT-3 (CoT) also struggled with uncommon terminology in language science and complex chemical changes involved in dinosaur fossil formation.
- The dataset is sourced from science textbooks and contains no user-usage data or personally sensitive information such as gender and race.
- SCIENCEQA is presented as a benchmark for multimodal learning, multi-hop reasoning, general artificial intelligence, and K-12 tutoring applications.