PaperlayerView exact revision on arXiv

Source-linked AI summary

T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering

Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, Heng Tao Shen

arXiv:2305.03453v4cs.CL

TL;DR

ScienceQA CoT methods rely on human rationales that are costly to collect and may omit essential external information. T-SciQ generates and mixes LLM-based teaching rationales for smaller multimodal models, reaching 96.18% accuracy on ScienceQA.

  • Problem

    Multimodal science question answering has received less attention than language-only CoT, while human-annotated CoT is costly and may omit essential external information.

    Source: Introduction · pp. 1–2

  • Method

    T-SciQ generates QA-CoT and plan-based QA-PCoT rationales with LLMs, mixes the teaching samples, and uses them to fine-tune smaller multimodal models.

    Source: Introduction · p. 2, Overview · p. 2, Generating Teaching Data · p. 3

  • Results

    4.5% over Mutimodal-CoTLarge (91.68% →96.18%), with Mutimodal-T-SciQLarge reaching 96.18% accuracy on ScienceQA.

    Source: T-SciQ v.s. Baselines · p. 5

  • Takeaways & Limitations

    T-SciQ provides an approach for teaching smaller models CoT reasoning on complex science question answering tasks with LLM-generated signals.

    Source: Conclusion · p. 7

    QA-PCoT may generate erroneous rationales when visual features are complicated.

    Source: Data Generation Process of PCoT · p. 14

Abstract

from arXiv · show

Large Language Models (LLMs) have recently demonstrated exceptional performance in various Natural Language Processing (NLP) tasks. They have also shown the ability to perform chain-of-thought (CoT) reasoning to solve complex problems. Recent studies have explored CoT reasoning in complex multimodal scenarios, such as the science question answering task, by fine-tuning multimodal models with high-quality human-annotated CoT rationales. However, collecting high-quality COT rationales is usually time-consuming and costly. Besides, the annotated rationales are hardly accurate due to the external essential information missed. To address these issues, we propose a novel method termed T-SciQ that aims at teaching science question answering with LLM signals. The T-SciQ approach generates high-quality CoT rationales as teaching signals and is advanced to train much smaller models to perform CoT reasoning in complex modalities. Additionally, we introduce a novel data mixing strategy to produce more effective teaching data samples for simple and complex science question answer problems. Extensive experimental results show that our T-SciQ method achieves a new state-of-the-art performance on the ScienceQA benchmark, with an accuracy of 96.18%. Moreover, our approach outperforms the most powerful fine-tuned baseline by 4.5%. The code is publicly available at https://github.com/T-SciQ/T-SciQ.

Introduction

ScienceQA requires multimodal understanding and external knowledge, while prior multimodal CoT methods rely on costly human rationales that may omit essential information. T-SciQ uses LLM-generated teaching signals and reports substantial gains over prior ScienceQA systems.

Introduction

  • ScienceQA contains 21,208 multimodal examples across broad topics and skills, requiring models to comprehend multimodal inputs and use external knowledge.Earlier scientific problem-solving datasets were described as limited in scale.Source: Introduction · p. 1
  • Prior CoT research focused mainly on language, while captioning complex scientific images can lose information needed for multimodal question answering.Multimodal-CoT instead models language and visual modalities through separate rationale-generation and answer-inference stages.Source: Introduction · pp. 1–2
  • Human-annotated CoT is time-consuming to produce for expert-intensive ScienceQA and may omit essential external information needed to derive answers.T-SciQ generates CoT and plan-based CoT teaching samples, mixes them, and fine-tunes a student model.Source: Introduction · p. 2
  • 4.5% over the most powerful fine-tuned baseline, 5.26% over the strongest instruction-tuning multimodal baseline, 9.64% over GPT-4 few-shot, and 7.78% over human performance.The reported comparisons are on the ScienceQA benchmark.Source: Introduction · p. 2

Related Work

Related work covers methods for eliciting and improving CoT reasoning, along with LLM-generated reasoning used to teach smaller models. T-SciQ differs by generating and mixing two types of teaching rationales for multimodal tasks.

Chain-of-Thought Prompting

  • CoT prompting elicits intermediate reasoning, while later methods improve demonstrations, reasoning chains, or prompt composition.Examples include Zero-shot CoT, Iterative Prompting, PoT Prompting, and Chameleon.Source: Chain-of-Thought Prompting · p. 2

LLMs as Teachers

Our T-SciQ Approach

T-SciQ has three stages: generate CoT and plan-based CoT teaching samples, mix them by skill using validation performance, and fine-tune a smaller multimodal student in two stages.

Overview

  • T-SciQ uses SciTeacher to generate QA-CoT and QA-PCoT samples, mixes them into a teaching dataset, and fine-tunes SciStudent.The framework has three stages: teaching-data generation, mixing, and fine-tuning.Source: Overview · pp. 2–3, Figure 2 · p. 3

Generating Teaching Data

  • Zero-shot prompting generates QA-CoT rationales using the question, context, options, correct answer as a hint, and an instruction.For examples without context, the context slot is replaced with “N/A.”Source: Generating Teaching Data · p. 3
  • For complex problems, three-step prompting generates a skill lecture, a general plan, and then a plan-based rationale for each example.The approach addresses the challenge that QA-CoT samples alone may not teach highly complex problems.Source: Generating Teaching Data · p. 3

Mixing Teaching Data

  • QA-PCoT is suited to complex problems and QA-CoT to simpler ones; validation performance selects one signal for all training examples in each skill.When validation errors are lower with PCoT for a skill, the method selects PCoT for that skill; otherwise it selects CoT.Source: Mixing Teaching Data · pp. 3–4

Fine-Tuning

  • Fine-tuning first trains a rationale generator on language and visual inputs, then appends the teaching rationale to language input for answer inference.The default architecture shares a model between stages, with different inputs and outputs.Source: Fine-Tuning · p. 4

Experimental Setup

Experiments use the official ScienceQA split and compare T-SciQ with VQA, pretrained multimodal, LLM-based, and in-context baselines. Answer accuracy is the evaluation metric, and the supplied tables cover main results and signal ablations.

Dataset

  • ScienceQA has 21,208 multiple-choice examples spanning three subjects, 26 topics, 127 categories, and 379 skills.The official train, validation, and test split contains 12,726, 4,241, and 4,241 examples, respectively.Source: Experimental Setup · p. 4

Baselines

  • Baselines include early VQA models, pretrained text-to-text and multimodal models, fine-tuned LLM-based multimodal models, and few-shot prompting systems.The comparison includes GPT-3.5, ChatGPT, GPT-4, and Chameleon CoT prompting baselines.Source: Experimental Setup · p. 4

Evaluation Metrics

  • Answer accuracy is measured by comparing the ground-truth option with the model's final prediction.Table 1 reports main test-set results across eight question classes.Source: Experimental Setup · p. 4, Table 1 · p. 5

Implementation Details

  • The default teacher is GPT-3.5 text-davinci-003, and the approach is tested with three student model variants.The student models are reported as 200× smaller than their teacher models.Source: Experimental Setup · pp. 4–5

Evaluation

  • Table 2 reports an ablation study of the impact of different LLM-provided signals across all topics.The setup passage identifies the table's scope but does not provide its results.Source: Table 2 · p. 5

Main Results

Mutimodal-T-SciQLarge reaches 96.18% accuracy, outperforming the reported fine-tuning, instruction-tuning, GPT-4 few-shot, and human comparisons. T-SciQ signals also improve results across two other base student models.

T-SciQ v.s. Baselines.

  • 96.18% accuracy lets Mutimodal-T-SciQLarge outperform Mutimodal-CoTLarge by 4.5%, LLaVa by 5.26%, Chameleon by 9.64%, and human performance by 7.78%.The respective comparison pairs are 91.68% → 96.18%, 90.92% → 96.18%, 86.54% → 96.18%, and 88.40% → 96.18%.Source: T-SciQ v.s. Baselines · p. 5

T-SciQ with Different Base Student Models.

  • UnifiedQA-T-SciQBase improves over UnifiedQABase w/ CoT by 5.3%, while Mutimodal-T-SciQBase improves over Mutimodal-CoTBase by 6.84%.The reported pairs are 74.11% → 79.41% and 84.91% → 91.75%, respectively.Source: T-SciQ with Different Base Student… · p. 5
  • Figure 3 examines Mutimodal-T-SciQBase across generated-data proportions and compares its accuracy curve with Mutimodal-CoTBase across epochs.Source: Figure 3 · p. 6

Further Analysis

The analyses report benefits from both teaching-signal types, increasing generated-data proportions, and T-SciQ across training epochs. They also examine signal mixtures, skill-level errors, and illustrative reasoning cases.

Effect of Different Signals of T-SciQ.

Impact of visual Features.

  • Figure 4 reports student accuracy with teaching signals from different base LLMs and prediction errors for specific skills.Source: Figure 4 · p. 6

Proportion of Generated Data in Training Data.

Performance Change with Epoch.

  • Mutimodal-T-SciQBase consistently outperforms Mutimodal-CoTBase across epochs, with relatively higher accuracy at initial stages under their similar two-stage training approach.Source: Performance Change with Epoch · p. 6

Effect of Teaching Signals Provided by Different Base LLMs.

  • Even the worst of nine mixtures of QA-CoT and QA-PCoT signals from different base LLMs outperforms annotated CoT signals by a significant margin.The tested QA-CoT sources include text-davinci-002, text-davinci-003, and ChatGPT.Source: Effect of Teaching Signals Provided by… · p. 6

Error Analysis.

  • Training with T-SciQ signals reduces errors across six selected skills, including multi-step reasoning and map-direction tasks requiring open-world knowledge.Source: Error Analysis · pp. 6–7

Case Study.

  • In the case studies, T-SciQ includes geographic information missing from human-annotated CoT and correctly decomposes an image-free multi-step question that Multimodal-CoT gets wrong.Source: Case Study · p. 7, Figure 5 · p. 7

Comparison on Other NLP Reasoning Datasets.

Across six reasoning datasets, T-SciQ surpasses Reason-Teacher on five and performs equally well on Coin Flip.

Comparison on Other NLP Reasoning Datasets.

Conclusion

T-SciQ uses LLM-generated chain-of-thought teaching data to train smaller multimodal models for science question answering. It combines standard and plan-based rationales and reports improvement over prior ScienceQA baselines.

Conclusion

  • T-SciQ uses large language models’ chain-of-thought reasoning to teach smaller multimodal models complex science question answering.Source: Conclusion · p. 7
  • Its teaching data combines QA-CoT samples with plan-based QA-PCoT samples generated through a three-step zero-shot prompting approach for highly complex problems.Source: Conclusion · p. 7
  • Evaluation on ScienceQA shows significant improvement over previous state-of-the-art baselines.The authors identify exploring extensive LLMs and parameter-efficient fine-tuning with LLM teachers as future work.Source: Conclusion · p. 7

More Detailed Analysis

The data-mixture strategy performs best across the four tested backbones; QA-CoT alone is the weakest of the generated data types, though it outperforms manually annotated data.

More Detailed Analysis

More Cases Analysis

The examples indicate a difference by problem complexity: QA-PCoT is stronger on multi-step reasoning, while QA-CoT is more proficient on straightforward problems.

More Cases Analysis

  • QA-PCoT-trained models perform better on multi-step reasoning problems, while QA-CoT-trained models are more proficient on straightforward problems.Source: More Cases Analysis · p. 10
  • Figures 6 and 7 compare prediction cases from MM-CoT with QA-CoT and QA-PCoT, respectively.Source: More Cases Analysis · p. 10
  • Figures 8 and 9 compare prediction cases from the QA-CoT and QA-PCoT teaching-data models.Source: More Cases Analysis · p. 10

Data Generation Process of PCoT

The PCoT data-generation process uses a three-step zero-shot prompting approach to decompose complex problems into simpler subproblems. It generates a skill lecture, derives a plan, and uses both to produce a step-by-step rationale.

Data Generation Process of PCoT

Loading 2305.03453v4…