PaperlayerView exact revision on arXiv

Source-linked AI summary

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

Shraman Pramanick, Rama Chellappa, Subhashini Venugopalan

arXiv:2407.09413v3cs.CLcs.AIcs.CV

TL;DR

Scientific-paper QA datasets have been limited in scale and focused on text, leaving figures and tables outside their scope. SPIQA provides a large-scale dataset and interleaved image-text tasks; experiments report gains from fine-tuning, while the dataset remains restricted to computer-science papers.

  • Problem

    Existing scientific-paper QA datasets are limited in scale and focus on textual content, although figures and tables contain information important to understanding research.

    Source: §1 · pp. 1–2

  • Method

    SPIQA curates 270K question-answer-rationale triplets and develops direct and retrieval-based CoT tasks over figures, tables, and paper text.

    Source: §1 · p. 2

  • Results

    Fine-tuning InstructBLIP and LLaVA on SPIQA significantly improves zero-shot performance; InstructBLIP gains 28 L3Score points on average over three test sets.

    Source: §5.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    SPIQA supplies training and evaluation splits for assessing and fine-tuning multimodal systems on complex scientific papers.

    Source: §1 · p. 2

    SPIQA covers computer-science papers only, so models trained on it may not perform well on other scientific literature.

    Source: §6 · p. 10

Abstract

from arXiv · show

Seeking answers to questions within long scientific research articles is a crucial area of study that aids readers in quickly addressing their inquiries. However, existing question-answering (QA) datasets based on scientific papers are limited in scale and focus solely on textual content. We introduce SPIQA (Scientific Paper Image Question Answering), the first large-scale QA dataset specifically designed to interpret complex figures and tables within the context of scientific research articles across various domains of computer science. Leveraging the breadth of expertise and ability of multimodal large language models (MLLMs) to understand figures, we employ automatic and manual curation to create the dataset. We craft an information-seeking task on interleaved images and text that involves multiple images covering plots, charts, tables, schematic diagrams, and result visualizations. SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits. Through extensive experiments with 12 prominent foundational models, we evaluate the ability of current multimodal systems to comprehend the nuanced aspects of research articles. Additionally, we propose a Chain-of-Thought (CoT) evaluation strategy with in-context retrieval that allows fine-grained, step-by-step assessment and improves model performance. We further explore the upper bounds of performance enhancement with additional textual information, highlighting its promising potential for future research and the dataset's impact on revolutionizing how we interact with scientific literature.

1 Introduction

Scientific-paper QA datasets are limited in scale and largely omit figures and tables, despite their importance to understanding research. SPIQA addresses this with a large-scale dataset and interleaved image-text QA tasks, alongside experiments and an LLM-based evaluation metric.

1 Introduction

  • Scientific-paper QA datasets are limited in scale and mostly use abstracts, conclusions, or paragraphs, overlooking figures and tables needed for comprehensive understanding.Reasoning across article figures and associated text also requires multimodal and long-context capabilities.Source: §1 · pp. 1–2
  • SPIQA contributes 270K question-answer-rationale triplets from 26K computer-science papers, with training, validation, and three evaluation splits.The papers span multiple computer-science domains.Source: §1 · p. 2
  • The three tasks test direct QA with figures and tables, direct QA with the full paper, and CoT QA that retrieves helpful figures before answering.The retrieval task supports step-by-step assessment of fine-grained reasoning and grounding.Source: §1 · p. 2
  • Experiments evaluate 12 prominent models, and LLMLogScore (L3Score) uses LLM confidence from log-likelihood token probabilities to assess answer equivalence.The paper also reports fine-tuning InstructBLIP and LLaVA 1.5 on SPIQA and significant improvement over zero-shot evaluation.Source: §1 · pp. 2–3

2 Related Works

Related work spans general document QA, scientific-paper QA, and standalone visual QA, but these strands do not jointly address reasoning across figures, tables, and paper text. SPIQA combines these sources and introduces a two-stage retrieval-based CoT task for finer-grained assessment.

2 Related Works

  • Table 1 compares scientific QA datasets by coverage of figures, tables, and full-paper text; dashes indicate unreported counts.Source: Table 1 · p. 3
  • Scientific-paper QA poses challenges beyond general document QA because systems must handle domain-specific terminology, theoretical implications, and experimental visualizations.Long-document QA has also drawn attention alongside LLM long-context reasoning.Source: §2 · p. 3
  • Scientific-paper QA ranges from templated cloze datasets to human-written sets, while QASA offers advanced free-form questions answerable from text paragraphs alone.Earlier human-written datasets often used abstracts and produced simple yes/no or short-span questions.Source: §2 · p. 3
  • Standalone scientific visual benchmarks focus on individual diagram types or figures and tables, while SPIQA asks questions requiring joint reasoning over figures, tables, and paragraphs.This integrated scope aims at a more holistic understanding of scientific documents.Source: §2 · p. 3
  • SPIQA distinguishes itself with open-ended computer-science QA combining visual content, textual content, and domain knowledge, plus a two-stage CoT task that retrieves relevant figures and tables before answering.This stepwise task supports finer-grained assessment of baseline reasoning capabilities.Source: §2 · pp. 3–4

3 SPIQA Dataset and Tasks

SPIQA combines paper text, figures, tables, and captions in a large computer-science QA benchmark. It defines three evaluation tasks and includes both automatically generated and human-written evaluation questions.

3.1 Collection Guidelines and Task Formulation

  • SPIQA covers article text, figures, tables, and captions, and annotates which visuals support each question for grounding and reasoning evaluation.Its three tasks provide figures and tables directly, the full paper, or retrieved relevant visuals before answering.Source: §3.1 · p. 4

3.2 SPIQA: Data Collection, Question Generation and Filtering

  • The corpus draws on peer-reviewed papers from 19 top-tier conferences published between 2018 and 2023, using TeX sources and PDF figure cropping.TeX provides high-resolution figures, table TeXs, and article text; PDFFigures 2.0 helps capture figures missed in TeX processing.Source: §3.2 · p. 4
  • Gemini 1.5 Pro generated 270,194 QAs from figure- or table-focused prompts, with manual checks for visual relevance, correctness, and appropriate specificity.Evaluation questions received additional filtering; annotators agreed on which questions to discard 88% of the time, and about 11% were removed.Source: §3.2 · p. 5
  • The human-written test-B and test-C sets contain 228 QASA questions and 493 QASPER questions whose answers benefit from figures or tables.The evaluation set was identified from evidence sentences and manually checked for whether the cited visuals helped answer each question.Source: §3.2 · p. 5

3.3 Dataset Analysis

  • SPIQA contains 270,194 QAs across 25,859 papers, alongside 152,487 figures and 117,707 tables.Questions from the same paper stay in the same split; training, validation, and test-A cover every source conference.Source: §3.2 · p. 4, §3.3 · p. 6
  • SPIQA questions average 12.98 words and answers 14.56 words, with substantial length variation across both.Test-A uses generated QAs, while test-B and test-C use human-written QAs.Source: §3.3 · p. 6

4 LLMLogScore (L3Score): An Improved Metric for Free-form QA

Free-form answers can differ in wording while remaining accurate, making conventional text-overlap metrics inadequate. L3Score evaluates answer meaning using an LLM’s yes/no log probabilities, and the paper presents examples where it is more appropriate than ROUGE and BERTScore.

4 LLMLogScore (L3Score): An Improved Metric for Free-form QA

  • L3Score uses GPT-4o to judge whether a candidate answer and ground truth have similar meaning in the question’s context, scoring from log-likelihood probabilities.The model is prompted to answer “yes” or “no” in a single word.Source: §4 · p. 7
  • L3Score renormalizes the log probabilities of the “yes” and “no” tokens so their probabilities sum to 1.The formula defines the score from the exponential of the “yes” log probability divided by the sum of the “yes” and “no” exponentials.Source: §4 · p. 7
  • Because GPT-4o exposes only its top-five log probabilities, a missing “yes” or “no” token may need approximation; if neither appears, L3Score is zero.The approximation uses the lower-probability top-five token and the probability mass remaining outside the top five.Source: §4 · p. 7
  • Figure 4 presents examples where L3Score is more appropriate than ROUGE and BERTScore for evaluating QA answers.The paper also provides further comparisons with other LLM-based metrics in Appendix H.Source: §4 · p. 7

5 Experiments

Experiments assess models across direct image-based, full-paper, and retrieval-based CoT QA setups. Results show gains from additional context, CoT prompting, and SPIQA fine-tuning, alongside persistent challenges on complex visual categories.

5.1 Experimental Seutp

  • Six open-source and six closed-source models are evaluated across direct QA with figures and tables, direct QA with full papers, and CoT QA.Open-weight models generally receive images one at a time in a multi-turn setup; closed models use full-resolution images, while open models use 224px images.Source: §5.1 · pp. 7–8

5.2 Main Results

  • Claude-3 achieves the highest L3Score on test-B at 49.54, 3.32 points above GPT-4o; GPT-4o performs consistently well on test-A and test-C.Among open-source systems, InternLM-XC leads on test-A and test-B, while CogVLM leads on test-C.Source: §5.2 · p. 8
  • Fine-tuned InstructBLIP and LLaVA 1.5 gain 28 and 26 average L3Score points over their zero-shot counterparts and perform almost as well as Gemini Pro Vision.The average is over three test sets.Source: §5.2 · p. 8
  • Gemini 1.5 Pro gains 8.31, 2.24, and 2.76 L3Score points on test-A, test-B, and test-C, respectively, with full-paper input versus figures and tables alone.The full-paper setup includes text, figures, tables, and captions, and requires long-context reasoning.Source: §5.2 · p. 9
  • GPT-4 Vision gains 6.70, 1.73, and 2.98 L3Score points on test-A, test-B, and test-C with CoT prompts versus direct QA.CoT QA retrieves reference images before generating an answer; the prompt guides step-by-step reasoning.Source: §5.2 · p. 9

5.3 Ablation studies and performance on sub-categories

  • Omitting captions lowers scores for every Gemini and GPT variant; GPT-4o loses 2 L3Score points on test-A.Full-resolution images generally perform better, and open-source models struggle with tables; GPT-4o scores 65.18 on tables, 8.50 points above GPT-4 Vision.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 Human Evaluation

  • On a 20% subset of test-A, the human evaluator retrieves the correct figure 94.89% of the time, about 8% above the best model.The paper notes that a more robust human baseline would require multiple evaluators with diverse expertise.Source: §5.4 · pp. 9–10, Table 7 · p. 9

5.5 Qualitative Results and Additional Analysis

  • L3Score correctly evaluates semantically valid answers that ROUGE-L and BERTScore score as zero, and its relative model comparisons remain consistent across scoring LLMs.The paper also reports that L3Score correlates with existing automated metrics, although its absolute value varies with the LLM used to compute it.Source: §5.5 · p. 10

6 Conclusion

SPIQA targets QA about complex figures and tables in scientific papers, and the paper introduces L3Score for evaluating free-form answers. The dataset and evaluation resources are released publicly, with coverage limited to computer science papers.

6 Conclusion

  • SPIQA is a large-scale dataset for QA about complex figures and tables, and L3Score evaluates free-form answers by their semantic context.Fine-tuning LLaVA and InstructBLIP on SPIQA improves over zero-shot evaluation.Source: §6 · p. 10
  • SPIQA covers computer science papers only, so models trained on it may not perform well on other scientific literature.Extending the dataset to other scientific domains is future work.Source: §6 · p. 10

A Dataset and Code Release

  • The dataset is released under CC BY 4.0; evaluation scripts and model responses are publicly accessible, while L3Score code and its library use Apache 2.0.The release includes the dataset card and README.Source: A Dataset and Code Release · p. 15

B Additional studies on L3Score and ablations.

Additional studies examine scorer choice, image resolution, and fine-tuning strategies. They report a resolution–cost trade-off and stronger gains from QA-based supervision than from caption-based unsupervised training.

L3Score with Different LLMs

  • Relative L3Score changes between models remain consistent across LLM scorers, but absolute values vary; the authors recommend using the same scorer for comparisons.GPT-4o scores about 40% higher than Gemini Pro Vision with both GPT-3.5 Turbo and GPT-4o scoring, though the difference between them varies.Source: B Additional studies on L3Score and… · p. 15, Table B.1 · p. 15

Image Resolution

Supervised vs. Unsupervised Tuning on SPIQA Training Set

C SPIQA vs. Existing Scientific QA Datasets

SPIQA differs from earlier scientific QA datasets by targeting large-scale, open-ended questions that require holistic use of figures, tables, and paper text. Its CoT setup also tests retrieval of relevant visual evidence before answer generation.

C SPIQA vs. Existing Scientific QA Datasets

  • Cloze-style benchmarks remove a named entity from a sentence, unlike open-ended questions readers ask while reading long scientific papers.Manually creating open-ended scientific QA is costly and requires expert annotators with domain-specific knowledge.Source: C SPIQA vs. Existing Scientific QA… · p. 16
  • Unlike QASA’s 1,798 questions on 112 AI/ML papers, SPIQA targets large-scale, open-ended QA across computer science domains and requires holistic understanding of figures, tables, and paper text.SPIQA’s CoT task has models identify helpful figures and tables before answering, enabling fine-grained assessment of reasoning.Source: C SPIQA vs. Existing Scientific QA… · pp. 16–17
  • SPIQA contains papers from 19 top-tier conferences published between 2018 and 2023, and its questions and answers vary in length.The dataset comprises 25,859 papers.Source: Figure D.1 · p. 17, Figure D.2 · p. 17

D Additional Dataset Analysis

SPIQA draws on papers from 19 computer-science conferences and contains 270,194 generated question-answer-rationale triplets. Its answer lengths vary, and the additional results compare full-paper and CoT QA with direct QA.

D Additional Dataset Analysis

  • SPIQA covers 25,859 papers from 19 conferences published between 2018 and 2023, spanning four broad computer-science groups.AI/ML contributes 46%, NLP 19%, computer vision and graphics 17%, and other areas 18%.Source: D Additional Dataset Analysis · p. 17
  • 270,194 question-answer-rationale triplets were generated with Gemini 1.5 Pro from papers’ figures, tables, and text.Questions, answers, and rationales average 12.98, 14.56, and 37.42 words, respectively.Source: D Additional Dataset Analysis · p. 17
  • 36.62% of answers contain five words or fewer, 56.70% contain 6–40 words, and the remainder exceed 40 words.The paper characterizes the set as including both direct and descriptive or explanatory QAs; rationales are released for future research despite not being used in the experiments.Source: D Additional Dataset Analysis · pp. 17–18
  • Full-paper and CoT QA both improve Gemini and GPT-4 performance over direct QA with figures and tables.Table E.2 reports BLEU scores for these tasks.Source: Table E.2 · p. 18

E Additional Quantitative Results

Additional results compare model performance across test splits, fine-tuning, and CoT or full-paper inputs. The appendices also describe L3Score and the test-question generation and filtering procedures.

E Additional Quantitative Results

  • GPT-4o achieves state-of-the-art scores on test-A and test-C, while Gemini 1.5 Flash leads test-B with 26.8 BLEU@1, over two points above other models.Open-source models generally underperform closed-source models, which the passage attributes primarily to training on natural images.Source: E Additional Quantitative Results · p. 18
  • Fine-tuning raises InstructBLIP-7B by 13.56 average BLEU@1 points and LLaVA-1.5-7B by 4.22 points across the three datasets.The passage describes these improvements as evidence of the training set’s potential for specialized scientific QA systems.Source: E Additional Quantitative Results · p. 19
  • On test-A, GPT-4o reaches 41.3 BLEU@1 with the full paper, while GPT-4 Vision gains 5.6 BLEU@1 points with CoT versus direct QA.Similar trends are reported for the other two test splits.Source: E Additional Quantitative Results · p. 19

F Prompt for L3Score Computation

G Detailed Annotation Guidelines and User Interfaces

  • For SPIQA test-A, Gemini 1.5 Pro generated image-based questions across approximately 26,000 papers and 270,000 images; 200 papers were selected as the test set and questions filtered for higher quality.Annotators marked which figures and tables helped answer questions to assess grounding and CoT reasoning. The interface asked whether a question was answerable from the figure and whether it should be kept or discarded, separately asking if the question or answer should be modified.Source: G Detailed Annotation Guidelines and… · p. 19, Figure G.3 · p. 21
  • Manual verification required figure- or table-based questions with correct, concise answers, avoiding trivial or overly specific questions and retaining only one similar question per paper.Questions answerable entirely from surrounding text were discarded.Source: G Detailed Annotation Guidelines and… · p. 19
  • Unclear answers were discarded, and questions saying “Based on the passage” were modified because evaluation presents all figures and tables together.Source: G Detailed Annotation Guidelines and… · pp. 19–20
  • A pilot of 150 questions found crowd workers lacked the domain expertise needed to judge question nuances, so authors filtered the final test-A set.The pilot compared annotations by two groups of three workers with those of the authors.Source: G Detailed Annotation Guidelines and… · p. 20

H Qualitative examples of the task and data in SPIQA

The SPIQA CoT task requires systems to select a useful image from a paper’s figures and tables, then answer the question. The examples illustrate this multi-image workflow and the dataset’s image density.

H Qualitative examples of the task and data in SPIQA

I Error Analysis

The error analysis separates image retrieval from answer comprehension: models can find the relevant figure yet still fail to answer correctly, especially when information is embedded in tables.

I Error Analysis

  • Models may retrieve the correct figure or table yet answer the question incorrectly.Source: I Error Analysis · p. 26
  • When tables are represented as images, models have difficulty parsing and comprehending their contents.Source: I Error Analysis · p. 26, Figure I.1 · p. 26
  • In the illustrated errors, L3Score correctly indicates that responses do not capture the expected answer.The figure marks correct metric evaluations in green and incorrect ones in red.Source: Figure I.1 · p. 26

Related papers

Loading 2407.09413v3…