PaperlayerView exact revision on arXiv

Source-linked AI summary

SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs

Shengzhi Li, Nima Tajbakhsh

arXiv:2308.03349v1cs.CLcs.AIcs.CV

TL;DR

Interpreting scientific graphs takes time and depends on paper context, while caption-prediction datasets can produce incomplete or unhelpful descriptions. SciGraphQA builds contextual, open-vocabulary multi-turn dialogues about academic graphs; fine-tuning on the dataset improves LLaVA-13B’s reported CIDEr score over the same model without fine-tuning.

  • Problem

    Scientific graphs are time-consuming to interpret in context, and caption-prediction datasets can produce descriptions that are incomplete or unhelpful.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    SciGraphQA uses a commercial LLM to generate multi-turn graph question-answer dialogues from figures and their captions, paper text, and other graph context.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    CIDEr score was 0.268 for fine-tuned LLaVa-13B versus 0.153 for the same model without fine-tuning.

    Source: §4 · p. 8

  • Takeaways & Limitations

    SciGraphQA is presented as a resource for developing multimodal large language models.

    Source: §5 · p. 9

    Most evaluated and fine-tuned models used a fixed CLIP/ViT vision encoder, which may limit spatial awareness for graph questions.

    Source: §5 · p. 9

Abstract

from arXiv · show

In this work, we present SciGraphQA, a synthetic multi-turn question-answer dataset related to academic graphs. SciGraphQA is 13 times larger than ChartVQA, the previously largest chart-visual question-answering dataset. It is also the largest open-sourced chart VQA dataset with non-synthetic charts. To build our dataset, we selected 290,000 Computer Science or Machine Learning ArXiv papers published between 2010 and 2020, and then used Palm-2 to generate 295K samples of open-vocabulary multi-turn question-answering dialogues about the graphs. As context, we provided the text-only Palm-2 with paper title, abstract, paragraph mentioning the graph, and rich text contextual data from the graph itself, obtaining dialogues with an average 2.23 question-answer turns for each graph. We asked GPT-4 to assess the matching quality of our question-answer turns given the paper's context, obtaining an average rating of 8.7/10 on our 3K test set. We evaluated the 0-shot capability of the most popular MLLM models such as LLaVa, mPLUGowl, BLIP-2, and openFlamingo's on our dataset, finding LLaVA-13B being the most performant with a CIDEr score of 0.08. We further enriched the question prompts for LLAVA by including the serialized data tables extracted from the graphs using the DePlot model, boosting LLaVA's 0-shot CIDEr to 0.15. To verify the validity of our dataset, we also fine-tuned LLaVa using our dataset, reaching a substantially higher CIDEr score of 0.26. We anticipate further accuracy improvement by including segmentation mask tokens and leveraging larger LLM backbones coupled with emergent prompting techniques. Our code and data are open-sourced.

1 Introduction

Scientific graphs are information-rich but time-consuming to interpret, and readers understand them in the context of a paper’s text. SciGraphQA turns that interactive, contextual process into a multi-turn question-answering dataset and benchmark for multimodal large language models.

1 Introduction

  • Scientific graphs require substantial effort to interpret because their abstract components carry semantic meaning and are typically read alongside the paper’s abstract and relevant paragraphs.The introduction contrasts graphs’ trend lines and color-coded legends with natural images’ objects, positions, and interactions.Source: §1 · p. 1
  • SciGraphQA is a multi-turn question-answering dataset drawn from scientific literature and graphs, designed as a benchmark for multimodal large language models.The paper also evaluates current models and introduces a baseline trained on the dataset.Source: §1 · p. 1

2 Related Work

The related work contrasts synthetic, fixed-vocabulary or caption-focused graph QA datasets with SciGraphQA’s open-vocabulary multi-turn questions about real academic graphs. It also reviews chart-to-table systems and multimodal models before describing their evaluation on scientific graphs.

2.1 Graph Visual Answering datasets

  • Earlier graph QA datasets used synthetic data and fixed-vocabulary templates, while PlotQA introduced open-vocabulary questions but retained synthetic charts and restricted chart types and question variety [1] [2] [5] [3].ChartQA used web-crawled charts and human annotation, with a fine-tuned T5 generating two-thirds of its questions and answers from human-written summaries [4].Source: §2.1 · p. 2
  • SciGraphQA uses real-world academic graphs for open-vocabulary multi-turn QA, scaling ChartQA-style real-world chart VQA to 295K question-answer pairs from 21.9 (13X).Its questions and answers were generated from figures, captions, context, and paper metadata using Palm-2.Source: §2.1 · pp. 2–3

2.2 Chart VQA expert systems

  • DePlot converts charts into linearized tables with titles, legends, and interpolated values, addressing text-reading limits in MLLMs that are especially consequential for science graphs [10] [11].The passage describes DePlot combined with an LLM as surpassing VisionTAPAS with OCR input on ChartQA and PlotQA [9] [10].Source: §2.2 · p. 3

2.3 Multi-modal large language models (MLLM)

  • Because MLLMs were trained on natural rather than text-heavy images, the study evaluates them zero-shot on scientific graphs and fine-tunes a baseline on SciGraphQA [16].The subsection also surveys differing modality-bridging designs in BLIP-2, Flamingo, LLaVA, and mPLUG-Owl [15] [16] [9] [18].Source: §2.3 · pp. 3–4

3 Methodology

SciGraphQA is built from context-rich, multi-turn dialogues about academic graphs, then used to evaluate zero-shot and fine-tuned multimodal models.

3.1 Dataset Construction

  • The dataset combines graph figures, captions, OCR text, paper titles, abstracts, and mentioning paragraphs with two in-context examples to generate 295K dialogues using PALM2.Prompts focused the conversations primarily on the graph, and the entries were curated after filtering.Source: §3.1 · p. 4
  • Context-conditioned generation produced broad questions and component queries, as well as follow-ups tied to earlier question-answer exchanges.This differs from LLaVA’s fixed-question prompting approach.Source: §3.1 · p. 4
  • A keyword heuristic removed questions lacking graph-related terms, while the authors also planned to release the unfiltered question-answers.The authors note that naturally occurring conversations can include follow-ups about abstract concepts.Source: §3.1 · p. 4
  • 295K filtered dialogues contain 59.1 million tokens and average 2.23 turns, with 111K samples containing at least three turns.Figure 2 plots the distribution of question-answer turns.Source: §3.1 · p. 4, Figure 2 · p. 6
  • 86% of answers received GPT-4 ratings of 8.5/10 or higher in the 3K test set, while 6.4% received ratings of 7/10 or lower.The authors interpret the ratings as confirming the validity of their generation and filtering process overall.Source: §3.1 · p. 4

3.2 Zero-shot evaluation

  • Zero-shot evaluation gives candidate models only the image and question; an augmented prompt variant prepends graph tables extracted by DePlot.The test set contains 3K image-question-answer pairs, and evaluation uses BLEU-4, CIDEr, and ROUGE.Source: §3.2 · pp. 4, 6

3.3 Fine-tuning

  • LLaVA-13B was fine-tuned first on all 295K samples, then on a 30K subset with prompts augmented by DePlot-extracted tables.The authors selected LLaVA-13B because it had the highest accuracy among the models in zero-shot evaluation.Source: §3.3 · p. 6

4 Results

The results compare multimodal models in zero-shot and fine-tuning settings, examining model scale, DePlot prompt augmentation, and training-set size. Fine-tuning LLaVA-13B on SciGraphQA improves its CIDEr score over the same model without fine-tuning.

4 Results

  • Across zero-shot and fine-tuning evaluations on a 3K-sample test set using BLEU-4, ROUGE, and CIDEr, performance increased proportionally to LLM-backbone size; BLIP2, with a 2.7B backbone, had least accuracy, while LLaVA-13B showed superior performance.The authors caution that scaling alone may not be a one-size-fits-all solution, and that adapter design and training-data choice play equally important roles.Source: §4 · pp. 6, 8
  • Within the 7B range, model performance differs, which the authors link to adapter designs and training datasets.They conclude that adapter design and training-data choice matter alongside model size.Source: §4 · pp. 6, 8
  • Randomly adding 3, 6, or 9 in-context examples did not significantly improve OpenFlamingo over zero-shot evaluation.The authors attribute OpenFlamingo’s underperformance to its retrieval-based RICE scheme, which they could not replicate.Source: §4 · p. 8
  • DePlot table prompts boost LLaVA-13B more than mPLUG-Owl-7B, while DePlot + GPT-3.5 beats most baselines except LLaVA-13B.The authors highlight expert chart-to-table augmentation as important.Source: §4 · p. 8, Table 2 · p. 8
  • On the SciGraphQA dataset, the SciGraphQA-baseline, a LLaVA-13B model fine-tuned on the dataset’s training set, outperformed the same model without fine-tuning, with CIDEr scores of 0.268 versus 0.153.Source: §4 · p. 8, Table 2 · p. 8
  • The study did not perform model architecture search or hyperparameter tuning.The authors also note that the fine-tuning baseline may be constrained by trainable-parameter count and LLaVA’s linear projector.Source: §4 · pp. 8–9
  • 50% of the training set delivered the largest improvement; using the full set also helped, and 30K DePlot-enhanced samples consistently raised CIDEr.The authors suggest LoRA use may limit gains from additional data; an early projector-loss plateau points to potential benefits from more trainable parameters or a stronger adapter.Source: §4 · p. 9, Figure 5 · p. 9

5 Discussion

SciGraphQA offers a large, context-rich dataset of multi-turn scientific-graph dialogues, while the discussion identifies constraints in model adaptation and evaluation. The authors describe LoRA and prompt augmentation as ways to address adaptation challenges.

5 Discussion

  • 300K dialogues from academic publications make SciGraphQA 13 times larger than ChartQA, with responses averaging 8.7/10 in GPT-4 quality ratings.The 8.7/10 rating was measured on a test set.Source: §5 · p. 9
  • Fixed CLIP/ViT vision encoders may limit spatial awareness for graph questions; segmentation-based masks are proposed as a possible improvement.mPLUG-owl was the exception to using a fixed encoder; the authors hypothesize that projecting segmentation masks into the LLM token space could help.Source: §5 · p. 9
  • Conventional NLP metrics compare reference answers with predictions; GPT-4 evaluation faced inconsistent ratings, only 100 prompts, and possible limits on reproducibility.The authors therefore report CIDEr, ROUGE, and BLEU-4 as freely accessible, open, and reproducible metrics.Source: §5 · pp. 9–10
  • LoRA mitigates fine-tuning cost, and prompt augmentation addresses remaining adaptation issues for out-of-distribution scientific graph QA.The authors identify fine-tuning cost, hyperparameter and input-quality sensitivity, and catastrophic forgetting as challenges.Source: §5 · p. 10

6 Conclusion

SciGraphQA is an open dataset of multi-turn conversations about real-world scientific graphs, generated with extensive paper and graph context. The authors present it as a resource for pre-training and instruction tuning MLLMs.

6 Conclusion

  • SciGraphQA comprises 295K open-vocabulary multi-turn samples drawn from over 290,000 academic papers and real-world, non-synthetic graphs.The papers are described as Computer Science and Machine Learning ArXiv papers.Source: §6 · p. 10
  • A commercial LLM generated dialogues using paper titles, abstracts, captions, and pertinent paragraphs to capture explanations of complex graphs.The authors contrast this contextual approach with VQA datasets focused on simple facts or graph-composition questions.Source: §6 · p. 10
  • The authors present the open dataset as a resource for pre-training and instruction tuning MLLMs on real-world scientific graphs.Source: §6 · p. 10

Palm-2-chat prompt

The prompt supplies graph-related text and asks for a challenging question and detailed answer. It directs the model to reason from graph data and context, then explain trends and relationships in natural language.

Palm-2-chat prompt

  • The prompt combines OCR-extracted text, the figure caption, and the first paragraph that mentions the figure as context for interpreting the graph.Source: Palm-2-chat prompt · pp. 11–12
  • It asks the model to pose a challenging question based on the supplied sources and provide a detailed answer requiring graph-data understanding and contextual reasoning.The prompt encourages guides and scholarly perspectives based on the graph and its context.Source: Palm-2-chat prompt · p. 12
  • Questions should avoid directly revealing specific graph details, while explanations should use natural language to describe trends, prominent points, and relationships between datasets.Source: Palm-2-chat prompt · p. 12

Related papers

Loading 2308.03349v1…