PaperlayerView exact revision on arXiv

Source-linked AI summary

SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation

Jonathan Roberts, Kai Han, Neil Houlsby, Samuel Albanie

arXiv:2405.08807v2cs.CV

TL;DR

LMMs’ ability to interpret scientific figures is not well characterised, despite the importance of figures in scientific research and the difficulty of objective evaluation. SciFIBench provides a curated multiple-choice benchmark and probes model alignment and reasoning faithfulness; current LMMs find it challenging, with leading models still below the human baseline.

  • Problem

    LMMs’ capacity to understand scientific figures is not well known, and objective evaluation is difficult because large-scale accurate annotations require substantial manual work and domain knowledge.

    Source: §1 · p. 1

  • Method

    SciFIBench uses arXiv figure-caption pairs, adversarially selected negatives, and human verification to create multiple-choice tasks, then evaluates LMMs and probes alignment and reasoning faithfulness.

    Source: §1 · p. 2, §4.7 · p. 9

  • Results

    SciFIBench is challenging: GPT-4o and Gemini 1.5 outperform VLM baselines but are beaten by the human baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors release SciFIBench to encourage progress in LMM scientific-figure interpretation and understanding research.

    Source: §1 · p. 2

    The benchmark’s limited size makes it of limited usefulness for fine-tuning or pre-training.

    Source: A Dataset intended uses and limitations · p. 16

Abstract

from arXiv · show

Large multimodal models (LMMs) have proven flexible and generalisable across many tasks and fields. Although they have strong potential to aid scientific research, their capabilities in this domain are not well characterised. A key aspect of scientific research is the ability to understand and interpret figures, which serve as a rich, compressed source of complex information. In this work, we present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories. The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control. We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark. Finally, we investigate the alignment and reasoning faithfulness of the LMMs on augmented question sets from our benchmark. We release SciFIBench to encourage progress in this domain.

1 Introduction

Scientific figure interpretation is important to research, but LMM capabilities in this area are not well characterised and objective evaluation is difficult. SciFIBench addresses this gap with a curated multiple-choice benchmark and evaluates LMMs, including their alignment and reasoning faithfulness.

Motivation

  • Scientific figures compress complex information, yet LMMs’ ability to interpret them is not well known and reliable objective evaluation is difficult.The authors also note that manually annotating a sufficiently large figure set with accurate descriptions is unfeasible and requires domain knowledge.Source: §1 · p. 1

Benchmark

  • SciFIBench uses arXiv figure-caption pairs to create 2,000 multiple-choice questions, with adversarially selected negatives and human verification for difficulty and answerability.The authors frame the benchmark around quality, evaluation efficiency, and robustness.Source: §1 · p. 2, Figure 1 · p. 2

Evaluation and contributions

  • The paper evaluates 28 open- and closed-source LMMs against human and VLM baselines, uses Gemini-Pro to parse model answers for automatic evaluation, and probes alignment and reasoning faithfulness.The contributions also include releasing SciFIBench to encourage further research.Source: §1 · p. 2

Key findings

  • Current LMMs find SciFIBench challenging: GPT-4o and Gemini 1.5 are the best-performing models, outperform VLM baselines, but are beaten by humans.Adversarial filtering increases question difficulty, while human filtering is crucial for high-quality, answerable questions.Source: §1 · p. 2

2 Related Work

Prior scientific-figure work includes question answering, figure classification, caption generation, and retrieval, while multimodal benchmarks cover other image domains and tasks. The paper positions its contribution as quantitative multiple-choice evaluation of LMMs interpreting scientific figures.

Scientific Figure Interpretation

  • Earlier scientific-figure benchmarks address reasoning questions or figure-type classification, while captioning and retrieval work examine other aspects of figure understanding.Examples include ChartQA, PlotQA, FigureQA, ACL-Fig, SciCap, and SciMMIR.Source: §2 · pp. 2–3
  • Prior LMM analysis on scientific figures was qualitative and limited to a small handful of figures; this work instead reports quantitative evaluation in a multiple-choice setting.The authors state that quantitative LMM evaluation for scientific-figure understanding had yet to be reported.Source: §2 · p. 3

LMM Benchmarks

  • Existing multimodal benchmarks span natural images, geospatial tasks, hallucination analysis, and visual mathematics, but target different image types or evaluation focuses.The authors also draw methodological inspiration from adversarial filtering in [30].Source: §2 · p. 3

3 SciFIBench

SciFIBench comprises two tasks and two source-based subsets of scientific figure-caption questions. Its curation combines embedding-based neighbor selection, difficulty filtering, and human checks to form a challenging, answerable dataset.

3 SciFIBench

  • SciFIBench contains 2,000 questions drawn from figures and captions extracted from arXiv papers, organized into two tasks and two source-based subsets.Table 1 describes the subsets in terms of category and source.Source: §3 · p. 3, Table 1 · p. 3

3.1 Tasks

  • Figure→Caption asks models to select the correct caption from five options, while Caption→Figure asks them to select the matching figure from five options.Source: §3.1 · p. 3

3.2 Curation methodology

  • For the CS source, the authors begin with SciCap’s single-sentence arXiv computer-science figure-caption pairs, filter out subfigures, normalize captions, and remove duplicates.The selected SciCap subset contains approximately 94,000 pairs from 2010–2020.Source: §3.2 · p. 3
  • Joint figure-caption embeddings and nearest neighbors support similarity filtering and construction of shuffled caption or figure choices for each task.The embeddings concatenate normalized text and image representations; the described vector has dimension 2048.Source: §3.2 · p. 4
  • Questions are ranked by negative-answer distance, then human reviewers select difficult but answerable items; model errors and category-balanced sampling inform the curated CS and General sets.The CS set limits questions from a paper dominating cs.AI to 10 per task, while the General set contains 500 questions per task.Source: §3.2 · p. 4
  • Curated questions are estimated to contain at most a few percent noise, and their adversarial distribution is more challenging than the noisy distribution.The noisy uncurated questions are estimated to have approximately 20–25% noise; lower mean L2 distance indicates harder questions.Source: §3.2 · pp. 5–6, Figure 4 · p. 5

4 Experiments

The experiments compare LMMs with human and VLM baselines on SciFIBench, then examine performance across negative-selection settings, data variants, figure properties, caption generation, and alignment. The strongest models still leave room for improvement, and model responses vary in how they follow instructions and react to answer annotations.

4.1 Baselines

  • The evaluation includes open- and closed-source LMMs, VLMs, and humans, using deterministic prompting and Gemini-Pro to extract answers from model outputs.Roughly half the LMM baselines accept interleaved text and images, enabling evaluation on Caption→Figure; answer-order shuffling changes performance by up to 5%.Source: §4.1 · p. 6, §4.2 · p. 6, §4 · p. 6

4.3 Main Results

  • 73.8% and 70.1% are GPT-4o and Gemini 1.5 Flash’s Figure→Caption scores; 65.4% and 66.1% are their Caption→Figure scores, respectively.These leading scores show that even frontier models leave room for improvement.Source: Table 2 · p. 7, §4.3 · p. 7
  • The weakest models hover around chance, and closed-source models outperform open-source models; random negatives usually yield higher accuracy than adversarial negatives.For open-source models on Caption→Figure, adversarial and random negative performance changes little; scores are near chance, suggesting the task is too challenging for them.Source: §4.3 · p. 7
  • Caption→Figure is harder overall; model size often fails to predict performance, while CLIP matches or exceeds leading open-source models and the mean human baseline exceeds all models.Model-size patterns differ by negative type: smaller checkpoints more often lead on adversarial negatives, while larger ones more often lead on random negatives.Source: §4.3 · p. 8

4.4 Curated vs. noisy data

  • Curated-data model rankings mostly persist on noisy questions, while Gemini 1.5 Flash’s few-shot performance depends on the prompt and can rise or fall versus zero-shot.With some prompts, noisy examples improve accuracy and make outputs follow the requested format more closely; with others, added examples reduce performance.Source: §4.4 · pp. 8–9

4.5 Error analysis

  • Model rankings remain broadly consistent across arXiv categories, but scores vary; most models do well on stat questions and worse on q-bio, with a slight decline as captions lengthen.The analysis finds no clear publication-year or figure-size effect, and refusal rates are low, though verbose models tend to be less decisive.Source: §4.5 · p. 9

4.6 Caption generation

  • Evaluators prefer captions generated by all tested closed-source models over the true captions, but prefer the true captions over every tested open-source model’s captions.Three evaluator models from different model families ranked shuffled outputs across 100 figure-caption samples, with strong agreement between evaluators.Source: §4.6 · p. 9, Table 3 · p. 9

4.7 Alignment

  • Marking the true caption as correct raises accuracy, while marking an incorrect caption as correct lowers it; responses to instructions to ignore the label vary by model.OmniLMM and TransCore-M show almost no change when told to ignore the annotation, whereas CogVLM changes drastically.Source: §4.7 · p. 10, Figure 10 · p. 10, Figure 9 · p. 10

5 Conclusions

The conclusion presents SciFIBench as a challenging benchmark for scientific figure interpretation and reports experiments on LMM, VLM, and human baselines. The appendix collects additional materials, implementation details, and results.

5 Conclusions

  • SciFIBench uses arXiv figure-caption pairs, adversarially selected hard negatives, and human verification to form a challenging, high-quality benchmark for scientific figure interpretation.The authors report benchmarking LMM, VLM, and human baselines and examining alignment and instruction following.Source: §5 · p. 10
  • The authors release SciFIBench for community use and hope it encourages further research in scientific figure interpretation.They describe the benchmark as supporting efficient future evaluation without extensive compute or API credits.Source: §5 · p. 10

Appendix

  • The appendix includes intended uses and limitations, data and code links, human-baseline and compute details, model versions, prompts, further results, and qualitative examples.It also includes examples of model outputs before and after automatic evaluation and generated captions.Source: Appendix · p. 16

A Dataset intended uses and limitations

SciFIBench is intended for evaluating LMM capabilities. The authors note that its focus on high-quality, feasible questions limits its volume and usefulness for fine-tuning or pre-training.

A Dataset intended uses and limitations

B.1 Data

The supplementary material describes SciFIBench’s dataset and code availability, plans for hosting and maintenance, and the procedure for its human baseline.

B.1 Data

  • SciFIBench is hosted on HuggingFace under a CC BY-NC-SA 4.0 license, with a dataset viewer and downloadable Croissant metadata.Source: §B.1 · p. 16

B.2 Code

  • Example code covers dataset interaction, inference on both tasks, and automatic LLM answer evaluation; the code is released under an MIT license.Source: §B.2 · p. 16

B.3 Hosting and maintenance plan

  • The authors plan to keep the dataset on HuggingFace, add a DOI for the camera-ready version, and maintain both dataset and code repositories.Source: §B.3 · p. 16

C Human baseline

  • The human baseline used a Google form with 50 SciFIBench CS questions, presented in the same multiple-choice order as for models; participants were uncompensated.Source: C Human baseline · p. 16

D Compute

The supplementary material gives inference compute requirements, API model versions, prompts for both benchmark tasks, and the procedure for parsing model answers.

D Compute

  • Typical open-source model inference on SciFIBench required one A100 80GB GPU for about 45 minutes; Emu2-Chat and IDEFICS-80b-Instruct required two A100s.Source: D Compute · p. 17

E API model versions

  • The appendix lists the specific API versions used, including GPT-4o-2024-05-13 and Gemini-1.5-Pro-preview-0409.Source: E API model versions · p. 17

F.1 Task prompts

  • The two task prompts ask models to select the caption best describing an image or the image best matching a caption, with inputs inserted as placeholders.Source: §F.1 · p. 17

F.2 Automatic evaluation

  • Gemini-Pro was prompted to extract the multiple-choice answer letter from each model output, returning None when no answer was found.Source: §F.2 · p. 18

G.1 Human baseline

The extended human baseline compares five participants with leading LMMs on a small SciFIBench CS question subset, while noting limitations in representativeness and possible learning across questions.

G.1 Human baseline

  • Five humans answered randomly sampled questions from the SciFIBench CS dataset, with Table 4 reporting accuracy on a 25-question-per-task subset.Source: §G.1 · p. 19, Table 4 · p. 19
  • The mean human beats the leading LMMs, and the median human beats or equals them, although some individual humans are outperformed by an LMM.Source: Table 4 · p. 19, §G.1 · p. 19
  • The small, potentially unrepresentative sample had substantial variance, and answering 50 questions in sequence may have influenced later responses.Source: §G.1 · p. 19

G.2 Curated vs. noisy data rankings

The appendix compares model rankings on curated versus noisy data, category-level performance, and qualitative examples of model outputs and Gemini-Pro postprocessing.

G.2 Curated vs. noisy data rankings

  • Model rankings showed only minor variations between the curated and noisy CS datasets.Figure 11 illustrates the ranking consistency described for the two datasets.Source: §G.2 · p. 19, Figure 11 · p. 19

G.3 Performance Across Categories

  • Average accuracy for selected models differed by 22.5% between the best-performing LG category and the worst-performing AI category, while model rankings remained fairly consistent across categories.Table 5 expands the Figure →Caption category results to include scores for the plotted and additional models.Source: §G.3 · p. 20, Table 5 · p. 20

H Qualitative results

  • The qualitative-results section presents examples of automatic answer extraction and model reasoning on Figure →Caption and Caption →Figure questions.The reasoning examples appear in Figures 15–18.Source: H Qualitative results · p. 21

H.1 Automatic postprocessing

  • Gemini-Pro extracted correct predictions from unformatted LMM outputs in the examples, though some outputs required more involved reasoning to identify the answer choice.CogVLM’s output in Figure 14 is one such case.Source: §H.1 · p. 21, Figure 13 · p. 22, Figure 14 · p. 23

H.2 Model reasoning

The section presents examples of LMM reasoning on both figure-to-caption and caption-to-figure questions, with outputs that vary in how they interpret graph details and match captions to figures.

H.2 Model reasoning

  • The examples remove the answer-format constraint to elicit reasoning steps and show selected model outputs for both Figure →Caption and Caption →Figure tasks.Each example is organized around the question and the model outputs.Source: §H.2 · p. 24
  • For a threshold-versus-nodes graph, one output identifies the axes and selects the threshold δ caption, while another reverses the axes in its description.The target caption describes explored nodes as a function of threshold δ.Source: §H.2 · p. 27
  • For a rank-25 CMC caption, an example matches the figure labeled Rank-25, while outputs for a control-signal caption differ on which image matches.The control-signal example includes a response choosing E and another choosing D.Source: §H.2 · pp. 30, 32

I Generated Captions

The section gives an example of model-generated captions for a figure whose caption concerns mean reaction time versus number of commands across conditions.

I Generated Captions

  • The example figure’s caption describes mean reaction time versus the number of commands for each condition.The example is identified as a generated-caption result.Source: I Generated Captions · p. 33, Figure 19 · p. 33
  • Several model outputs describe reaction time as a function of the number of commands and identify Glass and Glass+Skin conditions.Claude 3 Haiku, Gemini Pro Vision, and GPT-4 Turbo each mention the relationship and conditions.Source: I Generated Captions · p. 33
  • Other outputs vary in specificity, including one that says the reaction-time increase is less pronounced for Glass+Skin and others that omit or misstate the conditions.The responses include descriptions that reduce the figure to a general relationship or a linear trend.Source: I Generated Captions · p. 33
Loading 2405.08807v2…