PaperlayerView exact revision on arXiv

Source-linked AI summary

SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions

Sameera Horawalavithana, Sai Munikoti, Ian Stewart, Henry Kvinge, Karl Pazdernik

arXiv:2307.01139v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Scientific alignment of existing foundation models with scientific concepts and goals remains underexplored, while human-curated multimodal instructions are scarce. SciTune trains LLaMA-based multimodal models from scientific publications, with reported gains on scientific visual tasks and ScienceQA.

  • Problem

    Aligning existing foundation models with scientific disciplines, concepts, and goals is underexplored, and high-quality human-curated multimodal data are scarce.

    Source: §1 · pp. 1–2

  • Method

    SciTune trains LLaMA and CLIP-based models on human-curated multimodal instructions from scientific publications, then can tune them for scientific reasoning.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    LLaMA-SciTune models outperform state-of-the-art vision-language models on scientific visual tasks, and the ScienceQA model records 90.03% accuracy versus human performance of 88.40%.

    Source: §3.2 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    Human-generated scientific multimodal instructions remain valuable for science tasks despite their lower volume and relative scarcity compared with synthetic data.

    Source: §1 · p. 2, §5 · p. 9

    The experiments exclude instruction-tuned LLaMA variants and focus on base LLaMA decoders.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present \textit{SciTune} as a tuning framework to improve the ability of LLMs to follow multimodal instructions generated from scientific publications. To test our methodology, we train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in SciCap and VisText benchmarks. In comparison to the models that are finetuned with synthetic data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark. Our results demonstrate that human-generated scientific multimodal instructions remain highly valuable in tuning LLMs to perform well on science tasks, despite their lower volume and relative scarcity compared to synthetic data. We publicly release the SciTune codebase https://github.com/pnnl/scitune.

1 Introduction

Multimodal instruction tuning has expanded LLM capabilities, but scientific alignment remains underexplored amid scarce high-quality human-curated data and concerns about synthetic instructions. SciTune uses human-curated scientific multimodal instructions and reports strong results on scientific visual-understanding and reasoning tasks.

1 Introduction

  • Earlier multimodal instruction models extended language-only instruction models to visual-grounded reasoning tasks.Source: §1 · p. 1
  • High-quality multimodal data are scarce, and synthetic instructions may be biased, ungrounded, inaccurate, or less aligned with human values.The introduction also notes concerns about recursively generated data and synthetic models meeting standards in some scientific subdomains.Source: §1 · p. 1
  • The paper identifies a gap in methods aligning existing foundation models with scientific disciplines, concepts, and goals, and asks how far human-curated scientific multimodal instructions can achieve that alignment.Source: §1 · p. 2
  • SciTune builds on LLaVA with two stages: scientific concept alignment across visual and textual signals, followed by multimodal scientific instruction tuning.The trained models use LLaMA and CLIP; the paper reports strong ScienceQA and scientific image-understanding results.Source: §1 · p. 2

2 Methodology

SciTune uses human-curated scientific publication content to align visual and textual signals, then adapts the resulting model to scientific reasoning. Its methodology combines an early-fusion instruction format, a modular vision-language architecture, and autoregressive multimodal training.

2.1 Scientific Multimodal Instructions

  • SciTune instructions use early fusion: a system message supplies role and context, a sampled visual-grounded question supplies the instruction, and multimodal data supplies the content.Source: §2.1 · p. 2
  • The human-curated instructions come from scientific publications, with SciCap providing figures, captions, and relevant paragraphs across eight scientific categories.The training split contains 333,472 examples; the validation split is used for evaluation.Source: §2.1 · p. 2
  • SciCap examples become instructions for captions, figure types, OCR, and paragraph mentions, with paragraph mentions generally longer and more varied than figure-specific captions.Source: §2.1 · p. 2

2 Methodology

  • The framework aligns scientific visual and textual signals, supports zero-demonstration visual grounding, and can be further tuned for scientific reasoning.Source: Figure 1 · p. 3, §2.2 · p. 4

2.2 Multimodal Architecture

  • The architecture pairs a LLaMA decoder with a CLIP vision encoder and trains a linear-projection adapter while keeping both pretrained models frozen.The adapter architecture is described as modular, and experiments use LLaMA 7B and 13B variants.Source: §2.2 · p. 3
  • The experiments omit instruction-tuned LLaMA variants to focus on base decoders and compare against baselines built on base LLaMA.Source: §2.2 · p. 3
  • Training models instruction likelihood as conditional multimodal-token probabilities, and the model can generate figure type, caption, OCR, and paragraph content in one turn.The visual encoder output is projected into the language model, and the instruction is sampled from visual-grounded questions.Source: §2.2 · p. 4

3 Experiments

The experiments evaluate scientific concept alignment on zero-shot figure-type generation and captioning, then test multimodal reasoning on ScienceQA. LLaMA-SciTune outperforms human performance on average in ScienceQA and improves on comparison models in the visual tasks.

3.1.1 Scientific Figure Type Generation

  • 57% performance improvement over standalone CLIP in zero-shot figure-type classification shows stronger alignment of scientific visuals with figure types.SciTune uses the same CLIP vision encoder, while its multimodal adapter is optimized to align figure types with plots.Source: §3.1.1 · pp. 4–5, Table 1 · p. 5

3.1.2 Scientific Figure Captioning

  • LLaMA-SciTune-SciCap outperforms BLIP on both BLEU and ROUGE for generated captions in SciCap and VisText.The comparison uses BLIP trained on more than 14M image-text pairs.Source: §3.1.2 · p. 5, Table 2 · p. 5

3.2 Scientific Multimodal Reasoning Task Performance

  • 90.03% accuracy for LLaMA-SciTune-ScienceQA-13B (CTOM) exceeds the 88.40% human result on ScienceQA and outperforms humans in four subgroupings.The model reaches performance comparable to LLaVA, which used twice the training data and sometimes GPT-4 inference support.Source: §3.2 · pp. 5–6, Table 3 · p. 6
  • 86.11 average performance for the 7B CTOM variant exceeds 85.11 for the caption-only C variant, supporting the value of interleaved multimodal training data.CTOM includes captions, figure types, OCR, and figure mentions.Source: §3.2 · p. 6, Table 3 · p. 6
  • CTOM’s LLaMA-SciTune-ScienceQA-13B model has nearly 5% performance advantage over its 7B variant, an advantage 5x bigger than LLaVA’s reported 7B-to-13B scaling advantage.The authors suggest that a larger language decoder helps improve multimodal reasoning performance.Source: §3.2 · p. 6
  • Performance drops substantially when a lecture appeared five or fewer times in training, then recovers quickly after at least ten views; the drop is worse for 7B than 13B.The authors recommend testing on completely unseen data to assess robustness in other domains.Source: §3.2 · p. 7, Table 4 · p. 7
  • For incorrect answers, lecture text is generated more accurately than solution text, and 13B lecture ROUGE is 0.924 for incorrect versus 0.861 for correct answers.The authors say this pattern could indicate lecture memorization without successful application during the solution stage.Source: §3.2 · p. 7, Table 5 · p. 8
  • Manual analysis found linguistic weaknesses in commonsense retrieval and word semantics, while visual weaknesses included counting and retrieving object properties.Correct answers could still contain incorrect solutions: around 8% for 7B C and 2% for 7B CTOM in the sampled examples.Source: §3.2 · pp. 6–8

4 Related Work

Prior multimodal instruction-tuning work adapted language models for visual reasoning through learnable adapters or image-text instruction data. LLaVA and LLaVAR extended these approaches with large instruction datasets and OCR-focused training.

4 Related Work

  • LLaMA-Adapter used zero-initialized gated attention to add learnable multimodal prompts at higher LLaMA layers, while Adapter-V2 distributed learnable parameters across all layers.MiniGPT-4 combined a frozen Vicuna LLM and vision encoder with a projection layer, then fine-tuned on a curated visual conversation dataset.Source: §4 · p. 8
  • LLaVA used data-reformation techniques to create multimodal instructions from image-text pairs, including 158K unique instruction examples generated from ChatGPT/GPT-4.Its training also included 595K image-text pairs filtered from CC3M.Source: §4 · p. 8
  • LLaVAR extended LLaVA for text-rich images using additional image-OCR and GPT-4-generated conversation data.The added data comprised 422K image-OCR examples and 16K conversations.Source: §4 · p. 8

5 Conclusion

SciTune uses human-generated multimodal instructions from scientific publications to align LLMs with scientific concepts and goals. The resulting models perform strongly on scientific visual tasks and ScienceQA.

5 Conclusion

  • LLaMA-SciTune-SciCap outperformed state-of-the-art vision-to-language models in scientific visual classification and figure-caption generation.LLaMA-SciTune-ScienceQA surpassed human performance on the ScienceQA benchmark.Source: §5 · p. 8
  • SciTune instructions are curated by humans from scientific publications’ visual signals and accompanying textual signals.Examples include plots, charts, equations, captions, OCR, and paragraph mentions.Source: §5 · p. 8
  • Human-generated scientific multimodal datasets are smaller than synthetic datasets but provide reliable ground truth for tuning LLMs, leading to better generalization and downstream science-task performance.Source: §5 · pp. 8–9

A SciTune Multimodal Instructions

The appendix points to examples of SciTune instructions used in concept-alignment training and to questions used to sample sI instructions.

A SciTune Multimodal Instructions

B Training Details

The training details specify adapter-training settings and the separate ScienceQA fine-tuning used for comparison with LLaVA.

B Training Details

  • SciTune instruction adapter training used one epoch, batch size 128, learning rate 0.002, and context length 2048.The 7B and 13B variants took 6.5 and 11.2 hours, respectively, on 8 A100 GPUs.Source: B Training Details · p. 10
  • For a fair comparison with LLaVA, the models were fine-tuned on ScienceQA training examples for 12 epochs.Source: B Training Details · p. 10

C LLaMA-1 and LLaMA-2 Base Model Comparison

The authors repeat the CTOM pipeline with LLaMA-2 (13B) but observe no ScienceQA performance advantage over LLaMA-1, so they use LLaMA-1 for the remaining experiments.

C LLaMA-1 and LLaMA-2 Base Model Comparison

  • Table 6 distinguishes the system message, natural-language instruction, figure type, caption, OCR, and paragraph mentions in SciTune examples.Source: Table 6 · p. 11
  • The LLaMA-2 base model shows no ScienceQA performance advantage over LLaMA-1, despite LLaMA-2’s reported improvements.The authors leave explaining this difference with more empirical results to future work.Source: C LLaMA-1 and LLaMA-2 Base Model… · p. 12
  • The authors repeat the entire CTOM pipeline with the LLaMA-2 (13B) decoder and use LLaMA-1 for the rest of the experiments unless noted otherwise.Source: C LLaMA-1 and LLaMA-2 Base Model… · pp. 10, 12

D Visual Grounded Task Performance

The section presents generated captions for SciCap images and identifies the caption sources used for comparison.

D Visual Grounded Task Performance

E ScienceQA Chain of Thought Reasoning Examples

Figures 3 and 4 provide ScienceQA test examples generated by LLaMA-SciTune, including multimodal and unimodal question-answering outputs.

E ScienceQA Chain of Thought Reasoning Examples

Loading 2307.01139v2…