PaperlayerView exact revision on arXiv

Source-linked AI summary

VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models

Chenyu Zhou, Mengdan Zhang, Peixian Chen, Chaoyou Fu, Yunhang Shen, Xiawu Zheng, Xing Sun, Rongrong Ji

arXiv:2406.10228v1cs.CVcs.AIcs.CL

TL;DR

The paper addresses the limits of narrow-context vision-language tasks for evaluating comprehension of long, distracting image-text inputs. It introduces IITC and the VEGA dataset, with ITA and multi-task, multi-scale training; its trained approach outstrips Gemini 1.5 Pro and GPT4V and reaches 85.8% image-association accuracy.

  • Problem

    Many MLLM applications and benchmarks focus on narrow visual and textual contexts, while document comprehension requires handling long interleaved content, distractions, and image-referenced answers.

    Source: §1 · p. 1

  • Method

    The paper introduces IITC and the scientific-content VEGA dataset, using ITA and multi-task, multi-scale training to support image-text comprehension.

    Source: §1 · p. 2

  • Results

    The trained approach outstrips Gemini 1.5 Pro and GPT4V, and Qwen-VL-Chat reaches 85.8% image-association accuracy.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    IITC makes following complex instructions and extracting key cues from long interleaved image-text scenarios explicit MLLM evaluation targets.

    Source: §1 · p. 2

Abstract

from arXiv · show

The swift progress of Multi-modal Large Models (MLLMs) has showcased their impressive ability to tackle tasks blending vision and language. Yet, most current models and benchmarks cater to scenarios with a narrow scope of visual and textual contexts. These models often fall short when faced with complex comprehension tasks, which involve navigating through a plethora of irrelevant and potentially misleading information in both text and image forms. To bridge this gap, we introduce a new, more demanding task known as Interleaved Image-Text Comprehension (IITC). This task challenges models to discern and disregard superfluous elements in both images and text to accurately answer questions and to follow intricate instructions to pinpoint the relevant image. In support of this task, we further craft a new VEGA dataset, tailored for the IITC task on scientific content, and devised a subtask, Image-Text Association (ITA), to refine image-text correlation skills. Our evaluation of four leading closed-source models, as well as various open-source models using VEGA, underscores the rigorous nature of IITC. Even the most advanced models, such as Gemini-1.5-pro and GPT4V, only achieved modest success. By employing a multi-task, multi-scale post-training strategy, we have set a robust baseline for MLLMs on the IITC task, attaining an $85.8\%$ accuracy rate in image association and a $0.508$ Rouge score. These results validate the effectiveness of our dataset in improving MLLMs capabilities for nuanced image-text comprehension.

1. Introduction

The introduction presents IITC as a test of comprehension across long, distracting interleaved image-text contexts, where models must answer questions and identify the relevant image. It introduces VEGA and a multi-task, multi-scale training strategy, reporting an 85.8% image-association accuracy baseline.

1. Introduction

  • IITC tests whether models can find relevant cues amid irrelevant or misleading text-image mixtures, understand long interleaved content, and identify the referenced image when answering.The introduction contrasts this with basic multimodal tasks focused on a narrow scope of visual content.Source: §1 · p. 1
  • IITC requires locating relevant text and images in context, answering accurately, and outputting the corresponding image index.The paper presents a benchmark for this task and describes it as the first benchmark MLLM for these challenges.Source: §1 · pp. 1–2
  • The training strategy combines ITA correspondence learning with progressively increasing context lengths and image counts to build robustness to redundant content.The authors state that simply adding more IITC question-answering samples is insufficient for accurate comprehension.Source: §1 · p. 2
  • VEGA is a scientific-content dataset with IITC and ITA subsets, reaching up to 8 images and 8,000 tokens of interleaved content.The IITC instructions ask about one image and require the model to specify its index.Source: §1 · p. 2
  • Evaluation on VEGA assesses current state-of-the-art models, including GPT4V, Gemini-1.5-pro, and Qwen-VL-Chat, and validates IITC's challenge.Source: §1 · p. 2
  • 85.8% image-association accuracy establishes a strong IITC baseline for Qwen-VL-Chat fine-tuned on VEGA with multi-task, multi-scale training.The authors describe this as a significant improvement.Source: §1 · pp. 2–3

2. Related Work

Related work surveys multimodal models and benchmarks for reasoning and specialized capabilities. It identifies multiple-choice formats as a possible source of hints that can let models bypass the reasoning process.

2.1. MLLMs

  • MLLM work spans visual-language architectures and training approaches, including Q-Former, instruction tuning, multimodal chain-of-thought, adapters, and multimodal generation.The section also notes Qwen-VL and InternVL as models supporting multi-image input, and uses Qwen-VL-Chat as its training base.Source: §2.1 · p. 3

2.2. MLLM Benchmarks

  • Benchmarks assess OCR, robustness, hallucination, nuanced reasoning, and domain-specific skills, but prevalent multiple-choice formats can let models use hints instead of reasoning.Source: §2.2 · p. 3

3. Proposed Method

VEGA defines IITC and ITA tasks, builds two scientific-content subsets from SciGraphQA, and trains Qwen-VL-Chat baselines on the resulting data.

3.1. Task Definition

  • IITC requires an answer and the relevant image index, while ITA matches shuffled image-text pairs; evaluation uses image association and text-quality metrics.IITC is assessed with correct-image rate, ROUGE, and BLEU; ITA uses the fraction of correctly matched pairs.Source: §3.1 · pp. 3–4

3.2. Data Collection

  • VEGA builds on SciGraphQA, extending visual-textual contexts and organizing the resulting data into IITC and ITA subsets.Source: §3.2 · p. 4

3.2.1 Subset for IITC

  • For IITC, the authors filter ambiguous questions, expand paper contexts with inserted images, and modify answers to identify the referenced image.They use the original paper’s sequence for test contexts and select 700 high-quality questions for the final test set.Source: §3.2.1 · pp. 4–5

3.2.2 Subset for ITA

  • ITA pairs images with text from disparate articles and trains across caption, first-mention, and expanded-context scales using sets of three or five images.Testing uses only extended pairs; the expanded text for each image is capped at 2,000 characters.Source: §3.2.2 · pp. 5–6

3.3. Dataset Statistics

  • VEGA contains 50k scientific-literature entries, over 200k question-answer pairs, and 400k images, with IITC contexts capped at 4k or 8k tokens.Each IITC category has roughly 200k training instances and about 700 test samples; each image counts as 256 tokens.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4. Baseline Model

  • The Qwen-VL-Chat baselines jointly train on IITC and ITA data at 4k and 8k token limits, using progressively harder image-count and context-length levels.Training uses roughly 340K data points; the visual encoder is frozen while the language model and adapter are tuned.Source: §3.4 · p. 7

4. Experiments

The VEGA evaluations find current MLLMs challenged by IITC and ITA, while the Qwen-VL-Chat model trained on VEGA achieves state-of-the-art results across both tasks. Ablations compare training strategies, datasets, and context construction methods.

4.1. Experimental Setup

  • The evaluation covers two open-source models and four proprietary models; some unanswered proprietary-model cases are excluded, and Qwen-Max inputs are capped at 6,000 tokens.Evaluation images were resized to 448 × 448 pixels.Source: §4.1 · p. 7

4.2. Main Results

  • The VEGA-trained Qwen-VL-Chat 7B achieves state-of-the-art results across all metrics on both IITC and ITA.The evaluation describes the tasks as challenging for mainstream state-of-the-art MLLMs.Source: §4.2 · pp. 7–8

4.3. Ablation Study

  • Adding multi-task and multi-scale training improves image-text association accuracy by 2.3%; without multi-scale training, ITA accuracy is near random guessing.Multi-task training jointly uses IITC and ITA, while multi-scale training uses three textual scales during ITA training.Source: §4.3 · p. 8, Table 3 · p. 8
  • Training on VEGA outperforms training on SciGraphQA on both SciGraphQA and IITC, while retaining traditional VQA processing capability.The comparison also reports that SciGraphQA-trained models struggle with IITC.Source: §4.3 · p. 8, Table 4 · p. 8
  • VEGA* underperforms the baseline on SciGraphQA and IITC; its context combines images with their initial mention paragraphs from several papers.The authors suggest that combining text and image context from the same paper may introduce extraneous material.Source: §4.3 · p. 8, Table 4 · p. 8

5. Conclusion

The paper presents VEGA and a multi-task, multi-scale approach using Qwen-VL-Chat to improve IITC. It reports performance above Gemini 1.5 Pro and GPT4V, and identifies several directions for future dataset and evaluation improvements.

5. Conclusion

  • The approach combines multi-task and multi-scale learning with VEGA in the Qwen-VL-Chat framework to improve interleaved comprehension and image-index generation.The conclusion describes the dataset as tailored to IITC in real-world applications.Source: §5 · p. 9
  • The approach outstrips Gemini 1.5 Pro and GPT4V on IITC performance.The conclusion presents this comparison as evidence of VEGA’s potential to improve IITC performance.Source: §5 · p. 9
  • Future dataset work will add subtler instructions, more image-text interaction in answers, broader data sources, and newer IITC evaluation methods.Source: §5 · p. 9

A. Training Data

VEGA-Base training uses the full IITC subset and selected portions of the ITA subset. Figure 6 visualizes the training-data composition.

A. Training Data

B. Question Filtering

The supplied question examples include broad prompts about graph implications and significance, alongside requests about specific graph features and experimental details.

B. Question Filtering

C. Case Study

The case studies show that all three models sometimes failed or succeeded together, while VEGA-Base-4k answered correctly in cases where Gemini-1.5-pro and GPT4V did not. The examples highlight deceptive text, similar images, and irrelevant text as challenges for interleaved image-text comprehension.

C. Case Study

  • Across four cases, all three models failed in Case 1 and succeeded in Case 2, while only VEGA-Base-4k answered correctly in Cases 3 and 4.The case-study figures mark the ground-truth pictures in blue, incorrect model references in red, and correct references in green.Source: C. Case Study · p. 12, Figure 8 · p. 14, Figure 10 · p. 16
  • Deceptive textual content misled the models and contributed to incorrect answers in the cases where all three failed.Source: C. Case Study · p. 12
  • VEGA-Base-4k navigated similar-image distractions and irrelevant text in Cases 3 and 4, interpreting image information to answer correctly.Source: C. Case Study · p. 12, Figure 12 · p. 18, Figure 14 · p. 20
Loading 2406.10228v1…