PaperlayerView exact revision on arXiv

Source-linked AI summary

LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, Liqiang Nie

arXiv:2311.11860v2cs.CV

TL;DR

Existing MLLMs can have insufficient visual knowledge because they mainly use vision encoders pretrained on coarsely aligned image-text pairs. LION combines staged incorporation of spatial-aware knowledge with soft prompting of image tags, and reports gains across vision-language benchmarks, including around 5% accuracy on VSR over InstructBLIP.

  • Problem

    Many MLLMs rely on vision encoders pretrained on coarsely aligned image-text pairs, leaving visual knowledge insufficiently extracted and reasoned about.

    Source: §1 · p. 1

  • Method

    LION progressively incorporates spatial-aware knowledge through region-level tasks and stage-wise tuning, while soft prompting image tags as high-level semantic evidence.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    LION reports superiority across image captioning, VQA, and visual grounding, including around 5% accuracy over InstructBLIP on VSR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported evaluations support LION's use of dual-level visual knowledge across a range of vision-language tasks.

    Source: §5 · p. 9

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have endowed LLMs with the ability to perceive and understand multi-modal signals. However, most of the existing MLLMs mainly adopt vision encoders pretrained on coarsely aligned image-text pairs, leading to insufficient extraction and reasoning of visual knowledge. To address this issue, we devise a dual-Level vIsual knOwledge eNhanced Multimodal Large Language Model (LION), which empowers the MLLM by injecting visual knowledge in two levels. 1) Progressive incorporation of fine-grained spatial-aware visual knowledge. We design a vision aggregator cooperated with region-level vision-language (VL) tasks to incorporate fine-grained spatial-aware visual knowledge into the MLLM. To alleviate the conflict between image-level and region-level VL tasks during incorporation, we devise a dedicated stage-wise instruction-tuning strategy with mixture-of-adapters. This progressive incorporation scheme contributes to the mutual promotion between these two kinds of VL tasks. 2) Soft prompting of high-level semantic visual evidence. We facilitate the MLLM with high-level semantic visual evidence by leveraging diverse image tags. To mitigate the potential influence caused by imperfect predicted tags, we propose a soft prompting method by embedding a learnable token into the tailored text instruction. Comprehensive experiments on several multi-modal benchmarks demonstrate the superiority of our model (e.g., improvement of 5% accuracy on VSR and 3% CIDEr on TextCaps over InstructBLIP, 5% accuracy on RefCOCOg over Kosmos-2).

1. Introduction

Existing MLLMs often rely on image-level visual supervision, which can leave their visual understanding insufficient. LION adds spatial-aware and semantic visual knowledge through staged instruction tuning and soft prompting, and reports improvements across vision-language tasks.

1. Introduction

  • Existing MLLMs mainly adapt vision encoders pretrained with image-level supervision, which the paper identifies as limiting visual understanding and contributing to erroneous or hallucinated responses.Replacing or tuning the encoder may require massive-data pretraining or risk catastrophic forgetting.Source: §1 · p. 1
  • LION progressively incorporates spatial-aware knowledge through stage-wise tuning and mixture-of-adapters, and uses soft prompting to integrate image tags while mitigating errors from imperfect predictions.The paper describes these additions as promoting image-level and region-level VL tasks and alleviating hallucination.Source: §1 · p. 2
  • Around 5% accuracy on VSR and 3% CIDEr on TextCaps over InstructBLIP, and around 5% accuracy on RefCOCOg over Kosmos-2; LION leads across the reported tasks.Figure 2 presents the comparison across a wide range of VL tasks.Source: §1 · p. 2, Figure 2 · p. 2
  • LION evaluates image captioning, visual question answering, and visual grounding, with POPE and MMBench evaluations addressing hallucination and perceptual abilities.The paper reports positive effects from tag integration across various VL tasks.Source: §1 · p. 2

2. Related Work

Prior MLLMs commonly align visual features to LLMs through bridge modules, while visual-grounding work adds region-level understanding. LION distinguishes itself by studying interactions between region-level and image-level tasks.

2.1. Multimodal Large Language Models

  • Prior MLLMs use linear, MLP, or more complex bridge modules, while LION investigates region-level tasks as a way to enrich the MLLM visual branch.The cited prior work also includes empirical study of vision encoders from a pretraining perspective.Source: §2.1 · p. 2
  • LION combines holistic Q-Former features with fine-grained vision-aggregator features, and routes task-dependent adapter fusion for image-level or region-level tasks.The router operates in the frozen LLM.Source: Figure 3 · p. 3

2.2. Visual Grounding in the field of MLLMs

  • Visual grounding links particular regions to textual descriptors; unlike prior grounding-focused instruction tuning, LION studies its interaction and conflict with image-level tasks.The paper names Shikra [4], Kosmos-2 [35], Ferret [47], and Pink [45] as works using grounding datasets for region-level understanding.Source: §2.2 · p. 3

3. LION

LION enriches MLLMs with fine-grained spatial-aware knowledge and high-level semantic evidence through region-level visual tasks, stage-wise training, a vision aggregator, and soft prompting of image tags.

3.1.1 Reorganizing Visual Grounding Tasks

  • Visual grounding uses referring-expression comprehension and generation tasks, with object positions expressed in natural-language instructions and square-bracketed coordinates.The tasks are constructed from Visual Genome, which associates local image areas with short descriptions.Source: §3.1.1 · pp. 3–4

3.1.2 The Stage-Wise Instruction-tuning Strategy

  • A task-specific router aggregates main-branch and adapter features to reduce interference between image-level and region-level tasks with distinct input-output modes.The method treats each adapter as an expert and uses a specific router function for each task type.Source: §3.1.2 · p. 4
  • Stage-wise training separates image-level and region-level instruction-tuning before joint training, and the authors report mutual promotion between the task types.The three stages tune image-level components, then region-level components, then a unified model with a Mixture-of-Adapters.Source: §3.1.2 · pp. 4–5, Figure 4 · p. 4

3.1.3 Vision Aggregator

  • The vision aggregator combines hidden features from multiple vision-encoder layers through two transformer blocks to produce visual features for fine-grained understanding.The selected input features come from middle encoder layers.Source: §3.1.3 · p. 5

3.2. Soft Prompting of High-Level Semantic Visual Evidence

  • LION injects RAM-generated tags for objects, scenes, actions, and attributes through a trainable soft prompt that guides adaptive use of the tags.The tailored instruction uses the prompt as a hint and avoids extra feature-alignment computation by using the model’s output labels.Source: §3.2 · p. 5, Figure 5 · p. 5

4. Experiments

The experiments evaluate LION on image-level and region-level vision-language tasks, then examine its training strategy, visual-knowledge components, hallucination results, and qualitative outputs.

4.1. Evaluations on Image-Level VL Tasks

  • LION ranks first on 7 of 9 image-level benchmarks and second on the other 2, with superior performance across zero-shot evaluation benchmarks.Table 1 covers image captioning and VQA; captioning uses CIDEr, Visual Dialog uses MRR, and other VQA tasks use top-1 accuracy.Source: Table 1 · p. 6, §4.1 · pp. 6–7

4.2. Evaluations on Region-Level VL Tasks

  • Fine-tuned LION achieves the best average and most REC evaluation-set results, outperforming Ferret despite using a simple vision aggregator and existing datasets.REC measures whether the predicted box has intersection-over-union of at least 0.5 with the ground truth.Source: Table 2 · p. 6, §4.2 · p. 7
  • In zero-shot REC, LION improves over Kosmos-2 and Pink on most evaluation sets, with test-A RefCOCO and RefCOCO+ as exceptions.The authors attribute the particularly strong RefCOCOg results to handling its more flowery referring expressions.Source: Table 2 · p. 6, §4.2 · p. 7

4.3. Ablation Study

  • Single-stage tuning damages average REC performance; staged tuning improves it, while adding a router improves held-out and REC performance with slight held-in degradation.The three-stage process separates image-level and region-level learning before combining them in the final stage.Source: Table 3 · p. 6, §4.3 · p. 8
  • Spatial knowledge significantly improves OKVQA, GQA, IconQA, and VSR, while image tags substantially raise Flickr30K and AOKVQA performance.The authors associate the first group with region-level understanding and spatial reasoning, and the second with broader semantic knowledge.Source: Table 4 · p. 7, §4.3 · p. 8

4.4. Evaluations on Object Hallucination and MMBench

  • POPE results are superior especially in popular and adversarial settings, and MMBench results support the authors’ conclusion that progressive fine-grained knowledge alleviates hallucination.Table 5 evaluates object hallucination using F1 as its major metric.Source: Table 5 · p. 7, Table 6 · p. 8, §4.4 · p. 8

4.3. Ablation Study

  • Removing the vision aggregator degrades REC performance; Figure 6 reports that the module overall improves results across eight RefCOCO-family evaluation sets.The authors connect feature aggregation to extracting fine-grained spatial-aware visual knowledge.Source: §4.3 · p. 7, Figure 6 · p. 8

4.5. Qualitative Analysis

  • Qualitative examples contrast LION with InstructBLIP and Shikra on fine-grained attributes, spatial reasoning, and referring expressions, showing fewer hallucinated responses.The examples include correct color and character identification and a referring-object localization case where Shikra misunderstands “a soda bottle.”Source: §4.5 · pp. 8–9, Figure 7 · p. 9

5. Conclusion

LION combines fine-grained spatial-aware knowledge with high-level semantic visual evidence to address limited visual information in MLLMs. Experiments support its performance across image captioning, VQA, and visual grounding.

5. Conclusion

  • LION combines fine-grained spatial-aware visual knowledge and high-level semantic visual evidence based on region-level and image-level tasks.A stage-wise instruction-tuning strategy progressively incorporates fine-grained knowledge to mitigate conflict between the task types.Source: §5 · p. 9
  • Image tags provide high-level semantic evidence, and soft prompting is used to mitigate potential influence from incorrect tags.The tags are incorporated as visual evidence through a learnable token embedded in tailored text instructions.Source: §5 · p. 9
  • Experiments validate LION’s superiority on image captioning, VQA, and visual grounding tasks.Source: §5 · p. 9

A. Experimental Details

LION uses a frozen vision backbone, an aggregator and MLP to connect visual features to the LLM, with image tags and training data incorporated during a three-stage process.

A. Experimental Details

  • The frozen EVA-CLIP ViT-G/14 backbone feeds a two-layer cross-attention vision aggregator, then a two-layer MLP projects its output into the LLM feature space.The projected features are concatenated with Q-Former output and textual inputs; adapters have hidden dimension 64.Source: A. Experimental Details · p. 11
  • LION-4B and LION-12B use FlanT5-XL (3B) and FlanT5-XXL (11B), respectively.Image tags come from RAM-14M with a Swin-Large backbone, 384 × 384 images, a 0.8 threshold, and 4585 tag categories.Source: A. Experimental Details · p. 11
  • Training comprises three stages, with distinct batch sizes, step counts, and learning rates; the vision aggregator uses a constant 1e-5 learning rate in Stage 2.Stages 1 and 3 use batch size 64 for 10 epochs; Stage 2 uses batch size 256 for 10 epochs.Source: A. Experimental Details · pp. 11–12
  • Stage 1 uses image-level tasks and dialogue data, Stage 2 uses Visual Genome for REC and REG, and Stage 3 unifies the datasets and inserts image tags.The image-level training data include captioning, VQA, visual question generation, and LLaVA-Instruct-150K.Source: A. Experimental Details · p. 12

B. Instruction Templates

The appendix describes instruction templates for image-level and region-level tasks and gives task-specific evaluation formats and dataset summaries.

B.1. Task Templates for Instruction-Tuning

  • Image-level instruction templates follow InstructBLIP, while region-level templates follow Shikra and were generated with GPT-4.The appendix displays a selection of templates for each listed task.Source: §B.1 · p. 12

B.2. Instructions for Evaluation

  • Evaluation options are arranged alphabetically, and REC evaluation randomly selects a template from the training instruction lists.Source: §B.2 · p. 12
  • OKVQA, AOKVQA, and GQA use image-question prompts followed by a short-answer instruction.Source: §B.2 · p. 12
  • COCOCap, Flickr30K, and TextCaps use a prompt requesting a short image description.Source: §B.2 · p. 12
  • IconQA and VSR use distinct formats: a question prompt for IconQA and a true-or-false statement prompt for VSR.Source: §B.2 · p. 12
  • Visual Dialog evaluation supplies dialog history and a question before requesting a short answer.Source: §B.2 · p. 12
  • Table 7 lists the training datasets used for instruction tuning, while Table 8 summarizes the evaluation datasets.Source: Table 7 · p. 13, Table 8 · p. 13
  • Table 9 presents instruction-template examples and defines {expr} as the REC expression and {BBox} as the user-specified location’s bounding box.Source: Table 9 · p. 14

Related papers

Loading 2311.11860v2…