PaperlayerView exact revision on arXiv

Source-linked AI summary

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Xin He, Longhui Wei, Lingxi Xie, Qi Tian

arXiv:2401.03105v2cs.CVcs.MM

TL;DR

MLLMs rely on CLIP-like encoders whose descriptions can leave visual details unrepresented. IVE combines complementary encoders and visual tools to enrich visual inputs, and the paper reports improved visual perception across multimodal tasks.

  • Problem

    CLIP-like encoders do not fully describe visual inputs, leaving MLLMs with an information-loss problem that limits visual perception.

    Source: §1 · p. 3

  • Method

    IVE combines semantic, low-level, and document-related encoders with visual tools that extract structural data used as prompts alongside fused visual embeddings.

    Source: §1 · p. 3, Multi-task Encoders · p. 6, Structural Knowledge Enhancement · p. 8

  • Results

    IVE is reported to recognize image content more accurately in general multimodal dialogue and to achieve competitive results on DocVQA against recent state-of-the-art methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper concludes that integrating visual experts improves MLLM visual perception across a wide range of multimodal dialogue datasets.

    Source: §5 · p. 15

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are experiencing rapid growth, yielding a plethora of noteworthy contributions in recent months. The prevailing trend involves adopting data-driven methodologies, wherein diverse instruction-following datasets are collected. However, a prevailing challenge persists in these approaches, specifically in relation to the limited visual perception ability, as CLIP-like encoders employed for extracting visual information from inputs. Though these encoders are pre-trained on billions of image-text pairs, they still grapple with the information loss dilemma, given that textual captions only partially capture the contents depicted in images. To address this limitation, this paper proposes to improve the visual perception ability of MLLMs through a mixture-of-experts knowledge enhancement mechanism. Specifically, we introduce a novel method that incorporates multi-task encoders and visual tools into the existing MLLMs training and inference pipeline, aiming to provide a more comprehensive and accurate summarization of visual inputs. Extensive experiments have evaluated its effectiveness of advancing MLLMs, showcasing improved visual perception achieved through the integration of visual experts.

1 Introduction

The introduction identifies limited visual perception in CLIP-like encoders as an information-loss problem for MLLMs. IVE combines complementary encoders and visual tools, and is reported to improve perception across multimodal tasks.

1 Introduction

  • Recent MLLM improvements have largely relied on collecting varied instruction data and fine-tuning models, a learning framework shared by prior approaches.Examples include general multimodal, document-understanding, referring-expression, and other instruction datasets.Source: §1 · p. 2
  • CLIP-like encoders and detailed instruction data still leave image details undescribed, limiting MLLMs’ visual perception and response quality.Prior work commonly maps CLIP-like visual features into the language space through a lightweight projection module.Source: §1 · p. 3
  • IVE combines a CLIP-like semantic encoder with low-level and document-related encoders, and uses visual tools to extract structural data as hard prompts alongside latent embeddings.The structural data can include instance categories and locations or text inside images.Source: §1 · p. 3
  • IVE recognizes image content more accurately in general multimodal dialogue and achieves competitive results on DocVQA compared with recent state-of-the-art methods.The authors attribute the improved visual perception to integrating visual experts.Source: §1 · p. 3

2 Related Work

The related work reviews vision-and-language pre-training architectures and the common MLLM pipeline. It positions IVE as an approach that aggregates visual experts to address limited visual perception.

2.1 Vision-and-Language Pre-training

  • Two-stream pre-training methods independently encode image and text; current MLLMs predominantly use their visual modules to encode image embeddings.One-stream methods instead process image and text with a single transformer.Source: §2.1 · p. 4

2.2 Multimodal Large Language Models

  • Most MLLMs use CLIP-like encoders, an abstractor, a projection layer, and a pretrained LLM, while differing in the instruction data used for tasks such as general dialogue, document analysis, and referring questions.The passage describes these systems as constrained by CLIP-like encoders’ limited perception.Source: §2.2 · p. 4
  • IVE takes a different approach by aggregating available visual experts to provide a more comprehensive description of visual inputs.This approach focuses on visual perception rather than relying only on distinct instruction datasets.Source: §2.2 · p. 4

3 Our Approach

IVE extends the usual visual-perception, projection, and language-generation pipeline with complementary visual encoders and tool-derived structural knowledge. It uses these components during training and inference, followed by a three-stage training strategy.

3.1 Preliminaries

  • MLLMs project visual features into language space before the LLM generates a response to the query.The process is represented as a visual perception module, a projection module, and an LLM.Source: §3.1 · p. 5

3.2 Incorporating Visual Experts into MLLMs

  • RAM and Grounding DINO detect objects and locations, while EasyOCR detects image text; these structural cues are fed to the LLM alongside fused embeddings.The extracted structural knowledge is treated as prompts.Source: §3.2 · p. 5, Figure 2 · p. 6, Structural Knowledge Enhancement · p. 8
  • IVE combines semantic, low-level, and document-related encoders to capture complementary information from visual inputs.The semantic encoder uses EVA-CLIP and Q-Former; the low-level encoder uses VQGAN with Perceiver Resampler; the document encoder uses Pix2Struct.Source: Multi-task Encoders · pp. 6–7
  • IVE uses structural knowledge during both training and inference, aiming to mitigate tool noise while using the tools’ informative cues.The paper contrasts this with approaches that restricted visual tools to inference.Source: Structural Knowledge Enhancement · pp. 8–9

3.3 Training Pipeline

4 Experiments

The experiments evaluate IVE across VQA and multimodal benchmarks, with direct-transfer, dataset-specific fine-tuning, and ablation studies. Results report gains over recent methods and examine how the visual encoders and structural knowledge affect performance.

4.1 Datasets

  • Stage 2 training combines general, OCR, document/chart, grounding, captioning, and multimodal instruction datasets; Stage 3 fine-tunes on specific datasets.Evaluation covers VQAv2, OKVQA, TextVQA, OCRVQA, DocVQA, ChartQA, WTQ, and the MME Benchmark.Source: Training Dataset · p. 10, Evaluation Dataset · p. 10

4.2 Implementation Details

  • IVE uses EVA-CLIPg with a Q-Former for semantic information, VQGAN for low-level information, and Pix2Struct-Large for document information, producing 128 visual tokens for LLaMA2-chat.The three encoders contribute 32, 32, and 64 tokens, respectively; Perceiver Resamplers summarize the latter two encoders’ latent embeddings.Source: Model Configuration · pp. 10–11

4.3 Direct-transfer performance on Visual Question Answer

  • 60.3% on OKVQA exceeds mPLUG-Owl2’s 57.7%, while IVE reaches 62.0% on TextVQA and 71.1% on OCRVQA, outperforming Qwen-VL-Chat by 0.5% and 0.6%, respectively.The direct-transfer evaluation also reports consistent improvements over recent approaches on DocVQA and WTQ.Source: §4.3 · pp. 11–12, Table 2 · p. 12

4.4 Fine-tuning on Visual Question Answer

  • After dataset-specific fine-tuning, IVE improves over direct transfer by 5.2% on VQAv2 and 4.9% on OKVQA, and outperforms Pix2Struct by 3.6% on OCRVQA and 9.7% on ChartVQA.IVE also shows competitive results against CogVLM. On MME, it scores 1455.6 in perception and 384.1 in cognition, with superior stability across these two tasks versus mPLUG-Owl2 and LLaVA-1.5.Source: §4.4 · pp. 12–13

4.5 Ablation Study

  • Adding the low-level encoder improves results across datasets over the semantic-only encoder, and adding the document encoder raises TextVQA from 44.0% to 46.3% and DocVQA from 40.2% to 43.6%.The ablations use 50% of the mixed Stage 2 dataset for efficiency.Source: Effectiveness of Multi-task Encoders · pp. 13–14, §4.5 · p. 13, Table 5 · p. 13
  • Structural knowledge used only at inference degrades performance on some datasets, including VQAv2 and DocVQA, whereas using it in both training and inference improves results across datasets.With ground-truth structural knowledge on sampled VQAv2, using it in both phases yields only 0.4% gains over inference-only use; the authors attribute the broader benefit to learning to identify relevant information while mitigating noise.Source: Effectiveness of Structural Knowledge… · p. 14, Table 6 · p. 14

5 Conclusion

The conclusion frames information loss as a limitation of MLLMs and presents IVE as a visual-information enhancement approach. Experiments evaluate it across multimodal dialogue datasets, while future work will explore a unified interactive model with more visual experts.

5 Conclusion

  • The paper identifies information loss as a persistent limitation in current MLLMs.IVE is presented as a response aimed at enhancing their visual perception ability.Source: §5 · pp. 14–15
  • IVE aggregates available visual information through a mixture-of-experts mechanism in both training and inference.The conclusion describes this as the first such work.Source: §5 · p. 15
  • Experiments across a wide range of multimodal dialogue datasets evaluate IVE, and future work will explore unified interactive MLLMs with more visual experts.Source: §5 · p. 15

A Multi-task Instruct Tuning Data.

Appendix A summarizes the Stage 2 multi-task instruction-tuning data. The paper says these multimodal datasets are collected to enhance generalization across multimodal dialogue scenarios.

A Multi-task Instruct Tuning Data.

B Ablation Studies on Structural Knowledge Enhancement

The ablation examines structural knowledge on MME position perception: Grounding DINO supplies instance coordinates representing spatial relationships, and using this knowledge in both training and inference improves accuracy.

B Ablation Studies on Structural Knowledge Enhancement

C Visualizations.

The visualizations illustrate how IVE’s encoders and structural knowledge affect responses, alongside examples comparing its visual understanding with other models.

C Visualizations.

  • Adding a low-level information encoder to the semantic encoder benefits detailed recognition, while the document-related encoder enhances understanding of documents and charts.Source: C Visualizations · pp. 19–20
  • Automatically generated structural knowledge can cause incorrect responses, but using it in both training and inference helps IVE resist noise and answer correctly.Figure 3 contrasts results without structural knowledge and with it in both stages; Figure 4 visualizes module variants.Source: C Visualizations · p. 20, Figure 3 · p. 20, Figure 4 · p. 21
  • Examples show IVE identifying characters, providing richer details, reading screen text, and interpreting flow charts and tables compared with mPLUG-Owl2 and QWen-VL-Plus.Source: C Visualizations · p. 20, Figure 5 · p. 22
  • In the referring-grounding example, IVE identifies the categories and coordinates of three animals.Source: C Visualizations · p. 23

Related papers

Loading 2401.03105v2…