PaperlayerView exact revision on arXiv

Source-linked AI summary

MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, Yu Liu

arXiv:2404.13046v2cs.CV

TL;DR

Vision encoders can vary in performance across image-content domains, and no single encoder dominates all types of understanding. MoVA adaptively routes and fuses task-specific vision experts using a coarse-to-fine approach. The paper reports significant gains over current state-of-the-art methods across challenging benchmarks, while noting potential degradation when context-relevant experts fail.

  • Problem

    Vision encoders show inconsistent performance across tasks, and no single encoder dominates diverse image-content understanding.

    Source: §1 · pp. 1–2

  • Method

    MoVA combines context-aware routing of task-specific vision experts with fine-grained expert fusion through MoV-Adapter.

    Source: §1 · p. 2

  • Results

    MoVA reports significant performance gains over current state-of-the-art methods across a wide range of challenging benchmarks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    MoVA flexibly leverages context-relevant vision experts while avoiding biased information from irrelevant experts.

    Source: §5 · p. 10

    Performance may degrade when context-relevant vision experts fail, and hallucination may occur in MLLMs.

    Source: §5 · p. 10

Abstract

from arXiv · show

As the key component in multimodal large language models (MLLMs), the ability of the visual encoder greatly affects MLLM's understanding on diverse image content. Although some large-scale pretrained vision encoders such as vision encoders in CLIP and DINOv2 have brought promising performance, we found that there is still no single vision encoder that can dominate various image content understanding, e.g., the CLIP vision encoder leads to outstanding results on general image understanding but poor performance on document or chart content. To alleviate the bias of CLIP vision encoder, we first delve into the inherent behavior of different pre-trained vision encoders and then propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts according to the user instruction, input image, and expertise of vision experts. This benefits from the powerful model function understanding ability of the large language model (LLM). In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge from various experts. This coarse-to-fine paradigm effectively leverages representations from experts based on multimodal context and model expertise, further enhancing the generalization ability. We conduct extensive experiments to evaluate the effectiveness of the proposed approach. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods in a wide range of challenging multimodal benchmarks.

1 Introduction

MLLMs combine visual representations with LLM capabilities, but vision encoders can perform inconsistently across image tasks, limiting general multimodal understanding. The paper motivates adaptive selection and fusion of task-specific vision experts, proposing MoVA and reporting gains across challenging benchmarks.

1 Introduction

  • MLLMs project visual representations into an LLM, but encoder training data and optimization targets can yield inconsistent performance across tasks, including weak grounding and OCR.The introduction presents task-specific experts as a way to support generalization across image domains.Source: §1 · pp. 1–2
  • Across seven vision encoders, task-specific experts perform best in their proficient areas, while plain concatenation can underperform a single relevant expert because irrelevant expertise introduces bias.The introduction gives DINOv2's weaker performance on text-oriented tasks as an example.Source: §1 · p. 2
  • MoVA uses coarse-grained, context-aware routing to select relevant vision experts and fine-grained MoV-Adapter fusion to extract and integrate task-specific representations.Routing draws on the image, user instruction, and expert relevance; the adapter uses MoE cross-attention and dynamic expert-wise soft weights.Source: §1 · p. 2
  • MoVA reports significant performance gains over current state-of-the-art methods across challenging multimodal benchmarks.The evaluation includes MLLM benchmarks, visual question answering, visual grounding, and biomedical understanding.Source: §1 · p. 3

2 Related Work

Related work describes MLLMs as aligning visual features with language representations through off-the-shelf encoders and LLMs. It traces efforts to improve CLIP's limited location and relation understanding through fine-tuning or additional specialized vision encoders.

2 Related Work

  • MLLMs align visual features with linguistic representations using off-the-shelf vision encoders and LLMs, while CLIP is widely used in these systems.LLaVA uses an MLP projector to align visual tokens from a frozen CLIP encoder with the LLM embedding layer.Source: §2 · p. 3
  • CLIP has strong classification and recognition ability but limited downstream location and relation understanding, prompting encoder fine-tuning and specialist additions.Examples include Qwen-VL's joint optimization for grounding and OCR, SPHINX's use of DINOv2, and Vary's chart- and document-focused encoder.Source: §2 · p. 3

3 MoVA Methodology

MoVA combines a base vision encoder and task-specific vision experts with LLM-based coarse routing and fine-grained fusion. Its training uses pretraining followed by supervised finetuning.

3.1 Overview

  • MoVA routes relevant vision experts with an LLM, then uses the MoV-Adapter to extract and integrate their representations into the base visual representation.The enhanced visual representation and instruction tokens are fed to the LLM to generate a response.Source: §3.1 · p. 3

3.2 Coarse-grained Context-aware Expert Routing

  • At inference, the LLM selects experts using the image, question, and expert descriptions; routing annotations for training mark experts whose language-modeling loss is below the base model’s.The annotations are generated offline, and training replaces inference-time routing with those annotations for efficiency.Source: §3.2 · pp. 4–5

3.3 Fine-grained Expert Fusion with MoV-Adapter

  • The MoV-Adapter aligns selected expert features, extracts them through cross-attention, and uses dynamic gating to weight and integrate expert knowledge.Its blocks also use a shared BERT instruction token and transformer updates.Source: §3.3 · pp. 5–6

3.4 Training Paradigm

  • MoVA is pretrained on 15M diverse visual-instruction samples, then supervised-finetuned on visual instruction and domain-specific question-answering data.Pretraining optimizes the adapter and base vision encoder while preserving the initial LLM; during supervised finetuning, task-specific vision experts are frozen and other components are jointly optimized.Source: §3.4 · pp. 6–7

4 Experiments

MoVA is evaluated across MLLM benchmarks, VQA, visual grounding, and medical VQA, with comparisons and ablations examining its performance and design. The experiments report broad benchmark gains alongside the effects of routing and adapter choices.

4.1 Implementation Details

  • Training uses one-epoch pretraining followed by supervised finetuning, and the MoV-Adapter uses three transformer blocks.The base vision encoder input resolution is 672×672.Source: §4.1 · p. 7

4.2 MLLM Benchmarks

  • MoVA-7B achieves the best performance across seven MLLM benchmarks among similarly complex open-source models, processing only 20% of LLaVA-NeXT-7B’s image tokens.MoVA-34B is on par with Gemini-Pro and outperforms Qwen-VL-Plus on five MLLM benchmarks.Source: §4.2 · p. 7

4.3 Visual Question Answering

  • MoVA achieves state-of-the-art performance across general and text-oriented VQA benchmarks, including matching 18B CogAgent on text-oriented benchmarks with a smaller model.MoVA-7B exceeds SPHINX-2k with Vicuna-13B by 4.2% on VQAv2 and 1.9% on GQA.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Visual Grounding

  • MoVA-7B scores 90.22% on RefCOCO+ val, 2.46% above UNINEXT-H, and MoVA-34B further pushes visual-grounding performance.Table 5 reports Acc@0.5 on the RefCOCO REC task.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 Medical Visual Question Answering

  • MoVA-7B scores higher than LLaVA-Med and LLaVA-1.5 on VQA-RAD and SLAKE under the described one-epoch comparison.Each sample was seen once during training, and the comparison includes the one-epoch-finetuned LLaVA-Med variant.Source: §4.5 · p. 9, Table 6 · p. 9

4.6 Ablation Study

  • Random routing, using all experts, or replacing MoV-Adapter with linear layers lowers performance; three adapter blocks outperform two or four, and uniform gating also hurts.The routing and adapter component ablations are reported in Tables 7 and 11.Source: §4.6 · pp. 9–10, Table 7 · p. 9, Table 11 · p. 9
  • Dynamic K consistently outperforms constant K, while expert-combination enumeration gives comparable performance with fewer models for routing-data construction.The comparisons are reported in Tables 8 and 9.Source: §4.6 · p. 9, Table 8 · p. 9, Table 9 · p. 9
  • On 105 human-verified open-scenario samples, the reported results indicate that context-aware routing preserves expert-routing generalization.A lightweight MLP classifier fails to generalize in this setting.Source: §4.6 · pp. 9–10, Table 10 · p. 9

5 Conclusion

The conclusion attributes cross-domain generalization challenges to vision-encoder bias and presents MoVA’s adaptive routing and fusion as its response. It also notes hallucination and failure of context-relevant experts as limitations.

5 Conclusion

  • MoVA combines context-aware expert routing with fine-grained MoV-Adapter fusion to use context-relevant vision representations while avoiding bias from irrelevant experts.The paper reports significant performance gains over current state-of-the-art methods across a wide range of benchmarks.Source: §5 · p. 10
  • Hallucination may occur in powerful MLLMs, and failures of context-relevant vision experts may degrade performance.The authors identify both as limitations for future work.Source: §5 · p. 10

A Appendix / supplemental material

The appendix material identifies the vision-expert configurations and the datasets used for MoV-Adapter pretraining.

A Appendix / supplemental material

  • Table 12 lists MoVA vision-expert configurations and marks methods that use a convolution layer to compress output features.Source: Table 12 · p. 18
  • Table 13 describes datasets used in MoV-Adapter pretraining and defines placeholders for object categories, expressions, bounding boxes, and points.MMC-Instruction and ScigraphQA use their original questions as instructions.Source: Table 13 · p. 18

A.1 Vision Experts

The supplied appendix passages point to the vision-expert configurations and routing-prompt model descriptions.

A.1 Vision Experts

  • The vision-expert configurations use publicly available official checkpoint weights.Source: §A.1 · p. 18
  • Table 17 contains the model descriptions used in the routing prompt.Source: §A.1 · p. 18

A.2 Training Data Details

MoVA training uses pretraining and finetuning stages, with different GPU counts and durations for its model sizes. The listed tables cover routing-token, routing-component, and vision-encoder-combination effects.

A.2 Training Data Details

  • Vicuna-7B and Llama3-8B models use 64 A100 80G GPUs for 2 days of pretraining and 32 GPUs for 1 day of finetuning; the 34B model uses 128 GPUs for 5 days and 64 GPUs for 2 days.The passage describes the training as two stages but does not provide dataset details here.Source: §A.2 · p. 18
  • Table 16 reports the effects of the number of image tokens used in routing.Source: Table 16 · p. 19
  • Table 14 reports performance of various routing components.Source: Table 14 · p. 19
  • Table 15 reports results for different vision-encoder combinations used for routing.Source: Table 15 · p. 19

A.3 More Experiments

The additional experiments test MoVA on referring segmentation and examine routing components, routing inputs, and routing-data construction. They report a segmentation gain, favor LLM-based routing with CLIP features and 144 image tokens, and find high accuracy for the constructed routing data.

Image segmentation

  • 57.1% gIoU on RefCOCO testA is 2.6% higher than the 54.5% baseline when MoVA is extended to referring segmentation through sequential polygon generation.The setup uses a SAM-Huge backbone; unlike segmentation generalists, it does not add a pixel decoder or use high-resolution images for mask generation.Source: §A.3 · p. 19

Effect of LLM for expert routing

  • LLM-based expert routing achieves the best performance; replacing it with an MLP or BERT requires 1.6M rather than 2K routing samples to preserve performance, and neither routes in open scenarios.Source: §A.3 · p. 19

Vision encoder for expert routing

  • CLIP features alone perform slightly better than other vision-encoder choices for routing, so MoVA uses CLIP to trade off efficiency and performance.The passage attributes the choice to the routing task not requiring elaborate expert knowledge and says plain fusion adds biased information and cost.Source: §A.3 · p. 19

Number of image tokens in expert routing

  • 144 image tokens achieve performance comparable to routing methods using more tokens, while avoiding the additional processing cost of more tokens.Source: §A.3 · p. 19

Data construction

  • 94.6% routing accuracy was achieved by the loss-driven data-construction approach on a 500-sample test split evaluated with GPT4-V judgments manually checked by humans.GPT4-V regarded 473 samples as reasonable before the manual check.Source: §A.3 · p. 19

A.4 Qualitative Results

MoVA is shown on diverse, complex real-world understanding and reasoning examples, which the authors attribute to context-relevant expert knowledge.

A.4 Qualitative Results

  • Figures 4 and 5 present qualitative multimodal-understanding examples of MoVA across diverse and complex real-world understanding and reasoning tasks.The authors state that MoVA successfully solves these cases thanks to context-relevant expert knowledge.Source: §A.4 · p. 19

A.5 Potential Societal Impacts

The paper identifies potential positive and negative societal impacts of MoVA. It presents assistance in work and question answering as a possible benefit, while noting the risk of misleading or false content generation.

Positive impacts

  • MoVA could serve as an assistant that helps people improve work efficiency and answer questions.Source: §A.5 · p. 19

Negative impacts

  • MoVA could be used to generate misleading and false content, a potential issue the authors say is present in all MLLMs.They state that they will try to improve the model's safety in future work.Source: §A.5 · p. 19
  • The authors describe the potential negative impact as shared with multimodal large language models generally, rather than unique to MoVA.Source: §A.5 · p. 19

Related papers

Loading 2404.13046v2…