PaperlayerView exact revision on arXiv

Source-linked AI summary

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Bin Lin, Zhenyu Tang, Yang Ye, Jinfa Huang, Junwu Zhang, Yatian Pang, Peng Jin, Munan Ning, Jiebo Luo, Li Yuan

arXiv:2401.15947v5cs.CV

TL;DR

Dense LVLM scaling activates all parameters for every token, creating substantial training and deployment costs. The paper introduces MoE-Tuning and the MoE-LLaVA sparse architecture, which route tokens through selected experts. With approximately 3B activated parameters, MoE-LLaVA matches LLaVA-1.5-7B across visual understanding benchmarks and exceeds LLaVA-1.5-13B on POPE hallucination evaluation.

  • Problem

    Dense LVLM scaling requires every token to use all model parameters, making training and deployment computationally expensive.

  • Method

    MoE-Tuning uses staged multimodal adaptation and FFN-based expert initialization to train MoE-LLaVA with router-selected sparse expert paths.

  • Results

    Approximately 3B activated parameters achieve performance comparable to LLaVA-1.5-7B across visual understanding benchmarks, while 2.2B activated parameters outperform LLaVA-1.5-13B on POPE.

  • Takeaways & Limitations

    MoE-LLaVA establishes a sparse LVLM baseline that expands parameter capacity while keeping activated computation limited.

  • Takeaways & Limitations

    Training stability remains difficult, particularly when using 16-bit floating-point precision.

Abstract

from arXiv · show

Recent advances demonstrate that scaling Large Vision-Language Models (LVLMs) effectively improves downstream task performances. However, existing scaling methods enable all model parameters to be active for each token in the calculation, which brings massive training and inferring costs. In this work, we propose a simple yet effective training strategy MoE-Tuning for LVLMs. This strategy innovatively addresses the common issue of performance degradation in multi-modal sparsity learning, consequently constructing a sparse model with an outrageous number of parameters but a constant computational cost. Furthermore, we present the MoE-LLaVA, a MoE-based sparse LVLM architecture, which uniquely activates only the top-k experts through routers during deployment, keeping the remaining experts inactive. Extensive experiments show the significant performance of MoE-LLaVA in a variety of visual understanding and object hallucination benchmarks. Remarkably, with only approximately 3B sparsely activated parameters, MoE-LLaVA demonstrates performance comparable to the LLaVA-1.5-7B on various visual understanding datasets and even surpasses the LLaVA-1.5-13B in object hallucination benchmark. Through MoE-LLaVA, we aim to establish a baseline for sparse LVLMs and provide valuable insights for future research in developing more efficient and effective multi-modal learning systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.

1. Introduction

MoE-LLaVA addresses the cost of dense LVLM scaling with a sparse architecture and three-stage MoE-Tuning strategy. Experiments report competitive visual understanding and improved object hallucination performance with few activated parameters.

  • Motivation: Dense LVLM scaling is computationally expensive because every token uses all model parameters, whereas sparse MoE models keep activated parameters fixed.MoE methods scale capacity while limiting computation per token.
  • Method: MoE-Tuning uses three stages: train the visual projection MLP, train the LLM while excluding the vision encoder, then initialize experts from FFNs and train only MoE layers.In each MoE layer, only two experts are activated per token.
  • Challenge: Directly applying MoE to LVLMs causes significant performance degradation because multimodal conversion and sparsification occur simultaneously.The authors identify initialization as crucial for successful LVLM sparsification.
  • Method: MoE-LLaVA routes each token to selected experts while inactive paths remain silent, expanding parameter capacity without proportionally increasing computation.The architecture stacks sparse MoE layers with learnable routers.
  • Results: 3.6B sparse activated parameters outperform LLaVA-1.5-7B by 1.9% on ScienceQA, 0.4% on POPE, 0.9% on MMBench, 30.7% on LLaVAW, and 3.8% on MM-Vet.The reported gains span visual understanding benchmarks.
  • Results: 2.2B sparse activated parameters outperform LLaVA-1.5-13B on the POPE object hallucination benchmark and achieve comparable performance to InternVL-Chat-19B.The comparison concerns sparsely activated parameters rather than total model parameters.

2. Related Work

Related work develops LVLMs through stronger visual alignment, larger datasets, and larger models, while MoE research provides sparse expert routing. MoE-LLaVA applies this sparse scaling direction to reduce the cost of expanding multimodal models.

  • Large Vision-Language Models: LVLMs adapt visual signals to language models through image encoders, visual tokens, and projection layers, with later work improving alignment and visual instruction data.These approaches support increasingly capable visual understanding.
  • Large Vision-Language Models: Fine-grained LVLM research includes region understanding, multi-region understanding, and pixel-wise grounding, but scaling dense visual data and models is costly.MoE-LLaVA is motivated as a more accessible scaling direction.
  • Mixture of Experts: Mixture of Experts combines multiple sub-models and uses a router to assign token sets to experts, reducing interference between different sample types.Routing enables sparse computation while retaining multiple experts.
  • Mixture of Experts: Hard routers predefine experts for specific patterns or modalities and do not require learning the router.This design is used in multimodal and task-specific MoE systems.
  • Mixture of Experts: Soft routers dynamically allocate data among experts so each can specialize while the overall model remains sparse.Prior soft-router work has mainly developed in natural language processing.
  • MoE-LLaVA: The MoE-LLaVA architecture combines a vision encoder, visual projection layer, word embeddings, stacked LLM blocks, and MoE blocks.Its configuration includes multiple experts with only a subset activated.

3. Method

MoE-LLaVA combines a vision-language architecture with sparse MoE blocks, routing each token to selected experts while keeping others inactive. MoE-Tuning uses staged initialization and multimodal training, with autoregressive and load-balancing objectives.

  • Architecture: MoE-LLaVA combines a vision encoder, visual projection layer, word embeddings, stacked LLM blocks, and MoE blocks.
  • MoE-Tuning: Stage I trains an MLP to project image tokens into the LLM input domain while training the LLM to describe images.
  • MoE-Tuning: Stage II tunes the LLM with multimodal instruction data, and its weights initialize Stage III to reduce sparse-model learning difficulty.
  • Sparse routing: Each token is routed to top-k experts, whose outputs are aggregated by router-weighted summation while inactive experts remain silent.
  • MoE-Tuning: Stage III replicates FFNs as experts and trains router-based top-k dispatch with weighted aggregation across image and text tokens.
  • Training objectives: Training combines autoregressive loss on generated text with an auxiliary load-balancing loss for expert utilization.

4. Experiments

Experiments evaluate MoE-LLaVA across image understanding, hallucination, routing, and ablation settings. Results show competitive or superior performance with sparse activation, while initialization and architectural choices materially affect outcomes.

  • Image Understanding Evaluation: MoE-LLaVA performs competitively across five image question-answering benchmarks with fewer activated parameters than dense models.MoE-LLaVA-Phi-2.7B×4 surpasses LLaVA-1.5-7B by 2.7% on SQAI using 3.6B sparse activated parameters.
  • Image Understanding Evaluation: MoE-LLaVA-Qwen-1.8B×4 surpasses Qwen-VL-7B by 21.5% on MMBench, despite Qwen-VL-7B using higher image resolutions.
  • Object Hallucination Evaluation: MoE-LLaVA-1.8B×4 surpasses LLaVA-1.5-13B by 1.0%, 1.5%, and 0.8% under adversarial, popular, and random POPE sampling, respectively.The model uses 2.2B activated parameters, and its yes ratio remains relatively balanced.
  • Quantitative Analysis: Expert routing is initially balanced but becomes specialized across depth, with expert 3 dominating layers 17–27 and expert 1 withdrawing in deeper layers.Experts 2, 3, and 4 mainly collaborate in shallow layers.
  • Quantitative Analysis: Experts develop modality preferences while each remains capable of processing both text and image tokens.Text and image routing distributions are highly similar, and experts 2 and 3 are consistently assigned in deeper layers.
  • Ablation Study: The three-stage MoE-Tuning initialization improves sparse conversion, while FFN-only tuning matches full-parameter tuning at approximately 75% of the training time.Alternating MoE and dense layers outperform the all-MoE architecture in training efficiency, and more sparse experts improve POPE and SQAI performance by 1.1% and 0.6%.

5. Conclusion and Future Directions

MoE-Tuning adapts mixture-of-experts architectures to LVLMs, producing MoE-LLaVA with strong multimodal understanding and hallucination-mitigation performance at sparse activation costs. The authors also identify training stability, particularly with 16-bit precision, as an ongoing difficulty.

  • MoE-Tuning adapts MoE architectures to LVLMs while addressing performance degradation caused by sparsity.
  • MoE-LLaVA achieves performance comparable to LLaVA-1.5-7B with only approximately 3B activated parameters.
  • MoE-LLaVA demonstrates strong multimodal understanding and potential for hallucination inhibition.
  • Training stability remains difficult, particularly when using 16-bit float precision.

Impact Statements

The impact statement highlights risks involving misinformation, bias, discrimination, and overreliance, alongside licensing, data-use, privacy, and non-commercial-use conditions. It also notes that the project is mostly released under Apache 2.0.

  • MoE-LLaVA may generate realistic false or misleading information, creating information-credibility risks.
  • Internet-derived training data may contain biases and discriminatory content that could be reflected in generated responses.
  • Overreliance on MoE-LLaVA for information and problem-solving may reduce independent thinking, autonomy, and judgment skills.
  • The authors state that reproducibility can be achieved using the released code and models.
  • The project is mostly released under the Apache 2.0 license, while generated data and ShareGPT materials have separate terms or practices.
  • The service is intended for non-commercial research-preview use and is subject to the LLaMA model license.

A.1. More Model Architecture

The appendix defines MoE-LLaVA’s parameter accounting for architectures that replace selected FFN layers with MoE layers. It distinguishes total parameters from activated parameters through expert count and activation choices.

  • MoE-LLaVA parameter accounting combines embedding, layer, FFN, and MoE-layer terms.
  • MoE-specific parameters scale with the number of MoE layers, additional experts, hidden width, and FFN dimensions.
  • The architecture table describes variants such as 1.6B×4-Top2, with four experts and two activated per token.

A.2. Training Details

Training uses one epoch per stage, 336x336 images, and stage-specific batch sizes, while model-size ablations show gains for smaller models but weaker performance for OpenChat-MoE. The authors attribute the latter possibility to insufficient multimodal instruction data for sparse-pattern learning at 10B scale.

  • All training stages use one epoch because the models overfit when trained for two epochs.
  • The batch size is 256 for stage one and 128 for stages two and three, with 336x336 image resolution throughout.
  • Table 9 provides training hyperparameters for models based on Qwen, StableLM, Phi, and OpenChat.
  • For models smaller than 7B, MoE-LLaVA performance improves as model size increases across StableLM-1.6B, Qwen-1.8B, and Phi-2.7B.
  • OpenChat-MoE performs significantly worse than dense models, possibly because current multimodal instruction data is insufficient for sparse-pattern learning at 10B scale.

B.2. Training Capacity

MoE-LLaVA uses Batch Priority Routing during training to balance expert workloads by dropping tokens assigned beyond capacity. Its capacity ablation is presented in Table 11, where “Res.” denotes input image resolution and an asterisk marks training-data overlap.

  • Training Capacity: Batch Priority Routing dynamically adjusts expert assignments based on capacity and drops excess tokens when an expert is overloaded.This strategy is used for MoE layers during training to promote balanced workloads among experts.
  • Training Capacity: Table 11 reports an ablation study on MoE-LLaVA capacity.The table uses “Res.” for input image resolution and marks overlapping training data with an asterisk.

B.3. Routing Distributions

Routing analyses compare expert loads, modality preferences, and token paths across four MoE-LLaVA variants and six benchmarks. Larger OpenChat shows more balanced and diverse routing, whereas smaller variants exhibit concentrated or modality-specific expert patterns; the authors connect these differences to data limitations but leave the issue unresolved.

  • Routing Distributions: Routing distributions are evaluated for four MoE-LLaVA variants across ScienceQA-IMG, TextVQA, POPE, MMBench, VisWiz, and MM-Vet.The distributions are measured at the final training checkpoint.
  • Routing Distributions: OpenChat-7B×4-Top2 shows relatively balanced expert loads and modality preferences, while Phi, Qwen, and StableLM exhibit more disordered patterns.The smaller variants include workload concentration and modality-specific preferences among particular experts.
  • Routing Distributions: Phi’s expert 3 dominates layers 17–23, Qwen’s expert 1 prefers images, and StableLM’s experts 2 and 3 are active in middle layers.These examples illustrate the specific routing patterns reported for the smaller models.
  • Routing Distributions: The authors speculate that insufficient multimodal data may contribute to OpenChat’s poor performance but cannot investigate this further because of data and computational-resource limits.They leave the expected degree of expert specialization versus similarity as an open question.
  • Routing Distributions: OpenChat token paths are more disorderly and diverse, whereas the other variants show trends consistent with their more concentrated routing distributions.The authors attribute OpenChat’s diverse paths to more balanced expert assignment.
  • Qualitative Examples: MoE-LLaVA performs comparably to LLaVA and LLaVA-1.5 on showcased classic images despite using fewer parameters.Table 12 presents examples involving challenging image questions and verification prompts.
Loading 2401.15947v5…