PaperlayerView exact revision on arXiv

Source-linked AI summary

MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon, Lu Zhang, Ahmed Kirmani, Babak Damavandi, Anuj Kumar

arXiv:2608.17402v1cs.CV

TL;DR

Dense vision-encoder scaling improves capability but increases compute and latency, motivating more efficient alternatives. MoE-ViE studies fine-grained MoE designs with balancing, kernel, and video-finetuning innovations, outperforming dense and prior MoE encoders across image and video benchmarks while matching a 1.7× larger dense encoder at 76% of its latency.

  • Problem

    Dense vision-encoder scaling raises compute cost and inference latency, while MoE design for CLIP-style encoders remains underexplored at SOTA levels.

  • Method

    MoE-ViE combines fine-grained experts, loss-free expert balancing, an optimized MoE kernel, and frame-level distillation with frozen experts for video finetuning.

  • Results

    Across image and video benchmarks and model scales, MoE-ViE outperforms dense and prior MoE encoders, with its largest model matching a 1.7× larger dense encoder at 76% of its latency.

  • Takeaways & Limitations

    Fine-grained sparse scaling can improve CLIP-style vision encoders across image and video tasks while reducing latency relative to a substantially larger dense encoder.

  • Takeaways & Limitations

    Vanilla MoE implementations incur latency from fragmented expert computation, dynamic routing, synchronization, and repeated kernel launches.

Abstract

from arXiv · show

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.

1 Introduction

Dense vision-encoder scaling improves vision-language models but increases compute cost and inference latency, motivating sparse MoE scaling. MoE-ViE studies fine-grained architectures, efficient kernels, and video training strategies that preserve image understanding while achieving strong image and video results.

  • Motivation: Dense vision-encoder scaling typically increases compute cost and inference-time latency, creating system-level trade-offs.
  • MoE scaling: MoE increases total model capacity while activating only a small subset of experts per input, but its application to CLIP-style vision encoders remains unsettled.
  • Architecture: Fine-grained MoE expert designs achieve larger and more consistent improvements than conventional MoE used as a drop-in replacement for dense MLP blocks.
  • Training: Frame-level distillation and expert freezing mitigate representation shifts from video finetuning that can degrade pretrained image capabilities.
  • Results: MoE-ViE consistently outperforms dense counterparts across scales and achieves SOTA results at all scales on zero-shot image and video benchmarks.
  • Results: > 2.5× speedup in inference latency is provided by the MoE kernel, while the largest encoder matches results from PEcoreG, which is 1.7× larger.

2 Vision Encoder with MoE

This section introduces MoE vision encoders as a way to increase parameter capacity while keeping compute tied to a small routed subset of experts. It develops fine-grained and shared-expert designs, loss-free balancing, multi-scale validation, and specialized kernels for practical efficiency.

  • MoE formulation: MoE replaces a dense MLP with N experts and top-k routing, increasing parameter count while keeping per-token compute proportional to k when k ≪ N.The output combines the selected experts using router weights.
  • Expert topology: Fine-grained MoE uses narrower experts to enable more experts under comparable compute, while shared experts remain active to preserve global context.Specialized routed experts provide conditional, token-specific transformations; the shared-expert weight λ is set to 1 in all experiments.
  • Expert topology: The design replaces Softmax gating with Sigmoid gating to avoid potential expert competition and renormalizes selected scores to keep the mixture scale stable.The model routes over specialized experts while reserving the first m experts as shared.
  • Expert utilization: Loss-free balancing decouples expert-utilization control from the training objective by updating router biases from observed token loads.The proposed z-score update scales corrections by imbalance magnitude, addressing oscillations caused by constant-magnitude sign updates.
  • Scalability: Across multiple scales, MoE-ViE consistently outperforms dense counterparts under the same compute budget after contrastive pretraining on MetaCLIP.Models are trained for 12.8B samples at 224×224 resolution and evaluated with ImageNet-1K classification accuracy.
  • Efficient implementation: A specialized Triton kernel improves practical efficiency by combining expert MatMuls through Grouped GEMM and fusing MatMuls with activations to reduce synchronization and HBM traffic.These mechanisms address fragmented GEMMs, routing-induced CPU–GPU handshakes, and memory-bandwidth overhead in naive implementations.

3 Robust Video Finetuning

Naive video finetuning causes severe image forgetting, while mixing image data limits video gains. The method addresses this with frame-level distillation and freezing mechanisms that preserve image knowledge while adapting video-specific feature aggregation.

  • Naive video finetuning substantially degrades image performance, whereas mixing image data partially recovers accuracy but limits video-understanding gains.
  • Frame-level distillation uses an image-pretrained teacher on randomly sampled video frames and minimizes student–teacher logit cosine distance.The distillation loss is L_d = 1 - cos(S, T).
  • Freezing MoE experts during video finetuning preserves per-frame visual knowledge while allowing adaptation of frame-feature aggregation for video-specific alignment.The design reflects the view that MLP layers store acquired knowledge, whereas attention layers primarily handle feature interaction.
  • Freezing text-tower MLP layers prevents text drift from conflicting with vision distillation during video finetuning.Contrastive training updates both towers, while distillation keeps the vision tower near its image-pretrained teacher.

4 Experiments

Experiments show that MoE-ViE consistently achieves state-of-the-art or best-in-class performance across image, video, fine-grained, OCR, and aligned vision-language benchmarks. Its efficiency advantages include > 2.5× kernel speedup and roughly 76% of PEcoreG/14 latency, while video finetuning and expert specialization preserve and enhance capabilities.

  • Experimental setup: MoE-ViE replaces vision-tower MLP modules with MoE blocks while excluding the first transformer block, where specialization provided no observed gains.The first layer is retained to capture general, low-level visual information.
  • Experimental setup: MoE-ViE is contrastively pretrained on 3.5B image-text pairs using 32 experts per MoE layer, with each expert sized at 1/4 of the corresponding dense MLP.The data comprise 2B MetaCLIP pairs and 1.5B proprietary pairs.
  • Zero-shot evaluation: MoE-ViE achieves SOTA zero-shot video results at all evaluated scales and outperforms the larger PEcoreG model on classification and retrieval tasks.These results indicate that MoE scaling transfers effectively from image pretraining to video finetuning.
  • Zero-shot evaluation: +5.6% over PEcoreB, +0.2% over PEcoreL, and +0.6% over PEcoreG on ImageNet-A, despite PEcoreG having 1.7× as many activated parameters as MoE-ViE-H.MoE-ViE achieves the best zero-shot image results at each scale among models with comparable sizes.
  • Fine-grained and OCR evaluation: MoE-ViE-H outperforms PEcoreG on fine-grained classification and achieves high accuracy on the TextCaps OCR benchmark, including against a model with 1.7× larger activated size.The paper attributes these gains to MoE applied with the correct topology.
  • Efficiency: > 2.5× speedup from the MoE kernel over the vanilla implementation, with MoE-ViE running at roughly 76% of PEcoreG/14 latency.The comparison uses the same number of input tokens, # tokens = 576, for latency fairness.

5 Related Work

Prior work strengthens CLIP-style vision encoders through modified objectives and supervision, while Mixture-of-Experts scales transformer capacity using sparse, input-dependent computation. These directions include methods targeting efficient training and strong zero-shot image/video performance.

  • CLIP provides a foundational approach for building strong, transferable vision encoders.
  • SigLIP replaces Softmax with Sigmoid for InfoNCE loss computation, improving scaling and efficiency.
  • MaskCLIP extends masked image modeling to train vision models as self-supervised encoders.
  • SigLIP 2 and PE combine several of these ingredients and achieve high accuracy on zero-shot image and video benchmarks.
  • MoE replaces dense layers with routed experts, activating only a small subset per input for sparse, input-dependent computation.

6 Conclusion

MoE-ViE is an efficient vision encoder for image and video understanding that combines fine-grained MoE architecture, kernel optimization, loss-free load balancing, and refined video fine-tuning. It outperforms prior dense and MoE vision encoders across image and video benchmarks at all scales, while its largest model matches a SOTA dense encoder 1.7× larger.

  • 6 Conclusion: MoE-ViE combines fine-grained MoE architecture, kernel optimization, magnitude-aware loss-free load balancing, and refined video fine-tuning.These components target architecture, inference efficiency, expert utilization, and video adaptation.
  • 6 Conclusion: MoE-ViE outperforms prior dense and MoE vision encoders across image and video benchmarks at all scales.The conclusion attributes the gains to the combined design and optimization choices.
  • 6 Conclusion: 1.7× larger: the largest MoE-ViE model achieves comparable results to a SOTA dense encoder.The supplied passage states the size comparison but truncates the accompanying runtime figure.

A Architecture Design Ablations … B MoE Kernel

Architecture ablations show that limited sharing and expert activation improve efficiency, while finer expert granularity generally improves performance before gains saturate. The supplied passages do not provide findings for the MoE Kernel section.

  • A Architecture Design Ablations: The ablation study examines shared experts, activated experts, and expert width using MoE-ViE-B with patch size 32 trained on 12.8B MetaCLIP samples.These analyses are conducted under the stated default training setup unless otherwise noted.
  • A.1 Number of Shared Experts: Introducing shared experts generally improves performance over fully sparse routing, but adding more than a small number can yield no gains or degrade results.The comparison holds 32 total experts and trades shared experts against sparsely routed experts.
  • A.2 Weight of Shared Experts: Performance is largely insensitive to the shared-expert scaling weight across a broad range, making it a noncritical hyperparameter.The authors hypothesize that the weight mainly affects initialization because further training can reach comparable solutions.
  • A.3 Choice of Top-k: Performance improves only marginally beyond k = 8 activated experts, indicating diminishing returns from additional activation.The study uses 32 fine-grained experts and selects k ∈{4, 8} according to model size and compute budget.
  • A.4 Expert Width: Finer expert granularity consistently improves performance across benchmarks while total expert capacity remains fixed at 8× the MLP size.The expert width factor c determines each expert’s width relative to the dense MLP.
  • A.4 Expert Width: Gains saturate below 1⁄4 width, suggesting a practical lower bound for useful expert decomposition.This conclusion follows the expert-width sweep under fixed total capacity.

B.1 Limitation of Vanilla MoE Implementation · B.2 Optimized MoE Kernels · B.3 Summary

The paper identifies sequential vanilla MoE execution and dynamic routing overhead as sources of poor GPU utilization and latency, then addresses them with specialized Triton kernels. Grouped GEMM, jagged layouts, fused operations, and parallel execution translate MoE efficiency into lower-latency vision encoding while preserving encoding quality.

  • B.1 Limitation of Vanilla MoE Implementation: Vanilla MoE processes experts sequentially in a Python for loop, fragmenting large GEMMs into smaller, imbalanced operations with low arithmetic intensity and poor GPU utilization.These limitations increase latency and are compounded by per-expert index selection during dynamic routing.
  • B.2 Optimized MoE Kernels: The optimized implementation uses two custom GPU kernels written in Triton to address the vanilla implementation’s bottlenecks.Algorithm 1 specifies the resulting optimized procedure.
  • B.2 Optimized MoE Kernels: Grouped GEMM fuses all per-expert GEMMs into one GPU launch, using a 3D grid to parallelize over tokens, output columns, and experts.This improves arithmetic intensity and GPU utilization while avoiding multiple small GEMM launches.
  • B.2 Optimized MoE Kernels: A jagged layout handles variable per-expert token counts without padding, while GPU-side Sort/CumSum removes per-expert Where queries, loops, and CPU-GPU synchronization.The routing phase computes sorted expert assignments, indices, and jagged segment boundaries.
  • B.2 Optimized MoE Kernels: The training kernel applies SwiGLU as a separate PyTorch operator, enabling experiments with SwiGLU, GeLU, and SiLU without modifying or re-autotuning the Triton kernel.This differs from the inference kernel, which fuses activation functions into the GEMM.
  • B.2 Optimized MoE Kernels: Kernel fusion combines the Gate+Up projection with SwiGLU and performs the Down projection with routing-weighted scatter-add directly to the output.SwiGLU is fused in registers without writing intermediate gate and up projections to memory.
  • B.3 Summary: Grouped GEMM and kernel fusion are necessary to realize MoE’s algorithmic efficiency in hardware-efficient execution for vision encoders processing all image patches simultaneously.High-resolution tiling can produce tens of thousands of patches, making compute utilization across the token volume important.
  • B.3 Summary: Co-designing MoE architecture with kernel-level optimizations reduces vision encoder latency while maintaining strong encoding quality for end-to-end VLM applications.Jagged segments and per-expert early exit accommodate uneven routing distributions from spatially correlated image patches.

C Scaling Experts in MoE

Scaling the total number of experts consistently improves zero-shot image classification accuracy across four benchmarks while keeping eight experts activated. MoE-ViE adopts 32 total experts as a favorable balance between performance and training cost.

  • Experimental setup: Experiments train MoE-ViE-B with patch size 32 on 10.9B MetaCLIP samples, keeping k = 8 activated experts and each expert MLP at one-quarter dense width.
  • Scaling results: Increasing the total number of experts consistently improves zero-shot accuracy on ImageNet-1K, ImageNetV2, ObjectNet, and ImageNet-A.
  • Scaling results: 32 total experts are selected for larger vision encoders because they provide a favorable balance between performance and training cost.
  • Scaling results: The comparison scales from 16 to 128 total experts while keeping 8 experts activated.

D Implementation Details … E.1 Linear Probing

The paper details MoE-ViE’s OpenCLIP-based vision encoder implementation, progressive-resolution training, and evaluation protocols. Linear probing on ImageNet-1K shows MoE-ViE-H/14 outperforming all tested encoders, including those with more active parameters.

  • D Implementation Details: MoE-ViE builds on OpenCLIP, removes the class token, uses attention pooling, applies 2D RoPE, and replaces FFNs with SwiGLU layers.The text tower uses absolute positional embeddings.
  • D.1 Training: MoE-ViE-H training uses image resolutions 98×98, 154×154, and 224×224.The supplied passage truncates the remainder of the MoE-ViE-H schedule.
  • D.1 Training: Pretraining uses 2B MetaCLIP image-text pairs plus 1.5B proprietary examples, a 10−3 learning rate, LAMB, weight decay 0.05, and batch size 262144.Cosine decay, (β1, β2) = (0.9, 0.995), 2× batch-size reduction at the highest resolution, and InfoNCE loss are also used.
  • D.1 Training: Video finetuning uses learning rate 10−6 for 20.5M samples across all MoE-ViE sizes, with batch size 4096 and 8 uniformly sampled frames per video.The distillation-loss weight β is 0.5, and the same optimizer as pretraining is used.
  • D.2 Evaluation: Zero-shot classification and retrieval evaluation uses the CLIP benchmark, prescribed prompt templates, both center-crop settings, and retrieval-result reweighting.The maximum score across the two preprocessing methods is reported.
  • D.2 Evaluation: Alignment results are evaluated with lmms-eval using greedy decoding in all cases.
  • E.1 Linear Probing: MoE-ViE-H/14 outperforms all encoders tested on ImageNet-1K linear probing, including encoders with significantly more active parameters.The comparison results are presented in Table 15.

E.2 Alignment · E.3 End-to-End VLM Latency

The alignment experiments show MoE-ViE achieving strong image and video performance with Llama 3.1 8B Instruct, while end-to-end tests show that its efficiency benefits improve downstream VLM latency–accuracy trade-offs.

  • E.2 Alignment: MoE-ViE uses Llama 3.1 8B Instruct as the decoder and extends both pre-alignment and supervised fine-tuning stages.The alignment pipeline also includes a 1k-sample vision-encoder freeze step in Stage 2a.
  • E.2 Alignment: 81.3 average on image benchmarks and 63.1 on video are achieved by MoE-ViE using 42M samples.It surpasses PElang across all three evaluation axes despite PElang using 1.7× more active parameters and 70M mid-training images.
  • E.2 Alignment: Data quality and scale are identified as decisive factors for absolute alignment performance.The sparse MoE-ViE architecture is described as particularly well-suited to capitalize on these gains.
  • E.3 End-to-End VLM Latency: End-to-end inference latency is measured after aligning vision encoders to a Llama 3.1 8B Instruct decoder.All latency benchmarks use a single H100 GPU and compare different vision encoders within the resulting VLM.
  • E.3 End-to-End VLM Latency: MoE-ViE’s end-to-end latency trend matches the vision-encoder-only results reported in Table 7.The comparison evaluates different vision encoders after integration into the Llama 3.1 8B Instruct VLM.
  • E.3 End-to-End VLM Latency: MoE-ViE’s efficiency benefits extend beyond the vision encoder into downstream VLM applications.The results translate into improved latency vs. accuracy trade-offs after alignment with the LLM decoder.
Loading 2608.17402v1…