PaperlayerView exact revision on arXiv

Source-linked AI summary

SmolVLA-MoE: Sparse Mixture-of-Experts for Efficient Multi-Task Robotic Manipulation

Shiyang Zhang, Miao Yang, Zhengda Li, Yang Shao, Ruifang Zhai, Hui Peng, Peng Song

arXiv:2610.30732v1cs.CV

TL;DR

Lightweight VLA models use a single dense action expert, limiting multi-task capability under constrained computation. SmolVLA-MoE adds sparse, token-routed experts to action-expert FFN layers, and improves over SmolVLA across all four LIBERO suites while keeping the same activated parameter count. The evaluation is limited to simulation on LIBERO.

  • Problem

    Lightweight VLA models rely on one dense action expert for all tasks, limiting multi-task capability under constrained computational budgets.

    Source: §1 · p. 2

  • Method

    SmolVLA-MoE replaces action-expert FFN layers with parallel expert FFNs and a token-level router, using a shared expert weight to balance pre-trained capability inheritance and specialization.

    Source: §5 · p. 14

  • Results

    SmolVLA-MoE outperforms SmolVLA on all four LIBERO suites while retaining the dense baseline's ∼500M total activated parameter count.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The shared expert weight provides a control knob for balancing stability and specialization across task types.

    Source: §5 · p. 14

    Evaluation is limited to the LIBERO benchmark in simulation.

    Source: §5 · p. 14

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown promising results in multi-task robotic manipulation by leveraging visual and language understanding for action generation. However, existing lightweight VLA models employ a single dense action expert to handle all tasks, which limits their multi-task capability under constrained computational budgets. To address this issue, we propose SmolVLA-MoE, a novel framework that introduces sparse Mixture-of-Experts (MoE) into the FFN layers of the action expert in VLA models. SmolVLA-MoE replaces dense FFN layers with multiple parallel expert FFNs and a token-level router, enabling each token to adaptively select the most suitable expert for action generation. We further introduce a shared expert with a tunable weight parameter to balance the inheritance of pre-trained capabilities and task-specific expert specialization. Through sparse activation, SmolVLA-MoE activates only a single expert FFN during inference, with a total activated parameter count of approximately 500M including the frozen VLM backbone, while benefiting from approximately 800M total stored parameters. We evaluate our method on four task suites of the LIBERO benchmark. SmolVLA-MoE achieves an average success rate of 84.25%, outperforming SmolVLA (82.75%) and OpenVLA-7B (76.5%), while maintaining comparable inference efficiency. Our code is publicly available at https://github.com/dudu435/smolvla-moe.

1 Introduction

SmolVLA-MoE addresses the single dense action expert's multi-task limitation by introducing sparse, token-routed experts in the action expert's FFN layers. On LIBERO, it improves over SmolVLA across all four suites while maintaining comparable inference efficiency and the dense baseline's activated parameter budget.

1 Introduction

  • A single dense FFN must encode diverse task-specific action patterns, creating tension between model capacity and computational efficiency.Scaling that expert increases inference cost and undermines the lightweight design.Source: §1 · p. 2
  • Sparse MoE maintains multiple experts but routes each input to a sparse subset, keeping active computation constant as model capacity scales.Its application to VLA action generation remains largely unexplored in the supplied discussion.Source: §1 · p. 2
  • SmolVLA-MoE replaces action-expert FFN layers with parallel expert FFNs and a learnable token-level router, while keeping attention layers shared.Each token selects an expert based on input context.Source: §1 · p. 2
  • A tunable shared-expert weight balances inheritance of pre-trained capabilities against task-specific expert specialization.The paper reports systematic analysis of this control across task types.Source: §1 · p. 3
  • SmolVLA-MoE outperforms SmolVLA on all four LIBERO suites while maintaining comparable inference efficiency and the dense baseline's ∼500M activated parameter count.Only ∼100M activated parameters are in the action expert; inference activates a single expert FFN.Source: §1 · pp. 2–3

2 Related Work

Related work establishes lightweight VLA models as a response to deployment constraints, while noting that SmolVLA's single dense action expert limits multi-task capability. Sparse MoE offers conditional capacity scaling, but its integration into lightweight VLA action generation had not been systematically studied.

2.1 Vision-Language-Action Models

  • SmolVLA uses a compact VLM backbone and independent Flow Matching action expert, reducing model size while retaining competitive performance; its single dense expert limits multi-task capability.The discussion situates it alongside other VLA architectures and lightweight policy approaches.Source: §2.1 · pp. 3–4

2.2 Mixture-of-Experts Models

  • Sparse MoE scales capacity through specialized experts with constant per-inference computation, but lightweight VLA action generation had not been systematically studied.This work combines DeepSeek-MoE's shared experts, AdaMoE's adaptive routing, and Sparse Upcycling initialization.Source: §2.2 · p. 4

3 Method

SmolVLA-MoE adds token-routed expert FFNs to the action expert while retaining shared attention and a frozen VLM backbone. A shared expert, adaptive routing, and a load-balancing objective shape how the model combines pre-trained capabilities with task-specific specialization.

3.1 Preliminaries: SmolVLA

  • SmolVLA retains a frozen VLM backbone for visual and language representations and a trainable action expert that generates actions using Flow Matching.The action expert uses Transformer layers with cross-attention to the VLM representations and SwiGLU feed-forward networks.Source: §3.1 · pp. 5–6

3.2 FFN-Level Mixture-of-Experts

  • Replacing dense action-expert FFNs with independent expert FFNs keeps attention shared while allowing tokens to select specialized transformations.A replacement-frequency parameter controls which FFN layers become MoE layers.Source: §3.2 · p. 6

3.3 Routing Mechanism

  • A bias-free linear router selects each token’s top-K experts, and softmax normalizes their routing weights.A scaling adapter adds per-token adaptive scaling to the selected experts’ weights, decoupling selection from weighting.Source: §3.3 · pp. 7–8

3.4 Shared Expert with Tunable Weight

  • The always-active shared expert is combined with routed-expert output using α, which controls their relative contribution.The shared expert retains the original dense FFN weights, while routed experts receive noise perturbations during initialization.Source: §3.4 · p. 8

3.5 Training Objective

  • Training combines Flow Matching loss with a load-balancing loss weighted by λ, encouraging balanced expert utilization.The load-balancing term uses token allocation and average routing probabilities; λ defaults to 0.01 and controls the trade-off with action prediction accuracy.Source: §3.5 · p. 9

4 Experiments

Across LIBERO’s four task suites, SmolVLA-MoE improves average success over the dense SmolVLA baseline at the same activated-parameter budget. Ablations examine routing and shared-expert choices.

4.1 Experimental Setup

  • LIBERO provides standardized evaluation suites for multi-task robotic manipulation in simulated tabletop environments, with four suites of increasing difficulty; each task is evaluated over 10 rollouts using success rate.Source: §4.1 · pp. 9–10

4.2 Main Results

  • 84.25% average success rate: SmolVLA-MoE exceeds SmolVLA’s 82.75% at the same ∼500M activated-parameter budget.It also outperforms OpenVLA (7B, 76.5%) and Octo (0.09B, 75.1%).Source: Table 1 · p. 11, §4.2 · p. 11
  • SmolVLA-MoE matches SmolVLA on Spatial (87%) and Object (93%), while gaining 5 points on Goal and 2 points on LIBERO-10.Its 93% Object score exceeds π0 (3.3B, 86%), while LIBERO-10 remains 8 points below π0 (3.3B, 73%).Source: §4.2 · p. 12
  • At ∼500M activated parameters, SmolVLA-MoE trails π0 (3.3B) by 1.75 points despite using 6.6× fewer activated parameters.The additional ∼300M expert parameters are stored but not all activated during inference.Source: §4.2 · p. 12

4.3 Ablation Studies

5 Conclusion

SmolVLA-MoE combines token-level sparse experts with shared attention to improve multi-task LIBERO performance while maintaining the dense baseline’s activated parameter count. The evaluation remains limited to simulation on LIBERO.

5 Conclusion

  • FFN-level sparse experts and a learnable token-level router let each token select specialized action-generation experts while scaling capacity without increasing inference cost.The action expert’s attention layers remain shared across tasks.Source: §5 · p. 14
  • SmolVLA-MoE outperforms SmolVLA on all four LIBERO suites while maintaining ∼500M total activated parameters, including ∼100M in the action expert module.Top-1 routing outperforms top-2 on tasks requiring precise action generation; shared expert weight controls the balance between stability and specialization.Source: §5 · p. 14
  • Evaluation is limited to the LIBERO benchmark in simulation, with real-world tasks and other benchmarks left for future work.Proposed directions also include adaptive expert configurations, quantization, and knowledge distillation.Source: §5 · p. 14

Related papers

Loading 2610.30732v1…