PaperlayerView exact revision on arXiv

Source-linked AI summary

ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation

Jiawen Yu, Hairuo Liu, Qiaojun Yu, Jieji Ren, Ce Hao, Haitong Ding, Guangyu Huang, Guofan Huang, Yan Song, Panpan Cai, Cewu Lu, Wenqiang Zhang

arXiv:2505.22159v3cs.ROcs.CV

TL;DR

VLA models struggle with contact-rich manipulation because they underuse force information needed for fine-grained control under occlusion and dynamic uncertainty. ForceVLA integrates real-time six-axis force feedback with pretrained vision-language representations through FVLMoE and adds ForceVLA-Data for five contact-rich tasks. It improves average task success by 23.2% over π0-based baselines and reaches up to 80% success on individual tasks.

  • Problem

    Existing VLA models often overlook force sensing, limiting fine-grained control in contact-rich tasks under occlusion and changing interaction dynamics.

  • Method

    ForceVLA integrates vision, language, proprioception, and six-axis force feedback through the force-aware FVLMoE Mixture-of-Experts module during action generation.

  • Results

    23.2% average success improvement over strong π0-based baselines and up to 80% success on individual tasks were achieved across five challenging tasks.

  • Takeaways & Limitations

    ForceVLA demonstrates the importance of integrating force with visual and linguistic information for contact-rich manipulation.

  • Takeaways & Limitations

    ForceVLA uses estimated external wrench values and was mainly validated on robotic platforms with integrated, typically high-cost, force-torque sensing.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have advanced general-purpose robotic manipulation by leveraging pretrained visual and linguistic representations. However, they struggle with contact-rich tasks that require fine-grained control involving force, especially under visual occlusion or dynamic uncertainty. To address these limitations, we propose ForceVLA, a novel end-to-end manipulation framework that treats external force sensing as a first-class modality within VLA systems. ForceVLA introduces FVLMoE, a force-aware Mixture-of-Experts fusion module that dynamically integrates pretrained visual-language embeddings with real-time 6-axis force feedback during action decoding. This enables context-aware routing across modality-specific experts, enhancing the robot's ability to adapt to subtle contact dynamics. We also introduce \textbf{ForceVLA-Data}, a new dataset comprising synchronized vision, proprioception, and force-torque signals across five contact-rich manipulation tasks. ForceVLA improves average task success by 23.2% over strong pi_0-based baselines, achieving up to 80% success in tasks such as plug insertion. Our approach highlights the importance of multimodal integration for dexterous manipulation and sets a new benchmark for physically intelligent robotic control. Code and data will be released at https://sites.google.com/view/forcevla2025.

1 Introduction

VLA models generalize semantic and linguistic instructions but remain vulnerable in contact-rich manipulation because they underuse force feedback and cannot adapt reliably to changing interaction dynamics. ForceVLA addresses this gap with force-aware multimodal fusion, a supporting data pipeline, and experiments showing improved task success.

  • VLA models support semantic grounding, language following, and zero-shot generalization through pretrained vision-language encoders and flow-based action generation.OpenVLA bridges perception and control, while π0 combines stronger VLM backbones with flow-based action generation.
  • Contact-rich tasks require force-sensitive control, yet existing VLA models often overlook force sensing and struggle with insertion, tool use, and assembly under visual occlusion.These tasks involve interaction forces and can produce brittle behavior or failure when visual conditions are poor.
  • ForceVLA treats 6D end-effector force sensing as a first-class modality and uses FVLMoE to route visual-linguistic and real-time force information across modality- and phase-aware experts.The gating mechanism dynamically activates specialized subnetworks using high-level instructions and low-level interaction feedback.
  • ForceVLA integrates force, vision, language, and action through a force-aware Mixture-of-Experts fusion module for improved precision and stability in contact-rich manipulation.The module dynamically processes and deeply integrates multimodal features during action generation.
  • ForceVLA contributes a complete contact-rich data collection pipeline, including teleoperation tools, data converters, and a new dataset intended for open community use.
  • 23.2% average performance improvement over baselines and up to 80% task success were achieved across five challenging contact-rich manipulation tasks.The experiments also assessed generalization to novel objects, occlusions, and physical perturbations.

2 Related Works

Related work has advanced VLA generalization, force and tactile sensing, and sparse expert architectures separately. However, existing approaches generally do not explicitly model force or tactile modalities together with dynamic interaction-aware routing.

  • Robotic VLA domain: VLA research maps visual and language inputs to low-level control, with flow-based π0 architectures combining pretrained encoders and fast action decoders.Other directions add reasoning mechanisms, action-space compression, or 3D point clouds.
  • Contact-rich manipulation domain: Force and tactile sensing improve stability, accuracy, fine manipulation, and articulation in contact-rich tasks.Prior work includes force-feedback methods, foundational force-control studies, and tactile systems such as TLA and Tac-Man.
  • MoE architecture-related work: MoE architectures improve scalability and efficiency through sparse expert activation, and recent robotic applications use MoE layers to improve policy generalization and adaptability.Multimodal MoE work has also explored joint vision-language learning.
  • MoE architecture-related work: Existing MoE-based VLA methods largely omit explicit force or tactile modeling and lack mechanisms for dynamic interaction-aware adaptation.

3 Preliminary

The task formulation combines visual observations, proprioception, and six-axis external force-torque readings with language instructions to learn executable action chunks. The preliminary architecture uses sparse expert routing to process multimodal inputs, while the model overview connects fused features to flow-based action generation.

  • Problem Formulation: At timestep t, the robot observes visual inputs, a 7-dimensional proprioceptive state, and six-dimensional external force-torque readings.These observations are collectively denoted as O_t = {V_t, s_t, f_t}.
  • Problem Formulation: Given language instruction L, the policy π(A_t|O_t, L) predicts an executable action chunk of horizon H for contact-rich task completion.The proprioceptive state contains TCP pose and gripper width, while force readings are estimated at the TCP in the world frame.
  • Problem Formulation: The external wrench f_t contains R^3 force and R^3 moment components expressed as {f_tx, f_ty, f_tz, m_tx, m_ty, m_tz}.
  • MoE Architecture: An MoE layer uses a gating network to select a small subset of specialized experts for each input token and aggregates their outputs with routing weights.Sparse routing typically activates k = 1 or k = 2 experts from N total experts, with k much smaller than N.
  • ForceVLA Overview: The ForceVLA overview processes visual and language inputs with a pretrained VLM, fuses projected force signals through FVLMoE, and uses the resulting features to guide a flow-based action head.

4 ForceVLA

ForceVLA is an end-to-end multimodal policy that combines vision, language, proprioception, and 6-axis force feedback for contact-rich manipulation. Its FVLMoE module introduces force after VLM processing, adaptively fuses modalities, and guides flow-based action generation.

  • 4.1 Overview of ForceVLA: ForceVLA integrates vision, language, proprioception, and 6-axis force feedback into an end-to-end policy for contact-rich manipulation.Visual inputs and task instructions are encoded by a SigLIP-based VLM, while proprioceptive and force cues condition action prediction.
  • 4.1 Overview of ForceVLA: FVLMoE projects force readings into dedicated tokens and adaptively routes them with vision-language embeddings through modality-aware experts.Its output provides guidance for handling subtle contact dynamics and visually ambiguous scenarios.
  • 4.1 Overview of ForceVLA: Force features are introduced after the primary VLM processes visual and linguistic inputs, preserving the VLM’s initial multimodal representations.The paper contrasts this post-VLM design with force injection before or during the VLM’s initial fusion.
  • 4.1 Overview of ForceVLA: The FVLMoE input concatenates VLM-derived visual-linguistic tokens with a dedicated token obtained by linearly projecting raw 6-axis force-torque data.The resulting multimodal sequence is processed by an encoder for shared refinement and cross-modal interaction.
  • 4.1 Overview of ForceVLA: Fused multimodal features guide the flow-based denoising action head by modulating the generated action sequence at each prediction step.The guidance is additively injected into features derived from the robot state and noisy action trajectory.
  • 4.2 ForceVLA-Data: ForceVLA-Data synchronizes visual, proprioceptive, and force-torque streams for contact-rich manipulation training.The dataset contains 244 trajectories and approximately 140 thousand synchronized timesteps, with collection and processing resources intended for public release.

5 Experiments

Experiments evaluate ForceVLA on five contact-rich tasks, generalization settings, and fusion ablations. ForceVLA achieves the strongest overall performance, generalizes under object, perceptual, and physical variation, and benefits from post-VLM adaptive fusion.

  • 5.1 Experimental Setup: ForceVLA is evaluated on five contact-rich tasks spanning bottle pumping, plug insertion, USB insertion, whiteboard wiping, and cucumber peeling.The tasks probe fine-grained control, varied initial conditions, and the utility of multimodal feedback including force sensing.
  • 5.2 Main Results: 60.5% average success rate is achieved by ForceVLA across all five tasks, compared with 37.3% for π0-base without force feedback.The reported improvement over that baseline is 23.2%.
  • 5.2 Main Results: ForceVLA reaches 14.12 cm average peel length per stroke and requires 7 strokes, compared with 13.17 cm and 10 strokes for π0-base w/ F.These metrics assess surface-manipulation fidelity and peeling efficiency on cucumber peeling.
  • 5.2 Main Results: Force feedback raises π0-base performance from 37.3% to 40.2%, while ForceVLA reaches 60.5%, showing that fusion quality matters beyond force-signal availability.The comparison isolates the benefit of effective force integration through FVLMoE.
  • 5.3 Model Generalization: ForceVLA generalizes across object changes, variable heights, visual occlusion, and unstable sockets, including 80.00% success in Object Gen. 1 and 90.00% under Visual Occlusion.The model scaled interaction forces under variable depths and avoided torque-limit violations observed in other models.
  • 5.4 Ablation Studies: Early force fusion degraded performance, including 0% success for MoE before VLM, whereas post-VLM concatenation reached 60% and ForceVLA reached 80%.The ablation supports introducing force after VLM encoding and using FVLMoE for adaptive fusion.
  • 5.5 Qualitative Analysis: ForceVLA re-oriented or re-grasped after USB insertion misalignment and adjusted plug pose when an unstable socket shifted.Baselines repeated failed motions, applied excessive force, or lost tracking in these illustrated conditions.

6 Conclusion

ForceVLA combines force-aware multimodal fusion with ForceVLA-Data for contact-rich manipulation. Across five tasks, it improves success over π0-based baselines, while remaining limited by estimated wrench sensing and costly integrated force-torque hardware.

  • ForceVLA introduces FVLMoE, which dynamically fuses visual, linguistic, and force modalities for fine-grained, context-aware control.
  • 23.2% average success improvement over strong π0-based baselines was achieved across five challenging tasks.
  • Up to 80% success was achieved on individual contact-rich manipulation tasks.
  • ForceVLA-Data contributes multimodal data for contact-rich manipulation alongside the architectural approach.
  • Estimated external wrench values may lack the precision of direct high-fidelity measurements in scenarios requiring extreme haptic sensitivity.
  • Validation predominantly used robotic platforms with integrated, typically high-cost force-torque sensing, limiting broader accessibility.

A Data Collection System

The data collection system combines multiple camera viewpoints with VR teleoperation and synchronized software-managed sensor streams.

  • Figure 8 presents the data collection system setup.
  • The setup uses a wrist-mounted camera and a static third-person camera to capture diverse visual perspectives.
  • An operator teleoperates the robotic arm with a Quest 3 headset and hand-held controllers.
  • A nearby computer runs robot interfacing, sensor-stream synchronization, and VR communication software for data acquisition.

B Training Details

Training used multi-GPU compute with Adam optimization and gradient accumulation, while Table 4 summarizes the dimensionalities of the core processing modules.

  • Models were mainly trained on nodes with 8× NVIDIA RTX 4090 GPUs, 64 physical CPU cores, and 251 GB system RAM.
  • Adam training used β1 = 0.9, β2 = 0.95, and a learning rate decaying from 2.5 × 10−5 to 2.5 × 10−6 over 30,000 steps.
  • Multi-task training used data parallelism across 2 GPUs, with global batch size 16 and effective batch size 2048 via gradient accumulation.
  • Table 4 details dimensions and key parameters for Input Projections, the FVLMoE block, and the Action Output Head.

C Router Analysis

Router analysis shows task- and phase-dependent expert utilization, alongside frequent activation of a broadly used Expert 0. The analysis normalizes variable-length episodes into comparable temporal segments.

  • Variable-length episodes were divided into 100 temporally equidistant intervals, with segment statistics averaged across approximately 10 episodes per task.
  • Insert plug and peel cucumber show temporal expert specialization, whereas wipe board consistently favors one expert throughout execution.
  • Expert routing dynamically allocates computation according to task-specific semantics and temporal structure.
  • Expert 0 dominates nearly half of the tokens across multiple tasks, unlike the more selective activation of Experts 1 and 3.
  • The observed routing asymmetry is interpreted as evidence of functional specialization and possible architectural bias toward certain experts.

D Multi-task Evaluation

Joint multi-task training exposes major differences among the evaluated models: ForceVLA achieves the strongest average performance and matches or exceeds the best result on each individual task.

  • ForceVLA demonstrates superior average performance and excels or matches the best performance across all individual tasks.
  • 0% average success was achieved by both π0-fast variants, which failed to acquire skills in joint multi-task learning.
  • 5% average success was achieved by π0-base w/o F, which succeeded on only one task at 20%.
  • 42.5% average performance was achieved by π0-base w/ F after adding direct force input, primarily through success on insert plug.

E Real-world Experiments Visualization

The real-world visualizations present key frames from experiment videos, contrasting baseline failures with ForceVLA successes under similar conditions across standard, generalized, occluded, and unstable plug-insertion settings.

  • Comparison: The visualizations contrast baseline failure cases with successful ForceVLA completions under similar conditions.
  • Real-world task videos: Key frames document videos for Pump Bottle, Insert USB, Wipe Board, and Insert Plug tasks.
  • Generalization: Three additional visualizations cover Insert Plug Generalization tasks 1–3.
  • Robustness settings: Two visualizations cover Insert Plug Occlusion tasks 1–2, while another covers the Insert Plug Unstable task.
Loading 2505.22159v3…