Source-linked AI summary
FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation
Ruiteng Zhao, Wenshuo Wang, Yicheng Ma, Xiaocong Li, Francis E. H. Tay, Marcelo H. Ang, Haiyue Zhu
TL;DR
Vision-only VLA policies can miss contact dynamics, while force sensors add hardware dependence. FD-VLA distills a force token from visual and state inputs, injects it into a pretrained VLM, and reports stronger performance than direct raw-force inputs and other baselines, including sensor-free deployment.
Problem
Vision-only policies poorly capture subtle contact effects, motivating force-aware VLA methods that incorporate information unavailable from images alone.
Method
FD-VLA trains a Force Distillation Module to predict a force token from vision and robot state, then fuses that token with VLM inputs without force measurements at inference.
Results
FD-VLA outperforms models using direct raw force inputs and other baselines, while predicting force representations from vision and state without test-time force sensors.
Takeaways & Limitations
The approach supports force-aware contact-rich manipulation on robots without dedicated force sensors and adds force-vision-state cross-modal alignment.
Abstract
from arXiv · showhide
Force sensing is a crucial modality for Vision-Language-Action (VLA) frameworks, as it enables fine-grained perception and dexterous manipulation in contact-rich tasks. We present Force-Distilled VLA (FD-VLA), a novel framework that integrates force awareness into contact-rich manipulation without relying on physical force sensors. The core of our approach is a Force Distillation Module (FDM), which distills force by mapping a learnable query token, conditioned on visual observations and robot states, into a predicted force token aligned with the latent representation of actual force signals. During inference, this distilled force token is injected into the pretrained VLM, enabling force-aware reasoning while preserving the integrity of its vision-language semantics. This design provides two key benefits: first, it allows practical deployment across a wide range of robots that lack expensive or fragile force-torque sensors, thereby reducing hardware cost and complexity; second, the FDM introduces an additional force-vision-state fusion prior to the VLM, which improves cross-modal alignment and enhances perception-action robustness in contact-rich scenarios. Surprisingly, our physical experiments show that the distilled force token outperforms direct sensor force measurements as well as other baselines, which highlights the effectiveness of this force-distilled VLA approach.
I. INTRODUCTION
FD-VLA addresses the limitations of vision-only contact-rich manipulation by distilling force representations from vision and robot state instead of relying on force sensors. Its Force Distillation Module adds force-vision-state fusion, enabling force-aware reasoning and improved cross-modal alignment for sensor-free deployment.
- Vision-only policies often miss contact effects and struggle with occlusions, lighting shifts, and depth ambiguities.
- Force and tactile sensing provide direct information about contact dynamics, compliance, and physical interactions that vision often cannot capture.
- FD-VLA uses a Force Distillation Module to condition a learnable query on visual observations and robot states, producing a predicted force token.
- During deployment, FD-VLA injects the distilled force token into the VLA without requiring direct sensor measurements.
- The additional force-vision-state fusion before the VLM improves cross-modal alignment and perception-action robustness in contact-rich manipulation.
II. RELATED WORKS
Related work establishes the need for multimodal sensing in contact-rich manipulation and motivates cross-modal knowledge distillation. FD-VLA applies this direction by distilling implicit force representations from vision and state inputs, avoiding dedicated tactile hardware at deployment.
- Large-scale VLAs couple pretrained vision-language representations with action generation and generalize across diverse tasks and platforms.
- Contact-rich manipulation requires modeling forces, slips, and deformations that vision-only approaches often capture inadequately.
- Cross-modal knowledge distillation aligns heterogeneous sensory streams so one modality can benefit from complementary information in another.
- FD-VLA distills implicit force representations from vision and robot state inputs, eliminating the need for dedicated tactile hardware during deployment.
III. METHODOLOGY
FD-VLA integrates a predicted force token into a pretrained VLM while preserving its visual-linguistic semantics. Its FDM predicts force from visual observations and robot state, enabling sensor-free inference and force-aware action generation.
- Multimodal inputs: FD-VLA uses language, visual observations, robot state, and predicted force as multimodal inputs to the VLM.Visual, language, and state embeddings are projected to the VLM hidden dimension before fusion.
- Force prediction: The FDM predicts a latent force representation from real-time visual observations and robot proprioceptive state.A learnable force-prediction token is trained against latent representations of actual force signals, but force measurements are unnecessary at inference.
- VLM integration: The pretrained VLM remains frozen, while attention masking separates perceptual streams from state and force control tokens.This separation is intended to preserve pretrained visual-linguistic representations while allowing downstream action integration.
- Action generation: A transformer-based action expert with conditional flow matching maps fused multimodal embeddings to action sequences.The architecture uses the VLM features to support downstream control without additional sensing requirements.
- Framework motivation: FD-VLA integrates a distilled force token into a VLA model for contact-rich manipulation.The framework is designed to incorporate force without directly retraining the pretrained VLM.
C. Force Distillation Module (FDM)
The Force Distillation Module learns a compact force representation from vision and robot state, using actual force only as a training-time supervisory signal. Its attention-based prediction branch is designed to produce a denoised, task-relevant proxy for contact forces.
- Module purpose: FDM generates a compact, state-aware force representation from visual and proprioceptive context without tactile hardware at inference.The method exploits correlations among visual cues, joint torques, velocities, and positions to infer contact forces.
- Denoised representation: FDM mitigates raw-signal noise by learning a supervised latent embedding that filters high-frequency artifacts while preserving manipulation-relevant dynamics.The resulting embedding serves as a task-relevant proxy for physical force.
- Training supervision: The actual-force branch projects robot force measurements into the predicted-force embedding space for training supervision only.The projected actual-force embedding is not required during deployment.
- Prediction branch: The prediction branch uses a learned query token and attention over image and proprioceptive embeddings to generate the latent force feature.The force-latent generation is formulated as retrieval conditioned on vision-state context.
- Attention design: The query token is included in both key and value sets, enabling self-conditioning through the residual path.This design injects the learned contact prior into context aggregation.
- Distillation objective: An auxiliary distillation loss aligns predicted and actual-force feature representations.This alignment connects the prediction and actual-force branches during training.
D. Directional Attention Masking Mechanism
Directional attention masking separates frozen vision-language perception from state and force control while permitting control tokens to use perceptual context. This one-way information flow preserves pretrained alignment during multimodal fusion.
- Stream separation: The input tokens are divided into a frozen perceptual stream and a control stream containing state and predicted-force tokens.The perceptual stream contains vision and language features, while the control stream contains state and force features.
- Information flow: The directional mask allows perceptual tokens to attend only to perceptual tokens, while control tokens can attend to perceptual tokens.Control-token self-attention is constrained by token order through the mask.
- Preserved semantics: This masking preserves pretrained vision-language alignment while fusing proprioception and force with perceptual context for action prediction.The mechanism keeps the perceptual stream isolated from control-stream feedback.
E. Action Expert
The action expert predicts action chunks from VLM features and noisy actions using conditional flow matching. It learns a velocity field rather than stepwise residuals, supporting chunked action prediction.
- Action-chunk prediction: The action expert is a transformer that predicts an action chunk conditioned on VLM features.The predicted chunk contains actions from the current step through a future horizon H−1.
- Flow-matching objective: Conditional flow matching trains the policy to output a velocity field from noisy actions and VLM features.The noisy action interpolation uses Gaussian noise and a sampled time variable.
- Policy formulation: Learning a velocity field rather than stepwise residuals is suited to chunked action prediction.
F. Overall Training Objective
FD-VLA trains with a standard policy-learning loss jointly combined with a force-distillation loss, whose influence is controlled by λ.
- The overall objective combines a standard policy learning loss with a force-distillation loss.
- λ controls the weight assigned to force supervision.
IV. EXPERIMENTS
The experiments evaluate FD-VLA through real-world manipulation trials and report success rates for three contact-rich tasks against several VLA and control baselines.
- The evaluation uses real-world experiments and analytical studies to assess FD-VLA.
- Success rates are averaged over 30 evaluation episodes for Plug in Socket, Clean Whiteboard, and Press Button.
- FD-VLA is compared with SmolVLA, π0, and DP3, with SmolVLA and π0 evaluated both with and without force inputs.
A. Experimental Setup
The physical evaluation uses a UR5e arm with two RGB cameras, coordinated through real-time data exchange for precise manipulation.
- Experiments use a UR5e robotic arm as the main manipulation platform.
- An Azure Kinect captures the main RGB image and a RealSense D405 captures gripper RGB images.
- The arm and cameras exchange data in real time during manipulation.
B. Experimental Tasks
The study evaluates FD-VLA on three contact-rich tasks with standardized demonstrations, test trials, success criteria, and baseline comparisons. FD-VLA achieves the highest reported overall success and outperforms direct raw-force alternatives while removing test-time sensor requirements.
- Experimental Tasks: The tasks are Clean Whiteboard, Press Emergency Button, and Insert Plug, covering continuous contact, precise pressing, and force-regulated alignment.
- Experimental Tasks: Each task uses 50 training demonstrations and 30 independent test trials with task-specific success criteria.
- Experimental Tasks: FD-VLA is benchmarked against DP3, π0, and SmolVLA under standardized data, tasks, optimization budgets, and evaluation protocols.
- Experimental Tasks: 61.1% mean success rate is achieved by FD-VLA across all tasks, exceeding SmolVLA without force encoding at 23.3%, DP3 at 11.1%, and π0 without force encoding at 46.7%.
- Experimental Tasks: 15.6% improvement is achieved by SmolVLA with raw force inputs, yet FD-VLA remains 22.2% higher; π0 improves by 20.0%, while FD-VLA remains 14.4% higher.
- Experimental Tasks: FD-VLA predicts force representations from visual and state inputs, eliminating force sensors at test time while maintaining superior performance.
D. Ablation Study
The ablation compares force encoding variants under matched contact-rich-task rollouts and finds that FDM with a learnable force token improves performance over direct force encoding. The method also remains robust under novel backgrounds and visual perturbations.
- Ablation setup: Three variants were compared on three contact-rich tasks using 30 independent rollouts under matched initialization, time budget, and success criteria.The variants were no FDM with an MLP force encoder, FDM with an actual force token, and FDM with the proposed learnable token.
- Ablation results: Adding FDM clearly improves results, indicating that deep cross-modal interactions during early fusion drive the observed gains.
- Ablation results: Replacing the learnable token with latent features from ground-truth force measurements lowers mean success rate, because raw force signals are high-dimensional and noisy.The learnable token retains real physical supervision while avoiding the difficulty of directly integrating fixed force features.
- Visual generalization: Under novel backgrounds and photometric perturbations, FD-VLA remains robust without significant policy degradation.The robot uses structural cues under background changes and adapts to illumination and appearance changes.