PaperlayerView exact revision on arXiv

Source-linked AI summary

FD-VLA: Force-Distilled Vision-Language-Action Model for Contact-Rich Manipulation

Ruiteng Zhao, Wenshuo Wang, Yicheng Ma, Xiaocong Li, Francis E. H. Tay, Marcelo H. Ang, Haiyue Zhu

arXiv:2602.02142v2cs.ROcs.CV

TL;DR

Vision-only VLA policies can miss contact dynamics, while force sensors add hardware dependence. FD-VLA distills a force token from visual and state inputs, injects it into a pretrained VLM, and reports stronger performance than direct raw-force inputs and other baselines, including sensor-free deployment.

  • Problem

    Vision-only policies poorly capture subtle contact effects, motivating force-aware VLA methods that incorporate information unavailable from images alone.

    Source: §I · p. 1

  • Method

    FD-VLA trains a Force Distillation Module to predict a force token from vision and robot state, then fuses that token with VLM inputs without force measurements at inference.

    Source: §I · p. 1, B. Overall Framework · pp. 3–4

  • Results

    FD-VLA outperforms models using direct raw force inputs and other baselines, while predicting force representations from vision and state without test-time force sensors.

    Source: C. Main Results · pp. 6–7

  • Takeaways & Limitations

    The approach supports force-aware contact-rich manipulation on robots without dedicated force sensors and adds force-vision-state cross-modal alignment.

    Source: §I · p. 2

Abstract

from arXiv · show

Force sensing is a crucial modality for Vision-Language-Action (VLA) frameworks, as it enables fine-grained perception and dexterous manipulation in contact-rich tasks. We present Force-Distilled VLA (FD-VLA), a novel framework that integrates force awareness into contact-rich manipulation without relying on physical force sensors. The core of our approach is a Force Distillation Module (FDM), which distills force by mapping a learnable query token, conditioned on visual observations and robot states, into a predicted force token aligned with the latent representation of actual force signals. During inference, this distilled force token is injected into the pretrained VLM, enabling force-aware reasoning while preserving the integrity of its vision-language semantics. This design provides two key benefits: first, it allows practical deployment across a wide range of robots that lack expensive or fragile force-torque sensors, thereby reducing hardware cost and complexity; second, the FDM introduces an additional force-vision-state fusion prior to the VLM, which improves cross-modal alignment and enhances perception-action robustness in contact-rich scenarios. Surprisingly, our physical experiments show that the distilled force token outperforms direct sensor force measurements as well as other baselines, which highlights the effectiveness of this force-distilled VLA approach.

I. INTRODUCTION

FD-VLA addresses the limitations of vision-only contact-rich manipulation by distilling force representations from vision and robot state instead of relying on force sensors. Its Force Distillation Module adds force-vision-state fusion, enabling force-aware reasoning and improved cross-modal alignment for sensor-free deployment.

  • Vision-only policies often miss contact effects and struggle with occlusions, lighting shifts, and depth ambiguities.Source: §I · p. 1
  • Force and tactile sensing provide direct information about contact dynamics, compliance, and physical interactions that vision often cannot capture.Source: §I · p. 1
  • FD-VLA uses a Force Distillation Module to condition a learnable query on visual observations and robot states, producing a predicted force token.Source: §I · pp. 1–2
  • During deployment, FD-VLA injects the distilled force token into the VLA without requiring direct sensor measurements.Source: §I · pp. 1–2
  • The additional force-vision-state fusion before the VLM improves cross-modal alignment and perception-action robustness in contact-rich manipulation.Source: §I · p. 2

II. RELATED WORKS

Related work establishes the need for multimodal sensing in contact-rich manipulation and motivates cross-modal knowledge distillation. FD-VLA applies this direction by distilling implicit force representations from vision and state inputs, avoiding dedicated tactile hardware at deployment.

III. METHODOLOGY

FD-VLA integrates a predicted force token into a pretrained VLM while preserving its visual-linguistic semantics. Its FDM predicts force from visual observations and robot state, enabling sensor-free inference and force-aware action generation.

Multimodal inputs

  • FD-VLA uses language, visual observations, robot state, and predicted force as multimodal inputs to the VLM.Visual, language, and state embeddings are projected to the VLM hidden dimension before fusion.Source: B. Overall Framework · pp. 3–4

Force prediction

  • The FDM predicts a latent force representation from real-time visual observations and robot proprioceptive state.A learnable force-prediction token is trained against latent representations of actual force signals, but force measurements are unnecessary at inference.Source: B. Overall Framework · p. 4

VLM integration

  • The pretrained VLM remains frozen, while attention masking separates perceptual streams from state and force control tokens.This separation is intended to preserve pretrained visual-linguistic representations while allowing downstream action integration.Source: B. Overall Framework · p. 4

Action generation

  • A transformer-based action expert with conditional flow matching maps fused multimodal embeddings to action sequences.The architecture uses the VLM features to support downstream control without additional sensing requirements.Source: B. Overall Framework · p. 4

Framework motivation

C. Force Distillation Module (FDM)

The Force Distillation Module learns a compact force representation from vision and robot state, using actual force only as a training-time supervisory signal. Its attention-based prediction branch is designed to produce a denoised, task-relevant proxy for contact forces.

Module purpose

  • FDM generates a compact, state-aware force representation from visual and proprioceptive context without tactile hardware at inference.The method exploits correlations among visual cues, joint torques, velocities, and positions to infer contact forces.Source: C. Force Distillation Module (FDM) · p. 4

Denoised representation

  • FDM mitigates raw-signal noise by learning a supervised latent embedding that filters high-frequency artifacts while preserving manipulation-relevant dynamics.The resulting embedding serves as a task-relevant proxy for physical force.Source: C. Force Distillation Module (FDM) · p. 4

Training supervision

  • The actual-force branch projects robot force measurements into the predicted-force embedding space for training supervision only.The projected actual-force embedding is not required during deployment.Source: C. Force Distillation Module (FDM) · p. 4

Prediction branch

  • The prediction branch uses a learned query token and attention over image and proprioceptive embeddings to generate the latent force feature.The force-latent generation is formulated as retrieval conditioned on vision-state context.Source: C. Force Distillation Module (FDM) · p. 4

Attention design

  • The query token is included in both key and value sets, enabling self-conditioning through the residual path.This design injects the learned contact prior into context aggregation.Source: C. Force Distillation Module (FDM) · p. 4

Distillation objective

  • An auxiliary distillation loss aligns predicted and actual-force feature representations.This alignment connects the prediction and actual-force branches during training.Source: C. Force Distillation Module (FDM) · p. 4

D. Directional Attention Masking Mechanism

Directional attention masking separates frozen vision-language perception from state and force control while permitting control tokens to use perceptual context. This one-way information flow preserves pretrained alignment during multimodal fusion.

Stream separation

  • The input tokens are divided into a frozen perceptual stream and a control stream containing state and predicted-force tokens.The perceptual stream contains vision and language features, while the control stream contains state and force features.Source: D. Directional Attention Masking… · p. 4

Information flow

  • The directional mask allows perceptual tokens to attend only to perceptual tokens, while control tokens can attend to perceptual tokens.Control-token self-attention is constrained by token order through the mask.Source: D. Directional Attention Masking… · p. 5

Preserved semantics

  • This masking preserves pretrained vision-language alignment while fusing proprioception and force with perceptual context for action prediction.The mechanism keeps the perceptual stream isolated from control-stream feedback.Source: D. Directional Attention Masking… · p. 5

E. Action Expert

The action expert predicts action chunks from VLM features and noisy actions using conditional flow matching. It learns a velocity field rather than stepwise residuals, supporting chunked action prediction.

Action-chunk prediction

  • The action expert is a transformer that predicts an action chunk conditioned on VLM features.The predicted chunk contains actions from the current step through a future horizon H−1.Source: E. Action Expert · p. 5

Flow-matching objective

  • Conditional flow matching trains the policy to output a velocity field from noisy actions and VLM features.The noisy action interpolation uses Gaussian noise and a sampled time variable.Source: E. Action Expert · p. 5

Policy formulation

  • Learning a velocity field rather than stepwise residuals is suited to chunked action prediction.Source: E. Action Expert · p. 5

F. Overall Training Objective

FD-VLA trains with a standard policy-learning loss jointly combined with a force-distillation loss, whose influence is controlled by λ.

IV. EXPERIMENTS

The experiments evaluate FD-VLA through real-world manipulation trials and report success rates for three contact-rich tasks against several VLA and control baselines.

  • The evaluation uses real-world experiments and analytical studies to assess FD-VLA.Source: §IV · p. 5
  • Success rates are averaged over 30 evaluation episodes for Plug in Socket, Clean Whiteboard, and Press Button.Source: Figure 6 · p. 6
  • FD-VLA is compared with SmolVLA, π0, and DP3, with SmolVLA and π0 evaluated both with and without force inputs.Source: Figure 6 · p. 6

A. Experimental Setup

The physical evaluation uses a UR5e arm with two RGB cameras, coordinated through real-time data exchange for precise manipulation.

B. Experimental Tasks

The study evaluates FD-VLA on three contact-rich tasks with standardized demonstrations, test trials, success criteria, and baseline comparisons. FD-VLA achieves the highest reported overall success and outperforms direct raw-force alternatives while removing test-time sensor requirements.

Experimental Tasks

  • The tasks are Clean Whiteboard, Press Emergency Button, and Insert Plug, covering continuous contact, precise pressing, and force-regulated alignment.Source: B. Experimental Tasks · p. 6
  • Each task uses 50 training demonstrations and 30 independent test trials with task-specific success criteria.Source: B. Experimental Tasks · p. 6
  • FD-VLA is benchmarked against DP3, π0, and SmolVLA under standardized data, tasks, optimization budgets, and evaluation protocols.Source: C. Main Results · p. 6
  • 61.1% mean success rate is achieved by FD-VLA across all tasks, exceeding SmolVLA without force encoding at 23.3%, DP3 at 11.1%, and π0 without force encoding at 46.7%.Source: C. Main Results · p. 6
  • 15.6% improvement is achieved by SmolVLA with raw force inputs, yet FD-VLA remains 22.2% higher; π0 improves by 20.0%, while FD-VLA remains 14.4% higher.Source: C. Main Results · p. 7
  • FD-VLA predicts force representations from visual and state inputs, eliminating force sensors at test time while maintaining superior performance.Source: C. Main Results · p. 7

D. Ablation Study

The ablation compares force encoding variants under matched contact-rich-task rollouts and finds that FDM with a learnable force token improves performance over direct force encoding. The method also remains robust under novel backgrounds and visual perturbations.

Ablation setup

  • Three variants were compared on three contact-rich tasks using 30 independent rollouts under matched initialization, time budget, and success criteria.The variants were no FDM with an MLP force encoder, FDM with an actual force token, and FDM with the proposed learnable token.Source: D. Ablation Study · p. 7

Ablation results

  • Adding FDM clearly improves results, indicating that deep cross-modal interactions during early fusion drive the observed gains.Source: D. Ablation Study · p. 7
  • Replacing the learnable token with latent features from ground-truth force measurements lowers mean success rate, because raw force signals are high-dimensional and noisy.The learnable token retains real physical supervision while avoiding the difficulty of directly integrating fixed force features.Source: D. Ablation Study · p. 7

Visual generalization

  • Under novel backgrounds and photometric perturbations, FD-VLA remains robust without significant policy degradation.The robot uses structural cues under background changes and adapts to illumination and appearance changes.Source: E. Model Generalization · p. 7, Figure 7 · p. 7

Related papers

Loading 2602.02142v2…