Source-linked AI summary
FAVLA: A Force-Adaptive Fast-Slow VLA model for Contact-Rich Robotic Manipulation
Yao Li, Peiyuan Tang, Wuyang Zhang, Chengyang Zhu, Yifan Duan, Weikai Shi, Xiaodong Zhang, Zijiang Yang, Jianmin Ji, Yanyong Zhang
TL;DR
Contact-rich VLA control is limited by unified-frequency fusion that discards high-frequency force cues and delays corrective action. FAVLA separates slow multimodal reasoning from fast force-conditioned execution, achieving higher success and lower contact forces in contact-rich tasks.
Problem
Unified-frequency VLA pipelines downsample high-frequency force data and execute action chunks largely open loop, limiting timely correction during contact-rich manipulation.
Method
FAVLA uses a slow VLM for semantic and force-history context, a fast action expert for recent force-conditioned correction, a multilayer force adapter, and force-variation-based frequency scheduling.
Results
13.8% higher task success rate and 80.8% average success rate were achieved versus strongest baselines, with lower peak contact forces.
Takeaways & Limitations
Separating slow semantic reasoning from fast contact-aware control improves reactivity and supports lower-force manipulation on contact-rich tasks.
Abstract
from arXiv · showhide
Force/torque feedback can substantially improve Vision-Language-Action (VLA) models on contact-rich manipulation, but most existing approaches fuse all modalities at a single operating frequency. This design ignores the mismatched sampling rates of real robot sensors, forcing downsampling of the high-frequency contact cues needed for reactive correction. Combined with common VLM-action-expert (AE) pipelines that execute action chunks largely open loop between expensive VLM updates, unified-frequency fusion often yields delayed responses to impacts, stick-slip, and force spikes. We propose FAVLA, a force-adaptive fast-slow VLA that decouples slow perception planning from fast contact-aware control. FAVLA runs a slow VLM at a fixed low frequency to encode modalities to produce latent representations and to predict near-future force variation. A fast AE then executes at a variable high frequency, conditioning on the latest force sequence data to generate reactive actions. We further introduce a force adapter that injects high-frequency force features into multiple AE layers, and adaptively schedules the AE's execution frequency based on the VLM's predicted force variation. Extensive experiments on contact-rich tasks demonstrate that FAVLA significantly outperforms baselines, achieving superior reactivity and success rates, especially with a smaller contact force during manipulation.
1. Introduction
Contact-rich manipulation requires rapid force-aware correction, but unified-frequency VLA pipelines downsample contact signals and execute action chunks too open loop. FAVLA separates slow semantic reasoning from fast force-reactive execution and reports higher success with lower contact forces.
- Contact-rich tasks require rapid correction for impacts, stick–slip, and jamming that vision alone weakly observes.
- Unified-frequency approaches imbalance modality tokens and downsample high-frequency force data, weakening access to sparse contact cues.
- FAVLA combines a fixed low-frequency VLM with a high-frequency action expert using recent force and proprioceptive data for closed-loop refinement.
- 13.8% higher task success rate and 80.8% average success rate were achieved versus strongest baselines, alongside lower peak contact forces.
- A force adapter injects high-frequency force feedback into multiple action-expert layers for direct force-conditioned action correction.
- Predicted near-future force variation schedules the action expert’s execution frequency, increasing responsiveness when contact transitions are imminent or underway.
2. Related Works
Related work extends VLA and manipulation policies with multimodal pretraining and physical interaction signals, but typical force-based VLA models still use one operating frequency. FAVLA instead introduces force-injected fast–slow fusion for adaptive action adjustment.
- Vision–Language–Action Models: VLA models use large-scale multimodal pretraining to learn generalist robotic policies, including autoregressive and continuous-action approaches.
- Contact-Rich Manipulation Models: Contact-rich manipulation methods incorporate force or force-related feedback to support rapid corrective behavior while maintaining stable long-horizon execution.
- Contact-Rich Manipulation Models: Typical force-based VLA models align modalities to one frequency, downsampling high-frequency signals and potentially producing open-loop behavior with force spikes.
- Contact-Rich Manipulation Models: FAVLA contrasts with consistent-frequency VLM–AE pipelines by using force-injected fast–slow fusion to adapt actions to contact feedback.
3. Methods
FAVLA separates slow semantic reasoning from fast force-reactive control, allowing multimodal inputs to be processed at their respective sampling rates. Its force-injected action expert and force-adaptive scheduling generate reactive action chunks while preserving reusable visual-language context.
- Force-Injected Fast–Slow VLA Model Architecture: FAVLA combines a slow VLM backbone with a smaller fast action expert to process multimodal inputs at different frequencies and generate action trajectories.The slow component handles semantic scene understanding, while the action expert performs responsive control using the fast-slow architecture.
- Slow VLM Backbone: The slow backbone encodes vision, language, and historical force data, then caches intermediate key/value states for reuse by the fast module.A temporal convolutional tokenizer converts force history into tokens before multimodal encoding and KV-cache construction.
- Fast Force-Injected Action Expert: The fast action expert conditions flow-matching action chunks on latest force measurements, robot state, and the slow backbone’s cached context.A force adapter injects high-frequency force features through cross-attention at multiple transformer layers rather than appending them only to the input sequence.
- Force Variance Head: A force-variance head predicts near-future contact-force volatility from the slow VLM representation to support adaptive action-expert scheduling.The target is an EMA-smoothed, weighted variance of the future 6D force/torque signal, with normalization applied before training the prediction head.
- Training Objectives: FAVLA jointly optimizes action generation and auxiliary force-variance prediction in an end-to-end training objective.The loss combines the action loss with a weighted variance-prediction loss, whose balance is controlled by λ.
- Force-Adaptive Fast–Slow Inference Strategy: The force-adaptive inference strategy runs the VLM once per cycle and repeatedly executes the action expert within an action chunk using the KV cache and latest force observations.The strategy decouples low-frequency visual reasoning from high-frequency force feedback, while fixed noise and temporal ensembling improve consistency across overlapping chunks.
4. Experiments
The experiments evaluate FAVLA on diverse real-world contact-rich tasks, compare it with strong baselines, and test the contributions of force injection, force-variance prediction, and adaptive execution frequency. FAVLA achieves higher success rates, lower peak contact forces, and adaptive frequency changes before demanding interactions.
- Experimental setup: FAVLA is evaluated on USB Insertion, Gear Assembly, Box Flipping, and Board Wiping using vision, force/torque sensing, and teleoperated demonstrations.The first two tasks require millimeter-level precision; the latter two involve dynamic contact interactions.
- Overall performance: FAVLA reaches an average success rate of 80.8%, exceeding π0 by 38.0 percentage points and ForceVLA by 13.8 percentage points.It achieves the best performance across all four contact-rich tasks.
- Overall performance: FAVLA reports 93.3% success on Gear Assembly and 70.0% on Board Wiping, where force-sensitive execution is especially important.The paper attributes these gains to closed-loop force control and reactive adjustments from high-frequency feedback.
- Force control: FAVLA records 7.7 N peak force on Gear Assembly and 9.9 N on Box Flipping, with lower values indicating safer manipulation.The method uses high-frequency closed-loop action control for force regulation.
- Ablation studies: The component ablation improves Box Flipping and Board Wiping from 50% and 10% with vision-only input to 80% and 70% with all proposed components.Force injection raises performance to 65% and 60%, while force-variance prediction further improves Box Flipping to 70%.
- Ablation studies: The adaptive frequency mechanism outperforms fixed ratios across USB Insertion and Gear Assembly and increases updates before high-precision contact phases.It reduces execution to 1× during stable periods with little or no contact.
5. Conclusion
FAVLA combines slow semantic processing with fast force-aware action adjustment in an end-to-end VLA framework. Real-robot experiments report an 80.8% overall success rate, lower peak contact forces, and a 13.8-percentage-point advantage over baseline VLA methods.
- Conclusion: FAVLA fuses slow semantic modalities with fast proprioceptive and force signals in an end-to-end VLA model.Its action expert receives high-frequency force information through a force adapter and varies execution frequency using predicted force variation.
- Conclusion: FAVLA achieves an 80.8% overall success rate and outperforms baseline VLA methods by 13.8 percentage points while reducing peak contact forces.The evaluation covers real-robot contact-rich industrial tasks.
Impact Statement
The paper targets safer, more reliable, and more efficient robotic manipulation in contact-rich and high-precision industrial environments. Its force-adaptive reactive control may also benefit industrial automation and related applications.
- Impact: FAVLA aims to advance deep learning for contact-rich and high-precision industrial robotic manipulation.The stated application scope includes physical-environment operations and industrial automation.
A.1. Teleoperation System
The teleoperation system maps 3D SpaceMouse motion to end-effector motion and converts six force dimensions into audio cues. This provides directional force information despite the absence of direct force feedback.
- Teleoperation: A 3D SpaceMouse controls the robot in end-effector space through scaled relative-motion mapping.The operator’s SpaceMouse movement is mapped to the robot’s relative end-effector motion.
- Teleoperation: Six force dimensions are mapped to distinct 220–880 Hz audio frequencies, with volume proportional to force.The audio cues let the operator identify which direction is experiencing force.
A.2. Data Collection Details
Demonstrations were collected through teleoperation while synchronously recording robot states, force/torque measurements, and camera observations across multiple sensor frequencies.
- A human operator controlled the robot arm with a 3D SpaceMouse while the system recorded states, force/torque measurements, and camera observations.
- Initial robot-arm and object positions were randomized to increase trajectory diversity and support more generalizable manipulation strategies.
- The teleoperation setup used end-effector poses and 6D force/torque readings at 200 Hz, plus external and wrist cameras at 30 Hz and 640 × 360 resolution.
- All recorded streams were synchronized and downsampled to 30 Hz before conversion to the LeRobot v2.1 format.
A.3. Dataset Statistics
The collected dataset covers four contact-rich manipulation tasks with 260 trajectories and 198,250 frames, totaling approximately 1.84 hours of real-world data.
- 260 trajectories and 198,250 frames were collected across four contact-rich manipulation tasks.
- The dataset represents approximately 1.84 hours of real-world time.
- Although relatively small, the dataset uses high trajectory diversity to support effective task learning.
A.4. Task Description
The evaluation covers four tasks requiring precision or force-responsive contact control, using multimodal observations and qualitative analyses of generalization and failure cases.
- Task Description: USB Insertion requires sub-millimeter alignment before fully inserting a drive into a standard USB port.
- Task Description: Gear Assembly requires force feedback to align a gear with a hidden pillar and ensure proper tooth meshing during final engagement.
- Task Description: Box Flipping requires adapting force direction and magnitude to avoid both cardboard damage and slipping.
- Task Description: Board Wiping requires stable eraser contact and sufficient force to remove board markings effectively.
- Model and Evaluation: The robot receives camera images, proprioceptive state, 6-axis force/torque data, and language instructions, and outputs action chunks of size 32.
- Qualitative Results: Qualitative results show generalization to rotated sockets, unseen gear colors, unseen box sizes, and unseen marker colors.
- Failure Cases: Failure cases include premature USB release under misalignment, excessive gear-assembly force triggering a safety stop, box slipping from insufficient force, and incomplete board wiping.