Source-linked AI summary
DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
TL;DR
Existing dexterous VLA models struggle with visual occlusion, complex contact dynamics, and non-adaptive tactile integration. DeCAL unifies understanding, imagination, and action generation with contact-aware tactile gating and visuo-tactile latent co-imagination, and reports state-of-the-art real-world performance across tasks.
Problem
Dexterous VLA models struggle with severe visual occlusions, complex contact dynamics, homogeneous tactile fusion, and limited explicit modeling of future physical interactions.
Method
DeCAL uses collaborative Mixture-of-Transformers experts for understanding, imagination, and action, with adaptive tactile gating and joint future visuo-tactile latent modeling.
Results
DeCAL consistently achieves state-of-the-art performance across real-world tasks and outperforms the strongest baseline by more than 15% average success rate.
Takeaways & Limitations
The experiments support DeCAL’s effectiveness and generalization for contact-rich dexterous manipulation.
Takeaways & Limitations
The method relies on accurate tactile perception, while the current framework lacks large-scale visuo-tactile pretraining.
Abstract
from arXiv · showhide
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.
1 Introduction
DeCAL addresses the difficulty of contact-rich dexterous manipulation by combining adaptive tactile use, future interaction modeling, and unified understanding-to-action processing. Real-world evaluations report strong performance and generalization.
- Existing dexterous VLA approaches struggle with severe visual occlusions, complex contact dynamics, and limited physical modeling.
- Most tactile-augmented methods use homogeneous fusion and react to current inputs without explicitly modeling future contact transitions.
- DeCAL uses three collaborative experts for understanding, imagination, and action generation with directional knowledge sharing through joint attention.
- Adaptive Visuo-Tactile Fusion selectively injects tactile information through contact-aware gating, while latent co-imagination jointly models future visuo-tactile dynamics.
- DeCAL outperforms the strongest baseline by more than 15% average success rate and achieves 0.27s average inference latency per action chunk.
- The framework unifies visuo-tactile representation learning and reports effectiveness and generalization across real-world experiments.
2 Related Work
Related work has advanced VLA architectures and tactile learning, but existing systems remain largely vision-centric and lack explicit physical-dynamics modeling for dexterous manipulation.
- VLA models transfer semantic and world knowledge from pretrained foundation models to robotic manipulation through diverse robot datasets.
- Recent MoT-based VLA systems unify perception, generation, and action for complex manipulation tasks, yet remain predominantly vision-centric.
- Dexterous hands create self-occlusion and ambiguous contact observations, motivating tactile feedback and self-supervised tactile representation learning.
3 Robot System Setup
The robot system combines dual-arm, multi-fingered hardware with wrist and egocentric vision and fingertip vision-based tactile sensing for contact-dynamics perception.
- The platform uses two 6-DoF UR5 arms and two 22-DoF five-fingered dexterous hands.
- Visual observations come from two wrist-mounted cameras and one egocentric Intel RealSense D435 camera.
- Each fingertip has a 320 × 240 vision-based tactile sensor whose internal camera captures elastic-surface deformation during interaction.
- Tactile inputs include raw images, 6-DoF net force, and deformation maps representing contact patterns, force, and local deformation depth.
4 Method
DeCAL jointly performs scene understanding, future visuo-tactile imagination, and action generation through specialized transformer experts. Its tactile pathway uses contact-aware gating, while latent prediction supplies forward-looking physical context.
- 4.1 Preliminaries: DeCAL predicts action chunks from multimodal observations because existing VLA models lack explicit future interaction dynamics for fine-grained physical reasoning.
- 4.1 Preliminaries: The training objective jointly maximizes the likelihood of future visuo-tactile latents and future actions conditioned on current observations and language.
- 4.1 Preliminaries: A Mixture-of-Transformers architecture integrates understanding, visuo-tactile dynamics foresight, and action generation through three specialized experts.
- 4.2 Model Architecture: The generation expert predicts future visual and tactile observations using non-autoregressive parallel decoding for efficient real-time interaction modeling.
- 4.2 Model Architecture: The action expert uses Factorized Flow Matching to decouple arm and hand motion while jointly processing them for coordinated trajectories.
- 4.2 Model Architecture: A blockwise attention mask propagates information from understanding to generation and then action, while preserving bidirectional attention within earlier experts.
- 4.3 Adaptive Visuo-Tactile Fusion: Contact-aware gating derives tactile weights from a global tactile token, selectively modulating local tactile cross-attention according to overall contact state.
- 4.4 Visuo-Tactile Latent Co-Imagination: Visuo-tactile latent prediction produces future representations from historical and current observations, with visual latents and tactile latents trained through distinct supervision pathways.
5 Experiment
DeCAL is evaluated on six contact-rich dexterous tasks against VLA and tactile-specialist baselines, alongside ablations, generation tests, and out-of-distribution evaluations. It achieves the highest success rate across all tasks and generalizes across several distribution shifts.
- Experimental Setup: DeCAL is evaluated on six contact-rich dexterous tasks, using 100 demonstrations per task and 20 trials by default.The tasks include Wipe Vase, Erase Whiteboard, Assemble Parts, Twist Cap, Pipetting, and Screw Light Bulb.
- Experimental Setup: The evaluation compares DeCAL with two state-of-the-art VLA models, two tactile-specialist policies, and a tactile-augmented InternVLA-A1 variant.The metrics include success rate and progress success rate.
- Real-World Results: DeCAL achieves the highest success rate across all tasks and the best progress success rate on most tasks.Vision-based policies perform reasonably on simpler tasks but struggle with contact-rich manipulation requiring precise physical interaction.
- Analysis: Joint attention propagates learned dynamics to the action expert, supporting more stable and precise action generation for contact-rich manipulation.The experimental analysis connects this information flow to the action-generation stage.
- Generalization: DeCAL generalizes across unseen backgrounds, cluttered environments, lighting conditions, and objects, reaching a 75.0% success rate with an unseen object.This result is reported for the Twist Cap task and substantially outperforms all baselines.
- Ablations: Ablation studies examine Factorized Flow Matching, tactile gating, visual latent generation, and tactile latent generation.Removing Factorized Flow Matching causes noticeable inconsistency and poor coordination between the arms and dexterous hands.
- Visual Generation: DeCAL is also evaluated for future visual generation on Assemble Parts and Twist Cap using Cosmos feature similarity and LPIPS.The comparison evaluates predicted frames against ground-truth future observations, and DeCAL consistently achieves better visual generation than InternVLA-A1.
6 Conclusions
DeCAL is a physically grounded dexterous vision-language-action model that combines multimodal understanding, generation, and action for contact-rich manipulation. Its contact-aware tactile gating and future-oriented visuo-tactile latent imagination support the reported effectiveness and generalization on real-world dexterous tasks.
- Conclusion: DeCAL unifies understanding, generation, and action within a collaborative framework for coherent multimodal reasoning and fine-grained dexterous control.The framework is designed for contact-rich manipulation and leverages tactile feedback.
- Conclusion: Contact-aware tactile gating adaptively fuses visuo-tactile information, while future-oriented latent imagination models visuo-tactile dynamics.The paper describes these mechanisms as providing the policy with implicit world knowledge.
- Conclusion: Extensive real-world dexterous-task experiments demonstrate the effectiveness and generalization of DeCAL.
7 Limitations
The method depends on accurate tactile perception and demonstrations collected without fingertip force feedback, while also lacking large-scale visuo-tactile pretraining. These constraints may affect long-term operation, contact regulation, demonstration quality, and future scaling potential.
- Limitations: Accurate tactile perception can be degraded by sensor noise, calibration errors, and model drift during long-term and high-load operation.The authors identify potential performance degradation under these conditions.
- Limitations: The teleoperation system lacks fingertip force feedback, which may delay contact adjustment and limit tactile-aware demonstration quality.The authors also identify the absence of large-scale visuo-tactile pretraining as a current framework limitation.
- Limitations: The framework does not leverage large-scale visuo-tactile pretraining, leaving scaled diverse tactile dexterous data as future work.
A.1 Training objective
DeCAL jointly trains visual foresight, tactile foresight, and action prediction to model future multimodal dynamics and generate contact-aware actions.
- Training objectives: DeCAL combines visual foresight generation, tactile foresight generation, and action prediction in its training objective.The total loss is a weighted sum of visual, tactile, and action losses.
- Visual foresight: Visual foresight predicts future visual latents from historical frames and supervises them against pretrained COSMOS latent features.The reconstruction loss encourages attention to task-relevant regions.
- Tactile foresight: Tactile foresight predicts raw tactile images, deformation maps, and 6-DoF force vectors to capture local contact dynamics and global tactile interactions.Each representation is trained with a corresponding reconstruction loss.
- Action prediction: Action prediction uses flow matching to learn a transport vector field that maps noisy action chunks toward target actions.The noisy action chunk interpolates between Gaussian noise and the expert action over τ from 0 to 1.
- Inference: During inference, predicted visuo-tactile foresight is integrated with cached visual-language context before final action generation.Prefix tokens store prior context in KV caches, while middle embeddings incorporate predicted dynamics.
B Policy Implementation Details
DeCAL is trained from pretrained VLA weights on a dual-arm, dual-hand real-robot platform using glove-tracker teleoperation data.
- Training setup: Training initializes DeCAL from InternVLA-A1-3B pretrained weights and uses 8 NVIDIA H100 GPUs.Optimization uses AdamW with a 5.0e−5 learning rate, warmup, and decay over 100,000 steps.
- Robot platform: The real-robot platform has two 6-DoF UR5 arms, two 22-DoF SharpaWave five-fingered hands, three cameras, and fingertip tactile sensors.The cameras provide wrist-mounted and egocentric views, while the tactile sensors capture high-resolution contact information.
- Data collection: Robot demonstrations are collected through human teleoperation with a glove-tracker system.Tracker measurements are transformed into arm joint configurations through inverse kinematics, while Manus gloves retarget hand motion.
D Task Details
The evaluation covers six bimanual and dexterous tasks requiring sustained contact, precise alignment, force-sensitive manipulation, or coordinated handoffs.
- Task descriptions: Wipe Vase requires coordinated bimanual cleaning while maintaining sponge contact with a curved vase surface.Success requires removing all visible graffiti marks and returning both objects to the table.
- Task descriptions: Erase Whiteboard requires controlled wiping with stable contact until all visible markings are removed.The robot grasps an eraser and performs a top-to-bottom motion.
- Task descriptions: Assemble Parts requires contact-feedback-driven alignment and insertion of a plug into a socket.The task emphasizes continuous pose adjustment during insertion.
- Task descriptions: Twist Cap requires stabilizing a bottle while rotating and fully removing its cap with a three-finger grasp.The cap must then be placed on the table.
- Task descriptions: Pipetting requires thumb-actuated dispensing, a hand transfer, and precise placement into a holder.The task combines liquid release, object handoff, and placement control.
- Task descriptions: Screw Light Bulb requires coordinated grasping and rotation until the bulb is tightened and illuminates.Continuous contact feedback supports stable grasping and screwing.
E.3 Out-of-Domain (OOD) Stage-Wise Results
Under out-of-domain shifts, DeCAL remains more robust than the baselines, including on substantially different objects where contact geometry must be adapted.
- OOD stage-wise results: DeCAL achieves superior Twist Cap stage-wise performance across unseen backgrounds, lighting, cluttered scenes, and objects.All methods degrade under distribution shifts, but DeCAL shows better robustness across the four settings.
- OOD stage-wise results: 75% final-stage success under unseen objects exceeds the best baseline by 30% points.The unseen cups differ in height, diameter, and shape from training objects.
- Supplementary generalization: Novel-object changes require real-time adaptation of visual localization, grasping height, contact position, and wrist pose.The novel cup differs in color, diameter, shape, and height, making this shift more challenging than environment-level perturbations.
- Supplementary generalization: DeCAL maintains a 75% success rate on the novel object by leveraging tactile feedback and future-oriented latent imagination.The baselines show clear performance drops under the same object-level shift.
- Ablation analysis: Removing visual and tactile latent generation weakens contact-sensitive performance because the policy cannot anticipate evolving observations and contact states.The ablation can cause hesitation or oscillation during plug insertion in Assemble Parts.
- Ablation analysis: Removing Factorized Flow Matching weakens arm-hand coordination by using a shared denoising process for the full action space.The resulting inconsistencies are especially problematic in grasping and bimanual tasks.
G Failure Case Analysis
DeCAL’s failures fall into bimanual coordination problems and limited camera field of view. These issues can destabilize contact or prevent reliable visual tracking during large-range motions.
- Bimanual coordination: Bimanual coordination failures arise when arm motion, hand alignment, or grasp timing are insufficiently synchronized.In the Pipetting task, such discrepancies can destabilize contact and cause the object to slip or deviate from the desired pose.
- Limited field of view: Limited head-mounted camera coverage can move manipulated objects outside the visible region during large-range motions.This makes it difficult for the policy to maintain reliable visual tracking and state estimation.
- Potential mitigations: Wider-angle or fisheye cameras could provide broader workspace coverage, while active perception could select informative viewpoints during execution.These are proposed directions for improving tracking of task-relevant objects and reducing visibility-related failures.