PaperlayerView exact revision on arXiv

Source-linked AI summary

TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation

Yujie Zang, Yuhang Zheng, Xian Nie, Yupeng Zheng, Shuai Tian, Songen Gu, Chen Gao, Zining Wang, Shuicheng Yan, Wenchao Ding

arXiv:2606.11184v1cs.RO

TL;DR

Contact-rich manipulation requires anticipating evolving physical interactions, but prior methods rarely model the asymmetric timing between global force signals and local tactile responses. TacForeSight forecasts short-horizon tactile latents from wrist force/torque dynamics and uses them in a predictive policy. Real-robot experiments across five tasks and three perturbation settings show consistent improvements over existing baselines, especially under dynamic disturbances.

  • Problem

    Contact-rich manipulation requires continuous regulation of changing interactions, while prior methods rarely model the asymmetric spatiotemporal roles of global force and local tactile sensing.

  • Method

    TacForeSight combines a force-conditioned tactile world model with a predictive tactile-conditioned policy that uses forecast tactile latents as anticipatory contact priors.

  • Results

    TacForeSight achieves state-of-the-art performance across five contact-rich tasks and three in-process perturbation settings, with a 79.0% average completion score on the five tasks.

  • Takeaways & Limitations

    Latent-space tactile prediction provides a compact mechanism for proactive contact reasoning and robust contact regulation and disturbance recovery in real-time manipulation.

Abstract

from arXiv · show

Contact-rich manipulation requires robots to continuously perceive and regulate evolving physical interactions under dynamic contact transitions or complex surface geometries. Recent imitation learning methods improve contact-aware control by incorporating tactile or force feedback, but they rarely model the asymmetric spatiotemporal roles of global force and local tactile sensing. To address this, we propose TacForeSight, a lightweight force-conditioned tactile foresight framework for real-time manipulation. The core component is TacForceWM, a tactile world model that predicts short-horizon tactile latent dynamics from dual-finger tactile observations conditioned on high-frequency wrist force and torque signals. Another key component, the Predictive Tactile-Conditioned Policy, leverages the predicted latents as anticipatory contact priors, models the current-to-future tactile evolution via cross-attention, and adaptively fuses visuo-tactile features through a tactile-guided gating module. By forecasting purely within a compact latent space, TacForeSight enables proactive contact reasoning with efficient real-time inference suitable for high-frequency manipulation control. Real-robot experiments on five representative tasks and three in-process perturbation settings show that TacForeSight consistently outperforms existing baselines, particularly under dynamic contact disturbances. All models and datasets will be made publicly available on the project website at https://tacforesight.github.io/ProjectPage.

I. INTRODUCTION

TacForeSight addresses contact-rich manipulation by modeling the predictive relationship between global wrist force/torque signals and local tactile evolution. Its force-conditioned latent forecasting and predictive policy improve robustness across nominal tasks and dynamic perturbations while retaining real-time inference.

  • Contact-rich manipulation requires continuous regulation of evolving interactions because dynamic transitions and complex surfaces can cause slippage, misalignment, or contact loss.
  • Existing methods commonly fuse force and tactile signals or use them as passive feedback, limiting proactive modeling of temporally coordinated contact interactions.
  • Wrist-wrench changes can precede detectable fingertip tactile responses, motivating explicit modeling of force-to-tactile cross-modal dynamics.
  • TacForeSight combines TacForceWM, which predicts short-horizon tactile latent dynamics, with a predictive policy using future tactile latents as anticipatory contact priors.
  • The framework uses efficient latent-space forecasting and achieves a real-time inference rate of 20 Hz on an RTX 4090D GPU.
  • Experiments across five tasks and three in-process perturbation settings show state-of-the-art performance for contact establishment, maintenance, and recovery from dynamic disturbances.

B. World Model for Robotic Manipulation

TacForeSight extends robotic world modeling from visual prediction to short-horizon tactile evolution conditioned on wrist force/torque signals. Its cascaded design forecasts compact tactile latents and uses them to support lightweight flow-based action prediction.

  • Prior robotic world models mainly predict visual or video states, whereas contact-rich manipulation also requires anticipating tactile interaction changes.
  • TacForceWM predicts short-horizon tactile evolution in a compact latent space conditioned on high-bandwidth wrist force/torque signals.
  • The framework first pretrains a force-conditioned world model and then uses predicted tactile dynamics for lightweight flow-based action sequence prediction.
  • The tactile world model combines a tactile tokenizer, force encoder, and latent dynamics predictor to model force-conditioned evolution from tactile observations and wrist wrench signals.
  • The tokenizer converts dense dual-finger tactile fields into compact frame-level tokens using a hybrid CNN-Transformer.
  • Shared encoder weights, positional embeddings, and finger-specific identity embeddings support spatial and cross-finger interaction modeling before producing a joint frame-level tactile latent.

2) Force Encoder:

The force encoder converts high-rate wrist force/torque sequences into tactile-aligned conditions for latent tactile dynamics prediction. Causal temporal processing preserves ordering while capturing multiscale force variations.

  • The encoder extracts temporally consistent physical-interaction conditions from high-rate wrist force/torque signals.
  • It maps six-axis force/torque sequences sampled above the tactile rate into a tactile-aligned condition sequence.
  • Projection, dilated causal 1D convolutions, and causal downsampling capture multiscale force variations while aligning them with tactile latents.

3) Latent Dynamics Predictor:

The latent dynamics predictor forecasts temporally coherent future tactile representations from force-conditioned inputs in a compact latent space. Its training combines prediction of tactile latents and temporal differences with SIGReg-based latent regularization.

  • Latent forecasting: The predictor forecasts future tactile representations in latent space using a force-conditioned latent Transformer backbone.The force condition is injected through adaptive layer normalization to modulate intermediate features.
  • Latent forecasting: Chunk-based forecasting produces temporally coherent dynamics instead of frame-wise one-step predictions.Forecasting latent representations rather than raw tactile observations keeps prediction fast and lightweight.
  • Training objectives: The prediction objective supervises absolute future tactile latents and their first-order temporal dynamics.This is intended to capture tactile evolution and reduce over-smoothed latent predictions.
  • Training objectives: SIGReg regularizes the tactile latent distribution toward an isotropic Gaussian structure to avoid representation collapse.The final world-model objective combines prediction and latent regularization losses.

B. Predictive Tactile-conditioned Policy

The predictive tactile-conditioned policy combines visual, proprioceptive, current tactile, predicted tactile, and force features to predict future action sequences. Cross-attention explicitly links current tactile states with anticipated tactile dynamics before producing a compact tactile representation for policy conditioning.

  • Multimodal conditioning: The policy extracts visual, proprioceptive, tactile, predicted tactile, and aligned wrist force/torque features from multimodal observations.These features form the conditioning inputs for action prediction.
  • Action prediction: A conditional flow-matching action head predicts future action sequences from the fused multimodal policy features.The policy therefore uses tactile foresight as an anticipatory conditioning signal for action prediction.
  • Current–future tactile interaction: Current tactile latents serve as queries, while predicted future tactile latents provide keys and values in cross-attention.Learnable temporal embeddings preserve sequence order before attention.
  • Current–future tactile interaction: The residual cross-attention structure augments the immediate contact state with predicted tactile dynamics.The enhanced tactile sequence is averaged over time into a compact future-aware tactile representation.

3) Adaptive Visuo-Tactile Fusion:

The adaptive visuo-tactile fusion module uses predictive tactile representations to regulate visual and tactile contributions at the feature-channel level. The resulting fused representation is combined with proprioception and passed to a conditional flow-matching action head.

  • Adaptive fusion: Predictive tactile representations dynamically regulate modality contributions through adaptive channel-wise visuo-tactile fusion.This differs from direct concatenation and token-level fusion approaches.
  • Tactile-guided gating: A sigmoid-activated MLP generates a tactile-guided gate conditioned on the future-aware tactile representation.The gate controls how visual context and predictive tactile dynamics are balanced.
  • Adaptive fusion: The module projects visual and tactile features into a shared space and performs channel-wise adaptive fusion.The fused visuo-tactile feature is concatenated with proprioception to form the global policy condition.
  • Action prediction: The flow-matching action head refines a Gaussian-noise action chunk by integrating a learned ordinary differential equation.The output is a denoised future action chunk conditioned on the global policy representation.

IV. EXPERIMENTS

The experiments evaluate the method on five contact-rich manipulation tasks and three in-process perturbation settings. The setup includes both nominal tasks and disturbances that alter ongoing contact states.

  • Experimental setup: The evaluation covers five core tasks: Vase Wiping, Card Swiping, Tube Adjustment and Insertion, Bulb Insertion and Locking, and Wire Insertion.These tasks span sliding contact, constrained alignment, and flexible-wire insertion.
  • Experimental setup: Three perturbation settings introduce height, angle, and pose disturbances during wiping, swiping, and tube adjustment.The perturbations are evaluated as in-process disruptions to ongoing contact states.
  • Evaluation protocol: Table I reports manipulation performance on both contact-rich tasks and in-process perturbation tasks.The experiment section also includes execution sequences and ablation studies.

A. Experimental Setup

Experiments use a real robotic platform with multimodal sensing and compare TacForeSight against representative visuomotor, multimodal, tactile-aware, force-aware, and reactive policies.

  • Experimental Platform: The platform uses a 7-DoF UFactory xArm7 with a Robotiq 2F-85 gripper, wrist camera, 6-axis force/torque sensor, and two fingertip tactile sensors.Images and tactile observations are captured at 30 Hz, while wrist wrench data are recorded at 120 Hz.
  • Training: TacForeSight’s 11.8M-parameter tactile world model is trained on 2,700 force-tactile interaction episodes before its encoder and predictor are frozen for the 68.9M-parameter downstream policy.Training uses task-specific demonstrations and diverse contact interaction data.
  • Baselines: The comparison includes vision-proprioceptive DP, multimodal DP+Tactile+Force, tactile-aware KineDex, force-aware FoAR, and reactive visual-tactile RDP.FoAR uses a 2D RGB encoder here for fair comparison.

B. Benchmark Tasks

The benchmark covers five contact-rich tasks and evaluates both nominal execution and recovery from in-process disturbances using task-specific completion scores.

  • Benchmark Tasks: The five tasks span sustained sliding contact, sequential constrained manipulation, and precise flexible-wire insertion: Vase Wiping, Card Swiping, Tube Adjustment & Insertion, Bulb Insertion & Locking, and Wire Insertion.Wiping and Swiping emphasize stable sliding, while Tube, Bulb, and Wire tasks require alignment or precise contact control.
  • Evaluation Settings: Evaluation includes standard execution under nominal demonstrations and perturbation-aware manipulation with recovery demonstrations involving external disturbances.Successful perturbation trials require re-establishing stable contact and completing the task.
  • Evaluation Metrics: Each method and task uses 20 independent trials, with scores based on completed interaction length, sequential-stage completion, or full wire insertion.Perturbation-aware trials count as successful only when the policy recovers and completes the task.
  • Main Results: TacForeSight achieves the best performance across nominal tasks and in-process perturbation settings.The evaluation reports quantitative results in Table I and execution sequences in Fig. 3.
  • Nominal Performance: The method reaches a 79.0% average completion score across the five representative contact-rich manipulation tasks.Predictive tactile-latent priors provide anticipatory contact information for contact establishment, maintenance, and task completion.
  • Perturbation Performance: Under height, angle, and pose disturbances, TacForeSight achieves 90%, 85%, and 85%, respectively, with an 86.7% average score.Qualitative sequences show stable-contact re-establishment, end-effector correction, and task continuation after disturbances.

2) Predictive Tactile Representation Analysis:

The analysis examines whether tactile latents forecast contact transitions and discriminate interaction patterns, while ablations test conditioning modalities for tactile-world-model prediction.

  • Temporal Prediction: Temporal analysis projects current tactile latents, predicted future latents, and their differences into a low-dimensional PCA space during Bulb Insertion and Locking and Vase Wiping.The visualization targets anticipatory responses around contact-state transitions.
  • Temporal Prediction: Predicted tactile latents show contact-related variations approximately 200 ms before similar changes appear in current tactile latents.Latent differences become prominent around key contact-stage transitions.
  • Representation Discriminability: On unseen pressing, twisting, and sliding episodes, sequence-level embeddings are formed by temporal max pooling of frame-level tactile latents.The analysis evaluates representation generalization beyond the training distribution.
  • Representation Discriminability: The t-SNE embeddings form well-separated clusters according to the underlying contact pattern, indicating contact-discriminative representations of local deformation and force-induced changes.This supports representation generalization across the evaluated interaction episodes.
  • Conditioning Ablation: The world-model ablation compares no condition, RGB image, robot state, and wrist wrench conditioning using MSE, cosine similarity, and symmetric KL divergence.The supplied result passage states that wrist wrench conditioning performs best across all metrics.

2) Policy Ablation:

Policy ablations show that predicted tactile features, force conditioning, current–future cross-attention, and adaptive visuo-tactile gating contribute to perturbation performance and recovery efficiency.

  • Policy Variants: The policy ablation compares parallel fusion with variants removing force conditioning, predicted tactile features, current–future cross-attention, or adaptive gating.Each variant is evaluated under the same real-world protocol.
  • Ablation Results: The full model achieves 100% on Wiping, 85% on Wiping-P, and 90% on Swiping-P, outperforming the ablated variants in the reported settings.Parallel force–tactile concatenation performs poorly under perturbations, while removing predicted tactile features lowers Wiping-P and Swiping-P performance.
  • Ablation Results: Removing force conditioning or current–future cross-attention degrades performance, supporting their roles in using predicted tactile information.The ablation specifically links both components to effective predictive-tactile use.
  • Adaptive Gating: Removing the adaptive gate increases recovery time from 2.56 s to 4.06 s and reduces Wiping-P performance.Gate responses vary across contact stages when projected alongside the tactile resultant-force trajectory.
  • Conclusion: TacForeSight uses force-conditioned latent tactile prediction to move beyond reactive feature fusion toward proactive contact reasoning and disturbance recovery.The conclusion frames latent-space tactile prediction as an efficient mechanism for predictive physical reasoning in real-time manipulation.
Loading 2606.11184v1…