PaperlayerView exact revision on arXiv

Source-linked AI summary

TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance

Zhemeng Zhang, Jiahua Ma, Xincheng Yang, Xin Wen, Yuzhi Zhang, Boyan Li, Yiran Qin, Jin Liu, Can Zhao, Li Kang, Haoqin Hong, Zhenfei Yin, Philip Torr, Hao Su, Ruimao Zhang, Daolin Ma

arXiv:2601.20239v6cs.RO

TL;DR

Fine-grained contact-rich manipulation remains difficult because tactile feedback is underused. TouchGuide steers pretrained visuomotor policies with task-specific tactile guidance, and experiments across five tasks show stronger performance than prior visuo-tactile methods while TacUMI supports affordable, precise tactile-data collection.

  • Problem

    Fine-grained and contact-rich manipulation remains challenging because robots underuse tactile feedback.

  • Method

    TouchGuide uses two-stage inference-time action-space steering: visual sampling proposes a coarse action, then a task-specific CPM refines it using tactile feasibility.

  • Results

    TouchGuide outperforms state-of-the-art visuo-tactile policies across five challenging contact-rich tasks, reaching 36.2% versus RDP at 30.3% and PolicyConsensus at 24.7%.

  • Takeaways & Limitations

    The approach provides cross-policy tactile fusion without retraining the base policy and uses TacUMI to collect precise tactile data with direct feedback.

  • Takeaways & Limitations

    The CPM remains task-specific, increasing training time and cost and requiring retraining for new tasks.

Abstract

from arXiv · show

Fine-grained and contact-rich manipulation remain challenging for robots, largely due to the underutilization of tactile feedback. To address this, we introduce TouchGuide, a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space. Specifically, TouchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy at inference time. First, the policy produces a coarse, visually-plausible action using only visual inputs during early sampling. Second, a task-specific Contact Physical Model (CPM) provides tactile guidance to steer and refine the action, ensuring it aligns with realistic physical contact conditions. Trained through contrastive learning on limited expert demonstrations, the CPM provides a tactile-informed feasibility score to steer the sampling process toward refined actions that satisfy physical contact constraints. Furthermore, to facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system. TacUMI achieves a favorable trade-off between precision and affordability; by leveraging rigid fingertips, it obtains direct tactile feedback, thereby enabling the collection of reliable tactile data. Extensive experiments on five challenging contact-rich tasks, such as shoe lacing and chip handover, show that TouchGuide consistently and significantly outperforms state-of-the-art visuo-tactile policies.

I. INTRODUCTION

TouchGuide fuses visual and tactile information through inference-time steering in a low-dimensional action space, while TacUMI supports precise tactile-data collection. Across five challenging contact-rich tasks, the method outperforms prior visuo-tactile policies and generalizes across policies and tactile modalities.

  • Motivation: Fine-grained contact-rich tasks require visual scene understanding together with localized tactile feedback for precise control.Existing fusion methods use feature-level or policy-level strategies, but policy-level approaches often require multiple single-modality policies and may miss cross-modal correlations.
  • TouchGuide: TouchGuide fuses modalities in the low-dimensional action space by steering a pretrained diffusion or flow-matching visuomotor policy during inference.The policy first generates a visually feasible coarse action, then a task-specific Contact Physical Model refines it using tactile guidance and contact constraints.
  • TacUMI: TacUMI combines low cost, high-precision localization, direct tactile feedback, and synchronized visual-tactile acquisition for expert demonstrations.Its design uses Vive-tracker localization and rigid fingertips to support reliable data collection for fine-grained manipulation.
  • Related systems: TacUMI uses Vive Trackers to reduce cost while meeting precision requirements and minimizes finger linkage for more direct haptic feedback.These choices target the accuracy limitations of SLAM-based collection and improve tactile-data quality.

III. DATA COLLECTION SYSTEM: TACUMI

TacUMI is a UMI-like handheld gripper for collecting synchronized visual, tactile, and proprioceptive data with accurate low-cost localization. Its processing pipeline converts tracked poses into robot-relative training states while retaining absolute end-effector pose for proprioceptive conditioning.

  • System Overview: TacUMI is a UMI-like handheld gripper that collects tactile data without relying on proprioception.The system overview emphasizes a handheld collection device with unified hardware and software interfaces.
  • Accuracy and Low-cost: A Vive tracker with two Lighthouse base stations provides accurate localization at a base cost of about $720.This addresses the insufficient accuracy and tracking loss associated with SLAM-based localization in feature-sparse scenes.
  • Usability and Synchronization: TacUMI streams visual, tactile, and proprioceptive observations through one timestamped interface at 30Hz.It captures 640×480×3 visual data, 200×350×3 tactile images, and dense force fields computed online by an onboard CPU.
  • Data Processing for Training: The coordinate pipeline aligns the robot base frame with the Vive world frame and transforms tracked end-effector poses into the robot base frame.The system defines world, tracker, end-effector, and robot-base frames before collecting transformed pose sequences.
  • Data Processing for Training: Training uses relative end-effector pose rather than absolute pose for the action dataset format.The relative pose is considered more effective for training, while absolute end-effector pose is retained as robot state so π0.5 can perceive its real-world pose.

IV. METHOD: TOUCHGUIDE

TouchGuide performs inference-time visuo-tactile fusion by steering a pretrained diffusion or flow-matching visuomotor policy with a task-specific Contact Physical Model. The CPM learns feasibility from aligned visual-tactile observations and actions, then supplies guidance during action sampling.

  • TouchGuide Framework: TouchGuide applies classifier-guidance principles to steer pretrained diffusion-based or flow-matching visuomotor policies during inference.The framework supports diffusion policies and flow-matching action experts such as π0.5.
  • Contact Physical Model: The CPM produces a task-specific feasibility score from tactile feedback, visual observations, and coarse actions generated by the pretrained policy.DINOv2 encodes tactile and visual observations before Transformer fusion produces a latent observation representation.
  • Contact Physical Model: The feasibility score is the cosine similarity between latent observation and action representations.The action representation is produced by an action encoder consisting of a 1D CNN and MLP.
  • CPM Training: Contrastive learning pairs observations and actions from the same timestep positively and pairs actions from different timesteps negatively.The CPM is trained from limited expert demonstrations using a standard contrastive loss.
  • CPM Training: Noise-augmented ground-truth actions train the CPM to operate in the noisy action space generated during early policy inference.Geometric noise is used with diffusion policies, while flow-matching policies use linear interpolation.
  • Inference Procedure: During inference, the action trajectory is sampled from noise and processed through diffusion or flow-matching policy updates, with touch guidance applied during designated steps.Algorithm 1 returns the refined action trajectory after iterating through the sampling process.

C. Steering the Policies via Touch Guidance

TouchGuide uses CPM-derived feasibility gradients to steer diffusion and flow-matching action sampling toward actions that better adhere to contact physics. In action space, this is framed as moving the policy distribution toward the real-world task distribution.

  • Touch Guidance: TouchGuide applies touch guidance during the sampling process to generate actions that better adhere to contact physics.The guidance is applied to the base policy’s action generation rather than requiring a separately trained multimodal policy.
  • Touch Guidance: The CPM’s feasibility score replaces the classifier objective in guidance gradients for both diffusion and flow-matching policies.The resulting guidance updates steer sampling toward actions with higher CPM-assessed feasibility.
  • Action-Space Interpretation: TouchGuide steers the policy distribution Q_policy toward Q_real, defined as the optimal distribution for accomplishing the task in the real world.The paper presents this action-space view as the framework’s contact-physics steering interpretation.
  • Action-Space Interpretation: Feature-level and policy-level fusion methods lack an explicit mechanism to steer their action-space distributions toward Q_real.The paper characterizes feature-level fusion as potentially over-attending to vision and policy-level composition as multiplicative in action space.

V. EXPERIMENTS

The experiments evaluate TouchGuide and TacUMI across five challenging contact-rich tasks spanning long-horizon, collaborative, and low-data precise manipulation. The evaluation examines tactile benefits, comparisons with prior policies, cross-setting generalization, robustness to visual changes, and data quality.

  • Experimental Scope: Five contact-rich tasks evaluate TouchGuide and TacUMI across challenging manipulation settings.The experiments address tactile benefits, prior-policy comparison, generalization, visual robustness, and data-collection efficiency.
  • Task Suite: The task suite includes long-horizon shoe lacing, fragile chip handover, cucumber peeling, vase wiping, and low-data lock opening.Lock Opening uses only 20 demonstrations and requires high-precision sequential manipulation.
  • Task Suite: Shoe Lacing requires threading a shoelace through four eyelets within the allotted time.The task tests sequential, precise manipulation over multiple contact points.
  • Task Suite: Chip Handover requires grasping, collaboratively handing over, and placing a potato chip without breaking it.Success requires completing the handover and placement within the allotted time.
  • Task Metrics: Cucumber Peeling and Vase Wiping use scores based on task progress relative to human performance.Cucumber Peeling quantizes its score into seven values from 0 to 1, while Vase Wiping uses a similar definition.

3) Baselines:

TouchGuide augments pretrained visuomotor policies with action-space tactile steering and is evaluated against established visuo-tactile baselines across five contact-rich tasks. The method improves performance, generalizes across policies and settings, and benefits from noise-pretraining.

  • Baselines: TouchGuide uses DP and π0.5 as base policies and compares against RDP, PolicyConsensus, SafeDiff, and Tactile Dynamics.The evaluation covers five challenging fine-grained, contact-rich tasks.
  • Touch-guided steering: TouchGuide steers noisy base-policy actions with a CPM that evaluates physical contact feasibility during inference.This action-space guidance targets initial-contact errors that can cause severe downstream failures.
  • Main results: 36.2% improves over RDP at 30.3% and PolicyConsensus at 24.7% on the reported comparison.TouchGuide also adds negligible inference-time overhead.
  • Generalization: TouchGuide generalizes across different robots, policies, and tactile modalities without retraining the base policy.The reported settings include DP and π0.5, multiple robots, and force or tactile-image sensing.
  • Noise pretraining: 39.17% →62.50% shows the task success-rate improvement from noise pretraining in the ablation.Noise is added using diffusion scheduling or flow-matching interpolation before CPM training and inference.

2) CPM Modality Ablation:

The CPM benefits from both visual and tactile inputs, while TouchGuide remains robust to visual shifts and occlusions. TacUMI supports data collection, but task-specific CPM training remains a cost limitation.

  • CPM modality ablation: Without vision, success falls from 62.50% to 43.33%, while without touch it falls to 43.50%.The ablation reflects complementary sparsity patterns in visual and tactile signals across manipulation phases.
  • Robustness: TouchGuide consistently improves the base policy across unseen objects, unseen scenes, combined shifts, and transient visual occlusion.The action distribution analysis also reports that steered actions align more closely with real actions than base actions.
  • TacUMI: TacUMI is designed to collect accurate, reliable tactile demonstrations for robot learning.The study compares data-collection usability using attempts, duration, valid rate, and satisfaction.
  • Limitation: TouchGuide avoids retraining the base policy, but its CPM still requires task-specific training that increases time and cost.Future work targets tactile representations and a CPM that generalizes across tasks.

APPENDIX

The appendix explains flow-matching contact steering, evaluates guidance hyperparameters, and cautions that steering settings depend on the task, policy, and action representation.

  • Flow-matching guidance: TouchGuide applies Equation 12 during inference to steer the flow-matching action expert in π0.5.The derivation connects conditional flow guidance with classifier-guided velocity prediction.
  • Flow construction: The conditional flow uses an optimal-transport path, with intermediate states formed by linear interpolation.The appendix derives the guided velocity from the flow-matching probability path and conditional score.
  • Guidance scale: η = 10 maximizes mean success rate when KTouchGuide = 0.3 on Chip Handover with π0.5.Smaller scales provide insufficient steering, whereas larger scales can overshoot and perturb the original sampling process.
  • Guidance steps: KTouchGuide = 0.3 maximizes mean success rate at η = 10, corresponding to the last three flow-matching steps.Too few steps provide insufficient steering, while too many introduce excessive CPM-input noise.
  • Practical guidance: TouchGuide remains effective across a wide range of guidance scales and steps, but hyperparameters should be tailored to each task, policy, and action representation.The appendix cautions against directly reusing large hyperparameter values.

C. Ablation Study on Observation Encoder

The encoder ablation favors frozen pretrained DINOv2 for both visual and tactile inputs, while task analyses connect baseline failures to missing in-hand state and force information.

  • Observation encoder ablation: 62.5% success with 118 M trainable parameters is achieved using frozen pretrained DINOv2 for both encoders.Two randomly initialized ResNet18 encoders achieve 49.0% in contrast.
  • Shoe Lacing: Shoe-lacing failures include wrong threading direction, incorrect grasp point, and shoelace slippage caused largely by missing in-hand state.Camera views make grasp angle and grasp location difficult to infer reliably.
  • Chip Handover: Chip-handover failures involve excessive force, wrong handover position, and incorrect insertion point.The first two positional failures relate mainly to missing in-hand state, while excessive force reflects absent tactile force feedback.
  • Baseline failure analysis: RDP’s compressed 15-dimensional tactile embedding discards in-hand information, while reactive baselines can respond too late to decisive initial-contact errors.TouchGuide instead steers toward contact-physics-consistent actions before those errors become failures.
  • Contact-sensitive tasks: TouchGuide improves contact-sensitive manipulation by providing in-hand state and force information, including during cucumber peeling and vase wiping.For cucumber peeling, contact-physics steering produces a more suitable initial peeling angle.

5) Lock Opening:

Lock Opening is sensitive to grasp precision, initial key contact, and full insertion before rotation. TouchGuide addresses these failure modes with tactile in-hand state information, while TacUMI improves the quality and consistency of collected demonstrations.

  • Lock Opening: Tactile feedback helps detect precise grasping and confirms when the key bottoms out before rotation.Wrist-camera observations alone make grasp precision and full insertion difficult to determine.
  • Lock Opening: Initial key contact largely determines success because misalignment can deflect or knock the key off-axis.The insertion angle must adapt to the grasp location, which varies across trials.
  • Lock Opening: TouchGuide built on π0.5 achieves reasonably good success with only 20 demonstrations and improves performance over its Diffusion Policy variant.Reported success rates rise from 35.3% to 49.9% for Force and from 36.2% to 58.0% for Tactile Image.
  • Data Collection: TacUMI combines direct tactile feedback, accurate tracking, a $720 base cost, and 540g collection-side weight.Rigid fingertip interfaces support direct tactile feedback, while Vive Tracker localization avoids several SLAM and headset-related limitations.
  • Data Collection: TacUMI produces smoother, more consistent trajectories than SLAM-based UMI, whose tracking failures and interpolation can obscure behavioral structure.Policies trained on TacUMI data achieve higher success rates than those trained on alternative-system data.

4) Hardware Selection Details:

The system combines compact hardware and inference-time TouchGuide integration with a feasibility score motivated by a probabilistic action-distribution interpretation. TouchGuide adds tactile steering to diffusion and flow-matching policies with limited implementation overhead.

  • Hardware: TacUMI’s hardware bill of materials covers collection-side and execution-side components and their quantities.The bill is accompanied by assembly illustrations in the hardware-design figure.
  • TouchGuide Integration: TouchGuide steering can be added to Diffusion Policy and π0.5 with four lines of inference-time pseudocode.The diffusion implementation adjusts predicted noise, while the π0.5 implementation adjusts the flow-matching velocity.
  • Performance: TouchGuide causes only a small increase in inference latency while substantially improving mean task success rate.The comparison includes π0.5, Diffusion Policy, and their TouchGuide variants under a client–server setup.
  • Feasibility Score: The feasibility score is motivated by modeling optimal and policy actions as Gaussian latent representations with isotropic covariance.The constant 1/σ factor can be absorbed into the guidance scale η.
  • Feasibility Score: The feasibility-score gradient guides policy actions toward the real-action distribution, encouraging contact-physics compliance.This interpretation connects the CPM score to inference-time steering rather than retraining the base policy.

H. Task-specific CPM Capability Analysis

The CPM assigns higher feasibility to correctly matched action–observation pairs, separates mismatched and unseen configurations, and remains sensitive to small neighboring action changes. Supplementary hard-setting experiments further evaluate TouchGuide’s robustness.

  • CPM Feasibility: In-distribution mismatched observations receive low feasibility scores, while correctly paired observations receive much higher scores.The evaluation pairs randomly selected actions with randomly sampled observations from the dataset.
  • CPM Feasibility: The CPM produces clearly separable feasibility scores for out-of-distribution actions and observations with unseen placements or orientations.This result is reported as evidence of task-specific contact-physics capture and some OOD generalization.
  • Temporal Analysis: Feasibility decreases smoothly away from a base frame because adjacent absolute joint-space actions typically change only slightly.The visualization expands a 10 Hz temporal window around selected frames.
  • Temporal Analysis: The CPM remains sensitive to neighboring observations even when action changes are small.The authors interpret this sensitivity as useful for fine-grained manipulation.
  • OOD Evaluation: TouchGuide achieves strong performance in both normal and hard Chip Handover settings and makes original baselines more robust to OOD scenarios.The hard setting uses test conditions distinct from the expert-demonstration training distribution.

K. Extended Task Descriptions

The five tasks require tactile control for grasping, contact alignment, force regulation, state estimation, or sustained manipulation. Success is measured by completion criteria, success rates, or normalized task scores depending on the task.

  • Shoe Lacing: Shoe Lacing requires adapting threading to varied initial lace poses and grasped positions.The task includes threading through an eyelet, continuously feeding the lace, and pulling it out with the other arm.
  • Chip Handover: Chip Handover requires preventing chip crushing and coordinating highly precise two-arm transfer and placement.The task succeeds only if the chip remains unbroken, is handed over once, and reaches the can.
  • Cucumber Peeling: Cucumber Peeling requires tactile sensing for secure grasping, blade insertion, slip detection, and smooth force-controlled peeling.The task terminates after a complete intact strip of cucumber skin is removed.
  • Vase Wiping: Vase Wiping requires correct eraser grasping and tactile regulation of contact force along a curved surface.Excessive force can misalign the eraser and prevent subsequent wiping.
  • Lock Opening and Metrics: Lock Opening requires precise grasping, adaptive insertion, and sensing full key insertion before rotation.The task uses success rate, while Cucumber Peeling and Vase Wiping use completion normalized by human-expert completion.
Loading 2601.20239v6…