PaperlayerView exact revision on arXiv

Source-linked AI summary

VT-MUSE: Multimodal Unified Sequential Visuotactile Representation Learning for Manipulation

Congsheng Xu, Qiaochu Yang, Fangyuan Shi, Yifan Han, Baijun Chen, Yiming Wang, Haonan Zhao, Daolin Ma, Xiaokang Yang, Hesheng Wang

arXiv:2608.21290v1cs.ROcs.CV

TL;DR

VT-MUSE addresses the limited modeling of fine-grained visual–tactile dependencies and contact evolution in existing manipulation representations. It jointly learns a sequential latent representation from masked visual and full tactile histories, then injects that memory into a Transformer policy through gated cross-attention. The framework improves performance across simulated tasks and in real-world experiments.

  • Problem

    Existing methods often encode vision and touch independently before fusion and focus on current observations, leaving reusable representations of synchronized histories and contact evolution less explored.

  • Method

    VT-MUSE uses two-stage cross-modal temporal representation learning, masked visual reconstruction, tactile-depth prediction, and gated cross-attention into a downstream policy.

  • Results

    VT-MUSE consistently improves manipulation performance over visual, visuotactile, and pretrained-representation baselines in simulation and on a physical robot.

  • Takeaways & Limitations

    The learned representation captures global visual states and fine-grained tactile contact dynamics while supporting downstream policies without task-specific end-to-end training.

  • Takeaways & Limitations

    The study uses a fixed perception sample stride and context window, and pretrains on at most seven tasks, leaving flexible retrieval and larger-scale scaling for future work.

Abstract

from arXiv · show

We propose VT-MUSE, a Multimodal Unified SEquential representation learning framework for visuotactilemanipulation. Existing approaches often encode visual and tactile observations independently before fusion, limiting their ability to capture fine-grained cross-modal dependencies. Moreover, most methods focus on observations at the current time step and overlook the temporal evolution of contact. VT-MUSE addresses both limitations through a two-stage representation learning framework. In Stage I, modality specific encoders are jointly adapted via cross-modal temporal alignment and masked-view consistency. In Stage II, a conditional variational latent model processes masked visual sequences together with full tactile histories. Auxiliary decoders reconstruct the masked recent visual observations and predict tactile depth changes, encouraging the latent representation to retain both global visual context and local contact dynamics. The learned representation is subsequently integrated into a lightweight Transformer policy through gated cross-attention. On the simulation benchmark, VT-MUSE outperforms the strongest baseline evaluated on all tasks by 11 percentage points and also achieves substantial improvements in real-world experiments.

I. INTRODUCTION

VT-MUSE targets gaps in cross-modal and temporal visuotactile representation learning by jointly modeling visual–tactile histories under partial observation. Its two-stage encoder and gated-memory policy improve manipulation performance in simulation and on a physical robot.

  • Motivation: Vision supplies scene-level and long-range context, while touch captures local contact states and remains informative during visual occlusion.The modalities are complementary in insertion, reorientation, and compliant manipulation.
  • Research gap: Existing methods often learn tactile features independently from vision and fuse separately learned features only during policy training.This may overlook fine-grained correspondence between visual changes and contact evolution.
  • Research gap: Existing temporal approaches commonly use short windows, one-step targets, or policy-coupled dynamics models, leaving reusable synchronized history representations less explored.The desired representation should remain informative with missing visual observations and transfer to a separately trained policy.
  • Approach: VT-MUSE jointly adapts visual and tactile encoders, then trains a conditional variational model on masked visual histories and complete tactile histories.The second stage reconstructs missing recent visual observations and predicts tactile depth changes to retain global context and local contact evolution.
  • Approach: The VT-MUSE Policy injects sequential latent memory into intermediate policy layers through gated cross-attention instead of concatenating it with policy inputs.This preserves the policy’s action-generation pathway while enabling retrieval of relevant multimodal history.
  • Results: VT-MUSE shows consistent improvements over visual, visuotactile, and pretrained-representation baselines in simulated and physical contact-rich manipulation tasks.The paper also reports systematic representation and temporal ablations.

II. RELATED WORK

Related work spans tactile representation learning, cross-modal vision–touch alignment, temporal prediction, and policy designs that separate global reasoning from local control. VT-MUSE combines sequential representation learning with a downstream Transformer policy using a frozen memory interface.

  • Tactile representation learning: Tactile learning has progressed from masked reconstruction and robotic-play pretraining to large-scale self-supervised and cross-sensor transfer methods.These approaches target transferable tactile features across sensors, objects, and tasks.
  • Cross-modal learning: Cross-modal methods learn visual–tactile correspondence through predictive, contrastive, attention-based, and alignment objectives.These methods provide complementary routes for structuring multimodal representations.
  • Temporal and action modeling: Recent temporal approaches predict future visual, tactile, or action trajectories, while tactile-aware VLA models use semantic tactile tokens and future tactile prediction.The related literature includes world action models and tactile-aware vision-language-action systems.
  • VT-MUSE: VT-MUSE trains a sequential encoder before downstream policy learning and supplies perceptual memory to a Transformer-based action-chunking policy.The encoder and policy are separated rather than trained as one task-specific end-to-end model.
  • VT-MUSE: Its two stages align visual and tactile observations, enforce temporal and masking consistency, reconstruct masked vision, and predict tactile geometry changes.The frozen representation is passed to the policy through gated cross-attention and can be learned from pooled multi-task data.

B. Problem Formulation

VT-MUSE encodes temporally subsampled visual–tactile histories with selected recent visual observations masked during representation learning. The resulting perceptual state supports a policy that combines historical multimodal memory with current sensory observations for action-chunk prediction.

  • Observation model: Each trajectory observation combines an external RGB image, optical tactile observations, robot proprioception, and a discrete task identity.The task identity is used during multi-task representation learning.
  • Temporal window: For each target time step, VT-MUSE constructs a temporally subsampled observation window of length L with sampling stride s.This window supplies the sequential context for representation learning.
  • Partial observation: Selected visual observations near the end of the window are masked, and the encoder infers a temporally grounded representation from incomplete visual and synchronized tactile histories.The task identity is also provided to the encoder.
  • Policy interface: The encoder’s representation is supplied alongside current visual, tactile, and proprioceptive observations to a policy that predicts an action chunk with horizon K.The encoder is a perceptual state encoder rather than an action predictor.
  • Sequential encoding: Stage II processes interleaved multimodal tokens with a temporal memory Transformer, retrieves context using recent tactile queries, and trains reconstruction and tactile-flow objectives.The overview describes this design as part of the full two-stage VT-MUSE pipeline.
  • Policy interface: VT-MUSE preserves two information sources: temporally aggregated historical visuotactile context and current sensory observations for immediate closed-loop prediction.The historical representation and current observations therefore serve complementary policy roles.

C. Model Structure

VT-MUSE interleaves visual and tactile tokens across time, masks recent visual inputs while retaining full tactile history, and aggregates tactile-conditioned temporal context into a latent representation. This representation is decoded for visual reconstruction and tactile geometry-change prediction, then injected into the policy through gated cross-attention.

  • Separate pretrained ViT encoders process visual and tactile observations augmented with modality, temporal-slot, and relative-time embeddings.
  • Recent visual tokens are replaced by learnable mask tokens, while the tactile sequence remains fully observable.
  • A temporal Transformer interleaves synchronized visual–tactile tokens, and final tactile tokens query the multimodal memory through cross-attention.
  • The latent representation reconstructs masked RGB observations and predicts dense tactile geometry changes to preserve scene context and contact dynamics.
  • The policy projects the representation into Transformer hidden space and supplies it as separate cross-attention memory rather than concatenating it with proprioceptive state.
  • A learnable scalar gate initially limits the auxiliary representation’s influence during policy training.

D. Training Objective

Stage I adapts visual and tactile encoders with objectives that align synchronized modalities, promote temporal coherence, and stabilize representations under visual masking. Only the final three Transformer blocks of each pretrained ViT encoder are updated.

  • Synchronous visual–tactile alignment treats tokens from the same temporal slot as positives and tokens from different batch samples as negatives.
  • A symmetric InfoNCE objective aligns tactile-to-visual and visual-to-tactile token representations.
  • Normalized multimodal states are formed by averaging synchronized visual and tactile tokens, then contrasted across adjacent sampled states.
  • Temporal contrastive learning keeps neighboring states from the same trajectory close in representation space.
  • Two independently sampled visual masks support cosine consistency training for representations of the same observation window.
  • Mask consistency encourages stability under different patterns of missing or corrupted visual evidence.
  • Only the final three Transformer blocks of each pretrained ViT encoder are updated during Stage I.

2) Stage II, Latent Modeling:

Stage II freezes the Stage I modality encoders and jointly optimizes temporal-memory, latent-modeling, and auxiliary reconstruction components. The training uses masked RGB and tactile depth-flow reconstruction, while downstream policy learning combines action prediction with latent regularization.

  • Stage II initializes and freezes the Stage I visual and tactile encoders before optimizing the temporal memory, latent modules, and reconstruction heads.
  • Masked RGB-tail reconstruction and tactile depth-flow reconstruction use MSE losses during Stage II.
  • During downstream policy training, the Stage II conditional-prior encoder is frozen and a four-layer Transformer policy uses two intermediate cross-attention layers.
  • The policy objective combines L1 action prediction loss with KL regularization of the action posterior toward a unit Gaussian.

IV. EXPERIMENTS

The experiments test downstream effectiveness, multimodal supervision, and temporal modeling across simulated and physical-robot manipulation tasks. The benchmark spans eight tasks covering complementary contact patterns and interaction settings.

  • The experiments ask whether sequential visuotactile pretraining improves manipulation over current-observation fusion and existing representation-learning baselines.
  • Ablations examine how visual reconstruction and tactile depth-flow prediction contribute to representation quality and downstream control.
  • Temporal experiments evaluate the effects of temporal contrastive learning and observation-window length on manipulation performance.
  • VT-MUSE is evaluated on four simulated UniVTAC tasks and four physical-robot tasks.
  • The simulation set includes grasping, key withdrawal, contact-based hole exploration, and precision HDMI insertion.
  • The simulation study uses 500 trajectories per task for representation learning and 50 benchmark demonstrations per task for downstream policy learning.
  • Physical experiments use a Flexiv Rizon 4s arm with an XenseGripper on tube insertion, board wiping, drawer withdrawal, and toaster pressing.

3) Baselines:

VT-MUSE is compared with vision-only, pretrained-representation, and tactile foundation-policy baselines in simulation and with current-observation ACT baselines on physical robots.

  • Simulation baselines: Simulation compares VT-MUSE with ACT, ACT+UniVTAC, VITaL Pretraining, and FTP-π0.5, which reports only three of four tasks.FTP-π0.5 is excluded from the four-task average because it lacks one task result.
  • Physical-robot baselines: Physical-robot experiments compare VT-MUSE with ACT using current external RGB alone or current RGB plus tactile observations.Both ACT baselines use the same 50 demonstrations per task as VT-MUSE and no pretrained representation.
  • Simulation results: 55.25% average SR: VT-MUSE exceeds ACT+UniVTAC’s 39.00% by 16.25 percentage points across four UniVTAC simulation tasks.On the three tasks reported by FTP-π0.5, VT-MUSE reaches 63.33% versus 51.33%, a 12.00-point improvement.
  • Simulation results: VT-MUSE achieves the highest simulation success rates on LB, IHo, and IHD, while VITaL Pretraining remains stronger on PoK.The gains are especially pronounced on IHo and IHD, which require sustained contact, geometry estimation, and fine-grained alignment.
  • Evaluation protocol: Simulation policies are evaluated on the same 100 initial conditions per task, while physical-robot methods are evaluated for 20 trials per task.The two representation-learning stages require approximately 25 hours on eight NVIDIA A800 GPUs; each task-specific policy trains for approximately 30 minutes on one A800 GPU.

2) Training and Evaluation Protocol:

The evaluation uses controlled simulation and physical-robot protocols, while results show strong physical performance and substantial sensitivity to the representation-learning objectives.

  • Physical-robot evaluation: 95.00% aggregate SR: VT-MUSE succeeds in 76 of 80 physical-robot trials.Each physical-robot method is evaluated for 20 trials per task under the same task-specific initialization and success criteria.
  • Multi-task pretraining: The four-to-seven-task expansion is examined through t-SNE projections whose points represent temporal observation windows and whose colors identify tasks.The figure provides a qualitative analysis of multi-task pretraining rather than a numerical evaluation.
  • Physical-robot results: 63.75 percentage points: VT-MUSE outperforms ACT (Visuotactile) and achieves at least 18/20 successes on every physical task.The improvement appears on both geometry-sensitive and sustained-contact tasks.
  • Objective ablations: 16.50 percentage points: removing tactile depth-flow prediction lowers average SR from 55.25% to 38.75%.All variants retain visual and tactile inputs and use the same policy architecture, demonstrations, and evaluation protocol.
  • Objective ablations: 18.25 percentage points: removing visual reconstruction lowers average SR to 37.00%, while removing Stage II entirely yields 27.25%.These results indicate that cross-modal alignment alone is insufficient under the evaluated setting.

1) Auxiliary Prediction Diagnostics:

Auxiliary-prediction diagnostics show that the visual and tactile objectives provide complementary supervision, while temporal contrastive learning also contributes to downstream control.

  • Auxiliary prediction quality: Without tactile prediction, RGB reconstruction is strongest, whereas the complete model has the lowest tactile depth-flow error.Joint training therefore preserves information from both modalities rather than maximizing every individual reconstruction metric.
  • Control utility: 38.75% average policy SR: the visual-only objective slightly improves pixel-level reconstruction but trails VT-MUSE’s 55.25%.Reconstruction fidelity alone is not a reliable measure of control utility in this comparison.
  • Qualitative predictions: Removing an auxiliary objective severely degrades its associated prediction head, whereas the complete model retains meaningful visual structure and tactile depth information.Figure 5 compares visual reconstruction and tactile depth-flow prediction against ground-truth observations under different objectives.
  • Temporal ablation: 13.00 percentage points: removing temporal contrastive learning reduces average SR from 55.25% to 42.25%.The temporal architecture remains unchanged, isolating temporal contrastive supervision rather than model capacity.
  • Temporal ablation: Temporal contrastive removal reduces performance by 20 points on LB and 23 points on PoK, but by only 5 and 4 points on IHo and IHD.The effect is task-dependent, and the loss contributes information not recovered by the Stage-II prediction objectives alone.

1) Observation-Window Length:

VT-MUSE selects a five-observation temporal window as the best-performing tested setting, while broader pretraining-task diversity preserves organized latent features. The authors note that fixed sampling stride and limited task scale remain open limitations.

  • Observation-Window Length: 55.25% average SR is achieved with L = 5, exceeding L = 4 by 15.25 percentage points and L = 6 by 14.00 points.The sampling stride is held fixed across settings, and performance does not improve monotonically with longer windows.
  • Observation-Window Length: Increasing the temporal window beyond five observations does not further improve performance, indicating a balance between useful context and redundant or outdated interaction states.The authors therefore use L = 5 in the reported setting.
  • Multi-Task Pretraining: Latent features remain organized without obvious visual collapse as the pretraining set expands from four to seven evaluated tasks.This is qualitative evidence that the encoder accommodates the evaluated increase in task diversity, not a quantitative measure of task recognition or policy improvement.
  • Limitations and Future Work: VT-MUSE currently uses a fixed sampling stride and perception context window, and pretraining is limited to at most seven tasks.Future work calls for flexible sampling and larger-scale validation of how representation quality changes with more tasks.
Loading 2608.21290v1…