PaperlayerView exact revision on arXiv

Source-linked AI summary

T-WLA: Bridging Slow Vision and Fast Touch

Kaichen Zhou, Yuzhen Chen, Hannah Yao, Chonghe Jiang, Rao Fu, Xihang Yu, Wei Dai, Bashar Kabbarah, Haoyu Chen, Ishita Goluguri, Zhiyang Dou, Haonan Chen, Fangneng Zhan, Mengyu Wang, Wojciech Matusik, Antonio Torralba, Paul Pu Liang

arXiv:2610.28814v1cs.RO

TL;DR

Contact conditions can change between visual updates, while rerunning visual planning adds latency and touch alone does not specify how to revise the task’s ongoing motion. T-WLA reuses cached visual memory to guide tactile residual corrections to unexecuted actions. It improves hand-motion prediction and raises mean real-robot task success from 17.0% to 50.0%.

  • Problem

    Contact conditions can change before a visual policy updates its plan, while rerunning visual planning adds latency and tactile feedback alone does not specify how to revise ongoing task motion.

    Source: §1 · p. 2

  • Method

    T-WLA caches history-aware visual memory and a base action chunk, then uses a lightweight tactile editor to predict residual corrections to unexecuted actions between visual updates.

    Source: §1 · p. 2

  • Results

    T-WLA improves finger-joint prediction accuracy on EgoTouch and Tachin and increases mean real-robot task success from 17.0% to 50.0%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support joint visual-tactile modeling and predictive supervision for contact-rich manipulation.

    Source: §5 · p. 9

    The asynchronous correction formulation assumes the base plan remains a useful reference between visual updates.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

In contact-rich manipulation, contact conditions can change before a visual policy updates its action plan. Recomputing the plan through the visual policy adds latency, while tactile feedback alone does not specify how the ongoing motion should be revised for the task. We introduce T-WLA, a world-language-action architecture that uses shared visual memory to guide both action generation and tactile correction. At each visual update, the visual branch integrates the current image, task instruction, and retrieved history into a visual memory that conditions base-action generation. The same memory is cached for a lightweight tactile editor, which combines it with the active action chunk and incoming touch to predict residual corrections to unexecuted actions without rerunning the visual branch. Future-image and future-pressure prediction provide training-only supervision, encouraging the visual memory and tactile features to capture interaction dynamics relevant to action generation and correction. We evaluate T-WLA on tactile-conditioned hand-motion prediction using EgoTouch and Tachin, and on four real-world dexterous manipulation tasks. T-WLA reduces finger-joint MPJPE by 30% relative to the strongest reported baseline and achieves an average success rate of approximately 50.0% across real-robot manipulation tasks. Videos and code are available at https://anonymous-submission-20.github.io/T-WLA.github.io/.

1 INTRODUCTION

T-WLA addresses the gap between slow visual planning and fast tactile feedback by reusing visual memory to guide asynchronous corrections. It evaluates this approach in hand-motion prediction and real-robot manipulation.

1 INTRODUCTION

  • Tactile feedback reveals local contact conditions that images alone can make difficult to infer, and supports rapid correction when those conditions change between visual updates.For example, vision guides plug alignment and approach while touch provides feedback on contact and insertion resistance.Source: §1 · p. 2
  • A shared history-aware visual memory guides base-action generation and lets a lightweight tactile editor correct unexecuted actions between visual updates without rerunning the visual branch.The editor combines cached memory and the active action chunk with incoming touch to predict residual corrections.Source: §1 · p. 2
  • Future-image prediction regularizes visual memory, while future-image and future-pressure prediction regularize tactile features; these training-only objectives complement action supervision.Their prediction heads are removed at inference.Source: §1 · p. 2
  • T-WLA improves finger-joint prediction accuracy on EgoTouch and Tachin and increases mean real-robot task success from 17.0% to 50.0%.Ablations examine tactile input, visual history, and predictive supervision.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses memory and action-chunk execution, and separately explores tactile representations and ways to incorporate touch into manipulation policies. T-WLA connects retained visual history with fast tactile correction of an active chunk.

2.1 MEMORY AND ACTION CHUNK EXECUTION

  • T-WLA retains visual history across policy updates and uses high-rate tactile feedback to revise the unexecuted portion of an active action chunk.This is presented as complementary to approaches that improve chunk selection, accelerate action synthesis, or account for inference latency.Source: §2.1 · p. 3

2.2 TACTILE FOUNDATION MODELS

  • T-WLA connects a slow visual policy to a fast tactile editor through cached, history-aware visual representation, with predictive supervision for evolving interaction dynamics.The auxiliary predictors are used only during training and add no inference cost.Source: §2.2 · p. 3

3 METHODOLOGY

T-WLA caches history-aware visual memory and a base action chunk at each visual update, then uses touch to revise remaining actions between updates. Training adds future-image and future-pressure prediction to action supervision.

3.1 OVERVIEW

  • Each visual update produces a base action chunk and visual memory; the tactile editor reads this cached pair to revise unexecuted actions between updates without rerunning the visual branch.The base chunk is sampled once per visual update, and each correction is recomputed relative to that cached chunk.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 HISTORY-AWARE VISUAL MEMORY

  • Learned queries retrieve recent visual-memory history, which the visual backbone combines with current image, instruction, field-of-view, and state tokens.The updated memory retains at most the K most recent memory sets and conditions the action expert.Source: §3.2 · p. 4

3.3 ASYNCHRONOUS TACTILE CORRECTION

  • Between visual updates, the visual memory and base chunk remain fixed while tactile updates run the encoder and residual editor to adjust valid, unexecuted actions.The formulation assumes the base plan remains a useful reference between visual updates.Source: §3.3 · p. 5
  • Recent tactile frames are encoded as a temporal context so contact evolution can be represented beyond what a single pressure map resolves.The tactile-history encoder uses reference subtraction.Source: §3.3 · p. 5, Figure 3 · p. 5
  • A small Transformer combines cached visual memory, the base chunk, and elapsed time; tactile context modulates its action features, and a residual head corrects only valid unexecuted actions.The residual is added to the original cached chunk rather than accumulated on previously corrected actions; missing tactile input yields zero correction.Source: §3.3 · p. 5

3.4 PREDICTIVE SUPERVISION AND TRAINING

  • Future-image and future-pressure prediction provide training-only supervision for visual and tactile control representations, and the prediction heads are removed at inference.Future-pressure prediction is conditioned on visual memory, tactile context, and upcoming edited actions; staged training first trains the visual branch, then freezes it while training the tactile editor.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate tactile-conditioned hand-motion prediction and closed-loop manipulation, then ablate tactile input, memory, and predictive supervision. Across these evaluations, T-WLA improves finger-motion prediction and real-robot success, with results also showing task-specific variation and the effects of removing components.

4 EXPERIMENTS

  • The experiments ask whether visual memory and high-rate touch improve future hand-motion prediction and closed-loop success, and which components account for improvements.They also test whether the visual branch remains a strong standalone predictor without tactile observations.Source: §4 · p. 6

4.1 EXPERIMENTAL SETUP

  • The evaluation covers five EgoTouch environments, the complete held-out Tachin set, and four contact-rich robot tasks.The robot tasks are egg handling, single-sheet pickup, paper-cup nesting, and paper-bag opening.Source: §4.1 · pp. 6–7

4.2 TACTILE- AND VISION-CONDITIONED ACTION PREDICTION

  • Approximately 30.0% lower wrist-relative MPJPE on EgoTouch and 27.0% lower on Tachin than each dataset’s strongest reported baseline.T-WLA has the lowest reported errors across all three metrics in every EgoTouch environment and on Tachin; the qualitative figure also shows the lowest MPJPE in both displayed examples.Source: §4.2 · p. 7, Figure 4 · p. 7
  • T-WLA reaches 119.1 Hz, compared with 69.0 Hz for the fastest reported baseline, alongside lower hand-motion prediction errors.The reported benefit spans all five EgoTouch environments and both datasets; wrist translation and rotation improve more modestly than finger articulation.Source: §4.2 · p. 8

4.3 REAL-ROBOT EVALUATION

  • 50.0% mean success with tactile input and 47.25% without it; SmolVLA reaches 16.75% without tactile input across four real-robot tasks.Each task uses 30 teleoperated demonstrations; Table 2 reports closed-loop success rates and their unweighted mean.Source: §4.3 · p. 8, Table 2 · p. 8
  • Tactile input’s success-rate benefit varies by task, with the largest gain in single-sheet pickup at 6 percentage points; cup nesting remains at 17%.The authors note that cup nesting also requires alignment and placement, and aggregate success does not identify the limiting stage.Source: §4.3 · p. 9

4.4 ABLATION STUDIES

  • Removing tactile input raises finger-joint error by 22.9% on EgoTouch and 8.7% on Tachin; removing recurrent memory particularly increases wrist prediction error on Tachin.Shortening memory from K = 8 to Kshort = 4 raises error by 2.3% on average across six metrics, while removing both world-prediction objectives raises it by 1.0%.Source: §4.4 · p. 9

5 CONCLUSION

T-WLA caches history-aware visual memory to guide base actions and higher-frequency tactile corrections, with future-image and future-pressure prediction used as training supervision. The conclusion reports that joint visual-tactile modeling and predictive supervision improve contact-rich manipulation; the related-work discussion positions this approach against policies that update vision and action together.

5 CONCLUSION

  • T-WLA reuses cached history-aware visual memory for high-frequency tactile corrections to ongoing actions, while future-image and future-pressure prediction complement action supervision during training.The conclusion states that joint visual-tactile modeling and predictive supervision improve contact-rich manipulation.Source: §5 · p. 9

A ACTION FOUNDATION MODEL

  • Prior world-action models use future-state prediction, but update visual observations, representations, and actions on a single clock; T-WLA adds a faster tactile controller that reuses cached visual representation.The passage identifies this as a challenge for contact-rich manipulation, where local contact conditions can change faster than the visual policy updates.Source: A ACTION FOUNDATION MODEL · p. 16

B MODULE IMPLEMENTATION

The appendix specifies how visual history, tactile context, and action features are represented and combined, then details the losses and predictive conditions used to train the controller.

B.1 VISUAL MEMORY RETRIEVAL

  • The visual-memory bank retrieves from learned reference slots and up to eight prior visual output-memory sets, masking unavailable history and resetting at episode boundaries.It uses eight learned query tokens of width 960; each visual event writes one new memory set.Source: §B.1 · pp. 16–17

B.2 TACTILE ENCODING

  • Tactile context encodes each hand’s recent pressure history and subtracts an encoded fixed reference history, with invalid representations zeroed.Actual and reference pressure windows use the same encoder; the reference is a raw pressure window rather than a cached embedding.Source: §B.2 · p. 17

B.3 RESIDUAL EDITING

  • The residual editor combines visual-memory and active-action features, then uses each hand’s tactile context to modulate action features before readout.The effective correction is masked by editable positions, hand component masks, and hand validity; each update is relative to the same base chunk.Source: §B.3 · pp. 17–18
  • The contact-filtered variant gates corrections using observed pressure history; gated-out hands retain their base action and are excluded from corresponding loss supervision.The gate does not require contact in the latest frame and is not learned from future contact labels.Source: §B.3 · p. 18

C.1 LOSS FUNCTION

  • The edited-action loss averages nonempty component groups within each supervised sample, then averages supervised samples; the residual penalty is a single mean over effective action entries.Contact filtering changes the valid sets in both expressions rather than adding artificial zero-loss examples.Source: §C.1 · pp. 18–19

C.2 PREDICTIVE CONDITIONS

  • Predictive training uses tactile context to condition future-image prediction and detached visual memory, tactile context, and upcoming edited actions to predict future pressure.The future-image predictor is frozen while gradients update the tactile adapter and encoder; the pressure target is measured future pressure.Source: §C.2 · p. 19

C.3 TRAINING STRATEGY

Training proceeds from tactile pretraining to editor training with the visual branch frozen; a joint fine-tuning procedure is proposed as an extension.

C.3 TRAINING STRATEGY

  • Tactile pretraining aligns synchronized EgoTouch tactile and hand-pose embeddings with a symmetric contrastive objective, treating matched pairs as positives and other minibatch pairs as negatives.The objective averages row-wise cross-entropy over the similarity matrix and its transpose.Source: §C.3 · p. 19
  • During editor training, the visual branch supplies cached visual memory and base-action chunks and remains frozen, while the tactile encoder, residual editor, conditioning adapter, and pressure predictor are updated.The visual predictor also remains frozen.Source: §C.3 · p. 19
  • The authors propose jointly fine-tuning both control branches on EgoTouch, recomputing visual memory and base actions online while keeping the visual predictor frozen.Under this proposed extension, corrected-action loss can update the visual branch through the memory interface.Source: §C.3 · p. 19

D.1 ACTION LAYOUT AND TEMPORAL ALIGNMENT

The hand-motion interface defines action-channel structure and temporal alignment for visual chunks, tactile edits, and future-pressure conditioning.

D.1 ACTION LAYOUT AND TEMPORAL ALIGNMENT

  • Each hand-motion action position has 192 channels, comprising 138 motion-data channels and 54 padding channels; padding is excluded from supervision and valid sampled action components.Joint channels store 21 three-dimensional local positions per hand.Source: §D.1 · p. 19, Table 4 · p. 20
  • Table 5 covers visual-only future hand-motion prediction on GigaHands, OakInk2, and TACO; lower prediction error is better, and frequency is reported but not ranked.The GigaHands average weights its five official activity families equally.Source: Table 5 · p. 20
  • The visual branch and tactile editor operate at 39.7 Hz and 119.1 Hz, respectively; tactile updates revise unexecuted actions in the active visual action chunk.Elapsed time is measured from the start of the chunk.Source: §D.1 · p. 20
  • Future-pressure conditioning uses the next eight edited actions from an elapsed-time-indexed position, zero-padding and marking positions beyond the chunk invalid.The indexing rule is distinct from the loss horizon and the mask for already executed actions.Source: §D.1 · p. 20

D.2 EVALUATION MASKS AND AGGREGATION

Evaluation uses declared valid prediction horizons and sequence-level aggregation; the contact-filtered protocol evaluates a contact-conditioned subset rather than the complete held-out set.

D.2 EVALUATION MASKS AND AGGREGATION

  • A hand is eligible only when its complete declared eight- or sixteen-position evaluation prefix is valid; truncated prefixes are not substituted.Predictions and targets are compared in the evaluator’s physical action coordinates.Source: §D.2 · p. 20
  • Wrist translation increments are accumulated across the prediction window before centimeter error is computed, while Euler rotations are composed using the xyz convention.Rotations are composed by left multiplication for both predictions and targets.Source: §D.2 · pp. 20–21
  • Finger-joint MPJPE averages palm-local positions for joints 1–20 in centimeters, excluding joint 0, the fixed local origin.The evaluator does not subtract predicted wrist position from already local joint coordinates.Source: §D.2 · p. 21
  • Errors are aggregated within each sequence before equally weighting environment means for EgoTouch or eligible sequence means for Tachin.This differs from pooling all frames into a sample-weighted mean.Source: §D.2 · p. 21
  • The contact-filtered protocol admits samples when at least one supervised hand passes the causal gate, then excludes noncontact hands from action and future-pressure supervision.These results apply to a contact-conditioned held-out subset; comparisons require a common sample set, hand mask, and prediction horizon.Source: §D.2 · p. 21

E VISUAL-ONLY ACTION PREDICTION

The visual-only experiment tests future hand-motion prediction without tactile observations or supervision, isolating the visual pathway rather than cross-modal transfer. The real-robot setup uses a dual-arm dexterous platform and teleoperated demonstrations.

E VISUAL-ONLY ACTION PREDICTION

  • The visual-only setting excludes touch during training and evaluation and uses RGB, language, pose history, recurrent visual memory, and the visual action expert.It isolates the visual pathway rather than testing cross-modal transfer.Source: E VISUAL-ONLY ACTION PREDICTION · p. 21
  • T-WLA achieves the best completed result on eight of nine dataset-level metric comparisons without touch, with consistent gains across all five GigaHands activity families.On GigaHands, it records 0.65 cm average translation error, 3.64° rotation error, and 0.43 cm joint error; relative to SmolVLA, reductions are 25.3%, 15.9%, and 54.7%.Source: E VISUAL-ONLY ACTION PREDICTION · p. 21
  • On OakInk2, T-WLA reduces SmolVLA’s translation, rotation, and joint errors by 12.0%, 32.0%, and 67.3%, respectively.On TACO, it has the lowest completed rotation and joint errors, improving over SmolVLA by 7.6% and 55.3%; its 0.75 cm translation error is slightly above SmolVLA’s 0.72 cm but 22.7% below VITRA.Source: E VISUAL-ONLY ACTION PREDICTION · p. 21

F REAL-ROBOT SETUP AND DEMONSTRATION COLLECTION

  • The real-robot platform is a dual-arm Linker A7black with a 7-DoF arm and Linker Hand L6 dexterous hand on each side, with fingertip tactile sensing and an egocentric RealSense D435i camera.Operators used data gloves and wrist trackers to map motion to the robot; 30 teleoperated episodes were collected for each of four tasks.Source: F REAL-ROBOT SETUP AND DEMONSTRATION… · p. 21, Figure 6 · p. 22

G ADDITIONAL QUALITATIVE RESULTS

Additional qualitative examples compare predicted hand poses with ground truth on EgoTouch and Tachin, and show real-robot examples of paper-bag dragging and single-sheet pickup. Wrist-aligned renderings isolate finger articulation, while camera-frame views preserve global wrist motion.

G ADDITIONAL QUALITATIVE RESULTS

  • Across the displayed EgoTouch and Tachin examples, T-WLA more closely matches ground-truth fingertip configurations and has the lowest per-example wrist-aligned MPJPE among compared methods.These per-example diagnostics are distinct from dataset-level test averages and complement the reported pattern of larger gains in finger-joint error than in wrist translation or rotation.Source: G ADDITIONAL QUALITATIVE RESULTS · p. 22

G.1 ADDITIONAL EGOTOUCH QUALITATIVE EXAMPLES

  • Six additional EgoTouch examples show camera-frame predictions followed by wrist-aligned renderings that isolate differences in finger articulation.The camera-frame comparisons include the source RGB observation, ground-truth pose, and predictions from VITRA, SmolVLA, MemoryVLA, T-Rex, and T-WLA.Source: §G.1 · p. 22, Figure 7 · p. 23, Figure 8 · p. 24

G.2 ADDITIONAL TACHIN QUALITATIVE EXAMPLES

  • Across six additional Tachin examples, T-WLA matches ground-truth fingertip configurations more closely and has the lowest per-example wrist-aligned MPJPE in every row.Camera-frame views preserve wrist motion and finger articulation; wrist-aligned views remove global displacement, and their values are per-example diagnostics rather than dataset-level averages.Source: §G.2 · pp. 24–25, Figure 9 · p. 25, Figure 10 · p. 26

G.3 ADDITIONAL REAL-ROBOT QUALITATIVE EXAMPLES

  • Additional robot examples show paper-bag dragging and single-sheet pickup, which require sustained grasp under changing fingertip loads or a secure pinch without crumpling the paper.The examples supplement the closed-loop success rates reported in Table 2.Source: §G.3 · pp. 26–27, Figure 11 · p. 27

Related papers

Loading 2610.28814v1…