PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterous Tactile World Model

Ziyao Zeng, Xiatao Sun, Hao Wang, Yueyang Pan, Zhengxiang Yu, Fengyu Yang, Tianyu Liu, Zhiwen Fan, Daniel Rakita

arXiv:2609.34286v1cs.CVcs.LGcs.RO

TL;DR

Contact events that shape manipulation are difficult to observe visually, motivating a world model that combines egocentric video with touch from both hands. DTWM aligns each hand’s tactile signal with its video location, and improves future-frame prediction over a matched vision-only model and other visual-tactile models.

  • Problem

    Contact events that determine how manipulation unfolds are difficult to observe in video because hands can occlude contact and firm pressing can look like touching.

    Source: §1 · p. 1

  • Method

    DTWM predicts future egocentric manipulation frames from observed video and both hands’ tactile signals, injecting touch at each hand’s image location with a causally masked residual.

    Source: §1 · p. 2

  • Results

    DTWM reduces hand-motion underestimation from 23% to 9% and hand-region perceptual error by 7.4% versus a matched vision-only model, and outperforms other visual-tactile models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Full tactile signals provide useful force magnitude and location information beyond binary contact, and training with touch improves future-frame prediction even without touch at inference.

    Source: §1 · p. 2, §5 · p. 9

    The paired dataset is about twenty hours, hand locations come from a reconstruction that misses some frames, and the human-manipulation model cannot yet be placed directly inside a robot planner or policy.

    Source: §5 · p. 9

Abstract

from arXiv · show

World models for manipulation are typically trained from video, yet the events that determine how manipulation unfolds, such as making and releasing contact, are difficult to observe visually and are often easier to sense through touch. We present the Dexterous Tactile World Model (DTWM), a video world model for future-frame prediction of egocentric manipulation from both observed video and tactile signals from a glove worn on each hand. We condition a pretrained video diffusion transformer on each hand's tactile signal through a zero-initialized residual at the corresponding hand location in the video tokens, while a causal mask prevents predicted frames from accessing future information. Compared with a vision-only model matched in architecture, parameters, and training, DTWM reduces the underestimation of hand motion from 23% to 9%, while reducing the perceptual error in the hand region by 7.4% across three training runs per model. The benefit also increases over the prediction horizon, with the improvement in the later predicted chunks being about 4.1x larger than in the first. DTWM also outperforms other visual-tactile world models under the same setting, and training with touch improves future-frame prediction even when no touch is available at inference. Ablations show that the model benefits from both the magnitude and spatial location of force: replacing the tactile signal with binary contact states, either per hand or per location, increases prediction error. The observed course of the force indicates whether the interaction will persist or change.

1 INTRODUCTION

Visual ambiguity makes contact difficult to infer even though it determines how manipulation unfolds. DTWM predicts egocentric human manipulation from video and both hands’ tactile signals, adding touch at each hand’s image location.

1 INTRODUCTION

  • World models can synthesize data, initialize models of physical interaction, or serve as environments for training and evaluating robot policies.These uses motivate accurate prediction at contact, where grasp stability, slipping, or a finger reaching a surface determines what happens next.Source: §1 · p. 1
  • Unlike prior visual-tactile models evaluated in robot settings, DTWM predicts human hand motion from past observations alone and aligns touch with the hand’s image location.Prior models represent touch as global features, extra tokens, or a separate tactile image, creating an indexing mismatch between skin patches and image locations.Source: §1 · p. 2
  • DTWM adds each hand’s tactile reading to video tokens at that hand’s location through a zero-initialized residual, while a causal mask blocks future information.The tactile pathway is learned from observed signals; predicted frames receive no future hand pose, action, or tactile reading.Source: §1 · p. 2
  • Hand motion is underestimated by 23% with the matched vision-only model and 9% with DTWM; DTWM also reduces hand-region perceptual error by 7.4%.The hand-region reduction is averaged across three training runs per model; the advantage over the vision-only baseline is about 4.1x larger in later predicted chunks than in the first.Source: §1 · p. 2
  • DTWM outperforms three visual-tactile world models in the same setting, with perceptual error lower by 1.8% to 14.7%.Figure 1 also compares whole-frame and hand-crop visual quality, hand location, and motion.Source: §1 · p. 2, Figure 1 · p. 2
  • Replacing full tactile input with binary contact states per hand or location increases prediction error, indicating that force magnitude and location matter beyond contact alone.In the corpus, changing force is associated with release: a rising or decaying force makes release within the next half second 1.8 to 2.5 times as likely as steady force, beyond the current force level.Source: §1 · pp. 2–3

2 RELATED WORK

Prior tactile work spans sensing, recognition, grasping, and robot world models. DTWM instead uses whole-hand touch from both hands to predict future frames of egocentric human manipulation.

2 RELATED WORK

  • Prior tactile work uses touch for recognition, grasping, or representation, whereas DTWM feeds whole-hand touch from both hands to a video world model to predict future frames.Wearable gloves capture everyday tasks, and recent corpora pair egocentric bimanual video with synchronized pressure measurements.Source: §2 · p. 3
  • Existing visual-tactile world models use robot data and represent touch as extra tokens, global tactile features, or separate tactile images, without aligning it to the hand’s image location.TouchWorld uses the same corpus before adapting to robot hands; the other described models use fingertip or gripper sensors and support planning, data generation, policy evaluation, or subgoals.Source: §2 · p. 3
  • DTWM observes RGB frames, hand poses, and both hands’ tactile signals, places each tactile embedding at its projected hand location, and predicts future RGB frames.The pipeline samples one reading per frame from both gloves, comprising 255 pressure taxels and 30 finger-flexion channels in total.Source: Figure 2 · p. 4

3 METHOD

DTWM predicts future frames in chunks using only information available before the predicted interval. It aligns each observed hand’s tactile signal to that hand’s location in the video-token grid.

3.1 PROBLEM SETTING

  • The model predicts latent video in four-frame chunks, with each token attending within its chunk and causally to earlier chunks; observed latents remain clean while predicted latents are denoised.At deployment, it observes 12 pixel frames and predicts three chunks, or 36 frames, without access to future hand poses or actions.Source: §3.1 · p. 4

3.2 TACTILE CONDITIONING AT THE HANDS

  • Each hand’s tactile signal is embedded and spread over nearby video tokens at its projected hand location, bridging skin-indexed sensor channels and the image-indexed token grid.The hand location comes from a rendered skeleton; the hand-shaped tactile raster has 217 on-hand cells before duplicate channels are removed.Source: §3.2 · pp. 4–5
  • Compared with the vision-only baseline, DTWM preserves hand shape and motion; compared with other visual-tactile models, it better preserves hand shape and tracks ground-truth hand positions.The comparison figures show pressure in warm colors, finger flexion as blue bars, and ground-truth hand regions with dashed boxes.Source: Figure 3 · p. 5, Figure 4 · p. 6
  • A validity-gated, normalized Gaussian footprint distributes the tactile embedding across the token grid, and a zero-initialized residual injects it at selected transformer blocks.The residual pathway is disabled for predicted latents, preserving the causal boundary; an undetected hand contributes nothing.Source: §3.2 · p. 5

4 EXPERIMENTS

Experiments compare DTWM with matched vision-only and adapted visual-tactile models, then test which tactile information supports prediction and how benefits vary across visual quality, hand motion, training runs, and prediction horizons.

4.1 EXPERIMENTAL SETUP

  • Evaluation predicts 36 future frames from 13 observed frames, using 3,000 held-out clips split into 2,728 ID and 272 OOD clips.The vision-only baseline matches DTWM in architecture, parameter count, and training recipe, with tactile readings zeroed.Source: §4.1 · p. 6

4.2 COMPARISON WITH THE VISION-ONLY BASELINE AND OTHER VISUAL-TACTILE WORLD MODELS

  • 2.0% on ID and 2.2% on OOD clips: DTWM lowers full-frame LPIPS versus the matched vision-only baseline and achieves the lowest LPIPS and LPIPShand on both splits.DTWM also has the lowest perceptual error among the four visual-tactile conditioning designs under the shared signal and training setting.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 WHAT INFORMATION IN TOUCH MATTERS?

  • Removing pressure magnitude raises perceptual error, while a per-hand contact bit that also removes contact location and finger flexion performs worse than vision-only.The observed force course is informative: rising or decaying force is associated with 1.8 to 2.5 times the likelihood of release within the next chunk, beyond current force level.Source: §4.3 · p. 8, Figure 6 · p. 8
  • With the tactile pathway disabled at inference, the trained full-reading model still has lower full-frame LPIPS than vision-only on both ID and OOD clips.Disabling touch worsens all four perceptual scores relative to using the full reading.Source: §4.3 · p. 8

4.4 VISUAL QUALITY AND HAND MOTION

  • Against the matched vision-only baseline, DTWM reduces hand-motion optical-flow magnitude underestimation from 23% to 9%, while the hand silhouette area ratio moves closer to one.In examples, DTWM better preserves hand details, although both models remain less sharp than ground truth. PSNR and SSIM favor the vision-only baseline, but controlled blur analysis shows these metrics can improve as detail is lost while LPIPS worsens.Source: §4.4 · pp. 8–9

4.5 VARIATION ACROSS RUNS AND PREDICTION HORIZONS

  • DTWM has lower mean LPIPS across three runs per model and wins 7 of 9 run comparisons on both held-out and training episodes.The held-out per-frame improvement after the first chunk is about 4.1 times that within the first chunk.Source: §4.5 · p. 9, Figure 7 · p. 9

5 CONCLUSION

DTWM predicts egocentric human manipulation from video and both hands’ tactile signals, improving hand-motion and appearance prediction over visual-tactile and vision-only baselines. Its scope remains human manipulation, with a small dataset and imperfect hand-location reconstruction.

5 CONCLUSION

  • DTWM predicts egocentric human manipulation from observed video and both hands’ tactile signals, adding touch through a zero-initialized, causally masked residual at each hand’s location.The model predicts how the manipulation unfolds next.Source: §5 · p. 9
  • Against the vision-only baseline and existing visual-tactile world models under the same setting, DTWM better predicts hand movement and appearance, with its vision-only advantage growing over the prediction horizon.Training with touch also improves future-frame prediction when touch is unavailable at inference.Source: §5 · p. 9
  • A small, roughly twenty-hour corpus, missed hand locations in some frames, and human-only predictions limit the current model’s data scale and direct use in robot planners or policies.The authors identify robot embodiment as a natural next step.Source: §5 · p. 9

ETHICS STATEMENT

The study uses existing recordings of people wearing instrumented gloves during everyday tasks. It limits use and release of the recordings while noting the method’s potential dual uses.

ETHICS STATEMENT

  • The recordings show private interiors, public spaces, personal belongings, and sometimes other people, while gloves record wearers’ hands throughout.No new recordings were collected, and the corpus was used for research purposes only.Source: ETHICS STATEMENT · p. 10
  • The authors do not redistribute raw video; their planned release covers code, configurations, split lists, and per-clip scores, subject to the corpus license.The corpus authors state that data, code, and benchmark will be released publicly.Source: ETHICS STATEMENT · p. 10
  • The authors identify potential dual uses in robot learning, teleoperation, and assistive devices, but report no misuse specific to body-worn-sensor conditioning beyond existing video-generation and robot-learning concerns.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

An LLM-based coding assistant contributed to writing, literature search, scripts, analyses, figures, and several ablation models. The authors reviewed the assisted work and take responsibility for it.

AI USE STATEMENT

  • The assistant helped draft and revise paper text, search literature on recent visuotactile world models, write and execute scripts, analyze statistics, create figures, and implement and launch ablations.Source: AI USE STATEMENT · p. 10
  • No data were synthetic or model-generated, and no mathematical proof in the paper was produced by an LLM.Source: AI USE STATEMENT · p. 10
  • The authors reviewed all assisted work, checked reported scores against per-clip files, and take responsibility for the paper’s text, claims, and artifacts.They also checked bibliography entries against Crossref or arXiv records and prior-work statements against original papers.Source: AI USE STATEMENT · p. 10

A DATASET AND PREPROCESSING

The pipeline uses EgoTouch’s synchronized bimanual video and glove readings, constructs clips and session-aware splits, and conditions a pretrained video transformer on tactile signals aligned to hand locations. Further details specify sensor handling, training controls, temporal alignment, and hand-weighted loss.

A DATASET AND PREPROCESSING

  • EgoTouch provides everyday bimanual manipulation episodes with egocentric RGB video, per-hand pressure grids, and WiLoR joints used to render and locate the hand-conditioning stream.The corpus contains 1,933 episode directories from five scenes.Source: A DATASET AND PREPROCESSING · p. 16
  • Session holdout removes episodes recorded near evaluation sessions; the held-out-episode split covers 73 episodes, while task grouping identifies three out-of-distribution tasks among 3,000 clips.The held-out set is disjoint in episodes and sessions, but many held-out tasks have close training counterparts.Source: A DATASET AND PREPROCESSING · p. 16, Table 3 · p. 17

B TACTILE SENSOR LAYOUT

  • The hand-shaped 21×21 pressure raster is not the sensor topology: only 217 cells are defined, duplicate cells map to shared channels, and each hand has 15 flexion channels alongside pressure taxels.Undefined cells are zeroed and masked; readings are standardized using training-frame statistics.Source: A DATASET AND PREPROCESSING · p. 16, B TACTILE SENSOR LAYOUT · p. 17
  • Hand regions are projected from skeleton joints and treated as invalid when the hand is undetected or its projected point falls outside the image.The resulting region array was computed for all 1,926 episodes.Source: B TACTILE SENSOR LAYOUT · pp. 17–18

C MODEL AND TRAINING DETAILS

  • DTWM finetunes a pretrained Wan2.2-Fun-5B video transformer, extending its patch embedding for the hand-skeleton stream while preserving the pretrained function at initialization.The model processes 49-frame clips as 2,496 video tokens.Source: C MODEL AND TRAINING DETAILS · p. 18
  • The tactile model and capacity-matched control train the same added modules, but the control receives zeroed pressure input and therefore no sensor information.The transformer, autoencoder, and text encoder are frozen; adaptation uses LoRA and densely trained modules.Source: C MODEL AND TRAINING DETAILS · p. 18

C.1 FURTHER DETAILS

  • Zero-initializing the tactile projection keeps its residual zero initially while allowing projection gradients from the first training step.A prior gated, randomly initialized projection received no gradient at initialization.Source: §C.1 · p. 19
  • The loss weights predicted frames by temporal chunk and increases spatial weight around predicted hand locations, using those locations only as supervision.Observed latents receive zero weight, the next chunk weight 1, later latents 0.2, and the hand-centered multiplier is 1 + 5.0H.Source: §C.1 · p. 19

C.2 TRAINING SCHEDULES AND VARIANTS

The section specifies training variants and controlled implementations for comparing tactile inputs and visual-tactile world models. The comparisons preserve the shared training setup while varying how tactile readings enter the predictor.

C.2 TRAINING SCHEDULES AND VARIANTS

  • The three-run comparison trains the full-reading model and control three times each using the Table 4 recipe.Table 7 reports the individual-run results.Source: §C.2 · p. 19
  • The tactile ablations replace values in 21 × 21 grids: per-taxel thresholds remove force magnitude, while a hand-level contact bit marks whether more than 2% of pressure taxels exceed 0.05.Each variant is trained once and evaluated on the session-held-out set.Source: §C.2 · p. 20

C.3 OTHER VISUAL-TACTILE WORLD MODELS

  • Each existing visual-tactile model receives the same glove readings and DTWM training setup, so the comparison isolates how tactile signals enter the predictor.The original systems differ from DTWM in predictor, embodiment, and task, so their published scores are not directly comparable.Source: §C.3 · p. 20, Table 5 · p. 20
  • TouchWorld presents tactile grids as fixed-position grayscale panels, whereas VT-WM appends per-hand, per-frame tactile tokens to video tokens.The VT-WM design uses 26 tactile tokens per clip, masks future readings, and discards tactile tokens at output.Source: §C.3 · pp. 20–21
  • FeelWorld spreads each hand’s tactile feature uniformly over the token grid and gates it at five injection blocks, omitting spatial placement.The learned gate adds 15,365 parameters.Source: §C.3 · p. 21

D EVALUATION PROTOCOL

The evaluation predicts 36 future frames from 13 observed frames, with causal chunk attention and no tactile or skeleton input after observation. It pairs frame-based perceptual scoring with direct hand-behavior measures and reports a sharpness comparison.

D EVALUATION PROTOCOL

  • Evaluation observes 13 frames spanning 0.43 seconds, then predicts 36 frames in one pass with block-causal attention and no future skeleton or tactile inputs.The 49-frame clips are divided at K = 12; sampling uses 30 flow-matching steps.Source: D EVALUATION PROTOCOL · p. 21
  • LPIPS scores full frames and a skeleton-centered crop that usually includes the grasped object, while PSNR and SSIM score full RGB frames.The enlarged LPIPShand crop measures the manipulation region rather than hands alone and is undefined for clips without a usable skeleton.Source: D EVALUATION PROTOCOL · p. 21
  • Clip-paired score differences are bootstrapped 20,000 times for percentile intervals, which capture clip-sampling and evaluation noise but not independent-run variation.For sets larger than 400 clips, a normal approximation is used; run variation is reported separately.Source: D EVALUATION PROTOCOL · p. 21

D.1 HAND-BEHAVIOR METRICS

  • Hand behavior is measured from glove silhouettes and optical flow using overlap, centroid, area, trajectory, flow-direction, and flow-magnitude comparisons against ground truth.The crop is centered on ground-truth hand location, and scores cover predicted frames 13 to 48.Source: §D.1 · pp. 21–22

D.2 SHARPNESS

  • The full-reading model’s generated-frame sharpness ratio is 0.171, versus 0.146 for the capacity-matched control, a difference of +0.025 [+0.024, +0.026].Neither model exceeds ground-truth sharpness, so the ordering reflects reproduced detail rather than added texture.Source: §D.2 · p. 22

E COMPLETE RESULTS

On the held-out set, DTWM leads existing visual-tactile world models on all six Figure 1 metrics, while pixel metrics rank the vision-only baseline higher on PSNR and SSIM. The results also report hand behavior, prediction-window effects, and variation across runs.

E COMPLETE RESULTS

  • Across three independent runs, DTWM wins 7 of 9 run pairings on each split, and its worst run is below the baseline’s mean.The run-level table complements the clip-paired intervals, which do not capture independent-run variation.Source: Table 7 · p. 22, E COMPLETE RESULTS · p. 22, Table 6 · p. 22
  • The per-frame LPIPS improvement is smallest in the first predicted block and several times larger in later blocks.Table 8 reports cumulative windows and per-frame means for each block.Source: Table 8 · p. 23, E COMPLETE RESULTS · p. 23
  • DTWM leads every existing visual-tactile world model on all six metrics in the 3,000-clip held-out comparison.The metrics are those shown in Figure 1 and cover predicted frames 13 to 48.Source: Table 9 · p. 23, E COMPLETE RESULTS · p. 23
  • The hand-behavior table compares DTWM with the vision-only baseline over the full predicted window, and its caption reports silhouette-area and flow-magnitude ratios moving toward one with touch.The comparison covers the same 3,000 clips.Source: Table 10 · p. 23, E COMPLETE RESULTS · p. 23

E.1 PSNR AND SSIM

  • Across splits, the vision-only baseline has higher PSNR and SSIM, while DTWM has lower LPIPS; model ordering also differs between pixel metrics and LPIPS.These comparisons are reported in the added PSNR/SSIM tables.Source: §E.1 · p. 23, Table 12 · p. 24, Table 13 · p. 24
  • On 150 held-out clips, frames 13 to 48, Table 11 reports that blurring DTWM predictions improves PSNR and SSIM, while LPIPS penalizes blur.Source: Table 11 · p. 24

F QUALITATIVE RESULTS

Further in-distribution examples show the same qualitative pattern as Figure 3: hand appearance is more stable with the reading than without it.

F QUALITATIVE RESULTS

  • The examples are generated at the model’s 384 × 512 output resolution.This is the resolution produced by the model.Source: F QUALITATIVE RESULTS · p. 24
  • With the tactile reading, the hands stay in place; without it, they drift or dissolve.The further in-distribution examples show the behavior seen in Figure 3.Source: Figure 8 · p. 25
  • The further examples reproduce the same with-reading versus without-reading hand behavior described for Figure 3.With the reading, hands stay in place; without it, they drift or dissolve.Source: Figure 8 · p. 25

G CONTACT AND FORCE DYNAMICS OF THE CORPUS

Across 12,000 training clips, force and pressure-location changes were more common than contact-state changes, and observed force trends predicted release beyond current force level.

G CONTACT AND FORCE DYNAMICS OF THE CORPUS

  • Adding observed force trend raised release-prediction AUC from 0.723 to 0.735 for the first chunk and from 0.627 to 0.649 for the second.The first-chunk increase was +0.012, with a 95% interval of [+0.007, +0.018] over episodes.Source: G CONTACT AND FORCE DYNAMICS OF THE… · p. 26
  • Among hands in contact at frame 12, release in the first predicted chunk occurred in 7% of steady-force cases, 16% of decaying-force cases, and 12% of rising-force cases.Within each fifth of current-force levels, decaying or rising force was associated with release 1.2 to 4.7 times as often as steady force.Source: G CONTACT AND FORCE DYNAMICS OF THE… · p. 26
  • These statistics describe the corpus and do not establish which parts of the tactile reading the model uses.The evaluation layout uses 12,000 training clips, with observation ending at frame 12 and predicted chunks covering frames 13–28 and 29–44.Source: G CONTACT AND FORCE DYNAMICS OF THE… · pp. 25–26
  • In the first predicted chunk, force changed by more than 20% in 46% of cases and more than 50% in 22%, while contact state changed for one hand in 17% and both hands in 5%.Center of pressure moved by more than two cells in 48%; all rates were higher in the second chunk.Source: G CONTACT AND FORCE DYNAMICS OF THE… · p. 25, Table 14 · p. 27

Related papers

Loading 2609.34286v1…