PaperlayerView exact revision on arXiv

Source-linked AI summary

STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation

Xiangcheng Liu, Tianhao Wu, Le Zheng, Yidong Wang, Bowen Jiang, Mingjie Pan, Xinlin Ren, Yi Liu, Jianlan Luo

arXiv:2609.12549v2cs.RO

TL;DR

Dexterous manipulation remains difficult to learn at scale because real-world dexterous data is scarce and tactile signals are sparse. STAR pairs a real-world dataset with an integrated VTLA training recipe, and its policy achieves the highest reported success rate and task completion rate across the evaluated tasks.

  • Problem

    Scaling dexterous manipulation beyond simple skills remains difficult, and sparse tactile signals make effective tactile representations challenging to learn.

    Source: §I · pp. 1–2

  • Method

    STAR combines visual-tactile joint pre-training, sparse-global tactile tokens, and sparse future tactile prediction for VTLA models.

    Source: §I · p. 2

  • Results

    Ours achieves the highest SR and TCR across all tasks.

    Source: B. Comparison of Different… · p. 6

  • Takeaways & Limitations

    The policy demonstrates dexterous manipulation capabilities across four real-world tasks using 100 task-specific post-training trajectories per task.

    Source: §I · p. 2

    The dataset covers a limited set of tasks and remains small compared with parallel-gripper datasets.

    Source: §VI · p. 8

Abstract

from arXiv · show

Dexterous manipulation requires coordinated multi-finger control and effective tactile feedback, yet learning these capabilities remains challenging due to the lack of large-scale real-world data and the difficulty of extracting effective representations from sparse tactile signals. We build a robot platform and teleoperation system to collect a 200-hour bimanual dexterous manipulation dataset with synchronized visual, tactile, and language annotations, comprising 10,576 trajectories across 65 tasks, 69.5% of which involve dexterous multi-finger manipulation. We further propose STAR, an integrated training recipe for vision-tactile-language-action (VTLA) models that addresses the spatial, temporal, and informational sparsity of tactile signals through visual-tactile joint pre-training, sparse-global tactile token representation, and sparse future tactile prediction. Trained on this dataset, STAR achieves a 61% average success rate across four real-world tasks with 100 post-training trajectories per task, demonstrating dexterous performance under task-specific post-training.

I. INTRODUCTION

Dexterous manipulation remains difficult to scale beyond simple skills, while learning tactile representations is challenging because tactile signals are sparse. The paper introduces a real-world dexterous dataset and an integrated training recipe to address these gaps.

I. INTRODUCTION

  • Prior models generalize across objects and tasks but remain concentrated on simple skills such as pick-and-place [5]–[8].Earlier work demonstrated impressive dexterous behaviors on specific tasks, while scaling to general manipulation settings remains difficult –[4].Source: §I · p. 1
  • The dataset contains 200 hours of real-world manipulation data, with 69.5% of trajectories requiring dexterous multi-finger manipulation.The collection includes synchronized visual, tactile, and language annotations.Source: §I · p. 1
  • The training recipe combines visual-tactile joint pre-training, sparse-global tactile tokens, and sparse future tactile prediction to address tactile-signal sparsity.The authors identify spatial, temporal, and informational sparsity as the challenges motivating these components.Source: §I · p. 2

A. Data Source for Dexterous Hand

  • Human data scales readily but has embodiment gaps, while real-robot teleoperation data is difficult to collect and existing datasets emphasize simple skills.Human and wearable-device data lack robot embodiment and commanded actions with force information,; hardware robustness and teleoperation also constrain real-robot collection.Source: A. Data Source for Dexterous Hand · p. 2

B. Dexterous Manipulation

Current dexterous manipulation approaches remain limited on contact-rich tasks, and existing tactile methods often address only one sparsity challenge. The paper frames VTLA action generation as predicting future action chunks from multimodal observations.

B. Dexterous Manipulation

  • Current methods show high success mainly on pick-and-place and limited performance on dexterous tasks, while this VTLA model is fine-tuned on dexterous real-robot data [8],.Prior approaches may require task-specific designs or models, or rely on accurate human references that poorly capture dynamics.Source: B. Dexterous Manipulation · p. 2

C. Vision-Tactile-Language-Action Models

  • Existing tactile approaches address temporal or informational sparsity, while spatial sparsity is especially severe for dexterous hands; this work combines solutions across the training pipeline [13]–.Dexterous hands’ larger sensor areas create token redundancy compared with parallel grippers’ two pads.Source: C. Vision-Tactile-Language-Action Models · p. 2

III. PROBLEM STATEMENT

  • The model predicts a future action chunk conditioned on observations that include images, tactile signals, robot state, and language instruction.Actions and state include arm end-effector pose and hand joint positions; generation uses conditional flow matching from a Gaussian prior to the target action distribution.Source: §III · p. 3

IV. METHOD

The paper combines a newly collected dexterous-hand dataset with an integrated training recipe to address tactile-signal sparsity.

IV. METHOD

  • The method pairs a 200-hour real-world dexterous-hand dataset with a training recipe designed to address tactile-signal sparsity.The dataset addresses the lack of high-quality data with dexterous finger coordination and tactile modality.Source: §IV · p. 3

A. Real-World Dexterous Manipulation Dataset

The authors build a bimanual robot and teleoperation system for real-world dexterous data collection, then train policies using a recipe tailored to sparse tactile input.

A. Real-World Dexterous Manipulation Dataset

  • The collection platform uses 10-DoF dexterous hands with 268-dimensional tactile sensors on a bimanual wheeled mobile robot.A skeleton-based glove captures human motions for teleoperation.Source: A. Real-World Dexterous Manipulation… · p. 3
  • The real-world dataset contains 10k+ trajectories across 65 tasks; 69.5% require dexterous manipulation, 20% involve other primitive skills, and 10% comprise long-horizon trajectories composed of primitive skills.It includes images from left and right wrist cameras, robot states, command actions, and manually annotated task-level language instructions, synchronized to 30 Hz.Source: A. Real-World Dexterous Manipulation… · p. 3

B. Training Recipe

  • Tactile activity is sparse: the highest activated-taxel ratio across frames is 24%, and activated tactile frames account for 35%.The policy is fine-tuned on the dataset and then post-trained with task-specific data; its recipe includes joint RGB-tactile pre-training, sparse-global tokens, and sparse future tactile prediction.Source: B. Training Recipe · pp. 3–4, Figure 3 · p. 4

1) Visual-Tactile Joint Pre-training:

Visual-tactile joint pre-training pairs corresponding wrist images and tactile signals for masked-token prediction, discouraging trivial all-zero tactile predictions through visual context. Both modalities use 75% patch masking and patch-wise normalized-loss supervision.

1) Visual-Tactile Joint Pre-training:

  • The tactile encoder is pre-trained on the self-collected dataset because large-scale dexterous-manipulation tactile pre-training is unavailable.Tactile signals are represented as images, with the same encoder used for left- and right-hand pairs.Source: 1) Visual-Tactile Joint Pre-training · p. 4
  • Joint masked-token prediction concatenates image and tactile tokens from corresponding wrist-image and tactile-signal pairs.The visual context makes predicting all-zero tactile signals suboptimal, encouraging more informative tactile representations aligned with vision.Source: 1) Visual-Tactile Joint Pre-training · p. 4
  • Both RGB and tactile images mask 75% of patches, with patch-wise normalized loss used for supervision.The target normalization uses each patch’s pixel mean and standard deviation.Source: 1) Visual-Tactile Joint Pre-training · p. 4

2) Sparse-Global Tactile Token Representation:

The tactile representation filters inactive spatial tokens while retaining a per-hand global token, and sparse future tactile prediction supplies supervision across the action horizon. These choices target spatial and informational sparsity, respectively.

2) Sparse-Global Tactile Token Representation:

  • A contact gate retains activated spatial patch tokens and masks inactive ones, while always preserving a per-hand global tactile token.The gate uses calibrated tactile readings and can also use contact or change maps from sensors such as GelSight.Source: 2) Sparse-Global Tactile Token… · p. 5
  • The global token summarizes the tactile field and is prevented from attending to visual, language, and action tokens, while those tokens may attend to it.This asymmetric attention keeps the global token tactile-specific while making tactile information accessible to other modalities.Source: 2) Sparse-Global Tactile Token… · p. 5

3) Sparse Future Tactile Prediction:

  • Sparse future tactile prediction distributes five predicted frames across the action horizon at timesteps 10, 20, 30, 40, and 50.The auxiliary task is used during policy finetuning and post-training to capture longer-horizon contact dynamics.Source: 3) Sparse Future Tactile Prediction · p. 5
  • Predicting tactile observations across multiple future horizons is intended to encode future contact dynamics and alleviate informational sparsity.The paper motivates this task by noting that historical tactile observations provide limited information about the current contact state.Source: 3) Sparse Future Tactile Prediction · p. 5

V. EXPERIMENTS

The experiments examine whether dexterous manipulation data and the proposed training recipe improve performance, and how tactile input affects it. Evaluation covers four real-world tasks with task-specific post-training and held-out initial configurations.

V. EXPERIMENTS

  • The evaluation asks whether dexterous manipulation data improves performance, whether the training recipe addresses tactile sparsity, and how tactile input affects performance.These are the three questions framing the real-world experiments.Source: §V · p. 5

A. Experiment Setup

  • The four tasks are earbud flipping, stacked-book retrieval, postcard retrieval, and multi-object grasping.They require distinct manipulation sequences, including finger-driven rotation, hooking and grasping, sliding and placement, and sequential collection and disposal.Source: A. Experiment Setup · p. 5
  • The comparison includes GR00T N1.7, LDA-1B, and π0.5, with the same task-specific trajectory count, post-training steps, and evaluation protocol across methods.The policies are initialized from officially released pretrained weights; LDA-1B and π0.5 have their action layers reinitialized to match the action dimension.Source: A. Experiment Setup · p. 5
  • Policies are evaluated over 20 trials per task on held-out initial configurations, using the same randomly selected scenes across methods.Evaluation-task objects are held out from the pre-training dataset, and each method receives 100 task-specific post-training trajectories per task.Source: A. Experiment Setup · pp. 5–6

B. Comparison of Different Manipulation Policies

Across the baseline comparison, Ours achieves the highest success rate and task completion rate across all tasks. Ablations indicate that the recipe’s components contribute under different tactile demands, while sparse future prediction’s benefit depends on how prediction targets are distributed over time.

C. Importance of Training Recipe

  • Removing any training-recipe component degrades performance across ablations on earbud flipping and stacked-book retrieval, which have complementary tactile requirements.All variants use the same two-stage protocol and 100 task-specific post-training trajectories per task.Source: C. Importance of Training Recipe · p. 6, Table II · p. 7

B. Comparison of Different Manipulation Policies

C. Importance of Training Recipe

  • STTR is more beneficial for stacked-book retrieval, while GTTR matters more for tactile-intensive earbud flipping, despite similar overall SR and TCR.STTR reduces tactile-token redundancy; GTTR aggregates global tactile information.Source: C. Importance of Training Recipe · pp. 6–7
  • With five prediction targets, replacing SFTP’s 1.67-second span with DFTP’s 0.17-second span decreases performance on both tasks.The comparison couples sampling interval with maximum temporal extent, so it cannot isolate either factor’s contribution.Source: C. Importance of Training Recipe · p. 7

D. Role of Tactile

Tactile masking degrades performance on both evaluated tasks, although sensor coverage limits tactile’s benefits in some settings. The paper identifies dataset and method scope boundaries for further work.

D. Role of Tactile

  • Masking tactile inputs during inference degrades performance on both evaluated tasks, indicating that the policy uses tactile information for action generation.Without tactile input, the policy’s overall SR and TCR are still higher than π0.5’s, demonstrating the dataset’s effectiveness.Source: D. Role of Tactile · p. 7, Table III · p. 7
  • Tactile’s benefit is task-dependent: fingertip sensing helps earbud flipping, while uncovered finger edges constrain multi-object grasping.For multi-object grasping, edge-contact deviations contribute to a gap between TCR and SR.Source: D. Role of Tactile · p. 7, Figure 6 · p. 8

VI. LIMITATIONS

  • The dataset covers a limited task set and remains small relative to parallel-gripper datasets; language alignment, other hands, and robustness to sensor noise, drift, or missing taxels remain open directions.The conclusion reports dexterous performance across four real-world tasks with 100 post-training trajectories per task.Source: §VI · p. 8, §VII · p. 8

APPENDIX A. Real Robot and Teleoperation System

The appendix describes a bimanual dexterous-hand robot and its teleoperation setup, then details visual-tactile pre-training and tactile representations used in policy training. Sparse future tactile prediction supervises force at selected points across the action horizon.

1) Robot System Setup:

  • The bimanual wheeled robot has two dexterous hands, three cameras, 10 actuated hand DoF per hand, and 16 total hand DoF per hand.The cameras comprise two wrist views at 640×528 RGB resolution and one head-mounted view at 1280×800.Source: 1) Robot System Setup · p. 9

B. Teleoperation System Setup

  • Teleoperation uses skeleton gloves for hand control and trackers for the arms; retargeting adds fingertip-orientation constraints and disables automatic finger closing.The orientation constraint jointly optimizes fingertip positions and orientations for more intuitive control.Source: B. Teleoperation System Setup · p. 9, Figure 7 · p. 9

1) Visual-Tactile Joint Pre-training:

  • Visual-tactile pre-training standardizes both modalities, independently masks 75% of their tokens, and reconstructs them through modality-specific encoders and a shared fusion encoder.Inputs are standardized to 224×224 and divided into 196 tokens per modality.Source: 1) Visual-Tactile Joint Pre-training · p. 9

2) Sparse-Global Tactile Token Representation:

  • The tactile representation keeps active patch tokens and a per-hand global summary attendable, while masking inactive patches and limiting the global token’s attention to other modalities.The global token aggregates the full tactile field; patch tokens retain localized contact information.Source: 2) Sparse-Global Tactile Token… · p. 9

3) Sparse Future Tactile Prediction:

  • SFTP predicts five future normal-force tactile frames at steps 10, 20, 30, 40, and 50, while DFTP predicts five adjacent frames at steps 1–5.Both variants use the normal-force channel, divided into non-overlapping patches, as their prediction target.Source: 3) Sparse Future Tactile Prediction · p. 9

D. Training Details

Training uses a two-stage pipeline, with policy pre-training on the full dataset followed by task-specific post-training. The appendix also specifies the evaluated subtasks and baseline-model adaptations.

D. Training Details

  • Tactile pre-training uses nonzero tactile images and corresponding RGB images, while policy training initializes the VLM and action expert from π0.5 and uses a 50-token action horizon.Pre-training runs for 15,000 iterations at batch size 1024; policy inputs comprise 1,562 tokens.Source: D. Training Details · p. 10
  • The two-stage pipeline trains for six epochs on 200 hours of multitask data, then for 100 epochs on task-specific demonstrations.The action projections are reinitialized for a 64-dimensional space, of which 34 dimensions are active.Source: D. Training Details · p. 10

E. Sub-task Description

  • The evaluation tasks comprise earbud flipping, stacked-book retrieval, postcard retrieval, and multi-object grasping, with the latter scored across eight subtasks.Task completion rate is the percentage of completed subtasks; it can differ from all-or-nothing task success.Source: E. Sub-task Description · p. 10

F. Implementation Details of Baselines

  • For baseline comparison, LDA-1B and GR00T N1.7 receive the same three-view visual input and aligned state and action representations as Ours.Both use a 50-step action horizon; other hyperparameters follow their respective default settings.Source: F. Implementation Details of Baselines · p. 10
Loading 2609.12549v2…