PaperlayerView exact revision on arXiv

Source-linked AI summary

Spark-0: Scaling Spatially Aligned Tactile Pretraining for Robot Dexterous Manipulation

Tianxing Chen, Wenwei Lin, Chuqiao Lyu, Peicheng Xiang, Junyuan Tang, Tian Nian, Zijian Cai, Kehe Ye, Chenze Yu, Wenxuan Zhu, Eric J. Chen, Yan Qin, Yue Chen, Tongyu Wu, Junhao Gong, Kai-Chong Lei, Jiahao Zhang, Kailun Su, Qiwei Liang, Renjing Xu, Ping Luo, Wenbo Ding

arXiv:2610.22342v1cs.RO

TL;DR

Contact-rich dexterous manipulation is difficult because force, compliance, and incipient slip are weakly observable from images, while large-scale full-hand tactile data and benchmarks for both vision-only dexterity and tactile control are scarce. Spark-0 uses touch for visual-token supervision during pretraining and action conditioning, with HumanTouch data and SparkArena evaluation. It achieves the highest mean success rate on both SparkArena tracks and ranks first on all Ego Humanoid aggregate metrics.

  • Problem

    Contact-rich dexterous manipulation remains difficult because force, compliance, and incipient slip are weakly observable from images, and large-scale synchronized full-hand tactile data and benchmarks for visual dexterity plus tactile control are scarce.

    Source: §1 · p. 2

  • Method

    Spark-0 uses glove-derived contact maps and frozen 3D geometry features to supervise visual tokens during training, while a tactile pathway conditions the action expert; HumanTouch provides 1,000 hours of tactile human manipulation.

    Source: §1 · p. 2

  • Results

    Spark-0 achieves the highest mean success rate on both SparkArena tracks and ranks first on all eight Ego Humanoid aggregate metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results indicate that touch can serve as scalable supervision for dexterous visuomotor pretraining, not only as a policy input.

    Source: §7 · p. 10

Abstract

from arXiv · show

Generalist robot policies benefit from large-scale visual pretraining but often struggle with contact-rich dexterous manipulation, where force, compliance, and incipient slip are hard to observe from images. We present Spark-0, a spatially aligned vision-tactile-language-action (VTLA) model that uses touch in two complementary ways. During pretraining, image-space contact maps derived from tactile gloves, together with features from a frozen 3D geometry teacher, supervise the visual tokens; these objectives are removed at inference. In parallel, a tactile pathway encodes regional force histories and 3D taxel states to condition the action expert. A single checkpoint therefore supports both vision-only and tactile-conditioned deployment. To scale this supervision, we build HumanTouch, a 1,000-hour dataset of full-hand tactile human manipulation, and pretrain Spark-0 on it together with egocentric video, robot data, and simulation. We also release SparkArena, which pairs vision-only simulated tasks with tactile-conditioned real-robot tasks on one dexterous embodiment. Spark-0 achieves the highest mean success rate on both SparkArena tracks and ranks first on all eight aggregate metrics of the Ego Humanoid Manipulation Benchmark. Compared with vision-only pretraining on the same data and compute with touch discarded, retaining the touch of 1,000 hours of HumanTouch raises vision-only success on Ego Humanoid from 52.5% to 71.7%, and success on SparkArena-Real, where touch is also used at deployment, from 44.0% to 52.0%, indicating that touch serves not only as a policy input but also as scalable supervision for dexterous visuomotor pretraining. The Website is available at: https://xsparkai.com/Spark-0.

1 Introduction

The paper asks whether touch can supervise visual representations at scale, addressing scarce full-hand tactile data and limited evaluation of both visual dexterity and tactile control. It introduces HumanTouch, Spark-0, and SparkArena, and reports strong results across their evaluation benchmarks.

1 Introduction

  • The paper identifies three gaps: scarce synchronized full-hand tactile data, limited use of touch to supervise visual representations, and few benchmarks for visual dexterity and tactile control.These gaps motivate its question of whether touch can serve as scalable visuomotor pretraining supervision, not only as a policy input.Source: §1 · p. 2

Contributions

  • HumanTouch contributes 1,000 hours of full-hand tactile human manipulation, recorded across more than 100 tasks and 32 operators.Bilateral gloves, hand and wrist tracking, and head and wrist cameras are synchronized at 60 Hz; the gloves also mount on the target dexterous hand.Source: §1 · p. 2, Contributions · p. 2
  • Spark-0 uses touch both to supervise visual tokens during training and to condition the action expert, supporting vision-only or tactile-conditioned deployment from one checkpoint.The contact-map and 3D-geometry objectives are removed at inference, and raw touch bypasses the vision-language backbone.Source: §1 · p. 2, Contributions · p. 2

1 Introduction

  • Spark-0 achieves the highest mean success rate on both SparkArena tracks and ranks first on all eight Ego Humanoid aggregate metrics.With touch retained during pretraining, vision-only Ego Humanoid success rises from 52.5% to 71.7%, compared with vision-only pretraining on the same data and compute with touch discarded.Source: §1 · p. 2

Contributions

  • SparkArena pairs vision-only simulated dexterity with tactile-conditioned real-robot tasks on the same dexterous embodiment.The benchmark is designed to evaluate deployment both with and without touch, which the authors say existing benchmarks do not support.Source: Contributions · p. 2, §1 · p. 2

2 Related Work

Related work covers tactile datasets and representations, policies for contact-rich dexterous manipulation, and manipulation benchmarks. The paper positions Spark-0 as combining large-scale tactile pretraining, visual-representation supervision, and evaluation of both vision-only and tactile control.

2.1 Tactile data

  • HumanTouch addresses the scarcity of tactile datasets with synchronized hand pose and multi-view video by providing both at 1,000 hours, using gloves shared by human and robot acquisition.Prior tactile work includes datasets from human interaction, visuo-tactile grippers, and simulation, as well as methods for estimation, generation, completion, aggregation, and transfer.Source: §2.1 · p. 3

2.2 Policies for contact-rich dexterous manipulation

  • Prior VLA and tactile policies use touch as an action input, prediction target, or feedback, whereas Spark-0 also uses it to supervise visual representations.The paper distinguishes its approach from STAR, which addresses tactile sparsity with 200 hours of bimanual robot data.Source: §2.2 · p. 3

2.3 Manipulation benchmarks

  • SparkArena evaluates vision-only dexterity and real-robot tactile control on the same dexterous embodiment, unlike benchmarks that mostly assess dexterous or visuo-tactile tasks in simulation.The related-work discussion also notes benchmarks for gripper-based multitask, long-horizon, bimanual, memory-dependent, and sim-and-real manipulation.Source: §2.3 · p. 3

3 HumanTouch and Pretraining Data

HumanTouch supplies large-scale full-hand tactile human manipulation data, and the pretraining mixture combines it with egocentric video, robot data, and simulation. The dataset’s acquisition setup synchronizes tactile, motion, and video signals, while its tasks vary across contact patterns, objects, motions, and durations.

3.1 Acquisition hardware

  • The acquisition unit records bilateral tactile gloves, 25-joint finger tracking, 6-DoF wrist tracking, and head- and wrist-camera video synchronized at 60 Hz.The gloves have 460 sensing points per hand; 140 points over flexion regions are masked because bending produces spurious pressure, leaving 320 points for tactile input and contact maps.Source: §3.1 · pp. 3–4
  • Spark-0’s architecture uses a Qwen3-VL-conditioned flow-matching action expert, with raw touch bypassing the VLM and train-only objectives aligning visual tokens.Even layers cross-attend to VLM features; odd layers self-attend over state, tactile, and action tokens, and regional force and 3D taxel streams yield 24 tactile tokens.Source: Figure 3 · p. 4

3.2 Dataset scale and diversity

  • HumanTouch comprises 1,000 hours across more than 100 tasks and 32 operators, with variation in contact patterns, object properties, motion complexity, and episode duration.Figure 2 presents representative in-the-wild tasks spanning diverse objects, scenes, and motions.Source: Figure 2 · p. 3, §3.2 · p. 4

3.3 Pretraining mixture

  • The pretraining mixture assigns distinct roles to HumanTouch, egocentric video, real-robot data, and domain-randomized simulation.HumanTouch supplies full-hand contact supervision, egocentric video broadens scene, object, and motion coverage, robot data align the model with its target embodiment, and simulation adds visual variation.Source: §3.3 · p. 4

4 Spark-0

Contact-rich dexterous control needs information about where contact occurs and how it changes, which vision alone captures poorly. Spark-0 aligns visual features with contact and geometry during training, while a tactile pathway conditions actions when touch is available.

4.1 VLA backbone and action objective

  • Human and robot data are converted to the target bimanual layout, with states in the head-camera frame and actions relative to the current end effector.The relative action chunk is reanchored to the latest observed pose at inference, yielding 58-dimensional states and actions across both hands.Source: §4.1 · p. 5
  • The action expert uses layerwise visual-language features and state history to predict a 50-step action chunk through flow matching.Learnable action queries summarize visual-language context, and inference integrates predicted velocity with four Euler steps.Source: §4.1 · p. 5

4.2 Spatial alignment: supervising interaction regions

  • At a deep VLM layer, train-only geometry and contact alignment supervise visual tokens; both alignment heads are discarded at inference.A frozen geometry teacher supplies scene features, while glove-derived contact maps mark interaction regions in the head view.Source: §4.2 · pp. 5–6

4.3 Tactile fusion: conditioning actions on contact dynamics

  • A tactile pathway encodes 20-frame regional force histories and current 3D taxel states as tokens for the action expert.Twelve force tokens and twelve 3D tokens represent the six regions on each hand; tactile, state, and action tokens are jointly updated.Source: §4.3 · p. 6
  • Missing or invalid touch uses a learned placeholder rather than zero pressure, distinguishing unavailable sensing from absent contact.The fixed token layout allows vision-only and tactile samples to share a batch.Source: §4.3 · p. 6

4.4 Training objective and deployment

  • The same weights support vision-only and tactile-conditioned deployment, with alignment teachers and heads removed and tactile placeholders used when touch is absent.Geometry alignment applies across available views, contact alignment to valid maps, and tactile encoders learn through action loss on samples with touch.Source: §4.4 · p. 6

5 SparkArena Evaluation Suite

SparkArena evaluates vision-only dexterity and tactile-conditioned contact-rich control on the same bimanual dexterous embodiment. Its simulation and real-robot tracks use distinct task sets and report both task success and progress through subgoals.

5 SparkArena Evaluation Suite

  • Both tracks use the same bimanual dexterous embodiment, pairing vision-only simulation with real-robot control using online touch.The real platform uses tactile gloves with the HumanTouch taxel layout; simulation reproduces the kinematics and cameras without touch.Source: §5 · p. 7
  • SparkArena-Sim tests seven vision-only tasks, while SparkArena-Real tests five contact-rich tasks under paired initial conditions.The simulated tasks span tool use, actuation, grasping, retrieval, stacking, collection, and household manipulation; real tasks span insertion, packing, placement, stacking, and reorientation.Source: §5 · p. 7
  • Both tracks report success rate and completion score, distinguishing trials that achieve every task goal from progress through subgoals.Paired initial conditions address the noise of real-robot trials.Source: §5 · p. 7

6 Experiments

The experiments test Spark-0 with and without deployment-time touch and examine how pretraining and HumanTouch scale affect performance. Spark-0 leads across simulated, real-robot, and cross-embodiment evaluations, with full-scale tactile pretraining improving vision-only transfer.

6.2 Overall performance

  • Spark-0 reaches 36.86% success and a 39.74 completion score on SparkArena-Sim, exceeding FastWAM by 8.57 and 6.23 points, respectively.It achieves or ties the best success rate on four of seven tasks while running without touch.Source: Table 1 · p. 8, §6.2 · p. 8

6.1 Experimental setup

  • The evaluation compares Scratch, vision-only pretraining, and 100- or 1,000-hour tactile pretraining across three benchmarks.Policies are fine-tuned for each benchmark; SparkArena-Real uses touch at deployment, while SparkArena-Sim and Ego Humanoid do not.Source: §6.1 · p. 8

6.2 Overall performance

  • Spark-0 ranks first on all eight Ego Humanoid aggregate metrics, with its success advantage growing from short- to long-horizon tasks.It runs without touch; the advantage over the strongest baseline rises from 3.17/6.86 points on seen/unseen short-horizon tasks to 18.52/22.12 points on long-horizon tasks.Source: §6.2 · pp. 8–9, Table 2 · p. 9
  • Spark-0 reaches 52.00% success and a 62.80 completion score on SparkArena-Real, exceeding tactile-policy baseline FTP-1 by 16.00 and 15.00 points.Spark-0 uses touch, and has the highest completion score on four of five tasks.Source: Table 3 · p. 9, §6.2 · p. 9

6.3 Pretraining and tactile-scale analysis

  • Adding 1,000 hours of HumanTouch to the same RGB data raises Ego Humanoid success by 19.27 points over Visual PT, while SparkArena-Sim changes by only +0.29.Ego Humanoid is evaluated without touch, so this gain reflects tactile pretraining rather than online tactile feedback.Source: §6.3 · pp. 9–10, Table 4 · p. 10
  • Ego Humanoid success rises from 52.45% with Visual PT to 59.42% with 100H and 71.72% with 1000H.Only 1000H achieves the best success rate on all three benchmarks.Source: §6.3 · p. 10

7 Conclusion

Spark-0 combines train-only tactile supervision of visual tokens with tactile input to the action expert, supporting vision-only and tactile-conditioned use from one checkpoint. It leads on both SparkArena tracks and all Ego Humanoid aggregate metrics.

7 Conclusion

  • Spark-0 achieves the highest mean success rate on both SparkArena tracks and ranks first on all Ego Humanoid aggregate metrics.The largest gain from 1,000 hours of HumanTouch appears on unseen Ego Humanoid under vision-only deployment.Source: §7 · p. 10
  • HumanTouch provides 1,000 hours of full-hand tactile human manipulation, and SparkArena pairs vision-only simulated tasks with tactile-conditioned real-robot tasks.The conclusion presents both alongside Spark-0.Source: §7 · p. 10
  • Touch serves both as train-only supervision for visual tokens and as an action-expert input, so one checkpoint supports vision-only and tactile-conditioned deployment.The conclusion identifies touch as scalable supervision for dexterous visuomotor pretraining.Source: §7 · p. 10

A HumanTouch Dataset Details

HumanTouch records synchronized full-hand tactile manipulation data using a wearable rig, with the same tactile layout usable on the target robot hand.

A HumanTouch Dataset Details

  • HumanTouch contains 1,000 hours of manipulation across more than 100 tasks and 32 operators, recorded at 60 Hz.The rig records head- and wrist-camera video, bilateral tactile signals, finger-joint tracking, and wrist poses.Source: A HumanTouch Dataset Details · p. 15
  • Each glove has 460 sensing points per hand, but 140 points over flexion regions are masked because bending induces spurious pressure.The remaining 320 taxels form the model observation.Source: A HumanTouch Dataset Details · p. 15
  • The 320 taxels are grouped into six anatomical regions per hand: 240 palm taxels and five subsequent 16-taxel finger blocks.The tactile encoder and contact-map generation both use this layout.Source: A HumanTouch Dataset Details · p. 15
  • The wearable rig synchronizes multi-view RGB, hand motion capture, and bilateral tactile gloves.The same gloves can also be mounted on the target dexterous hand.Source: Figure 6 · p. 15, A HumanTouch Dataset Details · p. 15

B Architecture Details

Spark-0 combines a tactile-conditioned action expert with train-only spatial supervision of visual tokens. Its tactile encoder represents both regional force histories and 3D taxel states, while contact maps provide hand-level visual targets.

B.1 Action expert

  • Even action-expert blocks cross-attend to VLM states, while odd blocks use masked self-attention over State–Tactile–Action; each block has one attention operation and a feed-forward sublayer.Action tokens use learned positional embeddings; tactile tokens instead use stream, hand, and region embeddings.Source: §B.1 · p. 15

B.2 Tactile encoder

  • The tactile encoder combines temporal regional-force tokens with current-frame 3D taxel tokens, producing 12 tokens per stream.Regional tokens summarize six force signals over 20 frames; 3D tokens pool 320 taxels per hand within regions. Present streams are gated, while absent streams use learned placeholders.Source: §B.2 · p. 16
  • The 3D stream is dropped during training with probability 0.15, and low-quality rows are replaced by placeholders; the regional stream has no modality dropout.Rows are replaced when fewer than 25% of coordinates and pressures are finite or more than 85% are saturated above magnitude 1.5.Source: §B.2 · p. 16

B.3 Joint attention and tactile shortcut

  • During training, the tactile shortcut routes VLM context to Action rows through tactile rows by zeroing State and Action cross-attention outputs in selected even blocks.The shortcut probability ramps to 0.1 and the shortcut is disabled at inference.Source: §B.3 · p. 17

C Train-Only Spatial Supervision

  • Train-only geometry and contact objectives align visual tokens with frozen-teacher features and contact targets resampled to the same image-token grid.The objectives supervise layer-25 tokens; both heads are discarded at inference.Source: C Train-Only Spatial Supervision · p. 17

C.1 Contact-Map Generation

  • Contact detection uses pressure level and rise thresholds, with hysteresis for holding and releasing contact; taxel intensity combines current pressure with onset sharpness.After release, intensity decays linearly to zero over eight frames.Source: §C.1 · pp. 17–18
  • Active taxels are diffused over a MANO hand mesh and rendered into the head view, with the two hands merged by pixelwise maximum.The resulting targets cover nearly the visible hand and encode hand-level contact regions rather than precise contact points; sequence-wide noise estimation is non-causal because maps are training targets only.Source: §C.1 · p. 18

D Training and Deployment

The released training recipe uses a heterogeneous pretraining mixture, then fine-tunes from pretrained checkpoints and supports deployment with or without tactile input. SparkArena uses the Gen 1 Prototype embodiment, with task success defined by its scoring criteria.

D.1 Pretraining

  • Pretraining runs for 400k optimizer steps across seven sources, with touch present in HumanTouch (0.35) and same-glove robot data (0.09).Mixture exposure is set by sampling weight, not hours.Source: §D.1 · p. 18, Table 5 · p. 19

D.2 Fine-tuning and deployment

  • Fine-tuning warm-starts pretrained variants for 80k steps, while Scratch starts from Qwen3-VL initialization; deployment inserts learned placeholders when tactile input is unavailable.Deployment uses asynchronous real-time chunking and guidance to keep replanned actions consistent with actions still executing.Source: §D.2 · pp. 18–19

E.1 Gen 1 Prototype embodiment

  • SparkArena-Sim reproduces the Gen 1 Prototype’s kinematics and cameras without tactile input; the real platform uses two WUJI Hand 2 hands with Moxian gloves and the HumanTouch 320-taxel layout.The embodiment also has two arms, a head camera, and two wrist-embedded cameras.Source: §E.1 · p. 19

E.2 Success and scoring criteria

  • A trial receives the score for its highest reached stage and succeeds exactly at 100; tasks without intermediate stages use binary success.The criteria and stage scores are defined in Table 7.Source: §E.2 · p. 19

E.3 Real-robot task assets

SparkArena releases the real-robot setup and task assets alongside contact-sequence visualizations. The baseline set spans generalist robot policies, human-prior transfer, world-action models, and tactile policies.

E.3 Real-robot task assets

  • The release includes SparkArena-Real hardware and sensor configuration, task definitions, paired initial conditions, scoring protocols, and the SparkArena-Sim environment and evaluation code.Task assets are shown by task; equivalent items of matching size and material may be substituted for listed purchases.Source: §E.3 · p. 19, Figure 9 · p. 21, Table 8 · p. 21

E.4 Contact sequences

  • Figure 10 expands each SparkArena-Real task into a contact sequence, with each row representing a task and columns showing tactile response over time.Source: §E.4 · p. 20

F Baseline Details

  • The baselines span VLM-based robot policies, large-scale diffusion models, human-prior transfer methods, world-action models, and tactile policies.The described set also includes general-purpose and multi-stream policies.Source: F Baseline Details · pp. 20–21

G.1 Per-task Ego Humanoid results

Spark-0 (1000H) has a higher overall mean than the Visual PT and 100H variants on Ego Humanoid. Table 11 supplies per-task SparkArena-Real results for the baselines and Spark-0 variants.

G.1 Per-task Ego Humanoid results

  • 71.72% overall mean for Spark-0 (1000H), versus 52.45% for Visual PT and 59.42% for the 100H variant, across Ego Humanoid tasks.The aggregates are arithmetic means over tasks; Tables 9 and 10 report short- and long-horizon per-task results.Source: §G.1 · p. 21, Table 9 · p. 22, Table 10 · p. 23

G.2 Complete SparkArena-Real results

  • Table 11 reports per-task SparkArena-Real results for all baselines and Spark-0 variants.The real-robot entries in Tables 3 and 4 are taken from Table 11.Source: §G.2 · p. 21, Table 11 · p. 23

Related papers

Loading 2610.22342v1…