TacClaw-Infinity: A Large-Scale Vision-Tactile-Language-Action Dataset for Dexterous Manipulation

I. INTRODUCTION

• TacClaw-Infinity uses a tactile-enabled, robot-independent handheld interface to synchronously record visual observations, fingertip tactile signals, and end-effector trajectories.Data collection does not require a target robot.Source: §I · p. 2
• Over 10,000 hours, 1 million episodes, and 1 billion frames span more than 1,000 task types, collected across 50+ locations and 20+ organizations.The collection spans homes, laboratories, industrial workplaces, retail stores, and outdoor environments.Source: §I · p. 2
• A scalable pipeline reconstructs trajectories, applies multimodal quality control and embodiment-aware feasibility filtering, and adds temporal, semantic, and spatial annotations.Annotations include contact-centric subtask segmentation, segment-level language descriptions, and contact-anchored target-object localization and mask tracking.Source: §I · p. 2
• Policies trained solely on TacClaw-Infinity transfer across three heterogeneous robot embodiments without embodiment-specific demonstrations.Evaluations also assess two policy families and use controlled comparisons to isolate tactile observations’ contributions to downstream policy learning.Source: §I · p. 2

A. Large-Scale Robot Demonstration Datasets

• Large-scale datasets support generalization and vision-language-action policies, but remain largely vision-centric or rely on robot-side collection; synchronized high-resolution tactile data remain scarce.The review names RoboNet, RT-1, BridgeData V2, DROID, Open X-Embodiment, and RH20T,,,,,.Source: A. Large-Scale Robot Demonstration… · p. 2

B. UMI-Style and Tactile Data Collection

• Robot-free UMI work includes in-the-wild collection, simplified hardware and pose tracking, and more than 100,000 multimodal demonstrations [15].Related work also links object and environment diversity to generalization and foundation-model pretraining for open-vocabulary and zero-shot cross-embodiment manipulation, [17].Source: B. UMI-Style and Tactile Data Collection · p. 2
• VISTA, HiFi-UMI, and UMI-Bench address visual-domain mismatch, synchronization accuracy, kinematic feasibility, and standardized real-robot evaluation [18], [19], [20].Portable tactile UMI systems include ViTaMIn, FreeTacMan, and Touch in the Wild [4], [7],, with later systems exploring additional sensing and manipulation settings, [21],, [23], [24], [25], [26], [27], [28].Source: B. UMI-Style and Tactile Data Collection · pp. 2–3
• Jointly scaling synchronized vision–tactile–language–action data across distributed environments with temporal, semantic, and spatial supervision remains underexplored.TacClaw-Infinity combines multi-site tactile UMI collection with structured contact-centric annotations for single-hand and bimanual manipulation.Source: B. UMI-Style and Tactile Data Collection · p. 3

C. Structured Annotation for Long-Horizon Manipulation

• TacClaw-Infinity jointly annotates temporal stages, segment-level language descriptions, and spatial targets with a task-agnostic, contact-centric framework.The framework addresses the dataset- and task-family specialization of earlier segmentation and demonstration-parsing approaches,,,,,.Source: C. Structured Annotation for… · p. 3

A. Dataset Overview

• The dataset contains over 10,000 hours, 1 million episodes, and 1 billion frames across more than 1,000 task types.Each episode includes synchronized visual observations, fingertip tactile signals, end-effector trajectories, language descriptions, and temporal, semantic, and spatial annotations.Source: A. Dataset Overview · p. 3
• The corpus covers single-hand and bimanual manipulation across diverse objects, actions, environments, and physical interaction patterns.Source: A. Dataset Overview · p. 3

B. TacClaw and Data Collection Protocol

• TacClaw integrates wrist and stereo cameras, an IMU, encoder, and visuotactile sensor to record manipulation without requiring a target robot.The sensor captures three-axis tactile signals at up to 120 Hz; streams include visual observations, tactile modalities, and end-effector trajectories.Source: B. TacClaw and Data Collection Protocol · pp. 3–4
• Direct finger-to-jaw drive yields lower first-perceived contact force than trigger-driven or finger-loop mechanisms in a blindfolded comparison.The reported means are 11.6 gf for TacClaw, 49.8 gf for UMI, and 204.0 gf for the finger-loop gripper.Source: B. TacClaw and Data Collection Protocol · p. 3
• Collection across more than 20 organizations and 50 locations combines shared-catalogue skills with collector- and collaborator-defined tasks.This approach provides repeated demonstrations of common skills and contributes less frequent manipulation behaviors.Source: B. TacClaw and Data Collection Protocol · p. 4

C. Data Curation and Annotation Pipeline

• Demonstrations are evaluated in target-robot action spaces for reachability and manipulability, and trajectories failing either filter are excluded from policy training.Reachability uses inverse and forward kinematics; manipulability is assessed for reachable configurations using weighted-Jacobian isotropy, with 0.04 as the low-manipulability threshold.Source: C. Data Curation and Annotation Pipeline · p. 4
• Contact-aware annotation segments demonstrations into four phases, predicts frame-wise transitions, and generates a subtask description for each segment.The phases are free motion, contact onset, in-contact manipulation, and release; an ASFormer-based model uses tactile depth descriptors, visual features, and proprioceptive states.Source: C. Data Curation and Annotation Pipeline · pp. 4–5
• At contact, a projected inter-finger anchor identifies the manipulated object across synchronized views, where its mask is generated and propagated to a preceding contact-related event.The process uses a VLM-generated object noun and local box with SAM3, and rendered TacClaw masks as negative regions.Source: C. Data Curation and Annotation Pipeline · p. 5

a) Scene and Task Diversity

• More than 1,000 task types across 50+ locations vary in backgrounds, objects, operators, and manipulation conditions.The pick-cup task, for example, was demonstrated by 192 collectors across 8 organizations and 9 scene settings.Source: §IV · p. 5

b) Action and Object Diversity

• The action and object vocabularies include long-tail contact-rich operations such as insertion, alignment, wiping, rinsing, and plugging.The place-block task spans 56 devices and 6 scenes, with variation in color, material, shape, spatial relations, and behavior patterns.Source: §IV · p. 5

c) Tactile Signal Diversity

• Tactile features show substantial variation within object categories and partial separation between categories, reflecting varied contact states.Source: §IV · p. 5

d) Action–Object Coverage of Multi-site Collection

• At a 300-hour collection-time budget, multi-site collection covered 201 more distinct action–object pairs than single-site collection (95% confidence interval: [60, 335]), reflecting greater diversity across sites.The comparison sampled episodes without replacement to match budgets from 25 to 300 h and repeated sampling 2,000 times to estimate confidence intervals.Source: §IV · p. 6

V. EXPERIMENTS

• The evaluation covers device-level haptic fidelity, action-space compatibility, annotation quality, and zero-robot cross-embodiment transfer.Source: §V · p. 6
• The data-level experiments assess the reliability of collection and processing through interface, compatibility, and annotation evaluations.Source: §V · p. 6
• The experiments also assess the resulting dataset's utility for downstream robot learning.Source: §V · p. 6

A. Haptic Subject Experiment

• Nine blindfolded participants tested each interface by closing its jaws on a force gauge and saying “stop” at first perceived contact.Each participant completed two practice and eight recorded trials per interface, with randomized interface order.Source: A. Haptic Subject Experiment · p. 6
• 11.6±3.3 gf with TacClaw, 49.8±12.0 gf with UMI, and 204.0±58.2 gf with the finger-loop gripper were the perception thresholds; all nine participants had the lowest threshold with TacClaw.The Friedman test reports p < 0.001 and pairwise Wilcoxon pHolm = 0.012.Source: A. Haptic Subject Experiment · p. 6
• Direct drive transmits contact to the hand without an intervening mechanism and yields the lowest threshold among the three interfaces.The finger-loop gripper was also the least consistent, which the authors say may be related to static friction in the rail.Source: A. Haptic Subject Experiment · p. 6

• DM-Sparky achieves 100% bimanual reachability with full-body control, while Tianji reaches 97.23% and RealMan 95.15%; low-manipulability rates remain below 5% for all embodiments.Together, these results indicate strong cross-embodiment kinematic compatibility and that most reachable poses avoid poorly conditioned or near-singular configurations.Source: B. Action-space Validation · p. 6

a) Temporal

• +6.3 F1@25 and +6.6 C-F1@25 isolate the contribution of tactile sensing; contact-onset recall rises from 58.6% to 68.0%.The benchmark uses 30 held-out per-arm sequences; remaining errors concentrate on release, which is confused with free motion as tactile response decays gradually.Source: C. Annotation Quality · p. 6

b) Spatial

• 98.5% of frames in the protocol-driven recordings’ labelling window receive an object mask, but coverage does not establish mask accuracy, which is bounded by SAM3 tracking.Benchmarking mask accuracy and testing its downstream policy effect are left for future work.Source: C. Annotation Quality · p. 6

c) Semantic

• Roughly 83% of a 5% random sample of VLM-generated subtask descriptions correctly identifies the action and object; tasks with failed samples are held for manual re-annotation.Because review is sample-based, errors may remain in other descriptions.Source: C. Annotation Quality · p. 6

a) Experiment Setup

• The evaluation covers Tianji, RealMan, and DM-Sparky with π0.5 and StarVLA, across Cup Stacking, Bottle Sorting, and Vase Wiping.The tasks respectively probe pick-and-place coordination, tactile-dependent state discrimination, and dynamic contact regulation.Source: D. Zero-robot Cross-Embodiment Transfer · pp. 6–7, Figure 5 · p. 7
• Each task uses 1,600 training episodes and 50 evaluation trials, with checkpoints from epochs 3, 4, and 5 evaluated under the same model-selection protocol.Both policy families receive tactile observations through the same simple image-input formulation in tactile variants.Source: D. Zero-robot Cross-Embodiment Transfer · p. 7

b) Result analysis

• Policies trained solely on TacClaw demonstrations transfer across Tianji Marvin M6S Lite, RealMan RM75-b, and DM-Sparky One without embodiment-specific robot demonstrations.The reported transfer holds despite substantial differences in robot kinematics and hardware configurations.Source: D. Zero-robot Cross-Embodiment Transfer · p. 7, Table IV · p. 8
• Adding tactile observations raises π0.5 overall SR from 50.0% to 74.0% and StarVLA from 46.0% to 72.7%, averaged across three embodiments.Gains concentrate on Bottle Sorting and Vase Wiping; tactile input uses a simple image formulation without a specialized tactile architecture or controller.Source: D. Zero-robot Cross-Embodiment Transfer · p. 7

• TacClaw-Infinity is a large-scale structured Vision-Tactile-Language-Action dataset for contact-rich manipulation.It combines robot-independent tactile demonstration collection with distributed acquisition, scalable data curation, and temporal, semantic, and spatial supervision.Source: §VI · p. 8
• Evaluations validate the interface’s contact feedback, trajectory compatibility across embodiments, and contact-aware annotation; policies transfer across heterogeneous robots without embodiment-specific demonstrations.Controlled comparisons highlight the utility of tactile observations for contact-rich manipulation.Source: §VI · p. 8