Source-linked AI summary
RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Chenxi Wang, Junbo Wang, Haoyi Zhu, Cewu Lu
TL;DR
Robotic manipulation learning has been limited by datasets centered on simple, visually guided tasks, despite the need for diverse skills and richer sensing. RH20T addresses this gap by collecting a large real-world dataset of contact-rich manipulation with calibrated visual, force, audio, and action data, paired with human demonstrations and language descriptions. It provides over 110,000 robot sequences spanning diverse skills, robots, and environments, while its current limitations include expensive data collection and no evaluation of robotic foundation models.
Problem
Existing manipulation-learning datasets and methods largely focus on simple tasks with visual guidance, while diverse skills may require contact-rich interaction and multiple sensory modalities.
Method
The paper constructs RH20T from real-world robot manipulation sequences using diverse robots and environments, calibrated multi-modal sensing, and paired human videos and language descriptions.
Results
RH20T contains over 110,000 robot manipulation sequences with corresponding human demonstrations, spanning diverse contact-rich skills and multi-modal observations.
Takeaways & Limitations
RH20T provides a public resource for studying diverse robotic skill learning and multi-sensory manipulation across real-world tasks.
Takeaways & Limitations
The paper identifies expensive data collection and the absence of robotic foundation-model evaluation as current limitations.
Abstract
from arXiv · showhide
A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations. This feature is attractive for enabling robots to acquire new skills and improving task and motion planning. However, due to limitations in the training dataset, the current focus of the community has mainly been on simple cases, such as push or pick-place tasks, relying solely on visual guidance. In reality, there are many complex skills, some of which may even require both visual and tactile perception to solve. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception. To achieve this, we have collected a dataset comprising over 110,000 contact-rich robot manipulation sequences across diverse skills, contexts, robots, and camera viewpoints, all collected in the real world. Each sequence in the dataset includes visual, force, audio, and action information. Moreover, we also provide a corresponding human demonstration video and a language description for each robot sequence. We have invested significant efforts in calibrating all the sensors and ensuring a high-quality dataset. The dataset is made publicly available at rh20t.github.io
I. INTRODUCTION
RH20T addresses the limited diversity and modality of existing robotic manipulation learning by collecting real-world data for complex, contact-rich skills. Its dataset combines diverse robots, environments, skills, demonstrations, and calibrated multi-modal sensing at large scale.
- Existing one-shot imitation learning studies mainly evaluate simple real-world tasks such as pushing, picking, and placing.
- Large and diverse robotic manipulation datasets remain scarce because configuring robots, creating environments, and collecting trajectories require substantial effort and resources.
- RH20T uses a force-torque sensor and haptic device with force rendering to support precise and efficient data collection.
- The dataset covers around 150 complex skills across multiple robot arms, with robot-configuration diversity intended to aid generalization to other robots.
- Many complex manipulations require contact-rich interaction, including cutting, plugging, slicing, pouring, folding, and rotating.
- RH20T contains around 110,000 robot manipulation sequences and corresponding human videos, totaling over 40 million robot-image frames and over 10 million human-demonstration frames.
- Each robot sequence includes visual, tactile, audio, and proprioception information, and the dataset is organized to support broad skill learning.
II. RELATED WORKS
Prior robotic datasets and imitation-learning methods generally emphasize small sets of simple, visually guided tasks. RH20T extends this scope with diverse contact-rich skills, calibrated multi-modal episodes, and broad robot and environment coverage.
- Each RH20T episode includes visual, force, audio, and action data from calibrated multi-view cameras, plus a human demonstration and language description.
- RH20T provides over 110K robot episodes, 110K corresponding human demonstrations, over 50 million frames, and over 140 tasks.
- Compared with prior datasets, RH20T emphasizes contact-rich manipulation and includes the force modality.
- Existing one-shot imitation methods use descriptors such as images, language, robot videos, or human videos, but mainly address simple visual manipulations.
- RH20T is designed to support learning hundreds of skills requiring multi-modal perception within a single imitation-learning model.
- The open question is how to handle multi-modal perception at different frequencies coherently across diverse skills.
- Table I defines camera calibration as extrinsic calibration of all cameras and the robot, while noting that depth pairing may cover only part of the images.
- Table III indicates that fingertip tactile sensing is available only in robot configuration Cfg 7.
III. RH20T DATASET
RH20T is a large, diverse real-world robotic manipulation dataset designed for general manipulation from task descriptions and human demonstrations. It combines multi-modal sensing, varied robots and environments, paired human–robot data, and both short and compositional tasks.
- Diversity: RH20T contains 147 tasks spanning 42 skills, with hundreds of objects, four robot arms, four grippers, three force-torque sensors, and seven robot configurations.Environment diversity includes more than 50 table covers with varied textures and materials, irrelevant objects, and manipulations by tens of volunteers.
- Multi-Modal: Each episode combines visual, tactile, audio, and proprioceptive information with timestamps, while calibrated multi-view cameras support aligned sensing.Visual data includes RGB, depth, and binocular infrared images; force-torque, fingertip tactile, in-hand and global audio, and robot state information are also provided.
- Scale: Over 110,000 robot sequences and an equal number of human sequences provide more than 50 million images in total, averaging approximately 750 robot manipulations per skill.The dataset also includes statistics on manipulation counts and execution times, with most sequences lasting 10–100 seconds.
- Data Hierarchy: For each task, a tree hierarchy organizes intra-task similarities so robot manipulation sequences can be paired with human demonstrations across viewpoints, scenes, subjects, and environments.This creates dense human-demonstration and robot-manipulation pairs for learning viewpoint- and context-robust task semantics.
- Compositionality: RH20T includes both single-manipulation sequences and longer tasks composed of multiple short tasks, enabling investigation of whether short-sequence mastery improves long-sequence acquisition.An example composes grabbing a plug, plugging it in, switching on the socket, and turning on a lamp.
IV. EXPERIMENTS
The experiments evaluate whether RH20T pretraining improves ACT transfer in a new environment, few-shot learning, convergence, and generalization to novel objects and table covers.
- Scope: The experiments demonstrate dataset effectiveness for transferring a baseline model, rather than evaluating robotic foundation models directly.The authors attribute the latter evaluation gap to computational-resource limitations.
- Results: Pretraining on RH20T data improves final success despite differences in camera viewpoints, robot embodiments, and backgrounds.Including data from different tasks during pretraining further improves the overall success rate.
- Results: Pretraining on RH20T data accelerates convergence compared with training without pretraining.The comparison contrasts 500 training epochs with pretraining against 750 epochs without pretraining.
- Few-shot transfer: With 40 demonstrations, RH20T-pretrained models outperform models trained with 75 demonstrations without pretraining.With only 10 demonstrations, pretraining on multiple RH20T tasks still surpasses the 75-demonstration, no-pretraining counterpart.
- Generalization: On novel weights and table covers, models pretrained on multiple similar RH20T tasks consistently outperform models without pretraining.Both models use the original 75 demonstrations and are trained for 750 epochs.
V. DISCUSSION AND CONCLUSION
The paper presents RH20T as a dataset for diverse robotic skill learning and releases it to support progress in robotic manipulation and foundation-model research.
- Contribution: RH20T is presented as a dataset for diverse robotic skill learning, with potential applications in robotic manipulation in novel environments.The paper states that the dataset is open-sourced to promote development in the area.
- Limitations: The paper identifies expensive data collection as a current limitation.The authors also report that robotic foundation-model potential was not evaluated because available computing resources were insufficient.
- Future work: Future work aims to extend RH20T to dual-arm and multi-finger dexterous manipulation.This is stated as a planned direction for broader robotic manipulation coverage.
Appendix Task Specification of RH20T
The appendix lists RH20T task specifications spanning self-collected, Meta, World, Proposed, and RLBench sources, with varied manipulation objectives.
- Task specification: The task table records task descriptions and their sources, and task IDs are not necessarily continuous.The listed sources include Self, Meta, World, Proposed, and RLBench.
- Task specification: Other tasks require pressing buttons, turning devices on or off, waving flags, and manipulating household objects.The examples include desk lamps, power strips, telephones, plants, soccer balls, and cups.
- Task specification: The specification includes object transport and placement tasks, such as moving blocks, grabbing objects, and placing items at designated locations.Examples include moving a block left to right and placing a block at a marked location.
23. Turn
The task list covers a broad range of object manipulation behaviors, including opening, pouring, stacking, shaking, rotating, and handling everyday items.
- Task specification: Several tasks involve opening or closing containers and manipulating lids, drawers, boxes, and test tubes.Examples include opening a box, closing a drawer, and screwing a lid onto a container.
- Task specification: The appendix includes pouring, sliding, sweeping, stacking, shaking, and vertically placing objects.Examples include pouring from a cup, stacking blocks, sweeping squares, and shaking a test tube.
- Task specification: The listed objects include cups, brushes, pens, blocks, paper, lamps, drawers, boxes, and test tubes.These examples illustrate the range of everyday object interactions represented in the task specification.
48. Rotate
This section lists manipulation tasks involving rotation, turning, catching, shooting, and switching or receiving objects.
- 48. Rotate: The tasks include rotating a cup and a rack by 90 degrees.One task rotates a cup 90 degrees; another rotates a rack 90 degrees counter clockwise.
- 48. Rotate: The section includes placing and removing a dish from a dish rack.
- 48. Rotate: Additional tasks use a clamp for liquid and catch a moving object.
- 48. Rotate: The listed actions also include receiving an object and turning a power strip on or off.
60. Turn
This section covers turning a knob to increase or decrease volume, handling toilet paper, and pressing a button.
- 60. Turn: The tasks include turning a knob to increase or decrease volume.
- 60. Turn: Other listed tasks involve taking toilet paper from its holder and scooping up an object with a toilet shovel.
- 60. Turn: The section also includes pressing a button and building with small Lego or Megabloks.
66. Build
This section presents construction, opening and closing, scooping, folding, attaching, dialing, pushing, and object-placement tasks across several environments.
- 66. Build: The tasks include building with Lego or Megabloks and assembling pieces in RLBench or Meta-World.
- 66. Build: The section includes opening and closing a sliding drawer or window and placing objects into a drawer.
- 66. Build: Additional tasks approach and touch an object, put a box down, and open a photo frame or bracket.
- 66. Build: Other tasks scoop water with a spoon, fold paper, and attach a bubble ring to a fly swatter.
- 66. Build: The listed skills include picking up and placing an object, dialing a number, and pushing an object with obstacles.
91. Move
This section lists diverse object-moving and tool-use tasks, including cleaning, stirring, dragging, cutting, plugging in, and manipulating pipette tips.
- 91. Move: The tasks include taking objects from drawers, cleaning glassware or tables, stirring with a glass rod, and scrubbing or dragging plates.
- 91. Move: Other manipulation tasks include inserting and removing pencils from a sharpener and matching objects with corresponding shapes or tape.
- 91. Move: Several tasks involve turning lights or levers, removing or placing knives, and plugging power cords or bulb holders into sockets.
- 91. Move: The listed tasks also cover stacking blocks, placing objects into garbage, and moving objects between locations.
- 91. Move: The section includes inserting pipette tips into holders and transferring large, medium, or small tips between holders or racks.
- 91. Move: Food-preparation tasks include chopping garlic, chili peppers, carrots, lotus root, onions, cucumbers, oranges, and potatoes.