Source-linked AI summary
RoboHive: A Unified Framework for Robot Learning
Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Jay Vakil, Abhishek Gupta, Aravind Rajeswaran
TL;DR
Robot learning lacks broadly recognized benchmarks and unified software frameworks, despite spanning diverse tasks and domains. RoboHive brings together environments, sim-to-real hardware support, teleoperation, and benchmarking resources. It emphasizes physical fidelity and visual diversity, while its task and hardware coverage remains incomplete.
Problem
Robot learning has fragmented software frameworks and lacks universally recognized benchmarks, raising entry barriers and limiting rapid prototyping.
Source: §1 · p. 1Method
Results
RoboHive emphasizes high physics fidelity and rich visual diversity across its environments to support visual-generalization research and real-world transfer.
Source: §1 · p. 2Takeaways & Limitations
The framework pairs task benchmarks with common hardware support, teleoperation, and demonstrations for research spanning simulation and physical robots.
Source: §1 · pp. 1–2RoboHive provides trained agents for only a few algorithmic families, its task coverage still needs additions such as aerial and navigation suites, and hardware support is not representative and favors manipulation over locomotion.
Source: §5 · p. 10
Abstract
from arXiv · showhide
We present RoboHive, a comprehensive software platform and ecosystem for research in the field of Robot Learning and Embodied Artificial Intelligence. Our platform encompasses a diverse range of pre-existing and novel environments, including dexterous manipulation with the Shadow Hand, whole-arm manipulation tasks with Franka and Fetch robots, quadruped locomotion, among others. Included environments are organized within and cover multiple domains such as hand manipulation, locomotion, multi-task, multi-agent, muscles, etc. In comparison to prior works, RoboHive offers a streamlined and unified task interface taking dependency on only a minimal set of well-maintained packages, features tasks with high physics fidelity and rich visual diversity, and supports common hardware drivers for real-world deployment. The unified interface of RoboHive offers a convenient and accessible abstraction for algorithmic research in imitation, reinforcement, multi-task, and hierarchical learning. Furthermore, RoboHive includes expert demonstrations and baseline results for most environments, providing a standard for benchmarking and comparisons. Details: https://sites.google.com/view/robohive
1 Introduction
Robot learning faces fragmented software frameworks and benchmarks that raise entry barriers and limit rapid prototyping. RoboHive addresses this with a unified, open framework spanning diverse environments, hardware support, teleoperation, and benchmarking resources.
1 Introduction
- RoboHive addresses fragmented frameworks and benchmarks by providing a unified, open-source ecosystem for robot learning.The platform is intended to support reinforcement, imitation, and transfer learning.Source: §1 · pp. 1–2
- Its environments span object and dexterous manipulation, legged locomotion, musculoskeletal models, multi-task settings, and deformable objects.RoboHive organizes these environments into suites, including Arms, Hand, Locomotion, Myo, Multi-Task, and Deformable.Source: §1 · pp. 1–2
- A unified Robot-Class abstraction connects simulated and physical robots through sim-hooks and hardware-hooks, enabling sim-to-real porting by changing one flag.The abstraction interfaces with both simulated and physical robots.Source: §1 · p. 2
- RoboHive provides keyboard, 3D space mouse, and VR teleoperation, and releases RoboSet, a human-teleoperated dataset covering 12 kitchen-task skills.The authors identify these tools and datasets as useful for imitation and offline learning.Source: §1 · p. 2
- The platform emphasizes high physics fidelity and rich visual diversity, including support for visual-domain and scene-layout randomization in several environments.These features are intended to aid visual generalization and facilitate real-world transfer.Source: §1 · p. 2, Figure 2 · p. 2
- RoboHive supplies concise evaluation metrics, a gym-like API, and baseline results for commonly studied algorithms across its environments.The baselines were developed in collaboration with TorchRL and mjRL.Source: §1 · p. 2
2 Relationship to Prior Work
Robot-learning benchmarks span heterogeneous domains, and existing benchmark clusters offer different strengths but leave gaps in realism, breadth, or ease of customization. RoboHive unifies diverse tasks and refreshed environments while retaining attention to physical realism and visual diversity.
2 Relationship to Prior Work
- Robot-learning questions span heterogeneous domains, so the field relies on benchmark clusters with distinct strengths rather than one uniform benchmark.The passage identifies locomotion, navigation, driving, and varied forms of manipulation as examples.Source: §2 · p. 3
- Unlike state-based toy benchmark clusters [10], RoboHive covers diverse simulated and real-world tasks across manipulation, locomotion, musculoskeletal agents, deformable objects, and multi-task and multi-agent settings.Its domain breadth contrasts with benchmarks that are primarily simple, state-based toy domains.Source: §2 · p. 3
- RoboHive targets benchmark gaps by combining physical realism and task diversity, unlike photorealistic benchmark clusters that trade off physical accuracy and domain-specific realistic efforts.Compared with IsaacGym [29], the authors cite RoboHive’s physical realism and lower development expertise demands as advantages for customization and rapid prototyping.Source: §2 · p. 3
- RoboHive consolidates adapted environments whose unsupported dependencies and divergent community customizations had fragmented comparisons, refreshing them with metrics, baselines, and visual diversity.Associated datasets were also reparsed and checked for reproducibility.Source: §2 · p. 3
3 Design Philosophy & Framework Overview
RoboHive’s design centers on a unified framework for robot-learning research, linking simulated and physical robots with diverse task environments and human or algorithmic agents. Its environments expose configurable observations and evaluation criteria, while teleoperation supports data collection for learning.
3.1 Abstract Robot Class for Simulation & Hardware Backends
- A shared robot class exposes unified sensor and robot APIs across simulation and hardware, allowing users to switch between them by changing a single flag.Simulation models use MuJoCo and support configurable domain randomization, sensor noise, and delays; hardware interfaces support position, velocity, and torque control.Source: §3.1 · pp. 3–4
3.2 Environments
- RoboHive organizes varied manipulation, locomotion, and musculoskeletal tasks into Gym-compatible suites, with abstract sensors and configurable observations.Environments separate success criteria from rewards and support vectorized simulation rollouts.Source: §3.2 · pp. 4–6
- The Multi-Task Suite supports shared experience across tasks, while the Multi-Agent Suite includes interaction with passive objects and other agents.The Multi-Task Suite builds on Franka-kitchen and table-scene tasks; the Multi-Agent Suite adapts tasks from prior work.Source: §3.2 · p. 6
3.3 Agents
- RoboHive supports standard RL agents through its Gym API and provides teleoperation interfaces across simulation and the real world to collect datasets for offline RL and imitation learning.Users can teleoperate through interfaces including a keyboard, 3D space mouse, and VR controllers.Source: §3.3 · p. 6
4 Capabilities and Utilities of the RoboHive Framework
RoboHive combines diverse environments and human or algorithmic agents with simulation-to-reality support, data collection, and benchmarking utilities. Its visual imitation benchmarks reveal substantial gaps between visual and privileged-state policies, with additional challenges across locomotion and teleoperated data.
4.1 RoboHive as a repository of environments and agents
- RoboHive pairs diverse, physically realistic environments with human teleoperators and algorithmic agents, and reports environment data-collection performance across task suites.Human interfaces can generate datasets; pre-trained policies support interaction, dataset generation, and baseline comparisons.Source: §4.1 · p. 7, Table 1 · p. 7
4.2 Sim and Real Counterparts
- RoboHive demonstrates synchronized deployment of simulation-trained policies in simulation and on physical robots for Franka manipulation and D’Kitty locomotion.The framework addresses simulation-to-reality transfer with rendering, physics modeling, and attention to real-world latency and sensing delays.Source: §4.2 · p. 7, Figure 6 · p. 8
4.3 Teleoperation and Dataset Collection
- RoboHive provides pre-collected RoboSet datasets and teleoperation interfaces for collecting additional data; RoboSet covers over 12 skills and over 30 tasks across multiple kitchen scenes.The paper describes RoboSet data composition by domain and source.Source: §4.3 · p. 8, Table 2 · p. 8
4.4 Datasets and Environments for Visual Imitation Learning and Offline RL
- Visual imitation baselines compare learned image representations on Kitchen, ROBEL D’Kitty locomotion, and Adroit Hand tasks against privileged-state and proprioception-only policies.Evaluations cover both expert-policy demonstrations and human-teleoperated datasets, averaging methods across seeds and camera angles.Source: §4.4 · pp. 8–9
- Visual policies trail privileged-state policies across expert and human-teleoperated datasets; on D’Kitty locomotion, visual baselines also perform worse than proprioception-only policies.Kitchen results show a larger performance gap with human-teleoperated data than with expert-policy data, and behavior cloning with 25 demonstrations can be challenging for Hand tasks such as Pen and Relocate.Source: §4.4 · p. 9
4.5 Well packaged and maintained ecosystem
- RoboHive consolidates diverse environments under a common base class with consistent constructors and method signatures.The package includes continuous integration checks and documentation, and the paper reports about five years of active maintenance at writing.Source: §4.5 · pp. 9–10
4.6 Reproducibility
- RoboHive tracks baseline performance across software versions and publishes release notes with individual runs and comparisons using at least three seeds for selected environments and algorithms.These comparisons are intended to check for performance regressions.Source: §4.6 · p. 10
5 Limitations
RoboHive’s trained-agent coverage, task-family coverage, and hardware support are incomplete and uneven.
5 Limitations
- Trained agents cover only a few algorithmic families, focused on those most commonly used for the corresponding task families.The authors hope open-sourcing and community engagement will broaden coverage.Source: §5 · p. 10
- Although RoboHive covers many task families, aerial and navigation suites remain areas the authors hope the community will expand.Source: §5 · p. 10
- Supported hardware is not representative, with manipulation hardware outweighing locomotion, navigation, and aerial hardware support.Source: §5 · p. 10
A Datasets
RoboSet includes human-collected and policy-expert datasets, organized as information-rich trajectories for learning research.
A Datasets
- RoboSet includes both human-collected data and expert datasets acquired from trained policies.Source: A Datasets · p. 15
- Each trajectory records observations, actions, rewards, RGB views from multiple cameras, and other environmental information in HDF5 format.Source: A Datasets · p. 15
- The datasets support research including imitation and offline reinforcement learning, visual generalization, and policy learning generalization.Source: A Datasets · p. 15
A.1 Collection Process
RoboSet combines expert-policy rollouts with human teleoperation data and replayed trajectories adapted from prior datasets.
A.1 Collection Process
- For each expert dataset, a task-specific trained NPG policy generates 25 trajectories for each of three trained agents, including failures.Source: §A.1 · p. 15
- Human trajectories are collected through Puppet teleoperation with an HTC Vive headset and controller in end-effector space, and are mostly successful.Source: §A.1 · p. 15
- The hand-manipulation human dataset adapts DAPG trajectories [51] by replaying them in RoboHive and recording additional observations, rewards, and task information.Source: §A.1 · p. 15
A.2 Reproducibility
The authors replayed 100 original trajectories in RoboHive and compared their final states to check trajectory validity and reproducibility.
A.2 Reproducibility
- The authors compare the original and replayed trajectories to assess the validity and reproducibility of replay.Source: §A.2 · p. 15
- The authors conclude that replay preserves dataset quality, keeping past results and findings valid for the RoboHive version of the task.Source: §A.2 · p. 15
- The final states of the 100 original and replayed trajectories converge within approximation bounds and are visually indistinguishable.Figure 9 shows a histogram of the norm of the difference between each original trajectory’s last state and its replayed counterpart.Source: §A.2 · p. 15, Figure 9 · p. 16
A.3 Access & Maintenance
RoboSet is open-sourced for users to train and evaluate policies, and its consistent format supports work across task suites.
A.3 Access & Maintenance
- RoboSet is open-sourced, with dataset lists and example trajectories available on the project Wiki.Source: §A.3 · p. 16
- Users can download a task-specific dataset, train policies with a preferred method, evaluate with 25 environment rollouts, and compare against baseline experiments.Source: §A.3 · p. 16
- Consistent dataset formatting reduces the overhead of switching between task suites and environments.Source: §A.3 · p. 16
- RoboSet’s expert and human data supports imitation, offline reinforcement learning, multimodal or play-data methods, and visual pre-training and generalization.Source: §A.3 · p. 16
B Environment sampling performeance
The section reports per-environment collection speeds and describes the asynchronous, multi-process setup used to collect environment data. It also notes the rendering configurations and excludes Myo from the rendering analysis.
B Environment sampling performeance
- Collection-speed results are reported per environment in Table 3, using 32 processes across 8 A100 GPUs on a single machine.The table reports averages and standard deviations across three seeds.Source: B Environment sampling performeance · p. 16, Table 3 · p. 17
- Asynchronous TorchRL collection used a “first ready first served” approach, with each 500-step batch collected in a separate process under a random policy.The collection totaled 800K steps; data were converted to floating-point tensors and resized to 84 × 84 pixels.Source: B Environment sampling performeance · p. 16
- Myo was excluded from the rendering analysis because it does not support dedicated cameras.Multi-task rendering used four 256 × 256 cameras; Arm and Hand used three 244 × 244 cameras.Source: B Environment sampling performeance · p. 16
- The authors report TorchRL collection as substantially faster than single-process or multiprocessed alternatives from most other frameworks [8].The passage refers readers to [8] for further details.Source: B Environment sampling performeance · p. 16
Checklist
The checklist points to sections covering limitations, reproducibility, and asset documentation. It also records consent and data-content statements for the datasets.
Checklist
- The authors direct readers to section 5 for limitations and state that the paper expects no negative societal impact.They attribute this expectation to the work not involving datasets or problems concerning humans or public places.Source: Checklist · p. 17
- The checklist points readers to subsection 4.6 for materials in the supplement or at a URL, and subsection 4.4 for training details such as data splits and hyperparameters.It marks stating theoretical assumptions and including complete proofs as not applicable.Source: Checklist · pp. 17–18
- Existing assets are credited in Table 2, while the project page links asset licenses and the dataset is licensed CC-BY 4.0.The checklist also marks new assets as included.Source: Checklist · p. 18
- The authors report explicit consent from dataset authors and say the data are released under the original license.They also state that the datasets contain robot trajectories without personally identifiable or offensive content.Source: Checklist · p. 18