DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

1 Introduction

• Existing benchmarks do not jointly support broad dexterous tasks, multiple arm-hand embodiments, controllable visual variation, demonstrations, parallel simulation, and representative policy evaluation.This makes comparing policies across contact regimes, visual conditions, and embodiments difficult.Source: §1 · p. 2
• DexVerse provides 100 tasks across diverse manipulation skills, supports multiple arms and dexterous hands, and allows configurable visual variation and modular extensions.The benchmark includes object grasping and relocation, articulated-object interaction, tool use, bimanual coordination, and long-horizon multi-stage manipulation.Source: §1 · pp. 2–3
• 34% mean online success is the best achieved by baselines across 19 tasks, with no single method consistently dominating and several tasks receiving zero success.Policies struggle particularly on tool-use and precision tasks; the results identify open challenges in contact-rich dexterity, fine-grained visuomotor control, and general-purpose manipulation.Source: §1 · pp. 2–3
• The benchmark is accompanied by 3,180 VR-teleoperated demonstrations and evaluations of representative imitation-learning and vision-language-action policies.Demonstrations include synchronized proprioceptive, RGB, depth, point-cloud, and simulator-state observations.Source: §1 · pp. 2–3

2 Related Works

• Many general-purpose benchmarks rely on parallel-jaw grippers, while dexterous benchmarks often focus on narrower settings or specialize in particular tasks and data regimes.DexVerse contrasts with these benchmarks by unifying broader dexterous coverage, multiple arm-hand embodiments, visual variation, demonstrations, and policy evaluations.Source: §2 · p. 3
• DexVerse combines embodiment changes with controllable visual variations and synchronized proprioceptive, RGB, depth, point-cloud, and state observations in one evaluation setting.The related-work discussion identifies visual shifts and differences in kinematics, degrees of freedom, workspace, joint limits, and contact geometry as distinct generalization challenges.Source: §2 · pp. 3–4
• The task taxonomy groups 100 tasks into eight categories by dominant interaction pattern and manipulation challenge rather than object type.Figure 2 visualizes selected tasks from the environments.Source: Table 2 · p. 4, Figure 2 · p. 4

3 DexVerse Environment

• DexVerse covers 100 tasks across primitive, functional, articulation, non-prehensile, contact-rich, bimanual, multi-goal, and long-horizon manipulation.Tasks are specified by their interactive objects, initial-state distribution, observation and action interfaces, and simulator-predicate success conditions; the suite supports category-level analysis across manipulation regimes.Source: §3 · p. 5

3.1 Modular Environment Design

• Configuration-driven environments use reusable task-family templates and task-specific parameters to define scenes, assets, interfaces, initialization, success conditions, and randomization.The design reduces duplicated implementation and supports environment changes through configuration overrides.Source: §3.1 · pp. 5–6

3.2 Robot Embodiments

• DexVerse supports 3 robot arms and 6 dexterous hands, with embodiment specifications that allow feasible arm-hand combinations to be changed without rewriting task logic.The supported hands span diverse kinematics, degrees of freedom, joint limits, actuation ranges, and morphologies; each hand also has a floating variant.Source: §3.2 · p. 6

3.3 Visual Variation

• Configurable visual variation samples object and table materials, lighting, backgrounds, exposure, and color temperature, and supports camera-viewpoint changes while preserving task objectives and success conditions.Visual and non-visual variations—including initial poses, sampling ranges, state perturbations, and dynamics—can be enabled independently or jointly.Source: §3.3 · p. 6

3.4 Asset Sources

• Assets combine research datasets, simulation libraries, public 3D repositories, and generated meshes, with 100 HDR skyboxes and randomized table materials supporting scene appearance diversity.When suitable objects are unavailable, candidate meshes are generated from reference images and manually processed for simulation.Source: §3.4 · pp. 6–7

4 Dataset and Evaluation

• Apple Vision Pro teleoperation streams simulation feedback through Isaac Lab’s CloudXR interface and maps tracked wrist poses to robot end-effector targets via inverse kinematics.Human hand motion is converted into target joint poses for different dexterous hands using optimization-based dex-retargeting.Source: §4 · p. 7
• The dataset contains 3,180 trajectories: 55 demonstrations for each of 56 single-goal tasks and 20 for each of 5 long-horizon tasks.For each single-goal task, 50 demonstrations use the Shadow Hand and one uses each of the other five hand embodiments.Source: §4 · p. 7

4.1 Imitation Learning Policy Evaluation

• All four policy families train on the same 950 episodes across 19 tasks and are evaluated closed-loop in the same simulator under identical termination criteria.The corpus is recorded as action-state pairs; replay restores simulator states and regenerates requested observations locally, avoiding accumulated rollout drift and allowing observation presets to change without duplicating the dataset.Source: §4.1 · p. 7, §4 · p. 7
• The authors attribute the VLA results to a pretraining-distribution gap: web-image and low-DoF priors transfer to perception but not to DexVerse’s high-DoF multifinger control.π0.5 (0.34) matches DP3 (0.34), while OpenVLA (0.19) trails both diffusion baselines.Source: §4.1 · pp. 7–8
• 0.34 overall success rate is shared by DP3 and π0.5, ahead of DP (0.32) and OpenVLA (0.19), while per-skill strengths diverge.DP is strongest on Pick-and-Lift, DP3 on Tool Use, and π0.5 on Precision Contact; Table 3 summarizes per-task success rates.Source: §4.1 · p. 7, Table 3 · p. 8
• The leading observation modality varies by skill: DP leads Pick-and-Lift (0.51), 3D DP leads Functional Tool Use (0.35), and π0.5 leads Articulated-object Manipulation (0.35) and Precision Contact (0.29).The paper links these differences to appearance and low-dimensional state, point-cloud geometry, and language conditioning with a flow-matching action expert, respectively.Source: §4.1 · p. 8
• PushT is 0.00 for all four policies, while InsertPen, SlideUtilityKnife, and OpenLaptop remain at or near zero across methods.The authors identify sustained force regulation and sub-centimeter alignment as demands that behavior cloning without explicit force feedback or closed-loop contact correction cannot yet provide.Source: §4.1 · p. 8

5 Conclusion

• Current imitation-learning and vision-language-action policies remain far from general dexterous manipulation, especially in precise contact, bimanual coordination, functional tool use, and complex-object interaction.DexVerse is presented as a challenging, extensible testbed for studying contact-rich control, visuomotor robustness, and embodiment-aware learning.Source: §5 · p. 8
• DexVerse unifies diverse task categories, multiple arm-hand embodiments, configurable visual variation, VR teleoperation, multimodal demonstrations, and representative policy evaluation.The conclusion characterizes this release as a modular benchmark for multi-task, multi-embodiment dexterous manipulation.Source: §5 · p. 8
• The current release prioritizes a broad, reproducible, multi-task, multi-embodiment benchmark; real-robot transfer, more demonstrations, and broader cross-task and cross-embodiment standardization remain future directions.The authors identify these extensions as ways to strengthen DexVerse as a foundation for developing general, robust, and transferable policies.Source: §5 · p. 8

A Task List and Visualization

• Articulated-object tasks include opening doors, drawers, lids, and appliances, rotating knobs, and manipulating hinged objects.The listed success conditions use joint travel or target states, including coordinated opening of both cabinet doors.Source: Table 4 · p. 15, A Task List and Visualization · p. 16
• Object and tool tasks include grasping and relocation, pouring postures, hammering, drilling, and manipulating scissors or a utility knife.Several objectives require avoiding contact with functional regions such as a tool bit, nozzle, rim, or cooking surface.Source: Table 4 · p. 15, A Task List and Visualization · pp. 16–17
• Precision-interaction tasks involve aligning or inserting objects: threading a nut onto a fixed bolt, guiding a pipette tip into target glassware, inserting a charger plug, and seating a gear.For the plug, success requires less than 0.0025 m of lateral and vertical misalignment.Source: A Task List and Visualization · p. 18
• Non-prehensile tasks include pushing objects toward target poses or regions and using a wall to pivot a large cuboid upright.Examples include pushing a sphere up a slope and aligning a T-shaped object with its goal footprint.Source: Table 4 · p. 15, A Task List and Visualization · p. 18
• Bimanual and long-horizon tasks range from carrying objects with two hands to transferring a cube between hands and completing cooking, appliance, or sorting sequences.The listed sequences combine ordered actions with final conditions such as keeping ingredients contained or closing receptacles.Source: Table 4 · p. 15, A Task List and Visualization · p. 19

Multi-goal composite tasks (multigraspenvs)

• The composite set includes paired lift tasks with additional requirements such as keeping a tray, basket, or Dutch oven level, or maintaining a firm left-hand grip while the right hand releases a cube.These criteria extend the lift objective with pose or hand-contact conditions.Source: Multi-goal composite tasks… · p. 19
• Longer composites combine ordered or simultaneous objectives, including pouring ingredients before positioning a pot and turning on a stove, or retrieving food and operating a microwave.Other listed sequences include seasoning and oven operation, and pouring milk before placing a mug and activating a machine.Source: Multi-goal composite tasks… · p. 19
• 39 multigraspenvs variants pair nine articulated-object tasks with four rigid-object add-ons, with PourMug also serving as a primary task for three additional pairings.Each variant places the primary-task asset and addon object in the same scene, requiring interaction with both.Source: Multi-goal composite tasks… · pp. 19–20
• A variant succeeds only when the addon object reaches its lift threshold and the primary task’s articulated joint reaches its standalone success threshold.Both conditions must hold at the same time; the addon threshold is fixed across variants, while the primary threshold is task-specific.Source: Multi-goal composite tasks… · p. 20

B Imitation Learning Baseline & Implementation Details

• π0.5 uses RGB views and task language, predicts 10-step chunks of absolute joint targets, and is fully fine-tuned from its 3.3B-parameter checkpoint.Its proprioceptive-state input is disabled; bimanual observations use third-person and left- and right-wrist cameras.Source: B Imitation Learning Baseline &… · p. 20
• OpenVLA combines RGB views, task language, and joint positions, using a continuous regression head to predict 8-step chunks of absolute joint targets.LoRA adapters are trained while the base weights remain frozen; the regression head and proprioceptive projector are trained from scratch.Source: B Imitation Learning Baseline &… · p. 20
• Diffusion Policy trains a separate state-conditioned U-Net per task without visual input, predicting 16-step action chunks and replanning after each chunk.The implementation uses two observation steps and takes 20 denoising steps at inference.Source: B Imitation Learning Baseline &… · p. 20
• 3D Diffusion Policy conditions each per-task policy on point clouds and proprioception, using a U-Net denoiser and replanning after executing the first eight steps of each 16-step chunk.The point cloud is downsampled to 512 points; bimanual tasks fuse two views.Source: B Imitation Learning Baseline &… · pp. 20–21

• The proprio group contains robot joint positions, while contact contains per-fingertip 3D force vectors when task sensors are enabled.Source: C Observation Modes · p. 21
• State holds task-specific object and articulated-object poses, derived functional geometry, and long-horizon stage progress; goal holds commanded targets when tasks define them.Source: C Observation Modes · p. 21
• RGB and depth provide per-camera images, pointcloud provides single-camera or optionally merged multiview data, and privileged holds simulation-only quantities withheld from the policy.Source: C Observation Modes · p. 21
• Task configurations select observation groups and terms, giving policies a unified interface across state-based, image-based, point-cloud-based, and privileged observations.Source: C Observation Modes · p. 21

• Observation-mode presets enable fixed sets of groups and disable every other managed group.Source: Observation-mode presets · p. 21
• The privileged group is never enabled by a preset because its information is hardly observable in the real world and should be enabled only intentionally.Source: Observation-mode presets · p. 22
• Table 5 lists the groups enabled by each preset and its history length.Source: Table 5 · p. 22