ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes

1 Introduction

• Cluttered scenes combine diverse, closely spaced or overlapping objects with occlusions and complex grasp dynamics.Prior work addresses single-object grasp poses or closed-loop tabletop grasping, while clutter-scene pose-generation methods are open-loop.Source: §1 · p. 2
• Real-world imitation learning requires costly human demonstrations, while existing sim-to-real RL methods mainly address simpler scenes and simplified object states.Those states do not capture local geometric details important for dexterous grasping in clutter.Source: §1 · p. 2
• ClutterDexGrasp uses a simulation-trained teacher with clutter-density curriculum learning, a geometry-and-spatial scene representation, and an interaction-aware safety curriculum.Offline imitation learning transfers the teacher to a DP3 student operating on partial point clouds.Source: §1 · p. 2
• The system reports general, dynamic, and safe target-oriented grasping in simulation and zero-shot sim-to-real transfer capability.The authors describe this as the first such demonstration for cluttered-scene dexterous grasping.Source: §1 · p. 2

Dexterous Grasping

• Open-loop clutter methods generate collision-free grasp poses and use motion planning, while closed-loop RL methods largely target single-object scenes.Cluttered interactions and state representations remain challenges for closed-loop control.Source: §2 · p. 3
• Real-world imitation learning for varied clutter requires costly demonstrations, and newer vision-language-action and motion-capture approaches still require substantial human effort.The paper positions simulation RL with specialized representations and curriculum learning as an alternative without real-world demonstrations.Source: §2 · p. 3

Sim-to-Real Transfer with Reinforcement Learning

• Sim-to-real RL faces observation and dynamics shifts; domain randomization, system identification, safety-aware RL, and curriculum learning address parts of this challenge.The method combines these strategies with an interaction-aware safety curriculum that progressively tightens constraints.Source: §2 · p. 3

Diffusion Policy and Teacher-Student Distillation

• Diffusion policies can use point-cloud observations, and teacher-student distillation transfers privileged simulation knowledge to students with limited observations.The paper uses this paradigm to train a point-cloud student from teacher demonstrations across clutter densities.Source: §2 · p. 3

3 Problem Formulation

• The formulation uses reinforcement learning for a teacher policy and imitation learning for a student policy controlling an arm and dexterous hand in clutter.The teacher maps observations to actions; the student learns from teacher demonstrations.Source: §3 · pp. 3–4

State and Action Spaces

• The tabletop robot has a 7-DoF arm and 12-DoF hand, with actions comprising relative arm-joint changes and absolute positions for six actuated hand joints.The hand includes six actuated and six underactuated finger joints.Source: §3 · p. 4

Task Simulation

• Each trial constructs clutter by sampling objects from a prior distribution and then randomly selects one object as the grasping target.Source: §3 · p. 4

Observation

• The student observes robot joint positions and a point cloud, whereas the teacher additionally receives privileged geometry-spatial and state information.The student point cloud is specified as O_pc ∈ R^4×5120.Source: §3 · p. 4

Objective

• The teacher objective maximizes expected discounted reward, while the student objective minimizes negative log-likelihood on teacher demonstrations.The formulation notes that both policies must adapt across objects and layouts.Source: §3 · p. 4

4 Method

• The framework trains a privileged teacher with geometry-spatial representations and clutter-density and safety curricula, then distills demonstrations into a point-cloud student without real-world demonstrations.Source: §4 · pp. 4–5

4.1 Teacher Policy Learning in Simulation

• The teacher is trained in three curriculum stages with PPO, progressively increasing task complexity before fine-tuning under safety constraints.Source: §4.1 · p. 5

4.1.1 Geometry and Spatial Representation

• For each finger link, the geometry-spatial representation measures distances to sampled points on target and non-target objects, encoding local geometry and hand-clutter relationships.This compact representation supports collision-aware learning without computationally expensive point-cloud rendering.Source: §4.1.1 · p. 5

4.1.2 Two-Stage Clutter-Density Curriculum

• The two-stage clutter-density curriculum first learns general single-object grasping, then fine-tunes in clutter to learn contact-rich strategies such as moving obstacles.The paper reports that learning cluttered grasping entirely from scratch is infeasible.Source: §4.1.2 · p. 5

4.1.3 Interaction Safety Curriculum

• The safety curriculum progressively tightens a force threshold after success-rate milestones, penalizing excessive force and terminating those episodes.It is intended to reduce collisions and excessive force during contact-rich interactions.Source: §4.1.3 · p. 5

4.2 Student Point-Cloud Policy Distillation

• The student uses DP3 on demonstrations from varying clutter densities, while point-cloud alignment and system identification address perception and dynamics gaps.Source: §4.2 · p. 6, §4.2.1 · p. 6

4.2.2 Point Cloud Alignment

• Synthetic dense arm and hand point clouds augment observations to address perception gaps and occlusions and provide spatial cues for grasping and collision avoidance.Source: §4.2.2 · p. 6

4.2.3 System Identification for Sim-to-Real Transfer

• System identification iteratively compares simulated and real trajectories under identical commands and tunes arm and hand parameters until behavior matches closely.Source: §4.2.3 · p. 6

5.1 Simulation Experiments

• Simulation evaluation covers sparse, dense, and ultra-dense scenes; the reported setup uses 500 unseen layouts and 550 unseen-object layouts.The success rate counts a trial when the target object is lifted 0.1 meters; the real-world study uses 41 objects across 9 sparse, 5 dense, and 3 extreme scenes.Source: §5.1 · p. 6, Table 1 · p. 7, §5.2 · p. 8

5.1.1 Main Results

• The teacher reached 91.9% success on seen objects and layouts, while the distilled student lost less than 5% average success across seen and unseen combinations.Both policies also generalized to ultra-dense scenes despite training only on sparse and dense clutter.Source: §5.1.1 · p. 7
• The policy learned a two-stage clutter strategy: gently nudge occluding objects, then approach the target from the side once obstacles are partly cleared.These behaviors emerged without heuristic mode identification or switching.Source: §5.1.1 · p. 7, Figure 3 · p. 7

5.1.2 Effectiveness of Density Curriculum and Safety Curriculum

• Direct training in clutter failed at 0% success, whereas initializing from a general grasp policy achieved 87.0% under the same training duration.This comparison evaluates the density curriculum; the learning curve is shown in Fig. 8.Source: §5.1.2 · p. 7
• Without safety training, the teacher had 1.9% higher success but used excessive force; the safety curriculum reduced force and variance while filtering high-risk behaviors.The resulting interactions were more gentle and suitable for sim-to-real transfer.Source: §5.1.2 · pp. 7–8

5.2 Real-world Experiments

• The real-world policy ran at 15 Hz and continued grasping in each scene until three consecutive failures occurred.The study reports success rate and area under the curve to evaluate efficiency.Source: §5.2 · p. 8

5.2.1 Main Results

• 83.9% overall success across 167 real-world attempts on unseen layouts with unseen objects, with no scene stopped after three consecutive failures.Most successful grasps finished within 30 seconds, including in dense clutter; the cumulative success curve reached 80% within 40 seconds.Source: §5.2.1 · p. 8

6 Conclusion

• ClutterDexGrasp trains a general, dynamic, and safe teacher in simulation using geometry- and spatially embedded scene representation and density and safety curricula.The teacher's knowledge is transferred to a point-cloud student using a 3D diffusion policy and sim-to-real strategies.Source: §6 · p. 8
• The authors report strong simulation performance and zero-shot transfer to real-world environments, with generalization across varied objects and layouts.The conclusion describes this as the first system of its kind for target-oriented dexterous grasping in cluttered scenes.Source: §6 · p. 8
• The conclusion presents the framework as a route to robust dexterous grasping in cluttered environments without real-world training.The cited passage describes its approach and experimental findings but does not specify a further practical boundary.Source: §6 · p. 8

7 Limitation

• Tiny-object grasps remain imprecise because of the sim-to-real gap and frequent occlusions by the hand and surrounding objects.The authors identify multi-view fusion or active vision as possible ways to improve visibility and grasp planning.Source: §7 · p. 9

A.1 Geometric and Spatial (GS) Representation

• Success rates are comparable, but the GS representation lowers maximum contact force and yields strategic repositioning, with negligible performance drop entering safety fine-tuning.The GS-based policy repositions occluders from the side and approaches at angles that minimize collision.Source: §A.1 · p. 14

A.2 Clutter-Density Curriculum Learning

• Clutter-only training from scratch remains at zero success, while single-object pretraining followed by clutter fine-tuning enables learning strategic grasping behavior.This two-stage clutter-density curriculum is illustrated by the learning curves in Figure 8.Source: §A.2 · p. 14

B.1 Geometric and Spatial (GS) Representation

• For each finger link, the GS representation encodes 3D distances to sampled target and non-target object surfaces, supporting collision-aware grasping without point-cloud rendering.The implementation samples 200 target points and 50 non-target points, then computes vectors from 11 selected finger-link bases.Source: §B.1 · pp. 15–16

B.2 Teacher Observation Space

• The teacher observation space is detailed by index in Table 4.The supplied passage does not enumerate the table entries.Source: §B.2 · p. 16, Table 4 · p. 16

B.3 Teacher Reward Function

• The teacher reward combines grasping and target-approach rewards with a global penalty discouraging risky proximity to non-target objects.The reward description also defines the target-position and desired-lift terms used by the grasp reward.Source: §B.3 · pp. 16–17

B.4 Safety Curriculum

• The safety curriculum lowers its fingertip-force threshold from 200 to 50 as success improves, with updates spaced by a minimum iteration interval.Training also terminates when a fingertip exceeds the force threshold or penetrates the table surface.Source: §B.4 · p. 17

B.5 Teacher Coarse-to-Fine Data-Efficient Learning

• During approach, hand actions are frozen using privileged distance information; full action control begins once the hand is within 0.08 of the object.The policy models grasping as a coarse approach phase followed by fine interaction.Source: §B.5 · p. 17

B.6 Training Details

• Student-policy training hyperparameters are listed in Table 7.The supplied passage does not give the hyperparameter values.Source: Table 7 · p. 18
• Teacher PPO training used a single GeForce RTX 4090 GPU and an i9-13900K CPU.The paper directs readers to Table 6 for memory and training time and Table 5 for PPO hyperparameters.Source: §B.6 · p. 18

C Implementation Details of Student Policy Distillation

• The teacher policy is offline-distilled into a student that uses robot-joint positions and partial point-cloud observations available in the real world.The student observation is described as 13 robot-joint values and a 4096-dimensional partial point cloud.Source: C Implementation Details of Student… · p. 18

C.1 Student Observation

• The student point cloud combines a masked camera view, synthetic ground points, and a masked synthetic robot cloud.The components are described as 3584 camera points, 512 ground points, and 1024 robot points.Source: §C.1 · p. 19

C.2 Multi-Level Clutter Density Dataset Generation

• The DP3 student is trained by imitation learning on 20,000 successful teacher trajectories from 500 clutter scenes spanning different clutter densities.The demonstrations are evenly distributed across clutter levels, and batches of 120 are sampled for training.Source: §C.2 · p. 19

C.3 Point Cloud Observation Processing

• Point-cloud preprocessing crops to the workspace, downsamples to 3584 points, transforms to the robot frame, and replaces the table with synthetic points; added training noise did not improve real-world performance.The same preprocessing is applied to simulated and real-world data.Source: §C.3 · p. 19

C.4 System Identification

• System identification aligns simulated trajectories with real-world counterparts to calibrate the robot’s physical and dynamic parameters.The authors follow prior approaches cited as [60] [42].Source: §C.4 · p. 19

C.5 Training Details

• Student training hyperparameters are listed in Table 7.The supplied passage does not provide their values.Source: §C.5 · p. 19

D Simulation Environment Generation Details

• Training and evaluation scenes are formed by dropping randomly selected objects and retaining scenes where all objects settle on the tabletop.The simulated robot pairs a RealMan RM75-6F 7-DoF arm with an AgiBot 6-DoF dexterous hand.Source: D Simulation Environment Generation… · p. 20

E.1 Real-World Hardware Setup

• Real-world deployment uses SAM2 segmentation and runs at 15 Hz; torque-based grasp detection, force limits, and slower end-effector motion support execution stability.No safety-related human intervention was observed in the real-world experiments.Source: §E.1 · p. 20

F Failure Case Analysis

• The policy struggles with extreme object shapes and sizes because of hand-morphology limitations, including large and excessively flat objects.The authors describe failures on these object types.Source: F Failure Case Analysis · p. 20
• Some objects fall outside the arm’s working area or are pushed out of range during grasping, resulting in failures.Scene generation was not strictly constrained to the arm’s operational range.Source: F Failure Case Analysis · p. 20

• Student-policy performance improves as trajectory data scale increases across representative Sparse, Dense, and Ultra-Dense scenes.The evaluations use a consistent teacher policy and identical scenes while varying the trajectory data available for distillation.Source: G Data Scaling for Scene-Level… · p. 21
• The scaling evaluation compares student distillation using different amounts of trajectories collected by one teacher policy in identical scenes.Source: G Data Scaling for Scene-Level… · p. 21
• Larger datasets provide richer diversity and broader state-action coverage, which the authors attribute to more effective learning and generalization.Source: G Data Scaling for Scene-Level… · p. 21