Source-linked AI summary
Assembling Two Parts in One Hand
Liuao Pei, Tianyue Wu, Hui Zhang, Ping Luo, Jie Song
TL;DR
The paper asks how robotic hands can achieve human-like coordination when mating two objects within one hand, without a fixture or second arm. It uses a unified simulation-based reinforcement-learning formulation with relative-pose goals, functional finger rewards, human pose regularization, and observation robustness mechanisms. The same approach transfers from simulation to hardware across Bottle, Syringe, and Marker tasks, while positioning in-hand assembly as a dexterity benchmark.
Problem
In-hand assembly requires coordinated finger interactions to mate two objects within one hand, but existing dexterous manipulation has not fully exploited this capability.
Method
The paper uses simulation-based reinforcement learning with relative object-state goals, function-based finger rewards, human reference-pose regularization, and randomized historical proprioceptive and object observations.
Results
The unified recipe solves Bottle, Syringe, and Marker assembly tasks and transfers from simulation to hardware using a single camera, demonstrating effectiveness under state-estimation noise.
Takeaways & Limitations
In-hand assembly can serve as a benchmark for machine dexterity, and anthropomorphic dexterity involves functional finger specialization and coordinated contact use.
Takeaways & Limitations
The study covers only a fragment of the full workflow, includes three tasks, and relies on artificial supporting force in some experiments until policies engage.
Abstract
from arXiv · showhide
A hallmark of human dexterity is the cooperative use of fingers, where different fingers take on distinct yet coordinated roles to accomplish fine manipu- lation, such as capping a pen with the hand that holds it. We study this finger-level coordination through in-hand assembly: mating two rigid objects within a single dexterous hand, with no second arm and no fixture. We present a reinforcement learning formulation to solve this problem in a unified framework, which is driven by a goal relative pose between the two parts. Finger coordination is shaped by a function-based auxiliary reward and regularized toward a single human reference pose, while domain randomization and a fusion of historical proprioception and object observation confer robustness to occlusion-induced estimation noise. The same recipe solves three different assembly tasks (Bottle, Syringe, and Marker). Trained purely in simulation, the policies transfer zero-shot to hardware with a single camera, demonstrating robustness to state-estimation errors caused by oc- clusion. Our experiments also reveal that in-hand assembly places demands on hand morphology and can serve as a benchmark for modern robotic hand systems. Videos and code are available at https://ltbgbird.github.io/in-hand-assembly-page/.
1 Introduction
The paper studies fixture-free in-hand assembly, where one dexterous hand coordinates multiple fingers to mate two objects. It develops a unified simulation-based reinforcement-learning approach using task structure, human reference poses, and robustness mechanisms for occluded execution.
- In-hand assembly requires one multi-fingered hand to manipulate and mate two objects under articulation constraints, without a second arm or fixture.The setting demands forceful, precise coordination across four or five fingers and spans diverse geometries, mating mechanisms, and target motions.
- The approach formulates in-hand assembly as relative-object-state goal reaching and assigns different fingers roles such as object translocation and compliant support.A function-relevant reward triggers these distinct finger behaviors while preserving shared structure across assembly tasks.
- Reference snapshots from human manipulation constrain exploration, while simulation-based reinforcement learning automatically searches for coordinated assembly solutions.
- The system introduces in-hand assembly as a dexterity benchmark and demonstrates fixture-free assembly with a general-purpose anthropomorphic hand.The reported system integrates simulation-based RL, finger-function rewards, and human kinematic priors, with real-world execution using a single camera and historical proprioception.
2 Related Work
Prior work largely treats fingers as enveloping one object or as independent grippers, while robotic assembly commonly relies on fixtures, parallel-jaw grippers, or multiple arms. This paper instead uses coordinated fingers of a general-purpose anthropomorphic hand for fixture-free two-object assembly.
- Many multi-fingered-hand studies focus on grasping a single object, often reducing the hand to an enveloping grasp achievable by a low-DoF gripper.
- When one hand holds several objects, finger subsets typically enclose separate objects as independent grippers rather than coordinating their interaction.
- Existing systems therefore leave human-like coordination of multiple objects within one hand insufficiently captured.
- Learning-based robotic assembly usually uses a parallel-jaw gripper on an arm, with the other part held by a fixture or second arm.
- The paper addresses this gap with fixture-free assembly through coordinated fingers of a general-purpose anthropomorphic hand, providing a benchmark for intrinsic dexterity.
3 Method
The method formulates in-hand assembly as goal-relative-pose reinforcement learning for one dexterous hand, combining role-specific finger coordination, human-reference initialization, and partially observable control. It uses object observations, proprioception, recurrent policies, and robust deployment-oriented tracking and rewards across three assembly tasks.
- Problem formulation: In-hand assembly mates two rigid bodies with one dexterous hand, without a second arm, fixture, or vise.The hand must drive one object into a task-specific relative pose while both objects remain supported by the same hand.
- Finger coordination: Finger roles are regularized through separate reward components: thumb and index manipulate the held object, while the other three fingers cage and assist the fixed object.The auxiliary reward also constrains drift from a nominal grasp pose under a per-joint mask, using a single-frame human reference.
- Task setup: The three evaluated tasks are Bottle, Syringe, and Marker, differing in geometry, assembly pose, and tolerance.The Bottle task additionally involves relatively large parts and orientation adjustments requiring multiple fingers.
- Observation and policy: The observation combines object centroids, symmetry-axis directions, and 22 measured joint angles; recurrent policies address partial observability without velocities or contact-force signals.With randomized hand tilt, roll and pitch are added to the observation.
- Observation and action spaces: Actions are normalized joint-position-target deltas applied through a stiff implicit PD controller and clipped to joint limits.The controller operates at 15 Hz with α = 0.1 rad, producing smooth absolute joint angles.
- Rewards and deployment: The goal-reaching reward measures relative spatial and angular alignment, while auxiliary terms provide multiscale motivation and finger-organization constraints.Real-world tracking reduces estimated 6-DoF poses to centroid and symmetry-axis observations and rejects depth-inconsistent estimates during occlusion.
4 Results
The experiments evaluate how hand morphology, observation and reward design, estimation noise, and real-world conditions affect in-hand assembly. The results show that human-like hand structure, object-state information, recurrent history, and robust simulation training are important for reliable assembly.
- Hand morphology: Sharpa and Wuji hands perform best, while Allegro lacks the fifth finger needed for deep insertion and XHand often fails insertion and planar alignment.The authors attribute the strongest performance to human-like dimensions, high degrees of freedom, and wide joint ranges.
- Ablation study: Removing vision generally reduces performance, with the Bottle task particularly affected because assembly can require explicit spatial relationships between the objects.Object-state information supports precise relative alignment during assembly.
- Ablation study: LSTM history outperforms fixed-window historical inputs, while removing finger-function or reference-pose rewards changes learning efficiency and converged rollout behavior.The authors relate these effects to partial observability and the many possible behaviors enabled by the system’s operational freedom.
- Estimation noise: Performance is evaluated under zero-mean Gaussian noise and Gaussian noise with a constant per-episode mean shift, modeling random errors and systematic estimation offsets.The latter represents errors such as simulation–reality coordinate-frame misalignment or camera-parameter errors.
- Real-world capabilities: The real-world system uses a single depth camera, with fingers reorienting objects for alignment and the pinky often increasing insertion depth.These behaviors support assembly without a second arm or fixture.
- Robustness and limitations: Policies recover from induced object deviations, but failures arise from unfavorable initial cap poses, obstructed insertion paths, and unstable simulated pinch contacts at some wrist tilts.Rigid-body simulation cannot model planar patch contacts, creating a sim-to-real gap for pinch control.
5 Limitations and Future Work
The study covers only part of the in-hand manipulation workflow and currently includes three tasks. Future work targets broader geometries and assembly types to expand the benchmark.
- Limitations: The current study includes only three tasks, limiting the breadth of the proposed benchmark.
- Future Work: Future extensions will add more general part geometries and assembly types, including threading and friction-fit tasks.The goal is to expand the suite into a systematic benchmark for hardware and in-hand manipulation algorithms.
6 Conclusion
The paper introduces in-hand assembly as a benchmark requiring one dexterous hand to stabilize and assemble objects without external fixtures or a second arm. A reinforcement-learning solution with functionally specialized fingers is validated across three tasks and transfers this coordinated manipulation to real-world settings.
- The proposed reinforcement-learning solution uses lightweight rewards to distinguish finger roles and coordinated contact use.
- The approach is validated across three different real-world assembly tasks.
- The results suggest that anthropomorphic dexterity depends on functional finger specialization and coordinated contact, not merely many degrees of freedom.
A Training Implementation
Training uses simulation-based reinforcement learning with contact-rich physics, randomized conditions, recurrent policies, and reset procedures that generate feasible initial object states.
- Simulation setup: Simulation training uses IsaacSim with signed-distance collisions, contact patches, and Gauss-Seidel contact solving.
- Object initialization: Reset-only wrench perturbations settle objects around reference poses while the hand remains a static collision body, preventing direct teleportation into interpenetrating states.
- Termination: Episodes last 20s for Bottle, 10s for Syringe, and 20s for Marker, with early termination when an object moves 0.5m from the hand.
- Policy representation: The recurrent Gaussian policy uses a two-layer LayerNorm-LSTM with 1024 hidden units followed by a [512, 128, 64] ELU action head.
- RL setup: PPO policies are trained through the Isaac Lab wrapper, with scaled environments and minibatches for randomized hand-tilt variants.
- Domain randomization: Domain randomization models observation, joint-reading, actuation, object, and manipulator-property variation during training.
B Real-World Deployment
Real-world deployment initializes the hand and objects manually, stabilizes control through rate and thermal safeguards, and uses calibrated camera-based object tracking. The deployment pipeline addresses occlusion and changing camera placement through marker calibration and reduced pose representations.
- Initialization: Deployment begins with a fixed wrist, task-specific joint angles, and manually placed objects, while a human holds them until control activates.
- Control: Proprioception is sampled at 500 Hz and decimated to 15 Hz, while actions are clipped, slew-limited to 0.12 rad per step, and interpolated to 60 Hz.
- Control: A tracking-error stall-relax loop keeps motor currents below thermal limits during prolonged contact-rich manipulation.
- Camera calibration: Camera calibration estimates marker-to-camera extrinsics over 10 frames so object poses remain expressed in the simulation task-frame convention.
- Pose tracking: Object tracking uses a 640×480, 30 Hz RGB-D stream and passes centroid and body z-axis information rather than poorly constrained yaw.
C Marker Ablation and Noise Analyses
Marker ablations show that the full policy substantially outperforms a proprioception-only variant, while performance remains above baseline under moderate observation noise but declines as errors grow.
- Marker ablation: 1280.1 reward and 65% success were achieved by the full policy, versus 338.0 and 11% for the proprioception-only policy.Both variants were evaluated over 500 episodes under domain randomization.
- Marker ablation: Removing either the finger-function reward or the reference-pose reward reduces Marker performance.
- Observation noise: Marker performance remains stable under moderate noise but decreases as the noise magnitude increases.
D Noise Model and Physical Occlusion
The study distinguishes simulated observation noise from physical occlusion and measures pose-estimation errors as object visibility changes. Valid estimates are usually close to the unoccluded reference, but severe occlusion can cause large errors or tracking loss.
- Noise models: Gaussian noise models random pose-estimation errors, while constant offsets model systematic calibration errors; neither directly models physical occlusion.
- Physical occlusion: The occlusion analysis varies each stationary object’s visible fraction and measures FoundationPose translation error against an unoccluded reference.
- Physical occlusion: Most valid estimates differ from the unoccluded reference by less than 1.5 cm, but severe occlusion can produce large errors or tracking loss.
- Physical occlusion: Translation errors above 2 cm or tracking loss count as failures, reported separately from valid-estimate error distributions.
- Interpretation: The noise experiments assess policy robustness to pose errors, whereas controlled occlusion characterizes a physical source of those errors.