Source-linked AI summary
Knowledge-Guided Hierarchical Policy Learning for High-Precision Cylindrical Assembly under Tight Tolerances
Binbin Lian, Xinyu Liu, Tao Sun
TL;DR
Large cylindrical components with 0.1mm tolerances are difficult to assemble robustly under uncertain conditions. The paper proposes a hierarchical framework combining expert-guided BC, TD3, LSTM-based heuristic guidance, and simulation-to-real transfer. The method converges within 500 episodes, transfers with a 0.9 success rate, and produces smoother real-world motion, while several physical and sensing uncertainties remain insufficiently tested.
Problem
Large components exceeding Φ150mm with 0.1mm tolerances and uncertain environments remain challenging and underexplored for robotic assembly.
Method
The framework combines BC and TD3 in a lower-level network with LSTM-based heuristic guidance, simulated training, and real-world policy transfer.
Results
0.9 success rate is maintained after direct policy transfer, with only slight performance degradation.
Takeaways & Limitations
The framework supports efficient and safe learning for tight-tolerance cylindrical assembly and yields smoother real-world trajectories.
Takeaways & Limitations
Robustness to physical parameter changes, actuator precision, real noisy sensors, and visual disturbances remains unexplored, and vision-error effects lack quantitative analysis.
Abstract
from arXiv · showhide
A hybrid hierarchical learning framework is proposed to achieve high-precision assembly of 170mm cylindrical components with tolerance of 0.1mm. The lower-level network integrates expert experience through Behavior Cloning (BC), giving the robot human-like intuition, and incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm to enhance training stability and robustness. The upper-level network dynamically adjusts the lower-level decisions based on heuristic rules, ensuring flexibility in operations. A simulated model is constructed to learn before transferring to real world. An efficient and safe training is allowed. Comparisons show that the reward curve converges within 500 episodes, indicating high learning efficiency. It also demonstrates better adaptability to initial conditions and pose errors, achieving satisfactory success rates even under extreme conditions. Moreover, the method exhibits good stability under Gaussian noise interference. In the real world, the assembly trajectory of the cylindrical segment shows smoother motion and less fluctuation.
1. Introduction
The paper addresses high-precision assembly of large cylindrical components, where tight tolerances and uncertain environments challenge conventional and reinforcement-learning methods. It proposes a hybrid hierarchical framework combining expert-guided behavior cloning, TD3, heuristic guidance, and simulation-to-real transfer.
- Research motivation: RL offers model-free adaptability but can suffer from slow convergence and inefficient exploration, motivating combinations with imitation learning.Imitation learning transfers useful information from human demonstrations into knowledge models.
- Research motivation: Components exceeding Φ150mm with 0.1mm assembly tolerance and multiple connecting holes remain challenging and underexplored for RL-based assembly.Existing studies largely focus on small-scale peg-in-hole tasks.
- Proposed framework: The lower-level network embeds expert heuristics through BC, then uses TD3 to learn docking skills with greater stability and efficiency.Reward penalties constrain motion limits and collisions to improve safety and learning efficiency.
- Proposed framework: The upper-level LSTM selects lower-level behavior and controls execution step size using heuristic rules derived from expert data.This supports fast approach movements and fine adjustment during assembly.
- Simulation and transfer: A near-real simulation model enables learning before direct transfer to the real robot, supporting efficient and safe training.The paper records the framework’s organization across simulation, transfer, and real-world evaluation.
2. Preliminary
The assembly system uses a binocular-camera coordinate framework, a six-degree-of-freedom parallel robot, kinematic modeling, expert demonstrations, and synchronous collision detection. These components provide the geometric state representation, control model, training data, and safety mechanisms for cylindrical assembly.
- Cylinder segment assembly: The task assembles a Φ169.90 mm shaft with a Φ170.00 mm hole, leaving a 0.1 mm gap and requiring alignment of eight connecting holes.Segment I is carried by the six-degree-of-freedom parallel robot toward Segment II.
- Assembly system coordinate system: The binocular camera and encoded fiducial points parameterize spatial relationships among the world, fixtures, segments, and moving platform.Frame {W} is the camera-based base frame, while fixture and component frames capture assembly geometry.
- Knowledge-guided expert acquisition: Expert data consists of recorded frame positions and orientations during human–robot collaborative assembly.The operator guides Segment I into Segment II while the binocular camera captures demonstrations for behavior-cloning learning.
- Parallel robot system: The deployed robot is a 6-DoF parallel mechanism with a 6-SPS topology, using spherical joints and actuated prismatic joints.Its closed-loop kinematics relate platform, joint, and link vectors through rotational transformations.
- Kinematic model: Forward kinematics is formulated as six nonlinear equations and solved with Newton-Raphson iteration for the platform pose.The pose is represented as X = [x y z α β γ]^T, and iterations stop when the change is within ε.
- Collision detection mechanism: Collision detection constructs oriented bounding boxes and applies the separating-plane criterion to assess whether segment geometries overlap.The method tests 15 candidate normal vectors, including axial, radial, and cross-product directions.
- Collision detection mechanism: Collision checking runs synchronously with motion, canceling unsafe actions and regenerating policy actions; repeated warnings or collisions trigger failure or emergency stopping.More than 20 consecutive warnings fail an episode, while the robot’s protection system can stop the motors.
3.1. Hybrid hierarchical learning framework
The framework uses a high-level LSTM to select adjustment modes and step sizes, while a low-level BC-TD3 network learns and optimizes pose-adjusting actions for assembly control.
- Framework structure: The assembly process is divided into high-level and low-level networks combining supervised, imitation, and reinforcement learning.The hierarchical structure is intended to support efficient and precise task execution.
- High-level network: Position and orientation weights are normalized with softmax to select the corresponding adjustment sub-network.Assembly typically begins with orientation adjustment before precise positioning.
- High-level network: Step-size weights support coarse tuning initially and fine-tuning during the final assembly stage.The adjustment limits are changed dynamically to accommodate varying initial conditions.
- Low-level network: The low-level BC-TD3 network uses expert demonstrations to initialize a policy before TD3 optimizes pose-adjusting actions.BC supplies the initial strategy, while TD3 further improves execution.
- High-level network: The high-level LSTM generates position, orientation, and step-size weights to guide the low-level network.It retains relevant sequence information through gating mechanisms while dynamically adapting decisions.
3.2. Imitation learning
Behavior Cloning learns an assembly policy from expert state-action demonstrations, using visual pose information and position-orientation actions to reproduce expert decisions.
- Behavior cloning: Behavior Cloning treats expert actions as labels and learns a policy that mimics expert decision-making.The approach distills domain knowledge into the agent’s policy.
- Training data: The expert training set contains N state-action pairs collected from expert trajectories.Each pair associates a state si with the expert decision at that state.
- Policy optimization: The BC loss measures differences between the agent’s decisions and expert decisions across the training states.Gradient descent updates policy parameters toward expert-level decision-making.
- State and action spaces: The state space encodes visual points and the relative position and orientation between the two assembly segments.Camera measurements and transformation matrices represent the system configuration.
- State and action spaces: The action space separates translational position adjustments from rotational orientation adjustments.Both action types describe motions of the assembly frame.
3.3. Reinforcement learning
The reinforcement-learning component uses TD3 for stable continuous-control optimization and designs global and local rewards to guide accurate, safe assembly behavior.
- TD3: TD3 improves continuous-control stability with dual critics, delayed updates, and target policy smoothing.These mechanisms address Q-value overestimation and sensitivity to noise associated with DDPG.
- TD3: TD3 contains two critic networks and one actor network that estimate returns and select actions through policy optimization.Critics minimize value-function loss while the actor follows the policy gradient toward higher reward.
- TD3: The actor is updated more slowly than the critics to reduce value-update variance and avoid policy divergence.This delayed actor update supports a more robust policy.
- TD3: Target actions receive clipped Gaussian noise during target updates to smooth Q-value estimates and improve training stability.The noise is limited to the interval [-c, c].
- Reinforcement-learning formulation: The assembly task is modeled as a Markov Decision Process in which actions produce rewards and transition the system between states.The policy is improved by maximizing discounted return.
- Reward design: The global reward assigns 50 for successful assembly and -50 for failure, motion-limit violations, or component collisions.The terminal reward is awarded once per training round.
- Reward design: A -0.01 penalty is added at each step to encourage efficient exploration and shorter assembly paths.Separate local rewards are defined for position and orientation control.
- Reward design: The final reward combines one global episode reward with accumulated local position, orientation, and error rewards.The weighted components balance overall task success against local assembly accuracy and behavior constraints.
4. Simulation
The simulation integrates pose determination, policy-based control, state updates, and reward feedback to train and evaluate the hierarchical assembly framework. Results show faster learning, benefits from staged rotation-translation control and dynamic step sizing, reduced adaptability under larger errors, and robustness to Gaussian noise.
- Simulation model: The simulation model updates encoded-point positions, robot kinematics, and rewards from policy-generated actions to support iterative control learning.It determines poses from encoded points, executes actions through kinematic calculations, and feeds rewards back to the control network.
- Training setup: 10000 epochs are used for both LSTM and BC training, with Adam for LSTM and TD3 networks and SGD for BC.
- Learning efficiency: Within 500 episodes, the hybrid algorithm achieves successful assembly, whereas traditional single-agent methods require over 1000 episodes to reach a similar reward level.The hybrid reward curve rises faster and fluctuates less during training than TD3, NAF, and DDPG comparisons.
- Learning strategy: Separating rotation and translation into staged 3-DoF models improves learning efficiency compared with an unsplit 6-DoF model.The 6-DoF model does not achieve successful assembly within the maximum episode limit.
- Step adaptation: Dynamic step adaptation uses larger movements when far from the target and smaller adjustments near it, improving long-distance efficiency while maintaining accuracy.The reported relationship between initial distance and required steps is inverse exponential, while step length decreases as step count increases.
5. Experiments and analysis
Simulation-to-reality transfer aligns encoded-point coordinates and preserves the learned rotate-then-translate assembly sequence. Real-world comparisons report smoother trajectories, faster stabilization, lower performance degradation after transfer, and high success on a different cylindrical object.
- Simulation-to-reality transfer: Coordinate mapping aligns encoded-point frames between simulation and reality, enabling identical initial poses and comparable assembly motions.Both settings follow rotation until orientation error O equals zero, then translation until hole offset P equals zero.
- Real-world trajectories: The proposed method produces smoother real-world trajectories with less fluctuation than the methods in and.The comparison uses motion trajectories recorded during assembly experiments.
- Pose and position control: The proposed method stabilizes more quickly, maintains smaller pose and position errors, and requires significantly fewer adjustment steps than the compared algorithms.These results indicate fewer unnecessary adjustments during final-stage assembly.
- Comparative performance: Across 1000 simulation runs and 20 real-world trials, the proposed method outperforms the compared learning methods in steps, reward, and overall performance.The method also shows the lowest performance degradation between simulation and reality.
- Generalization: Direct policy transfer to a Φ200mm object with 0.15mm tolerance retains a success rate of 0.9 with only slight performance degradation.The transferred policy is evaluated on a cylindrical segment with 12 circumferential connecting holes.
6. Discussion
The framework performs well for cylindrical segment assembly but has important unresolved robustness, safety, energy-efficiency, scalability, and generalizability boundaries. Future work targets broader uncertainty testing, improved contact handling, quantitative energy evaluation, and expanded applicability.
- The framework achieves high efficiency, high accuracy, and better adaptability in simulations and real-world scenarios.
- Robustness to physical parameter perturbations, actuator-precision changes, noisy sensors, and real-world visual disturbances remains unexplored.The simulated Gaussian noise does not fully represent real non-Gaussian errors, and the relationship between vision pose-estimation error and assembly success lacks quantitative analysis.
- Bounding-box-only detection can cause misjudgments and collision-triggered emergency stops under tight-tolerance conditions.The system lacks force sensing and contact handling, reducing safety and reliability in real industrial scenarios.
- The evaluation lacks specific energy-efficiency metrics despite smoother motion profiles.Future evaluations are planned to measure energy consumption during learning and assembly.
7. Conclusion
The proposed hybrid hierarchical framework addresses tight-tolerance cylindrical assembly by decomposing motion and combining expert-guided imitation with reinforcement learning. Simulation-based training supports efficient and safer transfer to real-world operation.
- The framework divides assembly into position and orientation adjustments with dynamic step sizes, reducing high-dimensional task complexity.
- The lower-level network combines Behavior Cloning with TD3 to integrate expert experience and improve training stability and robustness.
- An LSTM-based upper-level network dynamically adjusts lower-level decisions using BC-guided heuristic rules.
- A simulation environment enables learning before real-world transfer for an efficient and safe training process.
CRediT authorship contribution statement
The authors’ contributions span manuscript writing, methodology, software validation, data curation, and project direction.
- Binbin Lian conducted manuscript writing and editing, while Tao Sun directed the project and also contributed to writing and editing.
- Xinyu Liu carried out the methodology, software validation, and data curation.