Source-linked AI summary
Learning Robotic Assembly from CAD
Garrett Thomas, Melissa Chien, Aviv Tamar, Juan Aparicio Ojea, Pieter Abbeel
TL;DR
Robotic assembly requires contact-rich skills, while conventional control and RL struggle with task variation, local minima, and sample demands. The paper uses CAD-derived geometric motion plans to guide RL and adds a trajectory-conditioned neural policy. The resulting approach learns precise, robust controllers, generalizes across object placements, and requires only minutes of robot time in the reported setting.
Problem
Contact-rich assembly is difficult for conventional controllers and RL because task variation, local minima, and high sample complexity impede reusable learning.
Method
The method combines CAD-based geometric motion planning with RL through a motion-plan-tracking cost, initialization procedure, and trajectory-conditioned neural policy.
Results
The approach learns precise assembly controllers despite significant state-estimation noise, generalizes across task configurations, and requires only minutes of robot time to learn a task.
Takeaways & Limitations
CAD geometry can provide useful prior knowledge for efficient and reliable RL-based autonomous assembly in variable manufacturing settings.
Abstract
from arXiv · showhide
In this work, motivated by recent manufacturing trends, we investigate autonomous robotic assembly. Industrial assembly tasks require contact-rich manipulation skills, which are challenging to acquire using classical control and motion planning approaches. Consequently, robot controllers for assembly domains are presently engineered to solve a particular task, and cannot easily handle variations in the product or environment. Reinforcement learning (RL) is a promising approach for autonomously acquiring robot skills that involve contact-rich dynamics. However, RL relies on random exploration for learning a control policy, which requires many robot executions, and often gets trapped in locally suboptimal solutions. Instead, we posit that prior knowledge, when available, can improve RL performance. We exploit the fact that in modern assembly domains, geometric information about the task is readily available via the CAD design files. We propose to leverage this prior knowledge by guiding RL along a geometric motion plan, calculated using the CAD data. We show that our approach effectively improves over traditional control approaches for tracking the motion plan, and can solve assembly tasks that require high precision, even without accurate state estimation. In addition, we propose a neural network architecture that can learn to track the motion plan, and generalize the assembly controller to changes in the object positions.
I. INTRODUCTION
The paper addresses autonomous assembly under high product variability and contact-rich dynamics by combining CAD-derived motion planning with reinforcement learning. This combination guides policy search, supports generalization to task changes, and improves learning efficiency and robustness.
- Manufacturing is moving toward small production volumes and high product variability, motivating autonomous systems that learn new behaviors without explicit programming.
- Contact-rich assembly is difficult because accurate modeling and estimation of contact dynamics are challenging, so real-world systems commonly target repetitive scenarios.
- Motion planning uses geometric and dynamic knowledge to generate trajectories, whereas RL learns reactive policies through interaction without requiring prior dynamics knowledge.
- RL can converge to undesirable local minima, generalize poorly to changed object positions, and require many real-world trials without guiding prior knowledge.
- The proposed method combines CAD-derived geometric motion plans with RL policy search to guide learning for assembly tasks.
- A motion-plan-tracking cost avoids locally optimal solutions, while a neural policy that receives observations and motion plans supports generalization to task modifications.
- Using CAD specifications, the approach learns PR2 assembly controllers that generalize to object-placement changes and succeed where noisy estimation and local minima challenge standard methods.
II. RELATED WORK
The paper distinguishes its CAD-guided RL approach from prior assembly planning, demonstration-based trajectory learning, and dynamics-aware planning. It formulates manipulation around moving grasped objects to goals while allowing contact and friction dynamics to remain unspecified.
- Earlier CAD-based assembly work planned step sequences but relied on task-specific hardware for execution; this paper focuses on learned execution.
- Prior reference-trajectory methods used human demonstrations and modified trajectory parameters, whereas this work obtains trajectories from motion planning and learns the tracking controller.
- The method extends iterative tracking by learning a neural policy that generalizes a controller across different motion plans.
- Its neural representation uses a soft-attention mechanism over the motion plan, similar in spirit to prior reference-demonstration policies.
- The motion-plan-tracking cost is a form of reward shaping that uses prior geometric task information rather than additional human feedback.
- Unlike kinodynamic planning, which requires a known dynamics model, the approach uses geometric planning and handles task dynamics with RL.
- The problem family includes assembly, disassembly, insertion, ejection, and high-friction manipulation, assuming a collision-free motion-planned path but no restrictions on contacts or friction.
- The formulation represents robot configuration, end-effector pose, and torque control over a finite horizon, with dynamics imposing state transitions and loss measuring task-space error.
B. Reinforcement Learning and Guided Policy Search
The paper describes guided policy search as a combination of trajectory optimization and supervised learning. Its motion-planning component supplies collision-free reference paths that can guide the learned controller toward task goals.
- RL solves the manipulation problem through system interaction, while policy search learns parameters of a reactive policy that minimizes task loss.
- Guided policy search uses trajectory optimization and supervised learning as its two main components.
- Trajectory optimization learns local dynamics models and uses iLQG to optimize a predefined loss.
- The formulation can extend to stochastic dynamics and add velocity or time-dependent cost constraints, although the presentation uses a deterministic setting.
- Supervised learning fits a time-independent neural policy to trajectory-optimizer actions, while an additional loss encourages feasibility under that policy.
- Motion planning computes a continuous collision-free path between configurations whose endpoint reaches the task-space goal through forward kinematics.
IV. COMBINING MOTION PLANNING AND POLICY SEARCH
The system combines CAD-informed motion planning with policy search to handle assembly dynamics, execution noise, and task variation. It uses planned trajectories, tracking-based initialization, and a motion-plan-conditioned neural controller.
- Motivation: A ring-insertion example illustrates how random policy search can settle at a local minimum, while motion planning supplies a collision-free geometric trajectory.The undesirable solution moves the ring horizontally until it touches the peg.
- System approach: The proposed approach combines motion-plan-based policy-search costs, tracking-controller initialization, and a neural-network policy conditioned on the motion plan.These components target both difficult contact dynamics and generalization to modified task configurations.
- System approach: The system estimates object poses, computes a reference trajectory with an off-the-shelf planner, and uses that trajectory within policy learning and execution.The described perception setup uses CAD information, robot perception, and April Tags on the PR2.
- Generalization: Repeated training across initial positions supports learning a neural-network controller for varied object placements while retaining a shared goal position.Motion plans and policy-executed trajectories provide training data for the generalized controller.
- Scope: The demonstrated scope focuses on assembling two objects, with one grasped and the other clamped in a fixed position.Extension to grasping and assembling several objects is deferred to future work.
B. A Motion Planning-Based Cost Function
The method shapes reinforcement-learning costs around a CAD-derived motion plan so policy search follows useful trajectories rather than locally optimal ones. The shaped cost is intended to remain aligned with reaching the original goal.
- Cost design: The geometric motion plan guides policy search toward the correct trajectory and away from trajectories that are locally optimal under the original RL cost.The plan is treated as a reference trajectory for reward shaping.
- Reference trajectory: A continuous motion-planning path is discretized into reference points, with trajectory points followed by repeated goal points.The total number of reference points is T = Ttraj + Tfinal.
- Cost design: The reference cost differs from the original task cost, but sufficiently large trajectory and goal segments make reaching the goal lower-cost than failing to reach it.The stated alignment depends on choosing large enough T and Tfinal.
- Evaluation: For iLQG, the motion-plan-based cost performs significantly better than the standard goal-distance cost by steering search away from locally optimal solutions.The comparison is between iLQG using ℓref and iLQG using the standard cost.
C. Warm Starting Policy Search with a Motion Plan
The method warm-starts policy search with a trajectory-tracking controller derived from the known motion plan. This informed initialization reduces the robot interaction needed to learn effective controllers.
- Warm start: Policy search is initialized from a controller that tracks the motion plan instead of from randomly selected policy parameters.A PD controller is given as an example of the tracking controller.
- Warm start: The tracking controller’s parameters can be learned by supervised learning before policy optimization proceeds.This provides an informed policy representation for initialization.
- iLQG initialization: For iLQG, a linear PD controller can directly replace the first iteration because both the iLQG output and PD control can be represented linearly with position and velocity observations.The replacement warm-starts iLQG with a policy that tracks the motion plan.
- Result: Warm-starting significantly reduces iLQG sample complexity, enabling performant controllers with relatively few task rollouts.The result is reported in the experiments.
D. A Trajectory-Tracking NN for Generalizing across Task Configurations
The paper proposes a neural network controller that uses reference trajectories alongside observations to generalize assembly behavior across task configurations. It locally attends to the trajectory while preserving global task information, producing actions through an MLP.
- Generalization objective: The controller is trained to generalize from observed configurations to similar but unseen initial and final object placements.The cost and initialization procedures alone apply to a single task configuration, motivating a learned controller for variation across configurations.
- Network inputs: The neural network accepts the robot state, current timestep, and reference trajectory as inputs to guide action selection.The reference trajectory is centered around the current pose and processed with the state by an MLP.
- Trajectory processing: A local window of trajectory points around timestep t is selected, reducing computation while retaining the nearby motion direction.The experiments use k = 5, selecting points from t−k through t+k after clipping indices to the trajectory bounds.
- Trajectory representation: Centering reference points on the current end-effector position makes the input represent direction toward the path rather than its absolute location.This transformation also normalizes the data during training and simplifies learning.
- Trajectory processing: Soft attention combines candidate directions into an attended direction, while velocity information helps the network determine how far along the trajectory to look.The attended direction is then combined with the current state to compute the action.
- Action selection: The resulting policy computes actions from the current state and a nearby trajectory direction, learning which direction to select.Global information remains encoded in the motion plan, avoiding the generalization issues associated with purely reactive policies.
V. EXPERIMENTS
The experiments evaluate motion-planning and policy-search components on precise assembly tasks using a PR2 robot. The setup includes challenging geometries, tight tolerances, pose estimation, and both fixed-control and learned-policy components.
- Experimental goals: The experiments test whether combining motion planning with policy search improves performance and solves challenging assembly problems.The evaluation targets tasks with tolerances of 1 mm or less and non-trivial motion trajectories.
- Assembly tasks: The task set includes interlinking U-shaped objects, inserting a gear onto a shaft, and inserting a cylindrical peg into a hole.The gear and peg tasks form parts of a full gear assembly designed as a realistic challenge.
- Robot and sensing: Experiments use Willow Garage’s PR2 with AprilTag-based pose estimation from the robot’s narrow stereo camera.The goal position was additionally recorded by manually moving the PR2 arm to the goal.
- Task assumptions: Each task starts with the object already grasped at a manually defined grasping position, excluding the grasping problem from evaluation.This setup isolates the assembly manipulation component.
- Control setup: The policy search is warm-started with a joint-tracking PD controller whose fixed gains are Kp = 50 and Kd = 9.The target joint positions and velocities are determined by the motion plan, and the gain matrix is proportional to the arm-link moments of inertia.
A. Evaluating Policy Search with Motion-Planning Cost
The experiment evaluates motion-planning-based costs within iLQG against a standard cost and MoveIt! for fixed-position assembly. Combining the motion plan with iLQG produced robust task-solving policies despite noisy state estimation, with about 30 minutes of robot interaction per task.
- Experimental setup: The experiment compares a motion-planning-based cost for iLQG against a standard distance-based cost and the MoveIt! motion-plan tracking controller.Each controller receives the same motion plan, and policies are evaluated from fixed easy and hard starting positions.
- Results: Combining the motion plan with iLQG produced robust policies that overcame state-estimation noise and reliably solved the considered tasks.
- Sample complexity: About 30 minutes of robot interaction were required on average for each task to achieve successful task completion.Table II reports the number of policy rollouts required for successful completion.
B. Generalization Across Task Configurations
The proposed network is evaluated on unseen U-shape task configurations by training on ten initial poses and testing on twenty. At test time, each configuration supplies a newly computed reference trajectory, and the network performs equally well on training and test sets.
- Experimental setup: The generalization experiment trains on 10 initial poses and tests on 20 unseen poses for the U-shape task in simulation.The poses are generated by varying feasible object positions and orientations, with training and test subsets visualized separately.
- Training: The network is pre-trained on auxiliary trajectories with random initial and goal positions so it learns to attend to the reference trajectory.This reduces reliance on current-state information tied to a single goal position.
- Evaluation: At test time, a new reference trajectory is computed for each configuration and passed to the policy.A trial succeeds when the end-effector is within 2 mm of the target position.
- Results: The network performs equally well on the training and test sets, indicating generalization to unseen configurations in this deterministic evaluation.Only one rollout is taken for each condition because the system is deterministic.
VI. CONCLUSION
The paper combines CAD-based motion planning with reinforcement learning for contact-rich manipulation and adds a neural policy that tracks motion plans across configurations. The method learns robust, sample-efficient controllers under high precision and state-estimation noise, while future work will use additional CAD information and image inputs.
- Contributions: The method combines CAD-based motion planning with reinforcement learning through a guiding cost function and initialization procedure.A neural network imitates the learned controller for tracking motion plans across task configurations.
- Results: The method learns robust controllers for high-precision tasks despite significant state-estimation noise, while requiring only minutes of robot time.The conclusion states that state-of-the-art methods did not complete the challenging tasks considered.
- Future work: Future work will incorporate additional CAD-file information, including tolerances, process planning, and guidelines, to improve reinforcement learning.