PaperlayerView exact revision on arXiv

Source-linked AI summary

From Imitation to Refinement -- Residual RL for Precise Assembly

Lars Ankile, Anthony Simeonov, Idan Shenfeld, Marcel Torne, Pulkit Agrawal

arXiv:2407.16677v4cs.ROcs.LG

TL;DR

Long-horizon assembly demands precise execution, but action-chunked BC can lack closed-loop corrections. ResiP adds an RL-trained residual policy to a frozen chunked BC model; on the one-leg task, it reaches 98% success with 50 demonstrations, an 18-point gain over BC with 100K demonstrations.

  • Problem

    Long-horizon manipulation tasks such as assembly require high-precision control, where small insertion imprecisions can compound into task failure.

    Source: §I · p. 1

  • Method

    ResiP augments a frozen, chunked BC policy with a single-step residual policy trained using on-policy RL to provide per-timestep corrections.

    Source: §I · p. 2, C. Reactive Control via ResiP · p. 3

  • Results

    98% success on the one-leg task with 50 expert demonstrations, an 18 percentage point improvement over BC with 100K demonstrations.

    Source: §I · p. 2

  • Takeaways & Limitations

    ResiP achieves reliable one-leg performance with sparse task-completion reward while using far fewer demonstrations than the cited BC comparison.

    Source: §I · p. 2

    Extending practical online learning to broader tasks with longer horizons and less obvious automatic state resets is left for future work.

    Source: APPENDIX VII EXTENDED LIMITATIONS AND… · p. 26

Abstract

from arXiv · show

Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks. However, we find that the ease of teaching comes at the cost of unreliable performance that saturates with increasing data for tasks requiring precision. The performance saturation can be attributed to two critical factors: (a) distribution shift resulting from the use of offline data and (b) the lack of closed-loop corrective control caused by action chucking (predicting a set of future actions executed open-loop) critical for BC performance. Our key insight is that by predicting action chunks, BC policies function more like trajectory "planners" than closed-loop controllers necessary for reliable execution. To address these challenges, we devise a simple yet effective method, ResiP (Residual for Precise Manipulation), that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities. ResiP augments a frozen, chunked BC model with a fully closed-loop residual policy trained with reinforcement learning (RL) that addresses distribution shifts and introduces closed-loop corrections over open-loop execution of action chunks predicted by the BC trajectory planner. Videos, code, and data: https://residual-assembly.github.io.

I. INTRODUCTION

Precision-critical assembly exposes a limit of chunked behavior cloning: performance can saturate as demonstrations increase, while open-loop execution leaves bottleneck moments without corrective control. ResiP adds a closed-loop RL residual to a frozen BC policy, retaining chunked planning while improving precision.

I. INTRODUCTION

  • BC performance on precision tasks can saturate as data grows; one diffusion-based insertion policy plateaued near 80% with 100,000 demonstrations.The authors attribute saturation to offline-data distribution shift and open-loop action chunks that prevent real-time corrections at critical moments.Source: §I · p. 1
  • ResiP augments a frozen chunked BC policy with a single-step, on-policy RL residual that corrects actions at each control step.The base policy plans at roughly one-second intervals, while the residual corrects at roughly 0.1-second intervals.Source: §I · p. 2
  • 98% success with 50 expert demonstrations, an 18 percentage point improvement over BC with 100K demonstrations, on the one leg task.The result uses a sparse task-completion reward.Source: §I · p. 2

II. METHOD

  • The pipeline trains a BC base in simulation, improves it with RL residual corrections, then distills the behavior into a vision policy using synthetic data and a few real demonstrations.The RL-based pipeline relies on constructing a digital twin before transferring the trained policy to the real world.Source: §II · p. 2, §I · p. 2

A. Problem Formulation

  • The task is modeled as sequential decision-making with end-effector poses and gripper commands as actions, and a sparse reward for achieving a successful relative pose.The reward is 1 at successful assembly and 0 otherwise.Source: A. Problem Formulation · p. 2

B. Base Policy Learning via Behavior Cloning

  • The BC base uses action chunks—multiple future actions per prediction—which reduce long-horizon task length but sacrifice closed-loop corrections.The policy is trained from 50 simulated teleoperated demonstrations per task using Diffusion Policy.Source: B. Base Policy Learning via Behavior… · pp. 2–3

C. Reactive Control via ResiP

ResiP uses a timestep-level residual policy trained with PPO to correct a chunked BC policy, adding reactivity while addressing distribution shift. For sim-to-real transfer, successful teacher trajectories are rendered with visual variation and combined with a small set of real-world demonstrations to train a student policy.

C. Reactive Control via ResiP

  • A PPO-trained residual policy makes per-timestep corrections to a chunked BC policy, providing reactivity intended to help precise manipulation and distribution-shift robustness.The residual observation combines simulation state and the base policy’s predicted action; the resulting policy combines the pretrained base and correction policies.Source: C. Reactive Control via ResiP · p. 3
  • The residual uses orthogonal initialization with small final-layer gain to center initial corrections around zero, assuming base-policy errors are unbiased in expectation.The design targets small corrections around the base policy’s behavior because most task failures stem from slight action imprecision.Source: C. Reactive Control via ResiP · p. 3

D. Sim-to-Real Transfer

  • Successful trajectories from a state-based teacher are re-rendered as sensory-observation data with variations in colors, textures, lighting, and camera perspectives.The rendering introduces visual variation not easily obtained through standard image augmentation or real-world data collection.Source: D. Sim-to-Real Transfer · pp. 3–4, Figure 4 · p. 3
  • The student is trained with about 400 rendered synthetic demonstrations combined with 10–40 real-world demonstrations, using the same BC loss as in simulation.The real demonstrations contain RGB observations but no ground-truth pose information.Source: D. Sim-to-Real Transfer · p. 4

III. EXPERIMENTAL SETUP

The experiments evaluate assembly tasks spanning long horizons and precise, varied manipulation, using sparse completion rewards and a specified robot-control setup. Performance is measured over randomized initial states, with additional evaluations under dynamic perturbations.

A. Tasks and Environment

  • The six tasks include three FurnitureBench assemblies, peg-in-hole, and custom mug-rack and biman-insert tasks; they span long horizons, tight tolerances, bimanual control, and multimodal success criteria.The custom tasks are mug-hanging and precise bimanual insertion.Source: A. Tasks and Environment · p. 4

1) Primary Evaluation:

  • Task rewards are sparse: each completed part assembly earns 1 and otherwise earns 0; some tasks require sequencing grasping, reorientation, insertion, and screwing over up to about 750–1,000 steps.Success is target geometric alignment, and reported rates use 1,024 episodes from the demonstration initial-state distribution and each method’s best checkpoint.Source: A. Tasks and Environment · p. 4, 1) Primary Evaluation · p. 5

B. System Configuration

  • Most tasks run at 10 Hz on one 7-DoF Franka Panda, while biman-insert runs at 50 Hz on two Panda arms totaling 14 DoF.Desired end-effector poses are converted to joint targets with differential inverse kinematics and tracked by a 1 kHz PD controller.Source: B. System Configuration · pp. 4–5

1) Primary Evaluation:

  • Across the task suite, ResiP significantly improves success rates over the base BC policy, unlike the alternative RL methods; DP generally outperforms chunked MLP BC, while unchunked MLPs solve only peg-in-hole.Table I reports success rates over 1,024 evaluation episodes using each method’s best-performing checkpoint.Source: Table I · p. 5

2) Robustness to Dynamic Disturbances:

  • Dynamic-disturbance robustness is evaluated by perturbing a randomly sampled 1% of objects at each timestep and comparing success rates with and without perturbations across 1,024 rollouts.The perturbations use the same randomization as initialization and represent unexpected contact forces and dynamic disturbances not seen during training.Source: 2) Robustness to Dynamic Disturbances · p. 5

3) Real-World Evaluation Protocol:

The experiments compare BC architectures, approaches to distribution shift and RL fine-tuning, and closed-loop versus chunk-level residual corrections. The real-world protocol also compares policies trained on real demonstrations alone against policies trained with rendered synthetic data.

3) Real-World Evaluation Protocol:

  • The real-world protocol tests policies across a grid of initial object and obstacle poses within ±1.5 cm, using 10 trials per method on the same initial object states.Success is target geometric alignment between assembly parts.Source: 3) Real-World Evaluation Protocol · p. 5

1) Behavior Cloning Baselines:

  • Diffusion Policy provides the strongest BC performance and serves as both the primary BC baseline and the foundation for ResiP.The comparison includes standard MLP BC and an MLP variant that predicts action chunks.Source: 1) Behavior Cloning Baselines · p. 5

2) Distribution Shift Analysis:

  • DP-DAgger gathers corrective demonstrations by querying a ResiP expert in states visited by the learned policy, then trains with the nominal DP architecture and BC loss.The approach evaluates whether online expert correction addresses distribution shift.Source: 2) Distribution Shift Analysis · p. 5

3) Reinforcement Learning Comparisons:

  • The RL comparisons include PPO fine-tuning of chunked MLP actions and IDQL, which samples diffusion-policy chunks and selects among them using learned Q-values.IDQL uses the on-policy method of.Source: 3) Reinforcement Learning Comparisons · p. 5

4) Closed-Loop Control Ablation:

  • On the harder round-table task, BC saturates at 56% success, DAgger at 71%, and ResiP reaches 94%.The caption describes the same scaling pattern on one leg, with lower saturation points on the more complex task.Source: Figure 6 · p. 6
  • ResiP-C predicts corrections for an entire action chunk from the current state and all predicted chunk actions, unlike standard ResiP’s per-timestep corrections.ResiP-C uses online PPO but predicts future corrections without intermediate states.Source: 4) Closed-Loop Control Ablation · pp. 5–6

5) Real-World Baselines:

  • Real-Only policies use 10 or 40 real-world demonstrations, whereas Real+Sim combines the same demonstrations with synthetic rendered data.The comparison follows the paper’s sim-to-real approach.Source: 5) Real-World Baselines · p. 6

IV. EXPERIMENTAL RESULTS

Across six assembly tasks, the experiments compare chunked behavior-cloning policies with ResiP, then examine failure modes and design factors behind performance. ResiP improves precision-critical task success, while its local corrections have limits when initial scene randomness is high.

A. Augmenting Trajectory Planning with Reactive Control

  • On round table, DP improves from 12% with 50 demonstrations to 56% with 10,000, while DAgger plateaus at 71%, below expert-level performance.Similar scaling behavior is reported for the one leg task.Source: A. Augmenting Trajectory Planning with… · p. 6
  • Starting from 50 demonstrations, ResiP reaches 94% success on round table and 98% on one leg, versus 12% and 54% for DP, and raises peg-in-hole success from 5% to 99%.The authors report consistent advantages over BC baselines and alternative RL methods across the task suite.Source: A. Augmenting Trajectory Planning with… · p. 6

1) Analyzing Failure Modes:

  • For low-randomization failures, residual corrections address slight grasp-pose shifts by making small sideways adjustments and delaying insertion until alignment is proper.The analysis also attributes downstream alignment improvements to subtle corrections of initial grasps.Source: 1) Analyzing Failure Modes · pp. 6–7
  • On medium-randomness round table trials, ResiP eliminates Peg Precision errors, but out-of-distribution failures increase in share and success saturates at 70%-77%.The authors attribute this boundary to initial poses too far from nominal for local corrections, or to resulting states outside the DP policy’s support.Source: 1) Analyzing Failure Modes · p. 7
  • ResiP also discovers side grasps when the table-foot center is unreachable and uses more decisive recovery adjustments than the nominal BC policy.The demonstrations showed only grasps from above the center.Source: 1) Analyzing Failure Modes · p. 7

B. What drives performance of ResiP?

  • The evaluation examines training stability and sample efficiency, online data collection, closed-loop control, and robustness to dynamic perturbations.These are the four factors identified for analysis in the section.Source: B. What drives performance of ResiP? · p. 7

1) Performance and Training Characteristics of RL Methods:

ResiP trains more stably than the alternative RL methods described and outperforms them in task performance. Experiments also find advantages from per-timestep corrections for learning progress and robustness, while online data collection alone does not close the performance gap.

1) Performance and Training Characteristics of RL Methods:

  • With 50 demonstrations per task, ResiP reaches 98% on one leg, compared with 70% for PPO-C and 57% for IDQL.The authors report similar trends across all tasks.Source: 1) Performance and Training… · pp. 7–8
  • ResiP trains stably, with local corrections and a small residual network limiting large policy updates; this stability supports consistent hyperparameter performance and multiple gradient steps per trajectory.PPO-C exhibits training instability requiring careful KL regularization, while IDQL exploration is limited by insufficient action variance from the base DP model.Source: 1) Performance and Training… · p. 8

2) Impact of Distribution Shift:

3) Impact of Closed-Loop Control:

4) Robustness to Dynamic Perturbations:

1) Real-World Performance:

In real-world one-leg assembly, co-training with simulation data outperforms real-only policies, and color-randomized synthetic data preserves basic functionality under an unseen part color. The paper also identifies distillation and broader initial-pose coverage as remaining challenges.

1) Real-World Performance:

  • On the real-world one-leg task, Real+Sim policies achieve 50-60% success, compared with 20-30% for Real-Only policies.Co-trained policies also show smoother behavior with fewer erratic movements exceeding hardware limits; imprecise alignment before release remains a primary failure mode.Source: 1) Real-World Performance · p. 9
  • With unseen black parts, Real-Only fails on every trial by hitting velocity limits, while Real+Sim retains basic functionality at reduced performance.The synthetic training data included color randomization.Source: 1) Real-World Performance · p. 9

2) Understanding Performance Limitations:

  • Increasing synthetic trajectories from 10k to 100k raises distilled-policy success only from 78% to 80%, which the authors identify as a distillation-process limitation.Direct DP training on the same 50 demonstrations performs similarly with state and vision inputs, whereas distilling the state-based ResiP policy produces a gap.Source: 2) Understanding Performance Limitations · p. 9, Figure 12 · p. 9

A. Training diffusion models with reinforcement learning

  • Diffusion-model RL is complicated by inaccessible final action probabilities during iterative denoising, while action chunking expands the action space; ResiP freezes the base model and trains a small residual.The cited discussion notes that chunks of eight actions increase action dimensionality eightfold.Source: A. Training diffusion models with… · pp. 9–10

B. Residual learning in robotics

  • Unlike prior residual-policy work, ResiP runs its base policy at lower frequency and supplies RL-trained corrections at every control step.For the same FurnitureBench suite, prior work modeled human corrections with supervised learning, while ResiP trains from scratch with simulation task rewards.Source: B. Residual learning in robotics · p. 10

VI. DISCUSSION

  • ResiP relies on base-policy macro behaviors and struggles with very high initial scene randomness and out-of-support part poses.The paper also notes that its imitation-learning scaling analysis used an RL expert dataset, whose distribution may differ from human demonstrations.Source: §VI · p. 10

APPENDIX I IMPLEMENTATION DETAILS

The appendix lists separate training hyperparameters for state-based BC, online RL, and image-based real-world distillation. It also specifies the action and proprioceptive spaces, rotation representation, and normalization procedure.

1) State-based behavior cloning:

2) State-based reinforcement learning:

3) Image-based real-world distillation:

B. Action and State-Space Representations

  • Actions are 10-dimensional absolute end-effector poses plus a gripper command, while proprioception is a 16-dimensional vector of end-effector state and gripper width.Orientations use a 6D representation; action, proprioceptive-state, and part-pose dimensions are independently scaled to [-1, 1] using limits calculated from the BC dataset and reused for RL.Source: B. Action and State-Space… · pp. 16–17

C. Image Augmentation

The appendix describes image augmentations and evaluates the pipeline across furniture, multimodal mug-rack, and high-precision insertion tasks. The peg-in-hole results show a large success-rate increase after fine-tuning.

C. Image Augmentation

  • Training uses color jitter and Gaussian blur on both camera views, with random cropping only on the front view.At inference, the front view is center-cropped and the wrist view resized; crops are resized to 280x240 to retain essential scene features.Source: C. Image Augmentation · p. 17, Figure 13 · p. 17

1) Furniture assembly tasks:

2) Real-to-sim task:

  • The mug-rack task tests multimodality, and the frozen diffusion base preserves its alternative behaviors during residual RL.The base model can sometimes hang the mug on both pegs; the task also demonstrates reward inference from data rather than hand-designed rewards.Source: 2) Real-to-sim task · p. 18, Figure 14 · p. 18

3) High-precision, Factory task:

  • Fine-tuning raises peg-in-hole success from 5% in pre-training to approximately 99%, using the FurnitureBench hyperparameters unchanged.The task has 0.2 mm clearance, 25 times tighter than FurnitureBench; the authors attribute success to locally adjusting peg alignment.Source: 3) High-precision, Factory task · p. 18, Figure 15 · p. 19

4) Bimanual, high-precision task:

The appendix describes a 20-dimensional bimanual insertion task and simulator modifications for controlled randomization and faster training. It also examines real-to-sim action coverage and illustrates how residual corrections address insertion errors.

4) Bimanual, high-precision task:

  • Biman-insert tests precise bimanual assembly with a 20-dimensional action space and tight insertion tolerances, using the same code and hyperparameters as the other tasks.The task runs at 50 Hz and uses a MuJoCo simulation with demonstrations collected through DART teleoperation.Source: Figure 16 · p. 19, Table XIII · p. 20

B. Details on randomization scheme

  • Initial scene variation comes from applying random forces and torques to parts initialized in feasible configurations, while obstacle and part positions are also randomized.The obstacle variation is intended to cover imperfect alignment between simulated and real environments.Source: B. Details on randomization scheme · p. 20

C. Adjustments to FurnitureBench simulation environments

  • FurnitureBench modifications enable approximately 4,000 environment steps per second across 1,024 parallel environments for online RL training.Changes include vectorized operations, disabling camera rendering in state-only mode, and correcting simulation-time stepping.Source: C. Adjustments to FurnitureBench… · p. 20

APPENDIX III RGB SIM2REAL TRANSFER

APPENDIX IV VISUALIZATION OF RESIDUAL POLICY ACTIONS

  • During insertion, the residual counteracts premature downward pushes and redirects the peg toward the hole, correcting a base-policy action that can lead to failure.The visualization distinguishes the base action, residual correction, and combined action.Source: APPENDIX IV VISUALIZATION OF RESIDUAL… · p. 23

APPENDIX V EXTENDED VISION-BASED RESULTS AND ANALYSES

The appendix evaluates distillation, real-world co-training, and appearance variation. Distilled and co-trained policies improve on their respective baselines, while performance gaps and reduced reliability on unseen parts remain.

A. Performance Impact of Distillation

  • A vision student distilled from approximately 1,000 ResiP teacher trajectories reaches 73% one-leg success, versus 50% for direct training on human demonstrations.The student remains below the RL teacher’s 98%; image- and state-based students perform comparably, suggesting modality shift is not the primary source of the gap.Source: A. Performance Impact of Distillation · p. 23
  • State-based student success rises from 78% with 10k trajectories to 80% with 100k, still below the teacher’s 98%.The authors report that the same trend appears in Fig. 6, with saturation occurring earlier.Source: A. Performance Impact of Distillation · p. 23

B. Real-World Evaluation

  • For real-world one-leg trials, incorporating simulation data raises task completion from 20–30% to 50–60%.The remaining common failure is precise alignment before releasing the table leg, consistent with insertion challenges in simulation.Source: B. Real-World Evaluation · p. 24

C. Quantitative Results Failure Mode Breakdown

D. Extension of Pipeline to Unseen Settings

  • With white-only real demonstrations, the baseline behaves erratically on black parts; co-training with 400 color-randomized synthetic demos enables task completion.The co-trained policy is still considerably less reliable than on white parts, indicating a remaining sim-to-real gap.Source: D. Extension of Pipeline to Unseen… · p. 25

APPENDIX VI EXPANDED RELATED WORK

The appendix situates ResiP among assembly learning and BC–RL combinations, then discusses limits of offline transfer, local corrections, and simulation. It also reports comparisons of policy architectures and learning trends.

APPENDIX VI EXPANDED RELATED WORK

  • Unlike concurrent work that supervises policies per skill with point clouds, ResiP bootstraps RL from BC and uses sparse assembly-completion rewards.IndustReal trains RL policies from scratch with shaped rewards and curricula; ResiP also combines residual RL with teacher-student distillation for RGB manipulation.Source: APPENDIX VI EXPANDED RELATED WORK · pp. 25–26

APPENDIX VII EXTENDED LIMITATIONS AND FURTHER WORK

  • Offline teacher-student distillation may limit transferred performance; online learning could improve it, but practical challenges remain for these tasks.The paper identifies real-world online learning for longer-horizon tasks and less obvious automatic resets as future work.Source: APPENDIX VII EXTENDED LIMITATIONS AND… · p. 26
  • Residual RL is bound to its pretrained policy and mainly makes local corrections, limiting generalization to states far from the training set.The frozen prior also stabilizes training and helps prevent collapse.Source: APPENDIX VII EXTENDED LIMITATIONS AND… · pp. 26–27
  • The 0.2 mm peg-in-hole result does not establish real-world transfer; tighter precision may require more accurate physics calibration, and smaller parts challenge vision-based manipulation.Source: APPENDIX VII EXTENDED LIMITATIONS AND… · p. 27

APPENDIX VIII WHY ACTION CHUNKING AND DIFFUSION POLICIES?

  • Chunked MLPs substantially outperform single-action MLPs, which fail across tasks; diffusion policies generally outperform MLPs, especially at intermediate difficulty.On one-leg medium randomness, diffusion policy reaches 26% versus 10% for MLP, while lamp low randomness is an exception where MLP-C performs better.Source: APPENDIX VIII WHY ACTION CHUNKING AND… · p. 27

A. Distillation scaling analysis

B. Interactive distillation with DAgger

The section compares interactive DAgger with BC, tests how closed-loop residuals respond to perturbations, and examines base-policy and residual-scaling choices. Its findings favor one-step residual corrections for sample efficiency and robustness, while showing that ACT can also serve as a residual-learning base.

B. Interactive distillation with DAgger

  • ∼10k gradient steps let DAgger surpass BC trained with ∼100k steps, and around 20k steps let it surpass the best BC distillation runs.At around 20k steps, the aggregated dataset contains ∼1.5k demonstrations; the comparison BC runs used more than 10k rollouts and 500k gradient steps.Source: B. Interactive distillation with DAgger · p. 28

A. Effect of fully versus partially closed-loop policies

  • The chunked residual reaches ∼85% success after about 250 million environment steps, while the one-step residual needs about 75 million.The one-step residual makes predictions every timestep, compared with every eight timesteps for the base model.Source: A. Effect of fully versus partially… · p. 28
  • Under part perturbations throughout an episode, partially open-loop policies drop around 20 percentage points in success, versus 12 points for the one-step residual.The perturbation applies random forces to 1% of parts at each timestep; the table reports success rates with and without perturbations.Source: A. Effect of fully versus partially… · p. 28, Table XIV · p. 29

B. Residual base policy ablation

  • With an MLP base, residual training can collapse at higher randomness; at low randomness, exploration-related performance drops eventually recover toward diffusion-level performance.The MLP has a similar initial success rate to diffusion, but the diffusion-based residual performs better overall in the figure comparison.Source: B. Residual base policy ablation · p. 29, Figure 27 · p. 29
  • ACT performs slightly worse than diffusion during pre-training, but fine-tunes stably and comparably as a residual-learning base.The comparison is on the one-leg task at low randomness.Source: B. Residual base policy ablation · p. 29, Figure 27 · p. 29

C. Residual action scaling parameter ablation

  • Across tested α values, success performance is very similar, though α = 0.2 is slightly better than α = 0.1; activation magnitudes and losses vary, potentially affecting stability in harder settings.The selected α = 0.1 was an intuitive choice intended to bias the residual toward local corrections in the normalized action space.Source: C. Residual action scaling parameter… · pp. 29–30

Related papers

Loading 2407.16677v4…