PaperlayerView exact revision on arXiv

Source-linked AI summary

ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback

Muhammad A. Muttaqien, Tomohiro Motoda, Ryo Hanai, Yukiyasu Domae

arXiv:2608.17323v1cs.ROcs.AI

TL;DR

Imitation-learning manipulation policies can degrade under small execution errors and distribution shifts, while correction often requires costly retraining. ORPA adds feedback-conditioned joint-space residuals at runtime, improving success rates and recovery across precision-sensitive ALOHA tasks over ACT and rule-based IK.

  • Problem

    Imitation-learning manipulation policies remain sensitive to execution errors, environmental variation, and distribution shifts during deployment.

  • Method

    ORPA augments a pretrained policy with feedback-conditioned, context-aware residual corrections directly in joint space without retraining or kinematic conversion.

  • Results

    ORPA consistently achieved higher success rates across evaluated tasks, including Cube Transfer gains from 60.0% with ACT to 92.3% with discrete feedback.

  • Takeaways & Limitations

    ORPA improves robustness and recovery during precision-sensitive manipulation under small perturbations and distribution shifts without modifying the underlying policy.

Abstract

from arXiv · show

Robotic manipulation policies trained via imitation learning, such as Action Chunking with Transformers (ACT), can achieve strong performance under ideal conditions but often remain sensitive to small execution errors and distribution shifts. Correcting these failures typically requires dataset aggregation and full-policy retraining, which is computationally expensive and unsuitable for real-time deployment. In this work, we propose Online Residual Policy Adaptation (ORPA), a framework that enables immediate, feedback-driven correction of robot actions without modifying the underlying policy parameters. ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space, allowing the system to adapt its behavior at runtime. We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.

I. INTRODUCTION

Imitation-learned manipulation policies such as ACT achieve strong performance but remain vulnerable to small execution errors and distribution shifts in precision-sensitive tasks. ORPA addresses this limitation by adding a feedback-conditioned joint-space residual module for immediate runtime correction, improving robustness and recovery on ALOHA tasks over ACT and rule-based inverse kinematics corrections.

  • Motivation: Transformer-based policies such as ACT, combined with low-cost bimanual platforms such as ALOHA, learn long-horizon manipulation behaviors from human demonstrations.These advances have supported scalable robotic manipulation research.
  • Problem: Small deviations in end-effector position, orientation, or timing can cause failures in precision-sensitive tasks, including collisions, unstable grasps, and failed object transfers.Performance can also degrade when objects shift, viewpoints change, or execution conditions differ from training.
  • Problem: Dataset aggregation and policy retraining can improve robustness but require repeated data collection, substantial computation, and modifications to the entire policy network.These limitations make retraining-based pipelines unsuitable for interactive or real-time deployment when failures stem from local execution errors.
  • Method: ORPA preserves pretrained policy parameters while using external feedback to generate temporally consistent corrective residuals directly in joint space during execution.The residual module refines robot behavior without replacing the original policy.
  • Results: ORPA improves task robustness and recovery performance over baseline ACT policies and rule-based inverse kinematics corrections on precision-sensitive ALOHA manipulation tasks.The evaluation includes spatial perturbations and coordination-sensitive bimanual scenarios.

II. RELATED WORK

Prior work spans imitation-learning policies, iterative dataset aggregation, interactive human correction, and rule-based geometric adjustments for improving robotic manipulation. ORPA combines lightweight online adaptation with feedback-conditioned residual correction while preserving the pretrained policy and avoiding full retraining or manually designed geometric rules.

  • Imitation Learning: Imitation learning learns complex robotic manipulation behaviors directly from human demonstrations, while transformer architectures improve modeling of sequential actions and multimodal observations.ACT introduced action chunking strategies for efficient and stable policy learning in bimanual robotic manipulation.
  • Online Adaptation: DAgger and HG-Dagger aggregate corrective demonstrations or samples and iteratively retrain policies to reduce compounding execution errors.Dataset aggregation and policy retraining incorporate new corrective supervision to improve future performance.
  • Interactive Human Correction: OLAF and YAY Robot use interactive human verbal corrections for language-guided supervision and iterative policy refinement in robotic manipulation.The supplied passage also situates these methods among approaches based on interactive corrective learning.
  • Geometric Correction: Rule-based geometric correction maps task errors to end-effector offsets and uses inverse kinematics to produce joint-space actions, but its assumptions limit context-sensitive correction.Appropriate corrections can depend on task phase, object configuration, robot pose, and execution context.
  • ORPA: ORPA combines lightweight online adaptation with feedback-conditioned residual correction, preserving the pretrained manipulation policy without full-policy retraining or manually designed geometric rules.This design targets real-time behavioral refinement while retaining learning-based flexibility.

III. EXPERIMENT SETUP · A. ALOHA Simulation · B. ALOHA Workspace

ORPA was evaluated in both MuJoCo-based simulation and real-world ALOHA environments using synchronized dual-arm manipulation, vision, joint-state information, and precision-sensitive tasks. The setup used 14-dimensional joint-space control at 50 Hz, multi-view RGB observations, and integrated gripper actuation.

  • III. EXPERIMENT SETUP: The experiments used a dual-arm bimanual ALOHA system with vision and joint-state information for coordinated, precision-sensitive manipulation tasks.The framework was evaluated in both simulation and real-world environments.
  • A. ALOHA Simulation: Simulation used a MuJoCo-based ALOHA environment integrated with ACT for synchronized dual-arm manipulation.The simulator models two 6-DOF manipulators with parallel-jaw grippers.
  • A. ALOHA Simulation: 14-dimensional joint-space actions, including gripper actuation, were controlled at 50 Hz in simulation.The simulated system uses two 6-DOF manipulators and supports multi-view RGB observations from four virtual cameras.
  • B. ALOHA Workspace: Real-world experiments used the ALOHA platform, comprising two collaborative manipulators, parallel-jaw grippers, and multi-view RGB vision.The platform is designed for bimanual imitation learning and manipulation research.
  • B. ALOHA Workspace: The real-world platform used two synchronized 6-DOF manipulators with a ROS-based joint-space interface operating at 50 Hz.The combined control space is 14-dimensional, including gripper actuation.
  • B. ALOHA Workspace: ORPA generates online residual joint-space corrections during execution using feedback conditioned on nominal actions, errors, and robot observations.The training pipeline introduces structured perturbations and trains a Feedback Encoder and Policy Updater to predict residual corrections.
  • B. ALOHA Workspace: Each manipulator used a parallel-jaw gripper supporting variable-width pinch and encompassing grasps with real-time actuation control.Gripper actuation is integrated into the policy action representation for coordinated bimanual manipulation.
  • B. ALOHA Workspace: Four synchronized RGB cameras provided multi-view workspace observations, with images resized to 640×480 and processed directly by ACT’s visual encoder.The camera arrangement included two wrist-mounted and two external tabletop cameras, without explicit detection or segmentation modules.

IV. ONLINE RESIDUAL POLICY ADAPTATION · A. ORPA Architecture

ORPA augments a pretrained ACT manipulation controller with feedback-conditioned residual corrections, while keeping ACT frozen during training and deployment. Its architecture combines ACT, a Feedback Encoder, and a Policy Updater to apply local joint-space adjustments online without explicit kinematics.

  • IV. ONLINE RESIDUAL POLICY ADAPTATION: ORPA builds on ACT, which predicts temporally coherent joint-space action chunks from multimodal observations including multi-view RGB images and robot joint states.Predicting action sequences rather than individual low-level commands enables smooth and stable execution of long-horizon manipulation.
  • IV. ONLINE RESIDUAL POLICY ADAPTATION: ACT remains a frozen base policy throughout ORPA training and deployment, generating the nominal future action chunk from the current observation.ORPA refines ACT actions online rather than modifying the underlying policy.
  • IV. ONLINE RESIDUAL POLICY ADAPTATION: ORPA introduces a feedback-conditioned residual adaptation module to address execution-error degradation without retraining the underlying policy.This design is intended for real-time correction.
  • A. ORPA Architecture: The ORPA architecture consists of a pretrained ACT policy, a Feedback Encoder, and a Policy Updater module.During execution, ACT produces a nominal action chunk while human corrective feedback is encoded into a compact latent representation.
  • A. ORPA Architecture: The encoded corrective feedback is combined with the current robot state and visual observations to condition online residual adaptation.The architecture uses external corrective feedback provided by a human operator during execution.
  • A. ORPA Architecture: ORPA adds a feedback-conditioned residual correction ∆a_t to the nominal ACT prediction, preserving ACT’s learned behavior while enabling local adjustments.The residual formulation separates nominal manipulation behavior from corrective action.
  • A. ORPA Architecture: Because ORPA operates directly in joint space, it applies corrective actions without explicit forward or inverse kinematics computations.This joint-space formulation supports direct runtime correction of ACT actions.

B. Feedback Encoder and Policy Updater · C. Synthetic Data Generation

ORPA uses a feedback encoder and policy updater to produce context-dependent residual corrections from human commands, observations, and ACT predictions. It also generates corrective supervision by perturbing successful demonstrations and assigning directional feedback labels without additional teleoperation demonstrations.

  • B. Feedback Encoder and Policy Updater: ORPA introduces lightweight transformer-based Feedback Encoder and Policy Updater components for online corrective adaptation.The Feedback Encoder converts human corrective feedback into a latent representation for policy adaptation.
  • B. Feedback Encoder and Policy Updater: Directional commands such as too left, too right, too high, and good are represented as discrete feedback tokens before encoding.The encoded representation captures the semantic meaning of the corrective instruction.
  • B. Feedback Encoder and Policy Updater: The Policy Updater combines encoded feedback with the current observation and ACT prediction to predict a residual action correction.This conditioning uses the manipulation context represented by the current robot observation and associated action trajectory.
  • B. Feedback Encoder and Policy Updater: Context-dependent corrections vary in magnitude and direction with task stage, object configuration, and robot state, unlike identical rule-based inverse kinematics corrections.ORPA conditions residual actions on both feedback and execution context.
  • C. Synthetic Data Generation: ORPA generates corrective training samples automatically from successful teleoperated demonstration trajectories after training the ACT policy.Controlled perturbations modify object positions and orientations in the manipulation environment.
  • C. Synthetic Data Generation: Each perturbation receives a directional feedback label, such as too left for leftward displacement or too high for upward displacement.The resulting samples contain perturbed observations, corrective feedback labels, and reference actions from successful demonstrations.
  • C. Synthetic Data Generation: The residual correction target is computed as the difference between the reference action from the successful trajectory and the action associated with the perturbed execution.This procedure provides corrective supervision from paired successful and perturbed actions.
  • C. Synthetic Data Generation: This synthetic-data procedure enables scalable corrective supervision without requiring additional teleoperation demonstrations.Corrective samples are generated from controlled perturbations of successful demonstrations.

D. Training Objective

ORPA trains a feedback-conditioned residual adaptation module while keeping the pretrained ACT policy fixed. The module predicts joint-space corrections from feedback and execution context, using mean-squared error against successful demonstration actions, while preserving the original policy parameters with minimal computational overhead.

  • D. Training Objective: The pretrained ACT policy remains fixed and first predicts a nominal action chunk from the current observation.ORPA optimization does not modify the underlying ACT policy.
  • D. Training Objective: The Feedback Encoder encodes the feedback signal, and the Policy Updater predicts a residual correction conditioned on feedback, observations, and the nominal action chunk.The corrected action combines the nominal ACT chunk with the predicted residual.
  • D. Training Objective: The residual adaptation module is optimized with a mean-squared error objective between the corrected action and the reference action from a successful demonstration.The reference action is obtained from the original successful demonstration.
  • D. Training Objective: Only the Feedback Encoder and Policy Updater are optimized, preserving the original ACT policy parameters while introducing minimal computational overhead.This optimization scope defines the lightweight training procedure summarized in Algorithm 2.

V. EXPERIMENTS AND ANALYSIS · A. Tasks · B. ACT Hyper-parameters Tuning

ORPA was evaluated on precision-sensitive bimanual manipulation tasks with randomized object-pose perturbations and online corrective feedback. The ACT policy used chunked joint-space prediction with specified control, observation, and tuning procedures, while detailed ORPA settings were summarized separately.

  • A. Tasks: ORPA was evaluated on precision pick-and-place, cluttered grasping, constrained placement, and coordinated bimanual manipulation tasks using the ALOHA platform.These tasks were selected because small spatial or temporal deviations can significantly affect success.
  • A. Tasks: Synthetic training failures were generated by randomly initializing objects within workspace boundaries and perturbing successful demonstration target poses.The resulting scenarios included objects positioned “too left” and “too far.”
  • A. Tasks: Object poses were randomly perturbed relative to demonstrations to simulate distribution shifts and execution errors during evaluation.The perturbations targeted realistic deviations from reference trajectories.
  • A. Tasks: Online corrective feedback included directional signals such as “too left,” “too right,” and “too high.”These feedback signals were provided during task execution.
  • B. ACT Hyper-parameters Tuning: The ACT policy used chunked joint-space action prediction with a Transformer-based architecture, 100-step action chunks, and a 50 Hz policy update frequency.Multi-view RGB observations at 640 × 480 resolution were encoded with convolutional visual backbones before proprioceptive fusion.
  • B. ACT Hyper-parameters Tuning: ACT hyperparameters were selected using validation success rate and trajectory stability during manipulation execution.The detailed settings were summarized in Table I as hyperparameters used for ORPA training.

C. Experiment Results · D. Comparison with OLAF · VI. REAL-WORLD VALIDATION

ORPA improves manipulation robustness under small execution perturbations and distribution shifts, outperforming baseline ACT and rule-based IK corrections across evaluated tasks. It also exceeds OLAF’s 78.5% success rate while avoiding the additional LLM inference latency required for each correction.

  • C. Experiment Results: ORPA achieved higher success rates than baseline ACT and rule-based IK correction methods across all evaluated tasks.The experiments target robustness under small execution perturbations and distribution shifts.
  • C. Experiment Results: ORPA maintained substantial gains over ACT on Bimanual Insertion, a more challenging task requiring precise peg-target alignment and physical contact.Bimanual Insertion is less tolerant of positional, rotational, and timing errors than Cube Transfer.
  • C. Experiment Results: ORPA outperformed rule-based IK by predicting feedback-conditioned residual actions from visual inputs, robot states, and policy outputs.This enables context-dependent correction magnitudes without a forward-kinematics and inverse-kinematics conversion cycle during execution.
  • C. Experiment Results: Rule-based IK applies the same correction regardless of task phase, so a correction helpful during one stage can interfere with later precise-contact or coordination stages.ORPA learns corrective behavior from demonstration trajectories and implicitly captures task-phase information.
  • D. Comparison with OLAF: 78.5% was achieved by OLAF for online correction, below ORPA’s reported performance.OLAF’s real-time deployment is limited by the additional LLM inference required for each correction.

A. Deployment Setup · B. Task Evaluation

ORPA was evaluated on the real-world ALOHA platform across three precision-sensitive manipulation tasks. Task scores used partial and complete completion values, while results emphasized improved correction and robustness under execution failures and observation perturbations.

  • A. Deployment Setup: ORPA was deployed on the real-world ALOHA platform for snack box transfer, bottle cap opening, and object sorting.The tasks require accurate grasping, coordinated motion, and precise object interaction.
  • B. Task Evaluation: Performance was evaluated over 30 trials per task using completion scores of 1.0 for full success and 0.5 for partial completion.Partial completion included outcomes such as successful grasping without successful placement.
  • B. Task Evaluation: Table III compares the original ACT policy with proposed ORPA variants across the three manipulation tasks.The evaluated tasks were snack box transfer, bottle cap opening, and object sorting.
  • B. Task Evaluation: Bottle cap opening was the most challenging task because of contact-rich interactions, precise alignment requirements, and timing sensitivity.These factors resulted in the lowest baseline performance among the evaluated tasks.
  • B. Task Evaluation: Action perturbation alone improved bottle cap opening task scores, demonstrating correction of execution failures during deployment.The passage attributes this improvement to residual policy adaptation.
  • B. Task Evaluation: Qualitative task sequences showed initialization followed by key manipulation subtasks, with green boxes marking subtasks where online adaptation usually occurs.The sequences covered snack box transfer, chip tube opening, and object sorting.
  • B. Task Evaluation: Perturbed observations further enhanced robustness and improved performance across all evaluated tasks.The results indicate that a small amount of additional observation-level correction data can improve robustness while maintaining data efficiency.
  • B. Task Evaluation: Real-world failures included timing mismatches, imperfect contact interactions, and execution inconsistencies, beyond translation and rotation errors.These additional failure modes were reported after discussing the benefits of observation-level correction.

C. Qualitative Results · VII. CONCLUSION

ORPA uses online human feedback to adapt robot actions through residual corrections while preserving the pretrained ACT policy. Qualitative experiments show failure recovery and successful completion across three manipulation tasks, and the conclusion emphasizes lightweight, retraining-free, context-aware adaptation.

  • C. Qualitative Results: ORPA generated residual corrections from feedback such as “too left,” “too right,” and “too high” without modifying the underlying ACT policy.The corrections refined the action trajectory during execution.
  • C. Qualitative Results: Unlike fixed rule-based corrections, ORPA adapted its corrective behavior to the current execution context and task phase.
  • C. Qualitative Results: ORPA preserved ACT’s smooth, temporally consistent behavior while adding corrective adaptation during deployment.
  • C. Qualitative Results: In snack box transfer, ORPA corrected grasping and placement errors caused by object displacement.
  • C. Qualitative Results: In bottle cap opening, ORPA improved manipulation stability during contact-rich interactions and timing-sensitive motions.
  • C. Qualitative Results: The original ACT policy recovered from the too low perturbation, resulting in successful task completion, whereas too left and too far caused failures.
  • VII. CONCLUSION: ORPA is presented as a lightweight framework that enables immediate action-level refinement through a feedback-conditioned residual module without retraining the underlying policy.
Loading 2608.17323v1…