PaperlayerView exact revision on arXiv

Source-linked AI summary

Tune to Learn: How Controller Gains Shape Robot Policy Learning

Antonia Bronars, Younghyo Park, Pulkit Agrawal

arXiv:2604.02523v1cs.RO

TL;DR

Controller gains are usually chosen for desired task compliance, but their role changes when they form the interface between a learned policy and the physical system. This paper systematically studies gain learnability across behavior cloning, reinforcement learning, and sim-to-real transfer, finding that each paradigm has a distinct gain preference.

  • Problem

    Controller-gain selection for data-driven manipulation policies lacks principles analogous to classical guidance, despite gains shaping the policy–environment learning interface.

  • Method

    The paper systematically evaluates position-controller gain regimes across behavior cloning, reinforcement learning from scratch, and sim-to-real transfer.

  • Results

    Behavior cloning favors compliant and overdamped gains, reinforcement learning succeeds across gain regimes with compatible hyperparameters, and sim-to-real transfer suffers with stiff and overdamped gains.

  • Takeaways & Limitations

    Optimal controller gains depend on the learning paradigm rather than solely on the desired task behavior.

Abstract

from arXiv · show

Position controllers have become the dominant interface for executing learned manipulation policies. Yet a critical design decision remains understudied: how should we choose controller gains for policy learning? The conventional wisdom is to select gains based on desired task compliance or stiffness. However, this logic breaks down when controllers are paired with state-conditioned policies: effective stiffness emerges from the interplay between learned reactions and control dynamics, not from gains alone. We argue that gain selection should instead be guided by learnability: how amenable different gain settings are to the learning algorithm in use. In this work, we systematically investigate how position controller gains affect three core components of modern robot learning pipelines: behavior cloning, reinforcement learning from scratch, and sim-to-real transfer. Through extensive experiments across multiple tasks and robot embodiments, we find that: (1) behavior cloning benefits from compliant and overdamped gain regimes, (2) reinforcement learning can succeed across all gain regimes given compatible hyperparameter tuning, and (3) sim-to-real transfer is harmed by stiff and overdamped gain regimes. These findings reveal that optimal gain selection depends not on the desired task behavior, but on the learning paradigm employed. Project website: https://younghyopark.me/tune-to-learn

I. INTRODUCTION

Controller gains should be treated as learning-interface parameters rather than direct determinants of task behavior. The paper studies how gain regimes shape learnability across behavior cloning, reinforcement learning, and sim-to-real transfer.

  • I. INTRODUCTION: Position controllers are widely used in robot learning, but gain-selection principles for data-driven policies remain underdeveloped compared with classical control guidance.Classical guidance addresses tracking bandwidth, disturbance rejection, and impedance, whereas the learning setting lacks analogous principles.
  • I. INTRODUCTION: Gains act as an inductive bias that shapes the closed-loop behaviors a policy can easily express and learn.With learned reactive policies, task-level stiffness or compliance can emerge from policy outputs rather than low-level gains alone.
  • I. INTRODUCTION: The study systematically evaluates controller gains across three robot-learning paradigms: behavior cloning, reinforcement learning, and sim-to-real transfer.The investigation asks whether different paradigms prefer different learning interfaces.
  • I. INTRODUCTION: Behavior cloning performs best with compliant and overdamped gains, while reinforcement learning succeeds across gain regimes when hyperparameters are compatible.These findings contrast paradigm-specific preference with RL’s broader tolerance to gain settings.
  • I. INTRODUCTION: Sim-to-real transfer is harmed by stiff and overdamped controller configurations.The result identifies transfer robustness as a distinct gain-selection concern from task-level compliance.

C. Gain Settings in Large-Scale Robot Datasets

The paper examines how controller gains appear in existing datasets and isolates their effects on data collection, behavior cloning, and teleoperation. Its controlled procedures preserve comparable task states while varying gain-induced actions and user-facing dynamics.

  • C. Gain Settings in Large-Scale Robot Datasets: Existing robot datasets exhibit tight command tracking with minimal lag and overshoot, suggesting stiff gains are an implicit data-collection default.The analysis examines DROID and Open X-Embodiment datasets because exact gain values are rarely reported.
  • A. Behavior Cloning: TPR isolates gain effects by preserving nearly identical state trajectories while varying the position-target action distribution across gain settings.The procedure converts high-frequency torque demonstrations into gain-specific position-command datasets and replays them at policy frequency.
  • A. Behavior Cloning: TPR maintains ≥90% success and joint-position MSE <10^-3 across gain configurations up to 25× decimation.At higher decimation, success degrades slightly for contact-rich tasks where trajectory matching is less robust.
  • A. Behavior Cloning: Teleoperation experience depends on the mapping from user inputs to desired position targets, so operators tune mapping parameters separately for each gain setting.The study adjusts α and β during practice to compare each configuration at its best achievable experience.
  • A. Behavior Cloning: Behavior cloning consistently favors compliant and overdamped gains across broad controller grids and manipulation tasks.Higher closed-loop rollout success concentrates in the compliant, overdamped region, whereas stiff or weakly damped controllers perform worse.

B. Reinforcement Learning

The study evaluates reinforcement learning across controller gain regimes by retuning environment and action-space hyperparameters for each setting. It examines solution existence, hyperparameter discoverability, and training efficiency and stability.

  • B. Reinforcement Learning: RL discovers successful policies across all tested gain regimes when environment shaping and hyperparameters are tuned compatibly.The study evaluates each gain configuration at its best achievable performance by optimizing relevant training choices.
  • B. Reinforcement Learning: The RL experiments vary action-space design parameters controlling policy-output scale, absolute versus relative actions, and the reference state for integrating relative commands.Policies are trained with PPO on tasks modified from IsaacLab.
  • B. Reinforcement Learning: Gain settings modulate the hyperparameter optimization landscape, but no regime is consistently easier to optimize across tasks.The study records success over 50 optimization trials per gain setting and notes that observed differences may depend on task or the limited search space.
  • B. Reinforcement Learning: Sample efficiency and training stability are comparable across gain regimes once a successful hyperparameter configuration is identified.The analysis compares reward trajectories across five random seeds for configurations exceeding 95% success during optimization.

C. Sim-to-Real

The sim-to-real study calibrates simulation separately for each gain setting, trains policies in the resulting environments, and evaluates zero-shot transfer on a real Franka Research 3. It compares trajectory errors and examines control-frequency and domain-randomization effects.

  • C. Sim-to-Real: Gain-specific system identification fits simulation parameters to real robot trajectories produced by sinusoidal position targets.The resulting environments are intended to reproduce the closed-loop dynamics of each controller configuration.
  • C. Sim-to-Real: Zero-shot sim-to-real transfer is evaluated by training policies in gain-specific calibrated simulations and deploying them directly on the real robot.The protocol isolates gain effects by avoiding real-world fine-tuning and measures mean squared trajectory error across real rollouts.
  • C. Sim-to-Real: Compliant controllers attenuate action errors, maintaining high success under open-loop action noise while stiff controllers fail under the same perturbations.Compliant gains also keep perturbed trajectories near the original, whereas stiff gains produce large deviations leading to task failure.
  • C. Sim-to-Real: The transfer evaluation includes domain randomization within 10% of identified parameters and policy-frequency ablations at 10, 20, 50, and 100 Hz.The frequency study changes the zero-order-hold duration while retraining policies for each frequency.

A. Behavior Cloning

Behavior cloning favors compliant and overdamped gains because they attenuate action-induced errors during closed-loop execution, while appropriate input mapping preserves teleoperation efficiency. This advantage remains in end-to-end data collection and training.

  • A. Behavior Cloning: Under state-matched demonstrations, compliant and overdamped gains consistently yield significantly better closed-loop policy performance across tasks.TPR holds the state distribution constant while varying the action distribution; formal tests reject the null for all six tasks.
  • A. Behavior Cloning: Higher imitation MSE under compliant and overdamped gains does not predict worse policy performance because those controllers attenuate execution errors.Lower stiffness reduces force from action errors, while higher damping dissipates perturbations faster.
  • A. Behavior Cloning: Under matched action perturbations, compliant and overdamped gains maintain higher open-loop success rates than stiff gains.The experiment injects identical noise into identical action sequences across gain configurations.
  • A. Behavior Cloning: Compliant and overdamped gains provide comparable teleoperation efficiency, success, and operator preference when paired with optimized input mappings.Scaling the mapping compensates for reduced responsiveness, so these gains do not impose a data-collection penalty.
  • A. Behavior Cloning: Compliant and overdamped gains achieve the highest end-to-end behavior-cloning success rate when data collection and policy training use the same gain setting.The result holds despite gain-dependent teleoperation state distributions.

C. Sim-to-Real

Sim-to-real transfer is worse with stiff and overdamped gains, primarily because closed-loop execution produces high-frequency oscillation. Lowering policy frequency reduces this failure mode, while system-identification accuracy alone is not a reliable transfer criterion.

  • System identification achieves its lowest modeling error under stiff and overdamped gains, but those settings have worse closed-loop transfer.This inverse relationship motivates evaluating gains within the full policy loop rather than by modeling error alone.
  • Stiff and overdamped gains produce lower sim-to-real transferability, with high-frequency oscillation as the main failure mode.The stiff-overdamped region has significantly larger trajectory error, and the oscillation persists despite domain randomization.
  • Lowering policy frequency reduces high-frequency oscillation during sim-to-real transfer.The proposed mechanism is that longer intervals between commands let joints settle before subsequent policy reactions.
  • Across the gain grid, jitter failures decrease from 21.8% at 100 Hz to 5.0% at 10 Hz.The sharpest reduction occurs between 50 Hz and 20 Hz.

APPENDIX

The analysis shows that position-controller gains shape how behavior-cloning action errors become state deviations. Although compliant gains can make action targets harder to fit, their attenuation of errors can yield better closed-loop performance.

  • Steady-state position error variance under stochastic action noise is proportional to Kp/Kd.The result is independent of mass, so smaller Kp/Kd minimizes the resulting variance.
  • Action errors enter the dynamics with force proportional to Kp, whereas Kd accelerates perturbation decay between errors.Higher stiffness injects more energy for the same prediction mistake, while damping suppresses the resulting motion.
  • Closed-loop task success depends on state deviation rather than action prediction accuracy alone.Thus, the gain-dependent attenuation factor must be considered alongside the policy’s RMS action error.
  • Compliant gains may increase prediction difficulty, but experiments across six tasks found attenuation dominates that difficulty.This explains why higher training loss can coexist with better closed-loop performance in compliant regimes.

B. Quantitative Stiffness Analysis of Decoupling-Gains Experiment

Learned policies can develop task-level stiffness that differs substantially from actuator stiffness. The quantitative analysis therefore supports separating controller gains from the behavior expressed by the learned policy.

  • Despite 32× lower actuator stiffness, the stiff-behavior policy achieves approximately 5× higher effective task-level stiffness.Effective Cartesian stiffness is measured through force-displacement system identification throughout training.
  • The study evaluates six manipulation tasks using retargeted demonstrations across gain configurations.The tasks include Bimanual Handover, Dishrack Unload, Dishrack Load, Dishwasher Open, Mug Hang, and Block Stack.
  • Across ablations of dataset size, architecture, action chunking, action representation, and control frequency, compliant and overdamped gains remain preferred.This consistency indicates that the observed gain preference is not confined to one nominal training configuration.

4) Scaling Law:

The appendix validates the retargeting and statistical procedures used to compare gain settings, including extensions to task-space control. These checks support controlled comparisons while identifying limits at high decimation and confirming stronger behavior-cloning performance for compliant-overdamped gains.

  • Retargeted trajectories maintain at least 90% success and joint-position MSE below 10^-3 through 25× decimation, or 20 Hz.Beyond 25× decimation, contact-rich task performance degrades because the zero-order-hold assumption becomes less accurate.
  • In task-space control, Kp and Kd are Cartesian gain matrices that separately regulate translational and rotational compliance.TPR extends by recording task-space wrench, pose, and velocity before retargeting to new gains.
  • Statistical analysis rejects the null hypothesis that compliant-overdamped gains do not outperform other regimes across all six behavior-cloning tasks.The analysis uses logistic regression and Bonferroni-corrected one-sided Barnard’s exact tests.
  • The user study evaluates teleoperated box pushing with randomized initial box poses and a fixed goal pose.Users operate a Franka Research 3 with a SpaceMouse, and trials fail on robot faults or pushing outside the workspace.

2) Experimental Design and Results:

The study evaluates gain effects across five manipulation and locomotion tasks using controlled action representations, success-based evaluation, and gain-specific hyperparameter tuning. It also measures user experience and tests whether findings persist across training configurations.

  • User study: User experience is measured after each trial on a 1–5 intuitiveness scale following randomized, blind presentation of gain settings.The study collected 1,297 trials from 12 users during one-hour sessions.
  • Task suite: Five tasks span FR3 reaching, lifting, drawer opening, and Unitree G1 velocity tracking, covering manipulation and locomotion settings.
  • Action representation: The policy maps scaled outputs α ⊙ πθ(s_t) plus a reference position q_ref to desired positions sent to the PD controller.Joint groups share scale factors α1 and α2, which are tuned separately for each gain setting.
  • Optimization and evaluation: Action scales and selected reward weights are optimized with Optuna for each gain configuration, using task success rate as the objective.Policies are evaluated at their highest-reward checkpoint over 100 simulated trials.
  • Training configuration: The experimental protocol includes PPO hyperparameters shared across tasks and additional task-specific settings, with gripper gains held fixed across experiments.

F. Sim2Real

The sim-to-real evaluation identifies actuator parameters per gain setting by matching real and simulated responses, then deploys constraint-aware policies through nested control loops. Simulation uses a lower physics rate than the real controller, creating a stated fidelity gap.

  • Real-world protocol: Real-robot validation applies a sinusoidal desired trajectory for four seconds while logging positions, velocities, and commands at 50 Hz.The low-level torque controller runs at 1 kHz.
  • Simulation fidelity: The simulator updates position commands at 50 Hz and physics at 100 Hz, whereas the real robot’s low-level control loop runs at 1 kHz.The paper identifies this fidelity gap as a contributor to sim-to-real discrepancy.
  • System identification: CMA-ES optimizes per-actuator simulation parameters for each gain configuration to minimize discrepancies between real and simulated response trajectories.The study visualizes identified responses for representative gain settings.
  • System identification: The identification loss combines spectral mean-squared errors for real-versus-simulated joint positions and velocities.Frequency-domain matching is intended to capture oscillatory behavior and damping characteristics.
  • Constraint handling: The outer optimization prioritizes configurations satisfying position, velocity, torque, and torque-rate constraints before comparing success rates.Zero violations are required for position, velocity, and torque; torque-rate violations up to 0.2 are allowed because hardware limits them at 1 kHz.
  • Deployment: Deployments use nested outer policy and inner impedance-control loops, with torque and torque-rate limits enforced before commands reach the robot.The learned policy runs at 50 Hz, while the inner torque loop runs at 1 kHz.

H. Policy Frequency Ablation

The paper analyzes policy-frequency effects alongside gain-dependent behavior cloning and sim-to-real measurements. Across the supplied results, compliant and overdamped gains favor imitation learning, while stiff and overdamped gains worsen transfer errors.

  • Policy frequency ablation: The policy-frequency ablation retrains policies at 10, 20, and 100 Hz while varying the zero-order-hold duration for position commands.The nominal policy frequency is 50 Hz, and system-identification parameters are reused across frequency conditions.
  • Sim2Real analysis: Policy-output differences between simulation and the real robot correlate well with trajectory error, implicating out-of-distribution real states rather than low-level controller instability.
  • Sim2Real results: Higher stiffness and damping are consistently significant predictors of increased sim-to-real trajectory error in log-transformed OLS regression.
  • Sim2Real results: Stiff and overdamped gains produce significantly larger sim-to-real trajectory errors across all three evaluated conditions.The reported hypothesis tests reject the null in every condition after Bonferroni correction.
  • System identification: System-identification comparisons plot commanded, real-world achieved, and simulated positions for representative gain settings in each gain regime.
  • Behavior cloning: Behavior cloning retains a preference for compliant and overdamped gains across dataset sizes, policy architectures, action chunk sizes, action representations, and control frequencies.The pattern is reported for single actions and chunks, absolute and relative joint positions, and 10 Hz and 50 Hz settings.
Loading 2604.02523v1…