PaperlayerView exact revision on arXiv

Source-linked AI summary

Online Sim-to-Real Adaptation via Closed-Loop System Modeling

Yuhao Huang, Samuel A. Moore, Boyuan Chen

arXiv:2609.28878v1cs.RO

TL;DR

Sim-to-real controllers can remain functional while tracking inaccurately under residual dynamics mismatch. OSRAM meta-trains and adapts a model of the robot-policy closed loop to optimize reference commands without changing the policy, improving prediction and tracking under unseen dynamics in simulation and on hardware.

  • Problem

    Residual mismatch between simulated and real dynamics can leave transferred controllers functional on hardware but tracking commands inaccurately.

    Source: §I · p. 1

  • Method

    OSRAM meta-trains a closed-loop command-response model across randomized simulations, fine-tunes it with limited deployment interaction, and uses it to optimize references without modifying the policy.

    Source: §I · p. 2

  • Results

    Across simulated and physical bipedal velocity tracking and loco-manipulation, OSRAM improves prediction and tracking under unseen dynamics.

    Source: §V · p. 7

  • Takeaways & Limitations

    Adapting reference commands offers an alternative to changing the policy or explicitly identifying the full physical dynamics.

    Source: §I · p. 2, §V · p. 7

    The current focus is relatively low-dimensional command spaces; extending OSRAM to high-dimensional references such as whole-body trajectories remains future work.

    Source: §V · p. 7

Abstract

from arXiv · show

Sim-to-real transfer has made substantial progress, but can still produce controllers that remain stable and functional on hardware while suffering from degraded tracking accuracy due to residual dynamics mismatch. Correcting these errors typically requires identifying the underlying system dynamics, adapting the control policy, or returning to simulation for additional training and finetuning, all of which can require substantial data and computation. We propose OSRAM (Online Sim-to-Real Adaptation via Closed-Loop System Modeling), a framework that instead adapts the reference commands provided to an existing controller. OSRAM treats the deployed robot and its policy as a unified closed-loop dynamical system and learns its task-level command-response behavior directly from tracking observations. A closed-loop dynamics model is meta-trained across randomized dynamics in simulation and rapidly finetuned after deployment using limited real-world interaction. The adapted model is then used to optimize future reference commands while leaving the underlying control policy unchanged. We evaluate OSRAM on bipedal velocity tracking and loco-manipulation in simulation and on hardware. Results show that closed-loop modeling improves prediction and tracking accuracy under unseen dynamics, while online reference adaptation reduces residual sim-to-real tracking errors across different control objectives and hardware configurations. These results demonstrate that adapting the behavior of the robot-policy closed loop provides a practical alternative to finetuning the policy or identifying the full physical dynamics for sim-to-real transfer. More information can be found at http://generalroboticslab.com/OSRAM.

I. INTRODUCTION

A transferred controller can remain functional on hardware yet track commands inaccurately because simulated and real dynamics differ. OSRAM addresses these residual errors by modeling the robot-policy closed loop and adapting reference commands, not the policy.

I. INTRODUCTION

  • Residual dynamics mismatch can degrade tracking accuracy on hardware even when a transferred policy remains functional.Mismatches may reflect modeling inaccuracies, actuator dynamics, sensor noise, latency, or environmental variation.Source: §I · p. 1
  • Domain randomization depends on a suitable training distribution: too little leaves vulnerability to unseen dynamics, while too much can sacrifice task performance or destabilize learning.Post-deployment identification and policy fine-tuning can also demand real-world data, simulator calibration, and additional computation.Source: §I · p. 1
  • OSRAM corrects residual tracking errors by adapting references sent to the existing controller, which can compensate for under-tracking, overshoot, or delay without changing the controller.Source: §I · p. 1
  • OSRAM models the robot and policy together from tracking observations, learning task-level command-response behavior that predicts how candidate references will be executed.Source: §I · pp. 1–2
  • OSRAM meta-trains a closed-loop dynamics model across randomized simulations, fine-tunes it with limited deployment interaction, and optimizes future references for bipedal tracking and loco-manipulation.The deployed policy remains unchanged, and explicit physical-parameter identification is not required.Source: §I · p. 2

II. RELATED WORK

Prior online adaptation estimates latent conditions or modifies controller behavior, while OSRAM adapts through the reference interface. It builds on input-output modeling and reference steering with nonlinear closed-loop modeling meta-trained in simulation and adapted after deployment.

A. Sim-to-Real Transfer with Online Adaptation

  • Online adaptation methods infer latent conditions or modify controller behavior, whereas OSRAM models the closed loop and adapts references without changing the deployed policy.Source: A. Sim-to-Real Transfer with Online… · p. 2

B. Input-Output Dynamics Modeling for Robotics

  • Input-output models predict observable responses from input and output histories, avoiding full internal-state reconstruction; OSRAM extends reference steering with nonlinear closed-loop modeling meta-trained across randomized dynamics.Modeling only task-relevant outputs can reduce the burden of learning high-dimensional state transitions.Source: B. Input-Output Dynamics Modeling for… · p. 2

III. METHOD

OSRAM models the robot and its controller as one closed-loop system, then adapts future reference commands while keeping the deployed policy fixed. It meta-trains this command-response model in simulation and specializes it with limited real-world tracking data.

III. METHOD

  • OSRAM meta-trains a closed-loop command-response model in simulation, finetunes it with real tracking observations, and adapts references without changing the deployed policy.The framework has simulation-training and post-deployment adaptation stages.Source: §III · p. 2, Figure 2 · p. 3
  • The model predicts task-level responses from reference-command and response histories, capturing the aggregate robot-policy behavior without modeling physical dynamics separately.This focuses adaptation on task-relevant variables rather than the full robot state.Source: §III · pp. 2–3
  • Randomized simulator configurations and stochastic reference excitation generate diverse trajectories, while task-specific inner-loop updates shape a shared initialization for rapid specialization.The simulated variations include mass, friction, and actuator dynamics.Source: §III · p. 3
  • Training combines one-step prediction with multi-step rollout consistency, discounting farther-horizon errors and balancing one-step accuracy against long-horizon consistency.The model's autoregressive rollout recursively feeds predicted responses back into the model.Source: §III · p. 3
  • After deployment, limited reference-response data finetune the model, which then optimizes future commands to make realized behavior follow the original desired trajectory.The policy remains unchanged, and the method does not reconstruct the underlying physical dynamics.Source: §III · pp. 3–4
  • Reference optimization penalizes tracking error, deviation from nominal commands, and abrupt command changes; MPPI samples candidate trajectories and replans after executing the first segment.The deviation regularizer discourages aggressive corrections and extrapolation beyond the model's observed operating regime.Source: §III · p. 4

IV. EXPERIMENTAL RESULTS

The experiments evaluate OSRAM in simulation and on hardware across bipedal velocity tracking and loco-manipulation. They test generalization to unseen dynamics, the value of closed-loop modeling, comparisons with online adaptation methods, and applicability across control objectives and configurations.

IV. EXPERIMENTAL RESULTS

  • Evaluation covers bipedal velocity tracking and loco-manipulation in simulation and on hardware.The experiments are organized around four questions about generalization, closed-loop modeling, comparisons, and breadth of application.Source: §IV · p. 4
  • The experiments ask whether meta-dynamics learning improves adaptation under previously unseen environment dynamics and whether closed-loop modeling improves prediction and adaptation quality.Source: §IV · p. 4
  • They also test whether OSRAM outperforms existing online adaptation methods and works across different control objectives, policy configurations, and command-response spaces.Source: §IV · p. 4

A. Experimental Setup

The evaluation uses a LimX Tron1 biped for velocity tracking and loco-manipulation, with simulation and hardware setups designed for comparison. It measures tracking accuracy, consistency, and task completion reliability.

A. Experimental Setup

  • Experiments use the LimX Tron1 with and without a loco-manipulation expansion kit for base-velocity tracking and loco-manipulation.Loco-manipulation adds an end-effector pose-tracking objective to the velocity-tracking reward; task rewards are referenced in Table I.Source: A. Experimental Setup · p. 4, Table I · p. 5
  • Hardware evaluation reuses the simulation setup, estimates global base position with motion capture, and adds mass and changes policy gains to increase dynamics mismatch.Source: A. Experimental Setup · pp. 4–5
  • Metrics include mean tracking error, RMSE, RMSE standard deviation over repeated trials, and success rate, with tracking errors averaged over successful trials only.Success rate counts all trials; repeated-trial consistency uses 20 simulation trials and 5 hardware trials.Source: A. Experimental Setup · p. 5

B. Adaptation to Unseen Dynamics

Under out-of-distribution dynamics, the closed-loop meta-dynamics model predicts more accurately than a forward dynamics model and achieves stronger velocity tracking when used for adaptation. The reported tracking errors are lowest with the meta-dynamics model across the tested target velocities.

B. Adaptation to Unseen Dynamics

  • The prediction comparison uses models pretrained on equal amounts of data and finetuned on the same OOD dataset with the same batch size and gradient steps.The OOD evaluation further randomized center of mass and actuator gains.Source: B. Adaptation to Unseen Dynamics · p. 5
  • For one-step prediction on out-of-distribution trajectories, the proposed model's MAE was 0.0116 versus 0.0594 for the forward dynamics model, an 80.5% reduction.The OOD trajectories further randomized the robot's center of mass and actuator gains.Source: B. Adaptation to Unseen Dynamics · p. 5
  • Closed-loop meta-dynamics achieved ERMSE values of 0.0733, 0.0929, and 0.1560 at target velocities of 0.4, 0.7, and 0.9 m/s, the lowest values reported across the compared models.The forward MLP model reduced ERMSE from 0.1593, 0.2127, and 0.2209 to 0.0800, 0.1472, and 0.1917 at those respective velocities, maintaining a 100% success rate; OSRAM also had the best Emean at all tested velocities.Source: B. Adaptation to Unseen Dynamics · p. 5, Figure 3 · p. 5

C. Why Model the Closed-Loop System?

The section compares closed-loop command-response modeling with open-loop dynamics models for prediction and velocity-tracking adaptation. OSRAM tracks reliably across tested velocities, including where the open-loop baselines fail.

Closed-loop versus open-loop models

  • OL-Full predicts next state from full robot observations and executed joint actions, while OL-Out uses command-related observations and joint actions.OL-Out removes information unrelated to the command but retains an open-loop prediction structure.Source: C. Why Model the Closed-Loop System? · p. 6
  • The closed-loop model achieved the highest prediction accuracy under identical training conditions; OL-Full and OL-Out yielded MAEs of 0.0789 and 0.0338, respectively.The models were pretrained on equal amounts of data and finetuned on the same target dataset with the same batch size and gradient steps.Source: C. Why Model the Closed-Loop System? · p. 6
  • At 0.9 m/s, OSRAM achieved an ERMSE of 0.1560 with 100% success, while both open-loop baselines had 0% success.At 0.4 and 0.7 m/s, OSRAM also obtained much lower ERMSE than OL-Full and OL-Out while maintaining 100% success.Source: C. Why Model the Closed-Loop System? · p. 6

D. Comparison with Online Adaptation Strategies

The comparison tests OSRAM against recurrent latent adaptation and residual-policy adaptation using the same target-domain data budget. Both baselines remain stable, but OSRAM tracks more accurately across the tested velocities.

Baseline comparison

  • Res-Adapt learned a residual policy over frozen base-policy actions using the same amount of target-domain data as OSRAM's model finetuning.RNN-Adapt used an observation-history encoder whose latent representation was provided to the velocity-tracking policy.Source: D. Comparison with Online Adaptation… · p. 6
  • OSRAM achieved lower ERMSE than RNN-Adapt and Res-Adapt at all three tested velocities: 0.4, 0.7, and 0.9 m/s.OSRAM's ERMSE values were 0.0733, 0.0929, and 0.1560, respectively.Source: D. Comparison with Online Adaptation… · p. 6
  • Both RNN-Adapt and Res-Adapt maintained a 100% success rate across all tested velocities, but their tracking accuracy was consistently lower than OSRAM's.RNN-Adapt ERMSE values were 0.1415, 0.1614, and 0.2082; Res-Adapt values were 0.1421, 0.1833, and 0.2062.Source: D. Comparison with Online Adaptation… · p. 6

E. Sim-to-Real Hardware Evaluation

Hardware evaluations cover velocity tracking and loco-manipulation. OSRAM improves coronal tracking for both tested policy configurations, and substantially reduces loco-manipulation trajectory errors.

Hardware evaluation

  • OSRAM reduced loco-manipulation x-direction RMSE from 36.94 ± 3.02 cm to 3.11 ± 0.31 cm and y-direction RMSE from 3.48 ± 0.45 cm to 2.72 ± 0.23 cm.The task commanded a global sine-wave end-effector trajectory; command-response signals were represented in the robot's local base frame.Source: E. Sim-to-Real Hardware Evaluation · p. 7
  • On hardware velocity tracking, OSRAM improved coronal tracking for both the baseline policy and the RNN-Adapt policy.The evaluation also reported gains over the baseline policy and additional gains when OSRAM augmented the RNN-based policy.Source: E. Sim-to-Real Hardware Evaluation · p. 7
  • Forward-velocity prediction MAE decreased rapidly during initial finetuning and plateaued after approximately 30 gradient steps.The plotted error was vx MAE versus the number of finetuning gradient steps.Source: E. Sim-to-Real Hardware Evaluation · p. 7

F. How Does Reference Adaptation Correct Tracking Error?

The finetuned closed-loop model predicts deployed-system responses, and OSRAM uses those predictions to choose corrective references. These changes move realized trajectories closer to the original targets while leaving the policy unchanged.

Prediction and reference adaptation

  • Hardware plots show velocity-tracking trajectories truncated at x = 3.5 m, forward-velocity prediction error versus finetuning steps, and loco-manipulation end-effector and global x-position trajectories.The truncation highlights lateral tracking error in the velocity-tracking plot.Source: Figure 4 · p. 7
  • The finetuned model's one-step prediction RMSE was 0.0557 m/s for velocity tracking and 0.0074 m for loco-manipulation.The passage reports these errors as evidence that the model captures command-response behavior sufficiently accurately for reference optimization.Source: F. How Does Reference Adaptation… · p. 7
  • OSRAM chooses modified references whose predicted responses better match the nominal target, increasing commands for under-tracking and reducing them for overshoot.The resulting realized trajectories move closer to the original desired trajectories without changing the underlying policy.Source: F. How Does Reference Adaptation… · p. 7

V. CONCLUSION, LIMITATION AND FUTURE WORK

OSRAM improves prediction and tracking in unseen dynamics across simulated and physical bipedal velocity tracking and loco-manipulation. The work identifies low-dimensional command spaces and sensitivity to optimization hyperparameters as limitations or directions for future work.

V. CONCLUSION, LIMITATION AND FUTURE WORK

  • OSRAM improves prediction and tracking in unseen dynamics across simulated and physical bipedal velocity tracking and loco-manipulation.Source: §V · p. 7
  • Relatively low-dimensional command spaces constrain the current focus, with high-dimensional references such as whole-body motion trajectories identified as future work.Source: §V · p. 7
  • Fig. 5 presents one-step model predictions and shows adapted references compensating for tracking errors so realized responses more closely follow nominal targets.Source: Figure 5 · p. 8
  • The sampling-based optimization backend can be sensitive to hyperparameter tuning, while gradient-based optimization is proposed as a possible more efficient alternative.Source: §V · p. 8
  • Because OSRAM does not assume a specific inner-loop policy form, the authors identify model-based controllers as a potential extension.Source: §V · p. 8
Loading 2609.28878v1…