Source-linked AI summary
Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning
Davide Bargellini, Alex Pasquali, Andrea Govoni, Riccardo Zanella, Gianluca Palli
TL;DR
Torque-level Sim2Real transfer is limited by discrepancies between simulated and real robot dynamics, motivating safer and more accurate transfer methods. The paper calibrates simulation parameters through Real2Sim trajectory matching and genetic optimization, trains a TQC policy with domain randomization, and evaluates it across simulators and on a real Panda robot. The reported results show reduced reality-gap effects, reliable TQC performance, and robust real-robot operation under practical disturbances.
Problem
Sim2Real transfer for torque-controlled robots is challenged by discrepancies in physical parameters, contact forces, and simulator models.
Method
The pipeline calibrates friction, inertia, and other dynamics by matching real and simulated trajectories, then trains a TQC policy in simulation with domain randomization.
Results
The calibrated pipeline reduced the reality gap on a 7-DOF Franka Emika Panda, with TQC providing the most reliable performance and real-robot experiments confirming robustness to practical disturbances.
Takeaways & Limitations
Physics-aware identification using tuned friction and inertia can support transfer of policies trained entirely in simulation to the real robot.
Abstract
from arXiv · showhide
Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge, particularly when dealing with low-level torque control, where even small modelling inaccuracies can lead to unstable or unsafe behaviours. In this work, we propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms by combining trajectory matching, parameter optimization via genetic algorithms, and domain randomization. Using the 7-DOF Franka Emika Panda robot, we first identify friction, inertia, and gravity compensation parameters by minimizing the error between real and simulated joint trajectories. These calibrated dynamics are then used to train a TQC-based reinforcement learning agent in simulation. The trained policy is evaluated in both Gazebo and MuJoCo environments, and finally deployed on the real robot. Our results demonstrate a significant improvement in tracking accuracy and policy robustness after parameter tuning, with smooth policy transfer from simulation to the Real-World across multiple target-reaching tasks. This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.
I. INTRODUCTION
Torque-controlled RL offers precise interaction and safer policy development in simulation, but transferring policies to real robots is hindered by the reality gap. This work targets that gap for torque-based reaching with a 7-axis Franka Emika Panda.
- Torque control can improve responses to physical forces, while simulation enables faster and safer policy development than direct real-world training.Direct real-world training with joint torques is described as time-consuming, resource-intensive, and safety-critical.
- Sim2Real transfer is challenged by discrepancies in physical parameters, contact forces, and simulator implementations.These discrepancies form a reality gap between simulated and real-world environments.
- Prior work addresses the reality gap through improved calibration of dynamics, friction, and sensor-feedback parameters.
- The study optimizes Sim2Real transfer for a 7-axis Franka Emika Panda using reinforcement-learning policies for joint-torque position reaching.The evaluated policy families include DDPG, SAC, TD3, and TQC.
II. RELATED WORKS
Related work emphasizes accurate physical modelling and methods that expose policies to realistic variation. The paper combines trajectory matching, friction and inertia tuning, genetic optimization, and reinforcement learning for torque-controlled transfer.
- Accurate estimation of robot and environment parameters is crucial because modelling discrepancies can hinder Sim2Real policy transfer.Joint-space training can exploit full-model dynamics for complex robot-environment interactions.
- Domain randomization varies simulated mass, friction, inertia, and material resistance to improve robustness to real-world uncertainty.
- The workflow includes parameter estimation, RL training with TQC and GridSearch optimization, policy validation, and Real2Sim or Sim2Real execution.
- Torque-control reinforcement learning must model motor dynamics and motor interactions that affect robot movement.
- The paper reduces the simulation-reality gap by tuning friction and inertia through trajectory matching and genetic optimization.Friction parameters are calculated from prior work, while inertia parameters are optimized against trajectories.
III. METHODOLOGIES
The methodology first calibrates simulator dynamics by matching real and simulated joint trajectories, then trains the control policy entirely in the calibrated simulation to improve transfer fidelity.
- Real2Sim trajectory matching calibrates key dynamics parameters by minimizing discrepancies between measured and simulated joint responses.A genetic algorithm performs the parameter optimization.
A. Reinforcement Learning Overview
The RL agent learns torque commands for Cartesian target reaching in simulation, with rewards combining accuracy and motion quality. Domain randomization supports transfer to real-world scenarios.
- The 7-DoF Franka Emika Panda agent outputs joint torque commands to reach Cartesian targets while maintaining smooth and stable motion.
- The agent is trained in simulation with domain randomization to support robust transfer of learned policies to real-world scenarios.
- The objective maximizes expected cumulative reward, where the discount factor controls the importance of future rewards.Rt denotes expected cumulative reward, and rt+k denotes the reward at time t+k.
- The reward uses tanh to bound velocity and acceleration contributions and limit extreme-value influence.The tanh terms are described as supporting numerical stability.
- The reward balances target proximity with velocity and acceleration penalties to encourage smooth, energy-efficient joint movements.The first term promotes goal proximity, while velocity and acceleration penalties discourage abrupt or unstable motion.
- Policy parameters are updated iteratively through a loss-gradient rule using a learning rate.Θt represents policy parameters, η(t) the learning rate, and ∇L the loss gradient.
B. The Parameter Estimation Problem
The paper estimates robot dynamics parameters by matching simulated and Real-World trajectories, using a genetic algorithm to minimize tracking error. The model includes inertia, Coriolis and centrifugal effects, gravity, directional friction, and manually introduced static-friction compensation.
- Dynamics Model: The manipulator dynamics model represents torque through inertia, Coriolis and centrifugal effects, and gravity.The standard equation is applied to an n-DOF manipulator.
- Friction Modeling: Friction is modeled separately from the robot description, with different friction levels for positive and negative joint velocities.The friction model uses joint-specific parameters and includes directional asymmetry.
- Friction Modeling: Static friction is handled by applying a manually tuned breakaway torque within a dead zone around zero velocity.The breakaway torque is joint-specific and follows the desired motion direction.
- Trajectory Matching: The parameter estimation problem minimizes the discrepancy between simulated and Real-World joint trajectories.The error compares ysim(t, p) with yreal(t) over the observation interval T.
- Genetic Optimization: A genetic algorithm iteratively adjusts each joint’s parameter set by simulating trajectories, computing error, selecting top-performing individuals, and applying mutation.The procedure returns the best parameter set after convergence or the generation limit.
IV. EXPERIMENTS AND RESULTS
The experiments compare reinforcement-learning algorithms and evaluate torque transfer before and after parameter calibration. TQC is selected for Sim2Real results, while gravity compensation, static friction, inertia, and friction parameters are adjusted to reproduce Real-World behavior.
- Algorithm Comparison: TQC provided the most robust and stable performance across metrics and showed lower hyperparameter sensitivity than DDPG, SAC, and TD3.The reported Sim2Real transfer results therefore use TQC.
- Parameter Calibration: Figures 3a and 3b show joint-level tests without parameter calibration, including inadequate gravity compensation and unmodeled dynamics in load-bearing configurations.Torque offsets were manually adjusted for each joint to make positive and negative trajectories visually similar.
- Parameter Calibration: Static friction thresholds were identified by increasing joint torque until movement began, separately for positive and negative directions.In simulation, actions below the threshold were nullified within a small velocity range.
- Optimization Results: The genetic algorithm produced the inertia and friction parameters reported in Tables III and IV.These parameters constitute the calibrated dynamics used after optimization.
B. Sim2Sim
The agent is trained in MuJoCo for fast policy learning and then tested in Gazebo with varied Panda model parameters. Training targets are randomized across the robot’s workspace, while domain randomization perturbs estimated friction and inertia parameters.
- Simulation Setup: The TQC agent is trained in MuJoCo to reach random target points across diverse workspace coordinates.MuJoCo is used because its higher simulation speed supports rapid policy training.
- Simulation Setup: Before Real-World deployment, the policy is tested in Gazebo using different Panda model parameters within the training domain.The tests use different target positions to examine behavior under varied conditions.
- Domain Randomization: Domain randomization perturbs all 21 joint-specific friction parameters by ±10% and all 42 joint-inertia-matrix elements by ±5%.The perturbations are applied around the previously estimated parameter values.
C. Sim2Real
Domain randomization prepared the torque-controlled agent for varied simulation conditions and Real-World uncertainties. Evaluations across MuJoCo, Gazebo, and the physical Panda showed robust goal-reaching, with minor target deviations.
- Sim2Real preparation: Domain randomization varied simulated friction and inertia to improve adaptation to Real-World uncertainties.The approach trained the agent across a broad spectrum of conditions before deployment.
- Real-World evaluation: The deployed agent achieved high reward scores and reliable control on the physical Franka Emika Panda robot.It required minimal adjustments after simulation training with domain-randomized parameters.
- Parameter calibration: The inertia and friction parameter tables organize optimized-model values as before-and-after comparisons.Table III covers inertia tensor components for seven links, while Table IV covers friction parameters.
- Simulator comparison: MuJoCo and Gazebo comparisons evaluated TQC rewards across four random goal scenarios under differing simulator dynamics.The figure emphasizes performance variations attributable to the distinct properties of the two simulation environments.
- Real-World evaluation: Real-World evaluation showed the end effector consistently approached and maintained positions near randomized goals, despite minor reaching uncertainties.The deviations were observed in two Real-World trial comparisons referenced as Fig. 8(b) and Fig. 8(c).
- Cross-environment evaluation: Across environments, the trained model demonstrated generalizability and robustness for potential Real-World applications.The supplied evaluation summary covers different simulation and physical environments without reporting a numerical metric.
V. CONCLUSIONS
The paper presents a Real2Sim2Real pipeline for torque-controlled reinforcement learning on a 7-DOF Franka Emika Panda. Physics-aware parameter tuning and domain randomization supported transfer across simulators and to the real robot, while future work targets broader manipulation and robot settings.
- Conclusions: The Real2Sim2Real pipeline combines trajectory matching, domain randomization, and genetic optimization to reduce the reality gap on a 7-DOF Franka Emika Panda.The pipeline targets Sim2Real transfer under low-level torque actuation.
- Conclusions: Physics-aware system identification using a custom friction-and-inertia function increased simulation fidelity and enabled simulation-trained policies to transfer to the real robot.The approach avoided relying solely on simulation friction parameters.
- Conclusions: TQC provided the most reliable performance among the evaluated reinforcement learning methods.The conclusion reports this as the comparative outcome across the tested RL methods.
- Validation: The framework was validated in Sim2Sim across MuJoCo and Gazebo as well as in Sim2Real deployment.This indicates evaluation beyond a single simulation domain.
- Validation: Real-robot experiments confirmed robustness under backlash, cable drag, and unmodelled delays.These disturbances represent practical conditions encountered during physical deployment.
- Future directions: Future work includes contact-rich manipulation, online system identification, other robot platforms, safety-aware reinforcement learning, and compliance control.These directions define the current scope boundary of the presented framework.