PaperlayerView exact revision on arXiv

Source-linked AI summary

DTC: Deep Tracking Control

Fabian Jenelten, Junzhe He, Farbod Farshidian, Marco Hutter

arXiv:2309.15462v2cs.ROcs.LGeess.SY

TL;DR

Legged locomotion requires both accurate foothold planning and robustness to model mismatch, uncertainty, and sparse-reward terrains. The paper trains a simulation-based neural tracking controller from trajectory-optimization reference motions, and reports strong real-world performance across challenging terrains, ground conditions, and planners. The approach combines model-based planning with learned robustness while retaining a stated scope boundary rather than providing a universal integration recipe.

  • Problem

    Model-based control provides accurate planning but depends on imperfect assumptions, whereas reinforcement learning is robust yet struggles to explore terrains with sparse foothold rewards.

  • Method

    A neural low-level tracking controller is trained in simulation to follow footholds and reference motions generated by trajectory optimization.

  • Results

    The hybrid controller outperformed an RL baseline on every tested sparse terrain, outperformed two model-based controllers on slippery and soft ground, and tracked motions from two different planners.

  • Takeaways & Limitations

    The pipeline combines trajectory optimization’s planning and foot-placement accuracy with reinforcement learning’s robustness and recovery behavior across complex environments.

  • Takeaways & Limitations

    The authors state that the architecture is not a universal recipe for optimally integrating model-based and model-free methods, and convergence required many epochs.

Abstract

from arXiv · show

Legged locomotion is a complex control problem that requires both accuracy and robustness to cope with real-world challenges. Legged systems have traditionally been controlled using trajectory optimization with inverse dynamics. Such hierarchical model-based methods are appealing due to intuitive cost function tuning, accurate planning, generalization, and most importantly, the insightful understanding gained from more than one decade of extensive research. However, model mismatch and violation of assumptions are common sources of faulty operation. Simulation-based reinforcement learning, on the other hand, results in locomotion policies with unprecedented robustness and recovery skills. Yet, all learning algorithms struggle with sparse rewards emerging from environments where valid footholds are rare, such as gaps or stepping stones. In this work, we propose a hybrid control architecture that combines the advantages of both worlds to simultaneously achieve greater robustness, foot-placement accuracy, and terrain generalization. Our approach utilizes a model-based planner to roll out a reference motion during training. A deep neural network policy is trained in simulation, aiming to track the optimized footholds. We evaluate the accuracy of our locomotion pipeline on sparse terrains, where pure data-driven methods are prone to fail. Furthermore, we demonstrate superior robustness in the presence of slippery or deformable ground when compared to model-based counterparts. Finally, we show that our proposed tracking controller generalizes across different trajectory optimization methods not seen during training. In conclusion, our work unites the predictive capabilities and optimality guarantees of online planning with the inherent robustness attributed to offline learning.

INTRODUCTION

Model-based trajectory optimization offers accurate, predictive locomotion planning but is vulnerable to modeling assumptions, while reinforcement learning provides robustness yet struggles with sparse foothold rewards. The proposed hybrid approach uses trajectory-optimization references to guide a learned tracking controller, combining precise foothold placement with robust, generalizable behavior.

  • Model-based control: Trajectory optimization plans legged-system motions using robot kinematics, dynamics, and terrain models, then tracks them with whole-body control.Common assumptions include perfect state estimation, known contacts, zero foot-slip, and perfect realization of planned motions.
  • Learning-based control: Reinforcement learning learns closed-loop neural policies without explicit models and can provide robust locomotion and recovery behaviors.However, gaps and stepping stones produce sparse rewards that require effective exploration strategies.
  • Complementary strengths: Optimization methods can improve exploration by providing footholds, segmenting terrain into steppable regions, and planning future environmental interactions.These properties can support generalization across diverse terrain geometries when model assumptions are sufficiently generic.
  • Learning from optimized motion: Prior expert-imitation methods use offline demonstrations, but online expert-data generation is computationally expensive and learned foothold generators may not capture model-based behavior or solve exploration.These limitations motivate using trajectory-optimization solutions to guide exploration directly.
  • Proposed hybrid approach: The proposed hierarchy uses trajectory optimization for foothold planning and a higher-rate learned controller that treats optimized footholds as demonstrations of precise foot placement.The policy observes planar footholds, desired joint positions, and the contact schedule, with rewards focused on footholds at planned touchdown.
  • Proposed hybrid approach: The hybrid pipeline combines offline learning with online replanning to support accurate, agile, robust locomotion and zero-shot deployment across trajectory-optimization methods and update rates.The authors report deployment across sparse-contact and uncertain environments, including search-and-rescue scenarios.

RESULTS

DTC combined trajectory-optimization references with learned recovery and tracking to achieve accurate foothold placement, robustness to model violations, and generalization across planners and sparse terrains.

  • Robustness: The controller recovered from movable or deforming obstacles in all five trials despite not being trained on such obstacles.The test included slippery, soft, rolling, and sloped objects redistributed between trials.
  • Accuracy: 2.3 cm average foothold tracking error with 0.48 cm standard deviation was achieved across heading velocities from ±0.4 to ±1.0 m/s.The error was measured over approximately 20-second flat-ground trials using eight heading velocities.
  • Model-based comparison: DTC crossed the obstacle parkour at 1 m/s, 20% faster than baseline-to-1, while baseline reinforcement-learning policies could not traverse it.The parkour was traversed back and forth in simulation.
  • Robustness: DTC navigated an invisible-gap trap despite model and execution mismatch, whereas baseline-to-1 failed after heuristic recovery strategies proved ineffective.The experiment was repeated five times for each controller with consistent outcomes; DTC learned recovery and reflex skills using proprioception and simulated map drift.
  • Sparse terrain: 100% success was achieved over ten repetitions for a 0.1 m gap, and over four repetitions for each experiment with four 0.6 m gaps and a 1.8 m beam.The harder experiments increased both gap width and beam length while retaining the same success rate.
  • Ablation: Increasing optimizer updates from 1 Hz to 50 Hz reduced failure rate by 7.11% and increased success rate by 4.25%.The 50 Hz upper limit was imposed by the policy frequency.
  • Terrain generalization: Across sparse and dense terrain types, DTC outperformed baseline-rl-2 and exceeded baseline-rl-3 particularly on sparse-structured terrains.DTC matched baseline-rl-3 on dense terrains, while the sparse-terrain advantage indicated that performance was not specific to the training environment.

DISCUSSION

The hybrid pipeline combines trajectory optimization’s accurate planning with reinforcement learning’s robustness, while generalizing across planners and improving performance on challenging terrain. Its scope remains limited by sample efficiency and the lack of a universal integration recipe.

  • The hybrid pipeline combines accurate foot placement and dynamic agility from trajectory optimization with robustness and reflex behaviors from reinforcement learning.
  • Extensive real-world tests showed that DTC outperformed the RL baseline on gaps, stepping stones, narrow beams, and tall boxes, and surpassed two model-based controllers on slippery and soft ground.Experiments covered approximately half a year and included sparse and difficult terrain conditions.
  • The learned policy tracked motions from two different planners using the same trotting gait, indicating generalization across trajectory optimizers.The authors attribute this to footholds being mostly invariant across optimizer choices.
  • The policy generalized from visual-perception training to blind locomotion and tolerated substantial visual-data noise and drift without gaited networks.An asymmetric actor/critic MLP also achieved robust behaviors comparable to more complex teacher/student training.
  • The proposed training environment was not more sample efficient than similar unifying reinforcement-learning approaches, and convergence required many epochs.The authors suggest that learning foothold accuracy may be intrinsically difficult.
  • The architecture is not a universal recipe for optimally integrating trajectory optimization and reinforcement learning in generic problems.The discussion identifies broader hybrid-control variants as possible directions for future research.

MATERIALS AND METHODS

The method uses trajectory optimization to provide reference footholds and motion information, while a neural policy tracks these references through closed-loop reinforcement learning. The design omits some planner outputs from policy observations to improve robustness and planner independence.

  • Trajectory optimization plans motions using robot and terrain models, whereas reinforcement learning learns closed-loop policies without explicit terrain or system models.Trajectory optimization offers planning and accuracy, while reinforcement learning offers robustness but weaker extrapolation to unseen environments.
  • The policy is an MLP stochastic action distribution whose actions are target joint positions tracked by a PD controller.
  • At each step, the optimizer supplies a future reference state, the policy samples an action, and forward simulation produces the next system state.The extracted reference includes footholds, touch-down joint positions, and next-step base trajectory quantities.
  • The reward decreases with the error between optimized and measured states, framing the policy as a learned model-reference adaptive controller.The optimizer acts as the reference model for the learned controller.
  • Asymmetric actor-critic training gives privileged observations to the value function while the policy learns from policy observations alone.Privileged inputs include optimized base states and simulator-only contact, force, friction, and elevation-drift signals.
  • DTC uses TAMOLS to optimize footholds and base pose simultaneously, then exposes the policy to planar footholds, desired joint positions, and contact schedules.These reference signals are intended to retain useful planner information while avoiding common model-based-control pitfalls.
  • The observation space combines proprioception, planner-derived foothold and gait signals, and targeted height scans along each foot’s path to its desired foothold.Targeted scans support collision-free swing trajectories while reducing exposure to uncertain map information.
  • Analytical inverse kinematics computes desired joint positions, while the policy excludes base-pose references to reduce mapping sensitivity and planner dependence.The reference twist is retained so the network can infer the desired walking direction.

A B C

The controller trains a policy to track trajectory-optimization references while using consistency, regularization, and noisy observations to improve feasible, smooth locomotion. The training pipeline combines online optimization with simulation-based policy learning across varied terrains.

  • A: The optimizer supplies desired footholds, base pose, twist, acceleration, and joint positions, while a height scan samples terrain between each foot and its target foothold.At each leg touch-down, the optimizer generates a new solution and the policy drives the simulated state toward it.
  • B: Base-pose tracking uses a soft reward based on measured-versus-desired pose, twist, and acceleration errors one policy step ahead.The exponential reward shape progressively shifts learning emphasis from the trajectory middle and head toward the tail.
  • B: Foothold tracking uses a logarithmic hard reward that increasingly concentrates gradients as foot-placement error approaches zero.The reward is issued at most once per gait cycle when intended contact is detected, preventing stance-foot dragging.
  • B: Consistency rewards similar consecutive optimization solutions to discourage hesitation and the lazy behavior caused by updating references too frequently.Similarity is measured through Euclidean distance between adjacent solutions, including base trajectories and stacked footholds over overlapping horizons.
  • C: Training combines PPO with GPU simulation and back-propagation, CPU optimization, domain randomization, disturbances, and a 12-type, 10-level terrain curriculum.Noise affects observations, terrain scans, and selected joint-position inputs; randomized friction, pushes, and external forces target robustness to uncertainty.

SUPPLEMENTARY METHODS

The supplementary methods specify the policy interfaces, simulation settings, randomized disturbances, and evaluation configurations. They also document network parameterization, shared training settings, and terrain-specific test protocols.

  • Interfaces: The policy receives state, optimizer-solution, and perceptual inputs, while privileged observations are supplied separately to the value function.The nomenclature identifies actions, state, policy observations, privileged observations, optimized footholds, and one-step-ahead optimization solutions.
  • Training settings: Simulation uses a 5 ms timestep and a 20 ms policy timestep, with reference velocities resampled during episodes across heading, lateral, and yaw motion.Velocity distributions include vx ∼U(−1, 1) m/s, vy ∼U(−0.8, 0.8) m/s, and vψ ∼U(−0.8, 0.8) rad/s.
  • Randomization: Height-map and drift noise are sampled from bounded uniform approximations, while pushes, friction coefficients, base wrenches, and foot forces are randomized.The reported ranges include friction µ ∼U(0.1, 1.2), base-wrench sampling from U(−15, 15), and external foot forces from U(−2, 2) N.
  • Implementation: The Gaussian actor and critic use multilayer perceptrons, and all policies share the same PPO hyperparameters with 4096 parallelized environments.Both MLPs use three hidden layers with 512 neurons each, and policies were trained with seed 1.
  • Evaluation: Performance evaluations disable the curriculum and select the most difficult terrain of each type, while specific experiments alter pushes, velocity changes, and optimizer access to true disturbances.The evaluation protocols separately test overall performance, drift, and optimizer-rate effects under controlled deviations from training.
Loading 2309.15462v2…