Source-linked AI summary
Mind the Phase: Effective Rank and Representation Health in Legged Locomotion
Felipe Tommaselli, Thiago H. Segreto, Juliano D. Negri, Ricardo V. Godoy, Marcelo Becker
TL;DR
PPO locomotion policies lack a training-time representation signal tied to hardware behavior. The paper analyzes effective rank, especially phase-conditioned policy-Jacobian rank, and finds a swing–stance architectural signature associated with smoother transfer and roughly 3× lower joint jitter on Spot.
Problem
PPO locomotion policies are commonly evaluated through reward curves, while their learned representations remain poorly characterized and lack a training-time signal predictive of hardware behavior.
Method
The paper studies effective rank of policy Jacobians and other representations, conditioning Jacobian rank on gait phase to analyze locomotion policies.
Results
Roughly two more effective-rank dimensions are allocated to swing than stance by layer normalization and residual connections, and a related recipe yields roughly 3× lower joint jitter through transfer to a physical Spot.
Takeaways & Limitations
Effective rank is presented as a training-time diagnostic of representation health that covaries with deployment smoothness rather than an objective to maximize.
Takeaways & Limitations
The results do not fully characterize generalization or method isolation, and the study omits architectures such as RNNs and Transformers, several normalization and hyper-parameter choices, and actor–critic decoupling.
Abstract
from arXiv · showhide
Reinforcement learning has become the leading paradigm in legged locomotion, enabling complex behaviors from backflips to parkour through massively parallel simulation. Under PPO's non-stationarity, shallow networks remain the de facto architecture, supported by carefully staged curricula and environments, yet the representations these policies learn stay poorly understood, leaving no training-time signal of how they will behave on hardware. In this work, we empirically study locomotion policies through the effective rank of the policy Jacobian and show that conditioning rank on the gait phase exposes architectural structure that global rank averages away. In particular, we find that standard architectural choices, namely layer normalization and residual connections, allocate roughly two more dimensions of effective rank to swing than to stance, which is fully absent in vanilla MLPs. Building on this, we propose a simple recipe that turns these representational signatures into smoother, more reliable sim-to-real transfer. In practice, this results in roughly 3x lower joint jitter that holds from simulation onto a physical Spot, suggesting that representation health is an effective training-time lens to track sim-to-real smoothness.
1 Mobile Robotics Group, University of S˜ao Paulo
The paper studies legged locomotion across simulated quadrupeds and a humanoid, with real deployment on a Boston Dynamics Spot.
- The study covers three simulated embodiments: two quadrupeds and one humanoid.
1 Introduction
The paper addresses poorly characterized policy representations in PPO locomotion by analyzing effective rank and conditioning it on gait phase. This reveals an architectural swing–stance structure and motivates a recipe that reduces sim-to-real joint jitter.
- PPO locomotion policies are usually monitored through reward curves, leaving their learned representations poorly characterized and without a training-time signal predictive of hardware behavior.
- Layer normalization and residual connections provide optimization stability, but their representational mechanism in legged locomotion remains opaque.
- Effective rank of the policy Jacobian connects local observation-to-action sensitivity with action smoothness during deployment.
- Conditioning Jacobian rank on gait phase exposes architectural structure that phase-agnostic averaging obscures.
- Roughly two extra effective-rank dimensions are allocated to swing rather than stance with layer normalization and residual connections, unlike vanilla MLPs.
- A training recipe based on this signature yields roughly 3× lower joint jitter from simulation to a physical Spot.
2 Related Works
Related work connects legged-locomotion progress to deep reinforcement learning, while identifying unresolved concerns about high-frequency control, PPO representation drift, and gait-health characterization.
- Deep reinforcement learning has become a standard approach for legged control and has produced athletic behaviors through simulation and sim-to-real transfer.
- Task-reward-only policies can exploit high-frequency control signals that are difficult to realize on hardware and may damage actuators.
- PPO constrains policy-distribution updates but does not comparably constrain the underlying representation, which can drift under non-stationarity.
- Prior gait descriptions emphasize stance–swing alternation and foot placement, concepts that remain reflected in locomotion reward functions.
3 Background
The paper formalizes PPO locomotion and studies several rank objects, especially policy-Jacobian rank, to characterize local policy sensitivity across gait phases. It frames swing and stance as distinct control regimes whose rank can indicate gait quality.
- 3.1 PPO Locomotion Training Setup: The PPO setting is a discounted Markov decision process in which policies map observations to actions over rollouts.
- 3.1 PPO Locomotion Training Setup: The on-policy actor–critic pipeline jointly optimizes policy and value networks using rollouts, generalized advantage estimates, clipped objectives, and minibatch updates.
- 3.2 Effective Rank: Feature rank measures the penultimate-layer representation, the most abstract representation before final task-specific compression.
- 3.2 Effective Rank: Effective rank interprets the singular-value spectrum as an entropy-equivalent number of active modes: flat spectra over k modes approach k, while single-value dominance approaches 1.
- 3.2 Effective Rank: Weight rank evaluates actor weight matrices as a parametric bandwidth measure, so high representation rank is uninformative when the weight rank has collapsed.
- 3.2 Effective Rank: Policy-Jacobian effective rank measures the dimensionality of local input–output sensitivity; lower values indicate dependence on fewer dominant observation directions.
- 3.3 Gait Patterns: Legged locomotion alternates between swing, when a foot is off the ground, and stance, when it bears load.
- 3.3 Gait Patterns: Phase-conditioned Jacobian rank is expected to shift across the gait cycle as legs load and unload, while degraded or reward-hacked gaits may show lower rank and phase misalignment on hardware.
4 Experiments
Experiments show that phase-conditioned Jacobian rank reveals architectural structure hidden by global averages, and that this representation signature tracks smoother sim-to-real behavior. Over-training can inflate rank while erasing the phase structure, which reward alone may miss.
- Experimental setup: PPO experiments use IsaacLab and RSL RL, with healthy runs as the primary focus and five independent seeds per hyperparameter configuration.The architecture comparison includes an in-house SimBa v1 variant with observation normalization, Pre-LN residual blocks, and branch scaling, evaluated on Spot, ANYmal-D, and Unitree H1.
- Phase-conditioned rank: Phase conditioning exposes a swing-dominant Jacobian-rank split that global averaging obscures.The analysis separates swing and stance to reveal gait-dependent structure in the learned policy.
- Phase-conditioned rank: SimBa-like policies show ∆ϕ = +2.14 versus −0.31 for MLPs, with every SimBa run positive and every MLP run negative.The pooled contrast is statistically strong (Mann–Whitney p = 2.5e−6, Cliff’s d = 0.98), while the same phase structure reappears on the sim-to-real platform.
- Feature rank: SimBa maintains roughly twenty extra dimensions of global feature effective rank over MLPs across every tested robot and on perceptive rough terrain.This gap is distinct from the phase-conditioned Jacobian split and persists across multiple platforms.
- Representation health: Over-training inflates feature rank by +66% while functional rank saturates, collapses the phase split to zero, and leaves standard gait reward largely unchanged.The Big SimBa configuration even shows a deceptive reward increase, whereas healthy training produces structured Gram-matrix geometry and collapsed training loses that structure.
- Sim-to-real transfer: SimBa policies are roughly three times smoother than MLP policies in simulation, and the advantage survives deployment on physical Spot and extends to H1.The reported difference is driven by substantially lower high-frequency joint jitter while matching or improving velocity tracking.
5 Limitations
The authors identify limits in both generalization and methodological isolation. Results were studied in a deliberately narrow flat blind-locomotion testbed, while several architectural and training variables remain unexplored.
- Scope of evidence: Generalization and method isolation are not yet fully characterized.The authors note that available sim-to-real artifacts were used to validate generalizability, but further iterations are needed.
- Experimental scope: The study uses flat blind locomotion as an isolated testbed, limiting how broadly its findings can be interpreted.This design was intended to make the results more expressive within a controlled setting.
- Unexplored factors: RNNs, Transformers, alternative normalizations, hyperparameters, and actor–critic decoupling were not evaluated.The authors identify these omitted variables as directions for future work.
6 Conclusion
The paper concludes that phase-conditioned policy-Jacobian rank reveals representation structure that global averages miss and that this structure persists across simulation and hardware. Effective rank is therefore treated as a diagnostic associated with deployment smoothness, not as an objective to maximize.
- Core finding: Phase-conditioned effective rank exposes architectural structure that phase-agnostic averages obscure.Layer normalization and residual connections allocate roughly two extra rank dimensions to swing, unlike vanilla MLPs.
- Transfer: The higher-rank family transfers roughly three times more smoothly from simulation to hardware.The reported transfer advantage persists in the physical deployment comparison.
- Interpretation: Effective rank should be read as a representation-health diagnostic that covaries with deployment quality rather than optimized directly.Over-training can inflate rank while erasing the phase split, which standard gait reward does not detect.
- Boundary: The authors identify limited generalization as an important boundary and direction for future work.They frame the current findings as a basis for further mechanistic exploration.
A.1 SimBa Implementation Details
The SimBa implementation uses a normalized residual trunk integrated into RSL-RL through two model variants. Residual contributions are depth-scaled, while runtime observation normalization replaces the original trunk’s explicit initial RMSNorm.
- Architecture: The SimBa trunk comprises a linear embedding, N Pre-LN residual blocks, a post-LN, and a linear output head.Each block expands and projects hidden dimension d through W1 and W2 around a ReLU transformation.
- Integration modes: SimbaModel replaces the RSL-RL MLP body while preserving observation grouping, normalization, and distribution infrastructure.This provides a drop-in integration path within the existing MLPModel framework.
- Integration modes: SimbaActorCritic replaces actor and critic networks with separate configurable SimBa trunks.The implementation uses asymmetric sizing: actor d = 256, N = 1; critic d = 512, N = 2.
- Normalization: RSL-RL runtime observation normalization makes an explicit initial RMSNorm redundant.The normalization flag maintains running observation statistics instead of prepending the original SimBa RMSNorm.
B Experiment Details
The experiments provide open-source infrastructure, tune Simba’s learning rate separately from the MLP baselines, and compare architectures across five depths. Models were primarily optimized to maintain healthy representations while keeping performance comparable for the analysis.
- Infrastructure: The training code and multi-thread IsaacLab infrastructure are released as open source.The first version is identified through the linked repository.
- Hyperparameters: Simba uses a learning rate of 1.5 × 10^-3, while MLP rates vary by model size.MLPs use 1 × 10^-3 for the three smallest models and 6 × 10^-4 for the two largest.
- Hyperparameters: Bayesian optimization searches the learning rate for the Simba architecture.Figure 9 displays the top 10 sampled points from a broader iterative exploration.
- Experimental design: The experiments primarily optimize both architectures to maintain healthy representations across multiple depths.Performance comparability is treated as the necessary condition for the analysis.
- Experimental design: Both architectures span five depths while maintaining comparable performance for the analysis.Figure 10 compares the classical MLP baseline and SimBa across task depths and model capacities.
C.1 Deployment Details
The policy is deployed on a physical Spot with externally hosted ONNX inference and telemetry-based phase-conditioned evaluation. Hardware experiments compare locomotion quality across policy configurations using high-resolution recordings and steady-state gait strides.
- Hardware deployment: Inference runs through ONNX on an external Intel i7 desktop connected to Spot via Ethernet, with 50 Hz control and 2–5 ms end-to-end latency.Robot states and actions are exchanged through Spot’s proprietary C++ SDK over gRPC.
- Contact estimation: Spot contact states are reconstructed by thresholding knee torque at 8.0 Nm because the robot lacks explicit binary foot-contact switches.Knee torque is linked to vertical contact force through the leg Jacobian, providing a physics-grounded contact sensor.
- Evaluation protocol: The policy is evaluated under a scripted sagittal velocity profile spanning 0.0 m/s stance, +0.5 m/s forward walking, and −0.5 m/s backward walking.Lateral and yaw commands remain zero throughout the rollout.
- Measurements: Logged trajectories include commands, velocities, gravity, joint states, torques, contacts, foot heights, and phase identifiers for phase-conditioned locomotion metrics.These measurements support velocity tracking, torso stability, cost of transport, and swing/stance gait analyses.
- Hardware results: Ten policy configurations produced 81,313 telemetry steps and 193 steady-state forward gait strides, with tight 95% bootstrap confidence intervals for hardware transfer.The 8.45M-parameter SimBa-XXL policy achieved 0.066 m/s velocity-tracking RMSE versus 0.176 m/s for the 1.89M MLP-XXL baseline, a 62.7% improvement.
C.2 Training Details
Training follows an Isaac Lab-style PPO setup with 200 Hz simulation and 50 Hz policy control, combining locomotion-focused rewards, smoothness and safety penalties, curriculum design, symmetry augmentation, and domain randomization. The objective targets stable command-driven locomotion and mechanical efficiency while excluding arm manipulation objectives.
- Training setup: The simulation runs at 200 Hz while the policy control loop runs at 50 Hz, with flat terrain enabled and episodes capped at 20 s.The setup closely follows Isaac Lab defaults with modifications intended to improve simulation-to-real transfer.
- Curriculum and augmentation: A three-environment curriculum progressively scales selected penalties, especially base-orientation and contact-force terms, while symmetry augmentation mirrors transitions across the sagittal plane.Mirroring doubles the usable batch size without additional simulation time and is intended to reduce overfitting or reward hacking.
- Objective: The reward is a weighted sum of terms designed for stable command-driven locomotion, mechanical efficiency, reduced structural stress, and natural posture.The objective explicitly excludes arm reachability, end-effector trajectory tracking, and force-position manipulation targets.
- Locomotion rewards: Velocity tracking rewards cover planar linear velocity and yaw rate, while gait and foot-clearance terms promote trotting synchronization and 0.1 m swing height.The yaw-rate terms use wider and narrower standard deviations for general and high-precision tracking, respectively.
- Regularization and safety: Smoothness and safety penalties discourage action changes, asymmetric timing, base motion or tilt, foot slip, joint deviations, joint effort, and excessive contact forces.The contact-force penalty is heavily scaled within environmental curriculum stages, while the joint-position penalty increases at standstill.
- Domain randomization: Domain randomization perturbs friction, robot masses, initial states, impulses, and periodic pushes across startup, episode resets, and active rollouts.These perturbations are phased into training using curriculum terms as optimization advances.