PaperlayerView exact revision on arXiv

Source-linked AI summary

SleepWalking: Privileged Representation Shaping for End-to-End Blind Locomotion in Legged Robots

Zheng Pan, Tenghui Wang, Peilin Li, Shiyu Zhou, Hao Sun, Yan Ma, Liang Yu, Liang He

arXiv:2608.30883v1cs.RO

TL;DR

Partially observable locomotion raises whether missing physical variables must be explicitly estimated or can instead be retained in a recurrent history representation. SWAQ uses privileged reconstruction during training to shape that representation while deploying a direct history-to-action controller, with reported performance, representation, and theoretical evidence supporting this approach.

  • Problem

    Partial observability leaves task-relevant terrain properties unavailable before interaction, motivating whether controllers need explicit physical-variable estimation or can recover such information from internal history.

  • Method

    SWAQ jointly learns a recurrent history-based control policy and training-only next-step reconstruction of privileged physical and terrain targets without routing reconstructed outputs into the deployed policy head.

  • Results

    SWAQ retains selected robot-dynamics and coarse terrain information linearly decodable near the action output, while aligned comparisons report higher return and terrain-curriculum progression than non-exteroceptive baselines.

  • Takeaways & Limitations

    Privileged physical supervision can shape what a recurrent history representation retains without making the supervised variables explicit inputs to the deployed controller.

Abstract

from arXiv · show

Partially observable locomotion requires a policy to act when task-relevant properties of the robot--environment state are not fully specified by instantaneous observations. Existing approaches often address this challenge by explicitly estimating missing physical variables or processing extended observation histories through structured architectures. We take a different view: partial observability is fundamentally an information-retention problem. The decisive question is not how task-relevant information enters the network, but whether the policy's internal state retains it. Guided by this perspective, we propose SleepWalking for Robot Locomotion (SWAQ), a one-stage end-to-end framework that uses next-step privileged physical reconstruction to shape what a recurrent history representation retains during policy learning, while the deployed actor uses only a direct history-to-action pathway. Under aligned training settings, SWAQ achieves a 15.0\% higher peak mean terrain level than DWAQ, the strongest non-exteroceptive baseline, while using 44.4\% fewer inference MACs per control step. Layerwise probes further show that information associated with the reconstructed physical variables remains linearly decodable through the policy head up to the layer preceding the action output. Complementary theoretical analysis relates privileged-variable recoverability to the achievable-return gap between history-based and privileged-information policy classes. These results suggest that semantic objectives can structure learning without requiring a corresponding architectural decomposition of the deployed controller.

I. INTRODUCTION

Blind locomotion exposes partial observability because terrain properties are unavailable instantaneously, motivating SWAQ’s information-retention approach and one-stage privileged reconstruction.

  • Motivation: Terrain is important for anticipating stairs, gaps, obstacles, and feasible footholds, but exteroceptive sensing can be noisy, incomplete, unavailable, and costly.Removing exteroception leaves terrain properties indirectly observable through physical interaction.
  • Problem: Partial observability requires history because similar instantaneous observations can correspond to different latent states requiring different actions.Task return alone provides only indirect supervision for the representation.
  • Approach: SWAQ jointly learns control and a recurrent representation from nonprivileged action-observation history, using state-and-terrain reconstruction during training.The reconstructed variables supervise the shared representation without being routed into the controller.
  • Approach: SWAQ imposes semantic task decomposition through training objectives rather than mandatory runtime interfaces, while retaining a direct history-to-action pathway.This contrasts with deployed controllers conditioned on externally specified terrain modes or estimated latent variables.
  • Related work: Prior approaches include teacher-student learning, terrain-mode conditioning, context-aided estimation, attention-based selection, and predictive world-action modeling.These approaches differ in whether information is supplied through sequential training, explicit deployment interfaces, spatial or temporal selection, or predictive objectives.
  • Comparison: One-stage estimator-conditioned methods can still introduce deployment interfaces whose errors or discarded information propagate to the downstream policy.SWAQ instead supervises the representation without feeding reconstructed quantities to the deployed actor.
  • Related work: Auxiliary objectives provide representation-level supervision beyond task reward, while temporally structured self-supervision can produce representations suitable for downstream control.SWAQ applies privileged reconstruction in a one-stage policy-learning framework.

2) POMDP and History-State Formulation:

The paper models blind locomotion as a POMDP and replaces unbounded action-observation history with a finite recurrent representation whose retained information must remain useful for control.

  • 2) POMDP and History-State Formulation:: Blind legged locomotion is formulated as a POMDP because the controller lacks direct exteroceptive measurements of surrounding terrain.The POMDP includes latent state, actions, observations, transition and observation kernels, reward, initial distribution, and discount factor.
  • 2) POMDP and History-State Formulation:: The latent state contains complete robot state and task-relevant environmental variables, whereas observations are primarily proprioceptive and omit local terrain geometry.Different latent states can therefore produce similar instantaneous observations.
  • 2) POMDP and History-State Formulation:: The complete nonprivileged history is represented as τt := (o0, a0, o1, . . . , at−1, ot).This history records observations and previous actions available to the deployed policy.
  • 2) POMDP and History-State Formulation:: Taking τt as the state induces a history-state MDP with transition behavior defined over histories and actions.The induced transition kernel is P^T(B | τ, a) := Pr(τt+1 ∈ B | τt = τ, at = a).
  • 2) POMDP and History-State Formulation:: History-dependent policies maximize expected discounted return under trajectories induced by the initial distribution, transitions, observations, and policy.The policy class is denoted Πhist, and its objective is expected discounted return.
  • 2) POMDP and History-State Formulation:: The key representation challenge is compressing growing history while preserving control-relevant information without adding deployment observations.This motivates finite-dimensional recurrent history representations.
  • 3) Action and Observation Spaces:: The actor receives proprioceptive measurements including base angular velocity, velocity command, joint positions and velocities, previous action, and gait phase.Actions specify relative targets for actuated joints and are converted to torque by a PD controller.
  • 4) Reward Design:: The adopted reward structure retains phase-based contact terms, task-tracking terms, and motion-regularization terms for locomotion learning.Swing penalizes vertical contact forces and stance penalizes tangential foot velocities.

B. SWAQ Architecture

SWAQ uses a recurrent history representation shaped by training-only prediction of next-step privileged physical and terrain variables. The deployed controller retains only the direct history-to-action pathway, unlike estimator-conditioned designs that expose intermediate representations at deployment.

  • Architecture: SWAQ uses a recurrent history encoder and shared trunk for both direct action control and training-only privileged reconstruction.The reconstruction pathway predicts next-step physical targets from the shared representation, but its outputs are not routed to the policy head.
  • Architecture: An LSTM recursively compresses the growing observation-action history into a finite-dimensional representation h_t.This avoids representing the complete history explicitly while preserving a recurrent decision state.
  • Training targets: Next-step targets combine robot base velocity with a privileged terrain height scan generated from simulator transitions.The realized action at is available as the previous action at decision epoch t+1, while the completed transition is used only during training.
  • Training targets: Predicting next-step quantities encourages the history representation to retain information predictive of ensuing robot-environment transitions rather than merely contemporaneous information.Robot-state targets provide physical grounding, while terrain targets promote aggregation of terrain evidence revealed through interaction.
  • Training rationale: The reconstruction loss requires the finite-dimensional representation to retain predictive information because a constant representation can only achieve a constant-predictor variance baseline.The objective therefore penalizes complete representation collapse when reconstruction error falls below that baseline.
  • Design comparison: SWAQ shapes policy-internal representations directly, whereas DWAQ supplies estimated velocity and latent terrain context through an explicit downstream control interface.SWAQ preserves only the history-to-action pathway during deployment; DWAQ and related estimator-conditioned designs expose intermediate estimates to deployed control.

3) Joint Representation-Policy Optimization:

SWAQ jointly optimizes policy learning and privileged reconstruction so that the recurrent encoder is shaped by both objectives. The experiments probe recoverability, persistence along the control pathway, theoretical return relations, and locomotion effects using aligned comparisons and ablations.

  • Joint optimization: SWAQ combines PPO with a weighted state-and-terrain reconstruction loss in a single joint training objective.The objectives are evaluated on the same rollout minibatches and combined before backpropagation.
  • Joint optimization: The reconstruction objective updates the recurrent encoder and reconstruction head, while PPO updates the encoder, policy head, and critic.Thus, the encoder receives coupled policy and privileged-reconstruction gradients, whereas reconstruction does not directly update the policy head or critic.
  • Experimental setup: SWAQ trains 4096 simulated robots with progressive terrain curriculum across stairs, rough ground, and slopes.The simulation uses a 5 ms physics step and a 20 ms control step, corresponding to 50 Hz control.
  • Experimental setup: The actor uses a single-layer 256-unit LSTM, a 256-unit shared MLP layer, and policy-head MLP layers of 256 and 128 units.The auxiliary decoder has hidden layers of 128 and 256 units, while the critic uses a separate 256-unit LSTM and three MLP layers.
  • Evaluation: Experiments test semantic recoverability, persistence through the control pathway, theoretical links to achievable return, and policy-performance effects.Controlled ablations and aligned policy comparisons quantify the contribution of reconstruction components and the complete SWAQ design.
  • Evaluation: Reconstructed terrain follows coarse terrain profiles across level ground and stair ascent and descent, though stair edges are smoothed.The paper attributes this smoothing to pointwise squared-error regression and reports that coarse information remains sufficient for tested locomotion.

2) Autoregressive Future-state Visualization:

Training-only diagnostics show how recurrent history representations reconstruct terrain and support autoregressive state rollouts, while preserving robot and terrain information through the control pathway.

  • Autoregressive Future-state Visualization: Reconstructed height samples follow terrain profiles across level ground, stair ascent, and stair descent, but smooth abrupt stair-edge discontinuities.The reconstruction is less accurate around stair edges, consistent with pointwise averaging.
  • Autoregressive Future-state Visualization: A one-second decoder-actor rollout recursively feeds reconstructed observations back to the recurrent actor for 50 control steps without ground-truth feedback.The rollout begins from the recurrent state induced by real observation history; reconstructed terrain is not supplied to the actor.
  • Autoregressive Future-state Visualization: Initial foot-riser contact destabilizes the reconstructed rollout, whereas steady stair ascent preserves forward progression and alternating leg control.The diagnostic does not predict that the physical robot will fall because reconstruction errors accumulate outside the one-step training distribution.
  • Contact-Triggered Representation Update: At first contact, a 0.79 kN left-foot force coincides with reconstructed grade of 0.175 rad versus approximately 0.3 rad ground truth and locally maximal height-map error.The collision provides new terrain evidence, but a single interaction does not determine local geometry accurately.
  • Contact-Triggered Representation Update: Subsequent contacts refine the terrain estimate: reconstructed grade transiently exceeds 0.5 rad, then approaches ground truth as stable support is established and height-map error decreases.A coherent ascending structure appears by tc+0.52 s, and convergence toward ground truth follows left-foot support at tc + 1.0 s.
  • Information Retention Along the Control Pathway: At Hidden 2, reconstructed targets remain linearly decodable through the action pathway, including robot state, policy observations, height scans, and forward grade.Examples include reconstructed aggregate height scan R2 0.92, height features R2 0.85, and forward grade R2 0.93.

C. Training Performance and Curriculum Progression

SWAQ improves blind locomotion training and curriculum progression while retaining a lightweight recurrent deployment pathway. Ablations and cross-embodiment trials support the role of complementary reconstruction objectives, though baseline-architecture differences limit isolation of the mechanism.

  • SWAQ reaches the high-return regime early, continues improving, and finishes with higher return than all comparison policies.
  • SWAQ achieves higher terrain levels than non-exteroceptive baselines while maintaining velocity-tracking performance close to height-map PPO-LSTM.
  • Removing both robot-state and terrain reconstruction causes the largest and most persistent degradation in return, curriculum progression, and velocity tracking.Removing either component alone has a smaller effect, indicating complementary reconstruction targets support the complete model.
  • SWAQ uses 0.479 M inference MACs per control step, compared with 6.583 M for the causal Transformer processing an explicit 16 × 50 history window.The recurrent policy instead consumes a 50-dimensional per-step input and stores historical information in a fixed-dimensional state.
  • Interaction-driven stair traversal appears on both humanoid and Go1 robots despite different morphologies, contact sequences, and body geometries.The policies use embodiment-specific observation and action spaces while retaining the same one-stage auxiliary-learning design and proprioception-only deployment setting.
  • Layerwise probes find selected robot-dynamics and coarse terrain information linearly decodable through the layer preceding the action output.The conclusion frames this as privileged physical supervision shaping recurrent information retention without making those variables explicit controller inputs.

A. Preliminaries and Notation

The preliminaries align next-step reconstruction targets with decision epochs and define recurrent observation-history policy classes on a common augmented Markov process. The analysis compares these classes under explicit measurability and policy-sufficiency assumptions.

  • The auxiliary pathway predicts the next-step target Y t+1 from Ht, with k = t+1 denoting the succeeding decision epoch.
  • The target Y k is formed from simulator variables available at the beginning of decision epoch k, after the preceding transition and before Ak is selected.The previous-action entry is Ak−1, and reset conventions initialize the initial target, previous action, and carried history.
  • At decision epoch k, the recurrent actor updates Hk from Hk−1 and the current nonprivileged observation Ok, then generates the action from Hk.The complete decision rule is therefore a measurable function of (Ok, Hk−1).
  • The analysis uses a common augmented pre-action state containing the latent robot-environment state and complete nonprivileged history, making the process Markov.
  • The induced policy classes distinguish latent-state Markov policies, observation-only policies, and history-conditioned policies defined through measurable coordinate mappings.The policy-class comparison relies on the assumed sufficiency of Markov policies in the fully observed latent-state MDP.

B. Information Retention and Exact Policy Emulation

The paper formalizes representation retention as the key issue in history-based control and shows that exact privileged-variable recovery can support emulation by hidden-state policies. Additional privileged information cannot reduce optimal return, while exact recovery preserves the nonprivileged achievable return under expressivity assumptions.

  • Information Retention: Reconstruction accuracy implies that privileged-variable information is retained in the shared representation.The result applies to temporally aligned hidden representations and reconstructed targets.
  • Scope: The information-retention result concerns the representation as a whole, not independently disentangled physical quantities in individual coordinates.The return-preservation result also depends on the hidden-state policy class being sufficiently expressive; restricted networks may incur approximation error.
  • Policy Classes: Additional privileged information cannot decrease the optimal achievable return because a privileged policy can ignore it.The proposition establishes only a non-strict inequality.
  • Exact Emulation: Exact recovery lets a sufficiently expressive hidden-state policy exactly emulate every privileged-information policy.The construction composes the privileged policy with the recovery mapping from the preceding hidden representation.
  • Exact Emulation: Exact privileged-variable recovery preserves the nonprivileged policy class’s achievable return under the stated assumptions.This follows by combining exact emulation with the ordering between privileged and nonprivileged policy classes.

C. Approximate Policy Emulation and Return Guarantees

The approximate-recovery analysis extends exact policy emulation by bounding hidden-state emulation error under regularity, reward, and distribution-coverage assumptions. It then relates reconstruction quality to a bound on the achievable-return gap.

  • Assumptions: Exact recovery is generally unavailable in learned systems, motivating an approximate-recovery analysis under additional regularity assumptions.The assumptions include bounded rewards, a Lipschitz privileged-information policy, and training-distribution coverage.
  • Approximate Emulation: Theorem A.4 bounds privileged-policy emulation error when reconstruction error, policy sensitivity, and occupancy-distribution mismatch satisfy the stated conditions.The reconstruction-induced hidden-state policy must also belong to the hidden-state policy class.
  • Approximate Emulation: The proof transfers reconstruction guarantees from the training distribution to the privileged policy’s aligned occupancy distribution before translating recovery error into action-distribution and return differences.The argument uses bounded density ratios, Lipschitz continuity, total variation, and the performance-difference lemma.
  • Return Guarantees: Corollary A.5 gives an achievable-return bound for hidden-state policies under approximate privileged-variable recovery.The result is stated for a restricted policy class with finite Lipschitz and coverage constants.

A. Supplementary Quadruped Representation Probes

Supplementary Go1 probes test whether the representation-retention pattern transfers across body geometry and contact dynamics. Reconstructed variables remain decodable near the action output, with terrain-related and robot-related targets showing distinct patterns.

  • Probe Setup: The frozen-actor ridge-probe protocol evaluates whether representation retention persists on the Unitree Go1.The test changes body geometry and contact dynamics relative to the humanoid setting.
  • Robot Targets: Aggregate proprioceptive observations and previous actions remain strongly decodable through Hidden 2.The figure reports held-out linear-probe R2 across successive control-pathway layers.
  • Terrain Targets: Reconstructed terrain quantities are more decodable than ground-truth counterparts, with forward grade retained most strongly.Negative R2 for the ground-truth signed step edge indicates performance below a test-set-mean predictor.
  • Cross-Platform Comparison: At Hidden 2, ground-truth projected gravity reaches 0.85 on Go1 versus 0.69 on the humanoid, while reconstructed height-scan decodability is 0.72 versus 0.92.Forward grade remains the most decodable scalar terrain descriptor on Go1.
Loading 2608.30883v1…