Source-linked AI summary
Remember Where You Walk: Learning Spatial Memory for End-to-End Legged Robot Exploration
Leon Magnus, Nico Bohlinger, Jan Peters, Oleg Arenz
TL;DR
Legged-robot exploration requires remembering environment information that leaves the field of view, while recurrent and finite-history approaches have limitations in retaining spatially distinct or older information. LSMM uses position-indexed local memory for end-to-end maze exploration. It matches baseline coverage and is the only architecture reported to reduce maze-reconstruction error by more than threefold.
Problem
Autonomous legged-robot exploration requires choosing where to go and remembering previously visited areas, but recurrent states can mix or overwrite spatial information and finite histories discard older observations.
Source: §I · p. 1Method
LSMM stores learned information in a position-indexed memory, updating a local window and jointly training the memory and policy with maze reconstruction as an auxiliary objective.
Source: §I · p. 1, §III · p. 2, C. Local Spatial Matrix Memory · p. 3Results
0.33 to 0.1 reconstruction error: LSMM alone significantly reduces error while exploring, more than three times lower than any baseline, while all architectures achieve comparable 65.4 to 82.8 % coverage.
Source: A. Simulation · pp. 4–5Takeaways & Limitations
In these mazes, LSMM builds a persistent layout representation and spreads visits more evenly, although recurrent and finite-history baselines achieve comparable pure coverage.
Source: A. Simulation · p. 4, §VI · p. 7LSMM uses a fixed, maze-sized memory and depends on an external position estimate; onboard-only localization is not yet demonstrated.
Source: §V · pp. 6–7
Abstract
from arXiv · showhide
Humanoids and other legged robots operating in complex environments must remember information that leaves their field of view and only becomes relevant much later. Reinforcement learning policies address this using memory networks such as LSTMs that are trained end-to-end with the policy to memorize such task-relevant aspects. Yet, such architectures often fail to preserve sufficient information over long horizons. In this work, we propose a network architecture based on Hadamard memory that specifically targets robot navigation. Our Local Spatial Matrix Memory (LSMM) enables RL agents to store generic task-relevant information at specific, position-based memory cells, enabling flexible and spatially grounded memorization. We evaluate LSMM on the task of autonomous maze exploration, where it matches the maze coverage of strong recurrent and finite-history baselines while being the only architecture that remembers the maze layout, reducing the reconstruction error threefold.
I. INTRODUCTION
Autonomous legged-robot exploration requires deciding where to go and remembering places that are no longer visible. LSMM uses position-indexed memory to address this challenge, and maze experiments evaluate both exploration and layout reconstruction.
I. INTRODUCTION
- Exploration requires a legged robot to choose where to go and remember where it has already been, beyond local-frame reasoning and control.The paper frames this as a long-horizon, partially observable navigation problem.Source: §I · p. 1
- Unstructured recurrent states can mix or overwrite spatial information, while finite-history approaches discard older observations and classical maps often separate mapping, planning, and locomotion.The cited discussion identifies these as limitations of existing memory and navigation approaches.Source: §I · p. 1
- LSMM aligns matrix memory with the environment, updates a small position-centered window, and uses a CNN encoding for both policy features and maze reconstruction.The architecture combines spatially indexed updates with learned representations and end-to-end locomotion control, without explicit path planning or waypoint commands.Source: §I · p. 1
- The work evaluates end-to-end maze exploration with recurrent and finite-history comparisons in simulation and real-world deployment, including reconstruction, efficiency, localization-noise robustness, and generalization.The experiments use the Unitree Go2 and procedurally generated mazes.Source: §I · pp. 1–2
II. RELATED WORK
Prior work addresses partial observability with temporal memories, spatial neural maps, or navigation pipelines with separate planners. LSMM combines spatial indexing with jointly learned memory and locomotion, without an analytical planner.
II. RELATED WORK
- Recurrent and attention-based methods maintain temporal memory, but aggregate observations without explicitly linking stored information to its spatial location.The related-work discussion describes the Hadamard Memory Framework as calibrating previous memory element-wise and adding new information.Source: §II · p. 2
- Neural Map, MapNet, and Active Neural SLAM use spatial memories or maps; LSMM instead feeds spatial memory features directly into locomotion without Active Neural SLAM’s analytical planner.LSMM updates a window around the estimated position, jointly trains memory and policy, and reconstructs maze layout through an auxiliary head.Source: §II · p. 2
- Classical and learning-based navigation approaches retain external target selection, whereas LSMM replaces the occupancy-map role and learns the next movement jointly with locomotion.The cited work describes frontier targets as selected by an external planner or operator.Source: §II · p. 2
III. SPATIAL MEMORY FOR END-TO-END LEGGED ROBOT EXPLORATION
- The paper proposes a single policy that remembers the environment, chooses where to go, controls the robot’s joints, and trains with an auxiliary maze-reconstruction objective.The stated training approach uses PPO and joint optimization of memory, encoders, and policy.Source: §III · p. 2
A. Problem Formulation
The task is goal-free exploration of unseen mazes, where the policy must remember visited areas that current observations do not reveal. Its actor receives proprioceptive and terrain observations alongside spatial memory, while training also uses a privileged critic and maze-reconstruction objective.
A. Problem Formulation
- The Go2 explores an unseen 10 × 10 m maze for 70 seconds, without a goal, waypoint, or velocity command at deployment.The robot starts at the maze center and must visit as many cells as possible; the paper models this as a partially observable decision process.Source: A. Problem Formulation · p. 2
- The observation includes a waypoint command used only in early training, proprioceptive state, 250 terrain-height samples, and an estimated position and heading.The proprioceptive state includes joint positions and velocities, previous action, trunk angular velocity, and projected gravity.Source: A. Problem Formulation · p. 2
B. Policy Architecture
- The memory branch uses proprioception, estimated velocity, and position to update LSMM; a CNN then extracts features from the updated memory.The figure caption specifies noisy position and velocity estimates for the memory update.Source: B. Policy Architecture · p. 2, Figure 2 · p. 3
- The actor combines its observation and memory features to produce a Gaussian action distribution from which actions are sampled.The policy parameters include the actor and the memory and encoder functions.Source: B. Policy Architecture · pp. 2–3
- Training uses a memoryless critic with privileged observations and an auxiliary head that predicts maze walls from actor features; both are dropped at deployment.The reconstruction objective requires memory to retain information gathered earlier in the episode.Source: B. Policy Architecture · p. 3
C. Local Spatial Matrix Memory
LSMM stores learned features in an environment-aligned spatial tensor and updates only a local window around the position estimate. This preserves information at unvisited locations until the robot returns.
C. Local Spatial Matrix Memory
- The 50 × 50 memory grid has three learned-feature channels, with each 0.2 × 0.2 m cell initialized to 0.5.Its first two dimensions align with the environment frame.Source: C. Local Spatial Matrix Memory · p. 3
- At each step, LSMM updates a 5 × 5 window centered on the estimated position, leaving locations outside that window untouched.The window covers 1 × 1 m around the robot; two MLPs compute calibration and update tensors from its current contents.Source: C. Local Spatial Matrix Memory · p. 3
- The calibration tensor acts as a forget gate and the update tensor writes new information; normalization keeps window values in [0, 1].Both tensors are sigmoid-bounded, and the normalization uses ϵ = 0.02.Source: C. Local Spatial Matrix Memory · p. 3
- A written cell retains its latent exactly until the robot revisits that location, regardless of elapsed time.Calibration lets the policy choose how much stored information to retain on a revisit.Source: C. Local Spatial Matrix Memory · p. 3
D. RL Training Setup in Simulation
Policy and critic training uses PPO across parallel MJX environments, with maze reconstruction and velocity estimation added to the objective. Maze-reconstruction gradients reach the memory write functions, and a high discount factor supports long-horizon exploration.
D. RL Training Setup in Simulation
- PPO training uses 5,000 parallel MJX environments and 300,000 pregenerated maze layouts.The full objective combines PPO’s clipped-surrogate and value losses with maze-reconstruction and velocity-estimation losses.Source: D. RL Training Setup in Simulation · p. 3
- Maze-reconstruction gradients reach the memory write functions through the memory encoder, and training uses γ = 0.995 for long-horizon credit assignment.The reconstruction targets encode which possible maze wall segments exist; the auxiliary velocity target is the simulated trunk’s linear velocity.Source: D. RL Training Setup in Simulation · p. 3
E. Rewards
Exploration rewards favor rarely visited cells and overall maze coverage, while shaping penalties address gait and navigation. Evaluation reports comparable explored-cell coverage across architectures.
E. Rewards
- Exploration episodes reward visits to rarely visited cells, with the cell reward decaying over time spent there, and reward overall maze coverage.Goal episodes instead reward movement toward and arrival at the commanded cell.Source: E. Rewards · p. 3
- All architectures reach comparable explored-cell coverage at episode end.The figure reports mean coverage over 100 unseen evaluation mazes and three seeds per architecture, with standard-deviation whiskers.Source: Figure 3 · p. 4
- Shaping penalties address stalling, wall proximity, joint limits, torques, action rates, and asymmetric gaits.Source: E. Rewards · p. 4
F. Curricula
Training uses a performance-based curriculum and extensive domain randomization, while evaluation compares memory architectures on unseen mazes. LSMM spreads its visits most evenly across the maze.
F. Curricula
- A performance-based curriculum increases exploration episodes and shaping penalties when coverage thresholds are met, while varying maze-wall complexity.At training’s end, effectively all episodes are exploration episodes, and the deployed policy does not depend on an external goal.Source: F. Curricula · p. 4
G. Domain Randomization
- Training randomizes robot, environment, actuation, and initial-state properties, and adds noise to observations and position estimates.Position estimates are perturbed by up to ±0.20 m, corresponding to one memory cell.Source: G. Domain Randomization · p. 4
IV. EXPERIMENTAL RESULTS
- LSMM spreads visits most evenly across mazes, with the lowest divergence from a uniform visit distribution.The figure defines lower D_KL(p ∥u) as more even exploration.Source: Figure 4 · p. 4
- Evaluation uses three seeds per architecture across 100 previously unseen mazes, with baselines matched for observations, auxiliary losses, and hardware budget.The budget permits hidden states of size 1,000 for LSTM and SRU, and 10 recent observations for the Transformer and fixed-window MLP.Source: §IV · p. 4
A. Simulation
In simulation, LSMM matches other architectures’ maze coverage while uniquely reducing reconstruction error; ablations show that additive updates and a single memory channel suffice for static maze layouts.
A. Simulation
- All architectures reached comparable final explored-cell coverage of 65.4 to 82.8%.LSMM nevertheless spread visits most evenly, with KL divergence to uniform of 0.008, versus 0.015 for SRU, 0.018 for LSTM, and 0.019 for Transformer; the fixed-window model concentrated visits near the start.Source: A. Simulation · pp. 4–5
- LSMM reduced maze reconstruction error from 0.33 to 0.1, more than three times lower than any baseline.It was the only architecture to significantly reduce error while exploring; its predicted map filled in wall by wall as it moved.Source: Figure 5 · p. 5, A. Simulation · p. 5, Figure 6 · p. 5
- LSMM’s auxiliary head predicted spatially varying environment properties with lower error than random guessing.The properties included gravity, ground friction, roughness, and ground damping.Source: A. Simulation · p. 5
B. LSMM Memory Ablation
- For static mazes, additive memory updates reduced reconstruction error similarly to LSMM’s calibrated updates.The authors suggest calibration is unnecessary here because walls do not need revision once written.Source: B. LSMM Memory Ablation · p. 5
- Reconstruction performance was largely insensitive to memory size across 1, 2, 3, 5, and 10 channels.The authors suggest one channel per cell is sufficient to encode the wall layout.Source: B. LSMM Memory Ablation · p. 5, Figure 7 · p. 6
C. Real World
LSMM explored an unseen outdoor maze autonomously on a Unitree Go2. Reconstruction error declined similarly to simulation, with faster reduction attributed to the smaller maze.
C. Real World
- The Go2 explored an unseen 5 × 5 m outdoor maze without a goal, waypoint, or velocity command.It ran the policy onboard and used an externally tracked AprilTag pose estimate.Source: C. Real World · p. 6
- Deployment visits were less uniform and more repetitive than in simulation, but covered enough maze to reduce reconstruction error similarly.The unvisited regions also received no matrix-memory updates.Source: Figure 9 · p. 6, C. Real World · p. 6
- Deployment reconstruction error declined similarly to simulation and faster because the smaller maze exposed a higher fraction of its layout early.The deployment figure shows significant error reduction within the first 30 seconds.Source: C. Real World · p. 6, Figure 10 · p. 6
- Corner-cutting and wall contact required strong penalties in the fragile maze; with solid walls, mild contact might aid reorientation.The authors describe contact as a possible way to ground predictions, provided impact forces are mild.Source: C. Real World · p. 6
V. LIMITATIONS & FUTURE WORK
LSMM matches baseline maze coverage while preserving and reconstructing maze layouts, but its fixed memory extent and reliance on an external position estimate constrain its current scope.
V. LIMITATIONS & FUTURE WORK
- Unknown-size environments require growing or shifting the fixed, maze-sized memory, and updates depend on an external position estimate.Training with ±0.20 m of localization noise makes the policy tolerate imperfect localization, but it does not yet estimate position from onboard sensing alone.Source: §V · pp. 6–7
- In flat, small mazes, reactive strategies reach similar coverage, and the authors expect spatial memory's exploration benefit to grow with environment size.Future work targets larger, rougher environments, internal position estimation, and using reconstructed maps to navigate back to previously seen locations.Source: §V · p. 7
VI. CONCLUSION
- LSMM matches recurrent and finite-history baselines in maze coverage while reducing reconstruction error by a factor of three.The baselines explore reactively and forget the layout they have seen, whereas LSMM builds a persistent map.Source: §VI · p. 7
- The deployed policy explores a real outdoor maze without an explicit map, planner, or waypoint commands.LSMM also spreads visits most evenly over the maze and stores physical properties of the environment.Source: §VI · p. 7
APPENDIX
The appendix specifies shared learning hyperparameters and the reward-term aggregation used in the experiments.
APPENDIX
- All memory architectures use the same learning hyperparameters for a fair comparison.Table I lists those hyperparameters.Source: APPENDIX · p. 7, Table I · p. 7
- Reward terms are coefficient-scaled, summed, and clipped below at zero before the constant alive bonus is added.Table II lists the reward terms and coefficients.Source: APPENDIX · p. 7, Table II · p. 7
- Table II defines notation for exploration or goal episodes, steps in the current cell, wall-side offsets, and airborne-leg measures.Its caption defines E/G, ncell, δi, nair pairs, and tair.Source: Table II · p. 7