PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Scene Representation for Locomotion on Structured Terrain

David Hoeller, Nikita Rudin, Christopher Choy, Animashree Anandkumar, Marco Hutter

arXiv:2206.08077v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Robots must estimate nearby terrain from noisy, partial depth measurements that can omit foothold regions. This paper combines current observations with previous scene estimates in a learning-based reconstruction model. In experiments, it outperforms baselines on most reported metrics and supports locomotion in structured urban settings.

  • Problem

    Terrain mapping for locomotion is challenged by noisy depth observations, occlusions, localization drift, and camera blind spots near footholds.

    Source: §I · p. 1

  • Method

    A learning-based 4D point-cloud network combines partial camera observations with prior estimates to reconstruct occluded scene regions using spatial context and temporal feedback.

    Source: §I · p. 2, B. Architecture · p. 3

  • Results

    The method outperforms the other real-robot baselines by around 10% in most metrics, with more correctly detected points and lower mean absolute reconstruction error.

    Source: B. Evaluation on the Robot · p. 7

  • Takeaways & Limitations

    Simulation and real-robot experiments show the module accumulates spatial and temporal evidence to estimate structured terrain that locomotion controllers can use in urban settings.

    Source: §V · p. 7

    Some points from dynamic obstacles remain in the reconstruction and vanish only slowly.

    Source: §V · p. 7

Abstract

from arXiv · show

We propose a learning-based method to reconstruct the local terrain for locomotion with a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the algorithm estimates the topography in the robot's vicinity. The raw measurements from these cameras are noisy and only provide partial and occluded observations that in many cases do not show the terrain the robot stands on. Therefore, we propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement. The model consists of a 4D fully convolutional network on point clouds that learns the geometric priors to complete the scene from the context and an auto-regressive feedback to leverage spatio-temporal consistency and use evidence from the past. The network can be solely trained with synthetic data, and due to extensive augmentation, it is robust in the real world, as shown in the validation on a quadrupedal robot, ANYmal, traversing challenging settings. We run the pipeline on the robot's onboard low-power computer using an efficient sparse tensor implementation and show that the proposed method outperforms classical map representations.

I. INTRODUCTION

Robots need accurate local terrain maps despite noisy, occluded depth measurements and localization drift, including blind spots near footholds. The paper proposes a learning-based reconstruction method that uses scene context and past estimates, trained on synthetic data and evaluated in simulation and on a real robot.

I. INTRODUCTION

  • Noisy depth observations, occlusions, localization drift, and camera blind spots make terrain mapping difficult, while existing mapping methods rely on tuning heuristics.The paper notes that a prior recurrent map representation is tied to its locomotion controller and cannot be reused by another module.Source: §I · p. 1
  • The method reconstructs the surrounding scene from robot pose, partial camera observations, and the latest map estimate, using context to infer occluded regions.Its autoregressive feedback maintains geometry from previous inputs and can extend partially observed stairs or remember objects hidden from view.Source: §I · pp. 1–2
  • The network is trained only on synthetic scenes with simulated depth noise and extensive augmentation, then transfers to real-world sensory data.The training scenes include walls, roadblocks, stairs, and boxes, with more than 200,000 point-cloud observations.Source: §I · p. 2
  • The paper focuses on structured urban terrain and does not claim to solve the complete 3D rough-terrain problem.Source: §I · p. 2
  • The method is designed to reconstruct structured urban environments under noisy point clouds, missing stereo data, and imprecise trajectory estimates.It initializes maps with a meaningful context-based guess without heuristics.Source: §I · p. 2
  • Simulation and real-robot evaluation show the method handles estimator drift and missing data and outperforms heuristic-based baselines.The paper identifies the deployment setting as challenging real-world locomotion.Source: §I · p. 2

II. RELATED WORK

Prior locomotion and mapping approaches either couple perception to a controller or have representational and drift limitations. This work uses learning-based completion of unseen scene regions while handling odometry drift.

a) Locomotion

  • Perceptive locomotion models can tie vision to a specific policy, requiring retraining for each new policy; this method instead works as a standalone module for different policies.The paper reports compatibility with both model-based and learning-based policies.Source: §II · p. 2

b) Environment Mapping

  • Elevation maps cannot represent full 3D scenes and are susceptible to odometry drift; a neural hole-filling extension still uses only the current time step.The cited elevation-map approach fuses range measurements and robot pose with a Kalman filter.Source: §II · p. 2
  • Classical 3D mapping methods lack semantic completion of missing information, whereas this method learns to complete unseen scene parts and handle odometry drift.Its voxel representation stores relative point coordinates for sub-voxel accuracy.Source: §II · p. 2

c) 3D Scene Completion

  • The terrain module transforms the previous point-cloud estimate into the current frame, concatenates it with the current measurement, and processes both with a fully convolutional encoder-decoder.The architecture uses skip connections to produce the current point-cloud estimate.Source: Figure 2 · p. 3
  • Prior scene-completion systems include dense voxel methods, hierarchical autoregressive completion, and RGB-based fusion without recurrence to retain past information.The paper also adopts voxel upsampling and pruning to produce high-resolution reconstructions while reducing computational complexity.Source: §II · p. 3

III. METHODOLOGY

The terrain-reconstruction pipeline combines the current point-cloud measurement with the previous estimate aligned to the current robot frame. A fully convolutional network then produces the current scene estimate.

III. METHODOLOGY

  • The previous point-cloud estimate is transformed into the current measurement frame using the pose difference between time steps.Source: §III · p. 3
  • The aligned previous estimate is concatenated with the current measurement and fed to a fully convolutional network to produce the current point-cloud estimate.Source: §III · p. 3
  • The network’s output becomes the estimate used by the pipeline at the current time step.Source: §III · p. 3

A. Input Pre-processing

The method voxelizes point clouds into a sparse representation to avoid the memory demands of a dense grid. Each occupied voxel stores a centroid offset, and temporal indexing lets the network process measurements across time.

A. Input Pre-processing

  • A sparse voxel-grid representation stores relative point coordinates for sub-voxel accuracy while avoiding the cubic memory cost of a dense grid.Source: A. Input Pre-processing · p. 3
  • The input uses a 64×64×64 grid spanning a 3.2 m×3.2 m×3.2 m map around the robot, with each occupied voxel represented by its centroid offset from a voxel corner.Source: A. Input Pre-processing · p. 3
  • A discrete time index distinguishes current-measurement points from points in the previous output, enabling 4D convolutions across space and time.The feature values encode normalized centroid coordinates within each voxel.Source: A. Input Pre-processing · p. 3

B. Architecture

The model uses a sparse, fully convolutional 4D encoder-decoder to reconstruct point clouds, with skip connections preserving detail. Its training objective combines voxel-occupancy supervision with sub-voxel position estimation, while pruning threshold α trades output density against confidence.

B. Architecture

  • The fully convolutional 4D U-Net-like encoder-decoder uses skip connections to preserve fine details from the encoder.Source: B. Architecture · p. 3
  • Four strided convolutions down-sample each spatial axis by 16 while preserving the temporal dimension for space-time convolutions at multiple resolutions.Source: B. Architecture · p. 3
  • The decoder forwards encoder feature maps through skip connections and upsamples tensors back to the input stride, using pruning to avoid dense intermediate grids.The latent tensor could otherwise expand to a fully occupied 64×64×64×1 grid, which the authors say would be prohibitively slow.Source: B. Architecture · pp. 3–4

C. Training Procedure

  • Training combines binary cross-entropy for occupied voxels at each layer with mean Euclidean distance for sub-voxel point positions.Voxel confidence is represented by the feature value in Tp.Source: C. Training Procedure · p. 4
  • At α = 0.5, pruning gives a good experimental trade-off between reconstruction precision and recall.Higher α is more conservative and can create holes; lower α produces denser outputs but may add points absent from the target.Source: C. Training Procedure · p. 4
  • Training rolls the network out for 12 time steps, and propagating gradients over time did not improve reconstruction performance experimentally.Source: C. Training Procedure · p. 4

D. Data Generation

The authors generate structured urban-like scenes in simulation and collect robot-mounted depth observations with ground-truth point clouds. Noise and data augmentation train the network to denoise and complete partial measurements, including by using evidence from earlier observations.

D. Data Generation

  • IsaacGym scenes include randomized stairs, boxes, walls, poles, and narrow corridors, encouraging the network to learn structure from urban-like environments.The scene parameters are randomized to reflect real-world diversity.Source: D. Data Generation · p. 4
  • The dataset contains 200,000 time steps, with only 43% of ground-truth points visible in measurements on average.Four simulated depth cameras are placed around the robot, matching the standard ANYmal C configuration.Source: D. Data Generation · p. 4

E. Data augmentation

  • Training measurements are noisified and augmented with point perturbations, tilt, altered patch heights, pruning, and added outlier clusters.Source: E. Data augmentation · p. 4
  • Robot-pose perturbations and trajectory mirroring further diversify the training data, while the network learns denoising and completion from past measurements.Source: E. Data augmentation · p. 4

IV. EXPERIMENTS

The experiments compare the approach with Elevation Mapping and Voxblox using simulation and real-robot validation. They assess voxelized reconstruction quality and height accuracy, which the authors identify as important to locomotion.

IV. EXPERIMENTS

  • The evaluation compares the approach against Elevation Mapping and Voxblox in simulation and on the real robot.The baselines have more than ten hyper-parameters and heuristics, while the approach requires tuning only pruning threshold α.Source: §IV · p. 5
  • Reported metrics are mean precision, recall, and F1 score on a robot-centric 64×64×64 voxel grid, plus mean absolute height difference from ground truth.Source: §IV · p. 5
  • The voxel grid uses 0.05 m cells, and height difference is included because the locomotion policy directly uses height information.Source: §IV · p. 5

A. Evaluation in Simulation

In simulation, the reconstruction retains previously observed objects after they enter camera blind spots, while spatial input alone can recover stair structure. Performance under measurement removal is strong up to a reported level, but degrades as observations become sparser.

A. Evaluation in Simulation

  • With 50% of measurement data omitted, the network achieves an F1 score of 88%; at that sparsity, recall is 62% and precision is 82%.The removal rate excludes points already absent because of camera blind spots.Source: A. Evaluation in Simulation · p. 5
  • Above 80% data omission, decoder sparsity handling fails and pruning produces empty tensors; recall falls as output density decreases.The authors report that holes appear as output density drops.Source: A. Evaluation in Simulation · p. 5
  • The spatial formulation preserves full 3D coordinates, unlike image convolutions that may treat neighboring pixels as near despite distant 3D points.Source: A. Evaluation in Simulation · p. 5
  • Two boxes no longer visible 1.5 seconds later remain correctly reconstructed using evidence from earlier measurements.The reconstruction also recreates a wall from only its visible bottom; all training walls share the same height.Source: Figure 4 · p. 5, A. Evaluation in Simulation · p. 5
  • Using only the current measurement, the network reconstructs vertical stair surfaces from spatial data alone.The authors note temporal information is necessary for elements such as the boxes in the trajectory example.Source: Figure 5 · p. 5, A. Evaluation in Simulation · p. 5

B. Evaluation on the Robot

On-robot evaluation compares the reconstruction with classical mapping baselines across stairs and a reflective wooden-box scene. The method remains robust to substantial measurement removal and state-estimator drift, with real-time map updates on the robot.

B. Evaluation on the Robot

  • Mean validation performance stays constant through 50% measurement removal, after which recall diminishes rapidly.Source: Figure 6 · p. 6
  • Our method outperforms the baselines by around 10% in most metrics, detecting more points correctly and achieving lower mean absolute error on the box and stairs datasets.The comparison is reported in Table I.Source: Table I · p. 6, B. Evaluation on the Robot · p. 7
  • On the Jetson Xavier, inference averages 70 ms and map updates run at 6 Hz, while the controller queries terrain heights at 50 Hz.The locomotion policy uses a 1.6 × 1.0 m terrain input.Source: B. Evaluation on the Robot · pp. 5–6
  • The robot evaluation compares two classical baselines against BLK2GO LiDAR ground truth across stairs and a reflective wooden-box scene.Stairs vary in size, texture, and illumination; portions of the wooden box can be invisible to the depth cameras.Source: B. Evaluation on the Robot · p. 6
  • After a 7 cm state-estimator drop, our map shifts with the measurements; Elevation Mapping cannot handle the drift, while Voxblox remains erroneous below the robot.The maps are expressed in the world frame, so the top surface should remain unchanged during the drift.Source: Figure 7 · p. 6, B. Evaluation on the Robot · p. 7

C. Locomotion in Structured Terrains

Map drift degrades locomotion tracking and task completion, even when most simulated robots survive. The proposed map also supports learning-based and model-based controllers on structured terrain.

C. Locomotion in Structured Terrains

  • Compared with Elevation Mapping, the cleaner map produces smoother motions; the baseline causes erratic stair movement and more joint shaking, worsening tracking despite recovery from instabilities.Source: C. Locomotion in Structured Terrains · p. 7
  • At 20 cm drift, fewer than 20% of robots reach the target on time; survival falls only to 70%, while drift above 10 cm triples knee collisions.The target-reaching task averages performance over 4,000 trajectories.Source: C. Locomotion in Structured Terrains · p. 7, Figure 8 · p. 7
  • A perceptive rough-terrain model-based controller using the map can overcome structured terrain, where inaccurate footholds can produce hard-to-recover trajectories, especially on stairs.The controller computes feasible footholds and tracks them with model predictive control.Source: C. Locomotion in Structured Terrains · p. 7

V. CONCLUSION

The terrain module reconstructs the robot’s vicinity from noisy, occluded point clouds and supports locomotion in structured urban settings. The conclusion also identifies remaining limitations in dynamic-obstacle persistence and simulated noise coverage.

V. CONCLUSION

  • Simulation and robot experiments show that the module accumulates spatial and temporal evidence to estimate structured terrain for learning-based and model-based locomotion controllers.Source: §V · p. 7
  • Dynamic-obstacle points can remain in the reconstruction and vanish only slowly, despite augmentation with random distractors.The authors suggest training with a dynamic-obstacle dataset or a better noise model.Source: §V · p. 7
  • Depth-camera outliers at object edges are difficult to reproduce in simulation, and robot-collected data may be needed for full autonomous deployment.Source: §V · p. 7
  • The demonstrated scope is structured urban terrain; the authors describe unstructured missions as more challenging and say robot data is probably key there.Source: §V · p. 8
  • Future work includes reconstructing unseen obstacle tops, combining point clouds with foot-contact data on compliant ground, and giving the policy direct access to the latent representation.Direct latent access could avoid decoding and provide the policy a bigger picture of the scene.Source: §V · p. 8
Loading 2206.08077v1…