PaperlayerView exact revision on arXiv

Source-linked AI summary

The Neverwhere Visual Parkour Benchmark Suite

Ziyu Chen, Henghui Bao, Haoran Chang, Alan Yu, Ran Choi, Kai McClennen, Gio Huh, Kevin Yang, Ri-Zhao Qiu, Yajvan Ravan, John J. Leonard, Xiaolong Wang, Phillip Isola, Ge Yang, Yue Wang

arXiv:2609.16443v1cs.ROcs.CVcs.LG

TL;DR

Real-world policy evaluation needs scalable, realistic environments as robots move beyond lab settings. Neverwhere provides over 60 3D-Gaussian scenes and a scene-creation toolchain; evaluations found comparable hurdle performance across simulation and reality, but higher simulated success on stairs.

  • Problem

    Real-world evaluation and hand-crafted simulations are insufficient for scalable assessment as robots move into open-world environments.

    Source: §I · p. 1

  • Method

    Neverwhere combines over 60 urban indoor and outdoor 3D-Gaussian scenes with a toolchain for creating simulation scenes from uncalibrated captures.

    Source: §I · p. 1

  • Results

    Hurdle performance was comparable in simulation and reality, while Neverwhere reported higher success on stairs than real-world evaluations.

    Source: §V · p. 5

  • Takeaways & Limitations

    Neverwhere provides a complementary foundation for scalable, practical robot evaluation and real-to-sim-to-real research.

    Source: §VII · p. 7

    Adding environments still requires manual reorientation, rescaling, and task labeling because scenes are not automatically aligned to real-world scale or z-up convention.

    Source: §VII · p. 7

Abstract

from arXiv · show

State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world performance before deployment increasingly difficult. This work intends to narrow this train/evaluation gap by developing a collection of hyper-photo-realistic, closed-loop evaluation environments - The Neverwhere Benchmark Suite - comprised of over sixty 3D Gaussian Splatting reconstructions of urban indoor and outdoor scenes. Our goal is to encourage large-scale and reproducible robot evaluation by making it easier to create and integrate Gaussian splats-based reconstructions into simulated continuous testing setups. We also underscore the potential pitfalls of relying exclusively on 3D Gaussian-generated data for training, by providing policy checkpoints trained over multiple Neverwhere scenes and their performance when evaluated in novel scenes. Our analysis illustrates the necessity of sourcing diverse data to ensure performance. Code and data are available on the project page: https://ziyc.github.io/neverwhere-bench/.

I. INTRODUCTION

The paper targets scalable, realistic evaluation of visual policies as robots move beyond lab settings. It introduces Neverwhere scenes, a scene-creation toolchain, and policy checkpoints for studying simulation-based training and transfer.

I. INTRODUCTION

  • The approach addresses three scene-reconstruction challenges: representing complex environments, rendering robot viewpoints, and producing detailed collision meshes from mobile video.The paper combines improved Gaussian geometry with multi-view stereo reconstruction for collision meshes.Source: §I · p. 1
  • Neverwhere provides over 60 high-quality 3D-Gaussian environments spanning diverse urban indoor and outdoor scenes.The collection includes stairs, speed bumps, carpeted labs, and outdoor areas with and without vegetation.Source: §I · p. 1
  • Its toolchain supports community-created scenes from uncalibrated captures, with minimal human intervention.The pipeline uses multi-view images and requires human input to reorient the mesh and label waypoints.Source: §I · p. 1, Figure 2 · p. 2
  • The authors provide baseline visual-policy checkpoints trained on Neverwhere scenes to support research on simulation training and real-world transfer.Source: §I · p. 1

II. RELATED WORK

Prior work improves physics simulation and supplies designed scenes, while Neverwhere emphasizes reconstructed visual scenes and accurate collision geometry. The paper positions curated, photo-realistic Gaussian-splat scenes as evaluation environments rather than primarily as training data.

II. RELATED WORK

  • Physics engines have expanded material coverage, but 3D content remains a bottleneck despite benchmarks such as BiGym and RoboCasa.Source: §II · p. 2
  • Neverwhere uses 3D Gaussian Splatting for detailed scene replicas and MuJoCo with collision geometry, without requiring LiDAR or depth measurements.The authors contrast this with prior scene-level efforts and describe Neverwhere as an open-source foundation for customizable evaluation setups.Source: §II · p. 2
  • Unlike generative platforms that emphasize diversity, Neverwhere prioritizes photo-realism and accurate modeling, arguing curated deterministic scenes suit evaluation better than training.Source: §II · p. 2

III. FROM PHOTONS TO PHYSICAL TWINS

The toolchain converts uncalibrated multi-view images into robot-simulation scenes by combining visual modeling with geometry reconstruction. It uses mesh-derived depth and initialization to improve Gaussian geometry, then supports scene alignment and task annotation.

Toolchain and formulation

  • The pipeline converts uncalibrated multi-view images into metric-scale digital scenes with geometry-improved Gaussians and aligned collision geometry.It combines scene calibration, reconstruction, and modeling; reliable robot interaction requires both visual fidelity and accurate scene geometry.Source: §III · pp. 2–3

Formulation

  • Depth extracted from mesh reconstruction regularizes Gaussian optimization to improve novel-view renderings from robot viewpoints without extra sensors.Vanilla Gaussian splatting can overfit training views when robot viewpoints differ substantially from handheld capture trajectories.Source: §III · p. 3

Geometry-guided Gaussian initialization

  • Colored point samples from collision meshes initialize Gaussians with stronger geometric priors than scattered SfM points alone.The samples are better distributed and surface-aligned, improving geometric accuracy.Source: §III · p. 3, Figure 3 · p. 3

Scene alignment and task annotation

  • Users manually rescale and reorient reconstructed scenes and label task waypoints; the alignment and annotation process takes less than two minutes.Source: §III · p. 3

Experiments on assets quality

  • The integrated 3DGS models benefit from the framework’s training strategies across scene-reconstruction and novel-view-synthesis evaluations.The reported averages cover seven scenes; the comparison includes Splatfacto and Splatfacto-MCMC, with PSNR, SSIM, and LPIPS metrics.Source: Table I · p. 3, §III · p. 4

Enhanced geometry from mobile captures

  • OpenMVS reconstructions show finer details and more complete scene coverage than mobile-app meshes, while also supplying depth supervision for Gaussian modeling.The text contrasts OpenMVS with on-device mobile reconstruction such as Polycam.Source: §III · pp. 3–4

IV. THE NEVERWHERE BENCHMARK SUITE

The benchmark supplies more than sixty quadruped parkour scenes built from scans of two university campuses. Its locomotion setup combines joint-position actions, robot-state observations, and multiple visual modalities.

IV. THE NEVERWHERE BENCHMARK SUITE

  • The suite covers indoor and outdoor campus scenes with varied obstacle layouts and visual appearances for quadruped parkour.Source: §IV · p. 4

Action Space

  • The quadruped action space contains twelve target joint positions, with three actuated joints on each of four legs.Source: §IV · p. 4

Observation Space

  • Policy observations can include RGB, depth, point clouds, or semantic maps alongside robot state; training also has privileged terrain-heightmap observations.The heightmap offers a top-down terrain view that can be processed into ScanDots for lightweight inference.Source: §IV · p. 4

A. Evaluation Tasks

Neverwhere evaluates locomotion across hurdles, gaps, ramps, and stairs, using annotated waypoint trails to define tasks and measure success. Its rendering pipeline supports multiple visual modalities for policy evaluation.

A. Evaluation Tasks

  • The rendering pipeline offers color images from Gaussian splats, depth maps, heightmaps, and LiDAR projections for visual policies.Source: Figure 5 · p. 4
  • Four scenarios—hurdles, gaps, ramps, and stairs—test locomotion across fixed-height obstacles, openings, inclined surfaces, and staircases.Scenes use varied real staircases and outdoor structures as well as constructed obstacles.Source: A. Evaluation Tasks · p. 4
  • Success Rate is the percentage of waypoints reached, with each scene’s annotated waypoints defining its reference trail.Yellow cones mark waypoints and red cones mark starting points.Source: A. Evaluation Tasks · p. 4, Figure 6 · p. 4
  • Table II reports completion rates for parallel LucidSim evaluations using 10 trials per scene.Source: Table II · p. 5

B. Rendering Wrappers

Neverwhere provides rendering wrappers for visual policies, including photorealistic Gaussian-splat images and depth observations. Additional modalities and user-added wrappers extend the observation options.

B. Rendering Wrappers

  • Gaussian Splatting provides photorealistic images for closed-loop evaluation, with semantic masks used to blend task-relevant targets such as cones into rendered views.The paper reports that visual cones provide strong cues that lead to better policies.Source: B. Rendering Wrappers · p. 5
  • Depth maps are converted from MuJoCo-rendered depth and can serve as policy observations or condition depth-guided generative models.Source: B. Rendering Wrappers · p. 5
  • The system also supports semantic masks, heightmaps, LiDAR point clouds, and user-added wrappers such as depth-conditioned visual generation.Source: B. Rendering Wrappers · p. 5

V. SIM-TO-REAL CONSISTENCY OF NEVERWHERE

Parallel real-world and Neverwhere evaluations show comparable performance on hurdles, while Neverwhere reports higher success rates on stairs. The aligned performance trends make the benchmark a meaningful proxy for visual-policy behavior.

V. SIM-TO-REAL CONSISTENCY OF NEVERWHERE

  • Hurdles performance is comparable between Neverwhere and the real world, while Neverwhere reports higher success rates on stairs.The authors suggest the stairs difference likely reflects greater physical complexity in real-world stairs.Source: §V · p. 5
  • Four task scenes were paired 1:1 between Neverwhere and the real world, with the same LucidSim baseline policy evaluated in both settings.Source: §V · p. 5
  • The authors conclude that aligned performance trends make Neverwhere a meaningful proxy for evaluating visual-policy behavior.Source: §V · p. 5

VI. CLOSED-LOOP TRAINING ON NEVERWHERE

Closed-loop training experiments show limited generalization from Neverwhere scenes, with substantial train–evaluation gaps for some tasks. Performance varies with task difficulty and observation type, while visual cones improve training and evaluation outcomes.

VI. CLOSED-LOOP TRAINING ON NEVERWHERE

  • The benchmark is designed primarily for deterministic evaluation rather than large-scale training, although the authors also run closed-loop training experiments.Multi-scene experiments split each task’s scenes 70% for training and 30% for evaluation; Figure 8 reports averages over 50 rollouts per scene.Source: §VI · pp. 5–6, Figure 8 · p. 6
  • Training uses behavior cloning with a privileged teacher, ACT student policies, and domain randomization for multi-scene runs.Depth policies receive random noise and pixel dropout; RGB policies receive rotations, cropping, blur, and color transformations.Source: §VI · p. 6
  • Single-scene policies perform well on their training scene but generalize poorly to unseen scenes, with slight gains on some visually similar scenes.Figure 9 shows each single-scene policy evaluated across the other scenes, with cross-scene results clustering.Source: §VI · p. 6, Figure 9 · p. 6
  • About 50% average train–evaluation gap on Stairs versus about 10% on Gaps indicates limited generalization, particularly for more challenging tasks.Figure 8 reports average success rates across train and evaluation scenes.Source: §VI · p. 6, Figure 8 · p. 6
  • Adding visual cones substantially improves training efficiency and overall performance on both training and evaluation sets; depth policies perform worse on training than RGB policies.The authors report that explicit visual cues help policies learn more robustly across scenes.Source: §VI · p. 6

VII. CONCLUSION

Neverwhere provides a scalable testbed for evaluating visual policies before real-world deployment. The paper also describes a scene-labeling workflow and reports a gap between task-specific training and evaluation performance.

VII. CONCLUSION

  • Neverwhere combines a benchmark suite and real-to-sim toolchain for scalable, reliable evaluation of visual policies before deployment.The authors describe the toolchain as extending naturally beyond locomotion to broader robotics domains.Source: §VII · p. 7
  • Adding environments still requires manual reorientation, rescaling, and task labeling because reconstructed scenes are not automatically aligned to real-world scale or z-up.Future work will explore learning-based alignment and scene-labeling methods.Source: §VII · p. 7

APPENDIX I SCENE LABELING WORKFLOW

  • Scene labeling corrects arbitrary orientation and scale through manual alignment, distance marking, and task waypoint definition.The workflow takes about one minute; orientation, scaling, and waypoint steps are estimated at approximately 20, 10, and 15 seconds.Source: Appendix I · p. 8

A. Task-Specific Closed-Loop Training

  • Task-specific closed-loop policies showed a significant performance gap between training and evaluation scenes.The authors suggest the limited number and diversity of training scenes may hinder robust generalization.Source: A. Task-Specific Closed-Loop Training · p. 9

B. Combined Task Closed-Loop Training

Combining scenes across tasks increased scene and task diversity, but the resulting policy still performed substantially worse on evaluation scenes than on training scenes.

B. Combined Task Closed-Loop Training

C. Ablation on Different Observation Cues

The observation-cue ablations show that visual cones and explicit direction information both improve evaluation performance, with the visual-cone cue absent from typical real-world settings.

C. Ablation on Different Observation Cues

Related papers

Loading 2609.16443v1…