Source-linked AI summary
LENS: Localization enhanced by NeRF synthesis
Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, Arnaud de La Fortelle
TL;DR
Camera pose regression is attractive for lightweight robot relocalization but is limited by biased, insufficiently diverse training data. LENS uses NeRF-W to generate geometry-consistent views at valid scene locations and combines them with real images for training, improving localization performance and narrowing the gap with structure-based methods. The pipeline remains constrained by slow offline NeRF computation and a real–synthetic domain gap.
Problem
Camera pose regressors are lightweight but have limited accuracy because training data are often small, biased, and insufficiently diverse across possible camera poses.
Method
LENS trains NeRF-W from registered real images, selects virtual cameras using learned scene density, renders novel views, and combines synthetic and real images to train a pose regressor.
Results
LENS strongly improves localization, reducing the gap with structure-based methods while retaining fast inference, low memory use, and scalability to large environments.
Takeaways & Limitations
Camera pose regression can perform well when trained with large and diverse datasets, enabling accurate real-time robot relocalization with LENS and CoordiNet.
Takeaways & Limitations
NeRF-W requires days of GPU training per scene and slow rendering forces offline dataset generation; synthetic-only training also performs poorly on real images because of domain gap.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRF) have recently demonstrated photo-realistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.
1 Introduction
Camera pose regression offers a lightweight alternative to computationally costly structure-based localization, but its accuracy depends strongly on limited and biased training data. LENS addresses this data limitation by augmenting training with geometry-consistent NeRF views sampled across the scene.
- Motivation: Structure-based localization performs strongly but is difficult to deploy on embedded robotic systems because of computational and memory costs.End-to-end pose regression is an alternative that predicts camera position and orientation directly from images.
- Motivation: Pose-regression accuracy remains limited and highly dependent on the quantity and diversity of training data.The limitation persists despite architectural improvements.
- Motivation: Training sets built from consecutive video frames lack the spatial diversity needed to cover possible camera poses and generalize to unseen positions.The paper identifies dataset bias as a primary explanation for poor generalization.
- Proposed approach: LENS augments pose-regression training with NeRF-generated views sampled at meaningful locations using the model’s learned scene geometry.The method uses a regular grid and discards poses near occluders or with wrong orientations.
2 Related work
Prior visual-localization work uses novel-view synthesis to refine poses, verify estimates, or enlarge reference databases. LENS builds on these directions by using NeRF-rendered views specifically to augment camera-pose-regression training, addressing a stated gap in prior research.
- Novel-view synthesis for localization: Novel-view synthesis has been used to refine pose estimates, verify structure-based predictions, and recover poses through NeRF differentiability.These approaches apply rendering during localization or optimization rather than solely for offline pose-regression data augmentation.
- Synthetic reference data: Synthetic images can enlarge reference databases with densely sampled viewpoints for matching or learning-based localization.This design raises separate questions about virtual-camera placement and view-synthesis method.
- Virtual-camera placement: Existing virtual-camera methods commonly use regular grids, coarse scene structure, visibility checks, or probabilistic sampling to remove irrelevant locations.The paper positions camera placement as a central design choice in synthetic-data generation.
- Research gap: NeRF provides photorealistic continuous scene representations, but prior work had not studied NeRF-rendered views for enlarging camera-pose-regression datasets.The paper contrasts NeRF with textured meshes and generative models for novel-view rendering.
3 Synthetic dataset rendering with LENS
LENS is an offline augmentation pipeline that trains NeRF-W from registered real images, samples valid virtual cameras from its density representation, renders novel views, and combines them with real images to train a pose regressor.
- Pipeline overview: LENS uses a small set of pose-labeled real images to generate a large, uniformly distributed synthetic dataset for pose-regressor training.The augmentation improves training without adding computation during localization inference.
- NeRF-W training: NeRF-W learns scene appearance and geometry from images, extending NeRF to model temporal and appearance variation in real-world scenes.The model uses learned latent spaces and separate networks for static and transient content.
- NeRF-W training: LENS trains one NeRF-W model per scene on a sparse set of registered images before using the learned representation for synthesis.NeRF-W models are trained with the available scene images and their corresponding training rays.
- Density-volume construction: The method queries NeRF-W on a regular 3D grid to form a density volume and retains sufficiently dense points as reachable scene structure.The grid spans an expanded bounding box around the training poses, with density threshold tσ used to select points.
- Virtual camera locations: Virtual-camera generation samples grid candidates, prunes locations near structures or otherwise irrelevant areas, and adjusts resolution until the desired camera count is reached.The goal is uniform coverage of locations and viewpoints that an agent could visit during localization.
- Virtual camera locations: Each virtual camera copies the nearest training-pose orientation with a small random perturbation, producing pose queries for novel-view synthesis.The resulting pose includes camera position and quaternion orientation.
- Novel-view synthesis and training: Novel views are rendered at virtual locations with randomly interpolated appearance embeddings, removing transient occluders and augmenting illumination variation.The rendered images are then combined with real images and sampled together to train the pose regressor.
4 Experiments
Experiments evaluate LENS with CoordiNet on Cambridge Landmarks and 7scenes, comparing related methods, synthetic-data scale, pose distribution, and design ablations. Results show gains from larger and more uniformly distributed synthetic datasets, while real–synthetic domain differences and offline NeRF computation remain important constraints.
- Evaluation setup: LENS with CoordiNet is evaluated on Cambridge Landmarks and 7scenes against pose-regression and structure-based localization methods.The comparison reports median 6DOF translation and orientation errors.
- Comparison with related methods: LENS achieves the best reported camera-pose-regression results on both benchmarks and reduces the gap with Active Search on 5 of 11 scenes.The method also reports approximately +60% relative localization improvement, compared with +45% for another synthetic-data approach.
- Synthetic dataset size: A synthetic-to-real ratio of 10 yields 59% translation improvement for ShopFacade, 58% for Church, and 63% for Fire.The reported relative improvement is correlated with the synthetic-to-real ratio rather than the total image count, and the translation curves show no plateau.
- Ablation study: Volume pruning removes distracting virtual views, while random appearance embeddings slightly reduce translation and rotation errors.The authors expect appearance augmentation to be more useful under greater illumination diversity, such as day-and-night imagery.
- Pose distribution: Replacing original training poses with uniformly distributed LENS poses reduces median error from 27cm/11.7° to 8cm/3.5° at the same training-set size.Increasing synthetic data further improves localization, and the largest training regime can reach structure-based-method accuracy on real data.
- Pose distribution: Video-only datasets create unbalanced pose-regression problems because their labels do not cover the scene’s full set of possible poses.The paper identifies scene-wide training-data distribution as crucial for pose-regression performance.
- Limitations: NeRF-W requires several days of GPU training per scene, forcing synthetic data generation offline before localization training.The resulting online localization remains fast and lightweight for real-time embedded robotics applications.
- Limitations: Training exclusively on synthetic images performs poorly on real images because of a domain gap involving objects, textures, blur, and rendering artifacts.Mixing real and synthetic images is the stated mitigation; domain adaptation and higher-quality rendering are alternatives.
5 Conclusion
LENS addresses camera pose regression limitations at the data level by using NeRF-W to generate diverse synthetic views across an entire scene. The resulting training strategy improves localization while retaining pose regression’s fast inference and low memory footprint.
- LENS trains relocalization models with high-quality NeRF-W images sampled uniformly across the scene.
- LENS improves camera pose regression and reduces its performance gap with structure-based methods.
- Pose regression performs well when trained with large and diverse datasets, supporting accurate real-time robot relocalization.