Source-linked AI summary
Predicting Semantic Map Representations from Images using Pyramid Occupancy Networks
Thomas Roddick, Roberto Cipolla
TL;DR
Autonomous driving needs detailed BEV maps, but constructing them from sensors is complex and monocular images are challenging because perspective and BEV outputs use different coordinate systems. The paper proposes an end-to-end multiscale transformer architecture with semantic Bayesian occupancy grids, reports improved performance on NuScenes and Argoverse, and supports accumulation across views and timesteps.
Problem
The paper addresses direct BEV semantic-map estimation from monocular images, where image and orthographic BEV representations occupy different coordinate systems.
Method
The method uses dense transformer layers within a multiscale pyramid to map image features into BEV space and predict semantic occupancy maps.
Results
The approach improves on leading methods across two autonomous-driving datasets and processes 23.2 frames per second on a GeForce RTX 2080 Ti.
Takeaways & Limitations
Semantic Bayesian occupancy grids let the method combine predictions across multiple cameras and timesteps to build a complete surrounding-environment model.
Takeaways & Limitations
High-resolution feature maps can make dense transformer layers parameter-intensive, while visibility masks exclude cells outside the camera field of view or fully occluded.
Abstract
from arXiv · showhide
Autonomous vehicles commonly rely on highly detailed birds-eye-view maps of their environment, which capture both static elements of the scene such as road layout as well as dynamic elements such as other cars and pedestrians. Generating these map representations on the fly is a complex multi-stage process which incorporates many important vision-based elements, including ground plane estimation, road segmentation and 3D object detection. In this work we present a simple, unified approach for estimating maps directly from monocular images using a single end-to-end deep learning architecture. For the maps themselves we adopt a semantic Bayesian occupancy grid framework, allowing us to trivially accumulate information over multiple cameras and timesteps. We demonstrate the effectiveness of our approach by evaluating against several challenging baselines on the NuScenes and Argoverse datasets, and show that we are able to achieve a relative improvement of 9.1% and 22.3% respectively compared to the best-performing existing method.
1. Introduction
The paper targets direct monocular-image prediction of detailed BEV semantic maps, replacing a complex multistage pipeline with an end-to-end architecture and probabilistic occupancy representation.
- BEV maps compactly represent static geometry and dynamic-agent pose for tasks including path planning, collision avoidance, and navigation.
- Existing BEV-map construction combines structure from motion, ground-plane estimation, road segmentation, lane detection, and 3D object detection.
- The paper focuses on estimating BEV maps from monocular images, motivated by the cost and limited resolution of LiDAR and radar sensors.
- The authors represent scenes with semantic Bayesian occupancy grids that can incorporate information across multiple sensors and timesteps.
- A pyramid of transformers operating at multiple image scales predicts BEV maps from monocular images.
- The paper evaluates on two large-scale autonomous-driving datasets and reports considerable improvement over leading literature methods.
- The method qualitatively combines predictions across cameras and timesteps, and processes 23.2 frames per second on one GeForce RTX 2080 Ti.
2. Related Work
Prior work predicts top-down representations from images using IPM, object-focused methods, RGB-D, or weakly supervised approaches; this paper presents an architecture overview centered on multiscale feature transformation into BEV space.
- Prior monocular-image methods use inverse perspective mapping, GAN refinement, 2D-to-top-down detection, or direct 3D bounding-box prediction.
- Few methods specifically generate semantic maps from images, and IPM handles local road layout better than objects above the ground plane.
- Many prior methods rely on weak stereo supervision, weakly aligned map labels, or synth-to-real domain transfer because ground-truth data are scarce.
- The architecture uses a ResNet-50 backbone, feature pyramid, dense transformer layers, and a topdown network to predict semantic occupancy probabilities.
3. Semantic occupancy grid prediction
The paper formulates monocular BEV prediction as semantic occupancy-grid estimation and introduces a dense transformer architecture that maps image features into BEV space. A multiscale transformer pyramid handles depth-dependent sampling, while Bayesian fusion supports combining observations across cameras and timesteps.
- Each BEV cell stores class-specific occupancy states, allowing road, crossing, and vehicle classes to coexist at one location.
- The model learns a CNN-based inverse sensor model that predicts semantic occupancy probabilities directly from a monocular image.
- The dense transformer maps perspective image features to orthographic BEV features using camera geometry and fully connected reasoning.
- PyrOccNet combines a backbone, feature pyramid, dense transformer stack, and top-down network to produce the final semantic occupancy prediction.
- The balanced cross-entropy loss up-weights occupied cells for small object classes, while maximum-entropy loss is applied only to invisible or fully occluded cells.
- Bayesian filtering fuses occupancy predictions after transforming local camera-frame outputs into global coordinates using camera extrinsics.
- The transformer pyramid uses downsampling factors s_k = 2^k+3 and concatenates depth-subset outputs into the final BEV feature map.
- At high resolutions, feature-map heights can make dense transformer layers parameter-heavy; cropping to a fixed world-space vertical range keeps heights roughly constant across scales.
4. Experimental Setup
The experiments use NuScenes and Argoverse data converted into camera-coordinate semantic occupancy grids, with visibility-aware evaluation using IoU.
- Datasets: NuScenes and Argoverse provide 3D object annotations and semantic map information used to evaluate camera-based occupancy maps.NuScenes contributes four map and ten object categories; Argoverse contributes a driveable road mask and seven object categories.
- Data generation: Ground-truth occupancy maps rasterise transformed vector annotations and projected 3D bounding boxes on a 25cm-per-pixel BEV grid.The grid extends 50m ahead of the camera and 25m to either side.
- Data generation: Visibility masks exclude grid cells outside the camera field of view or lacking an unblocked LiDAR ray.A cell is visible when it lies within the field of view and has at least one LiDAR ray passing through it.
- Baselines: Comparisons include VED, VPN, inverse perspective mapping, and depth-based unprojection baselines.The published networks receive minor architectural changes for compatible input and output dimensions.
- Evaluation: Intersection over Union is the primary metric, with predictions binarised at p(mc_i|z_t) > 0.5 and non-visible cells ignored.Precision-recall curves are also provided to account for the threshold choice.
5. Results
The proposed architecture improves incrementally over its components and outperforms prior approaches, while temporal and multi-camera fusion produces smoother scene-level maps.
- 5.1. Ablation study: Each reintroduced component improves mean IoU by roughly 1%, with the dense transformer producing a particularly pronounced gain.The ablation evaluates the dense transformer, transformer pyramid, and topdown network.
- 5.1. Ablation study: The topdown network particularly improves small rare classes such as motorbike and bicycle, though it adds no advantage for large classes such as driveable area.
- 5.2. Comparison to other methods: The proposed method outperforms previous approaches, especially on smaller classes, and qualitatively resolves individual cars and pedestrian crowds more finely.The authors attribute this to greater spatial-information preservation than fully connected bottlenecks.
- 5.2. Comparison to other methods: IPM performs reasonably on driveable area but fails on other classes because predictions elongate along camera rays.
- 5.3. Evaluation on the NuScenes dataset: 9.1% relative improvement over VPN is achieved on NuScenes despite its greater dataset diversity.The method also captures finer details such as bus shape and crossroads geometry.
- 5.4. Temporal and sensor fusion: Accumulating occupancy probabilities over 20-second sequences combines multiple views to resolve ambiguities and produce smoother static-scene predictions.Scene-level maps visualise drivable, crossing, walkway, and carpark classes.
6. Conclusions
The paper presents monocular BEV semantic occupancy prediction with dense transformer geometry and fusion across views for comprehensive environmental models.
- 6. Conclusions: PyrOccNet predicts birds-eye-view maps directly from monocular images using dense transformer layers within a multiscale transformer pyramid.The dense transformers warp image-based features into BEV space using camera geometry.
- 6. Conclusions: The method combines information across multiple views to build an exhaustive model of the surrounding environment.The authors identify future BEV applications including lane instance detection and future prediction.
A. Modifications to competing networks
Competing VED and VPN networks are modified minimally to match this paper’s input, output, and multilabel prediction settings.
- VED: VED receives a resized bottleneck, an added decoder layer, and bilinear upsampling to the 196×200 output size.Bottleneck dimensions are 3×6×128 for NuScenes and 4×7×128 for Argoverse.
- VPN: VPN receives an increased transformer bottleneck dimension and output upsampling to 196×200 using the authors’ existing code.
- Training compatibility: Both competing methods are trained with balanced cross entropy because this evaluation uses multilabel prediction rather than the original single-label task.
B. Precision-Recall curves for Argoverse and NuScenes experiments
The precision-recall curves evaluate the method on Argoverse and NuScenes. Across almost all classes, the method forms an upper envelope of achievable precision at a given recall.
- Figures 7 and 8 show the precision-recall trade-off on Argoverse and NuScenes, respectively.
- Across almost all classes, the method achieves the highest precision available for a given recall setting.The passage characterizes this performance as an upper envelope on precision.
- Figure 8 presents the precision-recall curves for the NuScenes dataset.