Source-linked AI summary
Cross-view Transformers for real-time Map-view Semantic Segmentation
Brady Zhou, Philipp Krähenbühl
TL;DR
Autonomous vehicles need semantic and spatial scene understanding for safe navigation, while explicit depth-based geometry is difficult. This paper uses camera-aware cross-view attention to learn camera-to-map-view correspondences without explicit geometric reasoning, achieving state-of-the-art nuScenes performance while running in real time.
Problem
Explicit depth-based geometry for mapping camera views to a canonical map view is difficult because image-based depth estimates are error-prone and projections are rigid.
Method
A cross-view transformer combines camera-aware embeddings and image features with learned map-view queries to infer map-view segmentation without explicit geometric reasoning.
Results
35 FPS on a single RTX 2080 Ti, with state-of-the-art vehicle and road segmentation on nuScenes.
Takeaways & Limitations
The approach offers a simple, real-time alternative to explicit geometric mapping, with learned attention aligning semantic and geometric cues across views.
Takeaways & Limitations
Far-away and partially occluded vehicles are sensed less accurately because they provide fewer corresponding image features and less training data.
Abstract
from arXiv · showhide
We present cross-view transformers, an efficient attention-based model for map-view semantic segmentation from multiple cameras. Our architecture implicitly learns a mapping from individual camera views into a canonical map-view representation using a camera-aware cross-view attention mechanism. Each camera uses positional embeddings that depend on its intrinsic and extrinsic calibration. These embeddings allow a transformer to learn the mapping across different views without ever explicitly modeling it geometrically. The architecture consists of a convolutional image encoder for each view and cross-view transformer layers to infer a map-view semantic segmentation. Our model is simple, easily parallelizable, and runs in real-time. The presented architecture performs at state-of-the-art on the nuScenes dataset, with 4x faster inference speeds. Code is available at https://github.com/bradyz/cross_view_transformers.
1. Introduction
The paper addresses map-view perception from multiple camera views without explicitly modeling geometry. Its cross-view transformer uses camera-aware positional embeddings to learn camera-to-map correspondences and achieves real-time, state-of-the-art performance.
- Prior map-view approaches explicitly or probabilistically estimate depth, but image-based depth is error-prone and depth projections create a rigid mapping bottleneck.
- The cross-view transformer learns camera-view to canonical map-view mappings without explicit geometric reasoning.
- Camera-aware positional embeddings combine calibration-dependent geometry with learned map-view embeddings to link image features and map locations across cameras.
- 35 FPS on a single RTX 2080 Ti accompanies state-of-the-art vehicle and road segmentation on nuScenes.
- The learned attention mechanism directly discovers accurate camera-to-map correspondences from data.
2. Related Works
Related work spans explicit depth and geometric projections, planar transformations, and learned map-view representations. The paper positions implicit attention-based geometry as a simpler alternative that retains calibration information while avoiding explicit depth estimation.
- Map-view semantic segmentation combines 3D recognition, depth estimation, and mapping across different input and output coordinate frames.
- Monocular 3D detection commonly predicts or estimates depth, while multi-camera mapping must additionally merge multiple input views.
- Classical and learned depth methods explicitly compute or regress depth using geometry, stereo, or image-based prediction.
- Explicit depth is camera-dependent and requires accurate calibration and fusion of multiple noisy estimates, motivating the paper’s attention-based alternative.
- Prior map-view methods use planar homographies, direct prediction, or learned cross-view representations such as VED and VPN.
- The paper reports implicit geometric reasoning performing as well as explicit geometric models while improving inference speed.
3. Cross-view transformers
The architecture encodes calibrated camera directions and learned map-view positions, then uses cross-view attention to aggregate multi-scale image features into map-view segmentation. It avoids explicit depth by learning geometric correspondences during end-to-end training.
- 3. Cross-view transformers: The task maps n calibrated monocular views into a binary semantic segmentation mask in orthographic map-view coordinates.
- 3. Cross-view transformers: A shared image encoder extracts multi-scale features, while calibration-dependent positional embeddings support cross-view aggregation into a shared map representation.
- 3.1. Cross-view attention: The method replaces explicit depth with positional embeddings that encode depth ambiguity and let the transformer learn a proxy for depth.
- 3.1. Cross-view attention: Camera-aware embeddings encode unprojected image directions and combine with image features so attention can use geometric and appearance cues.
- 3.1. Cross-view attention: Cross-view attention links map-view coordinates to visible image locations, allowing each map location to select cameras and within-camera positions.
- 3.2. A cross-view transformer architecture: The network projects low-resolution features into map-view, refines the representation, repeats at higher resolutions, and upsamples through three convolutional layers.
- 3.2. A cross-view transformer architecture: The final network is end-to-end trainable using ground-truth map-view annotations and focal loss.
4. Implementation Details
The implementation uses a multiscale EfficientNet-B4 encoder, cross-attention blocks, and a convolutional decoder to produce 200 × 200 map-view outputs. Training uses focal loss and AdamW, while the model achieves state-of-the-art performance with faster inference.
- Architecture: EfficientNet-B4 extracts image features at two scales with 8x and 16x downscaling.The encoder is fine-tuned from a pre-trained EfficientNet-B4.
- Architecture: The initial map-view positional embedding is a learned w × h × D tensor with D = 128 and a 25 × 25 grid.The grid size limits the quadratic cost of cross-attention.
- Architecture: Two four-head cross-attention blocks process the multiscale features, followed by three bilinear upsample-plus-convolution layers.The decoder reaches a final output resolution of 200 × 200, covering 100 × 100 meters centered on the ego-vehicle.
- Training: Training uses focal loss, batch size 4 per GPU, 30 epochs, AdamW, and a one-cycle learning-rate scheduler.Training converges within 8 hours on a 4 GPU machine.
- Evaluation: 4.5× faster inference and state-of-the-art vehicle map-view segmentation are reported on nuScenes across both evaluation settings.The comparison uses single-timestep models and evaluates Intersection over Union (IoU).
5. Results
The cross-view transformer achieves strong map-view segmentation across nuScenes and Argoverse while remaining simple, fast, and robust to partial camera dropout. Ablations show that camera-aware positional embeddings and complete attention components are central to performance, while accuracy declines for distant or occluded objects.
- The model is evaluated on vehicle and road map-view semantic segmentation using nuScenes and Argoverse datasets.
- Comparison to prior work: In both evaluation settings, the cross-view transformer and FIERY outperform alternative approaches by a significant margin, with comparable performance overall.The cross-view transformer has a slight edge in Setting 2, while FIERY leads in Setting 1.
- Comparison to prior work: 32 GPU hours versus 96 GPU hours for FIERY, while inference is 4× faster under comparable feature-extractor and decoder configurations.Both models intentionally use the same EfficientNet-B4 image feature extractor and similar decoder architecture.
- Ablations of cross-view attention: The full attention design performs best because camera-aware embeddings, image features, and iterative map-view embedding refinement jointly support view alignment.Partial attention variants degrade below the full model, and the camera-aware embedding provides geometric information for linking camera-view and map-view locations.
- Camera-aware positional embeddings: Camera-aware positional embeddings perform best, while removing positional embeddings makes localization and camera identification difficult.Learned per-camera embeddings perform well, but linear and Random Fourier camera-aware projections perform best.
- Accuracy vs distance: Accuracy decreases with distance and is lower for far-away or occluded vehicles, although the approach degrades more slowly nearby and slightly underperforms FIERY at longer ranges.The authors attribute distant-object difficulty to occlusion, fewer corresponding image features, less training data, and fewer geometric priors.
- Camera dropout: Performance decreases linearly as cameras are randomly removed, while overall degradation does not extend beyond the regions made unobserved.The observed area also shrinks roughly linearly with the number of cameras removed because different cameras overlap only marginally.
- Qualitative results: Attention visualizations show that the model can highlight closely corresponding locations between camera views and the map view.
6. Conclusion
The paper presents a cross-view transformer approach for map-view segmentation that achieves state-of-the-art performance while remaining simple and real-time.
- The approach uses a cross-view transformer architecture for map-view semantic segmentation.
- The proposed approach achieves state-of-the-art performance.
- The model is simple to implement and runs in real time.