Source-linked AI summary
Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D
Jonah Philion, Sanja Fidler
TL;DR
Autonomous-vehicle perception must fuse multiple cameras with different coordinate frames into a bird’s-eye-view representation for planning. Lift-Splat lifts images into 3D feature frustums and splats them into a shared grid, outperforming baselines across benchmark segmentation tasks while enabling trajectory-shooting motion planning without ground-truth depth.
Problem
Self-driving perception must fuse multiple sensors across coordinate frames into ego-frame predictions for downstream planning.
Method
Lift-Splat lifts each image into a frustum of contextual 3D features, splats frustums onto a reference plane, and shoots proposal trajectories for interpretable planning.
Results
Lift-Splat outperforms baselines across benchmark object- and map-segmentation tasks and improves car segmentation when additional cameras appear at test time without retraining.
Takeaways & Limitations
The model infers bird’s-eye-view semantics without ground-truth depth and enables end-to-end motion planning through trajectory shooting.
Takeaways & Limitations
The approach assumes access to each camera’s extrinsic and intrinsic calibration parameters.
Abstract
from arXiv · showhide
The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these representations into a single "bird's-eye-view" coordinate frame for consumption by motion planning. We propose a new end-to-end architecture that directly extracts a bird's-eye-view representation of a scene given image data from an arbitrary number of cameras. The core idea behind our approach is to "lift" each image individually into a frustum of features for each camera, then "splat" all frustums into a rasterized bird's-eye-view grid. By training on the entire camera rig, we provide evidence that our model is able to learn not only how to represent images but how to fuse predictions from all cameras into a single cohesive representation of the scene while being robust to calibration error. On standard bird's-eye-view tasks such as object segmentation and map segmentation, our model outperforms all baselines and prior work. In pursuit of the goal of learning dense representations for motion planning, we show that the representations inferred by our model enable interpretable end-to-end motion planning by "shooting" template trajectories into a bird's-eye-view cost map output by our network. We benchmark our approach against models that use oracle depth from lidar. Project page with code: https://nv-tlabs.github.io/lift-splat-shoot .
1 Introduction
Self-driving perception must fuse multiple sensors with different coordinate frames into bird’s-eye-view predictions in the ego-car frame for downstream planning. Lift-Splat addresses this mismatch with a symmetry-preserving, end-to-end differentiable architecture that lifts image features into 3D, splats them onto a reference plane, and supports trajectory shooting for interpretable planning.
- Motivation: Self-driving perception receives multiple sensors in different coordinate frames but must produce predictions in the ego-car frame for downstream planning.This differs from conventional vision tasks, which usually predict in the input image’s coordinate frame or in a coordinate-frame-agnostic form.
- Method: Lift-Splat preserves translation equivariance, permutation invariance, and ego-frame isometry equivariance by design while remaining end-to-end differentiable.The model is designed to retain the three symmetries identified for practical multi-view extensions.
- Motivation: Post-processing single-image detections blocks end-to-end differentiation to sensor inputs and prevents data-driven learning of cross-camera fusion.It also prevents downstream-planner feedback from automatically improving the perception system through backpropagation.
- Method: The model lifts images into frustum-shaped 3D contextual features, splats all frustums onto a reference plane, and shoots proposal trajectories into that plane for interpretable end-to-end planning.The reference plane is selected as convenient for the downstream motion-planning task.
- Evidence: Empirical evidence indicates that Lift-Splat learns an effective mechanism for fusing information across a distribution of possible inputs.This evidence is presented in Section 5.
2 Related Work
Prior work spans multi-camera sensor fusion, monocular image-to-3D detection, pseudolidar pipelines, image-projected 3D primitives, and models that infer directly in bird’s-eye-view coordinates. Lift-Splat builds on these directions while exploring cohesive 360° scene representations from camera input.
- Multi-camera sensor fusion: Large-scale datasets from Nutonomy, Lyft, Waymo, and Argo enable learning full 360° ego-centric scene representations using only cameras.The paper positions Lift-Splat as an exploration of this possibility through sensor fusion and monocular detection.
- Pseudolidar: Pseudolidar methods separately train monocular depth prediction and bird’s-eye-view detection, aligning learning with the coordinate frame where detections are evaluated.Their empirical success is attributed to operating in a frame where Euclidean distance is more natural relative to the image plane.
- Projected primitives and direct BEV inference: Other monocular detectors use 3D primitives projected onto available cameras, while MonoLayout and Pyramid Occupancy Networks infer bird’s-eye-view representations directly from images.Mono3D generates ground-plane proposals scored through image projections, Orthographic Feature Transform projects fixed voxel cubes, MonoLayout uses adversarial inpainting, and Pyramid Occupancy Networks uses a transformer.
3 Method
Lift-Splat-Shoot learns a bird’s-eye-view scene representation from arbitrary camera rigs without depth sensors by lifting image features across discrete depths and splatting them into a shared BEV grid. It also enables interpretable camera-only motion planning by scoring template trajectories on an inferred cost map.
- Problem setup: The model maps images from an arbitrary camera rig into a rasterized representation y ∈ R^C×X×Y in the BEV coordinate frame using camera intrinsics and extrinsics, without depth sensors.The camera matrices define the mapping from reference coordinates to each camera’s local pixel coordinates.
- Lift: For each pixel, the lift stage creates points at all discrete depths, then scales a predicted context vector by a depth distribution to represent ambiguous monocular geometry.The depth sweep is non-learned and creates a point cloud of size |D|·H·W per image.
- Lift: The model can represent either depth-specific context or depth-independent context, corresponding theoretically to pseudolidar and OFT-like behaviors.A one-hot depth distribution selects one depth, whereas a uniform distribution assigns the same representation across depths.
- Splat: Lifted frustums are splatted into BEV pillars by nearest-pillar assignment and sum pooling, producing a tensor for standard BEV convolutional inference.Packing and a cumsum trick avoid padding and accelerate sum pooling during training.
- Motion planning: Camera-only planning shoots candidate trajectories onto the learned cost map, scores them, and acts according to the lowest-cost trajectory.Planning is framed as classification over K templates, with templates obtained by K-Means on expert trajectories and labels assigned by nearest-neighbor L2 distance.
4 Implementation
The implementation uses separate image and bird’s-eye-view backbones connected by a lift-splat layer, with fixed input and grid resolutions. Sum pooling with a cumulative-sum procedure improves training efficiency by reducing memory usage from padding.
- Architecture: The model has an image backbone and a bird’s-eye-view backbone connected by the lift-splat layer.The image network featurizes each camera’s generated point cloud, while the second network processes the splatted point cloud in reference-frame pillars.
- Image backbone: EfficientNet-B0 pretrained on ImageNet is used for image processing in all experiments, including baselines.EfficientNets outperform ResNet-18/34/50 across models but require more optimization steps to converge.
- Bird’s-eye-view backbone: The bird’s-eye-view network combines a convolutional stem with the first three ResNet-18 metalayers to produce representations x1, x2, and x3 at different resolutions.It upsamples x3 by 4, concatenates it with x1, applies a ResNet block, and upsamples by 2 to restore the original input resolution.
- Resolution: Input images are resized and cropped to 128 × 352, while the bird’s-eye-view grid spans -50 to 50 meters in both axes with 0.5-meter cells.Extrinsics and intrinsics are adjusted after image resizing and cropping.
- Training efficiency: Sum pooling across pillars uses a cumulative-sum trick that avoids excessive padding-related memory usage during training.The method sorts points by bin id, cumulatively sums features, and subtracts values at bin-section boundaries.
5 Experiments and Results
Experiments on nuScenes and Lyft evaluate Lift-Splat on bird’s-eye-view segmentation, robustness to camera and calibration errors, generalization across sensors and rigs, and planning. The model outperforms its baselines across segmentation tasks, approaches lidar performance for drivable-area segmentation, and produces trajectories with desirable driving behavior.
- Segmentation evaluation: Lift-Splat is evaluated on object and map segmentation tasks using nuScenes and Lyft camera data.Both rigs use six cameras with shifting intrinsic and extrinsic parameters, which the calibration-conditioned model can handle.
- Segmentation results: The model outperforms frozen-CNN, OFT, and concurrent-work baselines on all segmentation tasks.The results support learning both an effective implicit depth distribution and contextual representations for downstream prediction.
- Robustness to sensor errors: Training with randomly dropped cameras improves robustness to camera dropout at test time, with one dropped camera during training yielding the best six-camera performance.Training with extrinsic noise likewise improves robustness to extrinsic perturbations.
- Generalization: Car-segmentation performance improves when cameras unseen during training are added at test time without retraining.The model is also evaluated for transfer from nuScenes training to the Lyft camera rig.
- Comparison with lidar: Lift-Splat performs slightly worse than PointPillars trained with one lidar scan across tasks but approaches lidar performance on drivable-area segmentation.Its car-segmentation performance is much worse than PointPillars at night.
- Motion planning: In the planning experiment, output trajectories follow road boundaries and stop at crosswalks or behind braking vehicles.Planning is benchmarked against PointPillars, while trajectory targets are cheaper to acquire than ground-truth 3D bounding boxes.
6 Conclusion
The work presents a bird’s-eye-view architecture for arbitrary camera rigs that outperforms baselines on benchmark segmentation tasks without ground-truth depth. It also addresses calibration noise and enables end-to-end motion planning, while future work targets matching or surpassing lidar-based networks.
- Contributions: The architecture infers bird’s-eye-view representations from arbitrary camera rigs and outperforms baselines on benchmark segmentation tasks.It represents bird’s-eye-view semantics without ground-truth depth during training or testing.
- Contributions: The training methods make the network robust to simple models of calibration noise.
- Motion planning: The model enables end-to-end motion planning using the trajectory shooting paradigm.Planning is framed as classification among 1K template trajectories, but the model still lags behind lidar-based approaches in generalization.
- Future work: Future work aims to meet or surpass networks that exclusively use ground-truth depth from pointclouds.