Source-linked AI summary
FB-BEV: BEV Representation from Forward-Backward View Transformations
Zhiqi Li, Zhiding Yu, Wenhai Wang, Anima Anandkumar, Tong Lu, Jose M. Alvarez
TL;DR
Camera-based BEV perception must address sparse features from forward projection and false-positive features from depth-agnostic backward projection. FB-BEV combines both directions with depth-aware refinement, achieving 62.4% NDS on the nuScenes test set.
Problem
Forward projection can produce sparse BEV representations, while backward projection can create false correspondences because it lacks depth guidance.
Method
FB-BEV combines forward projection with depth-aware backward projection, using depth consistency to refine sparse foreground BEV grids and establish more accurate 3D-to-2D mappings.
Results
62.4% NDS is achieved by FB-BEV on the nuScenes test set, the reported state-of-the-art result.
Takeaways & Limitations
The two-stage VTM addresses sparse forward-projected features and imprecise backward-projected relationships, supporting higher-resolution BEV perception.
Takeaways & Limitations
Backward projection remains vulnerable to false correspondences from occlusion and depth mismatch when depth is unavailable.
Abstract
from arXiv · showhide
View Transformation Module (VTM), where transformations happen between multi-view image features and Bird-Eye-View (BEV) representation, is a crucial step in camera-based BEV perception systems. Currently, the two most prominent VTM paradigms are forward projection and backward projection. Forward projection, represented by Lift-Splat-Shoot, leads to sparsely projected BEV features without post-processing. Backward projection, with BEVFormer being an example, tends to generate false-positive BEV features from incorrect projections due to the lack of utilization on depth. To address the above limitations, we propose a novel forward-backward view transformation module. Our approach compensates for the deficiencies in both existing methods, allowing them to enhance each other to obtain higher quality BEV representations mutually. We instantiate the proposed module with FB-BEV, which achieves a new state-of-the-art result of 62.4% NDS on the nuScenes test set. Code and models are available at https://github.com/NVlabs/FB-BEV.
1. Introduction
Existing view transformation modules trade off sparse BEV features against false correspondences from depth-ambiguous backward projection. FB-BEV combines forward and backward projection, using depth consistency to refine sparse regions while suppressing false-positive features, and reaches 62.4% NDS on nuScenes test.
- Forward projection: Forward projection estimates pixel depths to place image features in 3D, but LSS produces discrete, sparse BEV representations.With default nuScenes settings, only 50% of BEV grids receive valid image features through projection.
- Backward projection: Backward projection fills predefined 3D positions with image features, yielding denser BEV representations but creating false correspondences when depth is unavailable.Occlusion and depth mismatch make object distance along the longitudinal direction ambiguous.
- Forward-backward transformation: FB-BEV uses backward projection to refine sparse forward-projected regions and applies depth consistency to suppress false-positive features.The consistency measure compares depth distributions associated with a 3D point and its 2D projection.
- Forward-backward transformation: The proposed bidirectional strategy generates dense BEV features while addressing sparse features and inaccurate-projection false positives.It is designed to make BEV representations more suitable for large-scale BEVs.
- Results: 62.4% NDS is achieved by FB-BEV on the nuScenes test set, establishing the reported state-of-the-art result.The model was extensively evaluated on nuScenes for camera-based 3D object detection.
2. Related work
Related work organizes BEV perception methods by view transformation strategy, covering forward projection, backward projection, temporal enhancement, and projection-free approaches.
- Forward Projection Methods: LSS is the archetypal forward-projection method, while later methods improve depth supervision, efficiency, or multi-modality extensions.BEVDepth and BEVstereo emphasize accurate depth estimation, while BEVDet and BEVFusion extend or optimize the paradigm.
- Backward Projection Methods: Backward-projection methods avoid complex 3D accumulation by projecting predefined 3D points or voxels back onto image features.OFT initiated this paradigm, followed by multi-view voxel methods such as ImVoxelNet and M2BEV.
- Backward Projection Methods: BEVFormer aggregates features at different BEV heights with deformable sampling and temporal features, reducing resource consumption and advancing camera-based perception.BEVFormer uses Deformable DETR and Panoptic SegFormer for perception heads.
- Temporal Methods: Some methods enhance spatial perception by using longer temporal information in addition to view-projection paradigms.
- Projection-free Methods: Projection-free methods learn view transformation through attention or position features, with some reducing dependence on camera calibration parameters.PETR and PETRv2 use global attention and camera-parameter position features, while CFT uses view-aware attention.
3. Method
FB-BEV combines forward projection, foreground-region selection, and depth-aware backward projection to produce denser and more discriminative BEV features. Forward projection supplies depth-guided features, while backward projection refines selected regions using depth consistency.
- Overall architecture: FB-BEV combines forward and depth-aware backward projection to address sparse BEV features and false-positive features from inaccurate projection.The forward module generates an initial BEV representation, while the backward module refines selected regions using depth guidance.
- Feature refinement: The refined foreground grids are added back to B to obtain the final BEV features B′.The design can densely fill 3D space at arbitrary resolutions while using depth consistency to improve projection quality.
- Forward projection: F-VTM projects multi-view pixel features into 3D space using predicted depth distributions and produces the initial BEV features B.Its implementation follows the Lift and Splat paradigm, where pixels are lifted into voxels and voxel features are sum-pooled.
- Foreground region proposal: FRPN predicts a binary foreground mask and selects BEV grids whose mask logits exceed threshold tf for further refinement.The selected grids include both blank and non-blank foreground grids to maintain feature consistency.
- Depth-aware backward projection: Depth-aware backward projection refines selected BEV queries by projecting them onto image features and weighting relationships using depth consistency.Depth consistency compares a 3D point’s depth distribution with the predicted distribution at its 2D projection, distinguishing points along the same projection ray.
4. Experiments
Experiments compare FB-BEV with forward- and backward-projection baselines on nuScenes, including depth-aware projection, foreground selection, and BEV-scale effects. Results show improved accuracy, reduced sparsity, and efficient refinement.
- Benchmark Results: On nuScenes validation, FB-BEV exceeds BEVDet and BEVFormer by 2.4% and 2.7% NDS without temporal information or depth supervision.With temporal information, FB-BEV still leads both baselines by 1.3 points.
- Depth-aware Backward Projection: Depth-aware backward projection improves BEVFormer by 0.9% NDS and removes about 0.9% NDS from FB-BEV when omitted.Depth consistency enables backward methods to benefit from more accurate depth prediction.
- Depth-aware Backward Projection: Depth-aware projection suppresses longitudinal errors and height-wise feature interference by selecting features according to depth consistency.Foreground grids show higher consistency, while aggregation varies with height at the same BEV location.
- Effect of FRPN: FRPN improves detection accuracy and inference efficiency by restricting depth-aware backward projection to foreground BEV grids.Without FRPN, inaccurate depth predictions can direct attention toward background regions.
- Effect of BEV Scale: 80.5% of BEVDet grids are blank at 400×400 BEV scale, illustrating why forward projection degrades on large-scale BEV.Fixed discrete depth values produce fixed projection points, increasing blank-grid proportions as BEV scale grows.
5. Conclusion
The paper presents a forward-backward projection paradigm that addresses sparse forward-projection features and inaccurate backward-projection relationships. The two-stage strategy targets higher-resolution BEV perception and long-distance or occupancy applications.
- 5. Conclusion: FB-BEV combines forward projection with depth-aware backward projection to address sparse features and imprecise projection relationships.The approach is designed for higher-resolution BEV perception and may support ultra-long-distance detection or high-resolution occupancy perception.