Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images

Yigit Baran Can; Alexander Liniger; Danda Pani Paudel; Luc Van Gool

Source-linked AI summary

Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images

Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, Luc Van Gool

arXiv:2110.01997v1Oct 5, 2021cs.CV

TL;DR

The paper addresses structured BEV understanding of road networks and traffic agents from a single front-facing camera, despite the challenge of inferring the ground-plane scene from horizontally mounted imagery. It predicts directed lane graphs with BEV object instances and reports superior performance against compared baselines, with design choices examined through ablations.

Problem
Autonomous navigation needs structured road-network and traffic-agent representations in BEV, but recovering them from a single horizontally mounted onboard camera is challenging.
Method
The method uses a transformer-based model to predict directed BEV lane graphs with Bezier centerlines, connectivity, and object instances from one front-facing image.
Results
The network achieves superior performance to compared baselines across lane-graph and object-detection tasks.
Takeaways & Limitations
The structured lane graph and BEV object instances provide a comprehensive traffic-scene representation suited to downstream planning and prediction tasks.
Takeaways & Limitations
The BEV positional embedding assumes a flat surface with constant point height determined by camera height.

Abstract

from arXiv · show

Autonomous navigation requires structured representation of the road network and instance-wise identification of the other traffic agents. Since the traffic scene is defined on the ground plane, this corresponds to scene understanding in the bird's-eye-view (BEV). However, the onboard cameras of autonomous cars are customarily mounted horizontally for a better view of the surrounding, making this task very challenging. In this work, we study the problem of extracting a directed graph representing the local road network in BEV coordinates, from a single onboard camera image. Moreover, we show that the method can be extended to detect dynamic objects on the BEV plane. The semantics, locations, and orientations of the detected objects together with the road graph facilitates a comprehensive understanding of the scene. Such understanding becomes fundamental for the downstream tasks, such as path planning and navigation. We validate our approach against powerful baselines and show that our network achieves superior performance. We also demonstrate the effects of various design choices through ablation studies. Code: https://github.com/ybarancan/STSU

1. Introduction

The paper targets structured BEV scene understanding from a single front-facing camera by jointly recovering directed lane graphs and traffic-object instances. Its representation preserves road connectivity and direction for downstream planning, while addressing limitations of existing sensor- and map-intensive or unstructured approaches.

1. Introduction: Existing approaches commonly rely on HD maps, multiple sensors, lane detection, or unstructured BEV semantics rather than the required structured output.The paper positions its goal between topologically challenging HD-map reconstruction and single-image lane detection, while retaining object prediction in the same model.
1. Introduction: The road network is represented as a directed graph whose centerline vertices and connectivity edges encode traffic flow.The method estimates centerlines first, then predicts edge existence and direction; traffic flows from green start points to red endpoints.
1. Introduction: The method jointly estimates a structured BEV lane graph and object detections from a single onboard monocular camera image.It predicts lane centerline splines, graph connectivity, and BEV object instances such as cars and pedestrians.
1. Introduction: The work addresses recovering complex road topology from a single image while detecting objects directly in BEV coordinates.The authors characterize coherent topological recovery from one image as unexplored and combine it with BEV object detection.
1. Introduction: The proposed transformer-based formulation uses Bezier centerlines and an assignment matrix to define a complete lane graph.Hungarian supervision is applied at the transformer output, and the predicted segments plus connectivity form the HD-map graph.
1. Introduction: The authors report significantly superior results to compared baselines for the unified road-network graph estimation and object-detection task.The contribution statement summarizes the reported comparison across the proposed structured BEV tasks.

2. Related Works

Prior work extracts road networks from aerial imagery or reconstructs HD maps from aggregated visual and 3D data, while image-based BEV methods often produce unstructured semantics. This paper targets more precise structured lane-graph understanding from onboard imagery.

2. Related Works: Aerial-image road-network methods provide coarse predictions that may support routing but are insufficiently accurate for action planning.The related-work discussion distinguishes coarse aerial road networks from the precision needed for vehicle action planning.
2. Related Works: Offline HD-map reconstruction commonly aggregates 2D and 3D visual information rather than using a single onboard image.The paper identifies aggregated visual information and 3D point clouds as a key distinction from its image-only setting.
2. Related Works: The paper illustrates that its method handles complex crossroads and multiple object instances, including pedestrians marked with circles.Figure 2 provides the cited visual example of complex crossroad scenes and object-instance handling.
2. Related Works: Single-image BEV semantic methods understand map semantics but generally do not provide the structured outputs required by many downstream tasks.The paper contrasts semantic representations with structured lane graphs and object instances.

3. The Proposed Method

The method jointly predicts a directed BEV lane graph and oriented object instances from a single front-facing camera image using a transformer with centerline and object queries.

3. The Proposed Method: A single front-facing image is converted into a directed BEV road graph and object detections, both requiring projection onto the ground plane.The graph represents road connectivity, while objects are represented as BEV bounding boxes.
3.1. Lane graph representation: The lane graph uses Bezier curves for centerlines and an incidence matrix for directed connectivity between segments.Connectivity links a segment to another when the latter starts where the former ends, preserving traffic-flow direction.
3.3. Architecture: A transformer jointly processes learned centerline and object queries, producing lane detections, control points, associations, class probabilities, and oriented box parameters.The shared query processing exploits relationships among centerlines and between centerlines and traffic objects.
3.3.1 Lane branch: The lane branch forms graph vertices from detected centerlines and predicts pairwise directional associations through learned feature vectors and an MLP.Training matches predicted and ground-truth centerlines with the Hungarian algorithm before association prediction.
3.3.2 Object branch: The object branch predicts semantic classes and five normalized parameters for oriented BEV boxes, with an optional refinement network for segmentation.The five parameters specify center location, short and long side lengths, and heading angle.
3.3.3 BEV positional embedding: The model combines image-domain and BEV positional embeddings, using a flat-ground assumption and logarithmic BEV locations to provide spatial awareness.The two embeddings occupy separate halves of the feature channels.
3.3.4 Training: Training uses cross-entropy for detection or classification and L1 losses for control points and box parameters, with smooth sin/cos loss for object angles.Angles are restricted to [0, π] because 180° flips are difficult to distinguish.

4. Metrics

The evaluation measures whether predicted centerlines reproduce the directed lane graph, covering geometric accuracy, centerline detection, and connectivity.

4. Metrics: The metrics are designed to assess faithful reproduction of the real directed graph from complementary perspectives.They separately address centerline accuracy, missed centerlines, and estimated connectivity.
4.1. Precision-Recall: Precision-recall matches estimated and target centerlines by Bezier control-point L1 distance before evaluating interpolated points at distance thresholds.Control-point matching preserves direction, unlike matching only interpolated points.
4.2. Detection: The detection ratio is the fraction of unique ground-truth centerlines matched by at least one estimate, exposing missed lines that precision-recall can overlook.High precision-recall with low detection indicates accurate estimates covering only part of the ground truth.
4.3. Connectivity: The connectivity metric compares predicted and ground-truth incidence relationships using true-positive and false-positive or false-negative directed associations.It tolerates fragmented centerlines when their fragments remain correctly associated.

5. Experiments

Experiments evaluate monocular lane-graph and object understanding on NuScenes using front-camera images, generated baselines, and established BEV comparison methods.

NuScenes provides 1000 Boston and Singapore sequences with centerline HD-Maps and 3D boxes, while experiments use six frequent object classes and only the front camera.
5.3. Baselines: The study compares lane-graph extraction against two image-adapted baselines and evaluates instance estimation against VED, VPN, and PON.
5.3. Baselines: Polyline-RNN first estimates points, whereas Polygon-RNN combines backbone features and those estimates to form centerline curves.
Lane-graph evaluation uses precision-recall curves, connectivity measures, and mIOU, while object evaluation also reports precision-recall results.

6. Results

The proposed method achieves the strongest reported lane-graph performance and competitive object results, with ablations favoring the large split-log transformer configuration.

6.1. Lane graph: Our method achieves the best results across all reported lane-graph metrics, outperforming the baselines in precision-recall, detection, and connectivity.
6.1. Lane graph: Visual comparisons show more faithful overall graph representations than competing estimates, even though Poly(GT) can be closer to matched ground-truth lines in Euclidean distance.
6.2. Objects: Our method surpasses PON in half of the object classes and in the mean mIOU, with a particularly significant difference for cars.
6.2. Objects: Refinement outputs trade some instance separation against mIOU maximization, while remaining a practical compromise for object estimates.
6.3. Ablation study: The Large+Split Log model is selected because split-log embeddings improve object, precision, and detection scores while slightly reducing connectivity IOU.

7. Conclusion

The paper concludes that directed graphs with Bezier centerlines support structured monocular BEV lane-graph prediction, alongside BEV object instances, with superior performance on both tasks.

The proposed representation models local road networks as directed graphs with Bezier curve centerlines and supports fixed-sized learnable lane-graph prediction.
The same model provides BEV object instances, and experiments report superior performance against powerful baselines for lane-graph and object-detection tasks.

8. Summary

Supplementary material adds view-synthesis augmentation, connectivity-metric analysis, and additional visual and quantitative results.

The supplementary material presents view-synthesis augmentation, further analysis of the connectivity metric, and additional visual and quantitative results.

9. Training

Training augments images with artificial depthwise motion under a flat-world assumption by computing how pixel coordinates change after ego-vehicle motion.

Training: Artificial depthwise motion is applied as data augmentation during training.The augmentation models ego-vehicle movement in the depth direction.
Training: The flat-world assumption enables computation of new pixel locations for real-world points after motion by β.The transformation uses camera and image parameters including focal length, camera height, and image offsets.
Training: Original and transformed pixel coordinates are related through formulas involving focal length, camera height, depth motion, and image offsets.

10. Connectivity metric

The connectivity metric evaluates whether estimated centerline associations match the ground-truth graph, while distinguishing correct associations from missed connections.

Connectivity metric: An estimated association is a true positive when matched centerlines share a target or their matched targets are connected in the ground-truth incidence matrix.
Connectivity metric: The metric measures how closely the estimated connectivity pattern matches the ground-truth graph, allowing fragmented centerlines when their associations remain correct.
Connectivity metric: A missed connection is a false negative when a ground-truth association lacks matched estimates for either endpoint or lacks any positively associated estimated pair.

11. Lane graph results

Visual results show that the proposed method generally produces stronger lane-graph estimates than the baselines, including in scenes with turns, occlusions, and complex crossroads, while object estimates are often reasonable or accurate.

11.1. Lane graph merging method: Lane-graph visualizations merge predicted junction endpoints for display without changing the underlying directed graph, and results cover the full target region including occluded areas.The full-field-of-view presentation means occluded-region results should be interpreted with this scope in mind.
11.2. Visual results: In Scene 1, only the proposed method detects the left turn, and the connectivity metric favors it despite similar precision-recall performance with Poly(GT).
11.2. Visual results: In Scene 3, the method estimates a complicated road network with barely visible turns, and the connectivity metric demonstrates its superiority.
11.2. Visual results: In Scene 4, the method captures the initial parts of turns but fails to estimate the whole crossroads, while Poly(Est) achieves the best result except on detection score.
11.3. Additional visual results: The proposed method produces the strongest overall lane-graph estimates across the reported visual scenes, outperforming baselines including Poly(GT).The additional examples explicitly describe its road-network estimate as vastly superior to all other baselines.
11.3. Additional visual results: With object estimations included, the method produces faithful lane graphs and generally accurate object estimates, though it misses objects in non-visible regions and mislabels one truck as a car.