PaperlayerView exact revision on arXiv

Source-linked AI summary

PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark

Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, Junchi Yan

arXiv:2203.11089v3cs.CV

TL;DR

Existing 2D perspective-view lane detection is insufficient for complex autonomous-driving scenarios, while prior front-view-to-BEV transformations and realistic 3D lane datasets remain limited. PersFormer addresses these gaps with Transformer-based local-context feature transformation, unified 2D/3D detection, and the OpenLane dataset, achieving strong 3D results and supporting 2D detection.

  • Problem

    Perspective-view lane detection is not applicable to industry-level products, while prior spatial transformations and realistic large-scale 3D lane datasets are limited.

  • Method

    PersFormer generates BEV features by attending to local front-view regions using camera-parameter-guided Transformer transformation, while jointly detecting 2D and 3D lanes.

  • Results

    PersFormer outperforms previous methods in OpenLane 3D lane detection, achieves the best F-Score across Apollo scene sets, and improves ablation performance by 6.02% with its complete model.

  • Takeaways & Limitations

    OpenLane provides a large-scale realistic 3D lane benchmark, while fine-grained BEV features with explicit prior and supervision improve lane-detection performance.

  • Takeaways & Limitations

    Lane-line height is defined relative to the zero point in the ego-vehicle coordinate system.

Abstract

from arXiv · show

Methods for 3D lane detection have been recently proposed to address the issue of inaccurate lane layouts in many autonomous driving scenarios (uphill/downhill, bump, etc.). Previous work struggled in complex cases due to their simple designs of the spatial transformation between front view and bird's eye view (BEV) and the lack of a realistic dataset. Towards these issues, we present PersFormer: an end-to-end monocular 3D lane detector with a novel Transformer-based spatial feature transformation module. Our model generates BEV features by attending to related front-view local regions with camera parameters as a reference. PersFormer adopts a unified 2D/3D anchor design and an auxiliary task to detect 2D/3D lanes simultaneously, enhancing the feature consistency and sharing the benefits of multi-task learning. Moreover, we release one of the first large-scale real-world 3D lane datasets: OpenLane, with high-quality annotation and scenario diversity. OpenLane contains 200,000 frames, over 880,000 instance-level lanes, 14 lane categories, along with scene tags and the closed-in-path object annotations to encourage the development of lane detection and more industrial-related autonomous driving methods. We show that PersFormer significantly outperforms competitive baselines in the 3D lane detection task on our new OpenLane dataset as well as Apollo 3D Lane Synthetic dataset, and is also on par with state-of-the-art algorithms in the 2D task on OpenLane. The project page is available at https://github.com/OpenPerceptionX/PersFormer_3DLane and OpenLane dataset is provided at https://github.com/OpenPerceptionX/OpenLane.

1 Introduction

Perspective-view lane detection is insufficient for complex autonomous-driving scenes because BEV projection often assumes flat ground and ignores lane height. PersFormer addresses this with learned spatial transformation, unified 2D/3D detection, and the OpenLane benchmark.

  • 2D perspective lane detection is unsuitable for industry-level products dominated by complicated driving scenarios.
  • BEV representations better align lane locations with planning, control, interactive agents, and multimodal sensors.
  • Ignoring lane height causes projected lanes to diverge or converge on uphill, downhill, and uneven roads, producing improper downstream decisions.
  • PersFormer learns spatial feature transformation through attention over front-view local regions to generate fine-grained BEV features.
  • OpenLane provides 200,000 annotated frames, over 880,000 lanes, 14 lane categories, and diverse scene-related annotations for realistic 3D lane research.

2 Related Work

Prior BEV-oriented lane and vision methods separately omit attention, camera geometry, or explicit BEV modeling. PersFormer combines camera parameters with cross-attention to learn view transformation and improve BEV feature representation.

  • Vision Transformers use cross-attention as a learnable feature transformation across views, supporting BEV downstream tasks.
  • Earlier BEV methods often omit attention mechanisms, 3D vision geometry, or both, limiting their feature transformation design.
  • PersFormer combines camera parameters with data-driven learning, using IPM coordinates as references while attending to related front-view local context.

3 Methodology

PersFormer transforms front-view features into BEV representations with camera-parameter-guided Transformer attention, then jointly predicts 2D and 3D lanes using unified anchors and auxiliary BEV segmentation.

  • Overall Structure: PersFormer comprises a backbone, Perspective Transformer, and lane detection heads for predicting 2D/3D coordinates and lane types.The backbone produces multi-scale front-view features; the Transformer generates BEV features using camera intrinsic and extrinsic parameters.
  • Perspective Transformer: Perspective Transformer learns front-view-to-BEV feature transformation by attending to local front-view context around camera-parameter-based reference locations.It uses IPM-based coordinate mapping as a reference while deformable cross-attention dynamically adjusts keys to aggregate relevant local features.
  • Perspective Transformer: The transformation maps front-view coordinates using a scale factor, pitch rotation, camera intrinsics, and camera height to establish the BEV correspondence.The mapping provides a geometric prior for generating representative BEV features.
  • Perspective Transformer: BEV queries are explicitly tied to physical locations, with self-attention operating among BEV queries before cross-attention retrieves front-view features.For each BEV target point, projected front-view reference points and learned offsets provide surrounding context for feature aggregation.
  • Unified 2D/3D Detection: Unified anchors are placed in BEV, projected to front view, and optimized through predicted offsets to support both 2D and 3D lane detection.The framework also adds a U-Net BEV binary-segmentation task, using projected 3D lane ground truth to improve intermediate feature representation.

4 OpenLane: A Large-scale Realistic 3D Lane Benchmark

OpenLane is presented as a large-scale real-world 3D lane benchmark built on Waymo Open, with extensive annotations and diverse realistic driving scenarios.

  • Highlights over Previous Benchmarks: OpenLane follows Waymo Open’s data format and evaluation pipeline, reducing the need for additional benchmark-specific rules.The dataset is constructed on top of Waymo Open.
  • Highlights over Previous Benchmarks: 200K frames and over 880K carefully annotated lanes make OpenLane larger than existing lane datasets.These counts are reported as 33% and 35% more than CurveLanes, respectively.
  • Highlights over Previous Benchmarks: Frames can contain up to 24 lanes, and about 25% contain more than 6 lanes, reflecting complicated lane topologies.Examples include intersections and roundabouts, with opposite-direction lanes annotated when no curbside separates them.
  • Highlights over Previous Benchmarks: Fourteen lane categories cover diverse lane types, although double-yellow solid and single-white solid or dash lanes comprise almost 90% of all lanes.The resulting category distribution is described as a long-tail problem common in realistic scenarios.
  • Generation of High-quality Annotation: OpenLane additionally provides scene tags, weather and location information, and closest-in-path object annotations for downstream planning and control.The dataset also includes high-quality 2D labels with tracking IDs, categories, and 2D point ground truth.

5 Experiments

PersFormer is evaluated on OpenLane and Apollo, where it achieves strong 3D lane detection performance and competitive 2D results. Ablations attribute gains to unified anchors, multi-task learning, spatial feature transformation, and auxiliary segmentation.

  • PersFormer achieves the highest F-Score on the entire OpenLane validation set and every scenario set.
  • 11% improvement over LaneATT is reported for OpenLane 2D lane detection.
  • 6.4% improvement in F-Score over the previous best method is reported for OpenLane 3D lane detection.
  • PersFormer achieves the best F-Score on every Apollo scene set, with comparable X/Z error to previous methods.
  • Ablation Study: Jointly using the new anchor and multi-task strategy improves 3D performance by 2.5% and 2D performance by 2.6%.
  • Ablation Study: Adding spatial feature transformation and auxiliary binary segmentation increases the 3D improvement to 6.02% and the 2D improvement to 9.7%.

6 Conclusions

The paper introduces PersFormer, a Transformer-based unified 2D/3D lane detector, together with OpenLane, a large-scale realistic 3D lane dataset. Experiments show that fine-grained BEV features with explicit prior and supervision improve lane detection, while OpenLane aligns academic and industrial needs.

  • PersFormer is a Transformer-based unified 2D/3D lane detector, and OpenLane is a large-scale realistic 3D lane dataset.
  • Fine-grained BEV features with explicit prior and supervision significantly improve lane detection performance.
  • A large-scale real-world 3D lane dataset aligns academic and industrial demands.

A.1 Lane Detection Benchmarks

The appendix describes benchmark limitations, model architecture, unified anchor construction, prediction targets, and multi-task loss design. It also records scenario-tag statistics and dataset-generation details.

  • Lane Detection Benchmarks: Existing lane datasets vary from pixel-level annotations and highway-only scenes to broader conditions without segment or tracking annotations.
  • Model Architecture: The backbone jointly supports 2D and 3D branches and outputs four scaled front-view feature maps for transformation to BEV.
  • Anchor Design: BEV anchors are placed every 8 pixels along the x-axis, with seven incline angles covering straight, curved, and perpendicular lanes.
  • Anchor Design: Ground-truth lanes are assigned to anchors by minimum edit distance in both 2D and 3D tasks rather than proximity at a fixed reference position.
  • Prediction and Losses: The 3D head predicts sampled lane coordinates and visibility, while corresponding 2D predictions use front-view coordinates and visibility.
  • Prediction and Losses: The total loss combines 2D detection, 3D detection, and intermediate segmentation terms weighted by learnable α, β, and γ.

C.1 Dataset Statistics

OpenLane provides 200K annotated frames across balanced train/validation splits, with diverse weather and challenging lane-count distributions. Its statistics also show substantial altitude variation, motivating 3D lane detection.

  • Dataset scale and splits: OpenLane contains 1,150 segments, with 798/202/150 train/validation/test splits; 1,000 annotated segments provide 200K frames at 10 FPS.The test set is reserved for online leaderboard evaluation, while the original train/validation partition is retained for comparison with other tasks.
  • Altitude variation: More than 50% of OpenLane frames have over 1m altitude difference, while only around 20% are relatively flat with absolute variation below 0.5m.The altitude distribution demonstrates the prevalence of non-flat road geometry in the dataset.
  • Scenario diversity: OpenLane covers diverse weather conditions and maintains consistency between its training and validation distributions.The weather categories include clear, partly cloudy, overcast, rainy, and foggy conditions.
  • Lane complexity: 25% of OpenLane frames contain more than 6 lanes, while the maximum number of lanes per frame is 24.This exceeds the maximum number in most lane datasets.

C.2 Annotation Criterion

OpenLane annotates visible and contextually recoverable lanes, assigns lane categories and scene tags, and includes CIPO labels for complex driving scenarios. Its 3D annotations are generated from valid 2D ground truths when LiDAR support is available.

  • Lane annotation: OpenLane annotates lanes that are partly visible, extending occluded portions and completing invisible intermediate parts according to context.Fork lanes are represented as separate lanes with a shared split or merge point.
  • Lane annotation: Each lane is assigned one of 14 categories, including distinctions based on color, marking multiplicity, and line style.The category scheme includes single and double white or yellow lane markings.
  • 3D lane annotation: Valid 2D ground truths are transformed into 3D annotations except when no LiDAR points scan through them.The same annotation criterion therefore applies to 3D lanes.
  • Scene tags: Each segment receives weather, scene, and hours tags covering five weather types and three time-of-day divisions.The hours are daytime, night, and dawn/dusk.
  • CIPO annotation: CIPO annotations divide vehicles, pedestrians, and cyclists into four levels according to proximity, lane interaction, reaction distance, and likely path impact.Level 1 identifies the closest relevant object within reaction distance, while Level 4 covers objects unlikely to affect the future path.
  • 3D lane generation: Large-curvature cases require rotating the 3D point set before filtering and fitting, then rotating it back afterward.Rotation is used because direct point sorting may fail when one-to-one coordinate mapping does not hold.

D.1 Evaluation Metrics.

The 3D evaluation matches predictions and ground truth using edit distance and a 1.5m point-wise threshold, then derives recall and related metrics from matched lanes. The 2D task uses the classical CULane metric.

  • 3D metrics: A predicted lane is a true positive when 75% of its covered y-positions have point-wise distance below the 1.5m maximum allowed distance.Prediction–ground-truth matching is built on edit distance.
  • 3D metrics: 3D recall is computed as the percentage of matched ground-truth lanes.The evaluation follows Gen-LaneNet with small modifications and adds category accuracy on OpenLane.
  • 2D metrics: The 2D lane detection task is evaluated using the classical metric from CULane.This metric is separate from the modified Gen-LaneNet-based 3D evaluation.

D.2 Implementation Details

The experiments use standardized image and BEV resolutions, train for 100 epochs on eight V100 GPUs, and compare against reimplemented baselines. Additional results cover ONCE-3DLanes and feature-transformation ablations.

  • Training configuration: Images are resized to 360 × 480 and projected into a 208 × 108 BEV representation for fair comparison with prior methods.The retained settings include image resolution and BEV scale.
  • Training configuration: Models are trained for 100 epochs with batch size 8, using PyTorch and eight NVIDIA Tesla V100 GPUs.The implementation follows a common training setup for the compared models.
  • Additional results: PersFormer achieves the highest validation F-Score on ONCE-3DLanes, outperforming SALAD by over 10%.ONCE-3DLanes does not provide camera extrinsics, so PersFormer predefines a set of camera parameters.
  • Ablation study: An ablation on approximately 300 OpenLane segments compares naive one-to-one, learned, and IPM-based cross-attention mappings.The study argues that IPM-based cross attention is necessary in PersFormer.

Ablations.

The ablation identifies IPM guidance and BEV-query self-attention as important design choices, while qualitative comparisons report stronger lane reconstruction across OpenLane and Apollo scenarios.

  • IPM prior plays a vital role in guiding BEV feature generation compared with naive one-to-one mapping and learned reference-target mapping.
  • Multi-scale deformable attention provides competitive mapping from front-view features to BEV features.
  • The BEV-query self-attention module is important in the Transformer-style structure.
  • PersFormer achieves higher accuracy and captures more lanes for scene reconstruction than compared methods on OpenLane and Apollo.The OpenLane comparisons cover night, uphill/downhill, extreme-weather, intersection, and merge/split cases; Apollo comparisons cover curve and uphill/downhill cases.
  • OpenLane is distributed under non-commercial licensing, while the PersFormer and OpenLane toolkit code is released under Apache License 2.0.
  • Future work includes joint road-object detection and improved BEV representations, with possible adaptation of PersFormer to new tasks.
Loading 2203.11089v3…