PersFormer: 3D Lane Detection via Perspective Transformer and the OpenLane Benchmark

• Perspective-view lane detection is not suited to complex scenarios, while downstream planning and control need BEV lane locations for task alignment and compatibility with other sensors.Conventional one-to-one image-to-BEV transformations retain perspective scale variance, which can limit BEV feature quality.Source: §1 · p. 2
• Ignoring lane height makes projected BEV lanes diverge or converge on uphill or downhill roads, but public benchmarks lack height labels and synthetic data leaves realistic performance and domain adaptation unresolved.The paper identifies other non-flat-road cases, including bumps and sharp turns, as challenges to the planar-ground assumption.Source: §1 · p. 2, Figure 1 · p. 3
• PersFormer uses a Transformer-based spatial feature transformation that attends to local front-view regions and aggregates relevant context to form fine-grained BEV features.It adopts deformable attention to reduce computational memory requirements and dynamically adjust keys in cross-attention, unlike direct one-to-one Inverse Perspective Mapping.Source: §1 · pp. 2–3
• OpenLane provides 200,000 annotated frames and over 880,000 lanes across 14 categories, with scene, weather, and closed-in-path-object annotations.The authors describe it as the first large-scale realistic 3D lane dataset, designed to support research on 3D lane detection and other autonomous-driving topics.Source: §1 · p. 3
• PersFormer introduces a Transformer-based architecture for spatial feature transformation and simultaneously unifies 2D and 3D lane detection; experiments show it outperforms state-of-the-art 3D lane detection algorithms.Source: §1 · p. 3

Vision Transformers in Bird’s-Eye-View (BEV)

• Transformer cross-attention offers a learnable transformation across views, and Transformer-based view transformation has succeeded in detection, prediction, and planning.BEV feature projection and downstream processing have become more prevalent and improved performance; the cited applications include 3D detection [31], prediction, and planning.Source: §2 · pp. 3–4
• PersFormer combines camera-aware BEV feature generation with cross-attention for view transformation, addressing gaps in earlier BEV methods.Prior methods omit attention and/or camera geometry; 3D-LaneNet uses camera matrices with IPM, while DETR3D lacks explicit BEV modeling.Source: §2 · p. 4

Lane Detection Benchmarks

• OpenLane is presented as the first large-scale, realistic 3D lane dataset, with broader data-distribution and task diversity than scenario-specific existing benchmarks.The passage notes numerous prior datasets and directs readers to Tab. 1 for benchmark comparisons.Source: §2 · p. 4

3D Lane Detection

• PersFormer targets 3D lane detection limitations with improved feature representation and unified 2D/3D lane detection.Planar assumptions fail on uphill/downhill roads and bumps; multi-modal approaches have hardware and computation costs, while monocular IPM methods struggle in curving or crush turns. 3D-LaneNet uses STN for feature projection, and Gen-LaneNet decouples segmentation encoding from 3D prediction in a two-stage network.Source: §2 · p. 4

3.1 Approach Overview

• A ResNet backbone feeds a Perspective Transformer and modified LaneATT and 3D-LaneNet heads that predict lane coordinates and types.The framework represents lanes with categorical attributes and pointwise visibility predictions in both image and BEV views.Source: §3.1 · pp. 5–6, §3 · p. 5

3.2 Proposed Perspective Transformer

• Perspective Transformer uses camera parameters to initialize BEV-to-front-view correspondences, then learns local reference-point offsets for cross-attention to build BEV features.The IPM-based mapping supplies a geometric prior; attending to partial local key-value pairs avoids the computational burden of full attention and helps handle camera-parameter noise and scale variance.Source: §3.2 · pp. 6–7

3.3 Simultaneous 2D and 3D Lane Detection

• Seven groups of incline-angle BEV anchors are projected into the front view and shared across 2D/3D branches to align features for joint optimization.The 3D anchors predict lane x-position, height, and visibility at fixed y samples; corresponding 2D anchors predict image-space coordinates without height.Source: §3.3 · pp. 8–9

3.4 Prediction Loss

• A U-Net head predicts binary lane segmentation in BEV as an auxiliary task, while learnably weighted 2D, 3D, and segmentation losses train the framework.Classification uses cross-entropy, lane-shape regression uses L1, and visibility and segmentation use binary cross-entropy.Source: §3.4 · p. 9

4.1 Highlights over Previous Benchmarks

• OpenLane provides 200K frames and over 880K annotated lanes—33% and 35% more, respectively, than CurveLanes —using Waymo Open data and its format and evaluation pipeline.Source: §4.1 · pp. 9–10
• OpenLane annotates all lanes, including opposite-direction lanes, with 14 categories and scene tags plus closest-in-path object (CIPO) labels.A frame can contain as many as 24 lanes, and about 25% of frames have more than 6; double-yellow solid and single-white solid or dashed lanes account for almost 90% of lanes, producing a long-tail distribution.Source: §4.1 · p. 10

4.2 Generation of High-quality Annotation

• Waymo Open was chosen as the dataset basis because its reconstruction precision and scalability outperformed the other compared candidates.Accurate localization and occlusions are identified as key challenges in constructing a real-world 3D lane dataset.Source: §4.2 · pp. 10–11
• The annotation pipeline interpolates 3D positions for 2D lane points from filtered point clouds, splices lanes across frames, marks certain points invisible, and smooths outliers.It first generates 2D labels with tracking IDs, categories, and ground-truth points; point-cloud filtering uses 3D object boxes and a threshold, while visibility marking handles points projected beyond a lane’s 2D annotation endpoint.Source: §4.2 · p. 11

5.1 Results on OpenLane

• PersFormer achieves the highest F-Score on OpenLane’s full validation set and every scenario, beating the previous best 3D method by 6.4% in F-Score.The reported comparison covers the entire validation set and scenario-specific sets.Source: §5.1 · pp. 11–12, Table 2 · p. 12
• On OpenLane, PersFormer’s 2D head beats LaneATT by 11%; its 3D results improve far error, but near x-axis error remains unsatisfactory.The paper also reports satisfying lane-type classification accuracy and presents the first baseline result.Source: §5.1 · p. 12, Table 3 · p. 12, Table 4 · p. 12

5.2 Results on Apollo 3D Synthetic

• On Apollo 3D Lane Synthetic, PersFormer achieves the best F-Score on every scene set, with comparable X/Z error (m) to previous methods.The authors note that the dataset is limited in scale to 10K frames.Source: §5.2 · pp. 12–13, Table 5 · p. 13

5.3 Ablation Study

• Relative to a pure 3D baseline, multi-task learning improves performance by 1.7%, the new anchor by 0.98%, and their combination by 2.5% in 3D and 2.6% in 2D.The combined anchor and multi-task setup is Exp.5.Source: §5.3 · pp. 13–14, Table 6 · p. 14
• With Perspective Transformer and the new anchor, improvement reaches 4.9%; auxiliary binary segmentation raises it to 6.02%, while improving the 2D task by 9.7%.These are Exp.6 and the complete model, Exp.7, respectively; the authors interpret the ablations as evidence that fine-grained BEV features matter for 3D lane detection.Source: §5.3 · p. 14, Table 6 · p. 14

6 Conclusions

• PersFormer is a novel Transformer-based detector for 2D and 3D lanes.Source: §6 · p. 14
• OpenLane is a large-scale, realistic dataset for 3D lane detection.Source: §6 · p. 14
• Fine-grained BEV features with explicit prior and supervision can significantly improve lane-detection performance.Source: §6 · p. 14
• The real-world 3D lane dataset aligns the needs of academic and industrial communities.Source: §6 · p. 14

A.1 Lane Detection Benchmarks

• OpenLane and ONCE-3DLanes have similar frame counts (200K and 211K), but differ in lane density, camera extrinsics, segment tags, and annotation attributes.More than 25% of OpenLane frames have over six lanes, versus fewer than 10% for ONCE-3DLanes; OpenLane also includes lane attributes and tracking IDs, while ONCE-3DLanes provides only 3D positions.Source: §A.1 · p. 19

A.2 2D Lane Detection

• Segmentation-based methods face instance-discrimination, output-limit, or computation issues, while grid-based and anchor-based methods offer alternatives.The authors adopt an anchor-based formulation, redesigning anchors to balance accuracy and efficiency despite challenges in defining anchors and the computational cost of many variants.Source: §A.2 · pp. 19–20

B Algorithm

• Appendix B organizes PersFormer’s method details around its backbone, overall structure, unified anchor design, and component loss functions.The overview identifies these components but does not provide further architectural specifics in the supplied passage.Source: B Algorithm · p. 20

B.1 Backbone

• PersFormer uses an EfficientNet backbone to produce four scaled front-view feature maps, each transformed into a BEV feature map by the Perspective Transformer.The map resolutions are 180 × 240, 90 × 120, 45 × 60, and 22 × 30; the backbone supports joint 2D/3D branches and can use FPN or omit it.Source: §B.1 · p. 20

B.2 Anchor Details

• BEV anchors use seven inclines at each x-axis start spaced eight pixels apart, and are projected into image space to form corresponding 2D anchors.Ground-truth association uses minimum edit distance at fixed 3D y positions (5–100 m) or 72 sampled 2D heights, rather than the closest anchor at a single reference position.Source: §B.2 · pp. 20–21

B.3 Loss Function

• The loss combines 2D lane detection, 3D lane detection, and intermediate segmentation with learnable weights; 3D classification, regression, and visibility use cross-entropy, L1, and binary cross-entropy losses.The 2D losses mirror the 3D classification, regression, and visibility losses in 2D form, while segmentation also uses binary cross-entropy.Source: §B.3 · p. 21

C Details on OpenLane Benchmark

• The OpenLane appendix covers dataset statistics, annotation criteria, visualization examples, and the algorithms used to generate the dataset.OpenLane annotations include scene tags recorded by segment, as noted in the benchmark comparison.Source: §A.1 · p. 19, C Details on OpenLane Benchmark · p. 22

C.1 Dataset Statistics

• OpenLane has 1,150 segments split into 798/202/150 for train/validation/test; the 1,000 annotated non-test segments comprise 200K frames at 10 FPS.The test set is reserved for online leaderboard evaluation, and the original train/validation partition is retained for fair comparison with other tasks.Source: §C.1 · p. 22
• Weather is varied and balanced across training and validation, while lane categories are long-tailed, with common types accounting for almost 90% of lanes.Scene-tag counts indicate diverse data collection; single white solid and dashed lanes and double yellow solid lanes make up almost 90% of total lanes.Source: §C.1 · p. 22, Figure 7 · p. 23
• Only around 20% of frames have absolute height variation below 0.5m, while over 50% exceed 1m, underscoring the need for 3D lane detection.The authors conclude that the statistics and examples make OpenLane the most challenging among existing lane detection datasets.Source: §C.1 · p. 22
• 24 lanes is the maximum per frame, and 25% of frames have more than 6 lanes, exceeding the maximum in most lane datasets.The distribution of lane counts per frame is shown in Fig. 8.Source: §C.1 · p. 22, Figure 8 · p. 23

C.2 Annotation Criterion

• The 2D annotation principle is to identify all visible lanes between the left and right road edges.The protocol then addresses partially occluded or abraded lanes and complex lane-count changes rather than omitting those cases.Source: §C.2 · pp. 23–24
• OpenLane annotates visible lanes even when occluded or abraded, retains fork lanes for evaluation, and assigns 14 categories, segment-unique tracking IDs, and ego-relative ranks to four key lanes.For forks, adjacent lanes are annotated separately with a common start or end point; non-temporary traffic bollards count as curbsides.Source: §C.2 · pp. 24–25
• Valid 2D ground truths are converted to 3D annotations using the main-body generation method, except lanes without LiDAR points scanning through.The stated lane-annotation criteria therefore also apply to 3D lanes.Source: §C.2 · p. 25
• Scene tags label each segment by five weather types, five location types, and three time-of-day groups to support robustness investigations.Weather categories are clear, partly cloud, overcast, rainy, and foggy; locations are residential, urban, suburbs, highway, and parking lot; hours are daytime, night, and dawn/dusk.Source: §C.2 · p. 25
• CIPO annotations divide vehicles, pedestrians, and cyclists into four levels, from the closest reaction-distance object mostly in the ego lane to distant, off-road, or parked objects.Level 2 captures objects interacting with ego-lane boundaries, often during cut-in or cut-out; Level 3 includes nearby or drivable-area objects with under 50% occlusion, including opposite-direction vehicles.Source: §C.2 · pp. 25–26

C.3 3D lane Generation

• For large-curvature lanes, the generation pipeline rotates the points into a temporary coordinate system for filtering and fitting, then rotates them back.This avoids direct sorting when one-to-one mapping fails along either the x or y direction.Source: §C.3 · p. 26

D.1 Evaluation Metrics.

• For 3D lanes, Gen-LaneNet matching counts a prediction as a true positive when 75% of covered y-positions are within 1.5m, and F-score reports regression performance.Recall is the percentage of matched ground-truth lanes; precision is the percentage of matched predictions.Source: §D.1 · p. 27
• On OpenLane, category accuracy is measured over matched lanes, and only PersFormer’s accuracy is reported because other 3D methods do not support classification.Source: §D.1 · pp. 27–28
• For the 2D task, the evaluation adopts the classical CULane metric.Source: §D.1 · p. 28

D.2 Implementation Details

• The implementation resized images to 360 × 480, projected them to 208 × 108 BEV, and trained with batch size 8 for 100 epochs.The authors retained many image-resolution and BEV-scale settings for fair comparison and re-implemented 3D-LaneNet and Gen-LaneNet on OpenLane.Source: §D.2 · p. 28

3D Comparisons on ONCE-3DLanes.

• PersFormer achieved the highest validation-set F-Score on ONCE-3DLanes, outperforming SALAD by over 10%.The passage identifies ONCE-3DLanes as another real-world 3D lane dataset and F-Score as its evaluation metric.Source: Table 8 · p. 29, 3D Comparisons on ONCE-3DLanes · p. 29

D.2 Implementation Details

• Training used Adam with a 2×10^-4 base learning rate on 8 NVIDIA Tesla V100 GPUs.The optimizer settings were β1 = 0.9, β2 = 0.999, and weight decay of 10^-4.Source: §D.2 · pp. 28–29

D.3 More Experimental Results

• The supplementary experiments include further 3D comparisons on ONCE-3DLanes, additional ablations, and qualitative examples.The section introduces these as additional experimental results.Source: §D.3 · p. 29

3D Comparisons on ONCE-3DLanes.

• Because ONCE-3DLanes provides no camera extrinsics, PersFormer used predefined extrinsics with camera height 1.5 m and pitch 0.5 to fit its IPM process.The passage states this choice does not affect evaluation results because it is used only to fit the IPM process.Source: 3D Comparisons on ONCE-3DLanes · p. 29
• The feature-transformation ablation found that IPM-based mapping outperformed naive one-to-one and learned fully connected mappings on an approximately 300-segment OpenLane subset.The naive mapping ignores camera parameters, while the learned mapping uses several fully connected layers; both underperformed IPM-based mapping.Source: Ablations · p. 29
• Multi-scale Deformable Attention slightly underperformed the final design, with the authors citing hyperparameter tuning and small-scale-to-large-scale feature effects as possible factors.This was the Exp. 3 alternative for mapping multi-scale front-view features to multi-scale BEV features; the final design is Exp. 5.Source: Ablations · p. 29

Ablations.

• The IPM prior plays a vital role in guiding BEV-feature generation compared with naive one-to-one and learned reference-target mappings.The supplied passage gives qualitative ablation conclusions but no numeric results.Source: Table 9 · p. 30
• MSDeformAttn from Deformable DETR [72] is competitive for mapping multi-scale front-view features to multi-scale BEV features, and BEV-query self-attention is important.The self-attention ablation is identified as Exp. 4 in the ordinary Transformer design.Source: Table 9 · p. 30, Ablations · p. 30

Visualization

• PersFormer is reported to achieve higher accuracy and capture more lanes than the compared methods in varied OpenLane scenarios.The examples cover night, uphill/downhill, extreme weather, intersections, and merges/splits, against 3D-LaneNet [16] and Gen-LaneNet [20].Source: Visualization · p. 30, Figure 15 · p. 31, Figure 16 · p. 32
• On Apollo, qualitative examples show PersFormer capturing more lanes and achieving higher accuracy than the compared methods.The examples cover curve and uphill/downhill cases against 3D-LaneNet [16] and Gen-LaneNet [20].Source: Visualization · p. 30, Figure 17 · p. 33

E License of Assets

• OpenLane data is distributed under non-commercial Creative Commons and Waymo Dataset license terms, while PersFormer code and toolkit use Apache License 2.0.Pretrained ResNet weights use MIT; integrated Deformable-DETR and Gen-LaneNet code use Apache 2.0, and LaneATT code uses MIT.Source: E License of Assets · p. 30

F Outlook

• Because OpenLane is based on Waymo Open Dataset, the authors identify road-object joint detection as a possible future direction.They also say better BEV representation remains to be explored and PersFormer may be adapted to new tasks.Source: F Outlook · p. 30