Source-linked AI summary
End-to-end Lane Shape Prediction with Transformers
Ruijin Liu, Zejian Yuan, Tie Liu, Zhiliang Xiong
TL;DR
Lane detection pipelines based on feature extraction and post-processing struggle with efficiency, global context, and long, thin lane structures. The paper proposes an end-to-end transformer that directly predicts physically interpretable lane-shape parameters, achieving state-of-the-art accuracy with low resource use and strong adaptation to new scenes.
Problem
Existing lane detectors often use inefficient segmentation and post-processing while struggling to learn global context and long, thin lane structures.
Method
The method directly predicts parameters of a road- and camera-pose-based lane shape model using a transformer that captures non-local visual interactions.
Results
The method achieves state-of-the-art accuracy with low resource consumption and strong adaptability to the challenging self-collected FVL dataset.
Takeaways & Limitations
The end-to-end design provides physically interpretable lane outputs and supports efficient deployment across changing scenes.
Takeaways & Limitations
The lane model assumes globally consistent lane shape, sharing curvature-related parameters across lanes.
Abstract
from arXiv · showhide
Lane detection, the process of identifying lane markings as approximated curves, is widely used for lane departure warning and adaptive cruise control in autonomous vehicles. The popular pipeline that solves it in two steps -- feature extraction plus post-processing, while useful, is too inefficient and flawed in learning the global context and lanes' long and thin structures. To tackle these issues, we propose an end-to-end method that directly outputs parameters of a lane shape model, using a network built with a transformer to learn richer structures and context. The lane shape model is formulated based on road structures and camera pose, providing physical interpretation for parameters of network output. The transformer models non-local interactions with a self-attention mechanism to capture slender structures and global context. The proposed method is validated on the TuSimple benchmark and shows state-of-the-art accuracy with the most lightweight model size and fastest speed. Additionally, our method shows excellent adaptability to a challenging self-collected lane detection dataset, showing its powerful deployment potential in real applications. Codes are available at https://github.com/liuruijin17/LSTR.
1. Introduction
The paper replaces inefficient segmentation-and-post-processing with an end-to-end lane-shape predictor built around a transformer. Its physically interpretable outputs and non-local modeling target lane slenderness, global context, and deployment efficiency.
- Motivation: Existing CNN lane detectors commonly segment lanes before clustering and curve fitting, which is inefficient and overlooks global context.These limitations are especially relevant because lane markings are long, thin structures with few appearance clues and may be occluded.
- Method: The method directly regresses parameters of a lane shape model derived from road structures and camera pose.The parameters provide an explicit mathematical representation of each lane and physically interpretable outputs.
- Method: A transformer models pairwise visual-feature interactions to capture long-range dependencies, slender lane structures, and global context.The network is trained end-to-end with Hungarian loss, which supports one-to-one assignment and removes explicit non-maximal suppression.
- Results: The method achieves state-of-the-art accuracy with the lowest false positive rate, smallest model size, and fastest speed on TuSimple.The reported evaluation is conducted without additional bells and whistles.
- Results: On the challenging self-collected FVL dataset, the method shows strong adaptability to scenes absent from TuSimple, including night scenes.FVL covers multiple cities, urban and highway settings, and varied traffic and weather conditions.
2. Related Work
Prior lane detectors either rely on hand-crafted or dense-segmentation pipelines, or use end-to-end parameterizations with design and efficiency trade-offs. This work instead combines physically grounded lane-shape parameters with transformer-based non-local modeling.
- Traditional methods: Traditional feature-based methods extract lane segments with Hough-transform variants and cluster them, while model-based methods use geometry and road-surface priors.Model-based approaches describe lanes in more detail while retaining simplicity.
- Deep segmentation methods: Earlier deep methods generate dense segmentation maps and post-process them, but their performance is limited by learning long and thin lane structures.SCNN addresses global context through message passing, while other methods use extra scene annotations.
- End-to-end methods: The proposed method directly outputs lane-shape parameters and avoids intermediate representations and post-processing.This distinguishes it from segmentation-based pipelines.
- End-to-end methods: Line-CNN predicts offsets from hand-designed rays but requires ad-hoc heuristics and non-maximal suppression, whereas PolyLaneNet uses polynomial regression for efficiency.The passage notes that Line-CNN is slower than most lane detectors and that PolyLaneNet has an accuracy limitation.
- Proposed distinction: Unlike simple polynomial coefficients, this method’s parameters model road structure and camera pose and have explicit physical meanings.Its transformer additionally uses attention to model non-local interactions and reinforce global-context learning.
3. Method
The method predicts lane-shape parameters end to end with a transformer, using a physically motivated model of road geometry and camera pose. Hungarian matching trains fixed output curves against ground-truth lanes, while attention captures global context and slender structures.
- 3.1. Lane Shape Model: The lane shape model uses a cubic polynomial prior for lane lines on flat ground, then re-parameterizes projections for camera intrinsics, extrinsics, and pitch.Vertical offsets α and β encode the upper and lower lane boundaries.
- 3.1. Lane Shape Model: Shared curvature parameters model globally consistent road shape across lanes, while lane-specific terms represent individual lane offsets and boundaries.The approximation assumes equal curvature from left to right lanes.
- 3.2. Hungarian Fitting Loss: The model predicts a fixed set of curves, classifying outputs as lanes or non-lanes and padding the ground-truth set when fewer lanes are present.Each predicted curve contains a classification indicator and lane-shape parameters.
- 3.2. Hungarian Fitting Loss: Hungarian fitting loss performs bipartite matching between predicted curves and ground-truth lanes before optimizing lane-specific regression losses.Matching costs combine classification probabilities with L1 fitting terms and boundary-offset errors.
- 3.3. Architecture: A reduced ResNet18 backbone flattens low-resolution features into a sequence, and transformer encoder-decoder layers produce decoded outputs for direct parameter prediction.The decoder uses an empty N × C query matrix and learned lane embeddings to decode all curve parameters in parallel.
4. Experiments
Experiments show that the method is accurate, efficient, and transferable, while ablations clarify how shape consistency, attention, decoder design, and prediction count affect performance.
- 4.1. Comparisons with State-of-the-Art Methods: The method outperforms PolyLaneNet by 2.82% accuracy with 5 × fewer parameters and runs 3.6 × faster on TuSimple.It is also 0.69% lower in accuracy than Line-CNN but runs 14 × faster.
- 4.1. Comparisons with State-of-the-Art Methods: On TuSimple, the method achieves competitive accuracy, the lowest false positive rate, fewest parameters, and much faster speed than other two-stage approaches.The authors connect the low false positive rate to reduced risks of false alarming and rapid changes in real applications.
- 4.2. Ablation Study: Cubic curve approximation with shape consistency is the best tested shape model, and shape consistency further improves accuracy.The consistency constraint lets remote lane points contribute to a shared curvature estimate across lanes.
- 4.2. Ablation Study: Removing encoder self-attention drops accuracy by 1.01%, while excessive encoder layers degrade generalization despite similar training accuracy.The attention maps show non-local context identifying slender lane structures despite occlusion or weak local appearance clues.
- 4.2. Ablation Study: The decoder focuses on each output slot’s own slender structure, helping separate lane instances without additional non-maximal suppression.As decoder layers increase, overall performance gradually degrades due to overfitting.
- 4.2. Ablation Study: Seven predicted curves give the best ablation result; too few or too many predictions degrade performance because the balance of negatives and positives becomes unsuitable.Too few negatives make training inefficient, whereas too many negatives divert optimization from positive curve fitting.
- 4.1. Comparisons with State-of-the-Art Methods: Without supervised training on FVL, the model shows excellent transfer performance on challenging new scenes.The authors attribute this to avoiding distribution-dependent prior processing and aggregating more generalized contextual information.
5. Conclusion
The paper presents an end-to-end lane detector that predicts physically interpretable lane-shape parameters, using transformer blocks to model global context and slender lane structures. It achieves strong accuracy and efficiency while adapting robustly across datasets.
- 5. Conclusion: The detector directly outputs parameters of a lane shape model whose formulation reflects road structures and camera state.These parameters have explicit physical meanings and can support interpretation of road curvature and camera pitch under specified camera priors.
- 5. Conclusion: Transformer blocks learn global context for inferring occluded lane parts and capture long, thin structures, especially near the horizon.
- 5. Conclusion: The complete method achieves state-of-the-art lane-detection performance with the least parameters and running-time consumption.
- 5. Conclusion: The method adapts robustly to dataset changes, supporting easier mobile deployment and more reliable operation.
- 5. Conclusion: Future work includes complex, fine-grained lane detection and adding tracking functionality.
7. Appendix
The appendix derives the lane-curve formulation from perspective projection, camera geometry, and pitch transformation. It defines the image-to-ground mapping and uses these relationships to obtain the tilted-image curve function.
- 7. Appendix: Perspective projection maps an image-plane pixel (u, v) to a ground-plane point (X, Z) using camera parameters.
- 7. Appendix: The mapping uses fu and fv as pixel-size-to-focal-length ratios and H as camera height.
- 7. Appendix: Substituting the projection equations into the lane model and simplifying produces the polynomial curve expression used to combine parameters into Eq. 2.
- 7. Appendix: A pitch-angle transformation relates coordinates in an untilted image plane to pitch-transformed coordinates before deriving the tilted-camera curve function.