Source-linked AI summary
Towards End-to-End Lane Detection: an Instance Segmentation Approach
Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool
TL;DR
Lane detection systems struggle with road-scene variation, fixed lane counts, and lane changes. The paper proposes end-to-end lane instance segmentation with an image-conditioned perspective transformation, achieving 50 fps and competitive tuSimple performance while adapting lane fitting to ground-plane changes.
Problem
Existing lane detection approaches use hand-crafted heuristics or fixed lane classes, limiting robustness to scene variation, lane counts, and lane changes.
Method
The method combines end-to-end LaneNet instance segmentation with H-Net, which learns an image-conditioned perspective transformation for lane fitting.
Results
50 frames per second and fourth place on the tuSimple challenge, with a 0.5% difference from the first entry.
Takeaways & Limitations
The method detects a variable number of lanes, copes with lane changes, and adapts lane fitting to ground-plane slope changes.
Abstract
from arXiv · showhide
Modern cars are incorporating an increasing number of driver assist features, among which automatic lane keeping. The latter allows the car to properly position itself within the road lanes, which is also crucial for any subsequent lane departure or trajectory planning decision in fully autonomous cars. Traditional lane detection methods rely on a combination of highly-specialized, hand-crafted features and heuristics, usually followed by post-processing techniques, that are computationally expensive and prone to scalability due to road scene variations. More recent approaches leverage deep learning models, trained for pixel-wise lane segmentation, even when no markings are present in the image due to their big receptive field. Despite their advantages, these methods are limited to detecting a pre-defined, fixed number of lanes, e.g. ego-lanes, and can not cope with lane changes. In this paper, we go beyond the aforementioned limitations and propose to cast the lane detection problem as an instance segmentation problem - in which each lane forms its own instance - that can be trained end-to-end. To parametrize the segmented lane instances before fitting the lane, we further propose to apply a learned perspective transformation, conditioned on the image, in contrast to a fixed "bird's-eye view" transformation. By doing so, we ensure a lane fitting which is robust against road plane changes, unlike existing approaches that rely on a fixed, pre-defined transformation. In summary, we propose a fast lane detection algorithm, running at 50 fps, which can handle a variable number of lanes and cope with lane changes. We verify our method on the tuSimple dataset and achieve competitive results.
I. INTRODUCTION
The paper frames lane detection as an end-to-end instance segmentation problem to handle variable lane counts and lane changes, while learning image-conditioned perspective transformations for robust lane fitting.
- Camera-based lane detection supports road positioning and subsequent lane departure or trajectory planning decisions in autonomous driving.
- Traditional systems combine hand-crafted features, heuristics, and post-processing, creating robustness issues under road scene variations.
- Deep segmentation methods improve dense prediction but remain limited to a predefined number of lanes and cannot cope with lane changes.
- The proposed approach treats each lane as its own instance within the lane class using jointly trained segmentation and embedding branches.
- A learned, image-conditioned perspective transformation replaces a fixed transformation to make lane fitting robust to road-plane changes.
- The paper organizes the pipeline around semantic and instance lane segmentation, parametric lane conversion, experiments, and conclusions.
II. METHOD
LaneNet combines binary lane segmentation with instance embeddings to assign pixels to individual lanes, while H-Net learns an image-conditioned transformation for fitting lane curves despite non-flat ground planes.
- LaneNet performs end-to-end instance segmentation so it can handle lane switching and an unrestricted number of lanes.
- LaneNet combines binary lane segmentation with clustering-based embeddings that assign each lane pixel to its corresponding lane instance.
- H-Net estimates an image-conditioned ideal perspective transformation in which each lane can be optimally fitted with a low-order polynomial.
A. LANENET
LaneNet treats lane detection as end-to-end instance segmentation, combining binary lane segmentation with pixel embeddings that separate lanes into clusters. This removes a fixed lane-count constraint and supports lane changes.
- A. LANENET: LaneNet jointly trains segmentation and clustering branches in a multi-task network for real-time lane instance segmentation.The segmentation branch identifies lane pixels, while the embedding branch separates them into instances.
- A. LANENET: The segmentation branch outputs a binary map distinguishing lane pixels from background, including annotated lanes through occlusions and absent markings.Ground-truth lane points are connected into one line per lane, including dashed or faded regions.
- A. LANENET: The embedding branch brings same-lane pixel embeddings together and separates embeddings from different lanes to form unique lane clusters.Its clustering loss uses variance and distance terms to control within-lane compactness and between-lane separation.
- A. LANENET: Clustering repeatedly thresholds embeddings around mean-shifted cluster centers until all lane embeddings are assigned.The procedure uses a radius of 2δv with δd > 6δv.
- A. LANENET: LaneNet modifies ENet into two branches that share only the first two encoder stages, while later encoder and decoder components remain branch-specific.This design avoids the unsatisfying results associated with fully sharing the encoder between tasks.
B. CURVE FITTING USING H-NET
H-Net learns an image-conditioned perspective transformation before polynomial lane fitting, addressing errors caused by fixed homographies under ground-plane changes. The transformed fit is evaluated at target heights and reprojected into the original image.
- B. CURVE FITTING USING H-NET: Fixed transformation matrices cause lane-fitting errors when ground-plane changes shift the vanishing point.The problem occurs because the matrix is calculated once and kept fixed for all images.
- B. CURVE FITTING USING H-NET: H-Net predicts image-conditioned perspective parameters optimized so transformed lane points can be fitted with a 2nd- or 3rd-order polynomial.The learned projection adapts under ground-plane changes and has 6 degrees of freedom.
- B. CURVE FITTING USING H-NET: Lane pixels are transformed by H-Net’s matrix H, then a least-squares algorithm fits an n-degree polynomial through the transformed points.The transformed representation makes curved lanes fit with a 2nd- to 3rd-order polynomial.
- B. CURVE FITTING USING H-NET: The fitted curve is evaluated at selected y′ positions, and the resulting points are transformed back to the original image space.This produces lane positions at different image heights.
- B. CURVE FITTING USING H-NET: H-Net is trained end-to-end with a fitting loss built from transformed ground-truth lane points and a closed-form least-squares polynomial solution.Because the fitting solution is differentiable, automatic differentiation supplies the gradients.
A. Dataset
The tuSimple dataset provides large-scale lane-detection evaluation across highway scenes and weather conditions, with lane-position annotations at discretized y-positions. Performance is summarized using accuracy, false positives, and false negatives.
- A. Dataset: tuSimple contains 3626 training and 2782 testing images recorded on highways with varying lane counts, times of day, and good or medium weather.Each image also includes 19 preceding, unannotated frames.
- A. Dataset: Annotations provide lane x-positions at a number of discretized y-positions, including ego lanes and left/right lanes.A fifth lane may be added during lane changes to avoid confusion.
- A. Dataset: Accuracy is the average number of correct lane points per image, with correctness determined by a thresholded difference from ground truth.The evaluation also reports false-positive and false-negative scores.
- A. Dataset: False-positive and false-negative measures count wrongly predicted lanes and missed ground-truth lanes relative to their corresponding predicted or ground-truth totals.The definitions use Fpred, Npred, Mpred, and Ngt.
B. Setup
The setup evaluates lane detection speed and fitting behavior using LaneNet outputs, polynomial fits, and transformation-based evaluation on 512x256 images.
- Speed: 50 frames per second is achieved for lane detection at 512x256 resolution with 4-dimensional pixel embeddings and a 3rd-order polynomial fit.The full component breakdown is reported in Table IV.
- Visual results: Visual results show ground-truth lane points, LaneNet outputs, and final lane predictions after lane fitting across the image-processing pipeline.
- Evaluation setup: The evaluation compares 2nd- and 3rd-order polynomial fitting under different transformations using validation-set MSE and average misses per lane.Unfittable points are excluded from MSE but counted as misses.
C. Experiments
Experiments compare lane fitting without transformation, with a fixed transformation, and with H-Net’s conditional transformation, including evaluation on tuSimple.
- Interpolation method: Fixed transformation improves fitting over direct image-space fitting, while a 3rd-order polynomial performs better than a 2nd-order polynomial.
- Interpolation method: Ground-plane slope changes cause points near the vanishing point to be fitted incorrectly under a fixed transformation and counted as misses.
- Interpolation method: H-Net’s conditional transformation outperforms fixed transformation for lane fitting and fits all points despite ground-plane slope changes.The transformation is optimized for lane fitting rather than fixed across images.
- tuSimple results: 4th place on the tuSimple challenge is achieved with LaneNet, 3rd-order polynomial fitting, and H-Net’s transformation, within 0.5% of the first entry.The paper notes that training data and speed performance are unclear for other entries.
IV. CONCLUSION
The paper concludes with an end-to-end lane detector that runs at 50 fps, supports variable lane counts and lane changes, and uses image-conditioned perspective transformation for robust lane fitting.
- IV. CONCLUSION: 50 fps end-to-end lane detection is presented with support for a variable number of lanes and lane-change maneuvers.
- IV. CONCLUSION: An image-conditioned network generates perspective-transformation parameters optimized for low-order polynomial lane fitting.
- IV. CONCLUSION: The learned transformation adapts to ground-plane slope changes, unlike the fixed bird’s-eye-view approach.