Source-linked AI summary
Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks
Qin Zou, Hanwen Jiang, Qiyu Dai, Yuanhao Yue, Long Chen, Qian Wang
TL;DR
Single-frame lane detection can be unreliable under shadows, degraded markings, and occlusion, even though lane structure persists across continuous driving frames. The paper addresses this with a CNN–RNN architecture that encodes each frame, models feature sequences with LSTM, and predicts the current lane. Experiments report better performance than single-image baselines and competing models, especially in challenging situations.
Problem
Single-frame lane detection often performs poorly under heavy shadows, severe mark degradation, and vehicle occlusion, limiting robust detection in difficult driving scenes.
Method
A hybrid encoder–decoder network combines CNN-based frame feature abstraction with an LSTM that processes continuous-frame feature sequences for lane prediction.
Results
The proposed models achieve significantly better results than single-image baselines and show higher precision, recall, and accuracy than other models, including stable detection in diverse challenging situations.
Takeaways & Limitations
Using multiple continuous frames improves robust lane detection, while ConvLSTM provides advantages over fully connected LSTM for sequential feature learning and target-information prediction.
Abstract
from arXiv · showhide
Lane detection in driving scenes is an important module for autonomous vehicles and advanced driver assistance systems. In recent years, many sophisticated lane detection methods have been proposed. However, most methods focus on detecting the lane from one single image, and often lead to unsatisfactory performance in handling some extremely-bad situations such as heavy shadow, severe mark degradation, serious vehicle occlusion, and so on. In fact, lanes are continuous line structures on the road. Consequently, the lane that cannot be accurately detected in one current frame may potentially be inferred out by incorporating information of previous frames. To this end, we investigate lane detection by using multiple frames of a continuous driving scene, and propose a hybrid deep architecture by combining the convolutional neural network (CNN) and the recurrent neural network (RNN). Specifically, information of each frame is abstracted by a CNN block, and the CNN features of multiple continuous frames, holding the property of time-series, are then fed into the RNN block for feature learning and lane prediction. Extensive experiments on two large-scale datasets demonstrate that, the proposed method outperforms the competing methods in lane detection, especially in handling difficult situations.
I. INTRODUCTION
Single-frame lane detection can fail under shadows, degraded markings, and occlusion, motivating continuous-frame inference. The paper proposes an end-to-end CNN–RNN architecture that uses temporal features for lane prediction.
- Single-frame methods often perform poorly under heavy shadows, severe road-mark degradation, and vehicle occlusion.
- Lane continuity and overlap between neighboring frames make previous frames informative for predicting a damaged or degraded current lane.
- The proposed hybrid network combines a convolutional network for frame-wise feature abstraction with an LSTM recurrent network for time-series lane prediction.
- The method uses an encoder–decoder, fully convolutional architecture to produce a lane-segmentation map while recurrently processing features from multiple frames.
- Two new evaluation datasets cover 12 challenging situations and rural roads, while the TuSimple dataset is expanded with additional labeled frames.
II. RELATED WORK
Prior lane-detection work includes geometric, energy-minimization, and deep-learning approaches, but commonly processes one frame at a time. This paper instead formulates detection over continuous frames and integrates CNN and RNN processing end to end.
- Traditional methods: Traditional methods use geometric modeling, line fitting, energy minimization, or tracking filters to detect or follow lane markings.
- Deep-learning-based methods: Deep-learning approaches include encoder–decoder CNNs, fully convolutional networks with optimization, CNN–RNN pipelines, and GAN-based segmentation.
- Proposed approach: The proposed method models lane detection as a time-series problem using multiple continuous frames rather than only the current frame.
- Proposed approach: The method seamlessly integrates CNN and RNN components into an end-to-end trainable lane-detection network.
B. ConvLSTM for Video Analysis
The paper motivates recurrent processing for video-like lane detection because lanes and adjacent driving-scene frames contain temporal structure. Its encoder–LSTM–decoder design converts continuous frames into a current-frame lane probability map.
- ConvLSTM is an LSTM variant with convolution operations that processes two-dimensional time-series inputs for video analysis.
- Single-image lane models are insufficient under heavy shadow, degraded markings, and vehicle occlusion, motivating temporal prediction.
- The encoder processes continuous frames into feature-map sequences, LSTM predicts lane information, and the decoder produces the lane probability map.
B. Network Design
The network combines convolutional encoder-decoder processing with ConvLSTM temporal modeling to predict lanes from continuous driving frames. Two variants, SegNet-ConvLSTM and UNet-ConvLSTM, embed ConvLSTM within fully convolutional segmentation architectures.
- LSTM network: The RNN block receives encoder-extracted feature maps from multiple continuous frames modeled as a time series.The paper uses LSTM because it can forget unimportant information and retain essential information.
- LSTM network: ConvLSTM replaces each LSTM gate’s matrix multiplication with convolution for end-to-end feature extraction from time-series data.This design is used instead of computationally costly fully connected LSTM processing.
- LSTM network: The ConvLSTM equations define input, forget, and output gates using convolution, recurrent activations, cell memory, and elementwise products.Xt is the encoder feature map at time t; σ and tanh are nonlinearities, while ∗ and ◦ denote convolution and Hadamard product.
- LSTM network: The ConvLSTM uses encoder-sized feature maps, 3 × 3 kernels, two hidden layers, and 512 dimensions per hidden layer.Feature-map sizes are 8×16 for UNet-ConvLSTM and 4×8 for SegNet-ConvLSTM.
- Encoder-decoder network: The encoder-decoder framework formulates lane detection as semantic segmentation, using convolution and pooling for abstraction and deconvolution and upsampling for target recovery.The architecture is designed to support output maps with the same size as the input for end-to-end training.
- Encoder-decoder network: SegNet-ConvLSTM and UNet-ConvLSTM embed ConvLSTM blocks into SegNet and U-Net encoder-decoder networks with fully convolutional encoder and decoder blocks.Decoder operations correspond to encoder operations, while U-Net uses feature-map appending between corresponding sub-blocks.
C. Training Strategy
Training uses back-propagation through the end-to-end network, with sequence-aware updates, class-weighted segmentation loss, pretrained initialization, and staged optimization.
- Training procedure: The end-to-end network updates convolutional-kernel and ConvLSTM parameters through back-propagation toward the ground truth.The training strategy considers pretrained initialization, sequence length, loss construction, and optimizer selection.
- Initialization: ImageNet-pretrained SegNet and U-Net weights initialize the proposed networks to reduce training time and transfer suitable weights.Training from scratch eventually produces test accuracy close to that obtained with pretrained weights.
- Sequence training: For N continuous input images, each ConvLSTM weight update coefficient is divided by N; experiments set N=5 and vary N experimentally.This scaling accounts for the multiple-frame input during back-propagation.
- Loss function: A weighted cross-entropy loss balances the lane class in the pixel-wise discriminative segmentation task.The class weight is set as a ratio between pixel counts of the two classes across the training set.
- Optimization: Training begins with Adam and changes optimizer when training accuracy reaches 90%, using an initial learning rate of 0.01.The staged strategy addresses Adam’s high gradient descent rate and tendency toward local minima.
IV. EXPERIMENTS AND RESULTS
The experiments evaluate the proposed networks for lane-detection accuracy and robustness across different scenes, compare them with diverse methods, and analyze parameter effects.
- Experimental design: The experiments assess accuracy and robustness in different scenes, compare the proposed networks with diverse lane-detection methods, and analyze parameter influence.The supplied passage states the evaluation goals but provides no numerical results.
A. Datasets
The datasets combine highway sequences from TuSimple with rural-road sequences collected by the authors, using continuous-frame sampling and augmentation for training and robustness testing.
- Original dataset: TuSimple contributes 3,626 highway image sequences, each containing 20 continuous frames collected over one second.The 20th frame is labeled, and every 13th image is additionally labeled for dataset construction.
- Original dataset: The authors add 1,148 rural-road image sequences to expand scene diversity.The supplied passage identifies this addition as the authors’ own lane dataset.
- Sampling: Training samples contain 5 continuous images and the ground truth of the last frame, with input strides of 1, 2, or 3 frames.Labels on the 13th and 20th frames support construction of the training set.
- Data augmentation: Rotation, flipping, cropping, and random illumination changes produce 19,096 training sequences containing 38,192 labeled images.The augmentation operations broaden illumination conditions represented in training.
- Testing: Testing uses 5 continuous images to predict the last frame and includes a normal TuSimple test set plus hard samples for robustness evaluation.Testset #2 contains difficult situations collected specifically for robustness evaluation.
- Ground-truth representation: Lane annotations use thin lines, although lane width varies with viewing distance and pixel-wise segmentation requires refined object boundaries.This motivates treating lane representation carefully in the semantic-segmentation setting.
- Testing conditions: Images are sampled at 256×128 resolution and evaluated under wet, cloudy, and sunny driving conditions.These conditions are used to assess low-resolution applicability and detection performance.
C. Performance and comparison
The evaluation has two parts: assessing the proposed networks visually and quantitatively, then testing robustness in difficult situations.
- The first experiment evaluates the proposed networks through visual and quantitative performance measurements.
- The second experiment verifies whether the proposed framework remains robust in difficult situations.
- Together, the experiments assess both general detection performance and behavior under challenging conditions.
1) Overall performance:
The proposed ConvLSTM networks are compared with encoder-decoder baselines and related variants through visual and quantitative evaluation. They improve lane visualization and F1 performance, while accuracy is limited as a standalone metric by severe class imbalance.
- Overall performance: The comparison includes SegNet-ConvLSTM and UNet-ConvLSTM alongside original SegNet and U-Net baselines and modified network variants.The variants include FcLSTM and 3D-convolution models.
- Visual examination: The proposed networks identify lanes without missing or excessive detection and distinguish lane boundaries from other background boundaries.
- Visual examination: The proposed lanes more closely match ground-truth positions and lengths, supporting more reliable predictions in real ADAS scenes.
- Visual examination: The visual results show fewer fuzzy regions, less breakage, and better continuity under sheltering, irregular shapes, and vehicle occlusion.
- Quantitative analysis: About 3%: F1-Measures rise relative to the original versions, indicating benefits from multiple frames and ConvLSTM sequential processing.F1 balances precision and recall, while thinner predicted lanes can slightly reduce recall.
2) Robutness:
The proposed models are evaluated on diverse and challenging driving scenes, including rural, urban, highway, shadow, occlusion, and dirt conditions. They show robust lane detection across varied environments, with UNet-ConvLSTM generally strongest and SegNet-ConvLSTM preferable under strong interference.
- Robustness testing: Testset #2 contains 728 images spanning rural, urban, and highway scenes, varied camera positions, and weather conditions.The dataset includes lanes difficult to detect even for human observers.
- Robustness testing: The proposed models detect lanes in vehicle-, shadow-, and dirt-occluded scenes across varied illumination and road conditions.They also identify lanes when entire lane markings are covered by cars or shadows.
- Comparative performance: UNet-ConvLSTM achieves the highest precision across all scenes and the highest F1 values in most scenes.The comparison is reported against other methods in Table V.
- Model selection: UNet-ConvLSTM is recommended for general applications, whereas SegNet-ConvLSTM is preferred under strong interference in tunnels or occluded environments.This recommendation follows UNet-ConvLSTM's stronger performance in most tested situations and SegNet-ConvLSTM's advantage under interference.
- Changing environments: The method is also tested on sequences where a vehicle enters and exits shadow under a bridge to assess robustness during changing environments.Figure 8 reports the robustness test for this scenario.
D. Parameter analysis
Parameter analysis examines input-sequence length and sampling stride, which together determine the temporal range of the input. More consecutive frames improve performance, while sampling stride has little effect when frame count is fixed.
- Parameter settings: The number of input frames and the sampling stride determine the total temporal range between the first and last frame.These are the two parameters analyzed for the proposed methods.
- Number of frames: More consecutive input images increase both accuracy and F1-Measure under the same sampling stride.Multiple-frame inputs also significantly outperform single-image inputs.
- Sampling stride: Sampling stride has an insignificant effect when the number of input frames is fixed, with differences visible only in the fifth decimal place.Performances remain very close across the tested strides.
- Interpretation: Using multiple frames provides richer integrated lane information for the decoder than a single-frame feature map.ConvLSTM combines feature maps from consecutive images because the encoder's small feature map cannot retain all lane information.
V. CONCLUSION
The paper proposes an encoder-decoder hybrid neural network that combines CNN feature abstraction with ConvLSTM processing of continuous frames for semantic-segmentation lane prediction. It outperforms single-image baselines, performs robustly across diverse scenes, and benefits from longer input sequences, while future work targets smoother lane fitting and unresolved performance differences under dim interference.
- Contribution: The proposed hybrid neural network combines CNN and RNN components for robust lane detection in driving scenes.Its encoder-decoder architecture takes multiple continuous frames as input and predicts the current lane by semantic segmentation.
- Architecture: CNN features are extracted from each frame, processed sequentially by ConvLSTM, and reconstructed by a CNN decoder for lane prediction.Two continuous-driving-image datasets are constructed for evaluation.
- Results: The proposed architecture significantly outperforms single-image baselines and shows higher precision, recall, and accuracy than other models.Tests also report stable lane detection and fewer false recognitions in diverse challenging scenes.
- Results: Longer input sequences improve performance, supporting the use of multiple frames rather than a single image for lane detection.The conclusion also reports advantages of ConvLSTM over FcLSTM for sequential feature learning and target-information prediction.
- Future work: Future work includes lane fitting for smoother, more integral lanes, while SegNet-ConvLSTM's advantage under dim strong interference requires further investigation.The paper identifies this model difference as an unresolved issue.