Source-linked AI summary
RESA: Recurrent Feature-Shift Aggregator for Lane Detection
Tu Zheng, Hao Fang, Yi Zhang, Wenjian Tang, Zheng Yang, Haifeng Liu, Deng Cai
TL;DR
Lane detection remains challenging under occlusion, ambiguous visual cues, and sparse lane annotations, which hinder ordinary CNN feature extraction. The paper introduces RESA for recurrent horizontal and vertical feature aggregation and BUSD for coarse-to-fine up-sampling, achieving state-of-the-art performance on CULane and Tusimple.
Problem
Lane detection is difficult in complex scenes because occlusion, ambiguous lanes, and sparse lane annotations make subtle lane features hard for ordinary CNNs to extract.
Method
RESA recurrently shifts sliced feature maps across rows and columns to aggregate spatial information, while BUSD combines coarse-grained and fine-detailed up-sampling branches.
Results
The method achieves state-of-the-art performance on CULane and Tusimple, including 75.3 F1-measure on CULane and 96.8% accuracy on Tusimple.
Takeaways & Limitations
RESA and BUSD provide a lane-detection system that aggregates global spatial information and meticulously restores pixel-wise predictions.
Takeaways & Limitations
Sequential spatial information passing is time-consuming and can lose information during long-distance propagation.
Abstract
from arXiv · showhide
Lane detection is one of the most important tasks in self-driving. Due to various complex scenarios (e.g., severe occlusion, ambiguous lanes, etc.) and the sparse supervisory signals inherent in lane annotations, lane detection task is still challenging. Thus, it is difficult for the ordinary convolutional neural network (CNN) to train in general scenes to catch subtle lane feature from the raw image. In this paper, we present a novel module named REcurrent Feature-Shift Aggregator (RESA) to enrich lane feature after preliminary feature extraction with an ordinary CNN. RESA takes advantage of strong shape priors of lanes and captures spatial relationships of pixels across rows and columns. It shifts sliced feature map recurrently in vertical and horizontal directions and enables each pixel to gather global information. RESA can conjecture lanes accurately in challenging scenarios with weak appearance clues by aggregating sliced feature map. Moreover, we propose a Bilateral Up-Sampling Decoder that combines coarse-grained and fine-detailed features in the up-sampling stage. It can recover the low-resolution feature map into pixel-wise prediction meticulously. Our method achieves state-of-the-art results on two popular lane detection benchmarks (CULane and Tusimple). Code has been made available at: https://github.com/ZJULearning/resa.
Introduction
Lane detection is challenging because lanes are thin, sparsely annotated, and often occluded, limiting ordinary CNN feature extraction. The paper introduces RESA for spatial feature aggregation and BUSD for detailed up-sampling, achieving state-of-the-art results on CULane and Tusimple.
- Lane detection supports autonomous driving and ADAS but is difficult under occlusion, adverse weather, ambiguous pavement, and lanes’ long, thin structure.
- Pixel-wise CNN segmentation often misses subtle lane features because annotated lane pixels are sparse relative to background and shape priors are underused.
- RESA recurrently shifts sliced feature maps horizontally and vertically, allowing pixels to aggregate spatial information through parallel updates and multiple strides.
- BUSD combines coarse-grained and fine-detailed branches to recover low-resolution feature maps into meticulous pixel-wise predictions.
- 75.3 F1-measure on CULane and 96.8% accuracy on Tusimple establish the method’s reported state-of-the-art performance.
Related Work
Lane detection research moved from hand-crafted traditional features toward deep learning and spatial-context modeling. Prior work includes semantic and instance segmentation, spatial RNNs, Graph LSTM, and slice-wise convolutions for information propagation.
- Traditional lane detectors use hand-crafted or specialized features but have complex selection processes and poor scalability across road-scene variations.
- Deep learning methods learn lane features end to end and include CNN-based detection and instance-segmentation formulations.
- Spatial-context methods propagate information across images using spatial RNNs, Graph LSTM, or SCNN’s slice-by-slice convolutions.
Method
The model combines an encoder, RESA aggregator, and bilateral decoder to propagate spatial information globally and recover detailed pixel-wise lane predictions.
- Overall Architecture: The architecture uses an encoder, RESA, and decoder; the encoder reduces the input to 1/8 resolution before aggregation and up-sampling.The decoder predicts lane existence and pixel-wise probability distributions.
- RESA: RESA recurrently shifts sliced feature maps in four directions so each location can aggregate information across the feature map.Vertical and horizontal propagation use shared convolution weights for slices with the same stride.
- RESA: RESA uses dynamically increasing strides, allowing parallel updates to gather whole-feature-map information in floor(log2L) iterations.The shift stride is controlled by the iteration number, with strides such as 1, 2, 4, and 8.
- RESA: RESA is designed as a concise feature-enhancement module that preserves input shape and can be inserted after common CNN backbones.The paper states that its computational time can be almost ignored.
- Bilateral Up-Sampling Decoder: Bilateral Up-Sampling Decoder combines a coarse branch for rapid rough up-sampling with a fine branch that restores subtle details.The coarse branch uses bilinear interpolation, while the fine-detailed branch uses transpose convolution and non-bottleneck blocks.
Dataset
The evaluation uses CULane’s varied urban and highway scenarios and TuSimple’s stable-lighting highway videos, with distinct dataset sizes and training setups.
- Datasets: CULane contains 55 hours of videos across nine scenarios, including urban, highway, curves, nighttime, dazzle, crowd, no-line, and arrow conditions.Each lane is treated as a 30-pixel-wide line for evaluation.
- Experimental Setup: Training uses batch sizes of 8 for CULane and 4 for TuSimple, with 12 and 50 epochs respectively.Experiments use four NVIDIA 2080Ti GPUs and PyTorch 1.1.
- Experimental Setup: The experiments use ResNet and VGG as backbone networks.For ResNet, an extra 1×1 convolution reduces the output channels to 128; VGG follows the SCNN modification.
Main Results
RESA outperforms the compared lane-detection baselines on CULane and Tusimple, while producing smoother, more continuous, and more robust predictions under occlusion.
- RESA outperforms all listed baselines on CULane and achieves a state-of-the-art result.The comparison uses RESA-50 with a ResNet50 backbone.
- 96.82% accuracy is achieved by RESA-34 on the Tusimple benchmark, exceeding the state-of-the-art comparison.The authors also report that RESA has substantially lower false positives than other algorithms.
- RESA produces smoother, more continuous, and less noisy lane predictions than segmentation methods and SCNN in severely occluded CULane examples.The qualitative comparison attributes this robustness to aggregating spatial relationships across rows and columns with different strides.
- RESA captures structural lane priors more strongly than traditional segmentation modules and SCNN in the qualitative CULane comparison.
Ablation Study
Ablations show that both BUSD and RESA improve lane detection, while RESA benefits from more directions and iterations at increased computational cost.
- Effect of Each Component.: Both the Bilateral Up-Sampling Decoder and RESA strongly improve lane-detection performance over the baseline.The baseline uses a ResNet-34 backbone followed by direct 8x bilinear upsampling; the modules are added step by step.
- RESA Design Choices: Adding more directions to RESA produces higher F1-measure, while maximum aggregation performs similarly to addition aggregation.
- Iteration In RESA.: Performance improves as RESA iterations increase, but additional iterations increase computational time; the authors select iteration = 4.The choice balances performance and computational resources.
- Compare RESA with SCNN: RESA outperforms SCNN and brings significant improvement in comparisons using VGG16 and ResNet34 backbones.
- Runtime Comparison: The runtime comparison evaluates LSTM, SCNN, and RESA across convolution kernel widths 7, 9, and 11, with RESA using four iterations.
Conclusion
The paper introduces RESA and BUSD for lane detection, combining recurrent spatial aggregation with coarse-to-fine upsampling, and reports state-of-the-art benchmark performance.
- RESA recurrently shifts sliced feature maps vertically and horizontally so each pixel can gather global spatial information.The module uses lane shape priors and can be plugged into other networks.
- BUSD combines coarse-grained and fine-detailed features during upsampling to recover low-resolution feature maps into meticulous pixel-wise predictions.
- The method achieves state-of-the-art performance on the Tusimple and CULane lane-detection benchmarks.