Source-linked AI summary
Spatial As Deep: Spatial CNN for Traffic Scene Understanding
Xingang Pan, Xiaohang Zhan, Jianping Shi, Ping Luo, Xiaogang Wang, Xiaoou Tang
TL;DR
CNNs have limited ability to model spatial relationships across rows and columns, which matters for long, occluded structures with weak appearance cues such as traffic lanes. The paper proposes SCNN, which propagates messages between feature-map slices, and reports improved structured outputs across traffic-scene tasks, including stronger lane-detection performance than recurrent, MRF, and deeper CNN alternatives.
Problem
CNNs do not fully capture spatial relationships across image rows and columns, limiting understanding of long, occluded structures with weak appearance cues such as lane markings.
Method
SCNN performs sequential convolution, nonlinear activation, and residual message propagation across rows or columns treated as feature-map slices, with shared directional kernels.
Results
SCNN improves traffic-scene understanding across lane detection and semantic segmentation, with a 20-layer network outperforming ReNet, MRF, and ResNet-101 in lane detection.
Takeaways & Limitations
SCNN effectively preserves continuity in long thin structures and provides beneficial diffusion for large objects in semantic segmentation.
Takeaways & Limitations
Traditional all-pixel message passing is computationally expensive for real-time autonomous driving, while MRF methods require large kernels that are hard to learn and carefully initialize.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) are usually built by stacking convolutional operations layer-by-layer. Although CNN has shown strong capability to extract semantics from raw pixels, its capacity to capture spatial relationships of pixels across rows and columns of an image is not fully explored. These relationships are important to learn semantic objects with strong shape priors but weak appearance coherences, such as traffic lanes, which are often occluded or not even painted on the road surface as shown in Fig. 1 (a). In this paper, we propose Spatial CNN (SCNN), which generalizes traditional deep layer-by-layer convolutions to slice-byslice convolutions within feature maps, thus enabling message passings between pixels across rows and columns in a layer. Such SCNN is particular suitable for long continuous shape structure or large objects, with strong spatial relationship but less appearance clues, such as traffic lanes, poles, and wall. We apply SCNN on a newly released very challenging traffic lane detection dataset and Cityscapse dataset. The results show that SCNN could learn the spatial relationship for structure output and significantly improves the performance. We show that SCNN outperforms the recurrent neural network (RNN) based ReNet and MRF+CNN (MRFNet) in the lane detection dataset by 8.7% and 4.6% respectively. Moreover, our SCNN won the 1st place on the TuSimple Benchmark Lane Detection Challenge, with an accuracy of 96.53%.
Introduction
Traffic scene understanding remains difficult for long, occluded structures such as lane markings and poles, where standard CNNs struggle to preserve spatial continuity. SCNN addresses this by propagating information across feature-map rows and columns, improving continuity in lane detection and semantic segmentation.
- Motivation: Lane detection and semantic segmentation are important traffic-scene understanding tasks, but harsh conditions make them challenging.Examples include bad weather, dim or dazzling light, and traffic objects requiring detailed localization.
- Approach: SCNN is designed for structured objects such as lanes, poles, and trucks with occlusions because spatial information reinforces their inter-layer continuity.The figure comparison reports that SCNN fixes disconnected CNN outputs for lane markings and poles.
- Motivation: CNN-based methods struggle with long, occluded structures such as lane markings and poles, while humans can infer hidden positions from visible context.The paper highlights disconnected or messy predictions for these structures in challenging scenes.
- Approach: SCNN generalizes layer-by-layer convolution by treating feature-map rows or columns as layers and sequentially propagating spatial information.Each slice receives convolution, nonlinear activation, and sum operations before passing information onward.
- Results: SCNN is reported to outperform recurrent and MRF/CRF-based methods while also improving on deeper ResNet-101 results.The introduction attributes these gains to better preservation of smoothness and continuity in structured outputs.
Spatial Convolutional Neural Network
Spatial CNN addresses difficult lane-detection conditions by propagating spatial information through feature-map slices, offering a more efficient alternative to conventional spatial models. It is designed for structured objects with long continuity and supports richer hidden-layer representations.
- Dataset and motivation: The paper introduces a large-scale challenging traffic-lane dataset because existing benchmarks were small, simple, or lacked occluded and unseen lane annotations.The new data include diverse driving scenes and lanes annotated from contextual evidence when markings are occluded or absent.
- Limitations of prior methods: Traditional MRF/CRF methods pass messages between every pixel and are computationally expensive, while their large kernels are difficult to learn.These methods are typically applied to CNN outputs, although top hidden layers may contain richer information for modeling spatial relationships.
- Spatial Convolutional Neural Network: Spatial CNN generalizes layer-by-layer convolution by splitting a C × H × W feature map into slices and sequentially propagating convolutional updates across rows or columns.The updated slice is added to the next slice, enabling spatial message passing within a feature-map layer.
- Spatial Convolutional Neural Network: SCNN uses shared convolutional weights across slices, supports downward, upward, rightward, and leftward directions, and can be inserted into richer hidden layers.Its messages are propagated as residuals through nonlinear activation, connecting the design to recurrent and residual learning.
- Efficiency and design advantages: SCNN reduces message-passing computation by using sequential local propagation while allowing information to reach all pixels through four directions.For SCNN the message count is ndirWHw, compared with niterW^2H^2 for dense MRF/CRF; typical settings use four directions and kernel width no larger than 10.
Experiment
Experiments evaluate SCNN for lane detection and semantic segmentation, including its prediction pipeline, spatial-message-passing design choices, and comparisons with baseline and alternative models. SCNN improves lane and segmentation results while preserving continuous structure and reducing message-passing runtime relative to dense CRF.
- Lane Detection: Lane predictions are extracted from probability maps by sampling the highest response every 20 rows and connecting positions with cubic splines.Lane markings with existence probability above 0.5 are processed during testing.
- Evaluation: Lane detection uses IoU thresholds of 0.3 and 0.5, with F-measure as the final evaluation index.Predictions exceeding the threshold are counted as true positives.
- Ablation Study: 8.4% and 3.2% baseline improvements are obtained with kernel width w = 9 at the two evaluated IoU thresholds.The ablation reports that larger kernel widths are beneficial and that w = 9 gives a satisfactory result.
- Ablation Study: Applying SCNN to the top hidden layer performs better than applying it to the output, which contains less information.This comparison uses the SCNN DURL model with w = 9.
- Ablation Study: Sequential propagation significantly outperforms parallel propagation, indicating that information reaches pixels from farther positions than nearby pixels alone.The sequential scheme updates each slice after receiving information from former slices.
- Comparative Results: SCNN outperforms recurrent, graphical-model, and deeper CNN alternatives on challenging lane cases, while improving segmentation for structured and large objects.SCNN preserves lane continuity better than ResNet-50 and ResNet-101; on Cityscapes, improvements are reported for categories including wall, pole, truck, bus, train, and motor.
Conclusion
The paper presents Spatial CNN for spatial-level information propagation and evaluates it on lane detection and semantic segmentation. SCNN preserves long thin structures, benefits large objects, and outperforms several baselines in lane detection.
- SCNN is a CNN-like scheme for effective information propagation at the spatial level.
- SCNN is evaluated on lane detection and semantic segmentation in traffic scene understanding.
- SCNN effectively preserves the continuity of long thin structures and benefits large objects through diffusion effects in semantic segmentation.
- A 20-layer LargeFOV network with SCNN outperforms ReNet, MRF, and ResNet-101 in lane detection.
- The presented challenging lane detection dataset is intended to advance research on autonomous driving.