Source-linked AI summary
Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras
Liuyuan Deng, Ming Yang, Hao Li, Tianyi Li, Bing Hu, Chunxiang Wang
TL;DR
The paper addresses raw-fisheye road-scene segmentation amid severe distortion and scarce annotated surround-view data. It combines RDC, conventional-to-fisheye zoom augmentation, and multi-task training, reporting top accuracy on all 18 evaluated SVScape classes against fine-tuned baselines.
Problem
Raw fisheye images have strong distortion, and large-scale annotated semantic-segmentation datasets for surround-view cameras are scarce.
Source: §I · p. 1Method
The paper proposes RDC for geometric distortions and trains a segmentation model with real and zoom-transformed images in a multi-task architecture.
Source: §I · p. 2Results
The proposed approach achieves the top accuracy on all 18 evaluated SVScape classes compared with fine-tuned FCN-VGG16, ENet, and ERFNet.
Source: C. Semantic Segmentation Using… · p. 10Takeaways & Limitations
The authors report that combining a small number of real surround-view images with many transformed images is effective when training data is limited.
Source: C. Semantic Segmentation Using… · p. 10
Abstract
from arXiv · showhide
Understanding the surrounding environment of the vehicle is still one of the challenges for autonomous driving. This paper addresses 360-degree road scene semantic segmentation using surround view cameras, which are widely equipped in existing production cars. First, in order to address large distortion problem in the fisheye images, Restricted Deformable Convolution (RDC) is proposed for semantic segmentation, which can effectively model geometric transformations by learning the shapes of convolutional filters conditioned on the input feature map. Second, in order to obtain a large-scale training set of surround view images, a novel method called zoom augmentation is proposed to transform conventional images to fisheye images. Finally, an RDC based semantic segmentation model is built; the model is trained for real-world surround view images through a multi-task learning architecture by combining real-world images with transformed images. Experiments demonstrate the effectiveness of the RDC to handle images with large distortions, and that the proposed approach shows a good performance using surround view cameras with the help of the transformed images.
I. INTRODUCTION
The paper targets 360-degree semantic segmentation on raw surround-view fisheye images, where distortion challenges CNNs and large annotated training sets are scarce. It proposes restricted deformable convolution, zoom augmentation, and multi-task training that combines real and transformed images.
I. INTRODUCTION
- Fisheye distortion varies across spatial regions, while regular CNNs have limited ability to model large, unknown geometric transformations; undistortion also blurs boundaries and loses information.The paper therefore explores segmentation directly on raw surround-view images.Source: §I · pp. 1–2
- Large-scale pixel-level annotated datasets for surround-view cameras are scarce, and existing forward-looking data may not represent the differing image compositions or regions of use.The paper notes that sideways-looking cameras capture different views from forward-looking cameras, and Cityscapes was collected in European cities.Source: §I · p. 2
- The paper extends its earlier conference publication with further improvements and experiments on road-scene segmentation using surround-view cameras.Source: §I · p. 2
- Restricted Deformable Convolution (RDC) restricts deformable convolution for pixel-wise prediction tasks to address their spatial-correspondence problem.The paper proposes RDC as a module for handling images with large distortions.Source: §I · p. 2
- Zoom augmentation transforms conventional images into fisheye images, and conventional datasets are used to augment surround-view images.A CUDA-implemented layer supports online training.Source: §I · p. 2
- A multi-task architecture trains an end-to-end model for real-world surround-view images by combining a small number of real images with many transformed images.The approach uses AdaBN to bridge their distributional gap and Hybrid Loss Weightings with auxiliary losses to improve generalization.Source: §I · p. 2
II. RELATED WORK
Related work covers CNN-based segmentation, methods for receptive-field and geometric modeling, dataset scarcity, synthetic-data strategies, and efficient networks for driving applications.
II. RELATED WORK
- CNNs and fully convolutional networks advanced semantic segmentation beyond earlier handcrafted-feature methods, which used Random Decision Forests or Boosting and CRFs for uncertainty and context.Source: §II · p. 2
- Dilated convolution enlarges receptive fields without reducing spatial resolution, while deformable convolution learns filter shapes conditioned on input features; Fig. 3 contrasts their sampling with RDC's fixed center.The paper notes that deformable convolution does not address the spatial-correspondence problem in dense prediction [16].Source: §II · pp. 2–3, Figure 3 · p. 3
- Road-scene datasets such as Cityscapes provide annotated conventional-camera imagery, but collection and pixel-level annotation are costly and none of the cited datasets uses surround-view cameras.Source: §II · p. 3
- Weakly supervised and domain-adaptation approaches address annotation burdens, while this paper combines a small number of annotated local images with extensive synthetic data in multi-task learning.Source: §II · p. 3
- Very large networks can achieve high accuracy at high computational cost, motivating efficient alternatives; ERFNet is selected as the paper's efficient semantic-segmentation baseline.ERFNet applies factorized convolutions and can be trained from scratch.Source: §II · p. 3
III. RESTRICTED DEFORMABLE CONVOLUTION
Restricted Deformable Convolution (RDC) keeps the filter center fixed while learning offsets for its outer sampling locations, aiming to model geometric transformations without sacrificing spatial correspondence. Its formulation uses input-conditioned offsets and differentiable bilinear sampling, allowing end-to-end training.
III. RESTRICTED DEFORMABLE CONVOLUTION
- RDC fixes the filter center and learns offsets for outer locations, restricting deformable convolution to help preserve spatial structure in pixel-wise prediction.The method rationale is that deforming lower- or middle-layer filters can disrupt spatial structures and make input-to-label correspondence difficult.Source: §III · p. 3, A. Formulation · p. 4
A. Formulation
- RDC adds learned two-dimensional offsets to regular-grid sampling locations, while the center remains at its original position.The regular grid provides the initial filter locations; the central offset is set to (0, 0).Source: A. Formulation · p. 4
- Because learned outer sampling positions can be fractional, RDC uses bilinear interpolation over the four nearest input-feature positions.The paper describes this interpolation operation as differentiable.Source: A. Formulation · p. 4
- At each sliding position, RDC learns a filter shape from the input features, using 2(N −1) offset parameters and no offset parameter for the center.Offsets are predicted by a convolutional layer from the same input feature map, with offset-field resolution matching the output feature map.Source: §III · p. 3, A. Formulation · p. 4
- RDC is differentiable and supports standard backpropagation, propagating gradients to filter weights, offsets, and the input feature map.The gradients reach the offsets and input feature map through the bilinear operation.Source: A. Formulation · p. 4
B. Factorized Restricted Deformable Convolution
Factorized Restricted Deformable Convolution (FRDC) applies the restricted deformable idea to consecutive one-dimensional kernels. It reduces the number of learned offset parameters relative to RDC, at the cost of less flexibility.
B. Factorized Restricted Deformable Convolution
- The illustration contrasts regular convolution, factorized convolution, and factorized restricted deformable convolution, omitting nonlinearities in the latter two examples.Its sampling-point and offset symbols follow the definitions in Fig. 3.Source: Figure 5 · p. 5
- FRDC decomposes a two-dimensional restricted deformable filter into vertical and horizontal one-dimensional kernels, with direction-specific offsets.Each outer location learns one parameter controlling its dilation factor.Source: B. Factorized Restricted Deformable… · pp. 4–5
- FRDC uses N −1 additional parameters, half as many as RDC, and is therefore less flexible.The factorized construction is presented as a way to reduce memory and computational cost.Source: B. Factorized Restricted Deformable… · pp. 4–5
IV. THE GENERATION OF FISHEYE IMAGE DATASET FOR SEMANTIC SEGMENTATION
The paper proposes transforming conventional images into fisheye images to enrich the limited supply of large-scale fisheye semantic-segmentation datasets.
IV. THE GENERATION OF FISHEYE IMAGE DATASET FOR SEMANTIC SEGMENTATION
- The method addresses the scarcity of large-scale fisheye semantic-segmentation datasets by transforming conventional images into fisheye images.The paper describes this transformation as a way to enrich available datasets.Source: §IV · p. 5
- The transformation builds a mapping from the fisheye image plane to the conventional image plane.This mapping is used to remap the conventional scene into a fisheye image.Source: §IV · p. 5
- The proposed image transformation is intended to enrich datasets for fisheye image semantic segmentation.The passage identifies few large-scale datasets for this task as the motivation.Source: §IV · p. 5
A. Mapping Conventional Image to Fisheye Image
The mapping section relates conventional pinhole-camera projection to fisheye equidistance projection. Under specified focal-length and viewing-angle settings, it maps fisheye image points to conventional image points and applies the same mapping to images and annotations.
A. Mapping Conventional Image to Fisheye Image
- The section describes conventional images with pinhole perspective projection and fisheye images with equidistance projection.It presents the mapping in terms of these two projection models.Source: A. Mapping Conventional Image to… · p. 5
- The mapping from fisheye image points to conventional image points assumes identical focal lengths and a maximum viewing angle of 180°.The mapping relationship is determined by focal length, and the chosen base focal length approximately covers a hemispherical field.Source: A. Mapping Conventional Image to… · p. 5
- The mapping relates image points through their distances from the principal points in the conventional and fisheye images.The passage defines these distances using the respective image-point and principal-point coordinates.Source: A. Mapping Conventional Image to… · p. 5
- The same mapping function transforms each existing segmentation-dataset image and its corresponding annotation to generate fisheye data.The passage describes the mapping as being determined by the focal length.Source: A. Mapping Conventional Image to… · p. 5
B. Zoom Augmentation for Fisheye Image
Zoom augmentation treats focal-length changes as a way to warp conventional images into fisheye-style images, producing examples with different distortion levels. It gives existing conventional segmentation datasets a route to augment fisheye data.
B. Zoom Augmentation for Fisheye Image
- Zoom augmentation extends conventional segmentation datasets with derived fisheye-style training images rather than merely resizing them.Conventional scaling is reframed as changing the camera focal length.Source: B. Zoom Augmentation for Fisheye Image · p. 5
- Zoom augmentation warps conventional images into fisheye-style images by changing the simulated camera focal length.The operation can use a fixed or randomly changing focal length.Source: B. Zoom Augmentation for Fisheye Image · p. 5
- Smaller focal lengths produce greater distortion, so varying focal length yields fisheye images with different distortion levels.The examples show transformations with focal lengths ranging from 200 to 800.Source: B. Zoom Augmentation for Fisheye Image · p. 5, Figure 6 · p. 6
V. TRAINING STRATEGY
The training strategy combines real-world surround-view images with transformed images in a multi-task model. It shares model weights while retaining domain-specific batch-normalization statistics and balances task losses with separate main- and auxiliary-branch weightings.
V. TRAINING STRATEGY
- A multi-task architecture trains on real-world and transformed images together, addressing different label spaces and domain shift while reducing the overfitting risk of fine-tuning on limited real-world data.The model uses shared weights for source and target domains.Source: §V · p. 6
A. Sharing Weights with Private Batch Normalization Statistics
- AdaBN shares model weights across domains but keeps separate batch-normalization statistics for each domain.The passage identifies batch-normalization statistics as closely related to domain knowledge.Source: A. Sharing Weights with Private Batch… · p. 6
B. Hybrid Loss Weightings
- Hybrid Loss Weightings (HLW) allows different task weightings for main and auxiliary losses, with a larger auxiliary weighting providing stronger regularization.A smaller main-loss weighting can balance the stronger auxiliary contribution.Source: B. Hybrid Loss Weightings · pp. 6–7
VI. EXPERIMENTS
- Experiments measure semantic-segmentation performance with mean Intersection-Over-Union (mIoU).The experiments use two NVIDIA GTX 1080Ti GPUs and MXNet.Source: §VI · p. 7
A. Datasets for Surround View Image Semantic Segmentation
The experiments use Cityscapes and SYNTHIA-Seqs as sources for zoom-augmented surround-view data, alongside a separately captured and annotated real-world surround-view dataset.
A. Datasets for Surround View Image Semantic Segmentation
- Cityscapes contributes 3,475 images and SYNTHIA-Seqs contributes 34,696 images for zoom-augmented surround-view data.Both real and synthetic source datasets are transformed; their resulting image resolutions are 576 × 640 and 512 × 640, respectively.Source: A. Datasets for Surround View Image… · p. 7
- Zoom augmentation examples show Cityscapes front-view scenes and SYNTHIA-Seqs scenes from different perspectives after random focal-length changes.The zoom augmentation layer applies these transformations online using a CUDA implementation.Source: Figure 8 · p. 7, A. Datasets for Surround View Image… · p. 7
- The SVScape dataset contains 600 annotated surround-view images captured by four fisheye cameras mounted around a moving vehicle.The images are split into 350 for training, 100 for validation, and 150 for testing.Source: A. Datasets for Surround View Image… · p. 7
B. Evaluation for the Restricted Deformable Convolution
The evaluation tests restricted deformable convolution across VGG-, residual block-, and DeepLab-based models on conventional and fisheye-style datasets. RDC performs better than DC in several settings, but its gains depend on architecture and the number of modified layers.
B. Evaluation for the Restricted Deformable Convolution
- The evaluation replaces selected convolutional layers or blocks with DC, RDC, or FRDC variants across VGG, ERFNet, ENet, and DeepLab architectures.For FCN-VGG16, the final λ stages are replaced; in DeepLab, λ denotes the final 3 × 3 convolutional layers replaced.Source: B. Evaluation for the Restricted… · pp. 7–9
- FCN-VGG16 accuracy drops sharply and can collapse as more DC or RDC layers are added, although RDC degrades more slowly than DC.RDC's fixed central sampling point alleviates the spatial-correspondence problem but does not prevent poor performance in this architecture.Source: Figure 10 · p. 8, B. Evaluation for the Restricted… · p. 8
- DC, RDC, and FRDC models outperform their base models on Cityscapes residual block-based tests, with ERFNet-RDC-4 and ENet-RDC-4 achieving the best accuracies in their respective model families.Performance first improves and then saturates as λ increases.Source: Figure 11 · p. 8, B. Evaluation for the Restricted… · p. 9
- ERFNet-RDC-8 achieved the best score on Fisheye-Cityscapes, while ENet-FRDC-8* achieved the best among ENet variants.The results also show that DC, RDC, and FRDC models almost outperform their base models on this dataset.Source: Figure 12 · p. 8, B. Evaluation for the Restricted… · p. 9
- ERFNet-OPP produces worse accuracies than ERFNet on Cityscapes and Fisheye-Cityscapes, while residual identity mappings are reported as important for preventing modified models from collapsing.The authors conjecture that OPP may increase learning difficulty or affect gradient propagation from decoder to encoder.Source: B. Evaluation for the Restricted… · pp. 8–9
- DeepLab-RDC is consistently better than DeepLab-DC when λ is at least 3, and DeepLab-RDC achieves its best performance at λ = 7.The supplied table covers DC or RDC in ResNet-101's last two stages and dilation-ratio variants on Cityscapes.Source: B. Evaluation for the Restricted… · p. 9, Table I · p. 9
C. Semantic Segmentation Using Surround View Cameras
The proposed model combines real and transformed images in multi-task training for semantic segmentation on surround-view cameras. On SVScape, it leads the reported per-class accuracy comparison across all 18 evaluated classes.
C. Semantic Segmentation Using Surround View Cameras
- Multi-task training combines SVScape real images with transformed Cityscapes and SYNTHIA-Seqs images, using shared weights except for the classifiers.The main and auxiliary loss weightings balance the contribution of real and transformed samples.Source: C. Semantic Segmentation Using… · p. 9
- A smaller main-loss weight with a larger auxiliary-loss weight significantly improved performance, while decreasing equal weights degraded it.The authors attribute degradation under equal-weight reduction to weaker auxiliary-task regularization.Source: C. Semantic Segmentation Using… · p. 10
- AdaBN improved performance by 7.5%; random-focal-length zoom augmentation improved accuracy by 0.5% over fixed focal length, while disabling zoom reduced accuracy by 0.9%.The authors connect variable focal length to training with different degrees of distortion.Source: C. Semantic Segmentation Using… · p. 10, Table III · p. 10
- 74.2% was reached after applying HLW with the reported weighting setting.The preceding setting is partially specified in the supplied passage.Source: C. Semantic Segmentation Using… · p. 10
- The proposed approach achieves the top accuracy on all 18 evaluated SVScape classes.The authors report larger improvements for challenging classes than for general classes.Source: C. Semantic Segmentation Using… · p. 10, Table V · p. 11
- ERFNet-RDC-8 took 2 ms more than ERFNet for a forward pass, excluding CPU preprocessing and GPU-to-CPU transfer time.The reported measurements include CPU-to-GPU transfer and GPU processing time.Source: C. Semantic Segmentation Using… · pp. 10–11, Table IV · p. 10
VII. CONCLUSION
The paper concludes that RDC-based semantic segmentation can handle fisheye images and has been implemented for surround-view road scenes. It identifies combining RDC with deformable convolution and further domain adaptation as future work.
VII. CONCLUSION
- RDC models geometric transformations and is less prone to saturation, while deformable convolution performs better when limited to the last few convolutional layers.The authors propose combining RDC and deformable convolution in one network to further enhance transformation modeling.Source: §VII · p. 11
- Further improving performance on real surround-view images remains future work, including through weakly supervised or other domain adaptation methods.The paper also proposes combining RDC and deformable convolution in one network.Source: §VII · p. 11