Source-linked AI summary
WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving
Senthil Yogamani, Ciaran Hughes, Jonathan Horgan, Ganesh Sistu, Padraig Varley, Derek O'Dea, Michal Uricar, Stefan Milz, Martin Simon, Karl Amende, Christian Witt, Hazem Rashed, Sumanth Chennupati, Sanjaya Nayak, Saquib Mansoor, Xavier Perroton, Patrick Perez
TL;DR
WoodScape addresses the limited availability of detailed public fisheye datasets for autonomous driving. It releases a four-camera, nine-task dataset with large-scale annotations and advocates adapting models to fisheye imagery rather than relying on naive rectification. The dataset also supports baseline evaluation while exposing practical constraints from distortion and data privacy.
Problem
Public datasets for detailed evaluation of computer vision algorithms on fisheye automotive images remain limited, while standard models do not generalize easily because of nonlinear distortion.
Method
The paper constructs a multi-camera fisheye dataset from diverse locations, vehicles, and driving scenarios, providing calibration, synchronization, quality assurance, and annotations for nine tasks.
Results
WoodScape provides over 10,000 instance-level semantic images, annotations for nine tasks, and a proposed 3D box metric that improves training time by 95x.
Takeaways & Limitations
The dataset is intended to encourage native fisheye models and investigation of whether distortion should be learned or explicitly modeled.
Takeaways & Limitations
Undistortion remains constrained by field-of-view loss, transition or horizontal distortion, and privacy requirements for European data containing visible faces and license plates.
Abstract
from arXiv · showhide
Fisheye cameras are commonly employed for obtaining a large field of view in surveillance, augmented reality and in particular automotive applications. In spite of their prevalence, there are few public datasets for detailed evaluation of computer vision algorithms on fisheye images. We release the first extensive fisheye automotive dataset, WoodScape, named after Robert Wood who invented the fisheye camera in 1906. WoodScape comprises of four surround view cameras and nine tasks including segmentation, depth estimation, 3D bounding box detection and soiling detection. Semantic annotation of 40 classes at the instance level is provided for over 10,000 images and annotation for other tasks are provided for over 100,000 images. With WoodScape, we would like to encourage the community to adapt computer vision models for fisheye camera instead of using naive rectification.
1. Introduction
WoodScape addresses the scarcity of detailed fisheye automotive datasets with a four-camera, multi-task resource for 360° vehicle sensing. It provides broad annotations and is intended to support unified models for fisheye perception.
- WoodScape uses four fisheye cameras to provide comprehensive 360° sensing around a vehicle.
- The dataset complements narrow-FOV automotive datasets with fisheye imagery and a comprehensive range of annotation types.
- It contains over 10,000 images with instance-level semantic annotation and supports nine tasks across four cameras.
- WoodScape introduces soiling detection as a new task and proposes a 3D box metric that improves training time by 95x.
- The paper is organized around fisheye models, dataset design, supported tasks and baselines, and conclusions.
2. Overview of Fisheye Camera Projections
Fisheye cameras offer wide coverage but introduce nonlinear projection and severe distortion that challenge standard vision models. The paper reviews projection and correction options while motivating direct fisheye-model adaptation.
- Fisheye cameras can achieve full 360° vehicle coverage with typically four cameras, but their complex projection produces severe distortion.
- Fisheye distortion is represented by a radial mapping r(θ) from incident angle to image radius.
- WoodScape provides fourth-order polynomial intrinsic calibration parameters for each video sequence without requiring distortion correction.The model maps incident angle to image radius in pixels.
- Fisheye Camera Models: The paper compares Polynomial, Rectilinear, Stereographic, UCM, and eUCM projection models over incident angles up to 120°.
- Image Undistortion vs. Model Adaptation: Standard models struggle with fisheye images because nonlinear, spatially varying distortion breaks assumptions such as CNN translation invariance.
- Image Undistortion vs. Model Adaptation: Rectilinear correction can lose field of view and amplify peripheral resampling distortion, while cylindrical correction preserves vertical objects but introduces horizontal quadratic distortion.
3. Overview of WoodScape Dataset
WoodScape targets native fisheye, multi-camera, and multi-task autonomous-driving research with broad sensing, annotations, and carefully designed dataset splits. It combines four-camera surround view data with nine tasks, diverse acquisition settings, and sampling procedures intended to reduce redundancy.
- High-Level Goals: WoodScape encourages native fisheye processing, addressing the scarcity of public fisheye datasets and supporting accurate near-field, full-coverage sensing with four cameras.The dataset is motivated especially by low-speed manoeuvring scenarios such as parking.
- High-Level Goals: The dataset supports joint multi-camera and multi-task learning by providing simultaneous annotations across varied autonomous-driving tasks, with some task-specific exceptions.Existing datasets primarily support task-specific learning and commonly share only encoder features.
- Dataset Design: Data span the USA, Europe, and China across highway, urban, and parking scenarios, with multiple vehicle configurations, calibrations, synchronization files, and mechanical data.Sampling combines classifier-accuracy selection with image-similarity filtering, and the dataset is split 6:1:3 into training, validation, and testing.
- Dataset Acquisition: WoodScape contains four 1MPx RGB fisheye cameras with 190° horizontal FOV, supported by LiDAR, GNSS/IMU, GNSS positioning, and vehicle-bus odometry.The sensor suite is designed for surround-view acquisition and multimodal ground truth.
- Dataset Acquisition: WoodScape provides nine autonomous-driving tasks, including semantic segmentation, depth estimation, 2D and 3D detection, visual odometry, visual SLAM, motion segmentation, soiling detection, and driving controls.Soiled-lens detection is identified as a novel task, and synthetic data are also provided for additional investigations.
- Dataset Acquisition: The dataset includes 40 recognition classes, denser SLAM-derived point-cloud ground truth, and semantic-class distributions displayed with a 300-pixel minimum instance size.Figure 6 merges some classes for display, while the underlying labels distinguish categories such as bicycle and motorcycle.
- Dataset Design: The minimal consistent subset formulation seeks the smallest training subset matching the full set's selected accuracy, reducing non-informative samples and annotation effort.The formulation requires ground-truth labels to compute accuracy and motivates careful dataset sampling.
- Dataset Design: More than one third of the dataset is GDPR-sensitive because European recordings contain visible pedestrian faces and license plates, complicating public release.The paper considers manual blurring, GAN-based re-targeting, and stringent data-handling agreements as privacy options.
4. Tasks, Metrics and Baseline experiments
WoodScape supplies task annotations and baseline experiments spanning fisheye segmentation and object detection. The reported comparison shows that a fisheye-trained detector substantially outperforms a pretrained network tested on fisheye imagery.
- Metrics and Baseline Experiments: WoodScape reports metrics and baselines for each task, with each task's test set comprising 30% of its annotated samples.The paper summarizes these experiments in Table 2 and provides code and supplementary video results.
- Semantic Segmentation: Its semantic segmentation labels cover 40 object categories, exceeding the 30 categories provided by Cityscapes.Prior work used artificially warped Cityscapes images because fisheye datasets were unavailable, but warping cannot enlarge the original captured FOV.
- 2D Bounding Box Detection: The 2D detection dataset derives seven object categories from instance-segmentation boxes, using Faster R-CNN with a ResNet101 encoder and ImageNet initialization.Training combines KITTI and WoodScape detection data through fine-tuning.
- 2D Bounding Box Detection: 12 mAP is achieved by a pretrained person detector on fisheye images, compared with 45 mAP for a model trained on WoodScape.The authors attribute the difficulty to differing object orientations between peripheral and central image regions and identify explicit fisheye geometry as necessary.
4.3. Camera Soiling Detection
WoodScape introduces soiling detection for surround-view cameras exposed to environmental contamination and evaluates a baseline classifier for opaque and transparent soiling.
- Motivation: Soiling detection addresses mud, dust, water, and ice contamination that can degrade surround-view visual perception.The dataset treats this as a novel task motivated by cameras’ direct exposure to adverse environmental conditions.
- Task formulation: The classifier jointly predicts a binary soiling indicator array and a categorical opaque-or-transparent label.Opaque soiling typically arises from mud and dust, while transparent soiling typically arises from water and ice.
- Annotation and evaluation: 5k images receive polygon annotations separating soiled and unsoiled regions for classification or segmentation modeling.Performance is evaluated with an example-based accuracy measure based on the average Jaccard index.
- Baseline result: 84.5% precision was achieved for multilabel classification using a ResNet10 encoder and three-layer decoder baseline.The baseline network is described as small and is evaluated for the multilabel classification task.
4.4. 3D Bounding Box Detection
WoodScape provides 3D box annotations and introduces SRTs, a metric that separately accounts for scaling, rotation, and translation while addressing drawbacks of 3D IoU.
- Annotation: 3D box annotations cover 10k frames and three classes: pedestrian, vehicle, and cyclist.The dataset uses these annotations for 3D bounding-box detection.
- Metric: SRTs compares 3D boxes through independent rigid transformations for translation, rotation, and scaling.The metric is designed around transforming non-overlapping boxes with respect to one another.
- Metric: SRTs combines scaling, translation, and rotation components with weights that sum to one and includes a penalty term.The full metric is expressed as Ssrt = pt · (α Ss + β St + γ Sr), with α + β + γ = 1.
- Baseline experiments: Baseline experiments train Complex-YOLO for cars using either 3D-IoU or SRTs optimization and compare orientation and runtime.The experiments use a fixed 50:50 training-validation split and report results on moderate difficulty.
4.5. Monocular Depth Estimation
WoodScape supports monocular depth estimation with LiDAR-derived supervision across all four cameras and denser SLAM-based ground truth for static scenes.
- Dataset: More than 100k images from all four cameras are provided for monocular depth estimation, totaling 400k camera images.LiDAR supplies the ground truth, with colored examples encoding distance from blue to red.
- Ground truth: The dataset also provides denser point-cloud ground truth generated from SLAM’d static scenes.This supplements the sparse depth obtained from LiDAR.
- Projection: Ground-truth 3D points are projected onto camera images using the proposed camera model.This links the point-cloud supervision to the fisheye image domain.
4.6. Motion Segmentation
WoodScape annotates motion for surround-view driving scenes, emphasizing motion cues from ego-motion and dynamic agents as a complement to appearance-based recognition.
- Motivation: Vehicle ego-motion and nearby dynamic objects provide strong motion cues in automotive perception.The paper highlights motion as useful because rare objects may be difficult to recognize from appearance alone.
- Annotation: Motion masks are provided for vehicles, pedestrians, and cyclists across over 10k images.Motion segmentation is formulated as binary segmentation and evaluated with IoU.
- Multi-stream modeling: Previous and next images are included to support multi-stream models such as MODNet.The temporal context complements the motion-mask annotations.
4.7. Visual Odometry/SLAM
WoodScape provides centimetre-level GNSS and IMU annotations for visual odometry, including six-degree-of-freedom ground truth up to scale. A fisheye-adapted LSD-SLAM model is used as the baseline, with accuracies reported in Table 2.
- GNSS and IMU provide centimetre-level ground-truth annotations for visual odometry.
- The ground truth contains all six degrees of freedom up to scale.
- The visual-odometry metric measures the percentage of frames within translation- and rotation-error tolerance levels.
- Fisheye-adapted LSD-SLAM is used as the baseline model, with accuracies reported in Table 2.
4.8. Synthetic Data Domain Transfer
WoodScape plans a synthetic surround-view fisheye dataset to support corner-case testing, address recording restrictions, and provide dense annotations for domain transfer to real data. The paper targets semantic segmentation and depth estimation transfer.
- Synthetic data supports rigorous corner-case testing across diverse autonomous-driving scenarios.
- Synthetic data avoids restrictions such as recording videos of children and enables dense depth and optical-flow annotation.
- WoodScape plans a synthetic fisheye surround-view dataset for exploring transfer from synthetic to real domains.
- The planned domain-transfer study focuses on semantic segmentation and depth estimation.
- End-to-end steering models for fisheye cameras are described as immature for deployment but potentially useful for redundancy or auxiliary-task accuracy improvements.
5. Conclusions
The paper releases a nine-task, multi-camera fisheye dataset for autonomous driving and aims to encourage native fisheye modeling rather than naive undistortion. It also identifies future work on fisheye geometry and unified multi-task learning.
- WoodScape provides an extensive multi-camera fisheye dataset for autonomous driving with annotations for nine tasks.
- The release aims to encourage native fisheye models instead of undistorting images before applying standard models.
- The dataset can support studying whether deep networks learn spatial distortion or require explicit geometric modeling.
- Future work includes comparing undistortion methods, incorporating fisheye geometry into CNNs, and designing a unified multi-task model.