PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniDet: Surround View Cameras based Multi-task Visual Perception Network for Autonomous Driving

Varun Ravi Kumar, Senthil Yogamani, Hazem Rashed, Ganesh Sistu, Christian Witt, Isabelle Leang, Stefan Milz, Patrick Mäder

arXiv:2102.07448v3cs.CVcs.RO

TL;DR

Fisheye perception for near-field driving is challenging, and automotive systems need efficient designs across multiple tasks. OmniDet presents a real-time multitask network with camera-geometry adaptation and task-specific representations. Its multitask models outperform single-task baselines, with state-of-the-art KITTI depth and pose results.

  • Problem

    Automotive systems need efficient perception across increasing numbers of cameras and tasks, while fisheye distortion complicates near-field perception.

    Source: §I · p. 1

  • Method

    OmniDet uses a real-time multitask model with a shared encoder, synergized decoders, camera-geometry adaptation, and polygon-based object detection for fisheye images.

    Source: §I · p. 1, B. Generalized Object Detection · p. 2

  • Results

    The six-task WoodScape and five-task KITTI and Cityscapes models outperform single-task baselines, with state-of-the-art monocular depth and pose results on KITTI.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The model demonstrates multitask perception on surround-view fisheye images and competitive five-task performance on KITTI and Cityscapes.

    Source: §V · p. 7

Abstract

from arXiv · show

Surround View fisheye cameras are commonly deployed in automated driving for 360\deg{} near-field sensing around the vehicle. This work presents a multi-task visual perception network on unrectified fisheye images to enable the vehicle to sense its surrounding environment. It consists of six primary tasks necessary for an autonomous driving system: depth estimation, visual odometry, semantic segmentation, motion segmentation, object detection, and lens soiling detection. We demonstrate that the jointly trained model performs better than the respective single task versions. Our multi-task model has a shared encoder providing a significant computational advantage and has synergized decoders where tasks support each other. We propose a novel camera geometry based adaptation mechanism to encode the fisheye distortion model both at training and inference. This was crucial to enable training on the WoodScape dataset, comprised of data from different parts of the world collected by 12 different cameras mounted on three different cars with different intrinsics and viewpoints. Given that bounding boxes is not a good representation for distorted fisheye images, we also extend object detection to use a polygon with non-uniformly sampled vertices. We additionally evaluate our model on standard automotive datasets, namely KITTI and Cityscapes. We obtain the state-of-the-art results on KITTI for depth estimation and pose estimation tasks and competitive performance on the other tasks. We perform extensive ablation studies on various architecture choices and task weighting methodologies. A short video at https://youtu.be/xbSjZ5OfPes provides qualitative results.

I. INTRODUCTION

Fisheye perception is challenging because strong radial distortion and spatially varying appearance complicate near-field sensing, while automotive systems need efficient coverage of multiple perception tasks. OmniDet addresses this with a real-time multitask model and camera-aware adaptations, reporting improvements over single-task baselines and state-of-the-art KITTI depth and pose results.

I. INTRODUCTION

  • Fisheye radial distortion cannot be corrected without reduced field of view and peripheral resampling artifacts, and it increases appearance variation, especially for nearby objects.The passage describes fisheye perception as challenging and relatively less explored despite its prevalence.Source: §I · p. 1
  • Automotive systems need efficient designs as cameras and perception tasks increase; multitask learning shares computation and can regularize features across tasks,.Source: §I · p. 1
  • OmniDet presents the first real-time six-task perception model for surround-view fisheye cameras.Source: §I · p. 1
  • A camera geometry tensor encodes radial distortion to adapt the CNN to the 12 camera models in WoodScape.Source: §I · p. 1
  • The authors propose VarNorm task weighting alongside synergized decoders and a shared encoder.Source: §I · p. 1
  • The six-task WoodScape and five-task KITTI and Cityscapes models perform better than their single-task baselines.The tasks share an encoder and use synergized decoders.Source: §I · pp. 1–2
  • The model achieves state-of-the-art monocular depth and pose estimation results on KITTI.Source: §I · p. 2

II. PERCEPTION TASKS AND LOSSES IN OUR MTL

The system combines semantic, geometric, and lens-soiling tasks for near-field driving use cases. Its self-supervised distance and pose framework combines view-synthesis reconstruction, distance regularization, cross-sequence consistency, and feature-metric losses.

II. PERCEPTION TASKS AND LOSSES IN OUR MTL

  • The system covers object detection, semantic segmentation, motion segmentation, depth estimation, visual odometry, and lens-soiling detection for near-field use cases such as parking or traffic-jam assist.Source: §II · p. 2

A. Self-Supervised Distance and Pose Estimation Networks

  • Distance and pose estimation use self-supervised monocular structure-from-motion with polynomial-projection view synthesis, distance smoothness, cross-sequence consistency, and scale recovery.The framework builds on FisheyeDistanceNet.Source: A. Self-Supervised Distance and Pose… · p. 2
  • Feature-metric discriminative and convergent losses address homogeneous fisheye regions and inconsistent gradients by shaping feature-space convergence.The losses are computed on image feature representations learned with a self-attention autoencoder.Source: A. Self-Supervised Distance and Pose… · p. 2
  • The distance objective combines image and feature reconstruction with distance smoothness, cross-sequence consistency, discriminative, and convergent losses.The weights β, γ, ω, and µ scale the smoothness, consistency, discriminative, and convergent terms, respectively.Source: A. Self-Supervised Distance and Pose… · p. 2

B. Generalized Object Detection

The detection branch represents distorted fisheye objects with adaptively sampled polygons, while segmentation branches address semantic, motion, and lens-soiling masks.

B. Generalized Object Detection

  • Heavy radial distortion, especially at the periphery, makes standard bounding boxes unsuitable for fisheye object detection.The model uses 24-sided polygons with non-uniformly sampled vertices to represent objects.Source: B. Generalized Object Detection · p. 2
  • Curvature-adaptive sampling places polygon vertices non-uniformly to represent fisheye object contours more efficiently than uniform sampling.Dominant points are detected and then simplified; polygon annotation remains more expensive than bounding-box annotation.Source: B. Generalized Object Detection · p. 2

C. Segmentation Tasks

  • Semantic and soiling segmentation predict seven and four WoodScape classes, respectively, while motion segmentation predicts a binary moving-or-static mask from two frames.Training uses Lovasz-Softmax and Focal losses for the posterior probabilities.Source: C. Segmentation Tasks · pp. 2–3
  • Because the soiling dataset is independent, its decoder is trained with the shared encoder frozen after training on five other tasks.Asynchronous joint training achieved the same accuracy, and the output changed from tiled to pixel-level segmentation relative to SoilingNet.Source: C. Segmentation Tasks · p. 3

D. Joint Optimization

The paper evaluates strategies for balancing five task losses and proposes VarNorm; the network uses a shared encoder and cross-task decoder connections.

D. Joint Optimization

  • The weighting evaluation compares uncertainty loss, GradNorm, DTP, DWA, and geometric loss across five tasks.This extends the three-task experiments described in prior literature.Source: D. Joint Optimization · p. 3
  • VarNorm normalizes each task loss by its variance over the previous five epochs, assigning lower weight to greater dispersion and tending to homogenize task learning speed.The loss values are treated as distributions, with each task’s average loss over the last five epochs used in the formulation.Source: D. Joint Optimization · p. 3
  • The multitask network outperforms single-task networks with every evaluated dynamic weighting method, and VarNorm achieves the best results among them.Equal weighting performs worst; VarNorm is used in subsequent experiments.Source: D. Joint Optimization · p. 3

III. NETWORK DETAILS OF MTL OMNIDET

  • OmniDet uses a shared encoder with vector-attention pairwise and patchwise self-attention, plus cross-task decoder connections intended to improve task performance.The motion network uses a Siamese encoder whose previous-frame features can be reused rather than recomputed.Source: §III · p. 3

A. Camera Geometry Tensor Ct

Camera calibration can vary across camera instances and over time, motivating a geometry-conditioned model for multiple viewpoints. The camera geometry tensor supplies camera-specific maps to the network.

1) Motivation:

  • Manufacturing variation, temperature, and aging can change camera calibration, motivating adaptation across camera instances rather than separate models for each camera.A single model is described as offering memory, data-transmission, training-data, and maintenance advantages.Source: 1) Motivation · p. 3

2) Approach:

  • The camera geometry tensor Ct converts camera geometry properties into CNN inputs and generalizes camera-aware convolutions from pinhole to arbitrary geometries, including fisheye.The paper identifies adaptation to 12 WoodScape camera models as a contribution.Source: 1) Motivation · p. 3
  • Ct combines precomputed, calibration-derived coordinate and angle-of-incidence maps and is passed with input features through the self-attention modules and skip connections.It adds six channels; centered principal-point coordinate maps are bilinearly resized to match feature size and centered at (0, 0).Source: 2) Approach · pp. 3–4, 3) Centered Coordinates (cc) · p. 4

4) Angle of Incidence Maps (ax, ay):

The camera geometry maps encode both distortion-related angles and image-relative position. Incidence angles are derived from camera geometry, while normalized coordinates provide camera-independent spatial location.

4) Angle of Incidence Maps (ax, ay):

  • For fisheye models, angle-of-incidence maps are derived by inverting the radial distortion function; the paper’s polynomial model uses numerically computed fourth-order roots stored in a lookup table.The roots are precomputed for pixel coordinates to improve training efficiency.Source: 4) Angle of Incidence Maps (ax, ay) · p. 4

5) Normalized Coordinates (nc):

  • Normalized x and y coordinate channels vary linearly from −1 to 1 across image coordinates, independently of camera sensor properties.Values nearer to 1 indicate features nearer the corresponding image border.Source: 5) Normalized Coordinates (nc) · p. 4

B. Synergized Tasks

The model uses motion and semantic masks to reduce dynamic-object contamination in reconstruction, while semantic features guide distance estimation. Pixel-adaptive convolutions also transfer segmentation knowledge into the distance decoder.

1) Dealing With Dynamic Objects and Solving Infinite Depth Issue:

  • Motion masks exclude potentially moving objects from reconstruction, while distance is learned from non-moving dynamic objects to address infinite-depth values and loss contamination.The pixel-wise mask also excludes dynamic objects wrongly projected from reconstructed frames.Source: 1) Dealing With Dynamic Objects and… · p. 5
  • When motion segmentation is unavailable, semantic segmentation can provide a binary mask based on the designated dynamic-object classes.The method also describes masking reconstruction losses for images and features by pixel-wise multiplication.Source: 1) Dealing With Dynamic Objects and… · p. 5

2) Semantically-Guided Distance and Detection Decoder:

  • Pixel-adaptive convolutions distill segmentation-branch semantic knowledge into the distance decoder, allowing location-specific guidance in multi-level distance features.The convolution uses semantic guidance features to calculate correlations for an input neighborhood.Source: 2) Semantically-Guided Distance and… · p. 5

3) Linking Self-Attention and Semantic features to 2D detection:

The detection decoder fuses self-attention encoder features, semantic decoder features, and detection embeddings using pixel-adaptive convolutions. The implementation uses a single-stage learning process and feeds camera intrinsics into camera-geometry tensor creation.

3) Linking Self-Attention and Semantic features to 2D detection:

  • Pixel-adaptive convolution fuses semantic and detection embeddings into content-agnostic features, significantly improving detection-decoder accuracy.The process first passes self-attention encoder features through skip connections to the semantic decoder.Source: 3) Linking Self-Attention and Semantic… · p. 5

C. Implementation Details

  • Training uses a single-stage process with a patchwise self-attention encoder variant, Ranger optimizer, and 20 epochs.The batch size is 24; the initial learning rate is 4 × 10^-4 for 15 epochs and 10^-5 for the last five.Source: C. Implementation Details · p. 5
  • Camera intrinsics from shuffled surround-view images are used to create the camera-geometry tensor supplied to distance and pose networks.The distance-decoder sigmoid output is scaled to constrain distance between 0.1 and 100 units.Source: C. Implementation Details · p. 5

IV. EXPERIMENTAL RESULTS

The experiments train and test single-task and multi-task models on WoodScape, KITTI, and Cityscapes. The supplied passages describe dataset sizes, annotations, and input settings rather than quantitative benchmark outcomes.

A. Datasets

  • Single-task and multi-task models are trained and tested on WoodScape, KITTI, and Cityscapes.The datasets provide annotations for multiple tasks, with KITTI and Cityscapes serving as pinhole-camera datasets.Source: A. Datasets · p. 5

1) WoodScape:

  • WoodScape contains 10,000 images, and its polygon-regression task uses 24 high-curvature contour points for pedestrians and vehicles.The dataset split is 6:1:3; semantic segmentation covers road, lanes, and curbs.Source: 1) WoodScape · p. 5

2) Cityscapes:

  • Cityscapes training uses 2,975 images, with validation results reported on 500 images.The 2D boxes are extracted from instance polygons.Source: 2) Cityscapes · p. 5

3) KITTI:

  • KITTI depth estimation follows the Eigen et al. split, and motion labels are available only for cars.The dataset includes 42,382 stereo sequences with raw LiDAR scans, 7,481 box-annotated images, and 200 semantically annotated training images.Source: 3) KITTI · p. 6

B. Single Task vs Multi-Task Learning

The six-task WoodScape model and five-task models on KITTI and Cityscapes outperform their single-task counterparts, with state-of-the-art KITTI depth and pose results. Ablations examine loss choices, camera geometry, task weighting, semantic guidance, and detection representations.

B. Single Task vs Multi-Task Learning

  • The six-task and five-task multi-task models outperform their single-task counterparts, with state-of-the-art depth and pose estimation on KITTI.The KITTI depth and pose comparisons are reported in Tables V and VI, respectively.Source: B. Single Task vs Multi-Task Learning · p. 6

C. Ablation Study of our Contributions

  • Adding the camera geometry tensor to the ablation setting significantly increases accuracy across cameras with different intrinsics and viewing angles.The study also examines pairwise and patchwise self-attention encoder variants and reports gains from discriminative and convergent losses.Source: C. Ablation Study of our Contributions · p. 6
  • The WoodScape task-weighting comparison reports pixel accuracy as one measure.The supplied table caption identifies the comparison as task-weighting methods but gives no winning method or values.Source: Table III · p. 7
  • Motion-mask filtering yields superior gains with camera geometry tensor than semantic-mask filtering, whose classes may omit dynamic objects.The authors also report significant mAP gains from semantically guided features in the detection decoder.Source: C. Ablation Study of our Contributions · pp. 6–7
  • The detection-representation evaluation compares mIoU, attainable mIoU from ground-truth representations, and parameter counts.The compared forms include oriented boxes, ellipses, curved boxes, and 24-sided polar polygons alongside standard 2D boxes.Source: C. Ablation Study of our Contributions · p. 7, Table IV · p. 7

D. State-of-the-Art Comparison on KITTI

On KITTI, the depth method outperforms previous monocular methods, and the pose network outperforms previously listed methods. The paper also reports competitive results across five tasks on KITTI and Cityscapes.

D. State-of-the-Art Comparison on KITTI

V. CONCLUSION

  • The network achieves competitive results on five tasks across KITTI and Cityscapes.Source: §V · p. 7
  • Scaling to more tasks still faces challenges in dataset diversity and balance, corner-case mining, stable training, and unified map representation.Source: §V · p. 8

Related papers

Loading 2102.07448v3…