PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Corner Case Detection for Autonomous Driving

Jan-Aike Bolte, Andreas Bär, Daniel Lipinski, Tim Fingscheidt

arXiv:1902.09184v2cs.CV

TL;DR

Corner-case detection lacks a settled definition, and training data may rarely cover critical situations. The paper defines corner cases using unpredictability, object relevance, and location relevance, then proposes a video-based detection framework combining segmentation and prediction. Preliminary qualitative experiments show promising results, while evaluation is constrained by the absence of labeled corner-case test data.

  • Problem

    Corner-case detection lacks a universally accepted definition, while machine-learning training data may rarely or never cover some situations.

    Source: §I · p. 1

  • Method

    The paper defines a corner case as a non-predictive relevant object or class in a relevant location and combines semantic segmentation with video-prediction error.

    Source: §I · p. 2, A. Semantic Segmentation · p. 4

  • Results

    Preliminary qualitative experiments showed promising results, and a woman crossing immediately on a road curve received the system’s highest corner-case score.

    Source: §VI · p. 8, D. Detection System · p. 8

  • Takeaways & Limitations

    The proposed framework is intended for online warning and offline selection of unusual driving video for more focused training.

    Source: §I · p. 1, §VI · p. 8

    The authors report that no dataset of labeled autonomous-driving corner cases was available for evaluating detection metrics such as FAR, FRR, or ROC AUC.

    Source: D. Detection System · p. 7

Abstract

from arXiv · show

The progress in autonomous driving is also due to the increased availability of vast amounts of training data for the underlying machine learning approaches. Machine learning systems are generally known to lack robustness, e.g., if the training data did rarely or not at all cover critical situations. The challenging task of corner case detection in video, which is also somehow related to unusual event or anomaly detection, aims at detecting these unusual situations, which could become critical, and to communicate this to the autonomous driving system (online use case). Such a system, however, could be also used in offline mode to screen vast amounts of data and select only the relevant situations for storing and (re)training machine learning algorithms. So far, the approaches for corner case detection have been limited to videos recorded from a fixed camera, mostly for security surveillance. In this paper, we provide a formal definition of a corner case and propose a system framework for both the online and the offline use case that can handle video signals from front cameras of a naturally moving vehicle and can output a corner case score.

I. INTRODUCTION

Corner-case detection targets unusual situations that may be missing from training data, using video from a moving vehicle for both online warning and offline data selection. The paper proposes a formal definition and a modular detection framework.

I. INTRODUCTION

  • The motivation is that training data may rarely or never cover situations that matter, while video from a monocular vehicle camera can represent part of a situation.The authors argue that people can distinguish normal from unusual events using such video alone.Source: §I · p. 1
  • Offline detection can select unusual recordings for focused training, including oversampling corner cases to address underrepresented critical data.Online detection can act as a redundant warning about how unusual the current situation is.Source: §I · p. 1
  • Corner-case detection lacks a universally accepted definition and is closely related to anomaly and novelty detection.The introduction distinguishes anomalies from novelties according to whether a world model adequately represents normality.Source: §I · pp. 1–2
  • The framework combines semantic segmentation with video-based prediction error to score non-predictive, relevant objects near the vehicle’s expected future trajectory.The authors present this as a way to provide a perception module with self-awareness and a criticality measure.Source: §I · p. 2

II. RELATED WORK

The paper draws on anomaly detection, video prediction, and semantic segmentation to build a modular corner-case detector. It adapts prediction approaches to automotive video, where the camera moves with the vehicle.

II. RELATED WORK

  • The proposed detector combines image prediction and semantic segmentation, adopting subsystems that can also be improved independently.The paper frames this modular design as closely related to anomaly detection.Source: §II · p. 2

A. Anomaly Detection

  • Reconstruction-based anomaly detection commonly learns normality from largely normal, unlabeled data and assigns novelty or abnormality scores to test sequences.The paper distinguishes image-based methods from sensor-based or non-image methods that it considers unsuitable for its task.Source: A. Anomaly Detection · p. 2

B. Image Prediction

  • Earlier video-prediction approaches largely evaluated on stationary-camera datasets, while automotive prediction must account for vehicle and object motion.PredNet is described as capturing key aspects of both kinds of movement.Source: B. Image Prediction · pp. 2–3
  • The experiments evaluate PredNet and a predictive autoencoder based on Hasan et al., with an adversarially trained predictive approach proposed for corner-case detection.The paper selects these approaches for automotive image prediction experiments.Source: B. Image Prediction · p. 3

C. Semantic Segmentation

The paper describes semantic segmentation as pixel-level class labeling and bases its network on DeepLabv3. It defines an autonomous-driving corner case through unpredictability, object relevance, and location relevance.

C. Semantic Segmentation

  • Semantic segmentation labels each input-image pixel with a class, and the paper bases its network on DeepLabv3 with modifications.The related-work discussion notes that segmentation architectures address context and resolution, among other design concerns.Source: C. Semantic Segmentation · p. 3

III. CORNER CASE DEFINITION

  • Because corner cases lack a universally accepted definition and explicit metric, the paper motivates a predictive detection approach.The authors argue that a technically predictable situation should not count as a corner case for their purpose.Source: §III · p. 3
  • A corner case requires a non-predictable relevant object or class in a relevant location.The definition combines all three criteria rather than treating every unusual or unpredictable event as a corner case.Source: §III · p. 3
  • An unpredictable event is not necessarily a corner case when it is irrelevant to driving, such as an aircraft in the sky.Pedestrians, cyclists, and other moving objects on the ground are given as relevant classes; distance from the vehicle or its future trajectory also matters.Source: §III · p. 3

IV. TOWARDS CORNER CASE DETECTION: FRAMEWORK AND MEASURES

The framework targets non-predictable situations involving relevant objects in relevant locations, combining image prediction, semantic segmentation, and information fusion. Its segmentation module assigns each pixel a scene class and evaluates masks with mean intersection over union.

IV. TOWARDS CORNER CASE DETECTION: FRAMEWORK AND MEASURES

  • The detector combines prediction errors, semantic classes and locations, and information fusion to check whether unpredictable relevant objects occupy relevant locations.The framework focuses on moving objects as relevant classes.Source: §IV · p. 3

A. Semantic Segmentation

  • DeepLabv3 with a WideResNet38 backbone produces pixel-wise class scores and assigns each pixel to one of 19 classes.The segmentation output is a posterior score for each class at each pixel, followed by an argmax classification.Source: A. Semantic Segmentation · p. 4
  • Mean intersection over union measures segmentation-mask accuracy by averaging frame-wise IoU, computed from true-positive, false-positive, and false-negative pixels.The text notes that overconfident classifiers can assign unknown objects to known classes; a single-frame anomaly detector is proposed as future work.Source: A. Semantic Segmentation · p. 4

IV. TOWARDS CORNER CASE DETECTION: FRAMEWORK AND MEASURES

  • Figure 3 presents the corner-case detector as a high-level block diagram.Source: Figure 3 · p. 4

B. Image Prediction

The image-prediction module uses consecutive video frames to predict the current frame, then compares that prediction with the actual image. The paper evaluates prediction quality using MSE, PSNR, and SSIM.

B. Image Prediction

  • The predictive model receives n consecutive past frames and predicts the current frame, whose difference from the actual frame supplies pixel-wise prediction errors.Source: B. Image Prediction · p. 4
  • MSE and PSNR are used to evaluate the prediction models by measuring differences between predicted and actual images.Source: B. Image Prediction · p. 4
  • SSIM evaluates perceptual differences between predicted and original images based on visible image structures.Source: B. Image Prediction · p. 4

C. Detection System

The detection system combines semantic segmentation with image-prediction errors to produce a corner-case score for each frame. It weights errors for relevant classes by image location, normalizes scores over a selected time set, and applies a threshold.

C. Detection System

  • The system combines outputs from prediction and segmentation into a corner-case score between 0 and 1 for each input frame, using current and past frames.Source: C. Detection System · p. 5
  • Squared errors for relevant classes are weighted by pixel row and summed, with bottom-row errors weighted fully and top-row errors weighted zero.The relevant classes are the eight mobile classes identified for corner-case detection.Source: C. Detection System · p. 5
  • Scores are normalized over a selected set of time instants and thresholded; offline users can choose the threshold to control how many corner cases are selected.For localization, the image may be divided into same-size patches, such as 32×32 pixels, whose scores can also be normalized.Source: C. Detection System · p. 5

A. Dataset

The study trains segmentation on Cityscapes and trains image prediction on three dataset demo videos. The segmentation model achieves 78.4% mIoU on the Cityscapes evaluation set, and uses fused predictions from multiple input scales.

A. Dataset

  • Cityscapes provides street-scene images from 50 cities; segmentation and image prediction are trained on Cityscapes, while image prediction uses its three demo videos.Cityscapes supplies 19 classes used during training and inference.Source: A. Dataset · p. 5

B. Semantic Segmentation

  • Segmentation training uses two stages: 90,000 iterations at output stride 16, then 120,000 iterations at stride 8 with batch statistics frozen.The initial learning rate is reduced from 0.001 to 0.0005 for the second stage.Source: B. Semantic Segmentation · p. 5
  • The evaluation fuses class score maps from parallel fine-tuned models at input scales 0.75, 1.0, and 1.25 after resizing them to the original image size.The models are evaluated using mIoU at output stride 8.Source: B. Semantic Segmentation · pp. 5–6
  • 78.4% mIoU on the Cityscapes evaluation set, close to the best approaches known at the time, is reported for the segmentation network.The network was trained solely on finely annotated Cityscapes training images.Source: B. Semantic Segmentation · p. 6

C. Image Prediction

The section compares two video-prediction models and selects a predictive autoencoder for corner-case detection. Its training combines an MSE objective with an adversarial loss.

C. Image Prediction

  • Both models are tested in leave-one-out training on the three Cityscapes demo videos, with each unseen-video test averaged across combinations.The section limits this initial comparison to Cityscapes because the segmentation network was trained on that dataset.Source: C. Image Prediction · p. 6
  • The predictive autoencoder uses an encoder-based discriminator with patch-wise classification of real versus predicted images.The discriminator is added to address blurry predictions associated with standard MSE training.Source: C. Image Prediction · p. 6, Figure 5 · p. 6
  • Training the generator combines standard MSE with adversarial loss, weighted by λ = 0.25.The training description also reports improved stability with a cyclic generator learning rate.Source: C. Image Prediction · p. 6
  • The predictive autoencoder provides better MSE than PredNet and is selected for its simple, modifiable feed-forward architecture.Prediction quality is reported as not far apart for PSNR and SSIM; the stated selection also reflects the better MSE, which is closely related to the detector's measure.Source: C. Image Prediction · p. 7

D. Detection System

The detection system uses blurred-image prediction error to form a corner-case score, with examples shown qualitatively. The authors note that labeled corner-case data is unavailable for quantitative evaluation.

D. Detection System

  • Without labeled corner-case test data, the authors present qualitative examples rather than quantitative detection metrics.They identify the lack of a labeled dataset as the reason FAR, FRR, and AUC cannot be applied here.Source: D. Detection System · p. 7
  • Table II reports example PredNet and predictive-autoencoder results averaged over unseen Cityscapes videos in leave-one-out training.The caption notes that discriminator parameters bring the predictive autoencoder's total parameter count to 10,795,650.Source: Table II · p. 7
  • A 10 × 10 Gaussian blur before squared-error calculation reduces error in ordinary situations while leaving clearer periods of nonzero error for potential corner cases.The score based on blurred-image MSE focuses on relevant classes and suppresses high errors for absent or predictable relevant objects.Source: D. Detection System · p. 7
  • The pedestrian crossing in a road curve produces the highest corner-case score among the three described situations.The other examples are a car being overtaken and a nearby oncoming car that the system cannot predict.Source: D. Detection System · pp. 7–8

VI. CONCLUSIONS

The conclusion defines corner cases through non-predictability, relevance, and location, and presents a modular detector with promising preliminary qualitative results. It identifies labeled corner-case video as necessary for quantitative evaluation.

VI. CONCLUSIONS

  • A corner case is a non-predictable relevant object or class in a relevant location.The detector assigns a module to each of the definition's three aspects.Source: §VI · p. 8
  • The proposed detection framework shows promising results in preliminary qualitative experiments.The semantic-segmentation and image-prediction subsystems can be improved independently and compared using established metrics in their fields.Source: §VI · p. 8
  • The authors identify recording and labeling arranged corner-case videos as the next step for quantitative evaluation of the full detector.Source: §VI · p. 8

Related papers

Loading 1902.09184v2…