Source-linked AI summary
Detecting Unexpected Obstacles for Self-Driving Cars: Fusing Deep Learning and Geometric Modeling
Sebastian Ramos, Stefan Gehrig, Peter Pinggera, Uwe Franke, Carsten Rother
TL;DR
Small unexpected road hazards are difficult to detect because they vary in appearance and are rarely addressed by existing systems. The paper combines deep semantic labeling with stereo geometry in a Bayesian fusion framework using Stixels, achieving up to 50% relative improvement over the state of the art while operating in real time.
Problem
Small unexpected road hazards such as lost cargo are a challenging and important autonomous-driving problem requiring robust detection across varied appearances and contexts.
Method
The system combines pixel-wise semantic detection of free-space, unexpected obstacles, and background with stereo-based geometric detection, then fuses both using a Bayesian Stixel framework.
Results
50% relative detection-rate improvement and 13% fewer false positives are reported relative to the state of the art.
Takeaways & Limitations
The fused framework detects critical low-height obstacles at large distances while operating in real time on a self-driving platform.
Abstract
from arXiv · showhide
The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is used to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with relative performance gains of up to 50%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.
I. INTRODUCTION
Small, unexpected road hazards are difficult but important for autonomous driving because they vary widely in appearance and occur in an open-world setting. The paper combines learned appearance and contextual cues with stereo-based geometric detection, reporting improved detection and fewer false positives.
- Motivation: Small unexpected road hazards are important and challenging because they can cause serious safety risks for autonomous vehicles.The paper identifies lost cargo as an example and notes approximately 150 annual deaths associated with lost cargo on U.S. roads.
- Motivation: Stereo geometry alone is insufficient for small obstacles with varied shapes, sizes, and appearances, motivating complementary appearance-based learning.Traditional supervised learning also faces variability in object and road-surface appearance.
- Motivation: Road hazards can be detected as contextual anomalies because they are often surrounded by drivable road within an otherwise regular surface.
- Contribution: The proposed deep learning approach targets generic, small, unexpected obstacles and addresses unseen objects and background modeling as open-world challenges.
- Contribution: 50% relative detection-rate improvement and 13% fewer false positives are reported by probabilistically fusing learned and stereo-based detections.
II. RELATED WORK
Prior work includes stereo-based geometric detectors, appearance-based learning, free-space segmentation, and compact scene representations. These methods motivate combining complementary cues for small, generic obstacles, whose detection remains difficult at long range and small scale.
- Scope: Research on small, generic, unexpected driving obstacles remains limited compared with broader obstacle-detection work.
- Geometric methods: Classical geometric methods model ground and detect obstacles through deviations from planar or estimated ground surfaces.
- Geometric methods: Stixels represent scenes using an estimated global ground model and rectangular vertical obstacle segments.
- Geometric methods: Stereo-based methods perform reliably for medium-sized objects at close-to-medium range but degrade for longer ranges and smaller obstacles.
- Geometric methods: The paper builds on a stereo detector that compares obstacle and free-space hypotheses using constrained plane models and local tests.
- Learning methods: Appearance-based learning has been less investigated, with prior work using reconstruction-based anomaly detection and free-space segmentation.
- Scene representations: Semantic Stixels combine disparity-derived geometric cues with deep-learning semantic labels in a compact urban-scene representation.
III. METHODS
The complete system processes stereo images through parallel semantic and depth channels, converts both outputs into 3D Stixel representations, and probabilistically fuses them into final obstacle detections.
- Processing pipeline: The pipeline independently performs pixel-level semantic reasoning and depth estimation from an input image pair.
- Processing pipeline: Semantic and geometric channels each produce a 3D Stixel-based obstacle representation before fusion.
- Processing pipeline: A probabilistic fusion stage combines both channels into an overall 3D Stixel representation of potential road obstacles.
A. Appearance-Based Semantic Detection: Unexpected Obstacle Network (UON)
The Unexpected Obstacle Network uses pixel-wise semantic labeling to identify free-space, unexpected on-road obstacles, and background. It exploits contextual regularities, produces Stixels, and is trained with data and loss choices designed for robustness and class imbalance.
- Semantic classes: The semantic detector assigns every image pixel to free-space, on-road unexpected obstacles, or background.
- Semantic classes: Contextual learning uses obstacles’ small size and surrounding free-space to generalize across diverse appearances and distances.
- Semantic classes: Background includes irrelevant regions and standard urban obstacles assumed to be handled by general-purpose perception systems.
- Network: The Unexpected Obstacle Network is a Fully Convolutional Network with a GoogLeNet architecture chosen for accuracy and relatively low computational cost.
- Network: The network outputs per-pixel class probability maps that are converted into labels for downstream processing.
- Training: Training combines Lost and Found with Cityscapes to expose the model to small road obstacles and complex urban scenes.
- Training: Weighting the unexpected-obstacle class in the cross-entropy loss addresses pixel imbalance and balances detections against false alarms.
- Stixel generation: UON-Stixels are generated from the FCN argmax output by aligning labels to Stixel width and expanding obstacle labels vertically.
B. Stereo-Based Geometric Detection: Fast Direct Planar Hypothesis Testing (FPHT)
FPHT detects small obstacles from stereo images using local statistical tests that compare obstacle and free-space hypotheses, then represents detections as Stixels for probabilistic fusion with semantic and disparity cues.
- FPHT detection: FPHT applies generalized likelihood-ratio tests to local stereo patches, comparing free-space null hypotheses with obstacle alternatives under constrained 3D plane models.The model parameters are optimized in disparity space, allowing non-flat ground surfaces to be handled implicitly.
- Stixel generation: Detected obstacle points are clustered and reshaped into Stixel-like groups, producing a compact and flexible mid-level obstacle representation.The resulting FPHT-Stixels are used alongside UON-Stixels in the fusion stage.
- Fusion baselines: The system also evaluates logical AND and OR fusion, respectively targeting lower false-positive rates and higher detection rates.Corresponding Stixels are matched using at least 50% overlap.
- Probabilistic fusion: The fusion combines UON, FPHT, and disparity confidences within a Bayesian framework to estimate each Stixel’s probability of existence.The formulation assumes the individual confidence cues can be treated as independent because disparity is used only for coarse FPHT initialization.
- Probabilistic fusion: pUOpr = 0.5 is selected as the obstacle prior, deliberately favoring obstacle decisions despite dataset statistics suggesting pUOpr < 0.01.The prior is therefore a design choice rather than an estimate matching the observed obstacle frequency.
A. The Lost and Found Dataset and Extensions
The Lost and Found dataset targets difficult small-obstacle detection conditions, combining diverse stereo driving sequences with precise annotations and added distance and height ground truth.
- Dataset characteristics: The dataset contains 112 stereo video sequences at 2 MP resolution with pixel-accurate annotations for approximately 2100 frames.Its scenes include small obstacles, irregular road profiles, far distances, varied road appearance, and strong illumination changes.
- Ground-truth extensions: Additional ground truth was generated for object distance and height across the annotated frames.Sequence sets were fine-calibrated for small relative roll, yaw, and pitch changes before these extensions were produced.
- Distance distribution: Objects occur at distances up to 200 m, with most instances between 10 and 70 m; the test set reaches 110 m.Distance ground truth is derived from stereo disparity, using the median when mean deviation is below 1 px, which holds for over 99.4% of the data.
- Object scale: 97% of instances have an image-domain height below 60 px, reflecting the dataset’s emphasis on small obstacles.Object height ground truth was measured once per sequence for all objects and recorded orientations using stereo data.
B. Application-Specific Evaluation Metrics
The application-specific evaluation measures object-level Stixel overlap with pixel-accurate labels, using explicit overlap thresholds for true and false positives.
- False positives: A Stixel is a false positive when more than 50% of its area overlaps labeled free space, excluding a 10 px margin around ground-truth obstacles.The margin treats foreground-fattening errors as acceptable for this task.
- True positives: A detection is true positive when its Stixel overlaps the ground-truth label by more than 50%.The metric operates on overlap between pixel-accurate annotations and the algorithm’s Stixel results.
- Object-level detection: An object counts as detected when at least one corresponding Stixel is found, yielding an approximate upper bound on detection rate.The authors characterize this interpretation as optimistic.
C. Quantitative Results
On the Lost and Found dataset, semantic–geometric fusion achieves the strongest detection performance while maintaining practical false-positive and runtime characteristics. Performance is analyzed across distance, obstacle height, confidence thresholds, and computational stages.
- Detection over distance: More than 90% detection is achieved for objects up to 50 m using OR-fusion and probabilistic fusion with p_thresh = 0.7.These methods outperform the listed UON-Stixels, FPHT, AND-fusion, Stixels, and Semantic Stixels in the distance evaluation.
- Detection and false positives: About 50% detection-rate improvement is obtained over FPHT alone while reaching comparable false-positive rates.UON-Stixels produce 0.1 false positives per frame, while FPHT is evaluated at 0.5 false positives per frame after excluding nearby false positives.
- Height and distance: More than 90% detection is obtained for objects exceeding 20 px height with FPHT in the height-over-distance evaluation.The evaluation specifically examines object height in the image together with distance.
- Confidence calibration: Detection rate is largely maintained for confidence thresholds up to 0.7, allowing the threshold to reduce false positives.Varying the obstacle prior produces a very similar false-positive/detection-rate curve.
- Operational false positives: One false positive occurs every 2 km driven for the fusion system despite high apparent false-positive counts on the challenging dataset.The dataset includes unusual road markings and vertically curved road profiles that intentionally emphasize rare challenges.
- Runtime: The processing stages take 45 ms for FPGA SGM, 35 ms for CPU FPHT-Stixels, 40 ms for GPU UON, and 5 ms for CPU fusion.The reported timings use an Intel Xeon 3GHz CPU and Nvidia TitanX GPU for the corresponding stages.
D. Qualitative Results
Qualitative examples show that fusion combines complementary strengths of the detection methods on distant, tiny, curved-road obstacles and mixed object classes. It suppresses false positives while recovering detections missed by individual components.
- Lost cargo at 40 m: At around 40 m, UON and fusion detect all three lost-cargo objects without FPHT’s shadow-induced false positive.Both Stixel baselines miss the objects, while FPHT detects an object and also triggers on a horizontal shadow and slight decalibration.
- Small obstacles on curved roads: At 42 m, UON and fusion detect 5 cm obstacles on a strongly curved road without false positives.The Stixel baseline produces many false positives, Semantic Stixels miss the objects, and FPHT fails to detect them at that distance.
- Mixed object classes: Fusion recovers the child-on-bobby-car detection that UON only partially detects because vehicles and pedestrians belong to its learned background class.The Stixel baselines perform fairly well, and FPHT cleanly detects the object in this example.
V. CONCLUSIONS AND FUTURE WORK
The framework combines semantic and geometric cues to detect small unexpected obstacles, achieving improved detection with fewer false positives and real-time operation. Future work targets greater robustness through additional training data and tighter fusion integration.
- Contributions: The framework combines visual appearance, contextual information, and geometric cues for small unexpected-obstacle detection.It uses a deep learning system together with a probabilistic fusion scheme and a stereo-based system.
- Results: Up to 50% relative detection-rate improvement accompanies a 13% reduction in false positives versus the state-of-the-art.
- Results: The system operates in real time on a self-driving platform.
- Future work: Future work will collect additional training data, including bad-weather recordings, to further increase robustness.The authors also plan to integrate probabilistic fusion directly into the learning system.