PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling

Stefan Hoermann, Martin Bach, Klaus Dietmayer

arXiv:1705.08781v2cs.RO

TL;DR

Long-term prediction of interacting road users in complex urban scenes is difficult, especially when raw multi-sensor time series create timing and representation challenges. The paper combines Bayesian dynamic occupancy grids with a CNN, automatic label generation, and pixel balancing. The network predicts complex multi-user scenarios up to 3 s and can infer multiple future maneuvers and spatiotemporal occupancy distributions.

  • Problem

    Long-term prediction must handle complex urban scenes with multiple interacting road users, while raw sensor time series pose learning challenges from asynchronous and varying-rate inputs.

  • Method

    A CNN predicts from a Bayesian dynamic occupancy grid containing spatial occupancy and velocity estimates, with automatic labels and balancing for rare dynamic cells.

  • Results

    The network predicts complex scenarios with varied road users up to 3 s and infers possible driving maneuvers and future spatiotemporal occupancy distributions.

  • Takeaways & Limitations

    A single filtered DOGMa input can support long-term, multi-modal scene prediction while modeling interactions through spatial context.

Abstract

from arXiv · show

Long-term situation prediction plays a crucial role in the development of intelligent vehicles. A major challenge still to overcome is the prediction of complex downtown scenarios with multiple road users, e.g., pedestrians, bikes, and motor vehicles, interacting with each other. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation, and machine learning as long-term predictor. More specifically, a dynamic occupancy grid map is utilized as input to a deep convolutional neural network. This yields the advantage of using spatially distributed velocity estimates from a single time step for prediction, rather than a raw data sequence, alleviating common problems dealing with input time series of multiple sensors. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. Pixel-wise balancing is applied in the loss function counteracting the extreme imbalance between static and dynamic cells. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data and compared to Monte-Carlo simulation.

I. INTRODUCTION

Long-term prediction in complex urban scenes remains difficult because multiple road users interact, while raw multi-sensor time series introduce technical learning challenges. The paper combines Bayesian environment estimation with CNN-based prediction and automatic labeling.

  • Complex downtown prediction must account for interacting cars, trucks, bikes, and pedestrians.
  • Raw sensor time series create challenges including varying sample frequencies, asynchronous sensors, temporal scaling and translation, and out-of-sequence inputs.
  • The approach uses spatial context from a dynamic occupancy grid map to support prediction from a single time step.
  • A CNN models long-term motion from a Bayesian estimate of the current dynamic environment while implicitly exploiting object and infrastructure context.
  • Fully automatic labeling separates rare dynamic from static cells, enabling balancing during training without hand-crafted labels.

II. FILTERED DYNAMIC INPUT

The filtered dynamic input is a bird’s-eye DOGMa that fuses sensor data into probabilistic occupancy and velocity information. Its sensor-independent representation avoids feeding raw sensor streams directly to the CNN.

  • A DOGMa represents the 360° environment in bird’s-eye view using Bayesian filtering over multiple sensor inputs.
  • Each grid cell contains free-space and occupancy masses, eastward and northward velocity estimates, and velocity uncertainty information.
  • The occupancy probability is visualized so that higher values correspond to darker pixels in the illustrative grid map.
  • Using DOGMa inputs provides spatially distributed velocity estimates with covariance while avoiding raw-sensor timing and setup dependencies.
  • Sequential Monte Carlo filtering combines sensor-specific occupancy grids with particle-based estimation of spatial occupancy and velocity distributions.

III. AUTOMATIC OUTPUT LABEL GENERATION

Training labels are generated automatically from future occupancy courses, separating static and dynamic contributions at each cell. This addresses the extreme dominance of static environment data during learning.

  • Approximately 99.75% of the environment data is static, motivating segmentation and balancing of underrepresented dynamic cells.
  • Occupancy probability courses are extracted for fixed spatial cells while the ego vehicle moves through the dynamic environment.
  • The desired output contains future occupancy values across time and a segmentation into static and dynamic components.
  • A rise followed by a fall in occupancy probability identifies a dynamic object traversing a cell, separating it from the static occupancy level.
  • The static occupancy label is computed as the median of static estimates outside detected dynamic intervals over the prediction horizon.

IV. CNN ARCHITECTURE

The CNN uses an encoder–decoder structure to preserve spatially distributed outputs while exploiting distant relations in the DOGMa. Its channels predict future static and dynamic occupancy rather than merely classifying pixels.

  • The architecture processes one DOGMa through downscaling stages mirrored by upscaling stages to retain spatial output structure.
  • Learnable deconvolution kernels and bypass connections from corresponding downscaling stages support spatially detailed prediction.
  • The network predicts occupied cells at future time steps instead of performing ordinary pixel classification.
  • Separate output channels represent static occupancy and dynamic occupancy over the remaining prediction channels.

V. SPATIAL BALANCING LOSS

The loss separates static and dynamic occupancy channels, then increases weighting for rare dynamic cells and later prediction times.

  • Static occupancy is stored in the first channel, while dynamic occupancy occupies later time channels and static-cell labels there are set to 0.
  • The weighting scheme emphasizes rare dynamic cells relative to static cells because dynamic regions comprise only 0.25% of the environment.
  • The total loss combines static-channel loss Ls with dynamic-channel loss Ld.
  • Dynamic-cell weights equal 1 for static regions and 1 + λk for dynamic regions, with λk increasing across prediction-time channels.

VI. DATASET AND TRAINING

The study uses 2.5 hours of urban shared-space recordings collected across three days and two vehicle positions, divided into training, testing, and validation samples.

  • The dataset contains 77582 samples from 2.5 hours of recordings collected across three days and two vehicle positions.
  • 62800 samples were used for training, 7590 for testing, and 7192 for validation.
  • The network was trained with the ADAM solver using automatic learning-rate adjustment.

VII. EVALUATION

The evaluation compares learned prediction with particle propagation in a challenging downtown scene, using ROC curves against the actually perceived DOGMa at future times.

  • The experiment uses a downtown shared-space scenario with pedestrians and bikes crossing arbitrarily, numerous dynamic objects, and field-of-view occlusions.
  • The learning-based method is compared with a particle algorithm that forward-propagates 900000 particles from cells classified as dynamic by velocity heuristics.
  • ROC curves evaluate occupied-versus-free dynamic cells against the actually perceived DOGMa at each prediction time, rather than automatically generated labels.
  • As prediction time increases, achieving a common true-positive rate requires a growing false-positive rate, reflecting greater uncertainty in the predicted area.
  • The learning-based approach reaches a minimum true-positive rate of 0.81, compared with 0.75 for the particle approach when dynamic prediction is ignored.
  • At a 0.5 s prediction time, maximum true-positive rates are 98% for learning and 89% for particles.

A. Predicting Multi-Modal Scene Evolution

In a scene with an ambiguous turning maneuver, the network predicts multiple possible future trajectories, whereas particle propagation covers only the straight maneuver.

  • The example shows parallel traffic with a possible right-hand turn, using one DOGMa as the network input and predictions from 0.5 s to 3.0 s.
  • Because the vehicle’s maneuver is ambiguous, the prediction network outputs a bimodal occupancy distribution for its trajectory.
  • Propagated particles cover only the straight maneuver, while the learned prediction indicates possible driving maneuvers and future spatiotemporal occupancy.

B. Static Regions

Static and dynamic cells are difficult to distinguish from cell-wise features alone, causing errors in both automatic labeling and particle prediction. The neural network can avoid propagating some static-region misclassifications.

  • B. Static Regions: Cell-wise features make static-versus-dynamic classification difficult, leading to misclassified static cells in automatic labels and particle predictions.Particle errors can blur static areas or omit dynamic objects when parameters are poor.
  • B. Static Regions: The neural network does not predict movement for some misclassified static regions, unlike the particle approach.In the example, it avoids predicting motion around the corners of a static wall.

C. Predicting Partially Occluded Objects

The learned predictor uses spatial context to handle interactions and partial occlusions in complex urban scenes. It can enlarge predicted occupied regions, adapt pedestrian paths to approaching vehicles, and model vehicle slowing near obstacles.

  • C. Predicting Partially Occluded Objects: The neural network predicts enlarged occupied areas for objects entering the field of view, while particles underestimate partially occluded vehicles.The learned prediction covers an entire partially occluded vehicle, whereas particles predict only its front.
  • D. Reducing Prediction Variance with Object Interactions: Spatial context enables the network to model interactions, reducing predicted future occupancy when a blocked lane requires a vehicle to slow down.The particle filter overestimates motion and predicts a collision with the standing vehicle.
  • C. Predicting Partially Occluded Objects: For a possibly turning vehicle, learned predictions represent multiple future maneuvers, whereas particles cover only the straight maneuver.The learned distribution becomes multimodal from 1.0 s onward.
  • D. Reducing Prediction Variance with Object Interactions: As an approaching vehicle nears a pedestrian, the network removes the predicted crossing path, while the particle approach cannot predict movement for temporarily static objects.The prediction changes when the vehicle’s arrival time decreases from about 3 s to about 1 s.
  • D. Reducing Prediction Variance with Object Interactions: Adding an off-road pedestrian causes the network to predict earlier vehicle deceleration, demonstrating sensitivity to altered interaction context.Without interaction modeling, the original trajectory would reach the pedestrian’s position and cause a collision.

VIII. CONCLUSIONS

The paper presents a CNN-based, automatically labeled approach for predicting complex urban scenes up to 3 s ahead. Results indicate that it handles multiple maneuvers, road-user interactions, and partially observed objects, while future work targets probabilistic loss weighting and recurrent architectures.

  • VIII. CONCLUSIONS: The approach predicts the 360° perceivable scene with a single neural network and also segments static and dynamic areas.Training uses fully automatic label generation, giving the learning procedure an unsupervised character.
  • VIII. CONCLUSIONS: Figure 6 shows interaction-sensitive predictions for a crossing pedestrian and a stopping vehicle, while the particle approach predicts no pedestrian movement and a collision.The figure compares camera views, DOGMa inputs, and RGB-coded prediction outputs.
  • VIII. CONCLUSIONS: The network predicts complex scenarios with different road-user classes up to 3 s and represents alternative maneuvers such as turning or going straight.The conclusion also reports that interactions between road users reduce prediction uncertainty.
  • VIII. CONCLUSIONS: Future work extends the loss function to account for increasing variance in an object’s future position and plans experiments with recurrent neural networks.These directions are presented as ongoing studies rather than completed results.
Loading 1705.08781v2…