PaperlayerView exact revision on arXiv

Source-linked AI summary

FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, Alex Kendall

arXiv:2104.10490v3cs.CVcs.RO

TL;DR

Autonomous driving needs models that anticipate multiple possible futures of other agents, but prior approaches often depend on staged pipelines, LiDAR, HD maps, or top-down inputs. FIERY predicts future BEV instance segmentation and motion directly from surround monocular cameras, modeling stochastic multimodal futures end to end, and outperforms previous prediction baselines on NuScenes and Lyft.

  • Problem

    Autonomous driving requires anticipating diverse joint behaviors of dynamic agents because the observed present state may permit many valid futures.

  • Method

    FIERY jointly estimates future bird’s-eye-view instance segmentation and motion from surround RGB monocular videos using an end-to-end probabilistic model without HD maps.

  • Results

    FIERY outperforms previous autonomous-driving prediction baselines on the NuScenes and Lyft datasets.

  • Takeaways & Limitations

    FIERY predicts temporally consistent future instance segmentations and motion while modeling diverse futures directly from surround RGB videos.

Abstract

from arXiv · show

Driving requires interacting with road agents and predicting their future behaviour in order to navigate safely. We present FIERY: a probabilistic future prediction model in bird's-eye view from monocular cameras. Our model predicts future instance segmentation and motion of dynamic agents that can be transformed into non-parametric future trajectories. Our approach combines the perception, sensor fusion and prediction components of a traditional autonomous driving stack by estimating bird's-eye-view prediction directly from surround RGB monocular camera inputs. FIERY learns to model the inherent stochastic nature of the future solely from camera driving data in an end-to-end manner, without relying on HD maps, and predicts multimodal future trajectories. We show that our model outperforms previous prediction baselines on the NuScenes and Lyft datasets. The code and trained models are available at https://github.com/wayveai/fiery.

1. Introduction

FIERY targets autonomous-driving prediction directly in bird’s-eye view from surround monocular cameras, combining perception, sensor fusion, and prediction without HD maps. It models stochastic, multimodal futures of dynamic agents and outperforms prior prediction baselines on NuScenes and Lyft.

  • 1. Introduction: Future-state prediction helps motion planning resolve ambiguous outcomes caused by the motion of road users and pedestrians.The challenge is especially important in highly dynamic autonomous-driving environments.
  • 1. Introduction: Camera-based perception could provide a leaner, cheaper, higher-resolution visual recognition system than LiDAR sensing.The paper anticipates extending camera-based advances to wider monocular-vision prediction tasks.
  • 1. Introduction: FIERY operates in an orthographic bird’s-eye-view frame without auxiliary HD-mapping systems that generate rasterized scene representations.The BEV frame is chosen for its benefits to planning and control.
  • 1. Introduction: The model joins perception and sensor fusion with prediction by estimating BEV predictions directly from surround RGB monocular camera inputs.This replaces a staged pipeline that merges independently predicted sensor outputs.
  • 1. Introduction: FIERY predicts future instance segmentation and motion of dynamic agents, modeling probabilistic, multimodal futures directly from camera driving data without HD maps.Its future motion can represent interactions and diverse trajectories rather than merely extrapolating current behavior.
  • 1. Introduction: FIERY demonstrates quantitative benchmarks for future dynamic-scene segmentation and outperforms previous autonomous-driving prediction baselines on NuScenes and Lyft.This is the paper’s principal reported benchmark outcome.

2. Related Work

Prior work studies lifting camera images into bird’s-eye-view representations and predicting future trajectories, increasingly using end-to-end models. However, autonomous driving requires jointly modeling multiple plausible future behaviors rather than only deterministic or waypoint-level uncertainty.

  • Bird’s-eye view representation from cameras: Lifting 2D perspective images into bird’s-eye view is an inherently ill-posed problem addressed by prior camera-based methods.Some approaches generate semantic BEV maps directly from images using simulator-derived ground truth.
  • Bird’s-eye view representation from cameras: NuScenes and Lyft enabled direct supervision on real-world BEV semantic segmentation through labels generated from 3D object detections.Related approaches include Bayesian occupancy prediction and depth-based lifting from monocular RGB images.
  • Future prediction: Classical future prediction uses detect-track-predict pipelines, while newer end-to-end methods address cascading errors and latency but often rely on LiDAR, HD maps, or additional sensors.These approaches incorporate maps, constraints, radar, and other sensors for improved robustness.
  • Future prediction: Deterministic trajectories and waypoint-level uncertainty do not capture the jointly multimodal future behaviors that autonomous-driving agents may exhibit.From an observed past, multiple valid and probable futures can occur, motivating probabilistic multihypothesis prediction.

3. Model Architecture

FIERY lifts surround-camera features into a temporally aligned bird’s-eye-view representation, models stochastic futures, and decodes future instance segmentation and motion.

  • 3.1. Lifting camera features to 3D: FIERY extracts features and depth probabilities from each camera image, then lifts and fuses them into a common 3D reference frame.Known camera intrinsics and extrinsics support the lifting operation.
  • 3.1. Lifting camera features to 3D: The 3D features are vertically sum-pooled into 200 × 200 bird’s-eye-view maps covering a 100m × 100m area at 0.50m resolution.The BEV feature map uses 0.50m × 0.50m spatial columns.
  • 3.3. Learning a temporal representation: Past BEV features are warped into the present reference frame with known ego-motion, concatenated with spatially broadcast actions, and processed by a 3D convolutional temporal model.The temporal model outputs a spatio-temporal state.
  • 3.4. Probabilistic future prediction: A conditional variational model uses present and future distributions so latent samples encode possible futures and the present distribution covers observed futures.The future distribution accesses observed future labels during training, while the present distribution uses only the current spatio-temporal state.
  • 3.5. Future prediction in bird’s-eye view: A convolutional gated recurrent unit recursively predicts future states, which a BEV decoder maps to segmentation, centerness, offsets, and future instance flow.The flow field supports temporally consistent tracking by matching flow-warped centers with subsequent detected centers.
  • 3.7. Losses: Training uses top-k cross-entropy for semantic segmentation, ℓ2 centerness loss, ℓ1 offset and flow losses, and exponential discounting with γ = 0.95.The top-k loss backpropagates through the hardest 25% of pixels.

4. Experimental Setting

FIERY is evaluated on NuScenes and Lyft using full-surround camera data, future BEV video metrics, and multimodal-future evaluation over a two-second horizon.

  • 4.1. Datasets: NuScenes has 1000 20-second scenes annotated at 2Hz, while Lyft has 180 scenes lasting 25–45 seconds and annotated at 5Hz.Both datasets provide six cameras covering the ego-vehicle’s full 360° field of view.
  • 4.2. Metrics: Future Video Panoptic Quality measures recognition quality through temporally consistent instance detection and segmentation quality through accurate instance segmentation.A true positive requires IoU above 0.5 and an instance identity consistent with the ground truth over time.
  • 4.2. Metrics: The evaluation also reports Generalised Energy Distance to measure the model’s ability to predict multimodal futures.The metric is defined in Appendix A.3.
  • 4.1. Datasets: The model takes 1.0s of past context and predicts 2.0s into the future.This corresponds to 3 past and 4 future frames in NuScenes, and 6 past and 10 future frames in Lyft.
  • 4.1. Datasets: Training uses six camera images per past timestep at 224 × 480 resolution and produces 100m × 100m BEV predictions at 50cm resolution.The model is trained with Adam at a constant learning rate of 3×10−4 for 20 epochs on four Tesla V100 GPUs.

5. Results

FIERY is evaluated against prior bird’s-eye-view segmentation methods and classical future-prediction baselines, then analyzed through architectural ablations. Results show stronger present and future segmentation, with temporal context, reference-frame transformation, recursive prediction, future flow, and learned depth weighting each contributing to performance.

  • 5.1. Comparison to the literature: FIERY Static outperforms all previous baselines for present-time bird’s-eye-view semantic segmentation on NuScenes.
  • 5.1. Comparison to the literature: FIERY achieves higher intersection-over-union than its single-timeframe counterpart by accumulating information over time and handling partial observability and occlusions.The authors also report sharper and more accurate predictions qualitatively.
  • 5.1. Comparison to the literature: FIERY performs much better than Fishing Net’s camera and LiDAR variants for bird’s-eye-view semantic segmentation 2.0s in the future.
  • 5.2. Future prediction: FIERY largely outperforms the Static and Extrapolation baselines for future prediction on NuScenes and Lyft.The evaluation reports future instance segmentation and motion prediction, including IoU and Video Panoptic Quality.
  • 5.3. Analysis: The No temporal context variant performs similarly to the static model because it lacks past information needed to infer road-agent motion.
  • 5.3. Analysis: Removing transformation to the present’s reference frame causes a large performance drop, while removing recursive unrolling also substantially degrades performance.The reported explanations are easier correspondence learning after ego-motion is factored out and loss of the sequential constraint, respectively.
  • 5.3. Analysis: Without future flow, predictions are no longer temporally consistent, while uniform depth lifting performs worse than learned depth weighting.Future flow supports instance re-identification through predicted displacement, and learned depth weighting improves 3D feature lifting.
  • 5.3. Analysis: FIERY predicts the most accurate and diverse futures among M-Head, Bayesian Dropout, and Classical VAE probabilistic baselines.

6. Conclusion

FIERY predicts dynamic agents in bird’s-eye view from surround RGB videos while modeling future stochasticity with a variational distribution. It produces temporally consistent segmentations and motion, and represents diverse futures accurately.

  • FIERY is presented as a bird’s-eye-view prediction model for dynamic agents using surround RGB videos.
  • The end-to-end network models future stochasticity with a variational distribution.
  • FIERY predicts temporally consistent future instance segmentations and motion while modeling diverse futures accurately.

A.1. Comparison with published methods

The published-method comparison evaluates present-frame bird’s-eye-view semantic segmentation. Incorporating past observations improves predictions over using only the present frame.

  • Figure 5 compares FIERY’s present-frame bird’s-eye-view semantic segmentation predictions with previous published methods.
  • Past information produces better present-frame bird’s-eye-view segmentation predictions.This comparison is shown in Figure 6.

A.3. Probabilistic modelling

The probabilistic modelling analysis evaluates diverse future predictions using Generalised Energy Distance and compares alternative multimodal baselines. Cluster analysis links similar situations to similar learned states and diverse futures.

  • Generalised Energy Distance evaluates predicted futures against ground-truth futures and the diversity among predicted samples.The distance metric is d(x, y) = 1−VPQ(x, y).
  • With one ground-truth future, Generalised Energy Distance simplifies to 2E[d(ˆY, Y)] − E[d(ˆY, ˆY′)].
  • The comparison includes M-Head, Bayesian Dropout, and Classical VAE probabilistic baselines.M-Head outputs M different futures; Bayesian Dropout uses temporal and decoder dropout; Classical VAE samples latent codes from N(0, I_L).
  • Table 4 reports Generalised Energy Distance for 2.0s future prediction with M = 10 samples and states that FIERY predicts the most accurate and diverse futures.
  • Cluster analysis shows that similar scenarios map to similar states, supporting diverse and multimodal futures from deterministic training data.

A.5. Temporal horizon of future prediction

FIERY’s future-prediction performance is evaluated across temporal horizons and capture sizes. Performance appears to plateau beyond 6.0s as uncertainty increases and agents may become unobservable.

  • Performance seems to plateau beyond 6.0s into the future.
  • Longer horizons become increasingly difficult because future uncertainty grows and dynamic agents may not remain visible in past frames.
  • Figure 8 reports future Video Panoptic Quality from 1.0s to 8.0s across Short 30m × 30m and Long 100m × 100m capture sizes.
  • Cluster visualizations illustrate examples of learned spatio-temporal states and their predicted futures.

B. Model and Dataset

FIERY processes temporal surround-camera observations into bird’s-eye-view features, models future distributions, and decodes future instance segmentation and motion. The evaluation uses NuScenes and a constructed Lyft validation split.

  • Model: FIERY processes 18 images from three past observations using six cameras per timestep at 224 × 480 resolution.The model considers depths from 2.0m to 50.0m in 1.0m slices.
  • Training: The training objective uncertainty-weights segmentation, centerness, offset, and flow losses, with the probabilistic loss weighted by 100.The model contains 8.1M parameters and trains in one day on four Tesla V100 GPUs.
  • Model: The bird’s-eye-view encoder lifts image features into 200 × 200 spatial tensors and aligns past observations using ego-motion and a spatial transformer.EfficientNet-B4 produces image features that are projected into bird’s-eye view before temporal modeling.
  • Model: A 3D convolutional temporal model aggregates inputs, while present and future distributions use residual downsampling architectures with different channel inputs.The future distribution additionally receives future target representations, with 88 input channels versus 64 for the present distribution.
  • Model: Future prediction repeats a convolutional gated recurrent unit followed by three residual blocks, and the decoder predicts centerness, offsets, segmentation, and flow through multiple heads.Flow labels represent changes in instance-center positions between consecutive timesteps.
  • Dataset: Results are reported on the official NuScenes validation split and a randomly selected Lyft validation split containing 6,174 samples, compared with NuScenes’ 6,019.Lyft lacks an official validation set, so the paper constructs one with roughly the same number of samples as NuScenes.
Loading 2104.10490v3…