PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Point Cloud Forecasting enables Scalable Autonomous Driving

Zetong Yang, Li Chen, Yanan Sun, Hongyang Li

arXiv:2312.17655v1cs.CV

TL;DR

Visual autonomous-driving pre-training remains underexplored despite the need to combine semantics, 3D geometry, and temporal dynamics for perception, prediction, and planning. ViDAR pre-trains visual encoders by predicting future point clouds from historical images, and the reported results show gains across downstream tasks. The authors identify the limited scale of their nuScenes pre-training data as a limitation.

  • Problem

    Visual autonomous-driving pre-training is underexplored, while joint perception, prediction, and planning require features that capture semantics, 3D geometry, and temporal dynamics.

    Source: §1 · p. 2

  • Method

    ViDAR pre-trains visual encoders by forecasting future point clouds from historical visual images, using a History Encoder, Latent Rendering operator, and Future Decoder.

    Source: §1 · p. 2

  • Results

    ViDAR improves UniAD across perception, prediction, and planning, with ∼10% error reduction in minADE.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Visual point cloud forecasting produces features with geometric and temporal hints that benefit perception, tracking, and planning.

    Source: §1 · p. 2

    Pre-training is conducted mainly on nuScenes Image-LiDAR sequences, whose data scale remains limited.

    Source: §5 · p. 9

Abstract

from arXiv · show

In contrast to extensive studies on general vision, pre-training for scalable visual autonomous driving remains seldom explored. Visual autonomous driving applications require features encompassing semantics, 3D geometry, and temporal information simultaneously for joint perception, prediction, and planning, posing dramatic challenges for pre-training. To resolve this, we bring up a new pre-training task termed as visual point cloud forecasting - predicting future point clouds from historical visual input. The key merit of this task captures the synergic learning of semantics, 3D structures, and temporal dynamics. Hence it shows superiority in various downstream tasks. To cope with this new problem, we present ViDAR, a general model to pre-train downstream visual encoders. It first extracts historical embeddings by the encoder. These representations are then transformed to 3D geometric space via a novel Latent Rendering operator for future point cloud prediction. Experiments show significant gain in downstream tasks, e.g., 3.1% NDS on 3D detection, ~10% error reduction on motion forecasting, and ~15% less collision rate on planning.

1. Introduction

Visual autonomous-driving pre-training remains underexplored despite the need to learn semantics, 3D geometry, and temporal dynamics for perception, prediction, and planning. This section introduces visual point cloud forecasting and ViDAR as a pre-training approach, reporting gains in forecasting and downstream tasks.

1. Introduction

  • Visual driving models often depend on precise 3D annotations that are difficult to scale, while existing pre-training struggles to model multi-view geometry and temporal dynamics together.Depth estimation is limited in multi-view geometry, and multi-view scene rendering lacks temporal modeling; pre-training can use readily available data to improve downstream performance when labels are limited.Source: §1 · pp. 1–2
  • Visual point cloud forecasting predicts future point clouds from historical visual images as a scalable pre-training task for end-to-end autonomous driving.The task is intended to exploit semantics, 3D geometry, and temporal dynamics in raw Image-LiDAR sequences.Source: §1 · p. 2
  • Predicting the future from history supervises scene flow and object motion, while point-cloud reconstruction from images supervises multi-view geometry and semantics.The resulting features contain geometric and temporal hints intended to benefit perception, tracking, and planning.Source: §1 · p. 2
  • ViDAR uses a History Encoder, Latent Rendering operator, and Future Decoder to pre-train visual BEV encoders from historical sequences.Latent Rendering models a 3D geometric latent space and bridges encoder and decoder; an autoregressive transformer decoder predicts future point clouds at arbitrary timestamps.Source: §1 · p. 2
  • ViDAR reduces 1s future point-cloud estimation Chamfer Distance by ∼33% and improves UniAD across perception, prediction, and planning.With Image-LiDAR sequences only and the same data scale, it improves 3D detection by 1.1% mAP and semantic occupancy prediction by 2.5% mIoU; with 3D detection pre-training, it adds 4.3% mAP and 4.6% mIoU.Source: §1 · p. 2

2. Related Work

Prior visual pre-training approaches and point-cloud forecasting methods address different parts of the autonomous-driving problem. The paper positions visual point cloud forecasting as a visual-input pre-training paradigm that combines semantic, geometric, and temporal modeling for downstream tasks.

Pre-training for Visual Autonomous Driving

  • General-vision pre-training includes contrastive methods that learn from positive and negative pairs and masked-signal methods that recover discarded signals to capture global semantics.The paper contrasts this extensive body of work with the underexplored area of visual autonomous-driving pre-training.Source: §2 · pp. 2–3
  • Visual-driving pre-training must combine semantic understanding, 3D structure, and temporal modeling, but existing vision methods mainly model semantics, Image-LiDAR methods struggle with temporal modeling, and supervised strategies are not scalable.The paper proposes visual point cloud forecasting as a uniform process for these dimensions that is easily scaled up.Source: §2 · p. 3

Point Cloud Forecasting

  • Earlier point-cloud forecasting predicts future clouds from past point clouds using range images, 3D convolutions, or LSTMs, while later methods use 4D occupancy prediction or differentiable ray-casting.This work instead uses past images to predict future point clouds and presents the task as a visual-driving pre-training paradigm for downstream applications.Source: §2 · p. 3

3. Methodology

ViDAR encodes visual sequences into BEV features, renders them into a geometric latent space, and autoregressively predicts future BEV features and point clouds. Latent Rendering addresses ray-shaped features by making grid features discriminative, while the Future Decoder conditions predictions on ego-motion and prior features.

3.1. Overview

  • ViDAR encodes visual sequences as BEV features, transforms them into geometric embeddings, and autoregressively predicts future BEV features for occupancy and point-cloud prediction.Point clouds are recovered from predicted occupancy by casting rays and selecting the maximum-occupancy distance along each ray.Source: §3.1 · p. 3

3.2. Latent Rendering

  • Table 1 reports that direct differentiable ray-casting pretraining drops downstream detection performance, while Latent Rendering significantly improves it.The comparison includes an N/A baseline without forecasting pretraining.Source: §3.2 · p. 3, Table 1 · p. 4
  • Latent Rendering weights shared ray-wise features by each grid’s conditional probability, producing more discriminative geometric BEV features.The operator is introduced to address ray-shaped features, in which grids along the same ray tend to have similar features.Source: §3.2 · p. 4
  • Multi-group Latent Rendering runs renderers in parallel across feature channels, preserving diverse ray-wise information for downstream performance.Conditional probabilities account for prior grids along a ray, suppressing responses around a raised grid response.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Future Decoder

  • The Future Decoder uses ego-motion and previous BEV features in an autoregressive transformer to predict future BEV features.It aligns future query reference coordinates to the previous frame using ego-motion; ray-wise cross-entropy supervises future point locations against other sampled waypoints.Source: §3.3 · p. 5

4. Experiments

Experiments on nuScenes evaluate ViDAR for point-cloud forecasting, downstream driving tasks, and annotation efficiency. Results show gains across forecasting, perception, prediction, and planning, while ablations examine limited supervision and Latent Rendering.

4.1. Setup

  • Experiments use nuScenes and evaluate BEVFormer, OccNet, and UniAD for detection, occupancy prediction, and unified driving tasks.The default ViDAR history encoder is BEVFormer-base; training uses five historical frames to predict three future seconds.Source: §4.1 · pp. 5–6

4.2. Main Results

  • 42.6% mAP vs. 41.5% mAP and 29.57% mIoU vs. 26.98% mIoU: ViDAR outperforms supervised 3D-detection pre-training.The comparisons concern 3D detection and semantic occupancy prediction, respectively.Source: Table 3 · p. 6, §4.2 · pp. 6–7, Table 4 · p. 7
  • ~33% forecasting-error reduction over 4D-Occ for future 1s predictions with 1s history, using visual inputs only.The evaluation uses Chamfer Distance.Source: §4.2 · p. 6, Table 2 · p. 6
  • ~10% error reduction in minADE and 3.5% improvement in EPA for UniAD motion forecasting with ViDAR pre-training.Table 7 reports motion forecasting comparisons.Source: Table 7 · p. 7, §4.2 · p. 7
  • UniAD occupancy prediction improves by 2.4% IoU and 2.7% VPQ nearby, and 2.0% IoU and 2.5% VPQ in distant areas.With ViDAR, UniAD also outperforms BEVerse in all areas.Source: Table 8 · p. 7, §4.2 · p. 7
  • ~15% lower average collision rate within 3 seconds and 0.21m lower average planning displacement error for UniAD.ViDAR also enables UniAD to outperform VAD on nuScenes open-loop planning evaluation.Source: Table 9 · p. 7, §4.2 · p. 7
  • ViDAR substantially improves UniAD across detection, tracking, mapping, motion and future-occupancy prediction, and planning.The joint results are reported in Table 10.Source: Table 10 · p. 8, §4.2 · p. 8

4.3. Ablative Study

  • With half the supervised samples, ViDAR-pre-trained BEVFormer reaches 39.4% mAP versus 37.7% under full supervision; gains grow as supervision falls.The reported mAP improvements are 4.9%, 6.5%, 6.7%, and 7.3% for full, half, quarter, and 1/8th subsets.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The paper introduces visual point cloud forecasting as a pre-training task and presents ViDAR, including a Latent Rendering operator, to pre-train visual BEV encoders. The authors note that their pre-training data scale remains limited.

5. Conclusion

  • Visual point cloud forecasting predicts future point clouds from historical visual images as a pre-training task for end-to-end autonomous driving.The paper presents ViDAR as a general model for pre-training visual BEV encoders and designs Latent Rendering to address ray-shaped features.Source: §5 · p. 8
  • Visual point cloud forecasting enables scalable autonomous driving.Source: §5 · p. 9
  • Pre-training is mainly conducted on nuScenes Image-LiDAR sequences, whose data scale remains limited.The authors identify scaling up the data and studying diverse datasets as future work.Source: §5 · p. 9

A. Discussions

The discussion frames ViDAR as a cross-modal world model, explains its point-cloud prediction target, and identifies potential applications and research directions.

A. Discussions

  • ViDAR predicts future point clouds from historical visual sequences, unlike discussed autonomous-driving world models that use the same modality for inputs and outputs.It can generate different futures conditioned on different future ego-vehicle motions.Source: A. Discussions · p. 13
  • Point clouds are chosen over images because they depict 3D structure, object positions, and geometric properties precisely for 3D tasks.The paper connects this target choice to extracting 3D geometry from visual inputs for downstream models.Source: A. Discussions · p. 13
  • The authors propose ViDAR as a possible simulator for model-based reinforcement learning and suggest exploring diverse Image-LiDAR datasets and combinations with other single-modality world models.They report demonstrating ViDAR as pre-training for downstream end-to-end autonomous driving models.Source: A. Discussions · p. 13

B. Implementation Details

Latent Rendering converts BEV embeddings into geometric feature responses through probability estimation, ray-based conditional probabilities, and ray-wise feature aggregation and weighting.

B. Implementation Details

  • A projection layer estimates independent probability maps, with G denoting the number of Latent Rendering groups.The implementation sets G to 16 and λ to 1; multi-group rendering divides Fbev along channels and applies rendering to each part.Source: B. Implementation Details · pp. 13, 15
  • For each target BEV grid, the method casts a ray from the feature-map origin and samples prior waypoints at uniform λ intervals.The direction is the normalized vector from the origin to the target grid; sampled waypoints lie closer to the origin than the target.Source: B. Implementation Details · pp. 13–14
  • The conditional probability of a target BEV grid is computed from independent probabilities at its prior waypoints.The probability at a position is obtained by bilinear interpolation from p.Source: B. Implementation Details · p. 14
  • Feature expectation aggregates BEV embeddings along a ray into shared ray-wise features, which are multiplied by the conditional probability map to produce geometric feature responses.The feature grids indexed for aggregation lie in the same direction as the target grid.Source: B. Implementation Details · p. 14

C.1. Structure of Latent Rendering

The ablation examines the conditional probability and feature expectation functions in Latent Rendering, comparing downstream 3D detection performance against a simple baseline.

C.1. Structure of Latent Rendering

  • The ablation uses BEVFormer-small with ImageNet-cls pre-training as its baseline and evaluates downstream 3D detection.The simple-baseline pipeline comprises the History Encoder, Future Decoder, and differentiable ray-casting.Source: §C.1 · p. 15
  • For future point-cloud prediction, directly multiplying BEV embeddings Fbev by the aggregated conditional probability map ˆp alleviates ray-shaped features and improves NDS by 7.14% versus the simple baseline.Source: §C.1 · p. 15
  • Adding feature expectation to conditional probability further improves downstream performance by integrating the ray-wise feature.Source: §C.1 · p. 15

C.2. Future Predictions

The ablation tests whether future point-cloud supervision benefits temporal modeling in downstream tracking. Tracking performance improves consistently as the number of future supervision frames increases.

C.2. Future Predictions

  • The tracking ablation uses the first stage of UniAD and varies future point-cloud supervision during ViDAR pre-training.The 0-future setting reconstructs only the current-frame LiDAR point cloud.Source: §C.2 · p. 15
  • The authors use multi-object tracking to assess temporal modeling because it requires associating 3D objects across frames.Source: §C.2 · p. 15
  • Tracking performance improves consistently with more future frames used for ViDAR supervision.The comparison includes 0, 1, 3, and 6 future frames.Source: §C.2 · p. 15, Table 13 · p. 15

C.3. Pre-train Structure

The component ablation finds that ViDAR pre-training benefits the view-transformation component most. It also reports compatibility with 2D image pre-training and advanced image backbones.

C.3. Pre-train Structure

  • ViDAR pre-training primarily improves view transformation in BEVFormer-small, whose baseline is 44.11% NDS.The view-transformation component extracts BEV features from perspective multi-view image features.Source: §C.3 · p. 15, Table 14 · p. 15
  • The component ablation loads different sets of pre-trained parameters during downstream fine-tuning to test which BEV encoder components benefit.Source: §C.3 · p. 15
  • The authors report that ViDAR is compatible with 2D image pre-training techniques and that combining them with advanced image backbones yields consistent improvements.Source: §C.3 · p. 16

D. Qualitative Results

The qualitative results show that Latent Rendering produces BEV features that capture 3D geometry from multi-view visual sequences. Compared with differentiable ray-casting, it also avoids ray-shaped features and retains informative geometric responses.

D.1. Latent Rendering

  • Latent Rendering features from multi-view visual sequences represent the underlying 3D geometry in the latent space.The visualization pairs ground-truth point clouds with the corresponding BEV features.Source: §D.1 · p. 16
  • Compared with differentiable ray-casting, ViDAR mitigates ray-shaped features and preserves geometric information in BEV grid responses.The comparison presents ground-truth point clouds, ray-casting features, and ViDAR features.Source: §D.1 · p. 16
  • The resulting BEV features are described as discriminative and beneficial for downstream fine-tuning.Source: §D.1 · p. 16

D.2. Visual Point Cloud Forecasting

ViDAR forecasts future point clouds from historical visual sequences, capturing relative motion and adapting predictions to specified future ego motions. Qualitative end-to-end results also show geometric feature responses and more precise predicted trajectories for moving objects with ViDAR pre-training.

D.2. Visual Point Cloud Forecasting

  • From 1 second of historical visual input, ViDAR predicts point clouds over 3 seconds, capturing relative motion between the ego vehicle and parked or moving objects.The examples also show road-map estimation during a right turn; visualizations use ego coordinates.Source: §D.2 · p. 16, Figure 10 · p. 19
  • ViDAR can simulate different future point clouds conditioned on specific ego-vehicle motions, including turning left, going straight, or turning right.The paper describes these simulations as a potential visual world model for autonomous driving.Source: §D.2 · p. 16, Figure 11 · p. 20

D.3. End-to-end Autonomous Driving

  • UniAD with ViDAR pre-training produces more precise future trajectories for moving objects, supporting the planning process in safety-critical end-to-end driving scenarios.Source: §D.3 · p. 16, Figure 12 · p. 21
  • Latent Rendering produces BEV features that capture 3D geometry and mitigates ray-shaped features relative to differentiable ray-casting.The visualizations compare features with ground-truth point clouds and show the reduced ray-shaped responses.Source: Figure 8 · p. 17, Figure 9 · p. 18

Related papers

Loading 2312.17655v1…