Forecasting from LiDAR via Future Object Detection

1. Introduction

• Existing end-to-end approaches tend to predict one future per object, while multi-future forecasting has often required specialized methods such as multiple-choice loss or generative models.The paper motivates its approach against this difficulty in representing future uncertainty.Source: §1 · p. 1
• FutureDet detects objects at future timesteps from accumulated raw LiDAR scans and backcasts their trajectories to the current frame.Many-to-one matching of future detections to current detections represents multiple plausible futures without object tracks or HD maps as inputs.Source: §1 · p. 2
• ADE/FDE-style metrics can be gamed by ranking stationary objects highly and do not penalize forecasts that are not tied to objects.The paper argues that forecasting and detection should be evaluated jointly.Source: §1 · p. 2
• The contributions are FutureDet, multi-future prediction per current detection, joint detection-and-forecasting AP metrics, and evaluation on nuScenes showing gains over end-to-end and multi-stage methods.The proposed metrics generalize object-detection AP so a forecast counts as correct only when its object is correctly detected and forecasted.Source: §1 · p. 2

2. Related Work

• LiDAR detection methods use point-cloud encoders and proposal or point-based detection, while tracking associates detections with filters, descriptors, or offsets.Accumulating multiple scans can improve detection, and tracking methods resolve ambiguous associations over time.Source: §2 · p. 2
• Many forecasting benchmarks assume perfect detection and tracking outputs, often using past trajectories and HD maps as inputs.Vision-based and automotive forecasting methods commonly estimate future positions from supplied trajectories.Source: §2 · pp. 2–3
• Earlier sensor-data approaches jointly detect, track, and forecast from LiDAR, but do not reason about multiple future trajectories.Another approach forecasts LiDAR scans before applying off-the-shelf detectors.Source: §2 · p. 3
• The paper proposes anchoring multiple possible future detections to current detections and generalizes AP for joint detection and forecasting.It reports improved forecasting accuracy over the earlier end-to-end approaches discussed in this section.Source: §2 · p. 3

3. Rethinking Forecasting Evaluation

• The task is to estimate current object locations and their trajectories in future, unobserved scans from a sequence of past LiDAR observations.Earlier formulations instead estimate future positions from ground-truth tracks, sometimes with HD maps and ego-vehicle positions.Source: §3 · p. 3, §3.1 · p. 3

3.1. Preliminaries

• ADE/FDE comparisons assume methods share the same true positives, an assumption that fails when end-to-end methods produce different detections.Recall-based ADE/FDE can favor trivial stationary-object forecasts and do not penalize false forecasts or missed forecasts.Source: §3.1 · pp. 3–4

3.2. Average Precision is All You Need

• A joint forecast true positive requires a match both at the current timestamp and at the future final timestep.The evaluation uses center-distance thresholds averaged across multiple thresholds at both timestamps, and considers all detections.Source: §3.2 · p. 4
• Forecasting mAP averages AP across static, linearly moving, and non-linearly moving car subclasses to reduce susceptibility to trivial forecasters.The metric assumes ground-truth subclass labels, though forecasts need not output subclass labels.Source: §3.2 · p. 4
• For multiple futures, evaluation takes each detection's top-K ranked forecasts and scores the best match among them.This protocol does not penalize a model for proposing plausible alternatives anchored to one detection.Source: §3.2 · p. 4

4. Forecasting as Future Object Detection

• FutureDet detects objects in future LiDAR scans and estimates offsets back to the reference frame, rather than forecasting offsets forward from current-frame detections.The method uses an accumulated LiDAR sequence; its detection network predicts objects at future timesteps and backcasts them.Source: §4 · pp. 4–5
• Velocity offset vectors for future detections estimate trajectories across the intervening frames from the current timestep to each detection's future timestep.Source: §4 · p. 4
• Each future detection is backcast and matched to its best previous-timestep detection, allowing multiple future forecasts to attach to one current detection.Source: §4 · p. 5, Figure 2 · p. 5
• Forecasts attached to one detection are ranked using the last detection's confidence; performance increases slightly from K = 1 to K = 5.The authors note that better ranking strategies may further improve FutureDet.Source: §4 · p. 5
• Copy-paste ground-truth sampling increases trajectory diversity and improves linear and non-linear forecasting performance.The implementation trains CenterPoint to detect objects in future scans, treating them as additional object classes with regression offsets.Source: §4 · p. 5
• Gaussian BEV heatmaps provide a continuous likelihood field for future object locations that naturally represents uncertainty and multiple possible futures.CenterPoint finds detections as local maxima in its heatmaps using non-maximum suppression.Source: §4 · pp. 5–6

5. Experimental Evaluation

• The experiments use nuScenes car forecasting up to 3 seconds ahead and evaluate on the official validation split.The analysis focuses on LiDAR data and does not use ground-truth trajectories or HD maps as input.Source: §5 · p. 6

5.1. Metric Breakdown Analysis

• The constant-position model performs better than FaF* on static cars but obtains 0 APf on moving cars; FaF* reaches 7.5 AP non-linear.Trajectron++ outperforms the constant-position model for linearly moving objects at 8.1 AP.Source: §5.1 · p. 6

5.2. Ablation and Comparison to State-of-the-Art

• The comparison includes CenterPoint constant-velocity extrapolation, FaF* velocity integration, and Trajectron++ forecasting from tracked trajectories.FaF* is re-implemented with a CenterPoint backbone for an apples-to-apples comparison.Source: §5.2 · pp. 6–7

5.1. Metric Breakdown Analysis

• 31.5 mAPf for FaF* versus 22.1 mAPf for the constant-position baseline, while ADE/FDE metrics favor the trivial baseline.The reported metric analysis finds that only forecasting mAPf favors state-of-the-art models over constant position prediction.Source: §5.1 · p. 6, Table 1 · p. 7

5.2. Ablation and Comparison to State-of-the-Art

• At K = 5, FutureDet improves non-linear object forecasting accuracy by 4% over FaF*; FutureDet with a PointPillars backbone performs best overall.FutureDet also improves on the other baselines in non-linear object APf and motion category-averaged mAPf.Source: Table 2 · p. 7, §5.2 · p. 8
• FutureDet detects objects in future frames and backcasts them to the reference frame, naturally representing multiple possible futures.Qualitative forecasts also indicate that it learns road geometry without map information, as shown by curved trajectories.Source: §5.2 · p. 7
• Bundling multiple trajectories into one prediction slightly reduces recall at K = 1, while K = 5 improves moving-object detection and forecasting; road masks change results very little.The authors note that further exploration is needed to determine how best to fuse map information.Source: §5.2 · p. 8

6. Conclusion

• FutureDet presents end-to-end LiDAR forecasting as forecasting-by-detection, with multiple futures and joint detection-and-forecasting metrics based on average precision.The paper reports that its analysis finds a constant-velocity model to be a surprisingly strong baseline for future forecasting work.Source: §6 · p. 8
• Without explicitly enforcing trajectory diversity, FutureDet’s multiple forecasts can cluster, leaving diverse multimodal prediction an open challenge.The paper describes FutureDet as the first raw-sensor end-to-end method capable of multi-future predictions.Source: §6 · p. 8

A. Examining FutureDet’s Predictions

• FutureDet’s raw output requires further post-processing before it can be used in downstream tasks.The paper notes that making multiple bets about object positions may induce more false positives.Source: A. Examining FutureDet’s Predictions · p. 11
• Standard forecasting assumes perfect input trajectories, which depend on perfect tracks and detections that are nearly impossible to obtain in practice.The paper identifies handling false-positive detections and forecasts as a challenge for raw-sensor forecasting.Source: A. Examining FutureDet’s Predictions · p. 11
• Making multiple bets about object positions may induce more false positives.Source: A. Examining FutureDet’s Predictions · p. 11

B. Evaluating Pedestrian Forecasting

• Pedestrian matching uses tighter center-distance thresholds because pedestrians typically move less over the three-second forecasting horizon.Current-frame thresholds are {0.125, 0.25, 0.5, 1}m and final-timestep thresholds are {0.25, 0.5, 1, 2}m.Source: B. Evaluating Pedestrian Forecasting · p. 11
• Pedestrian motion is harder to forecast than car motion because pedestrians are more dynamic and often travel in crowds.The figure description says forecasts are often all linear and similar to one another.Source: B. Evaluating Pedestrian Forecasting · p. 11, Figure 5 · p. 11
• FutureDet improves over FaF* by 0.5 mAPf in joint pedestrian detection and forecasting on nuScenes.FaF* and the constant-velocity baseline beat Trajectron++ by 14.4% and 16.6% mAPf, respectively.Source: Table 3 · p. 12

C. FutureDet Architecture

• The implementation-details section focuses on the detector head architecture and the sampling strategy for improving nonlinear trajectory forecasting.Source: C. FutureDet Architecture · p. 12
• FutureDet reuses CenterPoint but adds a shallow network that transforms current features into future features for detecting temporally offset classes.The shared representation in CenterPoint captures object location but does not provide a robust representation of forecasted features.Source: C. FutureDet Architecture · p. 12
• Because 60% of nuScenes car trajectories are static, FutureDet copy-pastes linear and nonlinear trajectories to address training-data imbalance.Augmentation is sampled at the trajectory level so augmented trajectories remain consistent across detection heads.Source: C. FutureDet Architecture · p. 12

• Subclass AP applies standard precision–recall metrics when predictions and ground truth have subclass labels, but predictions here lack those labels.Source: D. Computing Motion Subclass AP · p. 12
• Matched true positives inherit the ground-truth subclass, while evaluation derives labels for unmatched false positives using the ground-truth labeling logic; the approach also applies to detection attributes.Source: D. Computing Motion Subclass AP · p. 12
• Trajectories are static when first-to-last box IoU is positive, linear when the final box overlaps a constant-velocity target, and nonlinear otherwise.Source: D. Computing Motion Subclass AP · pp. 12–13

• Autonomous agents are relevant to automating tasks considered unsafe, including those associated with many traffic accidents.Source: E. Broader Impact · p. 13
• Safe autonomous-vehicle navigation requires predicting the motion of surrounding agents.Source: E. Broader Impact · p. 13
• LiDAR can support understanding of world geometry and dynamics, while evaluating moving and static car trajectories is important for safe embodied robotics.Source: E. Broader Impact · p. 13