PaperlayerView exact revision on arXiv

Source-linked AI summary

MC-DeTra: Motion-Consistent Joint Object Detection and Socially-Aware Trajectory Forecasting in Bird's-Eye-View Images

Vladislav Diuzhev, Dmitry Yudin

arXiv:2609.11717v1cs.CVcs.RO

TL;DR

Dynamic-actor forecasting remains difficult in unified detection-and-forecasting systems, while DeTra lacks a public implementation. MC-DeTra reimplements DeTra and adds train-only auxiliary motion and occupancy supervision plus heading consistency; on Waymo, it improves dynamic forecasting while preserving detection and inference latency. The calibrated full model leads key dynamic displacement metrics, with occupancy contributing most strongly, although gains are modest and single-run uncertainty remains.

  • Problem

    Unified detection-and-forecasting models still face a difficult dynamic-actor forecasting problem, and DeTra has no public implementation.

  • Method

    MC-DeTra extends an openly released DeTra reimplementation with train-only Past Reconstruction, Occupancy Auxiliary, and heading-consistency objectives calibrated through shared-backbone gradient norms.

  • Results

    The calibrated full model achieves the best dynamic minFDE6, minADE6, and brier-minFDE6 while preserving detection, with controller A best on MR6.

  • Takeaways & Limitations

    Occupancy provides the largest auxiliary contribution, while the train-only mechanisms improve dynamic forecasting without adding inference latency.

  • Takeaways & Limitations

    Gains are modest, the trajectory–occupancy consistency term is disabled after being gradient-starved, and reported accuracy results are single-run ablations without statistical significance claims.

Abstract

from arXiv · show

Unified models for object detection and trajectory forecasting aim to merge perception and prediction for autonomous driving, refining actor trajectories directly over shared bird's-eye-view (BEV) images rasterized from LiDAR and high-definition maps. Their accuracy on dynamic, moving actors, however, remains the hardest part of the task, and the strongest such model, DeTra, has no public implementation. We contribute an openly released DeTra reimplementation with documented approximations, and on top of it MC-DeTra: a family of motion-consistency mechanisms that add supervision through two annotation-derived auxiliary signals -- each actor's observed past motion and the occupancy of the surrounding traffic that forms its social context -- and one inter-output consistency constraint that aligns an actor's predicted heading with its predicted direction of motion. Every proposed loss is train-only and inference-safe: it shapes the shared BEV representation during training and is removed at test time, adding no inference latency. On the Waymo Open Dataset, evaluated under a strict, detection-conditioned forecasting protocol, MC-DeTra improves dynamic, socially-situated trajectory forecasting while preserving or improving detection accuracy; a gradient-based loss-calibration analysis exposes how the auxiliary objectives compete at the shared backbone, and our ablation identifies which signals contribute most. We release code, configurations, and evaluation tooling at https://github.com/diuzhevVlad/MC-DeTra.

1 Introduction

MC-DeTra addresses the difficulty of forecasting dynamic actors by extending a unified DeTra-style detector-forecaster with train-only motion-consistency supervision. Its calibrated full model improves dynamic forecasting while preserving detection and adding no inference latency.

  • 1 Introduction: Unified detection and forecasting lets the model retain sensor context for prediction while avoiding the compounded errors of cascaded detection, tracking, and forecasting modules.DeTra refines trajectories whose first pose is the current detection and remaining poses are multimodal future waypoints over a shared BEV representation.
  • 1 Introduction: The paper targets dynamic-actor forecasting because static actors dominate urban LiDAR scenes and can mask the moving-actor shortfall most relevant to safety.A documented, openly released DeTra reimplementation and strict detection-conditioned evaluation protocol support reproducibility and extension.
  • 1 Introduction: The method adds Past Reconstruction, Occupancy Auxiliary, and heading consistency objectives to shape a shared BEV representation during training without affecting inference latency.The objectives use annotation-derived past motion and surrounding-traffic occupancy, plus consistency between predicted heading and motion direction.
  • 1 Introduction: MC-DeTra’s calibrated full model achieves the best dynamic minFDE6, minADE6, and brier-minFDE6, while controller A gives the best MR6 and detection is preserved.The train-only components add no inference latency, and all tested GPUs remain within the 2 Hz prediction budget.
  • 1 Introduction: The authors release code, configurations, and evaluation tooling for the DeTra reimplementation and MC-DeTra.The repository is available at https://github.com/diuzhevVlad/MC-DeTra.

2 Related Work

Prior work unifies perception and forecasting, models multimodal futures, and represents scene occupancy or maps, while MC-DeTra adds auxiliary occupancy supervision and gradient calibration to a joint detector-forecaster. Its mechanisms are additions absent from DeTra’s original supervision.

  • Joint detection and forecasting: DeTra is positioned as a strong LiDAR-based joint detector-forecaster that refines detection and future trajectories over a shared LiDAR-and-map representation.Related approaches span cascaded, structured-intermediate, camera-centric, vectorized, and sparse end-to-end systems.
  • Multimodal trajectory forecasting: Trajectory forecasting research addresses multimodal behavior through anchor-based, set-based, latent-variable, generative, and attention-based predictors.The supplied passage establishes the breadth of forecasting approaches but does not identify a single dominant method.
  • Occupancy forecasting and map encoders: MC-DeTra adds an auxiliary occupancy field to trajectory forecasting, using surrounding traffic layout rather than replacing trajectories with occupancy predictions.This grounds the shared backbone in where scene mass lies and how it moves.
  • Multi-task loss balancing: The method applies gradient-norm calibration at the shared LiDAR trunk to target each auxiliary term’s contribution relative to the forecast gradient.This adapts multi-task loss balancing to the joint detector-forecaster setting.
  • Multi-task loss balancing: Unlike DeTra, MC-DeTra decodes observed past motion, supervises an auxiliary occupancy objective, and constrains yaw toward motion direction.These mechanisms are additions rather than re-implementations of DeTra components.

3 Method: Motion-Consistent DeTra

MC-DeTra augments DeTra with train-only auxiliaries that shape a shared BEV representation using past motion, surrounding occupancy, and heading–motion consistency. Gradient-based calibration sets their influence while preserving inference behavior and exposing which signals help or hurt forecasting.

  • Base architecture: DeTra refines a trajectory volume whose first slot is the current detection and remaining slots are six multimodal future trajectories through shared BEV features and refinement transformers.The reimplementation uses N=600 actor queries, K=6 modes, and T=11 trajectory slots.
  • Auxiliary objectives: MC-DeTra adds Past Reconstruction, Occupancy Auxiliary, and Heading Consistency objectives to representations already computed by DeTra, removing them at inference.PR uses observed histories, OA models surrounding traffic occupancy and flow, and HC links predicted box yaw with predicted motion.
  • Past Reconstruction: PR reconstructs each actor’s deterministic observed history from the mode-averaged current query with a GRU-plus-MLP head and masked smooth-ℓ1 supervision.The loss is gated by matched true-positive actors and applies only to valid cached history slots; the head is training-only.
  • Occupancy Auxiliary: OA predicts occupancy and flow on the shared stride-4 BEV grid from rasterized boxes over past and future windows, grounding features in surrounding traffic.It uses focal loss for occupancy and masked smooth-ℓ1 loss for flow.
  • Heading Consistency: HC aligns predicted box yaw with the net displacement of the best predicted mode for matched actors moving more than τv=0.5 m.Because future waypoints are strongly constrained by the main forecast loss, the consistency is met mainly by rotating box yaw toward predicted motion.
  • Loss calibration: Gradient-norm analysis shows OA is the strongest useful auxiliary, HC is gradient-starved and harmful at high weight, and the hand-calibrated model outperforms the controller.The headline model keeps HC at weight 0.25; train-only auxiliaries preserve the plain reproduction’s latency and remain within the 2 Hz budget.

4 Experimental Setup

Experiments use a controlled Waymo Open Dataset protocol that isolates auxiliary-loss effects from detector strength and training variation.

  • Dataset and protocol: Every ablation row is fine-tuned from one base checkpoint with identical weights-only schedules, so rows differ only in enabled losses.Training uses effective batch size 8, cosine learning rate 10−4, 200-step warmup, gradient clipping at 5, and 10k steps.
  • Metrics: Detection follows the official Waymo vehicle BEV AP/APH protocol with Hungarian matching at IoU thresholds 0.3, 0.5, and 0.7.Forecasting uses vehicle-only, detection-conditioned evaluation at a fixed 80%-recall operating point.
  • Comparison strategy: Because forecasting is detection-conditioned and competing methods use different detectors, recalls, classes, and datasets, comparisons are restricted to controlled like-for-like deltas in one codebase.All configurations share one initializer and evaluation protocol.

5 Results

MC-DeTra improves detection-conditioned dynamic forecasting without a detection trade-off, with occupancy auxiliary supervision providing the strongest gradient contribution while heading consistency remains a small, metric-dependent adjustment.

  • Detection preservation: Detection remains neutral: every mechanism is within ±0.16 AP@0.7 of the no-auxiliary control, and the full model scores 71.11 versus 71.00.The approximately 1.2 AP@0.7 gain over the initializer comes from continued fine-tuning rather than auxiliaries.
  • Forecasting ablation: Dynamic minFDE6 improves by 2.4% over the initializer and 2.9% over plain fine-tuning with MC-DeTra, which also achieves the best dynamic minADE6, brier-minFDE6, and macro minFDE6.MC-DeTra uses PR/OA/HC weights 0.05/0.30/0.25; controller A gives the best dynamic and macro MR6.
  • Gradient-norm calibration: Heading consistency helps minADE6 and brier-minFDE6 at weight 0.25 but is neutral on minFDE6/MR6 and harmful when overweighted.The penalty is therefore metric-specific rather than uniformly beneficial.
  • Forecasting ablation: Across 135k jointly detected dynamic vehicles, MC-DeTra lowers minFDE6 for 54.6% and raises it for 45.4%, with mean shift −0.059 m.The >1 m tails are near-balanced at 12.2% versus 10.3%.
  • Gradient-norm calibration: Occupancy auxiliary contributes approximately 35% of the forecast gradient, compared with approximately 6% for past reconstruction and 0.02% for heading consistency.The gradient analysis identifies occupancy as the dominant auxiliary signal and heading consistency as gradient-starved.
  • Qualitative results: Figure 4 provides an illustrative single-actor comparison, whereas the population-level quantitative result is reported in Table 3.The paper distinguishes qualitative visualization from aggregate evaluation.

6 Discussion

The discussion interprets gradient competition in the shared backbone and emphasizes that gains are modest, dynamic-actor-focused, and still subject to important scope and uncertainty limits.

  • 6 Discussion: Figure 3 qualitatively shows past reconstruction, heading–motion alignment, and six distinct forecast hypotheses for a moving actor.The forecast fan uses mode width to indicate mode probability.
  • 6 Discussion: MC-DeTra’s gains are modest and concentrated on dynamic actors, with the full model presented as the best calibrated configuration rather than a decisive win.Per-actor improvements are broadly distributed with near-balanced tails, supporting a reliable small improvement rather than a large one.
  • 6 Discussion: The study remains limited by a vehicle-only scope, single-run ablations without statistical significance claims, and validation-based auxiliary-weight selection that may be mildly optimistic.The authors identify multi-seed training, paired bootstrap intervals, and held-out tuning as preferable follow-ups.
  • 6 Discussion: Future work includes effective scene-level trajectory–occupancy consistency, extension to pedestrians and cyclists, and testing gradient-norm control with more numerous or competitive auxiliaries.

7 Conclusion

The conclusion presents MC-DeTra as an openly released DeTra reimplementation augmented with train-only motion-consistency regularizers. In controlled ablations, the calibrated model leads key dynamic forecasting metrics while preserving detection, and the accompanying example illustrates improved endpoint reach.

  • 7 Conclusion: MC-DeTra combines Past Reconstruction, Occupancy Auxiliary, and Heading Consistency as train-only, inference-safe regularizers with gradient-norm calibration of shared-backbone competition.The method adds no inference-time components.
  • 7 Conclusion: In a selected example, DeTra* stops short of the ground-truth endpoint whereas MC-DeTra reaches it, illustrating but not establishing the population-level result.The figure shows six predicted futures for one actor detected by both models.
  • 7 Conclusion: The calibrated full model attains the best dynamic minFDE6, minADE6, and brier-minFDE6, while controller A achieves the best MR6 and detection remains preserved.The reported comparison comes from a controlled ablation from one initializer.
  • 7 Conclusion: Gradient norms attribute the gain mainly to occupancy, with modest help from past reconstruction and a small, metric-dependent adjustment from heading consistency.
  • 7 Conclusion: The authors release code, configurations, and evaluation tooling for future work.
Loading 2609.11717v1…