PaperlayerView exact revision on arXiv

Source-linked AI summary

Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, Wei Zhan

arXiv:2210.02443v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Camera-only 3D detectors have limited temporal history and coarse matching resolution, restricting temporal stereo depth estimation. The paper analyzes localization potential and proposes SOLOFusion, which combines long-term low-resolution fusion with short-term high-resolution fusion. It reports first place on nuScenes test and gains of 5.2% mAP and 3.7% NDS over previous best art on validation.

  • Problem

    Camera-only 3D detection methods use limited history and low-resolution matching, while depth estimation remains a central bottleneck.

  • Method

    SOLOFusion combines long-term low-resolution temporal stereo with short-term high-resolution temporal stereo, guided by localization-potential analysis.

  • Results

    SOLOFusion achieves first place on the nuScenes test set and outperforms previous best art by 5.2% mAP and 3.7% NDS on validation.

  • Takeaways & Limitations

    Long-term and short-term temporal fusion are highly complementary for efficient temporal multi-view 3D detection.

  • Takeaways & Limitations

    The derivation excludes transformations involving the vertical Y axis and focuses on rotation in the XZ plane.

Abstract

from arXiv · show

While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception. Observing that existing works' fusion of multi-frame images are instances of temporal stereo matching, we find that performance is hindered by the interplay between 1) the low granularity of matching resolution and 2) the sub-optimal multi-view setup produced by limited history usage. Our theoretical and empirical analysis demonstrates that the optimal temporal difference between views varies significantly for different pixels and depths, making it necessary to fuse many timesteps over long-term history. Building on our investigation, we propose to generate a cost volume from a long history of image observations, compensating for the coarse but efficient matching resolution with a more optimal multi-view matching setup. Further, we augment the per-frame monocular depth predictions used for long-term, coarse matching with short-term, fine-grained matching and find that long and short term temporal fusion are highly complementary. While maintaining high efficiency, our framework sets new state-of-the-art on nuScenes, achieving first place on the test set and outperforming previous best art by 5.2% mAP and 3.7% NDS on the validation set. Code will be released $\href{https://github.com/Divadi/SOLOFusion}{here.}$

1 INTRODUCTION

The paper argues that camera-only 3D detection is constrained by short temporal history and coarse feature resolution, which limit temporal stereo depth estimation. It introduces SOLOFusion, combining long-term coarse matching with short-term fine-grained matching, and reports strong nuScenes gains.

  • Motivation: Short-history camera-only methods use only a few frames in a 2–3 second window, leaving long-term observations underused for depth estimation.The paper identifies depth estimation as the main bottleneck in camera-only detection.
  • Temporal Stereo View: Temporal feature aggregation in existing methods can be viewed as temporal stereo matching over candidate 3D locations.The correct depth hypothesis is easier to identify when source-view projections are more separated.
  • Analysis: The optimal rotation and temporal difference vary across pixels, depths, cameras, and ego-motion, requiring many timesteps to maximize localization potential.The analysis also finds that lower image feature resolution further reduces localization potential.
  • Method: SOLOFusion uses a 16-frame BEV cost volume for efficient long-term fusion and adds short-term fusion with higher-resolution features.The short-term module replaces monocular depth priors in the long-term volume with a two-view depth prior.
  • Contributions: SOLOFusion balances short-term high-resolution and long-term low-resolution temporal stereo for efficient depth estimation.The design explicitly balances spatial resolution against temporal difference.
  • Results: 5.2% mAP and 3.7% NDS improvements over previous best art are reported on the nuScenes validation set, alongside first place on the test set.Figure 2 also reports 9.4% mAP and 12.9% mATE improvements over non-temporal baselines.

2 RELATED WORK

Related work spans monocular, multi-camera, temporal detection, and multi-view stereo methods. The paper positions its contribution around overcoming the limited temporal history and low-resolution fusion used by prior temporal detectors.

  • Single-View 3D Detection: Single-view methods predict 3D boxes from images using 2D CNNs, CAD models, keypoints, or disentangled 3D parameters.
  • Multi-Camera 3D Detection: Multi-camera methods commonly predict depth-bin distributions and construct point clouds or BEV representations for detection.Other approaches use query-based 3D prediction frameworks.
  • Temporal 3D Detection: Temporal extensions align or sample features from multiple frames, but most use low-resolution features and limited history.These design choices are associated with limited improvement from temporal fusion.
  • Multi-View Stereo: Multi-view stereo methods reconstruct depth, meshes, or point clouds using depth maps or 3D volumes.Several methods scatter pixel features into 3D grids to estimate occupancy probabilities.
  • Multi-View Stereo: Table 1 distinguishes methods by sampled-feature weighting, BEV temporal stereo, and whether temporal aggregation is explicit or carried through running features.
  • Paper Positioning: SOLOFusion analyzes the connection between temporal detection and multi-view stereo, then combines short-term and long-term fusion.The proposed short-term fusion is described as more efficient and extensible.

3 UNIFIED STEREO TEMPORAL FORMULATION FOR CAMERA-ONLY 3D DETECTION

The unified formulation treats temporal camera-only 3D detection as temporal stereo matching. It organizes methods around candidate locations, sampling, resolution, aggregation, timesteps, and candidate processing, then identifies limited history and low feature resolution as key constraints.

  • 3.1 COMPONENTS OF A UNIFIED FORMULATION: Temporal camera-only detectors share a core formulation in which image features from multiple timesteps are matched at candidate 3D locations.
  • 3.1 COMPONENTS OF A UNIFIED FORMULATION: The formulation separates candidate locations, sampling operation, sampling resolution, temporal aggregation, timesteps used, and candidate-location processing.
  • 3.1 COMPONENTS OF A UNIFIED FORMULATION: Table 1 summarizes temporal methods using these formulation components, with detailed justifications deferred to the appendix.
  • 3.2 CONNECTING MULTI-VIEW STEREO AND TEMPORAL 3D DETECTION: Multi-view stereo estimates whether a candidate location is occupied, while temporal 3D detection estimates whether a particular object occupies it.Both use multiple 2D views of candidate 3D locations to determine whether something of interest is present.
  • 3.2 CONNECTING MULTI-VIEW STEREO AND TEMPORAL 3D DETECTION: LSS-based methods use dense candidate locations, whereas query-based methods use sparse locations near likely objects.Their supervision and refinement procedures differ accordingly.
  • 3.3 ANALYSIS OF COMPONENTS: Most detection methods fuse few frames over short windows, and BEVFormer gains no further improvement beyond three training timesteps.This indicates that its sequential fusion framework hinders longer-term temporal fusion.
  • 3.3 ANALYSIS OF COMPONENTS: Most detection methods use 1/16-resolution feature maps for temporal stereo, while multi-view stereo methods commonly use 1/4 resolution.UVTR is noted as an exception among detection methods.

4 THEORETICAL ANALYSIS

The analysis treats temporal camera-only 3D detection as multi-view stereo and studies how motion, depth, pixel location, and feature resolution affect localization potential. It finds that many timesteps are needed because the best temporal difference varies across candidate locations and depths.

  • 4.1 DERIVATION OF LOCALIZATION POTENTIAL: Temporal camera-only 3D detection methods can be viewed as multi-view stereo matching over candidate 3D locations.They aggregate image features from multiple timesteps to determine occupancy or object existence at those locations.
  • 4.2 EFFECT OF TEMPORAL DIFFERENCE ON LOCALIZATION POTENTIAL: Closer depths generally prefer smaller temporal differences, whereas farther depths prefer larger differences.This pattern is observed when optimizing temporal frames over candidate locations and depths in nuScenes.
  • 4.3 IMPACT OF RESOLUTION ON LOCALIZATION POTENTIAL: Downsampling image features by a factor of 4 divides localization potential by 4, making matching more difficult at coarse resolution.The paper notes that standard multi-view matching commonly uses 1/4 resolution, while detection temporal stereo commonly uses 1/16 resolution.
  • 4.4 EFFECT OF TEMPORAL DIFFERENCE ON MULTI-VIEW DEPTH AMBIGUITY: Aggregating more timesteps dramatically increases the percentage of objects benefiting from multi-view stereo across cameras and depths.An object is counted when shifting its 3D center by 0.5m produces at least a one-pixel projection difference.
  • Summary: Because no single temporal difference is optimal everywhere, many timesteps provide candidate locations with more favorable multi-view setups.The paper summarizes this conclusion from both theoretical analysis and empirical verification.

5 METHOD

SOLOFusion balances coarse, efficient long-term temporal stereo with fine-grained short-term stereo. Its two-stream design improves temporal multi-view detection while managing the computational cost of high-resolution, long-history fusion.

  • 5 METHOD: SOLOFusion combines an LSS-based long-history stream for coarse features with an MVS-based short-history stream for high-resolution depth prediction.The long-term stream uses low-resolution temporal stereo for detection, while the short-term stream refines per-frame depth estimation.
  • 5.1 LONG-TERM TEMPORAL FUSION: A 1/16-resolution BEV cost volume offsets coarse matching by aligning and concatenating BEV features from previous timesteps.The resulting temporal representation leverages long history without requiring high-resolution features throughout the pipeline.
  • 5.2 SHORT-TERM TEMPORAL FUSION: The short-term module performs MVS using two consecutive frames and a 1/4-resolution plane-sweep volume.Monocular depth predictions guide candidate sampling to reduce the cost of stereo matching.
  • 5.2 SHORT-TERM TEMPORAL FUSION: Short-term high-resolution temporal stereo further improves the strong long-term low-resolution model, demonstrating complementarity between the two streams.The proposed sampling strategy covers multi-modal depth distributions with minimal computational cost.
  • Necessity of Balance: High-resolution, long-term fusion is considered infeasible because high-resolution short-term stereo already substantially increases GPU memory and runtime.SOLOFusion therefore targets a trade-off between spatial resolution and temporal aggregation.

6 EXPERIMENTS

Experiments on nuScenes show that SOLOFusion benefits from combining long-term, low-resolution fusion with short-term, high-resolution matching. This design improves accuracy while maintaining strong efficiency.

  • Main Results: SOLOFusion outperforms STS by 5.2% mAP in the ResNet101, CBGS setting and achieves 1st place on the nuScenes test-set camera-only track.It also improves mAVE by 12.3% over the cited comparison and uses weaker training and inference settings.
  • Time Window for Temporal Fusion: 6.1% mAP and 7.9% mATE improvements result from increasing low-resolution long-term fusion from 1 to 16 timesteps.Performance saturates at 16 timesteps because there is little visible-region overlap beyond that history.
  • Depth Hypothesis Sampling: Top-k sampling guided by monocular depth improves all metrics, while Gaussian-Spaced Top-k further improves mAP and mATE with minimal FPS loss.Matching all 112 hypotheses increases runtime by 6x, whereas 28 uniform hypotheses impose a 2.4x slowdown.
  • Short-Term and Long-Term Fusion: The short-term and long-term modules improve mATE by 5.2% and 7.2%, respectively, while their combination yields an approximately 12% improvement.The results support their complementary contributions across the two resolution–temporal-difference trade-off regimes.
  • Balancing Temporal Fusion and Resolution: At half resolution, SOLOFusion outperforms full-resolution BEVDepth by 2.2% mAP and 1.1% NDS.It achieves similar performance at roughly 5x the FPS and half the memory cost.
  • Runtime Analysis: SOLOFusion runs 2.5x faster than STS while providing a larger improvement in localization.The comparison concerns methods using short-term temporal stereo with ResNet50.

7 CONCLUSION

The paper reframes temporal 3D detection as temporal stereo matching and uses localization potential to analyze temporal differences and feature granularity. SOLOFusion combines short- and long-term temporal information, achieving state-of-the-art nuScenes performance.

  • 7 CONCLUSION: The paper defines localization potential to measure the ease of multi-view depth estimation, identified as a bottleneck in multi-view 3D detection.Its analysis finds that temporal differences and feature granularity are especially important under the temporal-stereo formulation.
  • 7 CONCLUSION: SOLOFusion exploits the synergy between short-term and long-term temporal information and the trade-off between temporal extent and feature resolution.The resulting framework is presented as a baseline for temporal 3D object detection.
  • 7 CONCLUSION: 5.2% mAP and 3.7% NDS improvements over prior art establish a new nuScenes state of the art and place SOLOFusion at the top of the leaderboard.The paper reports this outcome across its nuScenes evaluation.

A ADDITIONAL DETAILS FOR THE UNIFIED TEMPORAL STEREO FORMULATION FOR CAMERA-ONLY 3D DETECTION

The appendix unifies prior camera-only 3D detection and multi-view stereo methods as temporal stereo systems by identifying their candidate locations, feature sampling, temporal aggregation, and prediction mechanisms.

  • A.1 MULTI-VIEW STEREO METHODS: MVSNet matches projected image features for every (x, y, d) candidate at fine 1/4 resolution and predicts a depth distribution from aggregated multi-view features.Its candidate probabilities represent occupancy at 3D locations before weighted depth estimation.
  • A.1 MULTI-VIEW STEREO METHODS: MaGNet restricts candidates to the confidence interval of a predicted depth Gaussian and aggregates their temporal features with dot products and a 2D CNN.It iteratively updates the depth Gaussian using the processed features.
  • A.2 LSS-BASED 3D DETECTION METHODS: BEVDet4D uses BEV grid cells as candidates, pools densely generated depth hypotheses at coarse 1/16 resolution, and concatenates ego-motion-aligned past BEV features.This expresses its temporal aggregation as features corresponding to past image points projected into the current grid.
  • A.2 LSS-BASED 3D DETECTION METHODS: BEVDepth retains BEVDet4D’s main components but aggregates a more recent frame at 0.5s.Its distinction from BEVDet4D is the temporal frame used for aggregation.
  • A.2 LSS-BASED 3D DETECTION METHODS: STS extends BEVDepth with an MVS depth-estimation component whose components are mostly identical to MVSNet.Its inherited low-resolution temporal stereo remains limited by using a single temporal frame.
  • A.2 LSS-BASED 3D DETECTION METHODS: BEVStereo also extends BEVDepth with MVS depth estimation but adopts design intuitions from MaGNet rather than MVSNet.The appendix distinguishes the two methods by the stereo component informing their extensions.
  • A.3 QUERY-BASED 3D DETECTION METHODS: BEVFormer uses BEV-aligned queries with fixed 3D z-axis samples and coarse 1/16-to-1/64 image features, aggregating past queries through deformable attention.Although it continuously saves past BEV queries, the text reports no benefit beyond training with three historical frames.
  • A.3 QUERY-BASED 3D DETECTION METHODS: PolarDETR, PolarFormer, UVTR, and PETRv2 use object queries or intermediate representations as candidate locations while aggregating temporal image information through concatenation, deformable attention, or cross-attention.Their candidate and aggregation mechanisms differ, but each fits the appendix’s unified temporal-stereo organization.

B.1 DERIVATION AND ANALYSIS OF IMAGE A TO IMAGE B PROJECTION

The appendix derives the image-A-to-image-B projection and connects it to standard stereo, where projected depth separation determines the ease of depth estimation. It also explains how camera geometry, depth, and image resolution affect localization potential.

  • Projection Derivation: The derivation formulates the projection of image-A coordinates into image B under rotation and translation, then specializes it to standard stereo.The standard configuration sets θ = 0, tz = 0, and a positive baseline −tx.
  • Standard Stereo Connection: In standard stereo, the projection reduces to the familiar disparity relation disparity = txf / da.Depth hypotheses are matched along image-B’s epipolar line to recover depth from the camera transformation.
  • Localization Potential: Localization potential is larger when the baseline −tx or focal length f is larger, or when the evaluated depth is smaller.These conditions increase the separation between projections of nearby depth hypotheses.
  • Localization Potential: Projected depth differences are larger between 1m and 2m than between 59m and 60m.This illustrates why depth affects the ease of stereo matching.
  • Geometric Trade-offs: Increasing baseline or focal length is not a simple quality improvement because larger baselines reduce overlap and larger focal lengths limit the captured scene.The appendix explicitly qualifies the standard-stereo observations with these geometric trade-offs.
  • General Two-View Case: Unlike standard stereo, the general two-view case has localization potential that varies with image position because camera geometry is not constrained to aligned stereo.Standard stereo’s aligned cameras produce epipolar lines parallel to the x axis, simplifying setup selection.
  • Equation Derivation: The general projection is reparameterized using the viewing direction α and translation direction β, with total translation magnitude t̄.The substitutions sin α = x′a / r′a, cos α = f / r′a, sin β = tx / t̄, and cos β = tz / t̄ lead to Equation 3.
  • Equation Derivation: Trigonometric identities and substitutions transform the projection into the localization-potential formulation in Equation 3.The derivation uses sin(α − θ) and cos(α − θ) before differentiating with respect to da.

B.3 EFFECT OF VIEW ROTATION θ ON LOCALIZATION POTENTIAL

The analysis finds that localization potential depends on view rotation relative to each pixel and ego-motion, so no single rotation or temporal difference is optimal everywhere. Multi-camera and long-term multi-timestep projection provide more favorable matching configurations, especially during turns.

  • No singular camera rotation is best for all pixels or translations because the optimal rotation changes with pixel location and current ego-motion.
  • When θ approaches α, projected depth hypotheses become closer together, making depth estimation more difficult.
  • Multi-camera temporal stereo provides valid projections for all depth hypotheses and enables larger temporal differences without the overlap problem of standard stereo.
  • Optimal temporal differences vary across candidate locations during 30°, 60°, and 90° ego-vehicle turns.
  • Many past timesteps are necessary because the optimal time difference and projected camera vary substantially across candidate locations in turning scenarios.

B.6 ADDITIONAL ANALYSIS ON EFFECTS OF TEMPORAL DIFFERENCE ON MULTI-VIEW DEPTH AMBIGUITY

Longer temporal aggregation substantially reduces multi-view depth ambiguity by increasing projected separation between depth hypotheses. The analysis also describes a Gaussian down-weighting procedure that encourages depth-candidate exploration around the monocular prior.

  • Less than 20% of projection changes exceed one pixel for 40m–60m objects with single-timestep aggregation.
  • Using 16 past timesteps significantly eases multi-view depth estimation, including for frames with fewer than 16 available historical timesteps.
  • The front camera is the most difficult view because its points cannot use multi-camera depth estimation.
  • Temporal aggregation increases the distance between projected locations across cameras and depths, making depth hypotheses easier to distinguish.
  • The sampling update down-weights monocular depth probability near the selected candidate, balancing exploitation of the monocular prior with exploration of other candidates.

D.1 IMPLEMENTATION DETAILS

The implementation uses efficient sequential long-term fusion with 16 timesteps and supplements coarse matching with short-term high-resolution stereo. Comparisons without CBGS use additional iterations instead, creating a disadvantaged training setting.

  • The model uses T = 16 timesteps for long-term fusion and k = 7 depth hypotheses.
  • Past BEV feature maps are saved during training and inference to keep sequential long-term temporal fusion efficient.
  • Short-term high-resolution stereo matching uses a small FPN.
  • Because CBGS is not readily applicable, the comparison setting matches its cycle by increasing training iterations without other changes.
  • This comparison setting is disadvantaged because CBGS is known to substantially boost performance in rarer categories.
Loading 2210.02443v1…