PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping

Markus Käppeler, Rohit Mohan, Abhinav Valada

arXiv:2609.26325v1cs.CVcs.RO

TL;DR

Camera-only 3D detection and vectorized mapping remain challenged by depth ambiguity, while existing explicit point-cloud priors rely on LiDAR-built maps. This paper builds a geometric-semantic prior from earlier camera traversals and reports gains on both tasks: detection CDS rises from 0.287 to 0.299 and mapping mAP from 0.669 to 0.750.

  • Problem

    Camera-only detection and mapping remain challenged by depth ambiguity, while explicit point-cloud priors for camera-based perception have relied on LiDAR-built maps.

    Source: §I · p. 1

  • Method

    The framework builds a Pi3X point-cloud prior from previous camera traversals, enriches points with DINOv3 features, and fuses retrieved prior-map and camera features for joint detection and vectorized mapping.

    Source: §I · pp. 1–2

  • Results

    0.287 to 0.299 CDS for detection and 0.669 to 0.750 vectorized mapping mAP show gains on both tasks over the matched baseline.

    Source: C. 3D Object Detection and Vectorized… · pp. 3–4

  • Takeaways & Limitations

    The findings show that observations accumulated across camera traversals can provide persistent spatial context for subsequent perception without LiDAR for map construction or online inference.

    Source: §IV · p. 4

    The reconstructed prior map is less geometrically accurate than a LiDAR prior.

    Source: C. 3D Object Detection and Vectorized… · p. 4

Abstract

from arXiv · show

Camera-based 3D object detection and online vectorized HD mapping provide compact scene representations for autonomous driving, but both depend on accurate metric geometry and remain limited by depth ambiguity. Over long-term deployment, observations from repeated traversals can be accumulated into persistent point cloud priors that provide geometric context beyond the current observations. Existing explicit point cloud prior approaches, however, rely on LiDAR-based map construction and therefore require expensive 3D ranging sensors. We propose a framework that constructs a static point cloud prior map from previous camera traversals using Pi3X and augments each point with DINOv3 features. At runtime, a local prior patch is retrieved using global localization, encoded with a sparse voxel backbone, and fused in bird's-eye view (BEV) with lifted multi-view camera features. Task-specific sparse transformer heads then predict 3D objects and vectorized map elements from the fused representation. On Argoverse 2, the vision-based prior improves a strong baseline from 0.287 to 0.299 CDS and from 0.669 to 0.750 vectorized mapping mAP. Ablations show that semantic DINOv3 features are particularly important for vectorized mapping. These results demonstrate that vision-built geometric-semantic priors provide an effective form of long-term scene memory for camera-based perception, improving both tasks without LiDAR for prior-map construction or online inference.

I. INTRODUCTION

The introduction motivates persistent prior maps as a way to supplement camera observations with geometric context, then proposes a vision-built point cloud prior for camera-based 3D detection and vectorized mapping. The framework combines reconstructed geometry and image-derived semantics without requiring LiDAR for map construction or online perception.

I. INTRODUCTION

  • Camera-only methods retain rich visual semantics but face depth ambiguity, especially when mapping static structures occluded by traffic participants or other scene elements.LiDAR-only and camera–LiDAR fusion methods exploit accurate 3D geometry but require expensive ranging sensors at inference.Source: §I · p. 1
  • Prior traversals can be accumulated into persistent spatial maps, supplying geometric knowledge beyond current observations during online perception.Repeated visits by vehicles, including under open-set conditions, enable accumulation across agents and previous traversals.Source: §I · p. 1
  • Existing explicit point cloud priors for camera-based detection are built from LiDAR traversals, while PreSight uses computationally heavy per-scene NeRF optimization for vectorized mapping.Earlier prior-map work also applied neural-rendering or global occupancy maps to occupancy prediction –[24], and LiDAR-map priors to camera-based detection,.Source: §I · p. 1
  • A camera-built point cloud prior augmented with DINOv3 features improves both 3D object detection and vectorized mapping without LiDAR for map construction or online perception.The method uses feed-forward Pi3X reconstructions to combine metric scene structure with image-derived semantics, while accumulated static-map information can provide context when current observations are occluded.Source: §I · pp. 1–2
  • The framework retrieves a local patch from a global vision-built map, encodes it with a sparse voxel backbone, and fuses its BEV features with lifted camera features for task-specific transformer heads.Perspective-view image features also remain available to both heads through deformable aggregation.Source: §I · pp. 1–2, Figure 1 · p. 1

II. TECHNICAL APPROACH · A. Problem Setup

The framework jointly performs camera-based 3D object detection and vectorized mapping using an offline static point-cloud prior reconstructed from prior camera traversals. It uses a globally localized pose to retrieve a local map patch, without requiring LiDAR for map construction or online perception.

II. TECHNICAL APPROACH

  • The framework uses an offline static point-cloud prior reconstructed from previous camera traversals to support joint 3D object detection and vectorized mapping.At inference, it retrieves a local prior-map patch using global localization; neither map construction nor online perception requires LiDAR, unlike prior-map methods based on LiDAR,.Source: §II · p. 2

A. Problem Setup

  • Each time step uses synchronized surround-view images and their calibration parameters as inputs.Source: A. Problem Setup · p. 2
  • The model also receives a globally localized ego pose, while an offline global static map is constructed from previous traversals.Source: A. Problem Setup · p. 2
  • The model predicts oriented 3D boxes and vectorized map elements represented as polylines [1].Source: A. Problem Setup · p. 2
  • Each road boundary, lane divider, or pedestrian crossing is represented by a polyline with Np = 20 points.Source: A. Problem Setup · p. 2
  • Detection is performed within a 50 m radius, while mapping covers a 30 m × 60 m ego-centered region.These spatial ranges follow SparseDrive [1].Source: A. Problem Setup · p. 2

B. Vision-Based Prior Map Construction and Retrieval

The method builds a tiled static point-cloud prior from camera traversals, augments its points with compressed DINOv3 descriptors, and retrieves a local patch for the current ego frame while excluding nearby traversal data.

B. Vision-Based Prior Map Construction and Retrieval

  • Pi3X [28] reconstructs camera-frame point maps in 12-frame chunks, which are scaled to metric geometry and aligned globally using dataset-provided poses.Pi3X-predicted poses recover scale; dataset poses establish relative frame alignment and the global reference frame.Source: B. Vision-Based Prior Map Construction… · p. 2
  • The map retains confident static points, removes points within 0.6 m or beyond 60 m and dynamic objects, then average-pools them in 0.2 m voxels into tiled storage as in DualViewMapDet.The confidence threshold is 0.5.Source: B. Vision-Based Prior Map Construction… · p. 2
  • Each reconstructed point receives its pixel-aligned DINOv3 ViT-S feature [29], extracted with the multi-scale procedure of and PCA-compressed from 384 to 64 dimensions.Stored points retain their 3D position, Pi3X confidence, RGB value, and compressed descriptor.Source: B. Vision-Based Prior Map Construction… · p. 2
  • At runtime, the current ego translation and retrieval range define a local global-map patch; points from the current and temporally adjacent traversals are excluded before remaining traversal points are merged and transformed to the ego frame.The retrieved patch is drawn from multiple traversals.Source: B. Vision-Based Prior Map Construction… · p. 2

C. Camera–Map Encoding and Fusion

The section describes how multi-view camera features and a vision-built point-cloud prior are encoded in BEV, fused in a shared metric space, and used by task-specific transformer heads for detection and mapping.

C. Camera–Map Encoding and Fusion

  • Multi-scale camera features are lifted from perspective view into BEV using an LSS-style module that predicts per-pixel depth distributions and pools frustum features across cameras.The image backbone is VoVNet-99, and the lifting module follows LSS-style methods,, [35].Source: C. Camera–Map Encoding and Fusion · p. 2
  • Sparse transformer heads for detection and vectorized mapping sequentially aggregate fused BEV features and multi-view PV image features.The overview depicts the prior retrieved locally at the ego location and encoded by a BEV point-cloud encoder.Source: Figure 2 · p. 3
  • The prior-map encoder voxelizes ego-frame points at 0.2 m resolution, averages their attributes per voxel, and processes them with a sparse 3D backbone.Point representations include a compressed DINOv3 descriptor.Source: C. Camera–Map Encoding and Fusion · pp. 2–3
  • Camera and map BEV features share a spatial resolution and are fused by channel-wise concatenation followed by a 3 × 3 Conv-BN-ReLU block [7],.Lightweight ResNet blocks and an FPN further refine the fused metric representation.Source: C. Camera–Map Encoding and Fusion · p. 3

D. Sparse Detection and Mapping Heads

The detection and mapping heads use a symmetric sparse design that sequentially aggregates fused BEV and multi-scale PV image features. Their task-specific sampling uses 3D box anchors for detection and polyline points for mapping, combining metric camera–map context with fine-grained image information.

  • Detection maintains 3D box anchors and instance features, while mapping represents anchors as polylines with corresponding instance features.Source: D. Sparse Detection and Mapping Heads · p. 3
  • Both branches follow SparseDrive’s symmetric sparse perception design [1], augmented with sequential BEV–PV deformable aggregation [11],.Each decoder layer first aggregates the fused BEV representation, then multi-scale PV image features.Source: D. Sparse Detection and Mapping Heads · p. 3
  • Detection samples deformable keypoints from 3D box anchors, while mapping uses polyline points, exposing both tasks to metric BEV context and fine-grained PV image information.Source: D. Sparse Detection and Mapping Heads · p. 3

E. Training Objective

The network is trained end-to-end with detection, mapping, and LiDAR depth objectives, while the prior-map encoder receives only downstream task-loss supervision. Image–map grid masking is used to reduce reliance on locally available prior maps.

E. Training Objective

  • The complete network is trained end-to-end with SparseDrive detection and mapping objectives [1] and LiDAR depth supervision for the LSS module,.Source: E. Training Objective · p. 3
  • The prior-map encoder is optimized only through downstream task losses, and image–map grid masking reduces overreliance on locally available prior maps.Masking follows DualViewMapDet.Source: E. Training Objective · p. 3

III. EXPERIMENTAL EVALUATION

The evaluation measures how vision-built priors from previous traversals affect camera-based 3D object detection and online vectorized HD mapping on Argoverse 2.

  • The method is evaluated on Argoverse 2 for camera-based 3D object detection and online vectorized HD mapping.Source: §III · p. 3
  • The evaluation compares the method with strong camera-based baselines under standard protocols.Source: §III · p. 3
  • Ablations assess the contributions of prior geometry and DINOv3 semantics.Source: §III · p. 3

A. Implementation and Training Details · B. Dataset and Metrics

The implementation uses a shared BEV grid, multi-scale camera features, and camera-based inference, with LiDAR depth supervision during training. Experiments use Argoverse 2, with traversal-separated prior maps and task-specific detection and mapping metrics.

A. Implementation and Training Details

B. Dataset and Metrics

  • Argoverse 2 supplies synchronized surround-view cameras and repeated urban coverage; training maps use only training traversals, while validation maps exclude the current traversal.The experiments use globally corrected Argoverse 2 poses from DualViewMapDet.Source: B. Dataset and Metrics · p. 3
  • Detection is evaluated on 26 classes with official CDS, mAP, mATE, mASE, and mAOE; vectorized mapping uses mAP following StreamMapNet and MapTracker.Source: B. Dataset and Metrics · p. 3

C. 3D Object Detection and Vectorized Mapping

The vision-based prior improves both 3D object detection and vectorized mapping on Argoverse 2. Although it is less geometrically accurate than a LiDAR prior, its accumulated static structure and semantic features provide useful context for both tasks.

C. 3D Object Detection and Vectorized Mapping

D. Ablation Studies · IV. CONCLUSION

Ablations show that semantic point-cloud priors and reliable traversal coverage improve camera-based detection and mapping, while the conclusion presents the framework as a LiDAR-free persistent prior and identifies alignment and map-change detection as future work.

D. Ablation Studies

  • Adding DINOv3 features raises detection to 0.291 CDS and 0.388 mAP and mapping to 0.756 mAP.The particularly large mapping gain provides cues for static structures including lane dividers, road boundaries, and pedestrian crossings.Source: D. Ablation Studies · p. 4
  • Geometry-only priors improve mapping mAP from 0.669 to 0.683 but do not improve detection, showing that noisy camera-reconstructed geometry alone is insufficient.Source: D. Ablation Studies · p. 4
  • Using priors from all previous traversals yields 0.299 CDS and 0.396 mAP for detection, while mapping remains comparable at 0.750 mAP.This uses all prior traversals rather than only the two traversals with the largest point clouds per tile.Source: D. Ablation Studies · p. 4
  • The prior remains beneficial across evaluated coverage subsets, with CDS gains of +1.8–1.9 pp on coverage-restricted subsets versus +1.2 pp on the full validation set.The map-free baseline was re-evaluated on exactly the same samples for each subset.Source: D. Ablation Studies · p. 4

IV. CONCLUSION

  • The conclusion describes a persistent geometric-semantic point-cloud prior built from previous camera traversals, with Pi3X geometry enriched by DINOv3 features and fused with camera features in BEV before task-specific decoding.Source: §IV · p. 4
  • Experiments on Argoverse 2 show consistent gains over the SparseDrivestyle baseline, with the largest vectorized-mapping improvements when semantic DINOv3 features are included.Source: §IV · p. 4
  • The framework retains observations as an explicit cross-traversal spatial prior without LiDAR for prior-map construction or online inference; future work targets better BEV and feature-space alignment and map-change detection.Map-change detection is intended to improve robustness to outdated priors and identify when updates are required.Source: §IV · p. 4
Loading 2609.26325v1…