PaperlayerView exact revision on arXiv

Source-linked AI summary

From LiDAR Maps to Visual Localization: Unified Visual Association for Robust Point-Line-Plane Pose Estimation

Wentao Zhao, Zikun Chen, Yihe Niu, Haoyu Chen, Jingchuan Wang

arXiv:2609.27363v1cs.RO

TL;DR

Camera localization in a prior LiDAR map must bridge the image–point-cloud modality gap to support global localization and continuous tracking. The paper renders LiDAR geometry and reflectivity into quasi-images for shared visual association, then uses metric geometric constraints and distribution-aware optimization. Experiments report accurate localization and robust tracking on EuRoC and real-world sequences, with structural-factor benefits depending on map support.

  • Problem

    Camera localization in a prior LiDAR map must address the modality gap between camera images and point-cloud maps while supporting both global localization and continuous tracking.

    Source: §I · p. 1

  • Method

    The framework renders LiDAR geometry and reflectivity into quasi-images with 2D–3D provenance, then uses shared visual association and distribution-aware, observability-complementary pose optimization.

    Source: §I · pp. 1–2

  • Results

    Experiments on EuRoC and real-world robotic sequences demonstrate accurate localization and robust tracking under challenging motion, illumination changes, and dynamic occlusions.

    Source: §V · p. 8

  • Takeaways & Limitations

    The framework supports global localization and continuous 6-DoF tracking using a pre-built LiDAR map as the persistent prior.

    Source: §I · p. 2, §V · p. 8

    Line and plane constraint benefits depend on sufficient structural support; severely degraded or structure-poor maps leave localization predominantly reliant on point correspondences.

    Source: §V · p. 8

Abstract

from arXiv · show

Camera localization in a prior LiDAR map provides a persistent geometric reference for long-term robotic navigation, yet remains challenging because of the substantial modality gap between camera images and point-cloud maps. We present a unified localization framework that makes the LiDAR map visually addressable rather than relying on a dedicated image-LiDAR correspondence model. Map geometry and reflectivity are rendered into LiDAR-derived quasi-images with explicit 2D-3D provenance, enabling camera observations and rendered map views to share mature visual features and matchers for both global localization and continuous pose tracking. Point and line correspondences are established through this common visual interface, while the retained provenance recovers metric LiDAR geometry and line-supported planar constraints for pose estimation. To improve robustness under ambiguous associations and weak geometry, we further introduce a distribution-aware, observability-complementary optimization strategy. Instead of reducing matching ambiguity to a scalar confidence, candidate association distributions are propagated into directional pose-information uncertainty, and reliable structural factors are selectively reinforced according to their ability to complement the currently weak pose directions. Experiments on the EuRoC MAV benchmark and self-collected real-world sequences demonstrate accurate global localization and robust continuous 6-DoF tracking using only a pre-built LiDAR map as the persistent prior, including under severe illumination variations and dynamic occlusions.

I. INTRODUCTION

The paper addresses camera localization against a LiDAR map for both global initialization and continuous tracking, where existing approaches rely on camera-derived map appearance, learned cross-modal models, or shared geometric structures. It makes LiDAR maps visually addressable and combines shared visual association with metric geometry and observability-aware estimation.

I. INTRODUCTION

  • Prior-map localization must support global pose estimation without valid initialization and continuous tracking, but existing approaches do not directly provide both from a LiDAR prior without camera-derived map appearance.LiDAR maps offer metrically accurate, illumination-invariant geometry; appearance-based methods and learned cross-modal models have the limitations described in the introduction.Source: §I · p. 1
  • LiDAR-derived quasi-images let camera observations and map views share visual features and matchers, while 2D–3D provenance preserves metric geometry for point, line, and plane pose factors.The offline global-localization database is built from LiDAR geometry and reflectivity without reference imagery or an additional visual mapping traversal.Source: §I · pp. 1–2
  • Association ambiguity and geometric weakness differ: ambiguous candidates can yield similar pose information, while confident matches may contribute little along currently weak pose directions.The proposed strategy propagates association distributions into directional pose-information uncertainty and reinforces structural factors only when they complement weak pose modes.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior-map methods use camera-derived appearance, learned cross-modal representations, or shared geometry; the proposed alternative renders LiDAR into quasi-images for visual point–line matching while retaining metric provenance for PLP constraints.Geometry-centric methods depend on the quality and availability of structural primitives.Source: §II · p. 2

III. METHOD

The system uses an offline quasi-image database to initialize a map-relative camera pose, then renders local views from predicted poses for continuous stereo–inertial tracking. Temporal and map associations are jointly optimized with stereo and IMU constraints.

A. System Overview and Problem Formulation

  • The pre-built map represents each point with a 3D position and LiDAR reflectivity, and the system takes stereo observations over time.The supplied formulation identifies map-point position and reflectivity as the map attributes used by the method.Source: A. System Overview and Problem… · p. 2
  • Offline representative map viewpoints form a quasi-image database for visual retrieval and geometric verification of the initial camera pose.During tracking, a predicted pose renders a local pose-conditioned quasi-image, and the shared visual front-end establishes temporal and map associations.Source: A. System Overview and Problem… · p. 2
  • Stereo observations and IMU constraints are jointly optimized with temporal and map associations to estimate the current map-relative pose.The system formulation distinguishes global localization from continuous stereo–inertial tracking.Source: A. System Overview and Problem… · pp. 2–3

B. Quasi-Image Generation

The framework renders LiDAR geometry and reflectivity into quasi-images, then uses a shared visual front end to associate camera and map features. Retained provenance recovers metric point, line, and line-supported planar constraints, while candidate distributions are preserved for pose-information modeling.

B. Quasi-Image Generation

  • Cross-guided neighborhood aggregation uses depth continuity to guide reflectivity propagation and reflectivity consistency to guide depth completion.Max aggregation preserves salient reflectivity responses, while average aggregation improves depth continuity without propagating across inconsistent structures.Source: B. Quasi-Image Generation · p. 3
  • The quasi-image jointly encodes completed reflectivity and depth cues, with reflectivity providing appearance-like contrast and depth variations emphasizing geometric boundaries.During tracking, the predicted pose determines the rendering viewpoint for the pose-conditioned quasi-image.Source: B. Quasi-Image Generation · p. 3

C. Unified Feature Association

  • A shared, frozen pretrained PL-Net processes camera images and quasi-images to detect keypoints and line segments without modality-specific fine-tuning.Line descriptors sample point- and line-feature maps along each segment, combining line structure with point context.Source: C. Unified Feature Association · p. 3
  • The same association pipeline matches point and line features across camera and map views, then uses 2D–3D provenance to recover metric LiDAR geometry and line-supported planar constraints.Planar geometry is recovered from LiDAR points traced through quasi-image provenance.Source: C. Unified Feature Association · pp. 3–4
  • Visual–map associations retain multiple plausible hypotheses and their normalized scores for subsequent pose-information modeling rather than relying only on a hard selection.The highest-probability hypothesis supplies the nominal correspondence, while the full candidate distribution is retained.Source: C. Unified Feature Association · pp. 3–4

D. Observability-Aware Optimization

The method identifies weak directions in the current pose information, then evaluates whether plausible structural associations provide reliable information in those directions. Point, line, and plane factors are selectively reinforced according to their complementarity, with weights updated during iterative optimization.

Observability-aware optimization

  • Point, line, and plane constraints have complementary translational weaknesses: planes can complement point–line weakness, while points can complement line–plane weakness.These are local geometric relationships, not a claim that the complete estimator is exactly degenerate.Source: D. Observability-Aware Optimization · p. 4
  • Weak pose modes are identified from current pose information, with normalization used for observability analysis and structural reweighting rather than changing the residuals or state parameterization.The base information includes inertial, temporal visual, and map-point contributions.Source: D. Observability-Aware Optimization · pp. 4–5
  • A candidate’s score measures its first-order increase in a normalized pose-information eigenvalue, indicating whether it informs a currently weak direction.This uses the candidate-specific information contribution in the same normalized pose space as the observability analysis.Source: D. Observability-Aware Optimization · p. 5
  • The full association distribution captures directional-information variation; diffuse matches can have zero variation when all candidates provide the same information.The expected contribution and its sensitivity to alternative associations are tracked separately.Source: D. Observability-Aware Optimization · p. 5
  • Structural factors are reinforced when they provide reliable information along weak pose directions relative to their overall information magnitude.The complementary score combines reliable weak-direction information with normalization by the factor’s total pose information.Source: D. Observability-Aware Optimization · p. 5
  • Each outer iteration linearizes factors, evaluates weak modes and candidate complementarity, then holds structural weights fixed during the Gauss–Newton update before recomputing them.The joint objective includes inertial, temporal visual, and visual–map factors.Source: D. Observability-Aware Optimization · p. 5

A. Experimental Setup

Evaluation uses EuRoC MAV and self-collected indoor sequences that target motion, illumination changes, degraded structure, and dynamic occlusion. Global localization and continuous tracking use separate baselines, metrics, and localization pipelines.

Datasets and platforms

  • EuRoC V1 and V2 provide rapid motion and illumination variation, while the self-collected sequences target degraded structure, illumination changes, and pedestrian occlusions.Illum.-1 and Illum.-2 have substantial illumination changes; Occl.-1 has sustained pedestrian occlusions over large portions of the static scene.Source: A. Experimental Setup · p. 6
  • Prior maps and reference trajectories are generated offline by LiDAR–inertial odometry with loop closure and dynamic-object filtering; online localization uses stereo images and IMU.The platform carries a RealSense D435i and a 32-beam Velodyne LiDAR.Source: A. Experimental Setup · p. 6

Baselines and metrics

  • Global localization is assessed by recall and mean translation error, while continuous tracking is assessed by ATE RMSE against visual, visual–inertial, and prior-map baselines.Global localization uses retrieved virtual viewpoints, LightGlue correspondences, multi-view voting, and multi-hypothesis PnP verification; tracking jointly optimizes PLP map constraints with temporal visual and IMU factors.Source: A. Experimental Setup · p. 6

B. Evaluation on EuRoC

On EuRoC, the framework sustains strong global localization from LiDAR-derived views and achieves the best continuous-tracking performance on most sequences. Its tracking advantage is especially visible on V103 and V203.

Global Localization Performance

  • Table II reports Recall as the percentage of queries below 0.3 m translation error and mATE as mean translation error of relocalized poses.The compared methods use camera-derived reference information, whereas this method synthesizes its database from LiDAR geometry and reflectivity.Source: B. Evaluation on EuRoC · p. 6, Table II · p. 6
  • The full configuration maintains strong global-localization recall across all four EuRoC sequences despite matching RGB queries directly to LiDAR-derived views.Ours (Fast) retains useful localization accuracy with substantially lower latency, while Ours (Full) uses a more exhaustive search.Source: B. Evaluation on EuRoC · p. 6, Table II · p. 6

Continuous Pose Tracking

  • The method achieves the best tracking performance on most EuRoC sequences, with a marked advantage on challenging V103 and V203.Competing methods show larger trajectory deviations and pronounced error peaks under rapid motion and limited visual support.Source: B. Evaluation on EuRoC · p. 6, Table III · p. 6, Figure 4 · p. 7

C. Real-World Experiments

On self-collected sequences, incomplete structural support and noisier maps make line and plane constraints less reliable. The method remains more stable under illumination changes and sustained pedestrian occlusions by combining point matches with selectively reinforced structural constraints.

C. Real-World Experiments

  • The method remains more stable on self-collected sequences by combining visual point correspondences with complementary line and plane constraints.The reported setting includes strong illumination changes and sustained pedestrian occlusions.Source: C. Real-World Experiments · p. 7
  • Less complete structural support and greater registration and fitting noise make reliable line and plane constraints harder to obtain than on EuRoC.Structure-dominated map-based methods degrade when geometric primitives are sparse or noisy.Source: C. Real-World Experiments · p. 7
  • Distribution-aware association modeling and observability-complementary reweighting selectively reinforce reliable structural contributions.The reported robustness is under strong illumination changes and sustained pedestrian occlusions.Source: C. Real-World Experiments · p. 7

D. Ablation and Analysis

The ablations show that visual point matching provides a robust backbone when structural associations are sparse, while line and plane constraints add useful geometric information. Observability weighting and association uncertainty further reduce tracking error on both EuRoC and real-world sequences.

Feature Association and Geometric Ablation

  • On real-world sequences, sparse and fragmented structural associations make line-only tracking fail, while more abundant point correspondences remain a stable localization backbone.Mapping uncertainty, glass surfaces, local degeneracy, and dynamic occlusions further reduce usable line and plane associations.Source: D. Ablation and Analysis · p. 7
  • Adding heterogeneous PLP constraints reduces EuRoC average ATE from 3.74 cm to 2.35 cm, while line-only tracking remains feasible with complete structural support.The quasi-image front-end establishes consistent point and line correspondences despite the camera–LiDAR appearance gap.Source: D. Ablation and Analysis · p. 7, Figure 6 · p. 7, Table V · p. 8

Effect of Observability and Association Modeling

  • Observability modeling reduces average ATE from 5.08 to 3.78 cm on EuRoC and from 55.21 to 35.15 cm on real-world sequences; association uncertainty further reduces them to 2.35 and 24.27 cm.The ablation compares uniform structural weighting, observability-complementary reweighting, and the full association-uncertainty model.Source: D. Ablation and Analysis · pp. 7–8, Table VI · p. 8

E. Runtime Analysis

On EuRoC V103, the fast global-localization configuration cuts per-query runtime relative to the full search. Tracking averages 75.10 ms per frame, with information modeling and reweighting adding only 0.06 ms.

E. Runtime Analysis

  • The fast global-localization search takes 129.67 ms per query, versus 337.58 ms for the full search.The fast and full configurations use candidate search budgets of 16/32 and 40/200, respectively.Source: E. Runtime Analysis · p. 8
  • Tracking averages 75.10 ms per frame, corresponding to approximately 13.3 Hz.Keyframe refinement takes 111.80 ms.Source: E. Runtime Analysis · p. 8
  • Information modeling and reweighting add only 0.06 ms to tracking computation.Most computation is spent on quasi-image generation and PLP extraction and association.Source: E. Runtime Analysis · p. 8

V. CONCLUSION

Experiments on EuRoC and real-world robotic sequences demonstrate accurate and robust localization under challenging motion, illumination changes, and dynamic occlusions. The framework uses LiDAR-derived quasi-images and metric 2D–3D provenance, with distribution-aware optimization that reinforces reliable structural information along weak pose directions.

V. CONCLUSION

  • Experiments on EuRoC and real-world robotic sequences demonstrate accurate and robust localization under challenging motion, illumination changes, and dynamic occlusions.Source: §V · p. 8
  • LiDAR-derived quasi-images enable shared visual association while preserving metric 2D–3D provenance for heterogeneous point-line-plane pose constraints.Source: §V · p. 8
  • Distribution-aware, observability-complementary optimization reinforces reliable structural information along weak pose directions.Source: §V · p. 8
  • The benefit of line and plane constraints depends on sufficient structural support; severely degraded or structure-poor maps shift localization predominantly to point correspondences.Source: §V · p. 8

Related papers

Loading 2609.27363v1…