PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Which Correspondences to Trust: Confidence-Weighted Event-Camera Localization in LiDAR Maps

Panagiotis Kiousis, Kuangyi Chen, Jun Zhang, Friedrich Fraundorfer

arXiv:2610.11967v1cs.CVcs.RO

TL;DR

Existing event-camera localization pipelines use geometric consensus but do not explicitly model correspondence reliability or pose informativeness, and flow error can be biased toward distant, low-parallax points. CELL learns confidence through differentiable probabilistic PnP, reuses it across the pipeline, and improves localization over LEAR on M3ED and DSEC.

  • Problem

    Existing event-to-LiDAR pipelines do not explicitly model correspondence reliability or how strongly each correspondence constrains pose, while flow-error-based confidence can favor distant, low-parallax points.

    Source: §III · p. 3, C. Learning confidence through the pose · p. 3

  • Method

    CELL learns correspondence confidence through differentiable probabilistic PnP and uses it to reweight flow supervision, select correspondences, and weight edge-matching refinement.

    Source: §I · p. 1, C. Learning confidence through the pose · p. 3, E. Confidence-weighted edge-matching… · p. 4

  • Results

    On M3ED, median translation error falls by 10.9% on average, with a maximum reduction of 26.5%, and median rotation error falls by 6.6% on average, with a maximum reduction of 15.8%.

    Source: B. Main results · p. 6

  • Takeaways & Limitations

    The conclusion reports that the combined components improve localization over LEAR on both M3ED and DSEC.

    Source: §VI · p. 8

    The learned confidence needs sufficient training data, gains on near-saturated DSEC accuracy are modest, and performance may degrade when the initial pose is far from the true pose.

    Source: §VI · p. 8

Abstract

from arXiv · show

Localizing an event camera against a pre-built LiDAR map can be cast as dense optical-flow estimation between a rendered depth view and an event image, followed by a Perspective-n-Point (PnP) solver over the induced 3D-2D correspondences. Existing pipelines rely on geometric consensus during pose estimation, but do not explicitly model the reliability or pose informativeness, i.e., how strongly a correspondence constrains the camera pose, of individual correspondences. We show that the natural way to learn it -- using the per-correspondence error to constrain the learning of confidence -- suffers from a depth-dependent bias: small pixel errors reside predominantly at large depths and do not lead to high pose informativeness. Instead, in our method (CELL), we learn a per-correspondence confidence end-to-end through the pose, using a differentiable probabilistic PnP whose log-partition term encourages weight configurations that yield a better-constrained pose distribution. The learned confidence is used in three ways: (i) it reweights the flow supervision in a decoupled training scheme that keeps pose gradients out of the flow/edge backbone; (ii) it drives a probabilistic correspondence selection at test time; and (iii) together with the network's edge-probability it weights a final edge-matching refinement. We further design a partial-completion depth representation that adds signal without hallucinating across large gaps. On M3ED and DSEC our full system improves over the LEAR baseline on the majority of the evaluated sequences: it reduces the median translation error by up to 26.9% and the median rotation error by up to 15.8%.

I. INTRODUCTION

Event-camera localization in LiDAR maps uses dense event–depth flow to form 3D–2D correspondences for PnP, but existing pipelines do not explicitly model each correspondence’s reliability or pose constraint. CELL learns confidence through the pose objective because image-space flow error can favor distant correspondences that constrain translation weakly.

I. INTRODUCTION

  • The baseline estimates dense flow between a rendered depth view and event image, lifts flow to 3D–2D correspondences, and solves pose with PnP.Geometric consensus rejects inconsistent correspondences but does not explicitly represent their reliability or pose informativeness.Source: §I · p. 1
  • Small pixel-space flow errors can favor distant, low-parallax points that provide weak constraints on camera translation.Geometrically consistent correspondences can still differ substantially in pose usefulness.Source: §I · p. 1
  • CELL learns correspondence confidence through differentiable probabilistic PnP instead of treating image-space error as a proxy for pose quality.The pose objective weights correspondences according to how well they constrain the pose distribution.Source: §I · p. 1

II. RELATED WORK

  • The method also uses learned confidence to reweight flow supervision, select correspondences, and weight edge-matching refinement, while partial depth completion adds support without crossing large gaps.These components build on the LEAR pipeline; related event-to-LiDAR work includes EVLoc and LEAR.Source: §I · p. 1, §II · p. 2, Figure 1 · p. 2

III. PRELIMINARIES

The LEAR baseline renders sparse depth from a LiDAR map at an initial pose, estimates event-to-depth flow, and converts valid pixels into 3D–2D correspondences for PnP. CELL extends this pipeline with partial depth completion to increase correspondence support while retaining depth boundaries.

III. PRELIMINARIES

  • LEAR renders a sparse depth image from map points at a coarse initial pose and accumulates a short event window into an event image.Separate depth and event encoders feed an iterative feature-refinement module that updates flow and edge-probability maps.Source: §III · p. 2
  • Valid depth pixels are deprojected into 3D points, paired with flow-predicted event-image observations, and passed to PnP within RANSAC.The baseline does not explicitly model correspondence reliability before geometric estimation, and sparse depth limits correspondence support.Source: §III · p. 3

IV. METHOD

  • CELL reuses pose-supervised confidence to reweight flow supervision, guide test-time correspondence selection, and weight edge-matching refinement.It also uses conservative partial depth completion to expand the spatial support of the sparse projection.Source: §IV · p. 3

A. Partial depth completion

  • A single 3 × 3 cross-kernel dilation raises valid depth coverage from approximately 20% to 68% without filling large gaps.The partial completion supplies more valid 2D–3D correspondences while preserving object edges needed by edge matching.Source: A. Partial depth completion · p. 3

B. Per-correspondence confidence head

CELL predicts per-pixel confidence from flow-network features and trains it through a probabilistic pose objective rather than per-correspondence flow error. The learned confidence also reweights flow supervision, with the confidence branch decoupled from the flow/edge backbone.

B. Per-correspondence confidence head

  • The confidence head combines recurrent hidden-state snapshots, context, and motion features into per-pixel logits that produce a positive confidence map.The input features are detached so confidence-branch gradients do not propagate into the flow/edge backbone.Source: B. Per-correspondence confidence head · p. 3

C. Learning confidence through the pose

  • Flow-error supervision is depth-biased: distant, low-parallax points tend to have smaller pixel residuals despite weak pose constraints.The error-guided confidence baseline can therefore favor distant points rather than correspondences most informative for pose.Source: C. Learning confidence through the pose · p. 3
  • The pose loss combines a weighted residual at the target pose with a log-partition term that favors weights yielding a sharp pose distribution.Qualitative maps show confidence concentrating most strongly on near-depth, high-parallax object edges.Source: C. Learning confidence through the pose · p. 3, Figure 2 · p. 4
  • With flow detached, the pose loss trains only the confidence head, while detached confidence reweights flow supervision over valid pixels.This decoupled scheme keeps pose gradients out of the flow/edge backbone.Source: C. Learning confidence through the pose · p. 4

D. Confidence-guided selection at test time

CELL uses confidence-dependent stochastic sampling to favor reliable correspondences without deterministically discarding every low-ranked point. A nonzero sampling floor hedges against imperfect confidence and reduces the number of correspondences sent to PnP+RANSAC.

D. Confidence-guided selection at test time

  • A hard median-confidence cutoff can discard useful correspondences whenever the confidence ranking is imperfect.The alternative is to bias the correspondence set through confidence-dependent stochastic sampling because the solver does not support per-point weights.Source: D. Confidence-guided selection at test… · p. 4
  • With floor0.5, even the minimum-confidence correspondence is retained with probability 0.5, rising linearly to 1 for the most confident point.The nonzero floor favors reliable correspondences while hedging against confidence errors.Source: D. Confidence-guided selection at test… · p. 4
  • On average, floor0 and floor0.5 retain approximately 28% and 64% of correspondences, respectively, versus 100% when using all points.This reduces the workload of the subsequent PnP+RANSAC stage.Source: D. Confidence-guided selection at test… · p. 4

E. Confidence-weighted edge-matching refinement

The refinement aligns predicted depth-edges with observed event edges by optimizing camera pose, weighting each edge by both edge probability and correspondence confidence.

E. Confidence-weighted edge-matching refinement

  • Predicted edges are thresholded and skeletonized, then deprojected through the rendered depth image into 3D points for refinement.The pose is parameterized by axis-angle rotation and translation.Source: E. Confidence-weighted edge-matching… · p. 4
  • The pose objective minimizes robust chamfer distance to the event-edge distance transform while regularizing toward the initial pose.The optimization is solved with LBFGS using a Huber loss.Source: E. Confidence-weighted edge-matching… · p. 4
  • Each edge weight is the product of edge probability and confidence, combining evidence that an edge is genuine with correspondence pose reliability.The paper reports that this combination gives the best accuracy.Source: E. Confidence-weighted edge-matching… · p. 4

A. Setup

The evaluation uses M3ED and DSEC with shared LEAR settings, then reports improvements over LEAR on most evaluated sequences across both datasets.

A. Setup

  • M3ED is the primary benchmark across indoor flight, outdoor driving, and forest environments; DSEC adds complex driving scenarios but provides disparity images rather than LiDAR maps.DSEC depth is reconstructed by back-projecting disparity-derived depth through the pinhole camera model.Source: A. Setup · p. 5
  • The evaluation perturbs ground-truth initial poses by up to ±50 cm and ±5°; DSEC uses identity initialization, and PnP+RANSAC estimates test poses with a 12-pixel inlier threshold.EPro-PnP is used only during training.Source: A. Setup · p. 5
  • M3ED models are trained per scene with held-out sequences or a temporal 70/30 split, while DSEC pools eligible scenes for training and excludes single-sequence scenes.Models train for 100 epochs; reported metrics include median translation and rotation error, thresholded accuracy, and flow EPE.Source: A. Setup · p. 5

B. Main results

  • Across 22 evaluated sequences, the method reduces median translation error on 19 and median rotation error on 18, while improving localization accuracy on 14 versus LEAR.The comparison is against LEAR; reported translation and rotation changes are relative reductions, while accuracy changes are percentage-point gains.Source: B. Main results · pp. 5–6
  • On nine M3ED sequences, median translation error falls by 10.9% on average and up to 26.5%, while median rotation error falls by 6.6% on average and up to 15.8%.Localization accuracy improves on all nine sequences by 7.4 percentage points on average and up to 11.2 points.Source: B. Main results · p. 6
  • On 13 DSEC sequences, median translation error falls by 5.7% on average and up to 26.9%, while median rotation error falls by 4.3% on average and up to 14.7%.LEAR already reaches at least 98% accuracy on 11 sequences, and the method maintains or improves accuracy on most sequences.Source: B. Main results · p. 6

C. Flow-error compared to pose supervision

The paper contrasts flow-error- and pose-supervised confidence, then evaluates confidence-guided selection, finding that a broad confidence-biased rule outperforms harsher selection overall.

C. Flow-error compared to pose supervision

  • With top-10% selection, flow-error-supervised confidence degrades pose estimation more than pose-supervised confidence, tending to favor more distant correspondences.Pose-supervised confidence retains correspondences with lower average depth and substantially better localization.Source: C. Flow-error compared to pose… · p. 6

D. Confidence-guided selection

  • Across 22 sequences, floor0 gives the best or tied results on five of six columns, with mean median translation and rotation error reductions of 7.5% and 3.2%.These selection results use non-refined poses; the gains are additional to the improvement already present without selection.Source: D. Confidence-guided selection · p. 6
  • Across 22 sequences, floor0 gives best overall results, reducing mean translation and rotation errors by 7.5% and 3.2%; the hard >median rule remains competitive on rotation but trails on translation, while DSEC selection is largely neutral and only the hard cut hurts.These comparisons use non-refined poses, before edge-matching refinement.Source: D. Confidence-guided selection · p. 6

E. Edge-matching refinement weighting

The refinement weights predicted edges using correspondence confidence, edge probability, or both. The combined weighting is strongest overall, with confidence also providing a useful cue.

E. Edge-matching refinement weighting

  • The tested choices are uniform weighting, correspondence confidence alone, and the product of edge probability and confidence.Confidence is sampled at the edge pixel; combined weights multiply it by the predicted edge probability.Source: E. Edge-matching refinement weighting · p. 6
  • All three weightings improve accuracy over the baseline and non-refined pose; confidence adds +0.29 points versus +0.26 for uniform weighting relative to the non-refined pose.The comparison reports mean accuracy over all sequences.Source: E. Edge-matching refinement weighting · p. 6, Table IV · p. 7
  • Combined edge-probability and confidence weighting is strongest overall (+3.12/+0.32).The two cues are complementary: edge probability reflects whether an edge is present, while confidence is learned through pose supervision.Source: Table IV · p. 7, E. Edge-matching refinement weighting · p. 7

F. Depth completion: sparse vs. partial vs. full

Partial depth completion preserves object boundaries while adding useful correspondences, and the full system improves localization over the baseline. The reported tests also show gains with fewer flow iterations.

F. Depth completion: sparse vs. partial vs. full

  • Partial completion is best across every reported metric, while full completion drops accuracy below sparse input (50.6 vs. 51.1) and worsens median EPE (6.06 → 6.42 px).Partial completion fills approximately 68% of pixels while preserving boundaries; full completion fills approximately 99.8% and propagates depth across object margins.Source: F. Depth completion: sparse vs.… · p. 7

G. Incremental ablation

  • The full system reduces median translation and rotation errors by 3.77 cm and 0.186° and improves localization accuracy by 10.4 percentage points over the baseline.This is the cumulative result after adding the proposed components.Source: Table VI · p. 7, G. Incremental ablation · p. 7
  • Partial depth completion and decoupled confidence training provide the largest incremental accuracy gains (+2.3 and +6.5 percentage points), followed by selection (+1.3 points) and refinement (+0.3 points).The final refinement also reduces median translation error by 0.22 cm.Source: Table VI · p. 7, G. Incremental ablation · p. 7

H. Halving the iteration budget

  • At 12 IFR iterations versus the 24-iteration LEAR baseline, median translation error falls on 13/22 sequences and accuracy rises on 12/22, mainly on M3ED (7/9 scenes, +3.3 points on average).The flow-update cost is roughly halved; both median errors also improve on near-saturated DSEC.Source: Table VII · p. 7, H. Halving the iteration budget · p. 7

I. Runtime and memory

The confidence head and partial-depth dilation add negligible runtime and memory over the baseline network, while edge-matching is the substantial added stage. The paper concludes that its components improve localization on M3ED and DSEC, while noting data and initialization constraints.

I. Runtime and memory

  • The confidence head and partial-depth dilation add negligible runtime and memory; edge-matching refinement is the one substantial addition and can be disabled while retaining baseline-level network cost and improved localization.Flow estimation dominates network cost and scales linearly with the iteration count.Source: Table VIII · p. 8, §I · p. 8

VI. CONCLUSION

  • The method improves localization over LEAR on both M3ED and DSEC by reusing pose-supervised confidence for flow supervision, correspondence selection, and edge-matching refinement.The confidence is combined with edge probability for the final refinement.Source: §VI · p. 8
  • The learned confidence needs sufficient training data to become reliable, and localization may degrade when the initial pose is far from the true pose.The reported gains on DSEC are modest because its accuracy is already near-saturated.Source: §VI · p. 8
Loading 2610.11967v1…