PaperlayerView exact revision on arXiv

Source-linked AI summary

SimFuse3D: Source-Guided Target Simulation and Confidence-Guided Multi-Stage Localization Reweighting for Cross-Platform 3D Object Detection

Yongchun Lin, Xinliang Zhang, Yun Zou, Zhixuan Xiao, Liang Lei, Jianya Guo, Yuqiang Zhai, Xiaofeng Wang, HaiKuo Xu, Haoang Li

arXiv:2609.04886v1cs.CVcs.AI

TL;DR

Cross-platform LiDAR UDA suffers when retained pseudo-boxes provide useful locations but unreliable enclosed point observations. SimFuse3D repairs those observations with target-view-aligned source geometry and reweights localization supervision, improving every reported AP component over Pi3DET-Net across six transfers and ranking first in nearly all comparisons.

  • Problem

    Changed sensor height and viewpoint create box–point inconsistency: retained target pseudo-boxes may enclose sparse, cluttered, or geometrically inconsistent observations despite useful locations.

  • Method

    SimFuse3D retrieves labeled source objects, aligns and filters their geometry at target placements and viewpoints, and maps target confidence scores to bounded RPN and R-CNN localization weights.

  • Results

    Across six transfers and two detectors, SimFuse3D improves all 48 AP components over Pi3DET-Net, ranks first in 47 of 48 comparisons, and reaches the highest listed nuScenes→KITTI scores with both detectors.

  • Takeaways & Limitations

    SimFuse3D preserves useful pseudo-box placement while repairing unreliable observations, with adaptation-only components that leave the deployed detector unchanged.

Abstract

from arXiv · show

Changes in sensor height and viewpoint alter object-level point distributions, making cross-platform LiDAR unsupervised domain adaptation (UDA) difficult. Self-training uses labeled source scans and unlabeled target scans, yet a retained prediction may provide a useful target location while enclosing sparse foreground returns, background clutter, or points inconsistent with the predicted box. We refer to this mismatch as box-point inconsistency. We introduce SimFuse3D, which preserves the target placement and repairs the associated pseudo-object using measured geometry from labeled source scans. Object Memory retrieves a compatible labeled source instance. Target Simulation places its ground-truth box at the target location, aligns its points with the target viewing geometry, and filters the aligned crop to approximate the target observation. Confidence-Guided Multi-Stage Localization Reweighting (CMLR) maps each target pseudo-object confidence score to a bounded weight shared by RPN localization and R-CNN box regression. All components operate only during adaptation, leaving the detector architecture and inference graph unchanged. Across six cross-platform transfers, SimFuse3D exceeds Pi3DET-Net on every reported AP metric and ranks first among the compared adaptation methods on nearly all metrics. On nuScenes-to-KITTI, it ranks first among the compared adaptation methods with both evaluated detectors.

I. INTRODUCTION

Cross-platform LiDAR UDA is difficult because changed sensing viewpoints alter object point distributions, while accepted pseudo-boxes can retain unreliable observations. SimFuse3D separates target placement from object geometry, repairs pseudo-objects with aligned source measurements, and reweights localization supervision.

  • Changed sensor height, motion, and viewpoint alter incidence angles, occlusion, sampling density, and target distributions across platforms.
  • Box–point inconsistency pairs a usable pseudo-box location with sparse, one-sided, cluttered, or geometrically inconsistent points, corrupting localization and RoI supervision.
  • SimFuse3D retrieves a labeled source instance, aligns its measured geometry to the target placement and viewing direction, filters it, and uses the source annotation for dimensions.
  • CMLR applies bounded score-derived weights to target pseudo-objects at RPN and R-CNN localization stages without changing detector architecture or inference.
  • Across six transfers and two detectors, SimFuse3D improves all 48 AP components over Pi3DET-Net and ranks first in 47 of 48 listed comparisons.

II. RELATED WORK

Prior adaptation methods address domain discrepancy through statistics, representations, scan density, temporal refinement, or pseudo-label selection. SimFuse3D instead targets the retained pseudo-object’s box-interior observation under a single-source, LiDAR-only protocol.

  • Statistical normalization, semantic point generation, geometry-aware prototypes, spatial-temporal alignment, and beam resampling address different forms of domain discrepancy.
  • Target-supervision methods improve pseudo-label maintenance, denoising, reliability selection, multi-source refinement, or object-level point processing.
  • Several related methods assume source data are unavailable, synchronized images, external 2D detectors, or multi-source settings unlike SimFuse3D’s protocol.
  • PointNet-family descriptors are used by SimFuse3D only for source-instance retrieval, not as part of the detector.

III. METHOD

SimFuse3D adapts a self-training detector by retrieving source objects, simulating target-view geometry inside retained pseudo-boxes, and reweighting localization targets. Unmatched pseudo-labels remain available for supervision, while the detector’s other operations follow the baseline.

  • The target domain consists of unlabeled scans collected from a different sensing platform or dataset, and the detector periodically generates target pseudo-labels.
  • Each retained pseudo-label contains a seven-parameter box, class label, and detection score, with enclosed target points used to characterize its observation.
  • Box–point inconsistency occurs when retained boxes contain few, one-sided, or dimensionally mismatched points, so a useful location cue lacks an equally reliable training observation.
  • Object Memory queries enclosed target points, and accepted matches trigger Target Simulation to replace original in-box points with aligned source geometry at the target placement and viewpoint.
  • CMLR converts pseudo-label scores into bounded weights for RPN localization and R-CNN regression, while operations outside these steps follow Pi3DET-Net.

B. Object Memory

Object Memory builds a fixed retrieval store from labeled source objects, and matches target pseudo objects to compatible source instances using Point-NN similarity and neighbor voting.

  • Memory construction: Object Memory stores each labeled source object’s local LiDAR points, ground-truth box, class, metadata, and retrieval descriptor.Entries are constructed offline from labeled source scans and use fixed 256-point inputs.
  • Memory construction: A target pseudo object queries the memory using a Point-NN descriptor computed from its centered local points.The descriptor is fixed and used only for retrieval, not detector features.
  • Gated retrieval: Accepted matches receive measured source geometry and a replacement confidence of 1.0.The replacement uses a labeled source instance selected through similarity consensus.
  • Gated retrieval: Gated retrieval requires a sufficiently similar nearest source instance and a positive majority vote among qualified neighbors.When both conditions hold, the top-ranked memory entry is passed to Target Simulation and the voted class is assigned.

C. Target Simulation

Target Simulation preserves a target pseudo-box’s placement while replacing inconsistent enclosed observations with source geometry aligned to the target viewing configuration.

  • Geometry transfer: Target Simulation uses the target pseudo-box for placement and the retrieved source ground truth for object dimensions and local point geometry.Source points are not rescaled to the target pseudo-box dimensions.
  • Pose alignment: The source crop and box are yaw-aligned using source and target sensor-to-object ray directions, then translated to match the target box center and bottom face.The source height is retained during translation.
  • Pose alignment: For cross-platform pairs, pitch correction uses median source and target elevation-angle differences and is clipped by a task-specific bound.The correction changes points while the replacement box keeps source dimensions and heading adjustment.
  • Target-view filtering: The aligned source crop is filtered by target-view angular limits before points are cropped to the transformed source box.If the filtered crop is too sparse, the original target observation is retained.
  • Evaluation: Table I reports APBEV/AP3D (%) for cross-platform adaptation to Pi3DET quadruped and drone settings, with supervised target-platform results shown separately.The table compares adaptation methods under shared detector and evaluation configurations.

D. Confidence-Guided Multi-Stage Localization Reweighting

CMLR converts target pseudo-object confidence into bounded reliability weights and applies the same weight to proposal-stage and RoI-stage localization losses.

  • Motivation: CMLR addresses variable quality among retained target predictions that Target Simulation does not match.Unmatched predictions remain in training and can vary in reliability.
  • Confidence weighting: CMLR maps each retained target pseudo-object score to a bounded weight, preserving reduced supervision rather than removing low-score objects.High-score objects receive stronger supervision, while low-score objects remain with reduced gradient contribution.
  • Multi-stage reweighting: The same object weight is applied to RPN localization and R-CNN box regression, while classification terms and other unchanged losses remain in Lother.The weighted branches are the proposal and RoI localization stages.
  • Multi-stage reweighting: Normalization keeps the target loss scale stable across mini-batches with different numbers of pseudo objects.Source ground-truth objects and simulated replacements use unit weight.
  • Evaluation: Table II evaluates bidirectional adaptation between Pi3DET quadruped and drone domains using APBEV/AP3D (%) under shared detector configurations.The table identifies the best scores among listed adaptation methods.

E. Training and Inference

Adaptation starts from a source-pretrained Pi3DET-Net, refreshes target pseudo-labels during training, and retains the unchanged detector architecture and inference graph.

  • Training: The current detector acts as the pseudo-label teacher at scheduled refreshes on target training scans.Eligible predictions are paired with source instances and processed by Target Simulation; unmatched predictions remain unchanged.
  • Training: Source supervision follows the baseline, while target pseudo-object localization losses are reweighted during optimization.The reweighted branches are RPN localization and R-CNN regression, with classification losses unchanged.
  • Inference: At inference, only the adapted detector is used, without memory lookup, point replacement, or loss weighting.The inference graph is identical to the baseline.

A. Experimental Setup

The evaluation follows six cross-platform transfers, using source annotations for training and target annotations only for evaluation, with APBEV and AP3D as metrics. Experiments use PV-RCNN and Voxel R-CNN under the Pi3DET-Net protocol, including separate nuScenes→KITTI evaluations.

  • Datasets and metrics: Six cross-platform transfers use source annotations for training and target annotations only for evaluation, reporting car APBEV and AP3D over 40 recall positions.The transfers include nuScenes, Vehicle, Quadruped, Drone, and nuScenes→KITTI settings.
  • Evaluation tables: Table III evaluates nuScenes→KITTI adaptation with PV-RCNN at IoU 0.7, reporting AP percentages and bolding the best adaptation scores.
  • Evaluation tables: Table IV evaluates nuScenes→KITTI adaptation with Voxel R-CNN at IoU 0.7, omitting unreported methods and identifying the fully supervised target-platform reference.
  • Evaluation protocol: nuScenes→KITTI uses the standard car protocol at IoU 0.7, while the other transfers use IoU thresholds of 0.7 and 0.5.
  • Implementation: Experiments use PV-RCNN and Voxel R-CNN as in Pi3DET-Net, with reproduced Pi3DET-Net results and shared detector-column settings.The official Pi3DET nuScenes→KITTI result uses SECOND-IoU and is not directly comparable to these runs.
  • Ablation setup: Table V provides component ablations with Voxel R-CNN on the Vehicle-source transfers, reporting APBEV/AP3D percentages for Target Simulation and CMLR.

B. Comparison with Adaptation Methods

SimFuse3D is evaluated against source-only training and several adaptation methods across cross-platform and cross-dataset transfers. It consistently outperforms Pi3DET-Net, while ablations show complementary contributions from Target Simulation and CMLR.

  • Cross-platform comparison: SimFuse3D exceeds Pi3DET-Net in all 48 AP components and ranks first among listed adaptation methods in 47 comparisons across six transfers and two detectors.The only exception is PV-RCNN APBEV@0.7 on Vehicle→Drone, where MS3D++ obtains 64.99 versus SimFuse3D’s 64.39.
  • Cross-platform comparison: On nuScenes→Quadruped, Voxel R-CNN AP3D@0.5 rises from 55.25 to 59.63, a gain of 4.38 points.
  • Cross-dataset comparison: On nuScenes→KITTI with PV-RCNN, SimFuse3D records 85.94/76.16 versus 84.98/75.50 for ST3D++ w/ SN.The gains are 0.96 points in APBEV and 0.66 points in AP3D.
  • Reliability analysis: Figure 4 reports Q@IoU ≥0.5 (%) and sample count n for bins defined by teacher confidence and in-box point support.
  • Ablation study: Target Simulation alone improves seven of eight metrics, with the only decrease being 0.28 points in Drone APBEV@0.5.
  • Ablation study: Adding CMLR to Target Simulation improves all eight metrics over both the TS-only row and the baseline.CMLR retains potentially correct pseudo labels while reducing the influence of lower-confidence cases.

D. Pseudo-label Reliability and Point-Sparse Objects

Pseudo-label reliability improves with point support but remains imperfect, motivating separate treatment of target placement and point observations. SimFuse3D improves recall across point-support bins, especially for sparse objects.

  • Reliability analysis: Pseudo-label reliability rises from 76.2% for objects with 0–15 points to 93.4% for objects with at least 31 points.The trend is not monotonic in every interval, so confidence and point support are complementary rather than sufficient quality indicators.
  • Qualitative comparison: Figure 5 marks three sparse Vehicle→Drone vehicles recovered by SimFuse3D at IoU 0.5 but missed by Pi3DET-Net.
  • Point-sparse objects: Recall gains are positive in all 16 target-point-support entries, with the largest gain of 7.25 points at IoU 0.5 for the 0–15-point Vehicle→Drone bin.
  • Point-sparse objects: Overall Recall@0.5 increases by 4.79 points on Vehicle→Quadruped and 2.90 points on Vehicle→Drone.Gains at IoU 0.7 are smaller, indicating reduced benefit under stricter overlap.
  • Interpretation: SimFuse3D preserves useful pseudo-box placement while repairing unreliable point observations with measured source geometry and controlling retained-label supervision through CMLR.
  • Overall outcome: Across six transfers, SimFuse3D improves all 48 AP components over Pi3DET-Net by up to 4.38 points and leaves the deployed detector unchanged.Memory lookup, Target Simulation, and CMLR operate only during adaptation.
Loading 2609.04886v1…