Source-linked AI summary
Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection
Pu Luo, Cong Xu, Yumei Li, Kexin Zhang, Licheng Jiao, Wenping Ma, Lingling Li
TL;DR
Real roadside LiDAR is costly to collect and annotate, and synthetic scans differ from real scans in several observation and class-distribution properties. This solution combines density-aligned synthetic training with complementary detectors and class-aware fusion, alongside an independently optimized realism branch. It reports a Combined Score of 0.4692, Detection Score of 0.1797, and Realism Score of 0.9035.
Problem
Real roadside LiDAR collection and annotation are costly at scale, and synthetic and real scans differ in point sampling, object morphology, and class frequency.
Source: §1 · p. 1Method
The system combines a density-aligned, multi-source synthetic training pool with role-based DSVT and PointPillars models, class-aware fusion, and a separate realism branch.
Source: §1 · p. 1, §3.1 · p. 3, §4.1 · p. 3Results
The final system achieves a Combined Score of 0.4692 and Detection Score of 0.1797, with a Realism Score of 0.9035.
Source: §7 · pp. 4–5Takeaways & Limitations
The reported practical lesson is that low-level observation alignment and error-aware model complementarity can be more valuable than simply enlarging the training set or ensemble.
Source: §7 · p. 5Detector weights are trained only with synthetic data; public real data outside the forbidden-frame list are used only for permitted validation, model selection, and post-processing analysis.
Source: §1 · p. 1
Abstract
from arXiv · showhide
We present our solution to the LUMPI track of the UCF UrbanTwin Sim2Real LiDAR Challenge at the 6th DriveX Workshop, ECCV 2026. The detector must be trained only on synthetic data and is evaluated on 50 held-out real LiDAR frames; a separate 50-frame synthetic submission is evaluated for point-cloud realism. Our method addresses the Sim2Real gap at three levels. First, we align synthetic scans to the 50k-point test density and build a 30k-record training pool using UT-LUMPI geometry, RangeLDM-based sampling diversification, rare-class copy-paste, and pedestrian-oriented augmentation. Second, complementary DSVT detectors and Car/Bus PointPillars specialists are trained under the same synthetic-only constraint. Third, predictions are integrated by class-aware routing, asymmetric agreement fusion, constrained residual-recall supplementation, class-coverage auditing, and selective box-size calibration. The realism branch is optimized independently with radial-density matching, weak affine calibration, and calibrated set mixing. The final submission obtains a Combined Score of 0.4692, a Detection Score of 0.1797, a Realism Score of 0.9035, and 3D mAP@0.5 of 0.1258.
1 Introduction
Roadside LiDAR offers stable infrastructure-side observations, but real data are costly to collect and annotate at scale. This work studies synthetic-to-real transfer in a setting where geometric similarity alone does not resolve differences in scan statistics and class frequencies.
1 Introduction
- Real roadside LiDAR collection and annotation are costly at scale, while UT-LUMPI provides a synthetic counterpart generated from a digital twin of LUMPI’s target environment.LUMPI is a multi-perspective roadside sensing dataset recorded at a large intersection in Hannover, Germany.Source: §1 · p. 1
- Synthetic and real scans differ in point count, radial density, local sampling, object morphology, and class frequency, affecting voxel statistics and per-object point support.The eight evaluation classes also have different frequencies and error modes.Source: §1 · p. 1
- The solution combines density alignment and multi-source synthetic training with role-based DSVT and specialized PointPillars detectors, class-aware fusion, and a separately optimized realism branch.The detector is trained only on synthetic data; permitted public real data are reserved for validation, model selection, and post-processing analysis under the challenge protocol.Source: §1 · p. 1, Figure 1 · p. 2
2 Task and System Overview
The LUMPI track evaluates 3D detection and synthetic point-cloud realism. The system develops these objectives in separate branches, with detection using density-aligned data and complementary detectors while the realism branch optimizes global point-cloud statistics.
2 Task and System Overview
- Detection is evaluated by class-averaged 3D AP at IoU 0.5 with KITTI 40-point interpolation, while AP@0.7 is also reported.The realism evaluation compares 50 submitted synthetic frames with 50 held-out real reference frames after masking and uniformly sampling 10,000 points per side.Source: §2 · p. 2
- Realism is measured using normalized Chamfer Distance, Maximum Mean Discrepancy, Earth Mover’s Distance, and Fréchet Point-cloud Distance.The two sets are evaluated after masking to [−40, 40] × [−40, 40] × [−5, 5] m.Source: §2 · p. 2
- The upper branch builds density-aligned synthetic data, trains complementary detectors, performs class-aware fusion, and calibrates selected box geometries; the lower branch independently optimizes 50 realism frames.Detection benefits from category and observation diversity, whereas realism rewards global distributional similarity.Source: §2 · p. 2
3 Density-Aligned Multi-source Synthetic Training
The training pool starts from UT-LUMPI geometry and adds density alignment, sampling diversification, rare-class copy-paste, and pedestrian-oriented augmentation. These sources form an approximately 30k-record synthetic pool for training under the 50k-point test condition.
3.1 Four Data Sources
- Resampling synthetic scans to exactly 50,000 points aligns training with test density; the same ensemble scores 0.1464 mAP at original ∼215k density versus 0.1320 after 50k resampling.The higher-density validation score overestimates performance under the released test condition, so subsequent selection and post-processing use the 50k protocol.Source: §3.1 · p. 2
- RangeLDM-based scans add a sparser, less regular sampling domain while retaining UT-LUMPI as a geometry-stable source.RangeLDM generates LiDAR in range view using latent diffusion.Source: §3.1 · p. 2
- Rare-class copy-paste augments Bicycle, Motorcycle, Bus, Truck, and Van instances, which otherwise contribute few training examples despite equal class weight in class-averaged mAP.Instances are pasted into valid positions after random pose perturbation.Source: §3.1 · p. 3
- Person instances receive dedicated scale, orientation, and position perturbations to diversify small, sparsely sampled pedestrian observations.Source: §3.1 · p. 3
- The approximately 30k-record pool combines an 8k density-aligned base stream, 6k RangeLDM scans, 2k copy-paste scans, 10k UT-LUMPI scans, and 4k pedestrian-oriented scans.Synthetic GT database sampling further balances class frequency; increasing rare-class sampling beyond the final setting gave no additional gain, with geometric and observation diversity becoming the dominant limitation.Source: §3.1 · p. 3
4 Complementary Detectors and Class-aware Fusion
The detector suite uses synthetic-only models trained on different data distributions and architectures. Class-aware routing and asymmetric fusion address class-dependent reliability and correlated errors, while residual recovery, coverage checks, and selective calibration handle remaining gaps.
4.1 Role-based Detector Learning
- The synthetic-only detector suite uses DSVT models emphasizing density-aligned, mixed, pedestrian/sparse, or sampling-diversified data, plus Car and Bus PointPillars specialists.Three stable late-stage mixed-data DSVT members provide training-trajectory diversity, and all models predict the eight evaluation categories.Source: §4.1 · p. 3
4.2 Why Class-aware Fusion
- Uniform ensembling can add duplicates and false positives because class reliability varies and DSVT models make correlated errors; validation selects a primary and auxiliary source per class for complementarity.For matched same-class boxes, fusion keeps primary geometry and updates confidence; unmatched candidates are retained, and primary/auxiliary order is validated.Source: §4.2 · p. 3
4.3 Residual Recall, Coverage Audit, and Geometry Calibration
- Residual specialists are admitted only for independent correct detections, with duplicate suppression and confidence discounting; coverage auditing addresses missing Van validation ground truth, and geometry calibration is limited to validated class-specific biases.A 1,000-frame synthetic validation set with 4,282 Van boxes supports relative Van model selection; dimension calibration helps Bicycle and Person but is disabled for Van after degrading validation performance.Source: §4.3 · p. 3
5 Independent Realism Optimization
The independently optimized realism branch matches radial density, tests weak affine corrections, and mixes calibrated variants. It reaches a Realism Score of 0.9035, with most remaining penalty attributed to EMD.
5 Independent Realism Optimization
- The 50 realism frames were reweighted by radial distance, weak axis-wise scale and translation candidates were tested, and two calibrated variants were mixed at approximately 75%:25%.The radial-density adjustment aimed to better match the near/far profile of permitted public LUMPI training data without distorting the digital-twin layout.Source: §5 · p. 4
- Realism Score 0.9035 was achieved, with raw metrics CD=2.1805, MMD=0.000725, EMD=1.857, and FPD=0.3395.Most remaining penalty came from EMD; additional global rotation, translation, and anisotropic-scaling searches did not materially reduce it.Source: §5 · p. 4
- Independent 10k-point subsamples from permitted public real data produced EMD values around 0.9–1.0, indicating substantial sampling variability at this scale.The team stopped low-return global calibration and prioritized detection.Source: §5 · p. 4
6 Experiments
Experiments use permitted real validation for selection and post-processing while keeping detector weights synthetic-only. The main detection improvements are reported under the target-consistent 50k protocol, and the hidden-test result is summarized separately.
6.1 Validation Protocol and Ablation
- Validation Set 1 contains 150 labeled public real frames for model selection and post-processing, while Validation Set 2 contains 1,000 synthetic frames and supplies Van coverage.Real labels do not update detector weights.Source: §6.1 · p. 4
- Table 1 reports the main detection improvements under the target-consistent 50k protocol; the 0.1464 score on original high-density scans is only a density diagnostic, not the method baseline.The validation protocol uses permitted real frames outside the forbidden list.Source: §6.1 · p. 4
- The improvement is not explained by repeatedly increasing ensemble size; stages instead address density mismatch, class reliability, box bias, correlated errors, and residual misses or classes.These stages target distinct problems.Source: §6.1 · p. 4
6.2 Final Result
- The hidden-test result is summarized in Table 2, with Detection Score consistent with normalized raw mAP and Combined Score consistent with the challenge weighting.The supplied text does not state Table 2’s numeric results.Source: §6.2 · p. 4
7 Conclusion
The system combines density-aligned synthetic training, complementary detectors, class-aware fusion, and a separately optimized realism branch. It achieves a Combined Score of 0.4692, Detection Score of 0.1797, and Realism Score of 0.9035.
7 Conclusion
- The final system achieves a Combined Score of 0.4692 with Detection Score 0.1797 and Realism Score 0.9035.The conclusion presents low-level observation alignment and error-aware model complementarity as more valuable than simply enlarging the training set or ensemble.Source: §7 · pp. 4–5
- A 30k-record synthetic pool and role-based DSVT models and PointPillars specialists support class-aware routing, agreement fusion, residual-recall supplementation, coverage auditing, and selective geometry calibration.The realism branch is optimized separately and does not constrain detector training.Source: §7 · p. 4
- The system is centered on target-density alignment and class-dependent complementarity.Its practical lesson is that these can be more valuable than simply enlarging either the training set or the ensemble.Source: §7 · pp. 4–5