Source-linked AI summary
Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking
Duong Nguyen-Ngoc Tran, Ngoc Doan-Minh Huynh, Cu Quoc Le, Hoang-Khang Nguyen, Long Hoang Pham, Huy-Hung Nguyen, Quoc Pham-Nam Ho, Trinh Le Ba Khanh, Chi Dai Tran, Duong Khac Vu, Son Hong Phan, Hyung-Min Jeon, Jae Wook Jeon
TL;DR
Synthetic training leaves multi-camera 3D tracking exposed to calibration, shape-prior, and object-count mismatches in real warehouses. Syn2RealTrack addresses these gaps locally without retraining feature extractors and achieves 52.0118% 3D HOTA on AI City Challenge 2026 Track 1.
Problem
Synthetic training leaves multi-camera 3D perception mismatched with physical warehouse irregularities, while dense 3D boxes and cross-camera identities remain expensive to annotate.
Method
Syn2RealTrack separates calibration, shape-prior, and object-census gaps, reallocating trust between geometry and appearance while keeping feature extractors frozen.
Results
52.0118% 3D HOTA, placing Syn2RealTrack second in AI City Challenge 2026 Track 1.
Takeaways & Limitations
The system supports domain adaptation by shifting reliance from simulation-specific geometry toward appearance when real-scene cues are unreliable.
Takeaways & Limitations
The method remains sensitive to calibration, RGB-derived depth, severe occlusion, frequent entry and exit, and closed-world census assumptions.
Abstract
from arXiv · showhide
Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The resulting synthetic-to-real gap, which corrupts ground-plane localization and cross-camera identity association, is usually treated as one deficiency for a single domain-adaptation module to absorb; we argue instead that it enters the pipeline at three separable points: the camera calibration, the object shape prior, and the assumption that the object census is known, each admitting a different local remedy. Our online pipeline, Syn2RealTrack, follows this decomposition: lens distortion is recovered from images alone under a calibration that provides none, detections are fused across views by a visibility-weighted part-based descriptor that abstains on occluded parts rather than guessing, person height is measured in closed form from calibration instead of copied from a synthetic prior, and a closed-world cardinality prior is paired with a causal filter that removes the phantom boxes the prior manufactures. The system therefore adapts by reallocating trust between geometry and appearance without retraining a feature extractor. On the AI City Challenge 2026 Track~1 evaluation server it reaches a 3D Higher Order Tracking Accuracy (HOTA) of 52.0118%. The code will be released at https://github.com/SKKUAutoLab/aic26_mc3dp
1 Introduction
Syn2RealTrack addresses online multi-camera 3D perception in warehouse scenes, where occlusions, handovers, and costly supervision create a synthetic-to-real gap. It separates this gap across calibration, cross-view fusion, object geometry, and object-count assumptions, applying local remedies without retraining feature extractors.
- Motivation: Multi-camera rigs recover world-frame position, footprint, orientation, and persistent identity despite occlusions and camera handovers, but dense 3D boxes and cross-camera identities are expensive to obtain.The task is motivated by warehouse targets disappearing behind obstructions and the limitations of single-camera range estimation.
- Localizing the synthetic-to-real gap: The pipeline localizes the synthetic-to-real gap by exposing calibration, camera overlap, and object counts as configuration variables while feature extractors remain frozen.This configuration surface shifts trust away from simulation-specific geometry when real-scene cues are unreliable.
- Distortion-aware camera grouping: AnyCalib estimates omitted lens distortion, camera grouping uses Unified Camera Model parameter ξ, and fisheye-only zones are masked to avoid invalid matches.Reference and fisheye views are split, footpoints are undistorted, and provided geometry is retained.
- Cross-view fusion that can abstain: Visibility-weighted part similarity fuses grounded detections while abstaining on unreliable regions, propagates single-camera identities, and adds 0.06 HOTA over single-view association.Geometric assignment rejects inconsistent world placements after identity propagation.
- Measured rather than synthetic geometry: Closed-form calibration-based height replaces the synthetic person-height prior, while DA3 point clouds refine footprint and yaw without depth or extra supervision.Identity and class-specific size are preserved during refinement.
- A contained closed-world prior: Exact per-class counts are used only in suitable closed-world scenes, and a causal visible-ankle reprojection filter suppresses unsupported boxes at negligible association cost.The filter requires confirmation from covering cameras and produces a detection-driven gain.
2 Related Work
Prior work addresses synthetic-to-real transfer through appearance- and geometry-aware representations or camera calibration, whereas Syn2RealTrack keeps its feature extractors and provided camera parameters fixed. Its framework combines distortion estimation, per-view perception, and visibility-aware appearance tracking.
- Synthetic-to-Real Generalization and Calibration: Prior methods reduce multi-camera domain shift using view-consistent augmentation, robust BEV features, perspective-invariant rendering, or calibration from correspondences, perspective cues, and pixel rays.The paper characterizes domain shift as both photometric and geometric.
- Synthetic-to-Real Generalization and Calibration: Syn2RealTrack keeps its feature extractors and provided intrinsics and extrinsics fixed rather than retraining or recalibrating them.This design reallocates adaptation to other pipeline components.
- Data preprocessing: AnyCalib estimates per-camera distortion from 31 sampled frames, while the Unified Camera Model separates reference views (ξ ≤0.3) from fisheye candidates.Reference views are shown in blue and fisheye candidates in red.
- Per-view perception: RF-DETR, ViTPose++, and KPR supply detections, keypoints, and visibility-aware part descriptors for an appearance–IoU Kalman tracker.The tracker links identities using per-view perception outputs.
3 Methodology … 3.4 Multi-View Tracking
Syn2RealTrack decomposes online 3D multi-view multi-target tracking into calibration, shape-prior, and cardinality issues, then reallocates trust between geometry and appearance across synthetic and real warehouse scenes. Its pipeline combines distortion analysis, class-adaptive ground anchoring, visibility-aware association, and lifecycle controls for globally consistent identities.
- 3 Methodology: The system tracks seven object classes across C static calibrated RGB cameras, emitting oriented, class-labeled 3D boxes with time- and view-consistent identities.The addressed setting is online 3D multi-view multi-target tracking in warehouse scenes.
- 3.1 Distortion-Aware Camera Grouping: AnyCalib estimates dense rays and field-of-view maps to fit pinhole, Brown–Conrady, and UCM models, grouping distortion-affected cameras while preserving dataset calibration for BEV projection.The UCM parameter ξ serves as a fisheye score for distortion analysis and grouping only.
- 3.2 2D Object Detection: The detector uses category dimension statistics to classify shapes as static, fixed-shape, or dynamic-shape, supplying fallback shapes and removing redundant, tiny, unseen, or invalid boxes.Training scenes are matched to test scenarios, including AgilityDigit and FourierGR1T2, using real scenes from AI City and MTMMC.
- 3.3 Single-View Re-Identification and Tracking: Pretrained ViTPose++ extracts keypoints, while pose-aware descriptors and motion link detections within each camera into local tracklets before cross-view reasoning.The keypoint representation is designed to remain useful under changes in pose, scale, and occlusion.
- 3.4 Multi-View Tracking: Multi-view tracking anchors detections to the ground plane using class-specific strategies: skeleton for humanoids, box center for low robots, and top–bottom correction for tall or self-occluding objects.Skeleton anchoring uses ankle keypoints when reliable and extrapolates only when sufficient pose confidence exists.
- 3.4 Multi-View Tracking: Cross-view observations are clustered using BEV distance and visibility-weighted appearance, ignoring parts occluded in either view and fusing complementary visible descriptors without imputation.Global tracks combine predicted ground position with gallery appearance, then Hungarian assignment minimizes the resulting gated costs.
- 3.4 Multi-View Tracking: Track lifecycle rules prevent duplicate births, dead-reckon unmatched tracks, retire stale tracks, and apply a closed-world per-class cardinality prior when counts are known.Capped tracks coast rather than retire so each class count converges to its specified N_k; geometry and appearance trust then shift by scene conditions.
3.5 Height and Yaw Angle Estimation
The method lifts 2D tracks into 3D boxes by estimating yaw from recent motion and replacing the synthetic person-height prior with a calibration-based monocular estimate. Height samples undergo strict validity checks before track-level aggregation, while other classes retain calibrated warehouse-derived size priors.
- Yaw Angle Estimation: Yaw ψ is estimated by fitting a second-degree polynomial to the previous 45 track positions, differentiating at the current point, and taking the arctangent.
- Height Estimation: Class box dimensions use warehouse ground-truth medians as width, length, and height priors, with a manual estimate for the unmatched class.The lifted box sets Z = h/2 so a floor-standing object's centroid lies at half its height.
- Height Estimation: Person height is recovered per frame by inverting the camera projection using the detected top row, without learned depth or 3D ground truth.
- Height Estimation: Height samples are accepted only when detections satisfy truncation, confidence, ankle, pose, back-projection, zone, and range constraints; invalid samples are rejected rather than clamped.Accepted estimates are aggregated for each track using the cross-camera median, with a small positive bias remaining from ankle elevation.
3.6 Point-Cloud-Guided 3D Box Refinement
Point-cloud-guided refinement addresses coarse planar position and yaw errors after multi-view tracking, which can reduce 3D IoU despite correct identity association. In synthetic scenes, DA3 refines fixed-shape object footprints and yaw using tracked-box point clouds while reverting to tracker outputs when support is weak.
- Motivation: Post-tracking ground-plane projection can yield coarse centers and planar position or yaw errors despite correct association.Failure cases include elevated Transporters, occluded feet, distortion, and calibration residuals.
- Synthetic Scene Refinement: Synthetic-scene refinement uses DA3 point clouds to refine footprint and yaw while retaining valid class-size priors.Reliable calibration and controlled categories keep the class-size priors of Eq. (6) valid, so refinement targets only footprint and yaw.
- Synthetic Scene Refinement: For fixed-shape objects, tracked-box points are projected into a BEV density grid, filtered, and fitted near the tracked pose.Floor, background, and weak components are removed before fitting the prior-sized footprint; weak support triggers reversion to the tracker output.
3.7 Filtering Predictions Based on the BEV Map
The section filters emitted boxes against camera visibility in the BEV map to suppress unsupported phantom detections. The causal per-frame check removes unconfirmed person boxes while leaving surviving boxes and other classes unchanged.
- Per-class caps or weak detections can emit unsupported phantom boxes that inflate false positives.The filter tests each emitted box against cameras whose ground-plane coverage includes it.
- Definition of Camera Visibility Maps: Each camera’s visibility map records observed ground regions by intersecting its projected field of view with annotated BEV coverage polygons.Rasterization produces, for each BEV cell, the cameras that see that ground position.
- Object Verification and Filtering: Unconfirmed person boxes are removed, while boxes from all other classes pass unchanged after the causal per-frame check.Surviving boxes retain their identity, size, and pose; the check suppresses forced-assignment phantoms.
- Fisheye BEV Consistency: Fisheye-only ground regions are manually annotated as BEV polygons because they lack a reference-camera anchor.These zones are marked as dashed violet regions in the floor map.
4 Experiments & Discussion
Experiments evaluate Syn2RealTrack under the RGB-only MTMC Tracking 2026 protocol using HOTA, DetA, AssA, and LocA. The system ranks second on the official Track 1 leaderboard at 52.0118% 3D HOTA, while ablations show gains from configuration choices and monocular height estimation.
- Evaluation Protocol: The evaluation uses synchronized 1080p/30-fps RGB videos, calibrated cameras, top-down maps, and 2D/3D annotations under an RGB-only protocol.Experiments cover expanded synthetic and hidden real scenes using HOTA, DetA, AssA, and LocA, with higher values preferred.
- Leaderboard Result: 52.0118% 3D HOTA places SKKU-AL-T1 second on the official Track 1 leaderboard.The team trails first-place EVA by 4.53 points and leads third place by 14.00.
- Ablation Study: Monocular person-height estimation improves HOTA monotonically by +0.13 over the constant-prior baseline.Calibration alone surpasses the synthetic prior without depth or supervision, adjusting box height and centroid rather than identity.
5 Conclusion
Syn2RealTrack addresses synthetic-to-real gaps in calibration, shape priors, and object census, achieving 52.0118% 3D HOTA and second place in AI City Challenge 2026 Track 1. Its remaining sensitivities motivate future work on uncertainty-aware calibration and depth, stronger temporal prediction, and domain adaptation.
- 52.0118% 3D HOTA and second place were achieved in AI City Challenge 2026 Track 1.
- The system remains sensitive to calibration and RGB-derived depth, severe occlusion, frequent entry/exit, and closed-world census assumptions.
- Future work will target uncertainty-aware calibration and depth, stronger temporal prediction, and domain adaptation.
A.1 Dataset Preprocessing
The preprocessing framework filters redundant, small, or invalid 2D annotations and selects training data for each test scenario. It also incorporates additional real-world data and object-shape statistics to support reliable downstream evaluation.
- 2D Bounding Box Filtering: Redundant, small, or invalid 2D bounding boxes are filtered to reduce false-positive detections and improve detection reliability.The raw annotations contain redundant or invalid boxes, and removed boxes are visualized in Fig. 1.
- Scenario-Specific Training-Set Selection: Scenario-specific training-set selection reduces the number of training bounding boxes while preserving detection accuracy and reliability.Selecting data according to each test scenario is intended to shorten training and improve generalization to test scenarios.
- Object Shape Statistics: Object-shape statistics provide default shapes so evaluation remains meaningful when later matching steps fail.The framework distinguishes static, fixed-shape, and dynamic-shape objects before seeding each object with a default shape.
B Re-Identification and Pose Estimation
The method uses pretrained ViTPose++ to estimate human pose from detected person regions. Its keypoint outputs support subsequent feature extraction, association, and refinement.
- Pose Estimation: Pretrained ViTPose++ extracts body keypoints from detected person regions for pose estimation.The model uses a plain, non-hierarchical vision transformer encoder with a lightweight keypoint decoder.
- Pose Estimation: ViTPose++ combines task-agnostic and task-specific feed-forward networks to handle heterogeneous body-keypoint categories across pose-estimation tasks.This knowledge factorization accompanies the model’s scalable keypoint-decoding architecture.
- Pose Estimation: The pretrained model supplies pose information for subsequent feature extraction, cross-camera association, and refinement.Pose estimation is therefore integrated before these downstream tracking stages.
C 2D Single-View Tracking
The 2D single-view tracker maintains object identities across frames, including during occlusions and close interactions, while its bounding boxes continue to follow the objects.
- C 2D Single-View Tracking: The method tracks multiple objects through occlusions and close interactions, preserving each identity across frames while bounding boxes follow the objects.The result is visualized in Fig. 4.
D Point Cloud Construction
Syn2RealTrack constructs shared point-cloud guidance at each synchronized time step by estimating metric-scaled depth from all camera images and back-projecting it with calibration parameters. The resulting DA3-based point clouds support scene-level BEV visualization.
- Depth Anything 3 estimates depth jointly from synchronized multi-camera images because test-time depth maps are unavailable.The method uses DA3’s pose-conditioned multi-view setting with the provided intrinsic and extrinsic parameters, K and E.
- The updated DA3 Nested Giant-Large 1.1 checkpoint combines any-view geometry with metric-depth scaling.
- Estimated depths are back-projected into a shared world coordinate frame to generate point-cloud guidance.
- Scene-level BEV visualizations are constructed from DA3-estimated point clouds for Warehouse 026 and Warehouse 027.