Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

Introduction

• With feature extractors frozen across domains, Syn2RealTrack exposes calibration, camera overlap, and object counts as configuration choices that shift trust from simulation-specific geometry to appearance when cues are unreliable.The system treats the synthetic-to-real gap as three localized interfaces rather than one domain-adaptation problem.Source: §1 · p. 2
• AnyCalib [35] estimates omitted lens distortion; views are grouped by Unified Camera Model parameter ξ, footpoints are undistorted while provided geometry is retained, and fisheye-only zones are masked.The masking avoids invalid matches.Source: §1 · p. 3
• 0.06 HOTA is added over single-view association by fusing views with visibility-weighted part similarity and propagating single-camera identities before geometric assignment rejects inconsistent world placements.The visibility-weighted fusion can abstain on occluded parts rather than guessing.Source: §1 · p. 3
• A closed-form estimate from calibration and ankle ground points replaces the synthetic person-height prior without depth or extra supervision; an RGB-only Depth Anything 3 (DA3) point cloud [15] refines footprint and yaw while preserving identity and class-specific size.Source: §1 · p. 3
• In suitable closed-world scenes, exact per-class counts are paired with a causal filter that requires visible-ankle confirmation after reprojection into covering cameras, suppressing unsupported boxes at negligible association cost.The filter yields a detection-driven gain.Source: §1 · p. 3

Synthetic-to-Real Generalization and Calibration

• Syn2RealTrack estimates omitted lens distortion before ground-plane projection, rather than changing feature extractors or the supplied intrinsics and extrinsics.Prior transfer work uses view-consistent augmentation, robust BEV features, perspective-invariant rendering, or alternative calibration cues [6,10,17,42,35,37,39,45]. AnyCalib estimates distortion from 31 sampled frames (Fig. 2).Source: Synthetic-to-Real Generalization and… · p. 3, Figure 2 · p. 4

Geometry, Visibility, and Multi-View Association

• Class-specific anchors and ankle keypoints recover ground contact and person height, while RGB point clouds refine footprint and yaw [4, 14, 15, 19, 26, 41].The framework also uses trajectory-derived yaw and monocular person height; Depth Anything 3 guides footprint and yaw refinement (Fig. 2).Source: Synthetic-to-Real Generalization and… · p. 3, Figure 2 · p. 4
• Unlike association methods using visible-part features, occupancy, target-count, camera models, or learned BEV and trajectory relationships [21,22,28,29,34,40,43,47], Syn2RealTrack excludes parts occluded in either view.Its sequential association also uses local-identity memory, motion, and a bounded gallery; single-camera identities take precedence over geometric matching, and unsupported person boxes are suppressed when no external class census is available.Source: Synthetic-to-Real Generalization and… · p. 3, Figure 2 · p. 4

Multi-View Fusion and Tracking

• Earlier multi-view systems aggregate dense BEV features or exchange uncalibrated-view features [5,8,9,30,36], while prior AI City systems rely on benchmark depth and calibration [13, 23, 31, 38].The 2026 protocol adds real scenes and withholds depth at inference [32]; Syn2RealTrack instead preserves modular per-view perception and fuses sparse object observations late (Fig. 2).Source: Synthetic-to-Real Generalization and… · p. 3, Figure 2 · p. 4

3 Methodology

• Online tracking across C static calibrated RGB cameras emits oriented, class-labeled 3D boxes with time- and view-consistent identities for seven warehouse-scene classes [32, 38].The classes are Person, Forklift, NovaCarter, Transporter, FourierGR1T2, AgilityDigit, and PalletTruck.Source: §3 · p. 4

3.1 Distortion-Aware Camera Grouping

• AnyCalib [35] analyzes 31 evenly sampled frames per camera; UCM scores ξ ≤0.3 identify reference cameras, while the rest are fisheye candidates, and dataset pinhole calibration remains the BEV reference geometry.Reprojection displacements grow toward image borders for fisheye candidates.Source: §3.1 · pp. 5–6

3.2 2D Object Detection

• RF-DETR 2x-large [25] is trained for 200 epochs at 1920-pixel resolution on scene-matched training data, with NMS IoU 0.5 and a 0.1 confidence threshold at inference.Training scenes include AI City [31] and MTMMC [44], including real scenes for otherwise absent AgilityDigit and FourierGR1T2 categories.Source: §3.2 · p. 6

3.3 Single-View Re-Identification and Tracking

• ViTPose++ [46] keypoints drive six-region KPR [28] descriptors and appearance-augmented IoU/Kalman tracking, producing local tracklets whose identities cue global association.KPR is trained on 155 scene-specific identities for 110 epochs; its part-based features help disambiguate crossings and brief occlusions.Source: §3.3 · p. 6

3.4 Multi-View Tracking

• Multi-view tracking anchors per-camera tracklets to ground positions, merges observations across views within each frame, then binds merged observations to persistent global tracks.This three-step flow is stated for the per-camera tracklets and globally consistent identities.Source: §3.4 · p. 6

3.4.1 Class-Adaptive Ground-Contact Anchoring.

• Class-adaptive anchoring uses ankle keypoints for humanoids, box centers for low flat robots, and frame-position-dependent top–bottom anchors for tall or self-occluding objects.When ankles fail, leg-chain extrapolation is used only with enough confident keypoints; for lower-image boxes, a projected class height allows the anchor to descend below a clipped bottom edge.Source: §3.4.1 · p. 7

3.4.2 Cross-View Association on the Ground Plane

• Cross-view observations are clustered within class using BEV-distance gates and, for re-identification classes, visibility-weighted part cosine distance that ignores parts occluded in either view.Single-linkage merging rejects clusters that already contain the same camera, and fused descriptors combine visible parts without imputing occluded ones.Source: §3.4.2 · p. 8

3.4.3 Global Multi-View Tracking

• Global tracks first carry forward matching local identities, then assign remaining observations using motion-gated BEV distance and nearest-gallery appearance, solved by Hungarian assignment.The motion gate expands with missing age to cover distance at maximum plausible speed; for re-identification classes, pairs also face an appearance gate.Source: §3.4.3 · p. 8

Trajectory Lifecycle Dynamics.

• Unmatched observations spawn tracks only beyond a minimum birth separation, while unmatched tracks dead-reckon until age exceeds Amax; matched tracks are smoothed and refresh descriptors and age.Matched-track velocity is updated from displacement since the previous sighting, normalized by elapsed frames.Source: Trajectory Lifecycle Dynamics · p. 9
• With known class counts, fixed-cardinality mode coasts capped tracks until each class reaches Nk, assigns contested slots to the most confident observations, and reclaims stale slots.The prior is exact in simulation, while real scenes use adaptive tracking with all gates active; dropping Pass-B gates force-assigns each track to its best observation.Source: Trajectory Lifecycle Dynamics · p. 9
• After the sequence, an offline pass interpolates gaps shorter than Gmax between sightings and leaves longer gaps unfilled.This is the pipeline’s only acausal step and is applied once per sequence before submission.Source: Trajectory Lifecycle Dynamics · p. 9

3.4.4 The Synthetic-to-Real Configuration Surface

• Synthetic scenes rely on exact calibration, broad overlap, and known counts, whereas real scenes need smoothing and active appearance gating because calibration is estimated, overlap is sparse, and counts are unknown.The pipeline shifts trust between geometry and appearance rather than retraining the feature extractor.Source: §3.4.4 · pp. 9–10

3.5 Height and Yaw Angle Estimation

• Each track position is lifted to a 3D box using class extent priors, while yaw comes from the derivative of a quadratic fit to the previous 45 positions.The priors are per-class medians of warehouse ground-truth box scales; the box centroid is placed at half its height.Source: §3.5 · p. 10
• Person height is estimated per frame by inverting the calibrated projection from an ankle-midpoint ground point to the detection’s top row, without learned depth or 3D ground truth.The independent ground point avoids the circularity of using a height-dependent anchor.Source: §3.5 · p. 10
• Height samples are accepted only when detection, keypoint, back-projection, zone, and height-range checks pass; rejected samples are not clamped, leaving a small positive bias from ankle elevation.A track uses the cross-camera median of accepted estimates, or falls back to its class height prior when none are accepted.Source: §3.5 · p. 10

3.6 Point-Cloud-Guided 3D Box Refinement

• DA3 [15] builds a fused metric point cloud from synchronized, pose-conditioned camera images to refine footprint and yaw while preserving tracked identity and size.Ground-plane projection can misplace boxes when image points are not true ground contacts; Track 1 withholds depth at inference [32].Source: §3.6 · p. 10

Synthetic Scene Refinement.

• For synthetic scenes, fixed-shape objects receive prior-sized footprint fits from filtered BEV point density, while dynamic-shape objects retain their size prior and receive limited yaw or jump corrections.Fixed-shape boxes revert to tracker output when point support is weak.Source: Synthetic Scene Refinement · p. 11
• For real-world scenes, seeded coarse-to-fine mean-shift selects each person track’s densest BEV mode, while a causal temporal state bridges short losses and suppresses jitter.The tracker seeds the first frame, later frames reuse the refined state, and DA3 corrects person centers while keeping class-size priors; parked vehicles retain fixed hand-verified poses.Source: Synthetic Scene Refinement · p. 11

3.7 Filtering Predictions Based on the BEV Map

• The BEV filter checks emitted boxes against cameras covering their ground locations, addressing phantom boxes from per-class caps or weak detections without changing surviving boxes.If no camera covers a box, the filter instead uses cameras onto which it geometrically projects.Source: §3.7 · pp. 11–12
• Person boxes survive only when a covering camera shows a visible ankle; unconfirmed person boxes are removed, while boxes of other classes pass unchanged.The visibility test uses ankle keypoint scores against a threshold in the selected camera views.Source: §3.7 · p. 12
• In fisheye-only zones, objects retain their fisheye BEV positions; in overlapping camera regions, fisheye-versus-reference disagreement is measured in ground-plane meters.These zones are marked as manually annotated BEV polygons, and the measured disagreement is the error consumed by BEV tracking.Source: §3.7 · p. 12

4.1 Dataset and Evaluation Metrics

• The RGB-only evaluation used synchronized 1080p/30-fps videos, calibrated cameras, top-down maps, and 2D/3D annotations, assessing expanded synthetic and hidden real scenes with HOTA, DetA, AssA, and LocA.Higher metric values are better.Source: §4.1 · p. 13

4.2 Quantitative Results

• 52.0118% 3D HOTA placed SKKU-AL-T1 second on the official Track 1 leaderboard, 4.53 points behind EVA and 14.00 ahead of third place.The reported gaps indicate a much smaller gap to first place than to the rest of the field.Source: §4.2 · p. 13

4.3 Ablation Study

• Over the 50.8078 baseline, gIoU and dIoU alone added +0.31 and +0.15 HOTA, while combining them without appearance reached 50.8204.These are stage-wise evaluations from single evaluation-server submissions, not independent ablation effects.Source: §4.3 · p. 13
• Re-identification and pose guidance reached 51.2906 and 51.3205 HOTA, respectively; together they produced the best single-view configuration at 51.8568 (+1.05 over baseline).The passage identifies the combined configuration as the final setting.Source: §4.3 · pp. 13–14
• At 1920 inference resolution, RF-DETR beat YOLOv26 at both training resolutions, and matching training to inference resolution added 0.19 and 0.22 HOTA.The detector backbones differed by only 0.09 HOTA, so detector choice was not decisive.Source: §4.3 · p. 13
• The full pipeline raised HOTA from 50.8078 to 52.0118 (+1.20), with single-view association contributing +1.05 (87.1%).Cross-view clustering added +0.06 HOTA, monocular height +0.07, and BEV-map filtering +0.02; overall DetA rose from 42.41 to 45.31 while AssA stayed within 56.46–56.52 once all association cues were enabled.Source: §4.3 · p. 14, Table 7 · p. 14
• Monocular calibration-based person-height estimation improved HOTA by +0.13 over the constant prior without depth or supervision.It adjusts box height and centroid rather than identity, leaving association essentially unaffected; residual scene-dependent ground-plane error remains.Source: §4.3 · p. 14, Table 5 · p. 14
• Disabling BEV-map prediction filtering cost 0.13 HOTA; cumulatively, filtering added +0.23 DetA while lowering AssA by only 0.01.The causal per-frame test suppresses phantom boxes at camera-unsupported locations without terminating identities held elsewhere; a final coverage check added +0.0018 HOTA at negligible overhead.Source: §4.3 · p. 14, Table 6 · p. 14, Table 7 · p. 14

5 Conclusion

• 52.0118% 3D HOTA and second place in AI City Challenge 2026 Track 1 mark Syn2RealTrack’s reported result.The system addresses synthetic-to-real gaps in calibration, shape priors, and object census.Source: §5 · p. 14
• Syn2RealTrack remains sensitive to calibration and RGB-derived depth, severe occlusion, frequent entry/exit, and closed-world census assumptions.Source: §5 · p. 14
• Future work will target uncertainty-aware calibration and depth, stronger temporal prediction, and domain adaptation.Source: §5 · p. 14

Dataset Preprocessing

• Preprocessing combines shape-statistics lookup, scenario-specific training-set selection, and removal of redundant, small, or invalid 2D boxes.These steps address fallback object typing, training efficiency and generalization, and false-positive detections.Source: §A.1 · pp. 19–20

Lookup Table Mapping

• Per-object-type minimum, maximum, and mean sizes seed default shapes for static, fixed-shape, and dynamic-shape objects when later matching fails.Static objects stay in place; fixed-shape objects retain their shape while moving or at rest; dynamic-shape objects change shape as they move.Source: §A.1 · p. 19

Scenario-Specific Training-Set Selection

• Training data is selected for each test scenario to reduce the number of boxes while preserving detection reliability and including categories absent from other scenes.For real-world scenes, MTMMC [44] supplements the AI City training dataset [31] to increase training-data amount and diversity.Source: §A.1 · pp. 19–20

Filtering Small or Invalid 2D Bounding Boxes

• Redundant boxes and boxes covering unseen or invalid objects are removed, as are boxes with excessively small spatial extent.The filtering process is visualized in Fig. 1.Source: Figure 1 · p. 20, §A.1 · p. 20

2D Object Detection Training and Inference

• RF-DETR [25] 2x-large is trained for 200 epochs at 1920-pixel resolution with batch size 4, then inferred at the same resolution with batch size 16.Inference uses non-maximum suppression at IoU 0.5 and discards detections with confidence below 0.1.Source: §A.1 · p. 20

Pose Estimation

• ViTPose++ [46] extracts keypoints from detected person regions using a lightweight decoder and task-agnostic and task-specific feed-forward networks.The resulting pose information supports subsequent feature extraction, association, and refinement.Source: B Re-Identification and Pose Estimation · pp. 20–21

Re-Identification

• KPR [28] divides people into six pose-defined regions for region-level features that emphasize human cues and reduce background interference under occlusion.Although KPR supports negative keypoints for occluded or unreliable regions, this pipeline uses only positive keypoints in training and inference.Source: B Re-Identification and Pose Estimation · p. 21
• Training and validation were merged and relabeled into 155 identities; the model trained for 110 epochs in approximately 3 hours and 7 minutes.The unification addresses the dataset’s scene-specific identity labels.Source: B Re-Identification and Pose Estimation · p. 21

• Fig. 4 shows objects retaining their identities through occlusions and close interactions, with bounding boxes following them across frames in a single camera view.Source: C 2D Single-View Tracking · p. 22

D Point Cloud Construction

• Pose-conditioned DA3 [15] estimates depth from synchronized camera images and calibration parameters, which are back-projected into a shared world coordinate system and merged as colored point clouds.The method uses the updated DA3 Nested Giant-Large 1.1 checkpoint with any-view geometry and metric-depth scaling; the resulting clouds support BEV visualizations of two real-world scenes.Source: D Point Cloud Construction · p. 22