PaperlayerView exact revision on arXiv

Source-linked AI summary

CAT-Free: Multi-View Pedestrian Localization without Calibration, Annotations, or Target-Scene Training via Adaptive Geometric Filtering

Taigo Sakai, Hiroki Kouno, Naoki Kato, Kazuhiro Hotta

arXiv:2609.34302v2cs.CV

TL;DR

Multi-camera pedestrian localization can require camera calibration, position annotations, or target-scene training, making deployment preparation substantial. CAT-Free estimates camera configuration from synchronized RGB video and uses adaptive geometric filters to select cross-camera position estimates. It reports benchmark results without those scene-specific inputs, transfers without retuning, and relates estimated uncertainty to localization reliability.

  • Problem

    Deploying multi-camera pedestrian localization in a new environment can require measured camera positions and orientations, position annotations, or target-scene training.

    Source: §1 · p. 1

  • Method

    CAT-Free estimates camera configuration from synchronized RGB video, combines pedestrian observations across cameras, and applies adaptive geometric filters with thresholds estimated from each sequence.

    Source: §1 · p. 2

  • Results

    82.5, 84.5, and 65.7 MODA on WildTrack, MultiviewX, and GMVD, respectively, without supplied calibration, position annotations, or target-scene training.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Across changes in camera availability and time, localization uncertainty is strongly correlated with MODA (r = −0.98), providing a label-free indicator of unreliable localization conditions.

    Source: §4.2 · p. 6

    Reported results measure relative rather than absolute metric localization because evaluation uses global similarity alignment fitted to ground-truth camera centers; absolute metric deployment needs an independent source of global scale and coordinate alignment.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multi-camera pedestrian localization is useful for wide-area monitoring in public and commercial spaces. However, deploying these systems often requires considerable setup for each new environment. Existing methods typically require camera calibration, position annotations, or target-scene training. CAT-Free removes all three requirements. It uses synchronized RGB video as its only scene-specific input. Camera configuration is estimated directly from the video. Pedestrian locations are then estimated by combining observations from multiple cameras. Automatic camera estimation is not always accurate. This can produce unreliable pedestrian locations. CAT-Free therefore introduces two adaptive geometric filters. They remove unreliable position estimates. Their thresholds are estimated from each input sequence. CAT-Free achieves 82.5, 84.5, and 65.7 MODA on WildTrack, MultiviewX, and GMVD. It uses no supplied calibration, position annotations, or target-scene training. Published methods using such scene-specific information report 88.2--95.0 MODA on WildTrack and 83.9--96.5 on MultiviewX under their respective protocols. CAT-Free also transfers without retuning. It reaches 74.9 MODA on four additional sequences and 78.6 on an unseen 8-camera installation. Finally, localization uncertainty predicts MODA with $r=-0.98$. This provides a label-free estimate of localization reliability.

1. Introduction

CAT-Free addresses the setup burden of multi-camera localization by using synchronized RGB video without per-installation calibration, annotations, or target-scene training. It combines adaptive geometric filtering with camera-estimation and multi-camera observations, reporting benchmark performance, transfer, and a label-free reliability signal.

1. Introduction

  • New-environment deployment can require measured camera positions and orientations, position annotations, or target-scene training, increasing preparation cost when installations change.Source: §1 · p. 1
  • Calibration-free methods can still require new-environment training, while label-free methods may still need known camera parameters or adaptation to the new environment.The cited examples include CaMuViD, UMPD, DCHM, and MVUDA.Source: §1 · pp. 1–2
  • CAT-Free uses synchronized RGB video to estimate camera configuration and combine pedestrian observations across cameras, with no model training or fine-tuning for the new environment.Camera positions and orientations are estimated using a pretrained 3D reconstruction model.Source: §1 · p. 2
  • Two adaptive geometric filters reject unreliable positions using sequence-estimated thresholds for localization uncertainty and consistency with the recovered ground.The thresholds are estimated automatically from unlabeled input video, avoiding scene-specific threshold tuning.Source: §1 · p. 2
  • 82.5, 84.5, and 65.7 MODA on WildTrack, MultiviewX, and GMVD, respectively, without supplied calibration, position annotations, or target-scene training.The reported accuracy approaches published methods that use scene-specific information.Source: §1 · p. 2
  • 74.9 ± 3.2 MODA across four additional sequences and 78.6 ± 6.5 MODA on an unseen 8-camera installation, without retuning.The same localization and tracking settings are used across datasets.Source: §1 · p. 2
  • Localization uncertainty predicts MODA across changes in camera availability and time (r = −0.98), providing a label-free reliability indicator.The uncertainty is estimated from localization rather than ground-truth pedestrian positions.Source: §1 · p. 2

2. Related Work

Prior multi-view localization and tracking approaches reduce different parts of the deployment burden, but often retain requirements for calibration, annotations, or target-scene training. CAT-Free combines calibration-free camera reconstruction with cross-camera pedestrian observations and requires neither known camera parameters nor new-environment position annotations or training.

Supervised multi-view detection and tracking

  • Supervised multi-view methods and tracking extensions achieve high accuracy but require calibrated cameras and position annotations from each new environment.CaMuViD removes supplied calibration but still requires annotated data and training for each environment.Source: §2 · p. 2

Label-free learning and domain adaptation

  • UMPD and DCHM reduce manual annotation, while MVUDA adapts to new camera environments; these approaches still require known camera parameters or additional new-environment training.Source: §2 · p. 2

Calibration-free 3D reconstruction

  • CAT-Free estimates pedestrian positions on the ground by combining cross-camera observations, unlike VGGT-based tracking from reconstructed depth, and requires neither known camera parameters nor new-environment training or annotations.VGGT and DUSt3R can estimate camera positions, orientations, and scene structure from uncalibrated images.Source: §2 · p. 2

3. Method

CAT-Free estimates camera geometry and ground structure from synchronized images, then matches and combines pedestrian observations across views. Adaptive geometric filters select positions before temporal linking forms ground-plane trajectories.

3. Method

  • Fixed-weight pretrained models estimate camera configuration and ground, detect pedestrians and produce rays, and extract appearance features; matched detections are filtered before trajectory linking.The four stages use no ground truth, and both geometric-filter thresholds are estimated from the input sequence.Source: Figure 2 · p. 3

3.1. Camera configuration and ground estimation

  • Camera positions, orientations, and scene structure are estimated with VGGT, then refined using multi-camera pedestrian reference points and bundle adjustment; RANSAC estimates a ground plane from the reconstructed point cloud.The refinement retains the result with the smallest median geometric error, and image-derived checks reject wall-like plane estimates.Source: §3.1 · p. 3

3.2. Pedestrian observations and ray reliability

  • YOLO11x detects pedestrians and estimates pose, while OSNet extracts appearance features; ankle visibility and bounding-box height inform ray reliability weights.Confirmed ankle keypoints receive stronger pose-based reliability, while small boxes receive reduced weight.Source: §3.2 · p. 4

3.3. Matching pedestrians across cameras

  • Cross-camera groups are merged only when they contain no duplicate camera and their rays fit a common position; appearance similarity orders candidate merges, with a geometry-based exception for singleton pairs.Singleton merges may relax the appearance threshold when the pair is substantially more geometrically consistent than alternatives.Source: §3.3 · p. 4

3.4. Robust 3D position estimation

  • Pedestrian positions are estimated by reliability-weighted ray fitting with iteratively reweighted least squares and Huber loss, reducing the influence of large errors.Rays farther than 3δ are treated as outliers and can be excluded when at least two rays remain; positions are estimated freely in 3D before ground projection.Source: §3.4 · p. 4

3.5. Selecting pedestrian positions

  • Position selection first rejects geometrically inconsistent rays, then applies uncertainty and ground-plane consistency filters with sequence-estimated thresholds.The uncertainty threshold is estimated automatically using Otsu’s method in log space; temporal support, duplicate removal, and recovery further refine candidates.Source: §3.5 · pp. 4–5

3.6. Trajectories

  • Accepted positions are linked across consecutive frames using spatial proximity and OSNet appearance; short tracks are removed and some tracks are extended by up to two frames under multi-camera support conditions.Extension also requires no competing track in the area and sufficiently similar appearance for at least one supporting detection.Source: §3.6 · p. 5

4. Experiments

CAT-Free is evaluated for multi-camera pedestrian localization without supplied calibration, annotations, or target-scene training. The experiments report benchmark results, transfer, and ablations of its localization and filtering choices.

4.1. Datasets and protocol

  • Evaluation aligns predictions to ground truth with a 7-DoF Umeyama transform fitted to camera centers, so reported results measure relative rather than absolute metric localization.The alignment is performed after inference and is not used for localization, filtering, or tracking.Source: §4 · p. 6

4.2. Main results

  • 82.5, 84.5, and 65.7 MODA on WildTrack, MultiviewX, and GMVD, respectively, without supplied calibration, position annotations, or target-scene training.On WildTrack, CAT-Free is 1.7 points below DCHM; on MultiviewX, it is 6.1 points above DCHM and 0.6 points above MVDet, though protocols differ.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • With settings fixed, four later GMVD sequences reach 74.9 ± 3.2 MODA, and an unseen synthetic 8-camera installation reaches 78.6 ± 6.5 MODA after camera re-estimation from RGB.These transfer results use different evaluation conditions from the development sequences.Source: §4.2 · p. 6
  • Across eight settings varying camera availability or time, localization uncertainty correlates with MODA at r = −0.98, indicating likely unreliable localization without ground-truth pedestrian positions.The uncertainty measures how precisely a position is determined from the available camera views.Source: §4.2 · p. 6

4.3. Ablations

  • Multi-camera 3D position estimation improves MODA over the best two-ray estimate by +3.4, +9.3, and +5.1 on WildTrack, MultiviewX, and GMVD, respectively.Ground-constraining the estimated 3D position reduces mean MODA by 1.9 points from Q and 8.6 points from the full ordered-filter variant; using ground-truth rather than estimated camera configuration changes MODA by −2.5, −3.3, and −0.5 points.Source: §4.3 · pp. 6–8
  • The uncertainty filter removes poorly constrained positions before the ground-consistency filter rejects candidates inconsistent with the recovered ground; this ordering improves mean MODA by +2.49 over Q.Among candidates passing the uncertainty filter, ground distance yields AUC 0.952 on WildTrack and 0.788 on MultiviewX; on WildTrack it removes about 90% of false positives while reducing annotated-pedestrian coverage by 1.8 percentage points.Source: §4.3 · pp. 7–8

5. Limitations

The reported results have limits in absolute metric localization and processing mode.

5. Limitations

  • Evaluation requires global similarity alignment fitted to ground-truth camera centers, so results measure relative localization; absolute metric deployment needs independent global scale and coordinate alignment.Source: §5 · p. 8
  • The current implementation operates offline and uses the complete processed sequence for camera estimation, tracking, and adaptive-threshold estimation.Source: §5 · p. 8

6. Conclusion

CAT-Free localizes and tracks pedestrians from synchronized RGB video without supplied calibration, position annotations, or target-scene training. Results include benchmark performance, transfer without retuning, and uncertainty as a label-free reliability indicator.

6. Conclusion

  • CAT-Free reaches 82.5, 84.5, and 65.7 MODA on WildTrack, MultiviewX, and GMVD, transfers to later GMVD sequences and an unseen 8-camera installation, and has uncertainty correlated with MODA at r = −0.98.The transfer settings are fixed, with camera configuration re-estimated from RGB for the unseen installation.Source: §6 · p. 8

A. Additional Protocol and Results

The runtime analysis reports processing time and identifies the dominant stage and its scaling behavior.

A. Additional Protocol and Results

  • Cross-view association takes 61–71% of processing time and is the only stage that scales quadratically with detections per frame.Detection-through-tracking takes 135 s on WildTrack, 237 s on MultiviewX, and 158–230 s on a 100-frame GMVD sequence, using one RTX A6000 workstation process with cached detections, rig, and reconstruction.Source: A. Additional Protocol and Results · p. 10

B. Effects of Camera Geometry and Time

Localization uncertainty rises as camera geometry or viewing conditions worsen, and it aligns with MODA across camera-removal and temporal changes. The scoring stage ranks hypotheses using standardized image-derived signals.

B. Effects of Camera Geometry and Time

  • 91 MODA points separate the two-camera and seven-camera settings, with the two-camera run below zero because false positives outnumber recovered people.The sweep varies the available camera geometry while holding frames, detections, camera parameters, ground plane, and fixed constants unchanged.Source: B. Effects of Camera Geometry and Time · p. 10
  • Across camera-removal and temporal settings, log σ predicts MODA with Pearson r = −0.98, despite co-visibility alone failing to reconcile the temporal results.The worst held-out block has σ = 36.9 cm and MODA = 33.37, compared with σ = 20.3 cm and MODA = 66.81 at five cameras.Source: B. Effects of Camera Geometry and Time · p. 10
  • The temporal analysis computes σ at ground-truth positions, while camera-removal settings use a fixed camera prefix and time windows vary where people stand.The analysis reports that people stand farther from the cameras early in the recording; its camera diagnostic excludes estimated camera parameters as the primary cause.Source: B. Effects of Camera Geometry and Time · p. 10

C. The Scoring Stage

  • The scoring stage standardizes seven image-derived signals over the sequence, weights them by Otsu separability, and uses the combined score to rank hypotheses for NMS.Hypotheses above the Otsu threshold are kept; weaker ones are retained only when outside the NMS radius of previously accepted hypotheses.Source: C. The Scoring Stage · p. 10

D. Evaluation Alignment and Scoring Ablation

Evaluation-time alignment has little effect on mean MODA under leave-one-camera-out refits. Removing the auxiliary score also leaves development and unseen-installation results essentially unchanged.

D. Evaluation Alignment and Scoring Ablation

  • Mean MODA under leave-one-camera-out alignment refits is 81.87 on WildTrack, 84.56 on MultiviewX, and 64.64 on GMVD sequence 1.These means are within 0.6, 0.0, and 1.1 points, respectively, of the values using all cameras.Source: D. Evaluation Alignment and Scoring… · p. 10
  • Removing the auxiliary scoring stage changes development MODA by only +0.10, +0.07, and +0.02, and changes unseen-installation MODA by +0.02 on average.The development runs retain the same true-positive counts and have one fewer false positive; the score is retained to keep the post-development configuration fixed.Source: D. Evaluation Alignment and Scoring… · p. 10
  • Table 5 distinguishes fixed downstream settings shared across datasets from settings estimated for each input sequence.Its scope is downstream of detection and camera estimation; E denotes reconstructed scene extent and σθ angular noise standard deviation.Source: Table 5 · p. 11

E. Incremental and Additional Ablations

The ablations identify geometric filtering and trajectory choices that affect localization, while further analyses characterize out-of-grid predictions and filter failure cases. The broader-impact and reproducibility discussion bounds the evaluation and implementation.

E. Incremental and Additional Ablations

  • Adding ground-plane consistency is the first configuration to improve all three datasets over Q after weighted view support is replaced.The view-support replacement alone yields the largest single GMVD gain (+4.75 MODA) but lowers WildTrack by 2.52 points in isolation.Source: E. Incremental and Additional Ablations · p. 11
  • Trajectory processing adds 2.8 MODA points over per-frame positions, almost entirely through track extension, while robust-inlier refitting adds 0.63, 0.20, and 0.12 MODA points.On GMVD, retaining short static tracks adds 44 true positives and one false positive; it lowers IDF1 by 0.36 there and 0.46 on WildTrack.Source: E. Incremental and Additional Ablations · p. 11
  • On MultiviewX, 288 of 1,639 predictions lie outside the annotated grid, and counting them as false positives lowers MODA from 84.54 to 65.26.Their median distance to the nearest annotated person is 5.0 m, and requiring three recovered cameras removes only 6 of the 288 predictions.Source: E. Incremental and Additional Ablations · p. 11
  • The uncertainty filter alone passes 93–94% of simulated incorrect pairs; the ground-plane filter discriminates more strongly as person separation increases.At 2 m separation about 30% of incorrect pairs survive both filters, versus 70% at 0.5 m; at 1 m, 72% survive for perpendicular separation versus 41% when aligned with the camera line. Real sequences are more favorable because most hypotheses use more than two cameras and first pass angular consistency filtering.Source: E. Incremental and Additional Ablations · p. 12

G. Broader Impact

  • The authors identify lower barriers to unconsented surveillance as a potential consequence, and state that evaluation uses only public research datasets recorded for that purpose.The pipeline localizes positions rather than identifying people, but its re-identification embedding is biometric and tracks are personal data in most jurisdictions.Source: G. Broader Impact · pp. 12–13

H. Reproducibility

  • The implementation does not provide a single command from RGB input to final output; its reproduction scripts replay detection through tracking from cached outputs.The cached inputs include detections, camera parameters, and reconstruction outputs.Source: H. Reproducibility · p. 13

I. Prespecified Evaluation Protocols

The post-development evaluations were prespecified, then test transfer across time and to an unseen camera installation under distinct protocols. The supplied tables also characterize how localization uncertainty tracks difficulty.

I. Prespecified Evaluation Protocols

  • Each post-development evaluation was specified before any prediction for that experiment was scored.The operative content of the prespecified files is summarized in the protocols that follow.Source: §I · p. 13
  • The WildTrack evaluation uses 360 frames unused during development but retains the same scene and camera rig, so it tests transfer across time rather than camera estimation on an unseen installation.Thresholds are re-estimated on the evaluation frames, with no subset selection or threshold changes after evaluation.Source: §I · p. 14
  • The four additional GMVD sequences share the development sequence’s camera installation and reuse its camera estimates, testing transfer across time rather than to an unseen scene or rig.The method’s constants, filters, stopping rule, and trajectory rule remain unchanged; the experiment starts from cached detections, camera parameters, and reconstruction.Source: §I · pp. 14–15
  • Across eight camera-removal and temporal settings, log σ predicts MODA with Pearson r = −0.98 and Spearman −0.93 (p < 0.001).The table compares median σ at annotated positions with measured MODA, pooling a camera-removal sweep and two time windows at all seven cameras.Source: Table 11 · p. 15
  • The unseen-installation evaluation covers all five sequences on an 8-camera configuration, with camera refinement selecting iteration 20 by triangulation residual without ground truth.The method remains unchanged and thresholds are estimated per sequence; the protocol does not test an unseen scene or end-to-end estimation of the reconstruction.Source: §I · p. 15

J. Stage-Level Error Analysis

The stage-level analysis locates missed pedestrians in candidate generation, selection, tracking, and detection, and tests how matching cues and filter thresholds affect results. It also distinguishes labeled outcome analysis from the uncertainty statistic that can be computed before labels are available.

J. Stage-Level Error Analysis

  • Ground truth is used in these analyses only to label outcomes.Source: J. Stage-Level Error Analysis · p. 15
  • Among people without an initial candidate, GMVD has 415 visible people matched by fewer than two detections, 230 assigned to another candidate by greedy partition, and 185 not localized within 0.5 m even by correct rays.On MultiviewX, greedy partition accounts for 34 of 59 such people; on WildTrack, angular-consistency failures and greedy partition dominate.Source: J. Stage-Level Error Analysis · p. 15
  • Adding short-term temporal candidates raises coverage from 92.2% to 93.3% but lowers MODA from 80.99 to 62.92.On earlier WildTrack hypotheses, the correct partner was closer under OSNet appearance in 9.4% of cases, DINOv2 in 47.4%, per-camera motion in 45.3%, and estimated body height in 47.6%.Source: J. Stage-Level Error Analysis · p. 16
  • 31.8% of visible annotated GMVD boxes lack a detection at IoU ≥0.5 with detector confidence 0.30; lowering confidence to 0.10 recovers 8% of them.Misses rise from 20.6% at maximum interperson IoU below 0.1 to 44.2% at 0.3–0.5; enlarged overlapping tiles change recall from 68.5% to 68.8%.Source: J. Stage-Level Error Analysis · p. 16
  • The best fixed filter quantile differs by dataset, while log-Otsu is within 0.6 MODA points of the best fixed choice on WildTrack and MultiviewX.Table 15 compares per-sequence thresholds with a fixed quantile applied to every dataset.Source: Table 15 · p. 17
  • The filter order selected on development sequences performs best on every sequence in the unseen 8-camera installation evaluation.The comparison uses the GMVD configuration 2 setup and the variants from Table 3.Source: Table 16 · p. 17
  • 5.4%, 3.7%, and 22.6% of annotated people are missed during candidate generation on WildTrack, MultiviewX, and GMVD; selection rejects a further 3.0%, 5.2%, and 7.0%.Candidate generation is the larger measured loss on WildTrack and GMVD, while selection is larger on MultiviewX.Source: J. Stage-Level Error Analysis · p. 15, Table 18 · p. 17
Loading 2609.34302v2…