PaperlayerView exact revision on arXiv

Source-linked AI summary

On-the-Fly Homographies Calibration for Multi-Camera Tracking

David Voihanski, Mor Sinai, Ben Zion Bobrovsky

arXiv:2609.18582v1cs.CVcs.AI

TL;DR

Precise 3D calibration is often operationally infeasible in large facilities because of privacy, bandwidth, and site-coverage constraints. The paper refines manual homographies with centroid-based Projection Optimization using live detection metadata. It reports improved geometric fidelity and tracking consistency across three datasets, with specific tracking gains reported for Dataset C.

  • Problem

    Traditional multi-camera tracking often assumes accurate 3D calibration, but privacy restrictions, bandwidth constraints, and calibration-target logistics make it difficult to deploy at large facilities.

    Source: §I · p. 1

  • Method

    Centroid-based Projection Optimization refines manual homographies using live detection metadata to align multi-camera projections in BEV space.

    Source: §I · p. 1, C. Projection Optimization (PO) · p. 3

  • Results

    Across Datasets A, B, and C, PO consistently improves geometric fidelity and tracking consistency; on Dataset C, IDF1 improves from 0.806 to 0.906 and HOTA from 0.683 to 0.810.

    Source: D. Results · p. 5

  • Takeaways & Limitations

    The authors conclude that asynchronous refinement on lightweight metadata can maintain stable global trajectories without computational overhead in settings where static 3D calibration and centralized video processing are infeasible.

    Source: §VII · p. 6

    A single homography is limited in scenes with significant elevation changes, and PO cannot refine a camera projection relative to the global frame when its view is spatially isolated from the network.

    Source: §VI · p. 6

Abstract

from arXiv · show

Precise multi-camera tracking traditionally relies on rigorous 3D site calibration, yet this requirement is often operationally impossible in large-scale deployments. Privacy regulations frequently prohibit recording video for offline calibration; limited bandwidth precludes synchronizing high-resolution streams from hundreds of cameras; and covering immense physical sites with calibration targets is logistically infeasible. We present a multi-camera homography calibration system designed to overcome these barriers through "on-the-fly" geometric refinement. Starting from coarse manual homographies, we introduce a centroid-based projection optimization (PO) that continuously aligns the ground-plane geometry using live detection streams. Because PO operates asynchronously on already-transmitted, lightweight metadata, it adds zero computational latency to the real-time tracker. This allows the system to adapt automatically to camera movements or environmental changes without human intervention. This optimized geometry feeds a multi-camera bird's-eye-view (BEV) tracker that fuses detections and unifies trajectories across zones. Crucially, by operating strictly on live anonymous metadata, our solution ensures a privacy-safe, zero-overhead, and resilient tracking pipeline that maintains global consistency in dynamic environments where static, recorded-video calibration is impossible.

I. INTRODUCTION

The introduction argues that rigorous 3D calibration is often infeasible in large deployments and presents on-the-fly projection refinement from live metadata as an alternative. The system pairs that refinement with a global BEV tracker and trajectory-level cross-camera unification.

I. INTRODUCTION

  • Privacy rules, bandwidth limits, and the logistics of placing calibration targets make traditional calibration operationally infeasible in large facilities.The cited barriers include restrictions on recording video, difficulty transmitting synchronized high-resolution streams from hundreds of cameras, and target placement across thousands of square meters.Source: §I · p. 1
  • Projection Optimization (PO) refines imperfect manual homographies using live detection metadata to align multi-camera projections in BEV space and recover from camera movement.The method operates asynchronously on lightweight clustered cross-view metadata, without adding computational latency to the real-time pipeline.Source: §I · p. 1
  • The global BEV tracker associates multi-view observations using motion and appearance cues, then fuses them into a shared geometric state.A separate trajectory-level stage merges duplicate global tracks using temporal alignment, motion dynamics, and appearance costs.Source: §I · p. 1
  • The paper's core scientific contribution is PO calibration, while its custom BEV tracker serves as an evaluation vehicle rather than a novel tracking architecture.The authors state that coordinate-mapping refinement can benefit multi-camera trackers that rely on spatial gating or trajectory alignment.Source: §I · p. 1

II. RELATED WORK

The related-work discussion contrasts calibration approaches with different deployment constraints. The system uses metadata-based cross-view processing and treats manual homographies as a warm start for online refinement.

II. RELATED WORK

  • Prior approaches rely on surveyed 3D calibration, precise planar homographies, or recorded sequences and environmental assumptions, each of which poses deployment constraints addressed by online metadata-based refinement.The discussion describes camera movement invalidating static 3D calibration, residual misalignment in manual homographies, and assumptions or video requirements in self-calibration methods.Source: §II · p. 2

III. SYSTEM OVERVIEW

  • Edge nodes transmit detection metadata rather than video, and the central tracker matches detections to shared BEV tracks, updates global states, and periodically merges duplicates.The metadata includes bounding boxes, confidence scores, and appearance embeddings.Source: §III · p. 2

A. Projection from Manual Homographies

  • Manual homographies map image points to a shared BEV plane but can be inaccurate, so the system uses them as a warm start for online optimization.The passage attributes errors to annotation accuracy, occlusions, imperfect floor plans, floor-contact ambiguity, extrapolation, and lens distortion.Source: A. Projection from Manual Homographies · p. 2

B. Matching Point Acquisition

Live cross-view detections provide correspondences for refining manual homographies. PO iteratively aligns their BEV projections around shared cluster centroids.

B. Matching Point Acquisition

  • An online appearance-based Re-ID tracker associates detections across cameras to acquire correspondences without recording video.Matching favors precision over recall: ambiguous associations are rejected to reduce the risk of corrupting spatial alignment.Source: B. Matching Point Acquisition · p. 2
  • When a camera is nudged, continuing cross-view matches let the subsequent optimization realign its projection without a site visit or manual recalibration.The passage describes this as enabling long-term resilience.Source: B. Matching Point Acquisition · p. 2
  • Spatial subsampling enforces a configurable minimum pixel distance between selected points to spread constraints across the field of view and reduce redundancy.The passage motivates subsampling by the redundancy of dense point clusters under a smooth planar homography.Source: B. Matching Point Acquisition · p. 3

C. Projection Optimization (PO)

  • Each calibration cluster groups corresponding points that represent one physical landmark observed in at least two cameras.The initial BEV centroid is computed from the points’ projections under the original manual matrices.Source: C. Projection Optimization (PO) · p. 3
  • PO minimizes summed squared distances from projected points to cluster reference centroids by iteratively updating camera matrices.An optimizer such as Adam minimizes the loss across clusters and calibration points; the centroid approach extends to any number of camera views.Source: C. Projection Optimization (PO) · p. 3

D. Multi-Camera Tracking in BEV

The BEV tracker maintains a shared pool of global tracks that cameras can update using projected detections and appearance information. It smooths image points before cross-camera association to reduce projection variance.

D. Multi-Camera Tracking in BEV

  • A single global track pool holds latent states in a shared BEV plane, where each camera can update any global track.Each camera acts as an independent measurement source.Source: D. Multi-Camera Tracking in BEV · p. 3
  • Each camera supplies detections with a 2D box, confidence, ground-contact image point, and appearance embedding; the image point is projected into BEV.The ground-contact point can be a leg point or the box bottom-center.Source: D. Multi-Camera Tracking in BEV · p. 3
  • Smoothed single-view state estimates reduce projection variance and improve cross-camera matching recall under tight Mahalanobis gates.The passage attributes the improvement to temporal stabilization and better-calibrated uncertainty covariances.Source: D. Multi-Camera Tracking in BEV · p. 3

1) Per-camera association against the shared pool:

Each camera’s BEV measurements are matched independently against the shared global-track pool. The association combines motion and appearance costs and resolves gated assignments one-to-one.

2) Per-camera association against the shared pool:

  • Each camera independently matches its BEV measurements to the same global track pool after the tracks are predicted.Matching can be parallelized across cameras.Source: 1) Per-camera association against the… · p. 3
  • Association combines Mahalanobis motion distance with Euclidean appearance-embedding distance, then applies motion and total-cost gates before Hungarian one-to-one assignment.The tracker does not use IoU-based fallback matching because it tracks a BEV point rather than a 2D box.Source: 1) Per-camera association against the… · pp. 3–4

2) Cross-camera fusion into the global state:

Concurrent camera measurements update a global track sequentially, with online adaptation of measurement noise and an exponential moving average for appearance.

3) Cross-camera fusion into the global state:

3) Track initiation and lifecycle:

Track initiation filters low-confidence detections, while a later trajectory-level stage merges fragmented cross-camera tracks using temporal overlap, motion, and appearance.

3) Track initiation and lifecycle:

  • Only detections above class-specific opening thresholds can start tracks, which move from tentative to confirmed through configured hit/miss logic.This prevents low-confidence clutter from creating global IDs.Source: 3) Track initiation and lifecycle · p. 4

E. Trajectory-Level Cross-Camera Track Unification

  • Trajectory candidates observed by the same camera at the same timestamp are barred from merging, preventing distinct neighboring objects from collapsing into one identity.The merge stage runs after per-frame fusion to address fragmentation from handovers or severe occlusions.Source: E. Trajectory-Level Cross-Camera Track… · p. 4
  • For admissible pairs, the method averages relative state differences and combined covariance over overlapping timestamps to compute a Mahalanobis motion cost.This uses the full temporal overlap rather than relying on a single frame.Source: E. Trajectory-Level Cross-Camera Track… · p. 4
  • The motion cost is combined with appearance cosine distance; thresholded pairs are greedily assigned, and a retained track absorbs its duplicate.The retained track updates its appearance embedding by a weighted average proportional to each track’s accumulated multi-view detections.Source: E. Trajectory-Level Cross-Camera Track… · p. 4

F. Implementation Details

Implementation starts with operator-annotated image-to-BEV correspondences, then refines those homographies asynchronously using synchronized detection metadata and centroid-based constraints.

F. Implementation Details

  • Operators initialize each homography with at least four widely distributed image-to-BEV anchor pairs, adjusting them through an interactive projection preview.The number of anchors depends on available scene landmarks.Source: F. Implementation Details · p. 4
  • At 5 fps, cross-camera detections are synchronized within ±100 ms, and Euclidean-distance matching uses a strict similarity threshold to prioritize precision.The live system uses a PLR-OSNet backbone to produce 2560-dimensional appearance embeddings.Source: F. Implementation Details · p. 4
  • Projected points beyond a configurable distance from their cluster centroid are rejected as outliers before optimization.This spatial gate is intended to prevent false Re-ID associations from corrupting geometric alignment.Source: F. Implementation Details · p. 4
  • PO uses Adam at a 1 × 10^-5 learning rate for 300,000 epochs, with centroid updates every 15,000 epochs frozen after 30,000.The service runs asynchronously on already-transmitted metadata, adding no computational overhead or latency to real-time tracking.Source: F. Implementation Details · p. 4

V. EXPERIMENTS

The experiments use three time-synchronized proprietary retail datasets with global-identity labels and assess both tracking and BEV alignment.

A. Setup and Datasets

  • Three proprietary retail datasets use fixed, time-synchronized cameras at 5 FPS and global identity labels shared across views.Raw videos and annotations cannot be publicly released because of privacy and business constraints.Source: A. Setup and Datasets · p. 5
  • Dataset A (Supermarket) has 11 cameras, 53 global ground-truth tracks, and a 2:20-minute duration.Cameras are installed at approximately 45° and 3 m height.Source: A. Setup and Datasets · p. 5
  • Dataset B (Mall) has 5 cameras, 80 global ground-truth tracks, and a 3:50-minute duration.Cameras are installed at approximately 40° and 4.5 m height.Source: A. Setup and Datasets · p. 5
  • Dataset C (General store) has 8 cameras, 45 global ground-truth tracks, and a 2:50-minute duration.Cameras are installed at approximately 45° and 4 m height.Source: A. Setup and Datasets · p. 5

B. Evaluation Metrics

  • Evaluation reports IDF1, HOTA, and ID switches for tracking, alongside mean ℓ2 distance from projected points to their cluster centroids for BEV alignment.The paper also lists MOTA among its tracking metrics.Source: B. Evaluation Metrics · p. 5

C. Baselines and Evaluation Strategy

The evaluation holds the tracking pipeline fixed while comparing manual homographies with the same homographies refined by PO. Results report improved alignment and tracking consistency across environments, with limitations for nonplanar sites, disconnected camera views, and severely incorrect initialization.

C. Baselines and Evaluation Strategy

  • The evaluation compares raw manual homographies with Manual + PO while keeping the tracking pipeline identical to isolate the impact of geometric calibration.The authors state that improvements over the manual baseline are attributable to better cross-camera BEV alignment.Source: C. Baselines and Evaluation Strategy · p. 5

D. Results

  • Across all three environments, PO improves geometric fidelity and tracking consistency over raw manual homographies.Table I presents the quantitative tracking and alignment results.Source: D. Results · p. 5, Table I · p. 5
  • In Dataset B, PO reduces BEV alignment error from 42 cm to 14 cm, ID switches from 182 to 38, and raises IDF1 from 0.717 to 0.892.The passage attributes the high manual-baseline switch count to identity mixing in the crowded mall setting.Source: D. Results · p. 5
  • In Dataset C, PO reduces BEV alignment error from 42 cm to 24 cm, while IDF1 rises from 0.806 to 0.906 and HOTA from 0.683 to 0.810.ID switches remain relatively stable, changing from 25 to 24; the passage links the gains to fewer dropped associations and broken trajectories.Source: D. Results · p. 5
  • Figure 2 depicts projections from two cameras converging toward a shared cluster centroid, while Figure 3 shows global BEV consistency across the video timeline before and after optimization.The optimization is described as independent of time and global identity.Source: Figure 2 · p. 6, Figure 3 · p. 6

VI. DISCUSSION AND LIMITATIONS

  • The homography model assumes targets lie on a dominant ground plane, so a single homography cannot accurately map foot points in scenes with substantial elevation changes.Ramps and split-level flooring are named examples.Source: §VI · p. 6
  • PO requires a connected graph of overlapping views; an isolated camera cannot be refined relative to the global frame and falls back to its manual calibration.In disjoint cases, cross-zone associations rely more on visual Re-ID embeddings than spatial gating.Source: §VI · p. 6
  • PO is a local refinement step, not a global geometric solver, and may reach a degenerate local minimum from grossly incorrect initial homographies.The passage identifies flipped axes and massive scale errors as examples.Source: §VI · p. 6

VII. CONCLUSION

Centroid-anchored Projection Optimization refines coarse manual homographies using live anonymous detection metadata. The asynchronous refinement removes projection scatter without computational overhead, supporting stable global trajectories in dynamic environments where static 3D calibration and centralized video processing are infeasible.

VII. CONCLUSION

  • Live anonymous detection metadata lets centroid-anchored Projection Optimization refine coarse manual homographies, eliminate projection scatter, restore spatial distance gating, and reduce global identity switches.The refinement builds a robust spatial coordinate system from coarse, manual planar homographies.Source: §VII · p. 6
  • Asynchronous optimization on lightweight metadata decouples geometric self-healing from real-time tracking, incurring zero computational overhead.Source: §VII · p. 6
  • The resulting privacy-compliant pipeline maintains stable global trajectories in dynamic environments where static 3D calibration and centralized video processing are prohibitive or infeasible.Source: §VII · p. 6

Related papers

Loading 2609.18582v1…