PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniGlimpse: Asymmetric Identity Memory for Online Multi-View Tracking

Rangya Zhang, Jiaping Xiao, Yujing Shang, Yuhang Zhang, Mingsheng Li, Mir Feroskhan

arXiv:2610.36785v1cs.CV

TL;DR

Multi-view tracking must match identities across complementary views without relying on offline scene-specific training or camera calibration, even as online adaptation can amplify association errors. OmniGlimpse separates adaptive association proposals from canonical-space identity verification and permits only verified observations to update persistent memory or drive adaptation after bootstrapping. It improves identity metrics over evaluated online baselines on both M3Track and MvMHAT, while comparisons with offline-trained references are mixed.

  • Problem

    Multi-view identity matching must handle uncertain cross-view evidence and changing appearances, while offline appearance training or geometric priors can constrain deployment and reused assignments can reinforce errors.

    Source: §1 · p. 2

  • Method

    OmniGlimpse uses target-specific adaptive spaces to propose associations and frozen-encoder canonical identity distributions to verify them, gating persistent updates on verification after bootstrapping.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    OmniGlimpse outperforms evaluated online baselines in IDF1 and CVIDF1 on both benchmarks; on M3Track, IDF1 rises from .7424 to .8333 and CVIDF1 from .3918 to .6129 over OCLReID.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The results support online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.

    Source: §5 · p. 9

    Targets receive bounding-box initialization at first valid appearance in each view, with no further frame-level identity annotations afterward.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Multi-view tracking extends visual coverage beyond a single camera, providing complementary observations when targets are occluded or leave the view. This raises a question: how can observations from different views be matched to the correct identities without offline scene-specific training or camera calibration? Progress toward this goal remains limited because many offline formulations assume such deployment-specific knowledge, while online adaptation can amplify association errors. We study this problem as online multi-view identity maintenance (OMIM): targets are initialized once per view, with no further identity labels provided after each initialization. To this end, we introduce OmniGlimpse, an asymmetric propose-verify framework that separates adaptive association from stable identity consolidation. At the representation level, short-term local and cross-view evidence is mapped into target-specific adaptive spaces to propose associations, whereas long-term identity distributions are maintained for verification in a canonical space defined by a frozen encoder. At the update level, short-term evidence may guide current associations, whereas, after bootstrapping, only verified observations may update persistent memory or drive adaptation. Under a unified target-specified evaluation protocol, OmniGlimpse improves over the best evaluated online baseline for each metric by 9.1 IDF1 and 22.1 CVIDF1 points on M3Track. The corresponding gains on MvMHAT are 20.3 and 26.2 points. Despite using no offline scene-specific training, it outperforms offline-trained references in IDF1 and remains competitive in cross-view association. These results support an approach to online identity maintenance that adapts to changing appearances while consolidating reliable evidence into stable identity anchors.

1 INTRODUCTION

Multi-view tracking offers complementary observations but makes identity matching harder across appearance changes and similar-looking distractors. OmniGlimpse addresses deployment constraints and error-prone adaptation with an asymmetric propose–verify framework for online identity maintenance.

1 INTRODUCTION

  • Offline appearance training and geometric priors can constrain deployment, while uncertain cross-view evidence and reused identity assignments create additional association challenges.An incorrect pseudo-label can alter the representation and bias later matches; the tracker must judge which observations are reliable enough to change persistent identity state.Source: §1 · p. 2
  • OmniGlimpse maintains targets initialized once per view without further identity labels, separating adaptive association from stable identity consolidation.Short-term evidence proposes associations; after bootstrapping, only observations consistent with long-term identity distributions may update persistent memory and target-specific adaptive spaces.Source: §1 · p. 2
  • Short-term local and cross-view evidence is matched in adaptive spaces, while a frozen-encoder canonical space anchors long-term identity verification.The framework combines cross-view evidence with target-specific online adaptation without offline scene-specific training or camera calibration.Source: §1 · p. 2
  • On M3Track and MvMHAT, OmniGlimpse outperforms evaluated online baselines in IDF1 and CVIDF1 and remains competitive with offline-trained references.Ablations support online adapter learning and verification-gated updates within the complete system.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses online identity tracking, multi-view association, and adaptation or memory, but each line leaves distinct deployment or reliability concerns. OmniGlimpse connects adaptive association with canonical-space verification to govern persistent updates.

2 RELATED WORK

  • Online MOT uses motion and appearance cues; ByteTrack uses low-confidence detections to recover fragmented trajectories, while OC-SORT improves motion association under occlusion and non-linear motion.Source: §2 · p. 2
  • Target-specific adaptation methods respond to appearance change, but reusing recent assignments as supervision can let an incorrect match persist through later updates.Following Closely Ye et al. (2022) and OCLReID Ye et al. (2024) update target appearance models; OCLReID also uses short- and long-term experience to optimize its feature extractor online.Source: §2 · p. 3
  • Multi-view trackers associate through appearance and trajectory cues or exploit 3D and geometric information, while some depend on offline appearance learning or calibration.The cited approaches include MvMHAT Gan et al. (2021), ReST Cheng et al. (2023a), and OCMCTrack Specker (2024).Source: §2 · p. 3
  • Tracking and ReID systems use memories or prototypes to stabilize matching, but matching evidence may be too unreliable to supervise updates and representation changes can make stored features inconsistent.OmniGlimpse addresses these concerns with adaptive association and canonical-space verification governing persistent memory updates and learning after bootstrapping.Source: §2 · p. 3

3 METHOD

OmniGlimpse maintains target identities from initialization-only supervision through adaptive association, long-term verification, and verification-gated updates. A frozen encoder defines the canonical feature space, while target-specific adapters support changing appearances.

3.1 PROBLEM SETUP AND OVERVIEW

  • Targets are initialized with bounding boxes at first valid appearance in each view, with no later frame-level identity annotations; ∅ denotes absent, occluded, or insufficiently identifiable targets.The system infers each target's corresponding observation using only observations and initialization information available up to the current time.Source: §3.1 · p. 3

3.2 ASSOCIATION AND VERIFICATION

  • Short-term local and received cross-view features pass through the current local adapter for association, while long-term identity distributions verify in canonical features from a shared frozen encoder.Memory banks store and exchange canonical features rather than local adapter outputs, preserving a consistent coordinate system as adapters learn.Source: §3.2 · p. 4, Figure 2 · p. 4
  • Candidate associations use canonical-space initialization references and adaptive ridge classifiers; memory support assists when current bindings are missing, uncertain, or contested.Association scores remain proposals: selection alone does not authorize memory consolidation or learning.Source: §3.2 · pp. 4–5
  • Long-term verification compares candidates with per-target Gaussian identity prototypes in normalized canonical global features, accounting for multiple appearance modes and dimension-wise uncertainty.Verification uses the distribution available before admitting the current observation; before maturity, its result is unavailable, and a failed check blocks new or recovered associations from being committed.Source: §3.2 · p. 5

3.3 VERIFICATION-GATED UPDATES

  • Only admitted observations may update persistent memory or supply positive training samples; bootstrap can populate distributions but cannot authorize ordinary pseudo-label training.Admission decisions use current assignments and pre-update distributions before memory or parameter updates, and updated distributions are used for verification at the next step.Source: §3.3 · pp. 5–6

3.4 DISTRIBUTION-ANCHORED ADAPTATION

  • Target-specific residual adapters learn from admitted same-view and cross-view triplets, excluding unassigned detections and unresolved overlaps as negatives.Cross-view positives bring different appearances together in the receiver's adaptive space; a penalty against cached canonical Gaussian prototypes discourages departures from the canonical identity distribution.Source: §3.4 · p. 6
  • During adaptation, gradients update only local residual branches; refreshed features and Ridge classifiers support the next association step using only samples with training permission.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments test identity maintenance without post-initialization labels and report gains over evaluated online baselines on both benchmarks, alongside component controls and a qualitative example.

4.1 EXPERIMENTAL SETUP

  • Targets receive a bounding-box initialization at first annotated appearance in each view, with no later frame-level identity labels; M3Track and MvMHAT each provide 19 paired-view test instances or 13 test groups, respectively.All methods are evaluated on the same selected identities; offline target-selection procedures are detailed in Appendix B.2.Source: §4.1 · p. 7
  • Online baselines run independently per view without cross-view information exchange; single-view identity maintenance and cross-view association are evaluated with separate metric sets.Single-view metrics include IDF1 and identity switches (IDs); cross-view metrics include CVIDF1 and MDA.Source: §4.1 · pp. 7–8

4.2 OVERALL RESULTS

  • On M3Track, IDF1 rises from .7424 to .8333 and CVIDF1 from .3918 to .6129 over OCLReID, the strongest online baseline on both metrics.Against CrossMOT, IDF1 rises from .7945 to .8333 and CVIDF1 from .5786 to .6129, while MDA falls from .5635 to .5503.Source: §4.2 · p. 8
  • On MvMHAT, IDF1 rises from .5292 for OCLReID to .7323, while CVIDF1 rises from .3068 for ByteTrack to .5685.Against CrossMOT, IDF1 is higher (.7323 versus .6901), but CVIDF1 is lower (.5685 versus .6078).Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • CVIDF1 rises from .5608 to .6000 with cross-view assistance; the Full variant reaches .6129 and reduces IDs from 83 to 28, at lower IDF1 than Online Contrastive.Table 3 covers 19 non-dark M3Track test instances and includes incremental variants and full-model controls.Source: §4.3 · p. 8, Table 3 · p. 8
  • Disabling adapter optimization lowers IDF1 by 2.83 points and CVIDF1 by 4.19 points; removing verification-gated updates lowers CVIDF1 by 2.99 points and raises IDs from 28 to 57.The controls support learning from new observations while restricting which observations update persistent memory and supervise adaptation.Source: §4.3 · pp. 8–9
  • Using adapted rather than canonical verification queries lowers CVIDF1 by 1.69 points and raises IDs from 28 to 44.Because Gaussian statistics remain canonical in this control, it isolates query choice under canonical statistics rather than testing the entire two-space design.Source: §4.3 · p. 9

4.4 QUALITATIVE ANALYSIS

  • In the garden-scene example, cross-view assistance maintains the selected identity when the no-assistance configuration assigns it to a similar-shirt distractor.The example illustrates cross-view assistance, not isolated effects of long-term verification or online adaptation.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

The paper frames OmniGlimpse as an online approach to identity maintenance across views, using asymmetric memory and representation roles. Its evaluation shows stronger identity maintenance than evaluated online baselines, while the current scope is limited to designated targets with per-view initialization.

5 CONCLUSION

  • OmniGlimpse separates association proposals from verified updates and adapts appearance features while keeping verification in a stable shared space.The approach uses initialization supervision without scene-specific training or camera calibration.Source: §5 · p. 9
  • Experiments on M3Track and MvMHAT show stronger identity maintenance than evaluated online baselines and competitive performance against offline-trained references.Source: §5 · p. 9
  • The evaluation focuses on a small set of designated targets with per-view initialization; scaling to denser target sets and weaker initialization remains a direction for future work.Source: §5 · p. 9

AI USE STATEMENT

The authors report using generative AI for language polishing and literature retrieval, with retrieved materials checked and AI-assisted text reviewed and revised.

AI USE STATEMENT

  • Generative AI assisted with language polishing and literature retrieval; authors independently checked retrieved materials and reviewed and revised AI-assisted text.The authors state they take responsibility for the manuscript’s technical content, claims, citations, and conclusions.Source: AI USE STATEMENT · p. 10

A METHOD DETAILS

Candidate association combines local adaptive classifiers, canonical initialization priors, and short-term support; long-term verification then governs persistent updates and adaptation. The method uses eligibility checks to keep tentative evidence from becoming training data.

A.1 CANDIDATE ASSOCIATION DETAILS

  • Short-term support combines local, received cross-view, and initialization observations; adaptive orientation/cue classifiers and canonical priors score candidates, while ambiguous part evidence falls back to global support.Remote samples must be timely, and missing classifier classes disable the model rather than preserving stale weights.Source: §A.1 · pp. 12–13

A.2 LONG-TERM DISTRIBUTION DETAILS

  • Long-term distributions begin with initialization references; after bootstrap, confidence, competing margins, binding stability, and verification govern memory admission, including receiver-side checks for remote evidence.Verification uses pre-update statistics; remote evidence cannot update a receiver without qualifying local association evidence.Source: §A.2 · pp. 14–15

A.3 ONLINE ADAPTATION DETAILS

  • Adapter training uses admitted same-target positives and eligible negatives in same-view or cross-view triplets, begins after distribution maturity, and refreshes adaptive features and applicable classifiers before the next association step.Negative samples require an admitted different identity; unresolved observations are excluded, and empty triplet sets contribute zero.Source: §A.3 · pp. 15–16

B.1 IMPLEMENTATION SETTINGS

The implementation keeps the canonical encoder and other shared components fixed while optimizing local adapters. Configuration tables specify association, verification, admission, adapter-learning, and classifier settings.

B.1 IMPLEMENTATION SETTINGS

  • The canonical encoder, feature path, visibility estimator, and orientation predictor remain frozen; only local residual adapters are optimized.Evaluation uses one RTX A6000 GPU, while offline references use two.Source: §B.1 · p. 16
  • Remote observations are used only after receipt, with timestamps no later than the current processing time and ages within the synchronization window.Adapter-update intervals count newly admitted local observations with training permission, not elapsed frames or received messages.Source: §B.1 · p. 16
  • Table 4 lists association, verification, and admission settings, with capacities counted in observations or prototypes rather than feature coordinates.The passage identifies the configuration scope but does not provide individual setting values.Source: Table 4 · p. 17
  • Table 5 lists local adapter-learning and classifier settings, and defines Ridge regularization through the class-normalized objective in Eq. 12.The available passage gives the table’s scope, not its individual values.Source: Table 5 · p. 17

B.2 EVALUATION PROTOCOL AND BASELINE ADAPTATION

Evaluation selects two designated identities per scene and initializes online baselines at each target’s first annotated occurrence in each test view. Tracking and cross-view scores use specified matching and aggregation rules.

B.2 EVALUATION PROTOCOL AND BASELINE ADAPTATION

  • Each scene evaluates two designated identities, and online baselines receive a bounding box at each target’s first annotated occurrence in each view’s test segment.Visible target frames remain in evaluation even when initialization fails.Source: §B.2 · p. 18
  • Tracking metrics use global trajectory assignment for Identity matching, while cross-view correctness requires matched prediction labels to agree across synchronized views.Cross-view pair scores are averaged over scenes on M3Track and over scene–view pairs on MvMHAT; dataset tracking results combine sequence statistics.Source: §B.2 · p. 18
  • Pair CVIDF1 is the harmonic mean of pair CVIDP and CVIDR before averaging across pairs, while CVMA penalizes extra, missing, and inconsistent shared labels relative to GT boxes.Zero-denominator values contribute zero, and frames with a missing view’s ground truth remain in the averaging denominator.Source: §B.2 · p. 18
  • Online baselines run independently by target and view, then receive common labels across views so their outputs can be scored for cross-view correspondence.The aligned labels supply correspondence without requiring the baselines to infer it.Source: §B.2 · pp. 18–19

B.3 FULL-MODEL ABLATION CONTROLS

Full-model controls preserve the observations, initialization, and association procedure while changing specified components. The variants distinguish reliability screening, verification-gated writes, representation learning, and the verification feature space.

B.3 FULL-MODEL ABLATION CONTROLS

  • Full-model controls keep observations, target initialization, and association fixed, while Base disables reliability conditions, cross-view assistance, adapter optimization, and Gaussian verification.Base retains local association and classifier fitting; detection filtering, acceptance, allocation, and overlap handling remain common.Source: §B.3 · p. 19
  • Full w/o Rel. removes confidence, competing-margin, and binding-stability screening but retains Gaussian verification, verification-gated writes, cross-view assistance, and adapter learning.Initialization, restricted bootstrap, bank maturity, and valid-sample requirements remain unchanged.Source: §B.3 · p. 19
  • Without online adaptation, adapters stay at identity initialization and receive no optimization, while cross-view assistance, canonical verification, memory admission, and Ridge refitting remain active.This control removes representation learning, not all online state updates.Source: §B.3 · p. 19
  • Without verified updates, verification no longer gates persistent-memory writes or training eligibility, while confidence, margins, stability, and receiver-side local evidence requirements remain.Bootstrap and the mature-anchor requirement for adapter learning are unchanged.Source: §B.3 · p. 19
  • Adaptive verification substitutes the current local adapter’s normalized query for the canonical query while keeping Gaussian statistics, update rules, permissions, losses, and threshold fixed.Both variants use τℓ = 0.50; the comparison does not match acceptance rates.Source: §B.3 · p. 19

B.4 ADDITIONAL QUALITATIVE ANALYSIS

In the M3Track wood scene, the full model maintains the selected target identity despite a visually similar pedestrian. The ablations differ in whether an incorrect candidate is emitted, and the examples do not isolate the effects of individual design components.

B.4 ADDITIONAL QUALITATIVE ANALYSIS

  • The full model maintains the target identity throughout the displayed interval, while removing online contrastive adaptation assigns it to a nearby pedestrian in similar dark clothing.The configurations without cross-view assistance selects the distractor before and after a brief interval with no confirmed output.Source: §B.4 · p. 20
  • The interruption lasts seven dataset frames (1868–1874), while an incorrect tentative candidate remains available internally; it does not indicate identity correction.The absence of a cyan box in the third column of the last row corresponds to this interruption.Source: §B.4 · p. 20, Figure 4 · p. 20
  • These examples illustrate tracking behavior but do not isolate the effects of representation spaces or verification-dependent update permissions.The two ablations differ in whether an incorrect candidate is emitted, rather than showing two consistently distinct failure modes.Source: §B.4 · p. 20

Related papers

Loading 2610.36785v1…