PaperlayerView exact revision on arXiv

Source-linked AI summary

UGO: Unified Architecture for General Multi-Object Tracking by Segmentation

Jer Pelhan, Alan Lukezic, Matej Kristan

arXiv:2609.37339v1cs.CV

TL;DR

GMOT must track all instances of a category specified by one exemplar, yet existing methods rely on surrogate training and box-level reasoning that can struggle with interacting instances. UGO combines exemplar-conditioned detection and mask-based propagation, consolidates outputs into exclusive masks, and uses hierarchical memory. The paper reports state-of-the-art performance in GMOT and video counting while remaining competitive with specialist MOT methods.

  • Problem

    Existing GMOT methods rely on surrogate training and box-level localization, which can struggle to represent complex interactions and deformable targets in crowded scenes.

    Source: §1 · p. 1

  • Method

    UGO combines exemplar-conditioned detection and mask-based instance propagation, consolidating their outputs into exclusive masks and coordinating them with hierarchical adaptive memory.

    Source: §1 · p. 2

  • Results

    UGO achieves state-of-the-art performance in GMOT and video counting while remaining competitive with specialist MOT methods.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The paper presents a unified GMOT design paradigm that brings detection and instance propagation together with mask consolidation and memory feedback.

    Source: §1 · p. 2

    UGO’s higher false-positive rate leaves its MOTA 3% lower than QDTrack’s, with some errors reflecting ambiguity between similar bird species under exemplar-based category specification.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

General multi-object tracking (GMOT) tracks all instances of a user-specified category from a single first-frame exemplar. Prior work relies on bounding boxes and surrogate training, and struggles with non-rigid objects, crowded scenes, and distractors. We introduce UGO, a unified GMOT tracker that pairs a pretrained exemplar-conditioned detection head with an instance-propagation head in a common architecture. A novel training-free, energy-minimization consolidation method converts overlapping proposals into exclusive pixel-wise masks and detections, resolving over-segmentation, duplicates, and conflicts. A hierarchical memory spanning global and instance levels improves recall and per-instance segmentation accuracy using a new memory management protocol. UGO sets a new state-of-the-art on GMOT benchmarks and video object counting, and is competitive with specialist MOT methods, establishing a strong paradigm for unified, open-category multi-object tracking.

1 Introduction

GMOT tracks every instance of a category specified by one exemplar, combining SOT’s generalization with MOT’s instance coverage without category-specific training. UGO pairs exemplar-conditioned detection and instance propagation, then coordinates their outputs through mask consolidation and hierarchical memory.

1 Introduction

  • GMOT tracks all instances of a category specified by a single first-frame exemplar, combining SOT’s generalization with MOT’s coverage without category-specific training data.SOT tracks one selected instance; MOT detects and associates multiple instances of a predefined class.Source: §1 · p. 1
  • Existing GMOT methods rely on surrogate training and box-level localization, which poorly captures complex instance interactions and deformable objects in crowded scenes.These limitations can make association unreliable when instances overlap.Source: §1 · p. 1
  • Joint mask prediction does not scale to many objects and needs dedicated training sets, while video foundation models can falter under strong interactions and require new-instance initialization..Source: §1 · p. 1
  • Figure 1 contrasts oversegmentation and a redundant detection on an occluding fish with UGO’s consolidation, then illustrates segmentation tracking through clutter and occlusion.Source: Figure 1 · p. 2
  • UGO unifies exemplar-conditioned detection and instance propagation, using exclusive mask consolidation and hierarchical memory to coordinate tracking across instances.Its contributions also include explicit failure detection in memory management to support detection and identity propagation amid distractors.Source: §1 · p. 2
  • UGO’s stated contributions are a coordinated detection-and-tracking paradigm, training-free energy-based consolidation, and hierarchical adaptive memory; the paper reports state-of-the-art performance and a new category-agnostic tracking paradigm.Source: §1 · p. 2

2 Related Work

Related work spans classical MOT pipelines, query-based joint detection and propagation, general single-object trackers, and open-category tracking. Their reported constraints include category-specific training, limited scene-level coordination, and box-level conflict handling.

Multi-Object Tracking (MOT)

  • Classical MOT associates detector outputs using motion or appearance cues, while mask-level MOTS adds segmentation; these approaches excel with strong class-specific detectors but can fragment tracks when detections are missed.TrackFormer and TransTrack use DETR-like queries for joint detection and identity propagation, but query approaches have been noted to create a conflict between category semantics and instance specificity.Source: §2 · p. 2

General single object trackers (SOT)

  • General SOT trackers handle appearance change and occlusion, but independent trackers lack scene-level competition, cannot ensure mutual exclusivity, and cannot discover new instances.Running multiple SOT trackers on MOT data can lead to identity swaps.Source: §2 · p. 3

General & open-category tracking

  • GMOT methods use an exemplar to track unseen categories; PLGMOT and S-DETR add exemplar-conditioned detection, but remain limited by box-level reasoning and NMS-based conflict resolution.SAM3 extends SOT toward open-vocabulary tracking but lacks explicit pixel-level interaction reasoning among instances.Source: §2 · p. 3

3 A Unified General Multiple Object Tracker

UGO combines exemplar-conditioned detection and frame-to-frame instance propagation in one architecture. It consolidates their pixel-level outputs into exclusive masks and uses those decisions to update tracker memories.

3 A Unified General Multiple Object Tracker

  • UGO pairs SAM2-based instance propagation with GECO2 exemplar-conditioned detection, producing compatible per-pixel logits that feed consolidation and memory updates.Resolved conflicts also simplify trajectory management.Source: §3 · p. 3

3.1 Frame-to-frame instance propagation

  • For each tracked instance, the propagation head attends to its memory and produces a pixel-wise logit map for the current frame.Source: §3.1 · p. 3

3.2 General instance localization

  • Because only the exemplar is initially provided, GECO2 detects other category instances, and the segmentation head produces detection logits conditioned on category memory.Source: §3.2 · p. 4

3.3 Detector and tracker output consolidation

  • Consolidation assigns each pixel one label, preferring trackers to detections and favoring trackers supported by detections while penalizing masks that deviate from their initial masks.Source: §3.3 · p. 4
  • Starting from per-pixel logit argmax labels, optimization reassigns contested pixels and can turn off poorly supported instances, converging within a few iterations.Source: §3.3 · p. 5

3.4 Instance track lifecycle management

  • Consolidated masks initialize or update trackers; empty masks for ten consecutive frames trigger termination, and detector confirmation is required to validate new tracks.A track is valid after 30% of its masks are confirmed.Source: §3.4 · p. 5

3.5 Hierarchical Adaptive Memory

  • HAM separates category-level detector memory from per-instance tracker memory, with DRM for distractor resolution and RAM for frame-to-frame segmentation accuracy.DRM updates use disagreement between pre- and post-consolidation masks when the consolidated mask is detector-confirmed.Source: §3.5 · pp. 5–6

4 Experiments

UGO is evaluated against general and category-specialist trackers, video counters, and ablated variants. Results compare its tracking and counting performance and test the contributions of consolidation and memory design.

4.1 Comparison with general multi-object trackers

  • UGO beats S-DETR by 23% MOTA and 52% IDF1 on GMOT-40, and exceeds SAM3 by 6% HOTA and 22% MOTA.It also surpasses S-DETR by 25% in MT and has 39% lower ML; versus S-DETR, false negatives fall 42% at a cost of 31% higher FP, with 6% higher F1.Source: §4.1 · pp. 6–7, Table 1 · p. 7

4.2 Comparison with specialist trackers

  • On AnimalTrack, UGO surpasses QDTrack by 26% HOTA and 27% IDF1, tracks 2× more instances, and produces 4× fewer identity switches.Against SAM3, UGO is higher by 14% HOTA and 23% MOTA; its higher FP rate leaves MOTA 3% below QDTrack.Source: Table 2 · p. 8, §4.2 · p. 8

4.3 Comparison with video object counters

  • Against CountVid, UGO improves MAE/RMSE by 18%/25% on Penguins and 27%/27% on Crystals.UGO processes video in a single pass, while CountVid uses a multistage forward-backward tracking pipeline.Source: Table 4 · p. 8, §4.3 · p. 8

4.2 Comparison with specialist trackers

  • On MOT17, UGO exceeds standard MOT specialists in HOTA but trails the category-trained state-of-the-art method, particularly in detection accuracy.OC-SORT using a public human detector has 24% lower DetA; UGO’s association accuracy is comparable to OC-SORT*’s.Source: §4.2 · p. 8

4.4 Ablation study

  • Removing consolidation cuts HOTA and MOTA by 20% and 30%; replacing it with Hungarian matching cuts them by 18% and 90%.Without consolidation, trajectory fragmentations are nearly 2× as high.Source: §4.4 · p. 9
  • Logit rescaling leaves performance unchanged, while removing Θ1/Θ2 lowers HOTA by 18%/2% and MOTA by 7%/4%, respectively.Θ1 resolves competition and favors trackers; Θ2 suppresses uncertain or unstable masks.Source: §4.4 · p. 9
  • Replacing the proposed instance memory with SAM2 drops MOTA/HOTA by 2%/1% and raises identity switches 10%; replacing its management with [36] drops HOTA/MOTA by 4%/7%.Source: §4.4 · p. 9
  • Keeping only the initial exemplar drops MOTA/HOTA by 5%/2%, while fixed memory from four random first-frame boxes drops them by 4%/1%.The comparisons support adaptive exemplar updates and on-the-fly trajectory mining.Source: §4.4 · p. 9

5 Conclusion

UGO combines exemplar-conditioned detection and mask-based instance propagation, with consolidation producing mutually exclusive masks and hierarchical memory supporting global and per-instance modeling. The appendix frames consolidation as assigning each pixel among competing detection, tracker, and background labels.

5 Conclusion

  • UGO integrates exemplar-conditioned detection and mask-based instance propagation, while consolidation resolves conflicting outputs into mutually exclusive masks and hierarchical memory supports global and per-instance modeling.The conclusion reports state-of-the-art performance in GMOT and video counting and competitiveness with specialist MOT methods.Source: §5 · p. 9

A The mask consolidation algorithm

A.1 The labeling cost function

The labeling objective combines pixel logits with label-wise potentials that encode preferences among trackers and detections. A surrogate optimization separates auxiliary-potential updates from per-pixel label selection.

A.1 The labeling cost function

  • The cost uses label-wise potentials to prefer trackers over detections, favor trackers with strong detector support, and suppress labels whose final masks depart from their initial masks.The potentials depend on IoU between masks induced by the labeling and binarized initial masks.Source: §A.1 · p. 12

A.2 Reweighted surrogate objective

  • The surrogate decouples pixel logits from nonlinear label potentials, and its size term prevents large masks from being preferred simply because they cover more pixels.Source: §A.2 · p. 12

A.3 Step 1 minimization problem

  • The iterative scheme alternates between estimating auxiliary variables for the potentials and updating the pixel labeling, following Majorize-Minorize.The auxiliary-variable step uses the previous labeling estimate.Source: §A.2 · p. 12, §A.3 · p. 13

A.4 Step 2 minimization problem

  • With auxiliary variables fixed, the labeling update decomposes over pixels, so each pixel's label can be selected independently.Source: §A.4 · p. 13

B HAM Architecture Details

HAM couples a category-level memory for exemplar-conditioned detection with per-instance memories for propagation. Reliable tracks update the global memory, while consolidation corrections update instance memories.

B HAM Architecture Details

  • HAM maintains category-level memory for detection and per-instance memory for propagation, with consolidation coupling their updates.Reliable tracks update category memory; consolidation-driven corrections update instance memories.Source: B HAM Architecture Details · p. 13
  • Each instance memory contains distractor-resolving and recent-appearance buffers, supporting discrimination and temporal adaptation.Source: B HAM Architecture Details · p. 13
  • The overview assigns robust category-memory updates to stable tracks and robust instance-memory updates to consolidation results.Source: Figure 5 · p. 14

C Experimental Evaluation on Animal Track

On AnimalTrack, UGO combines strong target coverage and identity performance with more false positives, including ambiguity from visually similar species. Against SAM3, the evaluation uses text prompts because SAM3 does not support persistent exemplar-based category specification across frames.

C Experimental Evaluation on Animal Track

  • UGO achieves the highest HOTA, recall, and F1 among compared AnimalTrack trackers, with 40,157 false negatives versus at least twice as many for each specialist method, including QDTrack's 83,057.It produces more false positives than some specialist detectors, but has the strongest overall F1.Source: C Experimental Evaluation on Animal… · p. 14
  • SAM3 is evaluated with a ground-truth-category text prompt because its image exemplars do not persist as cross-frame category specifications in GMOT.Text prompts offer a semantic advantage for fine-grained categories; UGO cannot explicitly control category granularity from visual exemplars.Source: C Experimental Evaluation on Animal… · pp. 14–15
  • Because UGO specifies target categories only through visual exemplars, it cannot explicitly control tracked-category semantic granularity, inherited from the GECO2 detector [30]; Figure 6 shows two AnimalTrack sequences with its most false detections.The caption notes that ground-truth boxes are green and some instances are missing from ground-truth annotations.Source: C Experimental Evaluation on Animal… · p. 15, Figure 6 · p. 15
  • Against SAM3, UGO leads by 14% in HOTA and 23% in MOTA, while SAM3's fewer identity switches and fragmentations coincide with 20% lower relative detection recall.UGO outperforms SAM3 on the primary tracking metrics despite using visual exemplars rather than explicit semantic supervision.Source: C Experimental Evaluation on Animal… · p. 15
  • SAM3 uses association, detector-support, and hotstart thresholds, whereas UGO handles duplicate management, instance discovery, mask correction, and memory updates through consolidation.The threshold summary includes a 15-frame hotstart delay, an 8-frame unmatched threshold, and a 16-frame detector refresh interval.Source: C Experimental Evaluation on Animal… · pp. 15–16, Table 7 · p. 15

D Ablation Study

On GMOT-40, UGO’s conservative consolidation and memory gates remain robust across single-threshold sweeps. The study varies one gate at a time against the default configuration.

Overall robustness to thresholds

  • MOTA falls by only 2% at τmd = 0.45, the largest deviation reported; across the sweeps, HOTA varies by less than 1%.The authors conclude these gates are robust rather than delicately tuned hyperparameters.Source: Table 8 · p. 16, D Ablation Study · p. 17

Tracker preference strength θ0

  • HOTA varies by less than 0.21 points and MOTA stays within ≈0.41 points of default as θ0 varies; moderate tracker preference is sufficient.θ0 controls tracker preference over detector logits and scales detector support for tracker masks.Source: Table 8 · p. 16, D Ablation Study · p. 16

Overlap threshold τmd

  • At τmd = 0.45, relaxed tracker confirmation increases false positives and drops MOTA by 2%; at 0.7, stricter confirmation reduces MT and increases ML.The authors identify τmd = 0.6 as the stable balance, yielding the least ML.Source: Table 8 · p. 16, D Ablation Study · p. 17

Tracker-collapse threshold τlo

  • Changing τlo from its 0.2 default has minor effects: 0.1 retains more corrected tracker masks but slightly raises FP, while 0.3 may prune valid corrected tracks.The threshold controls removal of tracker labels whose final masks are inconsistent with initial masks.Source: Table 8 · p. 16, D Ablation Study · p. 17

High overlap threshold τhi

  • Lowering τhi increases candidate memory updates with a modest FN cost, while raising it to 0.95 makes updates rare and slightly lowers HOTA, MOTA, and IDF1.The default τhi = 0.8 provides a conservative update regime.Source: Table 8 · p. 16, D Ablation Study · p. 17

E Application to Single-target Tracking

UGO is adapted to single-target tracking by keeping the initialized target active through occlusion or temporary disappearance. On DiDi, it surpasses SAM2.1 and performs close to DAM4SAM.

Adaptation to single-target tracking

  • For the single-target setting, UGO disables instance termination for the initialized target so tracking continues through extended occlusion or temporary disappearance.This supports target re-detection in scenarios described as more frequent in SOT than in GMOT.Source: E Application to Single-target Tracking · p. 17

DiDi results

  • UGO surpasses SAM2.1 by 5.5% and has tracking quality only 1.3% lower than DAM4SAM on DiDi.The paper frames this as cross-paradigm generalization by a fundamentally multi-target tracker.Source: E Application to Single-target Tracking · p. 17

Qualitative examples

  • Figure 7 represents each tracked instance with a colored mask and a trajectory line.The figure provides qualitative tracking examples.Source: Figure 7 · p. 18

F Qualitative Results

UGO’s qualitative examples show tracked instances across varied categories and datasets, using colored masks and trajectory lines to depict their identities and motion.

Qualitative results

  • Sixteen sequences from different datasets show two representative frames each, with colored masks and trajectory lines visualizing tracked instances and their motion.The examples include airplanes, balls, balloons, animals, and people.Source: F Qualitative Results · p. 18
  • The examples span categories including airplanes, balls, balloons, birds, penguins, ducks, deer, bees, fish, and people.The paper presents this breadth as highlighting UGO’s cross-category generalization capability.Source: F Qualitative Results · p. 18
  • Each sequence is illustrated with two representative frames, with semi-transparent masks and trajectory lines indicating instance identity and motion.Source: F Qualitative Results · p. 18

Related papers

Loading 2609.37339v1…