PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking

Mingzhan Yang, Guangxin Han, Bin Yan, Wenhua Zhang, Jinqing Qi, Huchuan Lu, Dong Wang

arXiv:2308.00783v2cs.CV

TL;DR

MOT association becomes ambiguous under occlusion and clustering because spatial and appearance cues degrade together. Hybrid-SORT adds confidence and height states to velocity direction, using simple, training-free modeling that preserves SORT properties. It consistently improves five representative trackers and performs strongly across DanceTrack, MOT17, and MOT20.

  • Problem

    Occlusion and clustering make the commonly used spatial and appearance cues ambiguous simultaneously, creating a persistent MOT association challenge.

  • Method

    Hybrid-SORT combines strong cues with confidence, height, and velocity-direction weak cues through Tracklet Confidence Modeling and Height Modulated IoU.

  • Results

    Hybrid-SORT consistently improves five representative trackers and achieves superior performance on DanceTrack, MOT17, and MOT20 while retaining SORT characteristics.

  • Takeaways & Limitations

    Weak cues can compensate for strong-cue limitations, and Hybrid-SORT can be applied plug-and-play and training-free across diverse trackers and scenarios.

  • Takeaways & Limitations

    MOT17 and MOT20 offer limited evaluation sensitivity because they are much smaller than DanceTrack and contain mostly simple linear motions with relatively saturated performance.

Abstract

from arXiv · show

Multi-Object Tracking (MOT) aims to detect and associate all desired objects across frames. Most methods accomplish the task by explicitly or implicitly leveraging strong cues (i.e., spatial and appearance information), which exhibit powerful instance-level discrimination. However, when object occlusion and clustering occur, spatial and appearance information will become ambiguous simultaneously due to the high overlap among objects. In this paper, we demonstrate this long-standing challenge in MOT can be efficiently and effectively resolved by incorporating weak cues to compensate for strong cues. Along with velocity direction, we introduce the confidence and height state as potential weak cues. With superior performance, our method still maintains Simple, Online and Real-Time (SORT) characteristics. Also, our method shows strong generalization for diverse trackers and scenarios in a plug-and-play and training-free manner. Significant and consistent improvements are observed when applying our method to 5 different representative trackers. Further, with both strong and weak cues, our method Hybrid-SORT achieves superior performance on diverse benchmarks, including MOT17, MOT20, and especially DanceTrack where interaction and severe occlusion frequently happen with complex motions. The code and models are available at https://github.com/ymzis69/HybridSORT.

Introduction

MOT association typically relies on spatial and appearance cues, but occlusion and clustering make both ambiguous. Hybrid-SORT compensates with weak cues and simple modeling that generalizes across trackers while preserving SORT characteristics.

  • Motivation: Occlusion and clustering can make spatial overlap and appearance features unreliable for distinguishing objects.High overlap makes IoU ambiguous, while appearance features can be dominated by foreground objects.
  • Motivation: Confidence, height, and velocity direction provide complementary weak cues when strong cues become unreliable.Confidence can indicate occluding/occluded relations, while height is robust to poses and contains depth information.
  • Method: Tracklet Confidence Modeling and Height Modulated IoU exploit confidence and height states while maintaining Simple, Online and Real-Time characteristics.The methods model tracklet states with Kalman filtering and use them for association.
  • Generalization: Hybrid-SORT’s confidence and height designs produce significant, consistent improvements when applied to five representative trackers.The evaluated trackers are SORT, DeepSORT, MOTDT, ByteTrack, and OC-SORT.
  • Results: Hybrid-SORT achieves superior performance on DanceTrack, MOT17, and MOT20 while leveraging both strong and weak cues.The method is presented as plug-and-play and training-free for diverse trackers and scenarios.

Related Work

Related MOT work uses spatial and appearance cues, graph-based matchers, or Transformer architectures for association. Spatial cues suit short-term high-FPS matching, while appearance cues support longer-term association.

  • Spatial-based Heuristic Matcher: Spatial information is widely used in high-FPS benchmarks because short frame intervals make object motion approximately linear.SORT predicts tracklet locations with a Kalman Filter and associates detections using IoU.
  • Appearance-based Heuristic Matcher: Appearance information remains relatively consistent throughout a video and therefore benefits long-term association.DeepSORT and GHOST use independent ReID models, while later methods integrate detection and ReID through joint training.
  • Cue Discrimination: Figure 1 uses green solid arrows for reliable discrimination and red dashed arrows for unreliable discrimination, with larger values indicating greater reliability.The figure compares discrimination capacity across strong and weak cues.
  • Graph-based Matchers: Graph-based learnable matchers formulate association as edge classification between tracklet and detection nodes.MOTSolv and GMTracker use GNNs, while SUSHI hierarchically connects short tracklets.
  • Transformer-based Matchers: Transformer-based methods use attention with track and detection queries to jointly propagate trajectories and initialize detections.MOTRv2 adds a separate detector to address the conflict between detection and association.

Method

Hybrid-SORT retains the SORT association pipeline while augmenting spatial and appearance cues with confidence, height, and velocity-direction cues. Its components model these weak cues for more discriminative matching under occlusion and clustering.

  • SORT Pipeline: Hybrid-SORT uses Kalman Filter motion estimation and Hungarian bipartite matching, with optional ReID appearance modeling.Pairwise tracklet–detection costs form the association matrices.
  • Tracklet Confidence Modeling: Tracklet Confidence Modeling estimates confidence with Kalman Filter states for ordinary detections and Linear Prediction for abrupt occlusion transitions.The confidence cost is the absolute difference between estimated tracklet confidence and detection confidence.
  • Height Modulated IoU: Height Modulated IoU multiplies height-based IoU with conventional IoU to combine a weak height cue with strong spatial information.Height is modeled as a temporally stable, pose-robust state that provides some depth information.
  • Cue Integration: The pipeline combines IoU and cosine distance as strong-cue metrics with confidence, height, and velocity direction as weak cues.It also includes high-confidence, low-confidence, and lost-tracklet recovery association stages.
  • Robust Observation-Centric Momentum: Robust Observation-Centric Momentum reduces velocity-direction noise by using multiple temporal intervals and four box corners instead of one center.The original center-based formulation is vulnerable to fixed intervals and sparse points.
  • Appearance Modeling: The ReID module uses Exponential Moving Average appearance modeling and cosine distance, but the paper does not focus on ReID components.Hybrid-SORT and Hybrid-SORT-ReID follow the same SORT paradigm with or without ReID.

Experimental Setting

The evaluation covers DanceTrack, MOT20, and MOT17, using HOTA as the primary metric alongside MOTA and IDF1. Experiments reuse baseline detection and ReID models, with FPS measured on a single V100 GPU using FP16 and batch size 1.

  • Datasets: The method is evaluated on DanceTrack, MOT20, and MOT17 to test association performance across challenging and diverse tracking scenarios.DanceTrack emphasizes nonlinear motion, interactions, and occlusions; MOT20 emphasizes dense objects and severe occlusion; MOT17 mainly contains linear motion.
  • Metrics: HOTA is the primary metric, complemented by MOTA and IDF1 for broader performance assessment.HOTA provides a higher-order assessment, while IDF1 reflects association and MOTA is primarily influenced by detection performance.
  • Implementation: The experiments use YOLOX detections matching OC-SORT and BoT ReID features matching BoT-SORT for fair comparison.The appearance feature dimension is 2048.
  • Implementation: FPS is measured with FP16 precision and batch size 1 on a single V100 GPU with an Intel Xeon Silver 4214R CPU.The IoU rejection threshold is 0.15 on DanceTrack and 0.25 on other benchmarks.

Benchmark Results

Hybrid-SORT consistently improves over OC-SORT across DanceTrack, MOT20, and MOT17 with negligible additional computation. Its strongest reported gains occur on DanceTrack, while improvements on MOT17 and MOT20 are limited by dataset size, simpler motion, and performance saturation.

  • Overall comparison: Hybrid-SORT consistently outperforms OC-SORT across all three datasets with negligible additional computation while retaining SORT characteristics.The comparison is against a strong heuristic baseline, whereas some heavier, offline, or more complex methods perform better on certain datasets.
  • Limitations of benchmark comparisons: MOT17 and MOT20 show limited improvement because they are nearly 10× smaller than DanceTrack, mostly contain simple linear motion, and are relatively saturated.These dataset characteristics may limit how accurately performance differences are assessed.
  • DanceTrack: 7.6 HOTA: Hybrid-SORT significantly surpasses OC-SORT on DanceTrack with identical association inputs and nearly identical computational complexity.Hybrid-SORT-ReID reaches HOTA 65.7 on DanceTrack among heuristic trackers.
  • MOT20: 0.4 HOTA, 0.3 IDF1, and 0.9 MOTA: Hybrid-SORT surpasses OC-SORT on MOT20 with practically indistinguishable additional computation.Hybrid-SORT-ReID reaches HOTA 63.9 on MOT20 for the heuristic tracker.
  • Table 1: Table 1 reports DanceTrack test-set results, groups methods sharing detections in a gray block, and emphasizes the highest-ranking heuristic matcher.The caption specifies the table organization and highlighting convention.
  • MOT17: 0.4 HOTA, 0.9 IDF1, and 1.3 MOTA: Hybrid-SORT surpasses OC-SORT on MOT17 with negligible additional computation.With an independent ReID model, Hybrid-SORT reaches HOTA 64.0 on MOT17.

Ablation Study

The ablations show that confidence and height modeling improve association while preserving near-real-time operation, and that these weak-cue designs generalize across trackers and scenarios.

  • Component Ablation: 4.0 HOTA improvement from TCM comes with only a -0.7 FPS inference-speed change, while HMIoU adds 1.6 HOTA with a -0.1 FPS change.ROCM adds 0.6 HOTA but reduces speed by 1.5 FPS; ReID adds 3.7 HOTA while making inference near real-time.
  • Confidence Modeling: Kalman Filter boosts high-confidence association by 2.9 HOTA, whereas Linear Prediction decreases HOTA by 1.1 in that stage.For low-confidence detections, Kalman Filter and Linear Prediction improve HOTA by 0.7 and 1.1, respectively.
  • Confidence Modeling: Kalman Filter suits stable confidence in unobstructed objects, while Linear Prediction better models rapid directional confidence changes during occlusion.The paper attributes these differences to stable confidence for high-confidence detections and rapid increases or decreases when clustering starts or ends.
  • Height State or Width State: Height state benefits association, whereas width state harms it because pose and limb movements make width irregular and difficult to estimate precisely.Height changes are relatively short and continuous during squatting or standing, making them more suitable for Kalman Filter modeling.
  • Generality on Other Trackers: Across DanceTrack and MOT17, weak-cue designs significantly improve all four evaluated trackers, including 4.9 HOTA for DeepSORT with TCM on DanceTrack.HMIoU improves SORT by 1.6 HOTA on DanceTrack and 1.0 HOTA on MOT17; the method remains plug-and-play and training-free.

Conclusion

Hybrid-SORT addresses heavy occlusion and clustering by combining overlooked weak cues with strong cues through simple modeling. It improves association while retaining SORT characteristics and generalizes across trackers and scenarios.

  • Conclusion: Hybrid-SORT uses confidence state, height state, and velocity direction to compensate for limitations of spatial and appearance cues in heavy occlusion and clustering.The proposed weak cues are modeled simply and used alongside strong cues to improve association.
  • Conclusion: Hybrid-SORT retains Simple, Online and Real-Time characteristics and can be applied to existing trackers in a plug-and-play, training-free manner.The paper reports strong generalization across diverse trackers and scenarios.
  • Conclusion: With appearance information, Hybrid-SORT achieves superior performance over state-of-the-art methods while using a simpler pipeline and faster association.The paper positions these characteristics as useful for diverse scenarios and devices with limited computational resources.
Loading 2308.00783v2…