Source-linked AI summary
Spatial-Temporal Relation Networks for Multi-Object Tracking
Jiarui Xu, Yue Cao, Zheng Zhang, Han Hu
TL;DR
MOT association requires robust similarity scores that integrate heterogeneous appearance, location, and topology cues across time. STRN uses spatial-temporal relation networks and tracklet-object pair representations to unify this reasoning, achieving state-of-the-art accuracy across MOT15–17 online benchmarks with public detections.
Problem
Similarity scoring is difficult because appearance-only cues are unreliable under shared categories, occlusion, and pose or quality variation, while existing methods rarely jointly learn heterogeneous cues across time.
Method
STRN extends relation networks across spatial and temporal domains to combine appearance, location, and topology cues and study tracklet-object pair representations.
Results
STRN achieves state-of-the-art tracking accuracy over online methods on all MOT15–17 benchmarks using public detections.
Takeaways & Limitations
A unified feed-forward, end-to-end relation-network framework can combine multiple cues across time for MOT similarity measurement.
Takeaways & Limitations
The paper instantiates only online settings, although it states that the method is also applicable to offline and online association.
Abstract
from arXiv · showhide
Recent progress in multiple object tracking (MOT) has shown that a robust similarity score is key to the success of trackers. A good similarity score is expected to reflect multiple cues, e.g. appearance, location, and topology, over a long period of time. However, these cues are heterogeneous, making them hard to be combined in a unified network. As a result, existing methods usually encode them in separate networks or require a complex training approach. In this paper, we present a unified framework for similarity measurement which could simultaneously encode various cues and perform reasoning across both spatial and temporal domains. We also study the feature representation of a tracklet-object pair in depth, showing a proper design of the pair features can well empower the trackers. The resulting approach is named spatial-temporal relation networks (STRN). It runs in a feed-forward way and can be trained in an end-to-end manner. The state-of-the-art accuracy was achieved on all of the MOT15-17 benchmarks using public detection and online settings.
1. Introduction
MOT association needs similarity scores that combine appearance with location and topology across time, but heterogeneous cues make unified modeling difficult. STRN addresses this with spatial-temporal relation networks that reason over objects and tracklets in an end-to-end framework.
- Appearance-only similarity is limited because tracked objects often share a category and undergo occlusions, pose changes, and quality variations.
- Topology can preserve identity when appearance changes or becomes occluded, because spatial relationships among objects remain consistent.
- Existing approaches commonly use cue-specific mechanisms or sophisticated learning procedures because appearance, location, and topology are heterogeneous.
- STRN extends object-object relation networks from spatial to spatial-temporal reasoning to integrate appearance, topology, and location cues.
- The module strengthens object features spatially, aggregates referenced tracklet features temporally, and combines them with current-object features for similarity measurement.
- STRN is feed-forward, end-to-end trainable, and achieves state-of-the-art performance over online methods on MOT15–17 benchmarks.
2. Related Work
Prior MOT association methods use appearance, location, or topology cues with limited joint learning and rare temporal aggregation. STRN instead combines these cues across time in a unified, feed-forward, end-to-end framework while designing tracklet-object pair representations.
- Tracking-by-Detection Paradigm: Tracking-by-detection methods decouple framewise localization from cross-frame object association, with online and offline variants distinguished by their association information.
- Similarity Computation: Appearance similarity methods evolved from handcrafted descriptors to deep networks, with some approaches combining feature, cosine-distance, and location or motion information.
- Similarity Computation: Hard motion assumptions such as high consecutive-frame overlap can fail for fast-moving objects, limiting their application.
- Similarity Computation: Relation networks encode individual-object topology, addressing occupancy grids that count object distributions without differentiating individual objects.
- Similarity Computation: Most methods use only one or two cues, rarely aggregate information across time, and often require separate networks or multi-stage training.
- Similarity Computation: STRN jointly learns all cues across time in a unified feed-forward framework and studies tracklet-object pair representations as a factor in similarity quality.
- Relation Networks: Unlike attention methods focused on salient foreground regions within bounding boxes, STRN models topology between objects for multi-object tracking.
3. Method
STRN computes tracklet–object similarity by separating spatial topology reasoning from temporal feature aggregation, then combining relation, cosine, location, and motion cues. Its pair representation feeds a final similarity network, while temporal attention downweights low-quality detections.
- Tracking-by-Detection: The online tracker detects objects frame by frame, computes tracklet–object similarities, and solves a bipartite assignment with the Hungarian algorithm.This process yields object trajectories through successive frame-level assignments.
- Relation Features: The tracklet–object pair representation combines relation, cosine, location, and motion features before a two-layer sigmoid network computes similarity.The hybrid relation representation outperforms formulations using either relation-feature mode alone.
- Spatial-Temporal Relation Module: STRN extends object relation networks into the temporal domain to jointly encode appearance, topology, and location cues.The extension distinguishes spatial topology modeling from temporal information aggregation.
- Spatial-Temporal Relation Module: Spatial reasoning strengthens appearance features using learned topology information, while temporal reasoning aggregates strengthened features across multiple frames.Temporal aggregation is applied to tracklets and uses only the latest τ1 frames, with τ1 = 10 by default.
- Spatial-Temporal Relation Module: Temporal attention assigns low weights to blurred, wrongly cropped, or partly occluded detections, reducing their effect on tracklet representations.The learned weights therefore support quality-aware aggregation of recent detections.
- Relation Features: Direct concatenation supports multiple relation modes but makes compact individual feature learning more difficult, motivating explicit cosine similarity features.The cosine feature is computed between projected relation features and passed to the final similarity network.
4. Experiments
Experiments evaluate STRN on three MOT benchmarks using public detections, ablations of feature representations and relation reasoning, and online tracking comparisons. Results show that hybrid appearance and location features, spatial-temporal reasoning, and the complete approach improve association and tracking performance.
- Implementation Details: The implementation uses a ResNet-50 backbone, 128 × 64 cropped detections, and temporal aggregation over the latest 9 frames.Appearance features are produced after reducing the backbone feature-map channels and applying a fully connected layer.
- Feature Representation: Hybrid unary appearance and cosine features reach 29.8 MOTA, compared with 19.8 for unary appearance and 25.2 for cosine similarity alone.These comparisons are conducted without spatial-temporal relation reasoning.
- Feature Representation: Combining unary location and motion features produces a 2.5 MOTA boost and reduces ID switches from 515 to 129.Unary location alone yields a 1.9 MOTA improvement, while motion features alone yield a 1.2 improvement.
- Relation Module Ablation: Spatial relation reasoning improves MOTA by 2.5, while temporal reasoning adds 1.4 MOTA and outperforms average summation and maxpooling.The spatial module also substantially reduces false positives, and the temporal module uses a weighted average over frame features.
- Benchmark Results: With public detections, STRN achieves state-of-the-art online tracking accuracy across MOT15–17 according to MOTA and average rank.The association step computes tracklet-object similarities and solves a bipartite graph.
5. Conclusion
The paper addresses MOT object association by using spatial-temporal relation networks to combine appearance, location, and topology cues across time. STRN is feed-forward, end-to-end trainable, and achieves state-of-the-art accuracy across online MOT15–17 benchmarks with public detections.
- Conclusion: STRN combines appearance, location, and topology cues through spatial relation networks extended to temporal information aggregation.The framework is designed for robust similarity measurement in object association.
- Conclusion: STRN runs feed-forward and end-to-end, achieving state-of-the-art accuracy over online methods on all MOT15–17 benchmarks using public detection.