PaperlayerView exact revision on arXiv

Source-linked AI summary

TransCenter: Transformers with Dense Representations for Multiple-Object Tracking

Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, Xavier Alameda-Pineda

arXiv:2103.15145v4cs.CV

TL;DR

Transformer-based MOT lacks an approach that is both accurate and efficient because quadratic attention and noise-initialized sparse queries are poorly suited to crowded scenes. TransCenter uses dense image-related detection queries and sparse tracking queries generated by query learning networks, and it reports large gains on MOT17 and MOT20 while maintaining reasonable efficiency.

  • Problem

    Transformer-based MOT needs an accurate and efficient alternative to quadratic attention with insufficient noise-initialized sparse queries, particularly for crowded scenes.

  • Method

    TransCenter combines dense pixel-level multi-scale detection queries with sparse tracking queries produced by query learning networks and processed in its decoder.

  • Results

    TransCenter sets new state-of-the-art performance, improving MOTA by +4.0% on MOT17 and +18.8% on MOT20 in the reported comparisons.

  • Takeaways & Limitations

    The results support dense detection and sparse tracking representations as an accurate and reasonably efficient design for MOT, including very crowded scenes.

  • Takeaways & Limitations

    Some comparison methods lack source code or sufficient information for evaluating image memory, parameter count, and runtime metrics.

Abstract

from arXiv · show

Transformers have proven superior performance for a wide variety of tasks since they were introduced. In recent years, they have drawn attention from the vision community in tasks such as image classification and object detection. Despite this wave, an accurate and efficient multiple-object tracking (MOT) method based on transformers is yet to be designed. We argue that the direct application of a transformer architecture with quadratic complexity and insufficient noise-initialized sparse queries - is not optimal for MOT. We propose TransCenter, a transformer-based MOT architecture with dense representations for accurately tracking all the objects while keeping a reasonable runtime. Methodologically, we propose the use of image-related dense detection queries and efficient sparse tracking queries produced by our carefully designed query learning networks (QLN). On one hand, the dense image-related detection queries allow us to infer targets' locations globally and robustly through dense heatmap outputs. On the other hand, the set of sparse tracking queries efficiently interacts with image features in our TransCenter Decoder to associate object positions through time. As a result, TransCenter exhibits remarkable performance improvements and outperforms by a large margin the current state-of-the-art methods in two standard MOT benchmarks with two tracking settings (public/private). TransCenter is also proven efficient and accurate by an extensive ablation study and comparisons to more naive alternatives and concurrent works. For scientific interest, the code is made publicly available at https://github.com/yihongxu/transcenter.

1 INTRODUCTION

TransCenter addresses MOT with transformer-based dense detection and sparse tracking representations to improve accuracy in crowded scenes while preserving computational efficiency. It combines image-related queries, tracking queries, and tailored network designs, achieving strong benchmark performance and offering accuracy–efficiency variants.

  • Motivation: Transformer-based global modeling is motivated by MOT’s need to capture interactions among objects, especially in crowded scenes.The approach aims to reduce missed or noisy trajectory predictions through global estimations.
  • Motivation: Dense, non-overlapping representations address missed detections from insufficient sparse queries and redundant detections from overlapping anchors.Dense image-size queries are compatible with center heatmap predictions and help cover crowded scenes.
  • Efficiency: Efficient attention, sparse tracking queries, and query learning networks mitigate the computational cost of dense transformer inputs.The proposed design uses efficient encoder and decoder structures and avoids heavy detection cross-attention where possible.
  • Core approach: TransCenter introduces dense image-related queries and sparse tracking queries for transformer-based multiple-object tracking.Dense queries support detection, while sparse queries support temporal tracking.
  • Results: +4.0% MOTA on MOT17 and +18.8% MOTA on MOT20 establish new state-of-the-art baselines in the reported settings.The paper also provides TransCenter-Dual for crowded-scene performance and TransCenter-Lite for computational efficiency.

2 RELATED WORKS

Related work spans probabilistic, recurrent, graph-based, detector-based, and transformer-based approaches to modeling object detections, associations, and interactions. Transformer methods offer global dependency modeling but face attention-complexity and backbone-cost challenges when applied to vision and MOT.

  • Classical and recurrent MOT: Early MOT methods optimize associations probabilistically, while later approaches use recurrent networks to model trajectories, dynamics, interactions, and appearance.These approaches include end-to-end recurrent formulations and combinations of dynamics, interaction, and appearance branches.
  • Interaction modeling: Graph neural networks replace hand-designed graphs with learnable structures to model complex interactions among objects.This line of work extends interaction modeling beyond recurrent and probabilistic formulations.
  • Detection-based MOT: Tracking-by-detection methods rely on detectors, while keypoint-based detectors avoid overlapping manually designed anchors and predict object centers more directly.The shift toward keypoint representations is presented as an alternative to classic anchor-based detection.
  • Transformers in vision: Transformers model long-term dependencies through multi-head attention and have been applied to vision tasks including recognition, person re-identification, generation, and super-resolution.Their attention mechanism supports dependencies between different entries or image regions.
  • Transformer detection: DETR uses an encoder-decoder architecture in which object queries attend to encoded image features to produce one-to-one object-position predictions.The attention module transforms inputs into queries, keys, and values.
  • Efficiency: Transformer attention has heavy input-size-dependent memory and computational costs, motivating feature extraction, deformable attention, and reduced-complexity variants.Deformable DETR reduces complexity and speeds convergence, but CNN backbone costs remain relevant.
  • Transformer MOT: Concurrent transformer-based MOT methods apply DETR or deformable DETR architectures, establishing the context for TransCenter’s alternative query design.TransCenter is positioned within the recent adoption of transformers for MOT.

3 TRANSCENTER

TransCenter combines image-related dense detection queries with sparse tracking queries to address MOT detection and temporal association while balancing accuracy and efficiency. Its QLN and decoder designs produce dense object localization and efficient cross-frame tracking.

  • 3 TRANSCENTER: TransCenter uses image-related dense queries for detection and sparse queries for tracking, replacing noise-initialized sparse detection queries with representations tied to image features.Dense detection queries identify object positions, while sparse tracking queries exploit prior object positions for temporal association.
  • 3 TRANSCENTER: Image-related dense queries are multi-scale, adapt their number to input resolution, and provide query-pixel correspondence that avoids Hungarian matching.These properties address quadratic computation, scene-density variation, and query-ground-truth assignment.
  • 3 TRANSCENTER: The pipeline encodes images at t and t −1 into multi-scale memories, then QLN produces detection queries and memory plus tracking queries and memory for the decoder.The decoder correlates detection queries with detection memory and tracking queries with tracking memory to produce detection and tracking features.
  • 3 TRANSCENTER: PVTv2 serves as the encoder because its efficient attention heads reduce complexity, its scaling is flexible, and its feature pyramid supports dense multi-scale queries.TransCenter uses PVT-Lite for TransCenter-Lite and PVT-B2 for TransCenter and TransCenter-Dual.
  • 3 TRANSCENTER: Sparse tracking queries exploit previous object positions, reducing tracking-query processing from roughly 14k locations to fewer than 500 depending on the number of objects.This discretization is used to speed tracking attention in the decoder.
  • 3 TRANSCENTER: The selected TQSA-Single decoder cross-attends tracking queries and memory while directly using dense detection queries for output branches, balancing efficiency and accuracy.TransCenter-Dual retains detection cross-attention, whereas TransCenter-Lite additionally uses a lighter encoder.

4 EXPERIMENTAL EVALUATION

TransCenter achieves strong MOT performance across MOT17, MOT20, KITTI, and comparisons with transformer, center-based, and concurrent methods, while its dense detection and sparse tracking design supports efficiency. Ablations further show that dense image-related queries, encoder choices, and decoder design shape the accuracy–speed tradeoff.

  • MOT17 and MOT20: +19.2%/+8.4% MOTA and +11.1%/18.8% MOTA are reported on MOT20 for public/private detections with and without extra training, respectively.TransCenter also reduces false negatives while maintaining relatively low false positives and runs at around 8 fps in crowded scenes.
  • KITTI and method comparisons: +5.3% MOTA is achieved over CenterTrack in pedestrian tracking, while car tracking remains close in KITTI.The authors caution that KITTI’s sparse object locations do not fully expose dense-scene tracking capacity.
  • Efficiency-accuracy comparisons: TransCenter outperforms TransTrack by +1.7% MOTA in MOT17 and +8.4% in MOT20, while also using a smaller model and less inference memory.Using the same encoder, TransCenter-DETR still improves over TransTrack by +0.3% in MOT17 and +4.3% in MOT20.
  • Efficiency-accuracy comparisons: TransCenter-YOLOX improves MOTA over MO3TR-PIQ by +2.2% in MOT17 and over it by +5.6% in crowded MOT20.It differs from ByteTrack by 0.5% in MOT17 and outperforms it by +0.1% in MOT20, although the authors note an inference-speed discrepancy with CNN-based methods.
  • Ablation study: Dense image-related queries outperform limited noise-initialized sparse queries, whose naive increase produces noisier detections and higher false positives.The proposed dense representation uses 43,520 queries, corresponding to H/4 × W/4, rather than DETR’s default 100 queries.
  • Ablation study: Dual-decoder cross-attention improves MOTA by +0.8% on MOT17 and +1.0% on MOT20 but slows inference by around 50%.The PVT encoder instead raises speed from 2.1 to 8.1 fps on MOT17 and from 1.2 to 6.1 fps on MOT20, while improving MOTA by +4.2% and +3.8%.
  • Ablation study: External Re-ID is unnecessary in the reported experiment, while QLN and TransCenter Decoder choices support tracking efficiency and accuracy.The tracking procedure instead places unmatched tracks to sleep for at most T = 60 frames before discarding them.

5 CONCLUSION

TransCenter is introduced as an efficient transformer-based MOT architecture using dense image-related representations, dense detection queries, and sparse tracking queries. It outperforms competitors on MOT17 and crowded MOT20 scenes under comparable training conditions.

  • Qualitative evidence: The conclusion is supported by qualitative visualizations of tracking trajectories in very crowded MOT20 scenes under private detection.These trajectories are shown in Fig. 6.
  • Contributions: TransCenter combines dense pixel-level multi-scale detection queries with sparse tracking queries produced by query learning networks.The queries interact with image features in the TransCenter Decoder.
  • Performance: TransCenter outperforms competitors on MOT17 and by a large margin in crowded MOT20 scenes under the same training conditions.The paper also reports better performance than some methods trained with substantially more data.

APPENDIX A NUMBER OF DECODER LAYERS

The decoder-layer ablation evaluates TransCenter variants across MOT17 and MOT20 validation sets and identifies different layer counts as best for the full and Lite models.

  • Ablation setup: The ablation varies TransCenter Decoder depth from 3 to 6 layers for TransCenter, TransCenter-Lite, and TransCenter-Dual.Results are evaluated using MOTA on MOT17 and MOT20 validation sets.
  • Full models: Six decoder layers achieve the best MOTA for TransCenter and TransCenter-Dual on both MOT17 and MOT20.The search covers decoder depths within the range [3, 6].
  • Lite model: Four decoder layers achieve the best MOTA for TransCenter-Lite.This result is reported for the decoder-layer search described in the ablation.

APPENDIX B QUALITATIVE RESULTS AND VISUALIZATIONS

The appendix visualizes TransCenter’s center heatmaps, detection queries, and tracking queries, including comparisons across MOT17 and MOT20 scenes. The figures also compare TransCenter’s detection outputs with outputs from other MOT methods.

  • Visualization scope: The qualitative appendix covers center heatmap responses, detection queries, and tracking queries from TransCenter.These visualizations are organized into Sections B.1, B.2, and B.3.
  • Detection outputs: Figure 8 compares TransCenter’s center heatmap with heatmap outputs from CenterTrack and FairMOT and box centers from TransTrack.The comparison includes both heatmap-based and sparse-query-based MOT outputs.
  • Detection queries: Figure 9 uses gradient-weighted class activation mapping to visualize TransCenter detection queries across MOT20 and MOT17 scenes.Red and orange zones indicate higher response values, while blue zones indicate lower values.

B.1 Center Heatmap Response

The center-heatmap discussion compares TransCenter with sparse-query and prior center-based MOT methods. It frames dense image-related queries as a response to missed or overlapping detections associated with local or sparse predictions.

  • Comparison: TransCenter’s center heatmap is compared with CenterTrack, FairMOT, and TransTrack outputs in the qualitative analysis.The comparison includes heatmaps and bounding-box centers from the listed methods.
  • Observed issues: Sparse-query transformer methods can miss detections or produce heavily overlapping predictions, while prior center-based methods estimate centers locally.The passage identifies these issues as potentially causing false detections.
  • TransCenter design: TransCenter uses dense pixel-level image-related detection queries to produce a dense representation for heatmap-based tracking.The passage presents this design as intended to mitigate missed and overlapping detections.

B.2 Detection-Query Visualization

TransCenter’s detection queries are visualized with gradient-weighted class activation mapping by treating center-heatmap prediction as a binary 2D classification task.

  • Gradient-weighted class activation mapping is used to visualize TransCenter’s detection queries.The visualization follows the method described in.
  • The center heatmap is interpreted as a 2D classification task similar to image segmentation.
  • The center-heatmap response uses two classes: “person” and “background.”Gradients are calculated with respect to the detection queries and averaged over the image.

B.3 Tracking-Query Visualization

TransCenter visualizes sparse tracking queries through their sampling locations in the decoder, showing attention concentrated around the corresponding objects.

  • Sparse tracking queries cannot be plotted directly as images, so their sampling locations are visualized instead.The tracking queries interact with tracking memory in the TransCenter Decoder, where object features are sampled from the input memory.
  • Sampling locations are obtained by displacing previous-frame reference points representing normalized object centers.
  • Larger circles represent higher attention weights, while locations below a threshold such as 0.2 are filtered out.
  • Sampling locations surround corresponding objects, with attention weights decreasing as locations move farther away.This provides a qualitative indication that previous-frame tracking queries attend around the associated objects.
Loading 2103.15145v4…