PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of Detector Architecture, Model Scale, and Scene Density on Multi-Pedestrian Tracking

Pedro Manoel Herminio Alves, Dalton Cézane Gomes Valadares, Cláudio de Souza Baptista

arXiv:2610.19369v1cs.CVcs.LG

TL;DR

Pedestrian tracking must maintain each person’s identity through video despite interactions and occlusions, and it is unclear whether end-to-end detection’s distinction translates into tracking gains. The study compares YOLO11, RT-DETR, and YOLO26 paired with ByteTrack and BoT-SORT, and tests YOLO26 inference with NMS versus end-to-end inference. RT-DETR achieved the highest HOTA on every benchmark with both trackers, with a clearer advantage on more challenging benchmarks, especially MOT20.

  • Problem

    Pedestrian tracking must maintain each person’s identity through video despite interactions and occlusions, and it is unclear whether end-to-end detection’s distinction translates into tracking gains.

    Source: §I · p. 1

  • Method

    The study compares YOLO11, RT-DETR, and YOLO26 paired with ByteTrack and BoT-SORT, and tests YOLO26 inference with NMS versus end-to-end inference.

    Source: A. Experimental Design · p. 2

  • Results

    RT-DETR achieved the highest HOTA on every benchmark with both trackers, with a clearer advantage on more challenging benchmarks, especially MOT20.

    Source: §V · p. 5

  • Takeaways & Limitations

    Within the evaluated protocol, RT-DETR suits HOTA priority, while YOLO26-L with NMS offers a favorable HOTA–speed balance on MOT17 and SOMPT22.

    Source: §V · pp. 5–6

Abstract

from arXiv · show

Pedestrian tracking in video is important for applications such as surveillance and scene analysis, but it becomes more challenging in crowded scenarios with frequent occlusions and rapid motion. This paper analyzes how detector architecture, model scale, scene density, and inference regime affect the tracking of multiple pedestrians. We compare the YOLO11, RT-DETR, and YOLO26 detector families with the ByteTrack and BoT-SORT trackers on sequences from the MOT17, MOT20, and SOMPT22 benchmarks. For YOLO26, we also compare traditional overlap filtering via Non-Maximum Suppression (NMS) with end-to-end inference. The RT-DETR family achieved the highest mean scores on the HOTA tracking quality metric across all three benchmarks with both trackers, with a more pronounced advantage on MOT20, albeit at a higher computational cost. On MOT17 and SOMPT22, YOLO26 variants, especially YOLO26-L with NMS, offered favorable combinations of quality and speed compared with RT-DETR. In the YOLO26 inference comparison, the end-to-end mode did not yield consistent tracking gains over the NMS-based regime. Taken together, these results provide a practical guide for selecting detectors and trackers for this task based on application priorities.

I. INTRODUCTION

Pedestrian tracking must preserve identities across video despite interactions, occlusions, and entries and exits, and its difficulty rises in crowded scenes. This study compares detector families, scales, scene densities, trackers, and YOLO26 inference regimes to examine their effects on tracking.

I. INTRODUCTION

  • Tracking-by-detection separates frame-level box detection from temporal association, making tracking performance directly dependent on detector characteristics.Detectors provide bounding boxes and confidence scores; trackers associate detections across consecutive frames into trajectories.Source: §I · p. 1
  • Crowded scenes increase occlusions, trajectory fragmentations, and identity switches, so the same detector–tracker choice can behave differently across visual contexts.The task is to preserve each pedestrian’s identity across the video, not merely detect people in individual frames.Source: §I · p. 1
  • Whether end-to-end inference improves tracking over NMS remains unclear, despite growing attention to this distinction at the detection level.NMS removes redundant boxes, whereas end-to-end detectors directly produce final predictions with reduced or no external post-processing.Source: §I · p. 1
  • The study compares YOLO11, RT-DETR, and YOLO26 with ByteTrack and BoT-SORT on MOT17, MOT20, and SOMPT22, and tests NMS against end-to-end inference for YOLO26.Its stated contributions include comparing detector–tracker combinations across benchmarks with different average pedestrian densities and assessing inference regime from a tracking perspective.Source: §I · p. 1

II. RELATED WORK

Prior studies show that detector choice affects tracking and that end-to-end detection is advancing, but they do not examine these factors together across detector families, scales, output formats, and benchmarks with contrasting densities. The paper presents a systematic evaluation of YOLO11, RT-DETR, and YOLO26 with ByteTrack and BoT-SORT on MOT17, MOT20, and SOMPT22.

II. RELATED WORK

  • Prior work compared YOLO-family detectors and trackers on MOT17 and MOT20, while other studies examined newer YOLO and RT-DETR detectors or integrated RT-DETR into a tracking pipeline.The cited studies include detector–tracker combinations and detection-level comparisons, but the paper distinguishes its combined protocol from those narrower scopes.Source: §II · pp. 1–2
  • The claimed gap is a unified evaluation of detector family, model scale, detection-output method, and benchmark density within one protocol.Earlier work focused mainly on detector–tracker pairs or detection-level comparisons, without directly testing output format’s tracking effects or including a complementary benchmark scenario.Source: §II · p. 2
  • The study evaluates YOLO11, RT-DETR, and YOLO26 across available scales on MOT17, MOT20, and SOMPT22 with ByteTrack and BoT-SORT.The authors describe this as the first systematic evaluation of these detector families across their available scales on these benchmarks using both trackers.Source: §II · p. 2

A. Experimental Design

The experimental design varies detector family, scale, benchmark density, and YOLO26 inference regime while pairing detectors with two trackers. Models use public COCO-pretrained weights without benchmark-specific fine-tuning, and detector configurations include convolutional and Transformer-based alternatives.

A. Experimental Design

  • The protocol compares YOLO11, RT-DETR, and YOLO26 with ByteTrack and BoT-SORT across detector scales and benchmark density, plus YOLO26 NMS and end-to-end inference.The design targets four aspects: detector family, model scale, performance across benchmarks with different average pedestrian densities, and inference regime.Source: A. Experimental Design · p. 2
  • All models use public COCO-pretrained weights without benchmark-specific fine-tuning, focusing the comparison on detector, scale, tracker, and inference-regime choices.The authors state this keeps the protocol controlled and closer to evaluating models as released; the results therefore reflect pretrained weights in new scenarios.Source: A. Experimental Design · p. 2

B. Detectors and Trackers

  • RT-DETR is a Transformer-based detector with a hybrid encoder using global attention and a decoder with bipartite matching; YOLO26 enables an NMS-versus-end-to-end comparison within one convolutional family.The evaluated detectors use 640-pixel input, a 0.25 confidence threshold, person-class detections, and an IoU threshold of 0.7 for NMS inference.Source: B. Detectors and Trackers · p. 2
  • Evaluated detector scales are the officially released nano, small, medium, large, and extra-large variants where available, with YOLO26 tagged by inference regime.The text and tables use uppercase scale suffixes, such as YOLO11-L and RT-DETR-X, and tags (NMS) and (E2E) for YOLO26.Source: B. Detectors and Trackers · p. 2
  • ByteTrack uses position, motion, and low-confidence detections for association, whereas BoT-SORT adds motion compensation and OSNet-based appearance re-identification at greater computational complexity.The trackers were run through BoxMOT with default parameters; Table I summarizes detector parameter counts and FLOPs.Source: B. Detectors and Trackers · p. 2, Table I · p. 2

C. Datasets

The three benchmarks represent different pedestrian-tracking conditions: MOT17 as an urban reference, MOT20 as an extremely dense setting, and SOMPT22 as elevated-camera static surveillance. Three sequences per benchmark were evaluated, with reported average densities highest in MOT20.

C. Datasets

  • The benchmark set spans reference urban scenes (MOT17), extremely dense scenes with severe occlusions (MOT20), and elevated-camera static surveillance (SOMPT22).Figure 2 provides example frames from the evaluated sequences.Source: C. Datasets · pp. 2–3, Figure 2 · p. 3
  • Three sequences per benchmark were evaluated, with average densities of 28.2 pedestrians per frame for MOT17, 47.7 for SOMPT22, and 149.2 for MOT20.The selected sequences were 02, 04, and 05 from MOT17; 02, 03, and 05 from MOT20; and 07, 10, and 12 from SOMPT22.Source: C. Datasets · p. 3

D. Evaluation Metrics

HOTA is the main tracking-quality metric, with IDF1 and MOTA providing complementary views; IDSW and Frag aid error interpretation. The measurement protocol averages five runs and sequences, while speed comparisons are specific to the tested hardware.

D. Evaluation Metrics

  • HOTA jointly balances detection, temporal association, and localization, while IDF1 analyzes identity preservation and MOTA summarizes system errors.Source: D. Evaluation Metrics · p. 3
  • IDSW counts identity switches, Frag indicates trajectory interruptions, and computational efficiency is reported using FPS and latency.Source: D. Evaluation Metrics · p. 3

E. Measurement Protocol

  • Each configuration was run five times per sequence; quality metrics are means over runs and sequences, while FPS and latency are reported as mean and standard deviation.Quality metrics varied by approximately 0.03 HOTA points on average; FPS and latency should be treated as relative comparisons on the specified hardware.Source: E. Measurement Protocol · p. 3

IV. RESULTS AND DISCUSSION

RT-DETR achieved the highest HOTA on every benchmark with both trackers, with its advantage clearest on denser benchmarks. Error-counter comparisons show that lower IDSW and Frag do not necessarily coincide with higher HOTA.

A. Overview of Tracking Quality

B. Effect of Scene Density and Model Scale

HOTA declines as the evaluated benchmarks become denser, but RT-DETR’s decline is less severe than those of YOLO26-X with NMS and YOLO11-X. Scale benefits vary by detector family, benchmark, and tracker.

B. Effect of Scene Density and Model Scale

  • With ByteTrack, HOTA fell 55.0% from MOT17 to MOT20 for RT-DETR-X, versus 68.7% for YOLO26-X with NMS and 70.8% for YOLO11-X.With BoT-SORT, the respective drops were 65.9%, 74.6%, and 75.4%.Source: B. Effect of Scene Density and Model… · p. 4
  • RT-DETR’s relative advantage was clearer in denser benchmarks, especially MOT20, though those benchmarks also differ in factors beyond density.The proposed spatial-context explanation is a hypothesis, not experimentally confirmed; the detector’s internal mechanisms were not isolated.Source: B. Effect of Scene Density and Model… · p. 4
  • Increasing model scale did not guarantee higher HOTA: YOLO11 gains from nano to medium on MOT17 with ByteTrack reversed at larger scales and on MOT20.For YOLO26, NMS large or extra-large variants had the family’s highest mean HOTA in all six benchmark–tracker combinations, while end-to-end BoT-SORT peaks varied by benchmark.Source: B. Effect of Scene Density and Model… · pp. 4–5
  • The reported metrics show MOT20’s performance decline but do not separate the contributions of detection and association to that decline.Source: B. Effect of Scene Density and Model… · p. 5

C. Comparison of ByteTrack and BoT-SORT

ByteTrack and BoT-SORT preserve a similar detector-family hierarchy but yield different quality and error-counter profiles. Computational efficiency also varies by tracker, detector, and scale, creating quality–speed trade-offs.

C. Comparison of ByteTrack and BoT-SORT

  • On SOMPT22 and MOT20, ByteTrack achieved higher HOTA in every configuration, while BoT-SORT yielded lower IDSW and Frag values across configurations.For RT-DETR-X on MOT20, BoT-SORT reduced IDSW from 138.60 to 29.27 and Frag from 301.13 to 98.33; those reductions did not consistently raise HOTA.Source: C. Comparison of ByteTrack and BoT-SORT · p. 5

D. Trade-off between Quality and Computational Efficiency

  • ByteTrack was consistently faster than BoT-SORT across all configurations.Source: D. Trade-off between Quality and… · p. 5
  • On MOT17 with ByteTrack, RT-DETR-X reached 38.0 FPS, compared with 71.9 FPS for YOLO26-X with NMS and 89.1 FPS for YOLO26-M with NMS.RT-DETR had the lowest FPS among detectors, while nano and small YOLO variants reached the highest values depending on benchmark and tracker.Source: D. Trade-off between Quality and… · p. 5
  • On MOT17 and SOMPT22, YOLO26 variants offered quality–speed alternatives to RT-DETR; YOLO26-L with NMS was 0.9–6.5% below the best-mean-HOTA RT-DETR variant while achieving 27.6–96.8% higher FPS.Smaller variants are candidates when speed dominates, while RT-DETR is an option when maximizing HOTA is the priority in the evaluated scenarios.Source: D. Trade-off between Quality and… · p. 5

E. NMS and End-to-End Ablation in YOLO26

In the YOLO26 ablation, NMS generally outperformed end-to-end inference on tracking quality, while end-to-end speed gains were small and inconsistent. This pattern held across the evaluated benchmarks and trackers.

E. NMS and End-to-End Ablation in YOLO26

  • With ByteTrack, NMS achieved higher mean HOTA than end-to-end inference at every evaluated YOLO26 scale on all three benchmarks.Source: E. NMS and End-to-End Ablation in YOLO26 · p. 5
  • Averaged across five scales with ByteTrack, NMS exceeded end-to-end HOTA by 0.56 points on MOT17, 1.35 on SOMPT22, and 0.80 on MOT20.The end-to-end mode also increased mean fragmentations by 40.5, 108.6, and 112.2, respectively.Source: E. NMS and End-to-End Ablation in YOLO26 · p. 5
  • With BoT-SORT, end-to-end inference again increased fragmentation at all scales, while FPS differences were small and did not consistently favor either regime.The trend persisted despite isolated exceptions.Source: E. NMS and End-to-End Ablation in YOLO26 · p. 5
  • Overall, NMS generally produced higher HOTA and lower fragmentation, with no systematic tracking advantage for end-to-end inference under the evaluated conditions.Possible causes, including lower temporal stability or a different confidence distribution, were not investigated.Source: E. NMS and End-to-End Ablation in YOLO26 · p. 5

V. CONCLUSION

RT-DETR achieved the highest HOTA on every benchmark with both trackers, with a clearer advantage on the more challenging benchmarks, especially MOT20. The conclusions also identify tracker trade-offs and detector choices for prioritizing HOTA or speed.

V. CONCLUSION

  • RT-DETR achieved the highest HOTA on every benchmark with both trackers, with a clearer advantage on the more challenging benchmarks, especially MOT20.Source: §V · p. 5
  • ByteTrack was consistently faster and achieved higher HOTA on SOMPT22 and MOT20 in every configuration, while BoT-SORT yielded fewer identity switches and fragmentations.Source: §V · p. 5
  • Within the evaluated protocol, RT-DETR is preferable when HOTA is the priority, particularly on MOT20.YOLO26-L with NMS offers a favorable HOTA-speed balance on MOT17 and SOMPT22, while nano and small variants remain candidates when speed dominates.Source: §V · pp. 5–6
  • Future work may extend evaluation to more sequences and benchmarks, add two-stage or fine-tuned detectors and additional trackers, and compare inference-regime latency on CPU.Source: §V · p. 6

Related papers

Loading 2610.19369v1…