PaperlayerView exact revision on arXiv

Source-linked AI summary

UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking

Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei, Ming-Ching Chang, Honggang Qi, Jongwoo Lim, Ming-Hsuan Yang, Siwei Lyu

arXiv:1511.04136v4cs.CV

TL;DR

Existing MOT evaluations often separate detection from tracking, leaving the effect of detection accuracy on complete systems insufficiently examined. The paper introduces the UA-DETRAC benchmark and jointly evaluates detector–tracker combinations, finding that MOT performance depends strongly on detection settings and motivating detection-aware evaluation tools and metrics.

  • Problem

    Existing MOT evaluations separate detection and tracking, limiting comprehensive analysis of detection accuracy’s effect on complete MOT performance.

  • Method

    The paper introduces the UA-DETRAC dataset and performs extensive experiments evaluating complete MOT systems with detection and tracking components.

  • Results

    MOT-system performance and relative method rankings vary significantly under different object-detection settings.

  • Takeaways & Limitations

    MOT evaluation should consider object detection and tracking jointly rather than rely on a single predefined detection setting.

  • Takeaways & Limitations

    The proposed evaluation protocol can only be used to

Abstract

from arXiv · show

In recent years, numerous effective multi-object tracking (MOT) methods are developed because of the wide range of applications. Existing performance evaluations of MOT methods usually separate the object tracking step from the object detection step by using the same fixed object detection results for comparisons. In this work, we perform a comprehensive quantitative study on the effects of object detection accuracy to the overall MOT performance, using the new large-scale University at Albany DETection and tRACking (UA-DETRAC) benchmark dataset. The UA-DETRAC benchmark dataset consists of 100 challenging video sequences captured from real-world traffic scenes (over 140,000 frames with rich annotations, including occlusion, weather, vehicle category, truncation, and vehicle bounding boxes) for object detection, object tracking and MOT system. We evaluate complete MOT systems constructed from combinations of state-of-the-art object detection and object tracking methods. Our analysis shows the complex effects of object detection accuracy on MOT system performance. Based on these observations, we propose new evaluation tools and metrics for MOT systems that consider both object detection and object tracking for comprehensive analysis.

1. Introduction

The paper argues that evaluating detection and tracking separately is insufficient for understanding complete MOT systems. It introduces the UA-DETRAC dataset and a protocol that jointly analyzes detection accuracy and tracking performance.

  • Motivation: MOT systems combine object detection to locate targets in each frame with tracking to generate trajectories across frames.The paper explicitly defines an MOT system as detection + tracking.
  • Motivation: Existing evaluations commonly use fixed detection results, limiting analysis of how detection accuracy affects complete MOT performance.The paper describes this separation as insufficient for analyzing complete systems and potentially producing uninformative conclusions.
  • Dataset: The UA-DETRAC dataset contains 100 challenging real-world traffic videos, more than 140,000 frames, and 1.21 million labeled vehicle bounding boxes with attributes.Annotations include illumination, vehicle type, and occlusion, while the dataset targets vehicle detection and tracking.
  • Dataset: UA-DETRAC focuses on vehicle detection and tracking in traffic scenes while introducing challenges for object detection and tracking algorithms.The paper contrasts its focus with other self-driving-car datasets and emphasizes thorough MOT annotations.
  • Evaluation: The study evaluates complete MOT systems by combining detection and tracking methods across UA-DETRAC, MOT16, and KITTI-T datasets.The experiments vary detector thresholds and combine multiple detection and tracking algorithms.
  • Results: MOT-system performance and method rankings vary significantly with detection settings, including a case where CEM and DCT reverse their MOTA ordering.CEM scores higher than DCT at precision-recall values (0.433, 0.341), but lower at (0.674, 0.183).
  • Contribution: The proposed protocol and metrics explicitly account for the object-detection module when evaluating MOT systems.The protocol is motivated by the observed dependence of complete-system behavior on detection accuracy.

2. UA-DETRAC Benchmark Dataset

UA-DETRAC is a vehicle-focused benchmark built from diverse real-world traffic videos and designed to evaluate detection, tracking, and complete MOT systems.

  • The UA-DETRAC dataset consists of 100 videos selected from more than 10 hours of traffic imagery.The videos were acquired at 24 locations representing urban highways, traffic crossings, and T-junctions.
  • Data collection spans varied locations, traffic patterns, illumination conditions, and shooting angles.These design choices provide variation across the captured traffic scenes.
  • Videos are recorded at 25 frames per second with JPEG images at 960 × 540 pixels.
  • The benchmark evaluates both detection and tracking methods through a submission protocol similar to MOT15.It is intended to support performance evaluation of complete MOT systems rather than tracking alone.

2.1. Data Collection and Annotation

UA-DETRAC provides extensive vehicle annotations and structured train/test evaluation across diverse visual conditions, difficulty levels, and detection-tracking systems.

  • Data Collection and Annotation: More than 140,000 frames containing 8,250 vehicles and 1.21 million vehicle bounding boxes are annotated.The annotation effort involved more than 10 domain experts and multiple rounds of cross-checking.
  • Data Collection and Annotation: The dataset is split into 60 training and 40 testing sequences recorded at different locations but with similar traffic conditions and attributes.This setting reduces the chances of methods overfitting to particular scenarios.
  • Data Collection and Annotation: MOT evaluation combines six representative tracking methods with four representative object detection methods.The tracking methods include GOG, CEM, DCT, IHTLS, H2T, and CMOT; detectors include DPM, ACF, R-CNN, and CompACT.
  • Data Collection and Annotation: Annotations cover vehicle type, illumination, scale, occlusion ratio, and truncation ratio.Vehicle types include car, bus, van, and other vehicles; scale uses small, medium, and large categories, while occlusion uses no, partial, and heavy categories.

2.2. Relevance to Existing Benchmark Datasets

UA-DETRAC complements existing detection and tracking benchmarks by targeting vehicle surveillance with extensive video, bounding-box, and attribute annotations, while supporting both detection and MOT evaluation.

  • Object Detection Datasets: Existing detection datasets are mainly designed for object detection in single images, whereas UA-DETRAC targets vehicle surveillance in traffic scenes.Related datasets include PASCAL VOC, ImageNet, Caltech, KITTI-D, and KAIST.
  • Object Tracking Datasets: Unlike pedestrian-focused MOT15 and MOT16 benchmarks, UA-DETRAC is designed for vehicle detection and tracking in traffic scenes.MOT15 provides a unified platform and evaluation protocol, while MOT16 extends it with more challenging sequences and thorough annotations.
  • Object Tracking Datasets: UA-DETRAC differs from existing MOT datasets through significantly more video frames, annotated bounding boxes, and attributes.Its vehicles are captured from different viewing angles and are frequently occluded.
  • Object Tracking Datasets: UA-DETRAC is designed for performance evaluation of both object detection and multi-object tracking.The benchmark summarizes differences from existing datasets across multiple aspects.

2.3. Object Detection Algorithms

This section reviews object detection methods used in MOT, spanning hand-crafted features, part-based models, CNN-based detectors, proposal methods, and newer benchmarked algorithms.

  • The review covers object detection for MOT across pedestrians, faces, and vehicles, and describes 12 evaluated algorithms in UA-DETRAC.
  • CNN-based detectors such as R-CNN and Faster R-CNN use learned hierarchical features and region proposals to improve detection accuracy and speed.R-CNN combines region proposals with CNNs, while Faster R-CNN learns proposals through a region proposal network.
  • Traditional detectors include DPM, which models part positions as latent variables but is computationally expensive for practical applications.
  • The evaluated UA-DETRAC detectors are trained or retrained on UA-DETRAC-train and evaluated on UA-DETRAC-test, including GP-FRCNN and RD2.
  • The benchmarked detector set includes DPM, ACF, R-CNN, CompACT, Faster R-CNN, EB, YOLOv3, GP-FRCNN, CSP, HAT, FG-BR Net, and RD2.

2.4. Object Tracking Algorithms

This section reviews MOT algorithms, including probabilistic association, hypothesis tracking, optimization-based approaches, and tracking-by-detection systems.

  • The paper reviews ten state-of-the-art object tracking approaches evaluated in the study.
  • Tracking-by-detection methods link detections from consecutive frames, take per-frame detections as input, and generate target trajectories.
  • JPDAF considers all target-detection matches jointly, but its computational complexity becomes intractable as the number of targets increases.
  • MHT evaluates trees of potential track hypotheses over multiple time steps and is more effective than JPDAF for long-term association at higher computational cost.
  • Other approaches formulate detection-tracklet association as optimization using shortest paths, independent sets, cliques, network flows, linear programs, or assignment methods.
  • The experiments evaluate trackers including GOG, CEM, DCT, IHTLS, H2T, CMOT, TBD, IOU, and KIOU on UA-DETRAC.

3. UA-DETRAC Evaluation Protocol

The UA-DETRAC protocol evaluates complete MOT systems across detector precision-recall behavior and tracking metrics, rather than relying on a single fixed detection input.

  • Existing protocols commonly use one predefined detection setting, which does not fully reflect complete MOT performance because detection results significantly affect tracking.
  • The protocol generates detector precision-recall curves by varying detection thresholds and evaluates detections at a 0.7 bounding-box overlap threshold.
  • MOTA combines false negatives, false positives, and identity switches, whereas MOTP measures average dissimilarity between true positives and matched ground-truth targets.
  • PR-MOTA is computed as the average MOTA over the precision-recall curve, while analogous PR metrics are computed for MOTP, MT, ML, IDS, FM, FP, and FN.
  • For each precision-recall operating point, a tracker processes the detections to produce a three-dimensional PR-MOTA curve relating detection precision and recall to MOTA.
  • The protocol considers complete detector performance and therefore better reflects the behavior of whole MOT systems than fixed or synthetically perturbed detection inputs.

4. Analysis and Discussion

The analysis evaluates detection across challenging conditions and complete MOT combinations, showing that detector quality affects tracking in complex, tracker-dependent ways.

  • Object detection: Recent detectors exceed 60% AP overall, but scores drop 15–20% from easy to hard subsets; RD2 reaches 76.64% AP on hard scenes.Faster R-CNN, YOLOv3, and RD2 achieve 62.13%, 76.17%, and 85.35% AP, respectively, on the benchmark.
  • Object detection: Detection remains difficult in poor illumination: most methods fall below 80% AP in rainy and night scenes, whereas RD2 reaches 86.59% AP at night.RD2 performs better under cloudy and sunny conditions, reaching 89.67% and 90.49% AP.

5. Run-time Performance

The runtime analysis highlights a trade-off between accuracy and computational cost: deep-learning detectors are more accurate when resources are available, while lighter systems are preferable under constraints.

  • Runtime and accuracy: Deep-learning detectors are more accurate than DPM and ACF when computing resources are not constrained.The paper notes that runtime comparisons are difficult because methods use different platforms and some require GPUs for training and testing.
  • Application settings: EB+IOU and EB+KIOU provide the most accurate results with relatively high efficiency for surveillance applications.The evaluation considers both accuracy and speed across the reported MOT and runtime tables.
  • Application settings: Under constrained computing resources, ACF+CMOT and ACF+H2T achieve higher PR-MOTA and lower PR-IDS and PR-FM than alternatives.These systems are presented as more suitable when computational resources limit the use of deep-learning approaches.

6. Conclusions and Future Research Directions

The paper presents UA-DETRAC as a large-scale benchmark for jointly evaluating detection, tracking, and complete MOT systems. Its conclusions emphasize scenario-specific metrics, detector–tracker interaction, computational constraints, and future dataset expansion.

  • Contributions: UA-DETRAC contains 100 videos with rich annotations and supports performance evaluation of both detection and tracking methods.The benchmark is used for extensive experiments involving detection and tracking approaches.
  • Protocol and metrics: The proposed protocol is limited to tracking methods that take detection boxes as inputs, although likelihood maps can be thresholded into boxes.The authors also identify the need for a protocol that avoids tuning input precision and recall rates to reduce evaluation complexity.
  • Protocol and metrics: Evaluation metrics should reflect application priorities: surveillance emphasizes identity switches, whereas autonomous driving emphasizes avoiding false negatives.The paper discusses adapting PR-MOTA and PR-MOTP to different application scenarios and calls for comprehensive metrics.
  • Joint detection and tracking: Detection performance significantly affects tracking, while temporal coherency in tracking can also help detection.The authors identify combining detection and tracking in a unified framework as a promising direction for further gains.
  • Real-time issue: Deep-learning methods outperform other approaches in accuracy but require substantial computational resources, motivating work on efficient real-time systems.Pruning, compression, and low-bit representations are cited as approaches for adapting networks to embedded platforms.
  • Future data: The authors plan to extend UA-DETRAC with more sequences and richer annotations, including pedestrian detection and tracking studies.Future work will also examine how the quantity and type of training data affect detection and tracking performance.

Appendix

The appendix establishes the range of the PR-MOTA score by integrating MOTA along the precision–recall curve and bounding the result using precision and recall ranges.

  • Definition: PR-MOTA is defined as a line integral of the MOTA value along the precision–recall curve.The curve is divided into arcs whose lengths are used in the range proof.
  • Range proof: Because MOTA values lie in (−∞, 100], the lower bound of PR-MOTA is −∞.The upper-bound argument uses the bounded precision and recall coordinates on the PR curve.
  • Range proof: The proof concludes that PR-MOTA lies in (−∞, 100].The bound follows from the total variation limits of precision and recall along the curve.
  • Range proof: Equality at the upper bound requires idealized conditions: zero precision at nonzero recall, zero recall at nonzero precision, and perfect MOTA for every input detection.The appendix notes that these conditions generally do not hold in real-world applications.
Loading 1511.04136v4…