PaperlayerView exact revision on arXiv

Source-linked AI summary

CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification

Zheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Ratnesh Kumar, David Anastasiu, Jenq-Neng Hwang

arXiv:1903.09254v4cs.CV

TL;DR

City-scale traffic-camera applications need vehicle tracking across large, multi-camera areas, but existing benchmarks lack sufficient spatial coverage and realistic video, geometry, and viewpoint information. CityFlow addresses this gap with a city-scale benchmark supporting MTMC tracking and image-based vehicle ReID, together with calibration data and extensive evaluations. The paper concludes that CityFlow broadens benchmark scale and enables vehicle-based MTMC tracking analysis.

  • Problem

    Existing benchmarks lack the spatial coverage, camera scale, original videos, calibration, and viewpoint diversity needed for city-scale MTMC vehicle tracking.

  • Method

    CityFlow provides annotated urban videos, camera geometry and calibration information, a vehicle ReID subset, and evaluations across ReID, detection, MTSC, and MTMC tracking.

  • Results

    CityFlow is presented as the first city-scale vehicle MTMC benchmark and the largest existing ReID dataset in spatial coverage and camera/intersection scale.

  • Takeaways & Limitations

    CityFlow supports analysis of vehicle-based MTMC tracking by combining original-video annotations, camera geometry, calibration, and spatio-temporal information.

  • Takeaways & Limitations

    Viewpoint variations cause most image-based ReID failures, identifying viewpoint handling as a target for future methods.

Abstract

from arXiv · show

Urban traffic optimization using traffic cameras as sensors is driving the need to advance state-of-the-art multi-target multi-camera (MTMC) tracking. This work introduces CityFlow, a city-scale traffic camera dataset consisting of more than 3 hours of synchronized HD videos from 40 cameras across 10 intersections, with the longest distance between two simultaneous cameras being 2.5 km. To the best of our knowledge, CityFlow is the largest-scale dataset in terms of spatial coverage and the number of cameras/videos in an urban environment. The dataset contains more than 200K annotated bounding boxes covering a wide range of scenes, viewing angles, vehicle models, and urban traffic flow conditions. Camera geometry and calibration information are provided to aid spatio-temporal analysis. In addition, a subset of the benchmark is made available for the task of image-based vehicle re-identification (ReID). We conducted an extensive experimental evaluation of baselines/state-of-the-art approaches in MTMC tracking, multi-target single-camera (MTSC) tracking, object detection, and image-based ReID on this dataset, analyzing the impact of different network architectures, loss functions, spatio-temporal models and their combinations on task effectiveness. An evaluation server is launched with the release of our benchmark at the 2019 AI City Challenge (https://www.aicitychallenge.org/) that allows researchers to compare the performance of their newest techniques. We expect this dataset to catalyze research in this field, propel the state-of-the-art forward, and lead to deployed traffic optimization(s) in the real world.

1. Introduction

CityFlow addresses the need for city-scale vehicle tracking across cameras by combining MTSC tracking, vehicle ReID, and spatio-temporal association. It provides broad urban coverage, annotations, calibration data, experiments, and an evaluation server.

  • Motivation: City-wide traffic optimization requires tracking vehicles across multiple cameras and intersections under varied conditions.The paper identifies this capability as a central gap in using traffic cameras as city-wide sensors.
  • Problem formulation: MTMC tracking combines within-camera MTSC tracking, cross-camera image-based ReID, and spatio-temporal information.The components connect target trajectories between cameras through visual-spatio-temporal association.
  • Benchmark: CityFlow contains 40 cameras across 10 intersections, covering city streets, residential areas, highways, and varied traffic conditions.The benchmark is presented as a city-scale dataset with synchronized high-quality videos and broad scene coverage.
  • Annotations and resources: More than 200K bounding boxes are labeled, while homography matrices provide pixel-to-GPS spatial localization.The dataset also includes a vehicle ReID subset and camera calibration information.
  • Evaluation: The paper evaluates image-based ReID, object detection, MTSC tracking, and MTMC tracking using extensive baseline and state-of-the-art experiments.An evaluation server is released to support comparison of new techniques.

2. Related benchmarks

Existing ReID and tracking benchmarks provide useful datasets but generally lack the spatial scale, camera diversity, original videos, calibration, or realistic tracking conditions needed for city-scale MTMC vehicle tracking.

  • Benchmark landscape: Public benchmarks span image-based person and vehicle ReID and video-based multi-camera tracking, with comparisons of cameras, annotations, identity samples, videos, geometry, and viewpoints.These categories and dataset properties are summarized in Table 1.
  • ReID benchmarks: Person ReID has received substantial attention, while vehicle ReID remains less accurate because vehicles have small inter-class and large intra-class variability.Viewpoint changes contribute to variation in vehicle appearance.
  • Vehicle benchmarks: Vehicle benchmarks such as VeRi-776, VehicleID, and PKU-VD generally omit original videos or calibration and often provide limited viewpoint variation.Some datasets expose only front and back vehicle views.
  • Remaining gap: These vehicle benchmarks assume perfect within-camera MTSC tracking and therefore do not facilitate realistic MTMC vehicle-tracking analysis.Spatio-temporal information is needed because appearance alone is insufficient when vehicle models look similar.

3. CityFlow benchmark

CityFlow is a city-scale vehicle-tracking benchmark spanning many cameras, intersections, scenes, and traffic conditions. It provides detailed annotations, camera geometry, an image-based ReID subset, and an evaluation server for standardized comparison.

  • Dataset overview: 3.25 hours of video from 40 cameras across 10 intersections span 2.5 km between the furthest simultaneous cameras.The dataset covers intersections, roadways, and highways.
  • Dataset overview: CityFlow is the first public benchmark supporting city-scale video analytics and MTMC vehicle tracking.Its spatial coverage, diverse scenes, and traffic conditions address limitations of smaller existing benchmarks.
  • Dataset overview: 229,680 bounding boxes cover 666 vehicle identities, each observed through at least two cameras.Videos are at least 960p, mostly 10 FPS, with video-start offsets available for synchronization.
  • Data annotation: Trajectory-level annotation combines tracking-by-detection, manual correction of detection and tracking errors, and manual cross-camera association.This process addresses misaligned boxes, missed detections, false positives, and identity switches.
  • Data annotation: Camera homographies map image pixels to GPS-defined ground coordinates, with an average converged reprojection error of 11.52 pixels.Calibration uses manually selected landmarks and optimization methods including least median of squares and RANSAC.
  • Subset for image-based ReID: CityFlow-ReID contains 56,277 bounding boxes spanning separate training and test identities, with each vehicle averaging 84.50 images across 4.55 camera views.The subset includes 333 training identities, 333 test identities, and 1,052 query images.
  • Evaluation server: An online evaluation server provides continuous submissions, a leaderboard, and common ground truths for fair comparison.It was launched with the benchmark at the 2019 AI City Challenge.

4. Evaluated baselines

The evaluation compares established methods for vehicle ReID, object detection, and single-camera tracking, while examining feature, sampling, association, and spatio-temporal choices.

  • Image-based ReID: CNN architectures for person ReID are compared under identical learning rates, training epochs, batch sizes, and Adam optimization.The experiments vary network architectures while controlling these training settings.
  • Image-based ReID: Vehicle ReID experiments include batch-based triplet sampling variants and visual-semantic feature fusion with multiple distance metrics.The tested distance functions include Bhattacharyya, L2, L1, and L∞ norms.
  • Single-camera tracking and object detection: Object detection uses YOLOv3, SSD512, and Faster R-CNN with COCO-pretrained default models and a shared detection threshold of 0.2.The target classes are car, truck, and bus.
  • Single-camera tracking and object detection: Single-camera tracking evaluates online DeepSORT and MOANA alongside offline TC, covering appearance, Kalman filtering, Hungarian association, and tracklet clustering.Offline methods can use aggregated tracklets, whereas online methods compensate for unavailable future information with appearance features.
  • Spatio-temporal association: Spatio-temporal association methods exploit predictable vehicle movement, camera-transition timing, and physical distance between views.The evaluated approaches include ratio-based re-ranking, Gaussian transition models, and predefined temporal distributions.

5. Experimental evaluation results

CityFlow evaluates image-based ReID, MTSC tracking, detection, and MTMC tracking, showing that vehicle ReID remains challenging and that visual, tracking, and spatio-temporal components jointly affect MTMC performance.

  • Image-based ReID: Hard triplet loss is the most robust, while combining hard triplet and cross-entropy losses yields the best results; DenseNet121 achieves the highest accuracy among architectures.These comparisons are reported for person ReID methods evaluated on CityFlow-ReID.
  • Image-based ReID: CityFlow-ReID is challenging: the top mAP remains below 35% despite person ReID methods and metric-learning losses.The benchmark evaluates whether existing ReID methods transfer effectively to vehicle imagery.
  • Image-based ReID: Most ReID failures are caused by viewpoint variations, identifying viewpoint handling as a key target for future methods.The qualitative results are summarized through cumulative match curves and visualizations.
  • MTSC tracking and object detection: SSD512 performs best among object detectors, while the offline TC tracker performs better on most MTSC metrics.DeepSORT and MOANA have similar MOTA, but DeepSORT has much higher ID F1; MOANA successfully tracks most trajectories.
  • MTMC tracking: MTMC tracking first applies MTSC tracking, then samples three signatures per vehicle trajectory to compare appearance features with spatio-temporal information.False positives are included in MTMC accuracy because the evaluation considers trajectories spanning multiple cameras.
  • MTMC tracking: FVS is the most reliable spatio-temporal association method, using a manually specified probabilistic transition-time model.2WGMMF is comparable in most cases, whereas PROVID performs worse without probabilistic modeling.

6. Conclusion

The paper presents CityFlow as a city-scale benchmark for video-based MTMC tracking and image-based vehicle ReID, with broad spatial coverage and calibration resources. Extensive evaluations show challenging scenarios and support further visual-spatio-temporal research.

  • Benchmark contribution: CityFlow enables both video-based MTMC tracking and image-based vehicle ReID tasks at city scale.It is presented as the first benchmark supporting vehicle-based MTMC tracking with original-video annotations, camera geometry, and calibration information.
  • Benchmark contribution: The benchmark covers diverse scenes and traffic-flow conditions while providing spatial information that can help resolve ambiguity in image-based ReID.Its scale is characterized by spatial coverage and the number of cameras and intersections involved.
  • Research implications: CityFlow’s challenging, realistic scenarios support research on visual-spatio-temporal association and related problems such as vehicle pose estimation and viewpoint generation.The conclusion frames these directions as opportunities opened by the benchmark.

7. Supplementary

The supplementary material defines the tracking and ReID evaluation measures used in the benchmark, including CLEAR MOT, identity-based MTMC metrics, and rank-100 mAP. It also reports supplementary rank-100 mAP results for several ReID experiments.

  • 7.1. Metrics in CLEAR MOT: CLEAR MOT evaluates tracking using bounding-box intersection over union, with a 50% similarity threshold for true positives.MOTA combines false negatives, false positives, and identity switches; its range is (−∞, 100].
  • 7.1. Metrics in CLEAR MOT: MOTP measures localization misalignment from bounding-box overlap, while MOTA is considered more reliable because MOTP varies little across methods.MOTP averages overlap over matched predictions and ground truths.
  • 7.1. Metrics in CLEAR MOT: MT, ML, PT, and FAR characterize tracking coverage, respectively counting mostly tracked, mostly lost, partially tracked, and false-alarm behavior.MT requires at least 80% of a target’s lifespan, whereas ML covers less than 20%.
  • 7.3. Rank-K mAP for evaluating image-based ReID: Supplementary tables report rank-100 mAP for non-metric CNN-feature methods, person-ReID metric-learning methods, and vehicle-ReID sampling variants.Tables 10–12 correspond to earlier experiments in Tables 3, 4, and 6.
  • 7.2. Metrics in DukeMTMC: DukeMTMC metrics IDP, IDR, and IDF1 use identity-level false-negative, false-positive, and true-positive counts rather than frame-by-frame matching.IDF1 measures correctly identified detections against the average number of true and computed detections.
  • 7.3. Rank-K mAP for evaluating image-based ReID: Rank-K mAP reduces ReID submission matrices from Q × T to Q × K by averaging precision over each query’s top K matches.The evaluation server uses rank-100 mAP, with K chosen as 100 because it generally exceeds the maximum ground-truth trajectory length.
Loading 1903.09254v4…