PaperlayerView exact revision on arXiv

Source-linked AI summary

Detection, Tracking, and Counting Meets Drones in Crowds: A Benchmark

Longyin Wen, Dawei Du, Pengfei Zhu, Qinghua Hu, Qilong Wang, Liefeng Bo, Siwei Lyu

arXiv:2105.02440v1cs.CV

TL;DR

Drone-captured crowd analysis lacks large-scale benchmarks covering consistent video-based counting and tracking despite challenges such as viewpoint and scale variation, clutter, and small targets. The paper builds the DroneCrowd dataset and proposes STNNet, which jointly estimates density, localizes people, and tracks them using neighboring context. Experiments report favorable performance against state-of-the-art algorithms across these tasks.

  • Problem

    Drone-captured crowd counting and tracking lack publicly available large-scale benchmarks, while viewpoint and scale variation, background clutter, and small targets remain challenges.

    Source: §1 · p. 1

  • Method

    The paper constructs DroneCrowd and proposes STNNet to jointly estimate density maps, localize people, and track them using a neighboring context loss.

    Source: §1 · pp. 1–2

  • Results

    STNNet performs favorably against 12 state-of-the-art algorithms on DroneCrowd for density estimation, localization, and tracking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present DroneCrowd and STNNet as resources intended to facilitate research and development in drone-based crowd localization, tracking, and counting.

    Source: §6 · p. 8

    The reported crowd-tracking results remain far from satisfactory.

    Source: §5 · p. 8

Abstract

from arXiv · show

To promote the developments of object detection, tracking and counting algorithms in drone-captured videos, we construct a benchmark with a new drone-captured largescale dataset, named as DroneCrowd, formed by 112 video clips with 33,600 HD frames in various scenarios. Notably, we annotate 20,800 people trajectories with 4.8 million heads and several video-level attributes. Meanwhile, we design the Space-Time Neighbor-Aware Network (STNNet) as a strong baseline to solve object detection, tracking and counting jointly in dense crowds. STNNet is formed by the feature extraction module, followed by the density map estimation heads, and localization and association subnets. To exploit the context information of neighboring objects, we design the neighboring context loss to guide the association subnet training, which enforces consistent relative position of nearby objects in temporal domain. Extensive experiments on our DroneCrowd dataset demonstrate that STNNet performs favorably against the state-of-the-arts.

1. Introduction

Drone-captured crowd analysis faces viewpoint and scale variation, background clutter, small targets, and a shortage of large-scale benchmarks with consistent video annotations. The paper introduces DroneCrowd and STNNet to address this gap across density estimation, localization, and tracking.

1. Introduction

  • Drone-captured crowd algorithms face viewpoint and scale variation, background clutter, and small targets, while existing benchmarks mostly cover still images or inconsistent surveillance frames.The paper identifies the lack of publicly available large-scale benchmarks for drone-based crowd counting and tracking.Source: §1 · p. 1
  • DroneCrowd contains 112 clips and 33,600 frames across 70 scenarios in four Chinese cities, with more than 4.8 million head annotations and video-level attributes.The paper describes it as the largest and most thoroughly annotated dataset for density estimation, localization, and tracking to date.Source: §1 · p. 1
  • STNNet jointly performs density map estimation, localization, and tracking using feature extraction, density heads, and localization and association subnets.Its feature extractor uses two-branch CNNs for multi-scale features and computes correlations between consecutive frames.Source: §1 · p. 1
  • The neighboring context loss penalizes large temporal displacements in nearby objects’ relative positions to guide motion-offset prediction, and experiments report favorable performance against 12 state-of-the-art algorithms.Table 1 compares DroneCrowd with existing datasets.Source: §1 · p. 2, Table 1 · p. 2

2. Related Work

Prior crowd datasets cover image counting, surveillance video counting, or narrower tasks, while the paper positions DroneCrowd as a large-scale drone-captured dataset for counting, localization, and tracking. Related methods use density estimation and temporal information, but STNNet also produces target locations and motion offsets.

2. Related Work

  • Earlier datasets include image collections such as UCF CC 50, ShanghaiTech, and UCF-QNRF [10], while Wang et al. provide crowded-scene counting and localization images.The paper notes that these datasets remain limited in size and scenario coverage.Source: §2 · p. 2
  • Video counting datasets include UCSD, Mall, WorldExpo, and Fang et al.’s 15K-frame dataset [5], whereas DroneCrowd adds drone-captured sequences with head annotations and trajectories.DroneCrowd is described as 112 sequences with more than 4.8 million head annotations on 20,800 trajectories.Source: §2 · p. 2
  • Crowd-counting methods commonly estimate density maps, with prior approaches addressing scale variation, temporal consistency, or dependencies among counting and localization features [29] [15] [17] [20] [34].Video methods also use spatio-temporal models or people flows [38] [43] [18].Source: §2 · pp. 2–3
  • Unlike prior video methods described here, STNNet outputs both crowd density and target locations, and estimates motion offsets using neighboring-target context.The neighboring context loss trains the model to capture context among neighboring targets.Source: §2 · p. 3

3. DroneCrowd Dataset

DroneCrowd is a multi-scenario dataset of drone videos with dense head annotations, disjoint training and testing locations, and video-level attributes. The paper evaluates counting, localization, and tracking with task-specific metrics.

3.1. Data Collection and Annotation

  • DroneCrowd videos were captured at 25 FPS and 1920×1080 resolution across settings including campuses, streets, parks, parking lots, playgrounds, and plazas.The dataset includes DJI Phantom 4, Phantom 4 Pro, and Mavic drone-mounted cameras; frames contain 25 to 455 people, averaging 144.8.Source: §3.1 · p. 3
  • More than 20,000 people trajectories and 4.8 million head points were annotated across 112 clips, with over 20 experts annotating and double-checking the data.The annotation process took more than two months and used the vatic software.Source: §3.1 · p. 3
  • The split assigns 82 sequences to training and 30 to testing, with different locations used to reduce scene overfitting; attributes cover illumination, object scale, and frame density.The density categories are Crowded (more than 150 objects per frame) and Sparse (fewer than 150).Source: §3.1 · pp. 3–4

3.2. Evaluation Metrics and Protocols

  • Counting is evaluated with MAE and MSE, localization with L-mAP and specified L-AP distance thresholds, and tracking with T-mAP at tracklet-matching thresholds.Localization matching uses a 25-pixel distance threshold; tracking uses a 25-pixel matching distance and thresholds of 0.10, 0.15, and 0.20.Source: §3.2 · p. 4

4. Our Method

STNNet processes frame pairs to estimate density maps, object locations, and motion offsets, then associates detections into trajectories.

4. Our Method

  • Paired-frame inputs yield density maps, object locations, and motion offsets; association links detections into long video trajectories.The network processes a pair of frames at a time.Source: §4 · p. 4

4.1. Network Architecture

  • A shared two-branch VGG-16 extracts multi-scale features; U-Net-style fusion and frame correlations feed density heads, while attention-based branches and PointConv–MLP predict locations and offsets.The localization subnet uses classification and regression branches, followed by non-maximal suppression; the association subnet uses three stacked PointConv and MLP operations.Source: §4.1 · pp. 4–5, Figure 3 · p. 5, Figure 4 · p. 5

4.2. Multi-Task Loss Function

  • The multi-task objective combines density, localization, and neighboring-context losses to train the network’s maps, proposals, and motion offsets.The density loss uses pixel-wise Euclidean error, while localization combines classification and regression losses.Source: §4.2 · p. 5
  • Neighboring-context loss uses temporal prediction and relation constraints to align projected proposals with ground truth and preserve neighboring objects’ relative positions, computed with a cycle strategy.The relation constraint compares projected relation vectors with those of corresponding associated ground-truth points.Source: §4.2 · p. 6

4.3. Optimization

  • Training uses flipped and cropped 2 × 2 frame patches, β = 8 nearest points, and two stages: first density maps and proposals, then association fine-tuning with density heads fixed.Both stages use Adam with a learning rate of 10−6.Source: §4.3 · p. 6

5. Experiment

The experiment evaluates density-map estimation, localization, and tracking on DroneCrowd. STNNet leads on overall density-map error and localization, while tracking remains challenging.

Experiment

  • Evaluation reports density-map results and speeds for STNNet and 12 existing methods, component ablations, and qualitative comparisons with DM-Count, CSRNet, and CAN.Figure 6 presents qualitative results on DroneCrowd.Source: §5 · pp. 6–7, Figure 6 · p. 7

Density map estimation

  • 2.6 MAE and 8.3 MSE improvement over second-best DM-Count on the overall density-map test set.STNNet also leads in four attribute subsets; CAN leads in Cloudy and Crowded, and DM-Count in Sunny.Source: §5 · p. 7, Table 3 · p. 8
  • 2.8 MAE and 3.5 MSE lower than STNNet (w/o loc) show the localization subnet also benefits density-map estimation.The ablation removes the localization subnet from STNNet.Source: §5 · p. 7, Table 3 · p. 8

Crowd localization

  • 40.45% L-mAP versus second-best DM-Count at 22.28% L-mAP makes STNNet the most accurate localization method reported.Removing both localization and association subnets reduces L-mAP by more than 8%.Source: §5 · p. 7, Table 4 · p. 8

Effectiveness of neighboring context loss

  • 0.23% and 0.24% gains in L-mAP and T-mAP, respectively, show STNNet (w/o cyc) outperforming STNNet (w/o rel).The comparison tests the relation constraint in the neighboring context loss.Source: Table 4 · p. 8, Table 5 · p. 8, §5 · p. 8

Crowd tracking

  • 32.50% T-mAP, 15.49% above second-best DM-Count, is STNNet’s reported tracking result.The authors still describe tracking performance as far from satisfactory; social-LSTM and min-cost flow perform comparably.Source: Table 5 · p. 8, §5 · p. 8

6. Conclusion

The conclusion presents STNNet as a joint approach to density-map estimation, localization, and tracking in drone-captured crowds. It also highlights DroneCrowd as a large dataset for evaluating those tasks.

Conclusion

  • STNNet jointly addresses density-map estimation, localization, and tracking in drone-captured crowded scenes.The neighboring context loss captures relations among neighboring targets in consecutive frames and is reported as effective for localization and tracking.Source: §6 · p. 8
  • DroneCrowd is described as the largest dataset to date in annotated head trajectories for density-map estimation, localization, and tracking on drones.The authors collected and annotated the dataset to better evaluate performance on drones.Source: §6 · p. 8
  • The authors hope DroneCrowd and STNNet will facilitate research and development in drone-based crowd localization, tracking, and counting.Source: §6 · p. 8

Related papers

Loading 2105.02440v1…