Source-linked AI summary
Drone-based Object Counting by Spatially Regularized Regional Proposal Network
Meng-Ru Hsieh, Yen-Liang Lin, Winston H. Hsu
TL;DR
The paper addresses counting and localization in unconstrained drone videos, where existing regression methods lack precise object positions and prior methods rely on fixed cameras. It proposes spatially regularized Layout Proposal Networks and spatial kernels, and introduces CARPK, reporting improved proposal recall on PUCPR and a large drone-view dataset for counting.
Problem
Existing counting methods cannot precisely localize objects, while prior parking-lot methods assume fixed cameras or known object locations rather than unconstrained drone videos.
Method
Layout Proposal Networks use spatially regularized loss and spatial kernels to count and localize objects by exploiting recurring object-layout patterns.
Results
59.9% to 62.5%: the method improves average recall over state-of-the-art region proposal methods on the public PUCPR dataset.
Takeaways & Limitations
CARPK provides a large-scale drone-view counting benchmark with nearly 90,000 cars and bounding-box annotations in diverse parking-lot scenes.
Abstract
from arXiv · showhide
Existing counting methods often adopt regression-based approaches and cannot precisely localize the target objects, which hinders the further analysis (e.g., high-level understanding and fine-grained classification). In addition, most of prior work mainly focus on counting objects in static environments with fixed cameras. Motivated by the advent of unmanned flying vehicles (i.e., drones), we are interested in detecting and counting objects in such dynamic environments. We propose Layout Proposal Networks (LPNs) and spatial kernels to simultaneously count and localize target objects (e.g., cars) in videos recorded by the drone. Different from the conventional region proposal methods, we leverage the spatial layout information (e.g., cars often park regularly) and introduce these spatially regularized constraints into our network to improve the localization accuracy. To evaluate our counting method, we present a new large-scale car parking lot dataset (CARPK) that contains nearly 90,000 cars captured from different parking lots. To the best of our knowledge, it is the first and the largest drone view dataset that supports object counting, and provides the bounding box annotations.
1. Introduction
The paper targets object counting in unconstrained drone videos, where regression methods lack precise localization and prior parking-lot approaches assume fixed cameras or known object locations. It introduces spatially regularized Layout Proposal Networks and CARPK to improve proposal quality and support large-scale drone-view counting.
- Fixed-camera parking-lot methods assume monitored object locations are known and cast car counting as classification, limiting direct use in unconstrained drone videos.
- Regression-based counting methods produce counts but cannot generate precise object positions for downstream investigation and applications such as recognition.
- Layout Proposal Networks count and localize objects in drone videos by learning spatially regularized constraints from recurring object layouts.The method models general adjacent relationships between proposals rather than relationships specific to one scene.
- 89,777 cars: CARPK provides bounding-box annotations for all cars in diverse, unconstrained drone-view parking-lot scenes.The dataset was introduced to evaluate counting effectiveness and reliability at large scale.
- 59.9% to 62.5%: spatially regularized proposal learning improves average recall over state-of-the-art region proposal methods on the public PUCPR dataset.
- CARPK is presented as the first and largest drone-based parking-lot dataset supporting counting with full-image annotations.Its scale is intended to provide sufficient training samples for deep learning models.
2. Related Work
Object counting work spans regression and detection-based approaches, while region proposal methods reduce computational cost by evaluating a small subset of candidate positions. Existing proposal methods improve recall with fewer proposals but do not incorporate spatial layout or relations among recurring objects.
- 2.1. Object Counting: Regression-based counters map high-dimensional image inputs to non-negative counts but cannot generate precise object positions.This limits applications requiring localization, such as high-level understanding and fine-grained classification.
- 2.1. Object Counting: Deep learning has been applied to crowd, vehicle-density, and cross-scene counting, but regression-based methods still lack precise localization.Examples include global and local contextual models, aerial-image patch representations, and FCN-LSTM models for low-resolution city-camera videos.
- 2.2. Object Proposals: Region proposal methods reduce inference cost by examining a small subset of possible object positions instead of classifying every position and scale.Deep networks-based proposal methods surpassed earlier approaches based on low-level cues.
- 2.2. Object Proposals: DeepMask achieves the same performance as Selective Search with 100 rather than 1,000 proposals, while RPNs surpass 2,000-proposal results using 300 proposals.These comparisons illustrate the efficiency gains of learned proposal mechanisms.
- 2.2. Object Proposals: LPNs incorporate spatial layout and relations among recurring objects to seek higher recall with fewer region proposals.The paper identifies this as an omission in prior proposal methods.
3. Dataset
Existing car datasets lack the combination of scale, resolution, complete annotations, and unconstrained drone viewpoints needed for counting and localization. The paper addresses these limitations with CARPK and completed PUCPR annotations, while using spatial layout scores to favor likely car positions.
- Dataset motivation: Existing aerial car datasets vary in limitations including few cars, low resolution, point-only annotations, or insufficient support for counting and localization.OIRDS has 180 cars, VEDAI 2,950, and COWC 32,716 with 24 to 48 pixels per car and center-point annotations.
- Spatial layout scores: Spatial layout scores assign higher confidence and probability to predicted positions surrounded by more nearby cars.The figure presents the score as a way to favor likely car locations based on neighboring instances.
- PUCPR and PUCPR+: PUCPR originally annotated only part of the cars and could not support both counting and localization, so the authors completed annotations into PUCPR+ with nearly 17,000 cars.PUCPR also uses fixed sensors in the same place, producing a repeated scene and dataset-bias concern.
- CARPK: CARPK contains approximately 90,000 cars from four diverse parking lots, with drone-view images and one bounding box per car.The dataset is described as the first and largest drone-view parking-lot dataset supporting counting with full-image manual annotations.
- CARPK: The dataset includes edge cars when their marked regions are recognizable and identifiable as cars.Bounding boxes record top-left and bottom-right points for each labeled car.
4. Method
The method uses Layout Proposal Networks to generate object-agnostic proposals while incorporating recurring spatial layouts into proposal scoring and training. Spatially regularized weights reweight nearby proposals, and localization is learned through foreground-box offsets with smaller default boxes for drone-view objects.
- Layout Proposal Network: LPNs form a unified fully convolutional framework that accepts arbitrary-size images and outputs object-agnostic proposals likely to contain instances.The module also provides directional guidance to the detection component using recurring-instance spatial information.
- Layout Proposal Network: The network slides over shared convolutional features, using sibling localization and classification layers while adding spatially regularized weights to the loss.A 3 × 3 window reduces feature dimensions before separate 1 × 1 convolutional layers predict localization and foreground/background classification.
- Loss Design: The localization loss is active for foreground predicted boxes and measures offsets in center position, width, and height relative to default boxes.Foreground and background boxes are assigned using IoU thresholds, while localization offsets are computed against the matching ground-truth box.
- Implementation: For small objects, the method uses conv4-3 features and default box sizes of 16 × 16, 40 × 40, and 100 × 100.These boxes are approximately four times smaller than the default 128 × 128, 256 × 256, and 512 × 512 sizes.
- Spatial Pattern Score: Gaussian spatial kernels reweight positive proposals according to nearby ground-truth centers and their directional layout patterns.The kernels aggregate confidence from neighboring instances around each predicted box position.
5. Experiment
Experiments evaluate LPNs for region proposal and car counting on PUCPR+ and CARPK using cross-validation and cross-scene splits. The reported results show improved proposal recall and localization, with counting errors of 8.04 MAE and 12.06 RMSE on PUCPR+ and 13.72 MAE and 21.77 RMSE on CARPK at 1000 proposals.
- Evaluation Setup: LPNs are evaluated on PUCPR+ and CARPK for both region proposal quality and object counting.PUCPR+ uses five-fold cross-validation, while CARPK uses three parking-lot scenes for training and the remaining scene for testing.
- Region Proposal Evaluation: 16.30% relative improvement over RPN-small in dense PUCPR+ scenes exceeds the 8.27% improvement in less-dense scenes.At 300 proposals, LPNs reach 64.4% recall at IoU 0.7 versus 54.7% for RPN-small.
- Region Proposal Evaluation: 64.4% recall at IoU 0.7 versus 54.7% for RPN-small shows more precise localization with 300 proposals.The experiment compares LPNs and RPN variants under different proposal counts and component settings.
- Car Counting Evaluation: The proposed method outperforms the best RMSE on PUCPR+ while achieving comparable MAE to the state-of-the-art regression counting method.The authors attribute worse results from models fine-tuned on PASCAL VOC and MS COCO to differing object perspective views.
- Car Counting Evaluation: 8.04 MAE and 12.06 RMSE on PUCPR+, and 13.72 MAE and 21.77 RMSE on CARPK, are reported for the method using 1000 proposals.The method performs better with a larger number of proposals in the PUCPR+ counting experiment.
6. Conclusions
The paper contributes CARPK, the largest drone-view dataset for large-scale car counting, and a spatial-layout-based method for generating feasible region proposals.
- CARPK is the to-date largest drone-view dataset for challenging car-counting tasks across various parking-lot scenes.The dataset is designed for large-scale counting in diverse parking-lot environments.
- The proposed proposal-generation method leverages spatial layout information to produce regularized structures for object counting.The learned deep model uses prior knowledge of object layout patterns to improve object counting.
- Future work will incorporate global context, road scenes, and other objects to distinguish false car-like instances from real cars.