Source-linked AI summary
Where are the Blobs: Counting by Localization with Point Supervision
Issam H. Laradji, Negar Rostamzadeh, Pedro O. Pinheiro, David Vazquez, Mark Schmidt
TL;DR
Object counting must handle substantial variation and occlusion, while regression methods often outperform detection because detection requires estimating object geometry. LC-FCN instead learns point-supervised localization blobs, and achieves state-of-the-art counting results across several challenging datasets, including cases using weaker supervision.
Problem
Object counting must handle variation in object shape, size, pose, appearance, viewpoint, resolution, occlusion, and scene conditions, while detection requires difficult instance-geometry prediction.
Method
LC-FCN extends a fully convolutional network with a loss that uses point annotations to produce one blob per object, plus methods for splitting large blobs.
Results
LC-FCN achieves state-of-the-art counting results on challenging datasets including PASCAL VOC, Trancos, and Penguins, outperforming methods that explicitly learn to count.
Takeaways & Limitations
Point-level supervision can support localization-based counting without estimating exact object size and shape, including for heavily occluded objects.
Takeaways & Limitations
Selecting suitable heuristics for identifying negative regions remains an open area for future work, and watershed splitting improves validation performance and convergence speed.
Abstract
from arXiv · showhide
Object counting is an important task in computer vision due to its growing demand in applications such as surveillance, traffic monitoring, and counting everyday objects. State-of-the-art methods use regression-based optimization where they explicitly learn to count the objects of interest. These often perform better than detection-based methods that need to learn the more difficult task of predicting the location, size, and shape of each object. However, we propose a detection-based method that does not need to estimate the size and shape of the objects and that outperforms regression-based methods. Our contributions are three-fold: (1) we propose a novel loss function that encourages the network to output a single blob per object instance using point-level annotations only; (2) we design two methods for splitting large predicted blobs between object instances; and (3) we show that our method achieves new state-of-the-art results on several challenging datasets including the Pascal VOC and the Penguins dataset. Our method even outperforms those that use stronger supervision such as depth features, multi-point annotations, and bounding-box labels.
1 Introduction
Object counting matters across surveillance, traffic, ecological, cell-counting, and everyday-scene applications, but variability and occlusion make localization difficult. The paper proposes point-supervised localization-based counting that predicts one blob per object and uses blob counts as the output.
- Object counting supports applications including surveillance, traffic monitoring, ecological surveys, cell counting, and everyday-scene benchmarks.
- Object variability, occlusion, viewpoint, resolution, background, weather, and illumination require robust recognition for efficient counting.
- The model predicts object blobs from point supervision, and the count equals the number of predicted blobs.
- Regression methods often outperform detection methods because detection must estimate each instance’s location, shape, and size.
- The proposed loss encourages each predicted instance region to contain one point-level annotation, avoiding the need to estimate exact object size and shape.
- The method introduces a point-supervised single-blob objective and two procedures for splitting large predicted blobs between instances.
2 Related Work
Prior counting methods cluster objects, regress counts or densities, or detect instances, with regression often leading benchmark performance. The paper positions point-supervised localization as an alternative to both image-level counting and more demanding fully supervised detection or segmentation.
- Object-counting approaches are commonly grouped into clustering, regression, and detection methods.
- Clustering methods group objects using appearance and motion features, including tracked feature points.
- Regression methods learn counts or density maps and have often been faster and more accurate than detection methods.
- Density-based regression depends on selecting a Gaussian-kernel size that matches object sizes, creating a key modeling challenge.
- Detection methods count detected instances but commonly require bounding-box or per-pixel ground truth, and prior detectors can underperform regression methods.
- Related point-supervised instance-detection work showed promising detection results but performed worse for counting than regression-based methods.
3 Localization-based Counting FCN
LC-FCN extends a fully convolutional semantic-segmentation network with a localization-based counting loss trained from point annotations. Its loss combines semantic pixel supervision with terms intended to produce separate object blobs rather than grouping instances.
- LC-FCN extends a fully convolutional network with a localization-based counting loss for point-supervised object counting and localization.
- The image-level and point-level loss terms enforce semantic segmentation predictions for every image pixel.
- Those semantic losses alone can group multiple object instances into one predicted blob, so additional loss terms are needed for counting.
- The point-annotation matrix T marks each object location with its class label and assigns zero elsewhere, while S contains per-pixel class probabilities.
} Point-level loss
The supplied passages identify additional loss terms in the proposed objective but do not explain their individual functions.
- The proposed objective includes a segmentation loss term LS(S, T).
- The proposed objective includes a false-positive loss term LF(S, T).
} False positive loss
The method uses point annotations to guide pixel labeling, split blobs containing multiple instances, and suppress blobs without annotated objects. Connected components then convert class masks into object counts.
- False positive loss: The image-level loss promotes pixels from present classes and suppresses predictions for classes absent from the image.Present and absent class sets are derived from point-level annotations.
- False positive loss: The point-level loss supervises only annotated object pixels and ignores unannotated pixels.The supervised locations are the object-instance points, with Ti denoting each point's true label.
- False positive loss: Blobs containing n point annotations are trained to split into n blobs, using predicted background boundaries between object pairs.Connected components identify foreground blobs, and only blobs with at least two annotations are selected for splitting.
- False positive loss: The line split selects a perpendicular segment with the highest mean background probability as the likely separation edge.Candidate segments lie inside the predicted blob and intersect its boundaries.
- False positive loss: The false positive loss penalizes predicted blobs containing no ground-truth points, reducing false positive predictions.All pixels in such blobs are treated as false positives during optimization.
- False positive loss: The model counts objects by assigning each pixel to a class, extracting connected components, and counting the resulting blobs.Its FCN backbone and upsampling path produce per-pixel class probabilities before blob extraction.
4 Experiments
The experiments evaluate counting and localization across crowd, surveillance, traffic-monitoring, population-monitoring, and multiclass datasets using MAE, mRMSE, GAME, and F-Score. LC-FCN achieves strong results across these settings, including state-of-the-art performance on Penguins, Trancos, and PASCAL VOC.
- Evaluation metrics: MAE measures count deviation for single-class datasets, while mRMSE evaluates multiclass counting and GAME assesses localization.Because LC-FCN predicts blobs rather than density maps, the paper also uses F-Score to assess localization.
- Dataset results: Penguins results significantly outperform prior methods in all four settings despite those methods using depth features.The result suggests LC-FCN can distinguish individual penguins under heavy occlusion and crowding.
- Dataset results: LC-FCN achieves new state-of-the-art results for counting and localization on Trancos without requiring a perspective map or multiscale approach.The method uses predicted blob centers for regional counts and accurately localizes and counts extremely overlapping vehicles.
- Dataset results: On three crowd datasets, LC-FCN achieves competitive MAE performance and predicts blobs centered on individuals’ faces in ShanghaiTech B.The face-centered blobs follow the point-level ground-truth annotations, which are marked on each person’s face.
- Dataset results: On the Parking Lot and MIT Traffic datasets, LC-FCN outperforms image-level counting methods and Glance, respectively, while providing the first counting evaluation reported for each setting.The MIT Traffic results associate localization-based learning with ignoring background regions that do not contribute to the count.
- Dataset results: LC-FCN outperforms methods that explicitly learn to count on PASCAL VOC with respect to mRMSE while using weaker supervision than methods requiring full per-pixel labels.LC-FCN is evaluated on the full test set, whereas competing methods use means over ten random test-set samples.
Crowd Counting Datasets.
On three crowd datasets, LC-FCN produces blobs corresponding to individual people, with predictions centered on the annotated faces.
- LC-FCN achieves competitive MAE performance on Mall, UCSD, and ShanghaiTech B under the survey-paper evaluation setup.The reported experiment uses ResFCN with the Watershed split method.
- On ShanghaiTech B, the model predicts one blob on each individual’s face, matching the location of the ground-truth point annotations.The face-centered predictions arise because annotations are marked on each person’s face.
4.3 Ablation Studies
The ablation study shows that split-level and false-positive losses are complementary: splitting separates merged instances, while false-positive suppression removes unsupported blobs.
- Image-level and point-level losses alone group many object instances into a single blob, producing poor counting and localization performance.The qualitative results correspond to Fig. 5(b), while quantitative effects are reported in Table 5.
- The split-level loss encourages blobs containing multiple point annotations to separate into distinct object instances.With this term added, predictions contain several instance-like blobs rather than one large grouped blob.
- Adding only the split-level loss can create many false positives because blobs without point annotations are not penalized.The full loss therefore adds a false-positive term to suppress unsupported predictions.
- The full proposed loss substantially improves qualitative and quantitative counting and localization results by combining instance splitting with false-positive suppression.The improvement is shown in Fig. 5(d) and Table 5.
- Combining the false-positive loss with split-level supervision avoids the tendency toward overly large blobs and yields a net performance improvement.Using only the false-positive loss instead tends to produce one huge blob.
- The watershed split achieves better validation MAE than the line split on Trancos and Penguins and improves faster across epochs.The comparison is shown in Fig. 6.
5 Conclusion
LC-FCN counts objects from point-level annotations by learning one segmentation blob per instance and outperforms state-of-the-art methods on several heavily occluded datasets.
- LC-FCN is a fully convolutional network for object counting using point-level annotations only.Its loss encourages one blob for each object instance.
- The method outperforms current state-of-the-art models on PASCAL VOC 2007, Trancos, and Penguins, including datasets with heavy occlusion.The conclusion identifies these datasets as containing heavily occluded objects.
- Future work will explore alternative FCN architectures and splitting methods for overlapping objects with complicated shapes and appearances.