Source-linked AI summary
Rethinking Spatial Invariance of Convolutional Networks for Object Counting
Zhi-Qi Cheng, Qi Dai, Hong Li, JingKuan Song, Xiao Wu, Alexander G. Hauptmann
TL;DR
Object-counting research has often treated increased spatial invariance as a key to better performance, while the authors report that strict pixel-level invariance can overfit density-map noise. They replace standard convolutions with Gaussian convolution, using low-rank approximation and translation invariance; experiments report improved performance across counting tasks and benchmarks.
Problem
The paper asks whether increasing spatial invariance helps object counting and how density-map noise affects performance.
Source: §1 · p. 1Method
The method replaces standard convolution with Gaussian convolution during feature learning, using low-rank approximation and translation invariance.
Source: §1 · p. 2, §3.3.2 · p. 5Results
Takeaways & Limitations
The findings point to investigating how to relax spatial invariance appropriately for object counting rather than increasing it indiscriminately.
Source: §1 · p. 2, §5 · p. 8The noise analysis assumes center-label errors are independent and identically distributed and Gaussian.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Previous work generally believes that improving the spatial invariance of convolutional networks is the key to object counting. However, after verifying several mainstream counting networks, we surprisingly found too strict pixel-level spatial invariance would cause overfit noise in the density map generation. In this paper, we try to use locally connected Gaussian kernels to replace the original convolution filter to estimate the spatial position in the density map. The purpose of this is to allow the feature extraction process to potentially stimulate the density map generation process to overcome the annotation noise. Inspired by previous work, we propose a low-rank approximation accompanied with translation invariance to favorably implement the approximation of massive Gaussian convolution. Our work points a new direction for follow-up research, which should investigate how to properly relax the overly strict pixel-level spatial invariance for object counting. We evaluate our methods on 4 mainstream object counting networks (i.e., MCNN, CSRNet, SANet, and ResNet-50). Extensive experiments were conducted on 7 popular benchmarks for 3 applications (i.e., crowd, vehicle, and plant counting). Experimental results show that our methods significantly outperform other state-of-the-art methods and achieve promising learning of the spatial position of objects.
1. Introduction
The introduction questions the assumption that greater spatial invariance improves counting, arguing that strict pixel-level invariance can overfit density-map noise. It proposes replacing standard convolutions with Gaussian kernels, using low-rank approximation and translation invariance to make the approach practical.
1. Introduction
- Density maps convert discrete counting into continuous regression by smoothing object centers with Gaussian kernels, introducing annotation-position error and kernel overlap.The introduction illustrates these two noise sources with crowd counting.Source: §1 · p. 1, Figure 1 · p. 1
- Across four mainstream networks and three counting tasks, strict pixel-level spatial invariance caused large prediction variances and overfitting to density-map noise.The authors also report that existing models could not generalize and could not learn actual object positions and distributions in density maps.Source: §1 · p. 2
- Gaussian convolution replaces standard convolution during feature learning to model density-map generation and relax pixel-level spatial invariance.The authors report that this relaxation can avoid overfitting to density-map noise and learn object positions and distributions.Source: §1 · p. 2
- Low-rank approximation uses finite Gaussian kernels to approximate massive Gaussian convolution, while translation invariance accelerates inference.The method uses representative kernels selected from sampled ground-truth density-map kernels, with PCA and an attention mechanism.Source: §1 · p. 2
- Experiments across seven datasets and crowd, vehicle, and plant counting demonstrate the method’s effectiveness.The paper evaluates the approach with MCNN, CSRNet, SANet, and ResNet-50.Source: §1 · p. 2
2. Related works
Related work describes CNN approaches that increase spatial invariance and methods aimed at density-map noise. The paper positions its approach as changing convolution during feature learning rather than addressing noise only through losses or regularization.
2.1. Increase the spatial invariance with CNNs
- Mainstream counting research uses density regression and pursues spatial invariance through multi-column, dilated, deformable, residual, graph, attention, pyramid-pooling, and hybrid CNNs.The authors note that continued architectural optimization has encountered performance bottlenecks.Source: §2.1 · p. 2
- CNN counting methods vary in how they relax pixel-level invariance, fuse local features, or use attention and perspective information.The paper presents low-rank Gaussian kernels with attention as one solution that relaxes invariance and fuses local features; it leaves further exploration to follow-up work.Source: §2.1 · p. 2, Figure 2 · p. 3
2.2. Dealing with noise in the density map
- Density-map noise methods include changes to losses, metrics, matching, noise regions, regularization, labels, training, adversarial learning, and reinforcement learning.The authors say these approaches do not reveal the correlation between spatial invariance and density-map noise, and most address noise through loss or regularization.Source: §2.2 · p. 3
3. Methods
The methods trace density-map noise to Gaussian smoothing and relax pixel-level spatial constraints by replacing standard filters with Gaussian convolution, accelerated through low-rank approximation and translation invariance.
3.1. Traditional density map generation
- Density maps smooth labeled object centers with Gaussian kernels, but annotation-position errors and overlapping kernels introduce noise into the regression target.The method models center-label error as Gaussian and derives an updated density distribution with more complex mean and variance.Source: §3.1 · p. 3, §3.2 · p. 3
3.2. Noise in object counting task
- Strict pixel-level spatial invariance can overfit label errors, leading models to miss occlusion in dense regions and overestimate density in sparse regions.The analysis attributes this to models ignoring object position and shape.Source: §3.2 · p. 4
3.3. Low-rank Gaussian convolutional layer
- GauNet replaces standard convolution filters with Gaussian kernels so feature extraction simulates density-map generation and relaxes pixel-level spatial constraints.The Gaussian kernels encode object position through their means and shape through their variances.Source: §3.3 · p. 4
3.3.1 Accelerate with Low-rank approximation
- PCA selects representative kernels from random Gaussian samples, and learned attention weights fuse their local features to approximate massive Gaussian convolution.The low-rank approximation uses K kernels with K ≪ N.Source: §3.3.1 · pp. 4–5
3.3.2 Accelerate with translation invariance
- Translation invariance separates each Gaussian operation into convolution with a zero-mean kernel followed by translation, using bilinear interpolation for non-discrete shifts.This lets the convolution step ignore kernel means; interpolation weights support subpixel displacement.Source: §3.3.2 · p. 5
- Low-rank approximation reduces complexity to O(KCiCoHWKkwkh), and translation invariance further reduces it to O(4KCiCoHW).The factor 4 is associated with bilinear interpolation.Source: §3.3.2 · p. 5
4. Experiments
Experiments across crowd, vehicle, and plant counting test four baseline networks, revealing density-map noise and evaluating the proposed Gaussian layers. The variants improve benchmark performance, robustness, and learned spatial representations, with effects varying by setting.
4.1. Experiment settings
- The method was evaluated on five crowd datasets and one each for vehicle and plant counting, using MCNN, CSRNet, SANet, and ResNet-50 with MAE and MSE measurements.SANet was evaluated on image patches, while MCNN and CSRNet were tested on whole images.Source: §4.1 · p. 5
4.2. Reveal the label noise of object counting
- Four counting methods showed large prediction variance on SHTech-PartA and UCF-QNRF, which did not decrease as spatial-invariance performance increased and was linked to overfitting density-map noise.Figure 5 compares full-image, high-density, and low-density regions; Figure 4 visualizes examples under annotation shifts of 0, 8, 16, and 32 pixels.Source: §4.2 · p. 5, Figure 5 · p. 6, Figure 4 · p. 6
- High-density regions had more severe variance and density underestimation than full images, while low-density regions had less variance than high-density regions but more than full images.The authors speculate that fewer Gaussian kernels mean lower annotation noise in low-density regions and that high- and low-density areas may compensate for each other.Source: §4.2 · p. 6
4.3. Ablation study
- Low-rank approximation and translation invariance significantly improved the computational efficiency of Gaussian convolution.Table 2 reports MCNN cost; its caption identifies the Vanilla setting as using 256 Gaussian kernels per layer.Source: §4.3 · p. 6, Table 2 · p. 6
- Replacing MCNN’s first three convolutional layers produced larger improvements than replacing later layers, and the study generally replaced all eligible convolutional layers.The authors report roughly similar results across MCNN’s three column structures and fewer parameters than the original convolutional layer.Source: §4.3 · p. 6, Figure 6 · p. 6
4.4. Compare with state-of-the-art methods
- Across benchmarks, CSRNet, SANet, and ResNet-50 beat state-of-the-art on free-view crowd counting; surveillance gains were smaller, while vehicle and plant results also surpassed prior methods.The light MCNN approached some advanced methods; gains were minor for vehicle counting, while plant-counting MSE improvement indicated greater robustness.Source: §4.4 · p. 7, Table 3 · p. 7, Table 4 · p. 7
4.5. Robustness to annotation noise
- With increasing annotation-point shifts, all methods’ performance declined, but the proposed Gaussian convolutions remained more robust than other methods.The robustness experiment trained on datasets with annotation points randomly moved by 4, 8, 16, or 32 pixels.Source: §4.5 · pp. 7–8, Table 6 · p. 7
4.6. Visualization of convolution filters
- Visualized filters learned object-distribution perspective patterns, including planting intervals and pedestrian and vehicle distributions that original SANet filters did not capture in the same way.The visualizations covered free-perspective and surveillance views; original SANet showed noise or guessed density regions.Source: §4.6 · p. 8, Figure 7 · p. 8
5. Conclusion
The conclusion argues that strict pixel-level spatial invariance can promote overfitting to density-map noise. It presents locally connected Gaussian kernels with efficiency-oriented approximations and calls for further study of the relationship between Gaussian kernels and spatial invariance.
5. Conclusion
- Instinctively improving CNN spatial invariance can make models overfit density-map noise.The authors frame this as a relationship between spatial invariance and density-map noise.Source: §5 · p. 8
- The proposed locally connected multivariate Gaussian kernels replace convolution filters and approximately simulate density-map generation during feature extraction.Translation invariance and low-rank approximation are used to improve efficiency.Source: §5 · p. 8
- The authors report outperforming other state-of-the-art methods and identify the relationship between Gaussian kernels and spatial invariance as future work.They suggest future research avoid blindly improving spatial invariance for object counting.Source: §5 · p. 8
A. Implementation details
The passage introduces network architecture details for the baselines.
A. Implementation details
- The appendix first presents baseline network architectures to clarify implementation details.The passage does not specify any architecture changes.Source: A. Implementation details · p. 9
A.1. Network structure of baselines
The implementation adapts MCNN, CSRNet, SANet, and ResNet-50 by replacing selected convolution filters with locally connected low-rank Gaussian kernels, while retaining or modifying other components as specified.
A.1. Network structure of baselines
- MCNN replaces filters in its first four layers with locally connected low-rank Gaussian kernels, using three branches and a spatial pyramid pooling block to fuse their features.The first two layers use 16 kernels each; the final two layers use 2, 4, and 6 kernels across the smaller, middle, and maximum branches, respectively.Source: §A.1 · p. 9
- CSRNet retains VGG-16’s first ten convolutional layers and modifies the subsequent dilated branches, selecting the B branch and adding spatial pyramid pooling.The chosen B branch uses four kernels, and the pooling module uses eight low-rank Gaussian kernels.Source: §A.1 · p. 9
- SANet modifies its first four convolutional blocks with Gaussian kernels and retains its final deconvolutional layer.The first two layers use 16 kernels; the latter two use 8, 6, 4, and 2 kernels for 1 × 1, 3 × 3, 5 × 5, and 7 × 7 filters, respectively.Source: §A.1 · p. 9
- ResNet-50 replaces 3 × 3 filters in residual blocks with four locally connected low-rank Gaussian kernels, while retaining the initial 7 × 7 and bottleneck 1 × 1 filters.The encoder stride and decoder are adjusted to preserve density-map scale.Source: §A.1 · p. 9
A.2. Data preprocessing and training details
The seven counting datasets receive input-size and label preprocessing; training otherwise follows baseline and established framework settings, with some implementation differences.
A.2. Data preprocessing and training details
- Preprocessing adjusts input size and transforms labels across all seven object-counting datasets.SHTech-PartA, SHTech-PartB, and UCF-QNRF follow the C3 Framework; the other datasets follow released code repositories.Source: §A.2 · p. 9
- Loss functions and optimization generally follow each original baseline, while C3 Framework Batchsize training accelerates MCNN and SANet training.The modified convolutional layers are the stated exception to retaining the original baseline setup.Source: §A.2 · p. 9
- CSRNet takes its first ten convolutional layers from pretrained VGG-16 and trains with fixed-rate SGD at 1e−6; revised ResNet-50 baselines are pretrained on ImageNet using original ResNet-50 hyperparameters.ImageNet pretraining uses learning rate 0.1, momentum 0.9, weight decay 1e−4, and batch size 256; the learning rate is reduced by a factor of 10 at epochs 30, 60, 80, and 90. Fine-tuning follows C3 Framework settings: learning rate 1e−4, weight decay 0.995, with the learning rate reduced layer by layer.Source: §A.2 · p. 10
B. More experimental results
This section presents additional experiments intended to expose problems in object counting and demonstrate the method’s effectiveness.
B. More experimental results
- Additional experiments are presented to reveal problems in object counting and test the method’s effectiveness.Source: B. More experimental results · p. 10
B.1. More ablation studies
The ablations select different Gaussian-kernel settings across network components, with branch fusion retained for CSRNet and K=16 chosen for spatial pyramid pooling.
B.1. More ablation studies
- CSRNet’s fused four-branch backend uses K values from 2 to 32 in each column, while K=4 gave the best result for an individual branch.The individual branch result was best among the tested settings, but fusing all four branches improved performance further.Source: §B.1 · p. 10
- K=4 provides the best efficiency–performance balance in the residual block.Source: §B.1 · p. 10
- Spatial pyramid pooling in MCNN and SANet uses K=16 because performance varied as K increased, which the authors attribute to possible overfitting.Settings from the MCNN ablation were applied directly to the other baselines.Source: §B.1 · p. 10
B.2. Reveal the problem of object counting
Repeated training experiments find substantial prediction variance in original counting baselines, including across density regions and datasets. The modified baselines significantly reduce that variance.
B.2. Reveal the problem of object counting
- Variance is prediction error on the same image across trained models, and the experiments repeat training 20 times per baseline with randomized training-data loading.The authors interpret this variance as evidence that models do not converge stably.Source: §B.2 · p. 10
- Across SHTech-PartA, UCF-QNRF, and JHU-CROWD++, baseline prediction variance is about a quarter of total error and does not fall as spatial invariance increases.High- and low-density regions are analyzed separately in these datasets.Source: §B.2 · p. 10
- Even in datasets described as relatively low-density or high-density scenarios, prediction error remains very large.These full-image analyses cover SHTech-PartB, TRANCOS, MTC, and UCF CC 50.Source: §B.2 · p. 10
- The modified baselines significantly reduce prediction variance compared with the original networks.Source: §B.2 · p. 10