Source-linked AI summary
Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks
Xiang Li, Xiaolin Hu, Jian Yang
TL;DR
CNN semantic sub-features can be spatially disrupted by similar patterns and noisy backgrounds, hindering localization and identification. The paper introduces SGE, which applies similarity-guided attention within feature groups at negligible cost. Integrated into CNN backbones, SGE improves image classification and detection across reported benchmarks.
Problem
Noisy backgrounds and limited region-level supervision can disorder the spatial distribution of semantic feature groups, weakening representation learning and localization.
Method
SGE generates group-specific spatial attention from similarity between global statistical and local positional descriptors, scaling features to suppress noise and highlight semantic regions.
Results
SGE steadily improves image classification and detection, performing better or comparably to state-of-the-art attention modules on ImageNet and yielding more than 1% AP gains across several COCO detectors.
Takeaways & Limitations
SGE provides a lightweight way for feature groups to enhance semantic representations and suppress noise in practical CNN recognition systems.
Abstract
from arXiv · showhide
The Convolutional Neural Networks (CNNs) generate the feature representation of complex objects by collecting hierarchical and different parts of semantic sub-features. These sub-features can usually be distributed in grouped form in the feature vector of each layer, representing various semantic entities. However, the activation of these sub-features is often spatially affected by similar patterns and noisy backgrounds, resulting in erroneous localization and identification. We propose a Spatial Group-wise Enhance (SGE) module that can adjust the importance of each sub-feature by generating an attention factor for each spatial location in each semantic group, so that every individual group can autonomously enhance its learnt expression and suppress possible noise. The attention factors are only guided by the similarities between the global and local feature descriptors inside each group, thus the design of SGE module is extremely lightweight with \emph{almost no extra parameters and calculations}. Despite being trained with only category supervisions, the SGE component is extremely effective in highlighting multiple active areas with various high-order semantics (such as the dog's eyes, nose, etc.). When integrated with popular CNN backbones, SGE can significantly boost the performance of image recognition tasks. Specifically, based on ResNet50 backbones, SGE achieves 1.2\% Top-1 accuracy improvement on the ImageNet benchmark and 1.0$\sim$2.0\% AP gain on the COCO benchmark across a wide range of detectors (Faster/Mask/Cascade RCNN and RetinaNet). Codes and pretrained models are available at https://github.com/implus/PytorchInsight.
1 Introduction
CNN feature groups can represent different semantic entities, but noisy backgrounds and missing region-level supervision disrupt their spatial organization. SGE addresses this by applying similarity-guided, group-specific spatial attention with negligible overhead, improving semantic feature distribution and recognition and detection performance.
- Grouped CNN sub-features represent different semantics, but noisy backgrounds and limited region-level supervision can disorder their spatial distribution.
- SGE scales feature vectors at each spatial location with group-specific attention masks that suppress noise and highlight correct semantic regions.
- The attention masks use similarity between each group’s global statistical feature and local positional features, requiring almost no additional parameters or calculations.
- SGE improves spatial distribution within semantic groups, increases activation variance, strengthens semantic-region learning, and compresses noise and interference.
- On ImageNet, SGE performs better or comparably to several state-of-the-art attention modules while having lower model capacity and complexity.
- Across Faster, Mask, and Cascade RCNN detectors on COCO, SGE brings more than 1% AP gains and improves small-object detection over SE on RetinaNet by approximately 1% AP.
2 Related Work
Prior work has used feature grouping to organize semantic or spatial information in convolutional representations, while attention models refine feature responses across vision tasks. SGE builds on these directions by enhancing grouped sub-features spatially.
- Grouped Features: Grouped feature representations appear in handcrafted descriptors, group convolution, group normalization, multi-scale bottlenecks, and capsule-based entity representations.
- Grouped Features: Earlier grouped-convolution methods use grouping to reduce computation, improve accuracy under similar complexity, or model spatial relationships within channels.
- Grouped Features: Capsule-based approaches interpret grouped neurons as entity representations whose activity and vector parameters encode existence and properties.
- Attention Models: Attention models expanded from language applications into computer vision tasks including person re-identification, image recovery, lip reading, classification, and object detection.
3 Method
SGE enhances each channel-group's spatial semantic representation by deriving location-wise attention from global–local feature similarity, then scaling group features to emphasize relevant regions and suppress noise.
- Spatial Group-wise Enhance: Each channel group contains spatial feature vectors that can represent a specific semantic response, such as a dog's eyes, but noise and similar patterns disrupt localization.The feature map is divided into G groups, with m = H × W spatial positions.
- Spatial Group-wise Enhance: SGE computes a global group descriptor by spatial averaging and compares it with each local feature vector using a dot product.The coefficient can be interpreted through vector lengths and the cosine of the angle between global and local features.
- Spatial Group-wise Enhance: The coefficients are normalized across space, then scaled and shifted by group-specific γ and β parameters before sigmoid gating scales each original feature vector.The normalization is designed to avoid biased coefficient magnitudes across samples while retaining an identity transform through γ and β.
- Spatial Group-wise Enhance: SGE introduces only γ and β for each group, typically totaling 32 or 64 parameters in one unit, negligible beside millions of backbone parameters.These are the only parameters introduced by the module.
- Visualization and Interpretation: The activation-variance analysis compares feature-map distributions before and after SGE and plots standard deviation for each group in the last bottleneck.The analysis uses the variance of each group's spatial activation values.
- Visualization and Interpretation: On ResNet50 visualizations, groups 18, 22, and 41 correspond to nose, tongue, and eyes, respectively, across images with varied shapes, categories, and viewing angles.The experiment uses 14 × 14 feature maps from the fourth stage and G = 64.
4 Experiments on Image Classification
Experiments on ImageNet evaluate SGE against lightweight attention modules and ablate its initialization, group number, and normalization choices. SGE remains competitive with fewer parameters, while the ablations identify settings that support performance.
- Comparisons with state-of-the-art Attention Modules: ImageNet evaluation compares SGE with state-of-the-art lightweight attention modules using ResNet50 and ResNet101 backbones.The benchmark reports Top-1 and Top-5 validation accuracy under a single 224 × 224 central crop.
- Comparisons with state-of-the-art Attention Modules: SGE-ResNet50 is on par with the best CBAM Top-1 and SK/SE Top-5 results, while using fewer parameters and slightly fewer calculations.
- Ablation Study: The number of groups G balances semantic diversity against the sub-feature dimension available for each semantic response.Too many groups weaken each sub-feature representation, whereas too few limit semantic diversity.
- Ablation Study: Performance increases and then decreases as G grows, especially for Top-1 accuracy, leading to a recommendation of G = 32 or 64 and a default of G = 64.
- Ablation Study: Initializing γ to 0 tends to produce better results than the tested alternatives and reduces the attention mechanism's early training influence.The authors explain that this temporarily prioritizes learning a basic semantic representation.
- Ablation Study: Removing normalization considerably reduces performance, supporting its importance for learning robust importance coefficients across inconsistent sample feature distributions.The authors also relate this inconsistency to large variation in activation variance across samples.
5 Experiments on Object Detection
Object-detection experiments embed SGE into established two-stage and one-stage detectors while keeping comparisons controlled. SGE produces notable AP gains, especially for stronger detectors and small objects, with almost no additional parameters or calculations.
- Experiments on state-of-the-art Detectors: SGE improves detection performance by basically more than 1% AP across Faster RCNN, Mask RCNN, and Cascade RCNN.The detector comparisons replace only the pretrained ImageNet backbone while keeping the remaining detector components intact.
- Experiments on state-of-the-art Detectors: +1.5% AP on ResNet50 and +1.8% on ResNet101 are reported for SGE in Cascade RCNN.The gains are larger on stronger baseline detection models.
- Experiments on state-of-the-art Detectors: The detection gains are obtained with almost no additional parameters and calculations in the SGE-embedded backbones.
- Comparisons with state-of-the-art Attention Modules: SGE greatly improves RetinaNet detection accuracy for small objects, while remaining close to optimal performance for medium and large objects.The cited comparisons include 41.2 vs 41.3 for medium objects and 49.9 vs 50.4 for large objects against SE and SK.
- Comparisons with state-of-the-art Attention Modules: SGE's spatially varying importance improves precise spatial-area representation, whereas SE and SK assign the same coefficient to every location within each channel.
6 Conclusion
SGE enables feature groups to autonomously strengthen learned semantic representations and suppress noise with nearly no additional parameters or computational complexity. Its simplicity accompanies steady gains in image classification and detection.
- SGE lets each feature group autonomously enhance its learned semantic representation and suppress possible noise.
- SGE introduces nearly no additional parameters or computational complexity.
- SGE steadily improves performance on both image classification and detection tasks.