Source-linked AI summary
See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification
Tao Hu, Honggang Qi, Qingming Huang, Yan Lu
TL;DR
Random data augmentation can be inefficient and introduce uncontrolled background noise, while fine-grained recognition needs discriminative local-part features without costly part annotations. WS-DAN learns attention maps from image-level labels, uses them for cropping and dropping, and localizes objects for refinement. Experiments on common fine-grained datasets report higher classification and localization accuracy than state-of-the-art methods and baselines.
Problem
Random augmentation may introduce uncontrolled background noise, while fine-grained recognition requires discriminative local parts and object-part annotations are costly.
Method
WS-DAN learns attention maps from image-level supervision, applies attention cropping and dropping, and uses attention regions to localize and enlarge objects.
Results
Experiments on common fine-grained visual classification datasets report improved classification and object localization accuracy over state-of-the-art methods and baselines.
Takeaways & Limitations
Weakly supervised attention and data augmentation are combined so each provides spatial information or training pressure that supports more discriminative features from multiple local regions.
Takeaways & Limitations
The method assumes object location annotations such as bounding boxes or keypoints are unavailable and relies only on category annotations.
Abstract
from arXiv · showhide
Data augmentation is usually adopted to increase the amount of training data, prevent overfitting and improve the performance of deep models. However, in practice, random data augmentation, such as random image cropping, is low-efficiency and might introduce many uncontrolled background noises. In this paper, we propose Weakly Supervised Data Augmentation Network (WS-DAN) to explore the potential of data augmentation. Specifically, for each training image, we first generate attention maps to represent the object's discriminative parts by weakly supervised learning. Next, we augment the image guided by these attention maps, including attention cropping and attention dropping. The proposed WS-DAN improves the classification accuracy in two folds. In the first stage, images can be seen better since more discriminative parts' features will be extracted. In the second stage, attention regions provide accurate location of object, which ensures our model to look at the object closer and further improve the performance. Comprehensive experiments in common fine-grained visual classification datasets show that our WS-DAN surpasses the state-of-the-art methods, which demonstrates its effectiveness.
1. Introduction
WS-DAN uses weakly supervised attention maps to guide data augmentation toward discriminative object parts, addressing noisy random cropping in fine-grained classification. It also localizes and enlarges whole objects for closer inspection, and experiments report improvements over state-of-the-art methods and baselines.
- Randomly cropped regions often contain background noise, lowering training efficiency and feature quality.
- Attention-guided augmentation addresses fine-grained categories’ subtle differences and viewpoint-related occlusion by emphasizing diverse local parts.
- WS-DAN generates attention maps from image-level supervision to represent the spatial distribution of discriminative object parts.
- Attention cropping crops and resizes one attention part, while attention dropping erases one region to encourage features from multiple parts.
- WS-DAN localizes the whole object from attention maps, enlarges it, and performs coarse-to-fine prediction on testing images.
- Experiments on common fine-grained visual classification datasets report improved classification and object localization accuracy over state-of-the-art methods and baselines.
2. Related Works
Prior fine-grained recognition methods use specialized localization, pooling, metric-learning, and augmentation strategies. WS-DAN is positioned against limitations including annotation cost, single-part focus, noisy random augmentation, and incomplete weakly supervised localization.
- Basic CNNs achieve only moderate fine-grained performance because they struggle to focus on subtle object-part differences without specialized design.
- Fine-grained methods often use part-location or attribute annotations to focus on local features, but these annotations add labeling cost.
- Existing approaches include geometric alignment, recurrent attention, bilinear or covariance pooling, and metric-learning objectives.
- Random cropping and dropping improve robustness but may erase background or entire small objects, producing uncontrolled noisy data.
- Weakly supervised localization uses image-level supervision, but standard classification often focuses on the most discriminative region rather than the whole object.
3. Approach
The WS-DAN approach combines weakly supervised attention learning with attention-guided augmentation and object localization. Its training and testing process is summarized in Figure 2.
- WS-DAN comprises weakly supervised attention learning, attention-guided data augmentation, and object localization and refinement.
- Figure 2 presents the overview of WS-DAN’s training and testing process.
3.1. Weakly Supervised Attention Learning
WS-DAN uses category annotations to learn attention maps that localize discriminative object parts, then extracts separate part features through bilinear attention pooling. Attention regularization encourages consistent part assignments across images.
- Weakly Supervised Attention Learning: Weak supervision predicts object-location distributions using only category annotations, without bounding boxes or keypoints.
- Spatial Representation: Each convolution-generated attention map represents a discriminative object part or visual pattern, such as a bird head, car wheel, or aircraft wing.
- Bilinear Attention Pooling: Bilinear Attention Pooling multiplies each attention map with feature maps, then applies pooling or convolution to extract one local feature per part.
- Bilinear Attention Pooling: The resulting object representation is a part-feature matrix formed by stacking the extracted attention features.
- Attention Regularization: Attention regularization penalizes within-part feature variance so each map consistently activates the same object part across examples.
3.2. Attention-guided Data Augmentation
Attention-guided augmentation selects and normalizes an attention map, then uses it to crop or drop a discriminative part. These operations focus augmentation on object regions and encourage multiple-part feature extraction.
- Augmentation Map: Randomly selecting and normalizing one attention map provides the guide for augmenting each training image.
- Attention Cropping: Attention cropping thresholds the selected map, finds a bounding box around its positive region, and enlarges that part for detailed feature extraction.
- Attention Dropping: Attention dropping thresholds the selected map to create a mask that removes the corresponding part region from the image.
- Attention Dropping: Dropping one part encourages the network to discover other discriminative parts, improving classification robustness and localization accuracy.
3.3. Object Localization and Refinement
At test time, WS-DAN first predicts from the raw image, localizes the object with attention maps, and enlarges the object region for a second prediction. The final result averages both predictions.
- Object Localization: The model predicts coarse classification probabilities and attention maps from the raw image, then computes an object map and its bounding box.
- Refinement: The localized bounding-box region is enlarged and passed through the same trained model to obtain a fine-grained prediction.
- Prediction Fusion: The final classification probability is p = (p1 + p2)/2, averaging coarse-grained and fine-grained predictions.
4. Experiments
Experiments evaluate WS-DAN on four fine-grained classification datasets, test its components and attention-guided augmentation, and compare it with random augmentation and state-of-the-art methods. Results indicate improved classification and localization, with performance stabilizing near 32 attention maps.
- 4.1. Datasets and Experiments Settings: Experiments use CUB-200-2011, FGVC-Aircraft, Stanford Cars, and Stanford Dog datasets, with Inception v3 as the backbone.The setup uses Mix6e feature maps, 1 × 1 convolutional attention maps, GAP pooling, and fixed cropping and dropping thresholds of 0.5.
- 4.2. Ablation Study: WS-DAN comprises weakly supervised attention learning, attention cropping, attention dropping, and object localization and refinement; experiments test each component and their combinations.The component study is conducted on CUB-200-2011, as summarized in Table 2.
- 4.4. Comparision with Random Data Augmentation: Attention-guided data augmentation is more efficient than random data augmentation in the CUB-200-2011 comparison.The experiment evaluates first-stage object localization using mean Intersection-over-Union, where higher mIoU indicates more accurate localization.
- 4.5. Comparison with Stage-of-the-Art Methods: WS-DAN achieves state-of-the-art accuracy on CUB-200-2011, FGVC-Aircraft, Stanford Cars, and Stanford Dog, significantly improving over the Inception v3 backbone.The comparisons are reported in Tables 4–7.
- 4.6. Object Localization Results: WS-DAN surpasses state-of-the-art methods and baselines in object localization results, with localization error significantly lower than other methods.The localization evaluation uses error for images whose predicted boxes have less than 50% IoU with ground truth, including CUB-200-2011 and Stanford Dogs.
- 4.7. Effect of the Number of Attention Maps: 89.4% accuracy is reached with increasing attention maps, while performance becomes stable at around 32 maps on CUB-200-2011.The experiment reports that more object parts generally contribute to better performance and that increasing attention maps can improve accuracy.
- 4.4. Comparision with Random Data Augmentation: Random augmentation introduces substantial background into training images, whereas attention guidance makes cropping and dropping more efficient by using object-part locations.The comparison is visualized on CUB-200-2011 and FGVC-Aircraft images in Figure 4.
5. Conclusion
WS-DAN combines weakly supervised learning with data augmentation: attention maps provide object spatial distributions, while augmentation encourages learning from multiple local regions. The resulting approach is reported to surpass state-of-the-art methods.
- 5. Conclusion: WS-DAN combines weakly supervised learning with data augmentation so spatial attention guides augmentation and augmentation supports attention learning.The two components benefit from each other and promote extraction of more discriminative image features from multiple local regions.
- 5. Conclusion: WS-DAN surpasses state-of-the-art methods by extracting more discriminative features from multiple local regions.This conclusion summarizes the reported outcome of the proposed approach.