Source-linked AI summary
Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up
Weifeng Ge, Xiangru Lin, Yizhou Yu
TL;DR
Image-level supervision can make convolutional networks focus on the most discriminative parts while missing complementary object information. This paper mines complementary parts weakly supervised and fuses them with a bi-directional LSTM, achieving state-of-the-art performance and outperforming existing methods by a large margin.
Problem
Image-level supervision can cause models to focus on the most discriminative parts while missing other object parts, motivating weakly supervised solutions to object detection and classification.
Method
The method weakly supervises object detection and instance segmentation, searches for diverse complementary parts, and fuses their partial descriptions with a bi-directional LSTM for classification.
Results
The proposed method achieves state-of-the-art performance on multiple image classification tasks and outperforms existing state-of-the-art methods by a large margin.
Takeaways & Limitations
Complementary parts provide an alternative to focusing only on the most discriminative regions in weakly supervised fine-grained image classification.
Takeaways & Limitations
Building the complementary part model more efficiently and accurately remains an open problem.
Abstract
from arXiv · showhide
Given a training dataset composed of images and corresponding category labels, deep convolutional neural networks show a strong ability in mining discriminative parts for image classification. However, deep convolutional neural networks trained with image level labels only tend to focus on the most discriminative parts while missing other object parts, which could provide complementary information. In this paper, we approach this problem from a different perspective. We build complementary parts models in a weakly supervised manner to retrieve information suppressed by dominant object parts detected by convolutional neural networks. Given image level labels only, we first extract rough object instances by performing weakly supervised object detection and instance segmentation using Mask R-CNN and CRF-based segmentation. Then we estimate and search for the best parts model for each object instance under the principle of preserving as much diversity as possible. In the last stage, we build a bi-directional long short-term memory (LSTM) network to fuze and encode the partial information of these complementary parts into a comprehensive feature for image classification. Experimental results indicate that the proposed method not only achieves significant improvement over our baseline models, but also outperforms state-of-the-art algorithms by a large margin (6.7%, 2.8%, 5.2% respectively) on Stanford Dogs 120, Caltech-UCSD Birds 2011-200 and Caltech 256.
1. Introduction
The paper addresses the tendency of image-label-trained networks to focus on dominant parts by mining complementary object parts weakly supervisedly and fusing them for classification. Its pipeline combines object detection, instance segmentation, complementary-part modeling, and bi-directional LSTM encoding, achieving state-of-the-art results on multiple tasks.
- Image-label-trained convolutional networks tend to focus on the most discriminative parts while missing other potentially complementary object parts.
- The paper targets two challenges: detecting objects without ground-truth boxes and exploiting detections to improve image classification using only image-level labels.
- The proposed representation selects a subset of object proposals to capture more complete object descriptions through weakly supervised complementary parts modeling.
- The detection and segmentation pipeline iterates Mask R-CNN object detection and segmentation with CRF-based instance-segmentation enhancement.
- A bi-directional LSTM fuses complementary information from different object parts to make the final classification decision.
2. Related Work
Related work covers weakly supervised object detection and segmentation, part-based fine-grained classification, and context encoding. The paper instead uses object proposals as complementary information and integrates them with a bi-directional LSTM.
- Weakly Supervised Object Detection and Segmentation: Weakly supervised object detection and segmentation localize and segment objects using image labels only.
- Weakly Supervised Object Detection and Segmentation: The paper uses CAM-derived coarse masks and proposals for Mask R-CNN, then iteratively rectifies object locations and masks with CRF.
- Part Based Fine-grained Image Classification: Unlike methods that build strong part detectors, the complementary parts model uses information hidden in object proposals produced during detection.
- Part Based Fine-grained Image Classification: Different object proposals contain complementary information that can form a more complete object description for classification.
- Context Encoding: The method uses a bi-directional LSTM to learn a deep hierarchical representation of image patches, improving substantially over a single-layer LSTM.
3. Weakly Supervised Complementary Parts Model
The method builds complementary object-part models from weakly supervised instance detection and segmentation, then fuses their information for classification. It iteratively refines object proposals and selects diverse parts before applying context encoding.
- Weakly Supervised Object Detection and Instance Segmentation: Image-level labels drive an iterative pipeline that combines CAM initialization, CRF segmentation, and Mask R-CNN detection and instance-segmentation refinement.CRF-generated segments and bounding boxes serve as pseudo-groundtruth annotations for Mask R-CNN training.
- Complementary Parts Model: Suppressed proposals from non-maximum suppression are retained because they may contain useful object-part information beyond dominant regions.The method treats these proposals as candidate complementary parts rather than discarding them.
- Complementary Parts Model: A complementary parts model represents an object with a root part, a center part, and surrounding parts described by appearance features and geometric information.The root covers the object and context, while other parts cover the core and different surrounding regions.
- Complementary Parts Model: The selected hypothesis maximizes part scores while penalizing semantic similarity and spatial overlap, encouraging diverse complementary proposals.The stated penalties use λ0 = 0.01 for semantic similarity and β0 = 0.1 for spatial overlap.
- Complementary Parts Model: A greedy search replaces the intractable exhaustive subset search and reduces the overall time complexity from exponential to linear, O(nk).The search sequentially selects an optimal proposal for each part from the suppressed proposal set.
- Image Classification with Context Encoding: Multiple patch losses improve classification accuracy over using only the final output loss from the second LSTM.The multiple-loss setting assigns γi = 1 for patch outputs, whereas the single-loss setting assigns γi = 0 for those outputs.
4. Experimental Results
Experiments evaluate implementation details, benchmark performance, and ablations across fine-grained and generic image classification. The proposed complementary-parts model with stacked LSTM improves accuracy over baselines and state-of-the-art methods, while ablations support the roles of parts search, multiple losses, and LSTM fusion.
- Implementation Details: The implementation uses GoogleNet for classification, ResNet-50 with FPN for Mask R-CNN refinement, and stacked LSTM sequences containing n + 2 image-patch features.All experiments use n = 9 parts; Mask R-CNN is pretrained on COCO and fine-tuned on target datasets.
- Fine-grained Image Classification: 93.9% accuracy is achieved on Stanford Dogs 120 with multiple-loss training, compared with 85.2% for the directly fine-tuned GoogleNet baseline.The corresponding single-loss stacked-LSTM model reaches 92.4%; with SJFT, multiple-loss training reaches 97.1%.
- Fine-grained Image Classification: 90.3% accuracy is achieved on CUB200 with multiple losses and no extra training data, while SJFT contributes only 0.1% additional gain in the reported comparison.The no-extra-data baseline is 82.6%, and the multiple-loss model reaches 90.4% with SJFT.
- Generic Object Recognition: 93.5% accuracy is achieved on Caltech 256 with multiple losses, outperforming current state-of-the-art by 5.2%.Without extra data, accuracy rises from the 84.1% baseline to 93.5%; with SJFT, the multiple-loss gain over baseline is 8.0%.
- Ablation Study: 90.3% accuracy is best when mining n = 9 parts; using image features only, uniform grid cells, or random suppressed proposals lowers accuracy to 82.8%, 78.3%, and 86.9%.Object features only reduce accuracy to 90.0%, and randomly shuffling part order leaves accuracy unchanged.
- Ablation Study: Multiple losses improve Stanford Dogs accuracy from 92.4% to 93.9%, while replacing stacked LSTM with concatenation or averaging decreases accuracy by 5.8% and 8.7%.The ablation supports the reported contribution of multiple-loss training and LSTM-based context fusion.
5. Conclusions
The paper presents a weakly supervised complementary-part pipeline for fine-grained image classification, using bi-directional stacked LSTM to fuse partial object descriptions. It outperforms existing state-of-the-art methods by a large margin, while efficient and accurate complementary-part modeling remains open.
- The proposed pipeline mines complementary parts containing partial object descriptions instead of focusing only on the most discriminative parts.
- It fuses the mined partial object descriptions with a bi-directional stacked LSTM for classification.
- Experimental results show that the method is effective and outperforms existing state-of-the-art methods by a large margin.
- Building complementary-part models more efficiently and accurately remains an open problem for future investigation.