PaperlayerView exact revision on arXiv

Source-linked AI summary

Mask-CNN: Localizing Parts and Selecting Descriptors for Fine-Grained Image Recognition

Xiu-Shen Wei, Chen-Wei Xie, Jianxin Wu

arXiv:1605.06878v1cs.CV

TL;DR

Fine-grained recognition must distinguish highly similar subordinate categories, where subtle differences make useful parts and descriptors important. Mask-CNN uses part annotations to learn masks that select convolutional descriptors and combines object- and part-level cues in a four-stream model. On CUB200-2011, it reports 85.5% classification accuracy alongside the smallest parameter count and lowest-dimensional feature representation among compared methods.

  • Problem

    Fine-grained categories have subtle inter-class differences, and prior part-based methods used convolutional descriptors without assessing their usefulness for recognition.

    Source: §1 · p. 1

  • Method

    Mask-CNN learns object and part masks from part annotations to select convolutional descriptors, then aggregates image, head, torso, and object cues in four streams.

    Source: §1 · p. 1

  • Results

    On CUB200-2011, M-CNN reports 85.5% classification accuracy, with the least parameters and lowest-dimensional feature representations among compared methods.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported combination of high accuracy and compact representations makes Mask-CNN computationally and storage efficient compared with state-of-the-art methods.

    Source: §1 · p. 2, §5 · p. 8

    Part localization is lower than Deep LAC when compared with its use of bounding boxes during testing; M-CNN uses no supervision at testing.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Fine-grained image recognition is a challenging computer vision problem, due to the small inter-class variations caused by highly similar subordinate categories, and the large intra-class variations in poses, scales and rotations. In this paper, we propose a novel end-to-end Mask-CNN model without the fully connected layers for fine-grained recognition. Based on the part annotations of fine-grained images, the proposed model consists of a fully convolutional network to both locate the discriminative parts (e.g., head and torso), and more importantly generate object/part masks for selecting useful and meaningful convolutional descriptors. After that, a four-stream Mask-CNN model is built for aggregating the selected object- and part-level descriptors simultaneously. The proposed Mask-CNN model has the smallest number of parameters, lowest feature dimensionality and highest recognition accuracy when compared with state-of-the-arts fine-grained approaches.

1 Introduction

Mask-CNN selects useful convolutional descriptors using learned object and part masks, then combines object- and part-level cues in a four-stream model. The paper reports high CUB200-2011 classification accuracy, compact representations, and strong part localization.

  • Fine-grained categories differ in subtle visual details, making discriminative parts useful for distinguishing subclasses and aligning same-class objects.Prior part-based approaches used convolutional descriptors without assessing their usefulness for recognition.Source: §1 · p. 1
  • The model uses FCN-generated object and part masks to select useful convolutional descriptors, then aggregates image, head, torso, and object streams.The FCN is trained using part annotations and image-level labels; the method does not use annotations at testing time.Source: §1 · pp. 1–2
  • Mask-CNN is presented as the first end-to-end model to select deep convolutional descriptors for object recognition, especially fine-grained recognition.Source: §1 · p. 2
  • Compared with state-of-the-art methods, M-CNN has the least parameters (60.49M) and smallest feature dimensionality (8,192-d), achieving 85.4% classification accuracy on CUB200-2011; SVD whitening compresses features to 4,096-d and improves accuracy to 85.5%.Source: §1 · p. 2
  • M-CNN’s part-localization performance outperforms other part-based fine-grained approaches requiring additional bounding boxes; its head localization is about 10% higher than the state of the art, with 84.62% head and 89.83% torso localization.Source: §1 · p. 2

2 Related Work

Prior fine-grained recognition work develops discriminative representations, aligns objects, or uses part-based representations. Mask-CNN differs by using segmentation masks not only to localize objects and parts, but also to select convolutional descriptors.

  • Prior approaches fall into representation-learning, object-alignment, and part-based groups, with the latter motivated by subtle differences in object parts.Source: §2 · p. 2
  • Compared with part-based methods using bounding boxes and part annotations, M-CNN needs only part annotations during training and no test-time supervision, while jointly capturing object- and part-level information.Source: §2 · p. 3
  • Unlike segmentation-based approaches that use masks only to localize objects or parts, M-CNN also uses them to select useful deep convolutional descriptors.Source: §2 · p. 3, Figure 2 · p. 3
  • M-CNN localizes only head and torso, making its segmentation task easier than the fifteen-part approach in part-stacked CNN.M-CNN is reported to achieve 1.8% higher CUB200-2011 accuracy than with the same baseline network, using fewer training annotations and none at testing.Source: §2 · pp. 3–4

3 The Mask-CNN Model

Mask-CNN uses an FCN to predict object and part masks, then a four-stream model to select descriptors and jointly capture object- and part-level information. Its part masks are learned from keypoint-derived segmentation targets, and its streams combine whole-image, head, torso, and object features.

3 The Mask-CNN Model

  • Mask-CNN combines FCN-predicted object and part masks with four jointly trained CNN streams to capture object- and part-level information.The four streams represent the whole image, head, torso, and object; the FCN segmentation network's parameters remain fixed during M-CNN training.Source: §3 · p. 4, §3.2 · p. 6

3.1 Learning Object and Part Masks

  • Head and torso masks are smallest convex polygons around their key points, and FCN learns them as three-class segmentation targets alongside background.At inference, FCN predicts masks without annotations; the two part masks are combined to form the object mask.Source: §3.1 · p. 4
  • In qualitative testing examples, FCN masks localize parts even though the ground-truth masks are not very accurate.Figure 4 overlays predicted head masks in red, torso masks in blue, and background pixels in black.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2 Training Mask-CNN

  • The whole-image stream resizes its object mask to the convolutional grid and uses it to keep object-region descriptors while zeroing background descriptors.Selected descriptors are average- and max-pooled into two 512-d vectors, then concatenated as a 1024-d stream feature.Source: §3.2 · p. 5
  • Head and torso streams crop the smallest bounding boxes containing their masks, while the object stream crops the region from the combined object mask.The final 4,096-d representation concatenates whole-image, head, torso, and object features; the four CNNs are learned simultaneously.Source: §3.2 · pp. 5–6

4 Experiments

Experiments evaluate CUB200-2011 classification against baselines and prior methods, then assess part localization and foreground-object segmentation.

4.1 Dataset and Implementation Details

  • With VGG-16 initialization, each of the four streams is fine-tuned separately before joint training, with horizontal flipping used to double training data.At test time, predictions for each image and its flipped copy are averaged.Source: §4.1 · p. 6

4.2.1 Baseline Methods

  • M-CNN achieves the best baseline accuracy; 4-stream Pooling is about 1% lower, while 4-stream FCs has the lowest accuracy.The authors suggest the FC baseline’s lower accuracy might be caused by overfitting.Source: Table 1 · p. 7, §4.2.1 · p. 7

4.2.2 Comparisons with state-of-the-art methods

  • 85.5% classification accuracy with 4-stream M-CNN+ after SVD whitening, compressing its feature from 8,192-d to 4,096-d.The uncompressed 8,192-d representation achieves 85.4%.Source: Table 2 · p. 7, §4.2.2 · p. 7
  • Under comparisons that omit test-time part annotations, raising the whole-image stream input to 448 × 448 improves accuracy by 2.1%, while enlarging the object stream slightly lowers it.Source: §4.2.2 · p. 7

4.3 Part Localization Results

  • 84.62% head localization; using PCP at >50% IoU, M-CNN beats part-based R-CNN and strong DPM on torso but trails Deep LAC, which uses test-time boxes.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Object Segmentation Performance

  • 72.41% mean IU on foreground-object segmentation; M-CNN generally segments birds but misses finer parts, while Figure 5 shows it finding both birds despite one being annotated.No segmentation refinement was applied before or after prediction.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Conclusion

Mask-CNN removes fully connected layers and uses object/part masks to select convolutional descriptors, which are aggregated into object- and part-level cues. On CUB200-2011, it achieves 85.5% accuracy with the least parameters and lowest-dimensional feature representations; future work includes weakly supervised part detection and general object categorization.

5 Conclusion

  • 85.5% classification accuracy on CUB200-2011 accompanied Mask-CNN’s least parameter count and lowest-dimensional feature representations.The conclusion reports these as the model’s outcomes on the dataset.Source: §5 · p. 8
  • Mask-CNN localizes objects and parts, generates masks to select convolutional descriptors, and aggregates object- and part-level cues in a four-stream model.The model is described as operating without fully connected layers.Source: §5 · p. 8
  • Future work includes weakly supervised part detection using image-level labels and exploring descriptor selection for general object categorization.The authors say weak supervision would require less labeling effort to achieve comparable classification accuracy.Source: §5 · p. 8

Related papers

Loading 1605.06878v1…