PaperlayerView exact revision on arXiv

Source-linked AI summary

SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization

Hongbo Sun, Xiangteng He, Yuxin Peng

arXiv:2208.14607v1cs.CV

TL;DR

Fine-grained visual categorization must distinguish similar subordinate categories, while many approaches focus on discriminative regions and neglect their interdependencies and holistic object structure. SIM-Trans incorporates object structure information into a vision transformer using SIL and MFB. It achieves state-of-the-art results on two FGVC benchmarks, with 91.8% classification accuracy on CUB-200-2011.

  • Problem

    Many FGVC approaches focus on finding discriminative regions but neglect their interdependencies and holistic object structure, despite their importance for localizing discriminative information and understanding objects.

    Source: §1 · pp. 1–2

  • Method

    SIM-Trans adds object structure information to a vision transformer through SIL, which models relations among significant patches, and MFB, which combines multi-level features and contrastive learning.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    SIM-Trans achieves state-of-the-art results on two typical FGVC benchmarks; on CUB-200-2011 it reaches 91.8% classification accuracy.

    Source: §1 · p. 2, §4.2.2 · p. 5

  • Takeaways & Limitations

    The authors present SIL and MFB as lightweight modules that can be plugged into vision-transformer backbones and trained end-to-end.

    Source: §1 · p. 2

    The authors leave embedding structure information at the transformer's initial layer for future study to improve FGVC performance.

    Source: §5 · p. 8

Abstract

from arXiv · show

Fine-grained visual categorization (FGVC) aims at recognizing objects from similar subordinate categories, which is challenging and practical for human's accurate automatic recognition needs. Most FGVC approaches focus on the attention mechanism research for discriminative regions mining while neglecting their interdependencies and composed holistic object structure, which are essential for model's discriminative information localization and understanding ability. To address the above limitations, we propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning to contain both the appearance information and structure information. Specifically, we encode the image into a sequence of patch tokens and build a strong vision transformer framework with two well-designed modules: (i) the structure information learning (SIL) module is proposed to mine the spatial context relation of significant patches within the object extent with the help of the transformer's self-attention weights, which is further injected into the model for importing structure information; (ii) the multi-level feature boosting (MFB) module is introduced to exploit the complementary of multi-level features and contrastive learning among classes to enhance feature robustness for accurate recognition. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily, which only depends on the attention weights that come with the vision transformer itself. Extensive experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks. The code is available at https://github.com/PKU-ICST-MIPL/SIM-Trans_ACMMM2022.

1 INTRODUCTION

Fine-grained categorization requires distinguishing subtle differences between similar subcategories, yet many approaches select discriminative regions without modeling their relations or holistic object structure. SIM-Trans adds structure information to a vision transformer through SIL and MFB, and is reported to achieve state-of-the-art performance on two FGVC benchmarks.

1 INTRODUCTION

  • FGVC distinguishes subordinate categories with large intra-class variance and small inter-class variance, where subtle differences such as tail and beak shape matter.“Artic Tern” and “Common Tern” are examples within the basic category “Bird.”Source: §1 · p. 1
  • Existing approaches often select salient patches while ignoring their relations and holistic object structure, which can impair localization of the object extent.For example, a model may confuse a bird's legs with surrounding twigs.Source: §1 · pp. 1–2
  • SIL uses transformer attention to select discriminative patches and models their spatial context relations, while MFB combines features from the last three layers with contrastive learning.The resulting structure information is injected into the backbone through graph convolution.Source: §1 · p. 2
  • SIM-Trans combines structure information learning (SIL) and multi-level feature boosting (MFB) to learn fine-grained representations containing appearance and structure information.SIL models spatial context relations among discriminative regions; MFB exploits complementary multi-level features and contrastive learning.Source: §1 · p. 2
  • The authors report that SIM-Trans achieves new state-of-the-art performance on two typical FGVC benchmarks.The modules are described as lightweight, plug-in components for vision-transformer backbones that can be trained end-to-end.Source: §1 · p. 2

2 RELATED WORK

The related work reviews fine-grained categorization and vision transformers, noting that both region-focused FGVC approaches and transformer methods generally leave object structure or patch relations unmodeled.

2.1 Fine-grained Visual Categorization

  • Recent FGVC approaches discover discriminative regions and extract features, but generally ignore holistic object structure that can help localize the whole object extent.The cited approaches include sparse attention, reinforcement learning for region selection, progressive mask attention, and counterfactual attention learning.Source: §2.1 · p. 2

2.2 Vision Transformer

  • Vision-transformer approaches encode images as patch-token sequences, but generally ignore spatial relations among patches that matter for discovering significant patches.RAMS-Trans is described as using recurrent attention and multi-scale learning to find and amplify significant regions.Source: §2.2 · p. 2

3 APPROACH

SIM-Trans extends a vision transformer with structure learning and multi-level feature boosting to combine object appearance and spatial composition for fine-grained recognition. Its modules use transformer attention and features to model patch relationships and strengthen representations.

3 APPROACH

  • SIM-Trans adds structure information learning (SIL) and multi-level feature boosting (MFB) to a vision transformer for fine-grained categorization.The framework uses transformer self-attention weights to learn spatial context and combines features from multiple levels with contrastive learning.Source: §3 · pp. 2–3

3.1 Vision Transformer Backbone

  • The backbone splits images into patch tokens using sliding windows, which alleviate the loss of neighboring information caused by hard patch separation.The patch sequence is projected into embeddings, with a learnable CLS token and position embeddings added before transformer processing.Source: §3.1 · p. 3
  • The backbone concatenates the CLS-token outputs of the last three transformer layers as the image representation for classification.The backbone, SIL, and MFB are jointly trained end to end.Source: §3.1 · p. 3, §3.3 · p. 5

3.2 Structure Information Learning

  • SIL uses patch-to-CLS attention to localize object-relevant patches, then uses polar coordinates to describe each patch relative to the most-attended reference patch.The mean attention value thresholds insignificant patches before structure learning.Source: §3.2 · p. 4
  • SIL builds a graph from spatial-context patch features and attention-derived edge weights, then applies two-layer graph convolution to produce structure features.The reference patch’s structure feature is added to the CLS token feature, incorporating object composition into the transformer.Source: §3.2 · p. 4

3.3 Multi-level Feature Boosting

  • MFB concatenates CLS-token features from the last three transformer layers and uses contrastive learning to increase same-class similarity and decrease different-class similarity.Negative pairs whose similarity is α smaller than that of positive pairs are filtered out.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Experiments evaluate SIM-Trans on CUB-200-2011 and iNaturalist 2017, compare it with state-of-the-art methods, and analyze qualitative visualizations, component ablations, and a hyperparameter.

4.1 Datasets and Evaluation Metric

  • Experiments use CUB-200-2011 and iNaturalist 2017, with accuracy as the metric; CUB has 11,788 images across 200 bird subcategories.CUB is split into 5,994 training and 5,794 test images.Source: §4.1 · p. 5

4.2 Experiments and Analyses on CUB-200-2011

  • 91.8% classification accuracy on CUB-200-2011 is 1.2% above ViT and optimal CNN method CAL.SIM-Trans also outperforms RAMS-Trans by 0.5%.Source: §4.2.2 · pp. 5–6, Table 2 · p. 6
  • Compared with ViT, SIM-Trans filters irrelevant background, completes object extent, and highlights discriminative regions in its attention visualizations.Figure 2 compares original images, ViT attention, and SIM-Trans attention.Source: Figure 2 · p. 6, §4.2.3 · p. 6

4.3 Experiments and Analyses on iNaturalist 2017

  • Visualizations on CUB-200-2011 and iNaturalist 2017 highlight entire objects and discriminative regions, including objects from different subcategories and supercategories.Figure 3 places original images in rows one and three and corresponding focus regions in rows two and four.Source: §4.2.3 · p. 6, Figure 3 · p. 7, §4.3.1 · p. 7
  • 69.9% classification accuracy on iNaturalist 2017 exceeds ResNet152 by 10.9%, ViT by 2.9%, TASN by 1.7%, and RAMS-Trans by 1.4%.The benchmark contains more than 5,000 categories and 95,000 test images.Source: Table 3 · p. 7, §4.3.1 · p. 7

4.4 Effectiveness of Each Component in Our SIM-Trans Approach

  • On CUB, SIL adds 0.5% over the 90.6% ViT baseline; three SIL layers perform best, while MFB adds another 0.7%.Contrastive learning in MFB adds 0.4% classification accuracy.Source: Table 4 · p. 7, Figure 4 · p. 8, §4.4 · p. 8

4.5 Hyper-parameter Experiments

  • α=0.3 gives the best classification performance; at higher values, the threshold cannot filter easy negative pairs and performance degrades.α controls the hardness degree of negative pairs in contrastive learning.Source: Figure 5 · p. 8, §4.5 · p. 8

5 CONCLUSION

SIM-Trans combines structure information learning and multi-level feature boosting to represent both object structure and appearance, achieving state-of-the-art FGVC results on two benchmarks. The authors identify initial-layer structure embedding as future work for improving structure modeling.

5 CONCLUSION

  • SIM-Trans achieves state-of-the-art FGVC performance on two typical fine-grained benchmarks, with promising results on large-scale iNaturalist 2017.The conclusion reports the overall benchmark outcome without giving specific accuracy values.Source: §5 · p. 8
  • SIL models spatial context among significant patches, while MFB exploits complementary multi-level features and contrastive learning to improve representation robustness.Together, the modules promote feature discrimination and support representations containing both appearance and structure information.Source: §5 · p. 8
  • Initial-layer structure embedding is identified as future work for improving structure modeling and FGVC performance.Source: §5 · p. 8

Related papers

Loading 2208.14607v1…