Source-linked AI summary
TransFG: A Transformer Architecture for Fine-grained Recognition
Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang
TL;DR
Fine-grained visual classification is difficult because class differences are subtle, while region-localization methods can complicate the pipeline and retain redundant information. TransFG extends ViT with attention-guided part selection and contrastive learning, achieving state-of-the-art results on most evaluated benchmarks. The paper reports that selected patches can capture meaningful discriminative regions and that transformer models have strong potential for fine-grained tasks.
Problem
Fine-grained visual classification is challenging because subclasses have subtle differences, while existing region-based methods complicate processing and may retain redundant object information.
Method
TransFG applies ViT to fine-grained classification, uses a Part Selection Module to select discriminative patches, and applies contrastive learning to separate confusing subcategories.
Results
TransFG achieves state-of-the-art performance on most datasets among five evaluated fine-grained benchmarks, including 92.3% accuracy on Stanford Dogs and a 2.0% improvement over SOTA.
Takeaways & Limitations
The results support transformer-based models as a promising direction for fine-grained recognition, with small selected patches providing meaningful discriminative regions.
Abstract
from arXiv · showhide
Fine-grained visual classification (FGVC) which aims at recognizing objects from subcategories is a very challenging task due to the inherently subtle inter-class differences. Most existing works mainly tackle this problem by reusing the backbone network to extract features of detected discriminative regions. However, this strategy inevitably complicates the pipeline and pushes the proposed regions to contain most parts of the objects thus fails to locate the really important parts. Recently, vision transformer (ViT) shows its strong performance in the traditional classification task. The self-attention mechanism of the transformer links every patch token to the classification token. In this work, we first evaluate the effectiveness of the ViT framework in the fine-grained recognition setting. Then motivated by the strength of the attention link can be intuitively considered as an indicator of the importance of tokens, we further propose a novel Part Selection Module that can be applied to most of the transformer architectures where we integrate all raw attention weights of the transformer into an attention map for guiding the network to effectively and accurately select discriminative image patches and compute their relations. A contrastive loss is applied to enlarge the distance between feature representations of confusing classes. We name the augmented transformer-based model TransFG and demonstrate the value of it by conducting experiments on five popular fine-grained benchmarks where we achieve state-of-the-art performance. Qualitative results are presented for better understanding of our model.
Introduction
Fine-grained visual classification is difficult because subclasses differ subtly, while existing localization pipelines can be cumbersome to train and operate. TransFG applies vision transformers with discriminative token selection and contrastive learning, achieving state-of-the-art performance on most evaluated datasets.
- Fine-grained visual classification is challenging because subclasses have small inter-class and large intra-class variations, with limited annotations for long-tailed classes.
- Existing localization methods propose discriminative regions, resize them, and reuse the backbone to extract local features, complicating the pipeline and training.
- Vision transformers use self-attention and long-range receptive fields to capture important regions, subtle differences, and spatial relations for fine-grained recognition.
- TransFG introduces a Part Selection Module to find discriminative regions and remove redundant information, together with contrastive loss for more discriminative representations.
- TransFG is evaluated on five fine-grained benchmarks and outperforms existing state-of-the-art CNN methods on most datasets.
Related Work
Prior fine-grained recognition work mainly uses localization or feature-encoding methods, while transformer research has expanded from language to multiple vision tasks. TransFG extends pure vision transformers to fine-grained classification and illustrates its framework through patch-based token processing and selection.
- Fine-grained recognition methods mainly divide into localization approaches that reuse detected parts and feature-encoding approaches that learn richer representations.
- Earlier localization methods used annotated parts, but expensive and unavailable annotations motivated weakly supervised proposals using image-level labels.
- Feature-encoding methods improve classification by learning representations through techniques such as cross-layer bilinear pooling and grouped bilinear pooling.
- Transformers progressed from natural-language processing to detection, segmentation, tracking, and pure image-patch processing in vision.
- TransFG extends ViT to fine-grained classification, using image patches, embeddings, and a Part Selection Module before the final transformer layer.
Method
TransFG adapts Vision Transformer processing for fine-grained recognition by preserving local patch structure, selecting discriminative tokens through accumulated attention, and adding contrastive feature learning.
- Image Sequentialization: Overlapping sliding-window patches preserve local neighboring structures that non-overlapping patching can disrupt.Adjacent patches share an overlap of size (P −S) ∗P; smaller steps improve performance but increase computational cost.
- Patch Embedding: Image patches are projected into a D-dimensional embedding space and augmented with learnable positional embeddings.The projection maps vectorized patches into latent embeddings, while position embeddings retain spatial information.
- TransFG Architecture: The pure Vision Transformer can be applied directly to fine-grained visual classification, but it does not adequately capture the local information required for FGVC.TransFG addresses this limitation with a Part Selection Module and contrastive feature learning.
- Part Selection Module: The Part Selection Module integrates attention weights across previous Transformer layers to identify discriminative patch tokens.Accumulated attention is used to select one token per attention head, which is concatenated with the classification token before the last Transformer layer.
- Contrastive Feature Learning: Contrastive learning minimizes similarity between classification tokens from different labels and maximizes similarity for samples sharing a label.The model jointly optimizes cross-entropy and contrastive losses.
Experiments
TransFG is evaluated across five fine-grained benchmarks and large-scale settings, with comparisons, ablations, and visualizations assessing accuracy, scalability, component effects, and interpretability.
- Experimental Setup: TransFG is evaluated on CUB-200-2011, Stanford Cars, Stanford Dogs, NABirds, and iNat2017, with additional iNat2021 competition results.Experiments use 448 × 448 inputs except 304 × 304 on iNat2017.
- Quantitative Analysis: 1.3% higher Top-1 Accuracy than StackedLSTM and 1.4% higher than ViT are reported on CUB-200-2011.The paper also describes TransFG as maintaining a simpler pipeline than the multi-stage StackedLSTM model.
- Quantitative Analysis: 92.3% accuracy on Stanford Dogs exceeds the reported state of the art by 2.0%.The authors associate this result with discriminative part selection and contrastive-loss supervision.
- Quantitative Analysis: 90.8% accuracy on NABirds is 0.9% above ViT and 1.6% above the reported state of the art.NABirds is described as challenging because it has many images and 355 more categories than the referenced comparison setting.
- Quantitative Analysis: 3.5% higher performance than the reported state of the art makes TransFG the only method above 70% accuracy on iNat2017 with 304-pixel inputs.The paper also reports nearly a 10% advantage of ViT over ResNet152 on iNat2017.
- Ablation Study: Applying the Part Selection Module raises accuracy from 90.3% to 91.0%, while contrastive loss raises TransFG accuracy from 91.0% to 91.5%.The contrastive loss is described as increasing separation between similar subcategories and reducing separation between same-category representations; margin α = 0.4 performs best.
- Qualitative Analysis: Visualizations show Top-4 selected patches and global attention maps across four benchmarks, with bird parts highlighted while occluded twigs are ignored.Selected patches are ranked by attention score and enlarged for visualization.
Conclusion
TransFG is a fine-grained recognition framework that uses self-attention to capture discriminative regions and achieves state-of-the-art results across four common benchmarks. Its smaller selected patches, relational processing, contrastive loss, and visualizations support meaningful and interpretable classification.
- Self-attention captures the most discriminative regions, while selected patches are smaller and more meaningful than bounding boxes produced by other methods.
- Transformer layers model relationships among selected patches instead of classifying each patch separately.
- Contrastive loss increases the discriminative ability of the classification tokens.
- Qualitative visualizations show the interpretability of the method.
- The authors identify TransFG as a starting point for future transformer-based fine-grained recognition work.