Source-linked AI summary
Feature Fusion Vision Transformer for Fine-Grained Visual Categorization
Jun Wang, Xiaohan Yu, Yongsheng Gao
TL;DR
FGVC requires subtle discriminative features, but existing localization and attention approaches can be costly or overemphasize object-dominated regions. FFVT addresses this with multilevel token fusion and parameter-free MAWS, achieving state-of-the-art performance across four FGVC benchmarks, with reported backbone gains on three datasets.
Problem
FGVC must distinguish subordinate categories with subtle differences, while prior methods face annotation, computational, or feature-coverage limitations.
Method
FFVT fuses local, low-level, middle-level, and high-level token information, while MAWS selects informative tokens using mutual attention without extra parameters.
Results
FFVT achieves state-of-the-art performance on four FGVC benchmarks and improves the backbone by 5.42%, 4.67% and 0.80% on CottonCultivar80, SoyCultivarLocal and CUB, respectively.
Takeaways & Limitations
FFVT enriches transformer classification with multilevel local information and provides an alternative to CNN-based FGVC approaches.
Takeaways & Limitations
Localization-based approaches can produce inaccurate or oversized boxes that cover background and miss discriminative regions that are not rectangular.
Abstract
from arXiv · showhide
The core for tackling the fine-grained visual categorization (FGVC) is to learn subtle yet discriminative features. Most previous works achieve this by explicitly selecting the discriminative parts or integrating the attention mechanism via CNN-based approaches.However, these methods enhance the computational complexity and make the modeldominated by the regions containing the most of the objects. Recently, vision trans-former (ViT) has achieved SOTA performance on general image recognition tasks. Theself-attention mechanism aggregates and weights the information from all patches to the classification token, making it perfectly suitable for FGVC. Nonetheless, the classifi-cation token in the deep layer pays more attention to the global information, lacking the local and low-level features that are essential for FGVC. In this work, we proposea novel pure transformer-based framework Feature Fusion Vision Transformer (FFVT)where we aggregate the important tokens from each transformer layer to compensate thelocal, low-level and middle-level information. We design a novel token selection mod-ule called mutual attention weight selection (MAWS) to guide the network effectively and efficiently towards selecting discriminative tokens without introducing extra param-eters. We verify the effectiveness of FFVT on three benchmarks where FFVT achieves the state-of-the-art performance.
1 Introduction
FGVC distinguishes subordinate categories with subtle inter-class and large intra-class variation, motivating methods that capture discriminative local features efficiently. FFVT fuses information across transformer layers and uses MAWS to select informative tokens, achieving state-of-the-art results on four benchmarks.
- Motivation: FGVC differentiates subordinate categories within basic-level classes and is challenging because of small inter-class and large intra-class variance.The task has applications including autonomous driving and intelligent agriculture.
- Motivation: Earlier localization-based methods require costly annotations or region proposals, which can cover background and miss non-rectangular discriminative regions.The passage specifically identifies leaf veins as features that cannot be simply annotated by rectangles.
- Method: FFVT aggregates local information from low-, middle-, and high-level tokens to enrich the classification representation.The framework is proposed as a pure transformer approach for FGVC.
- Method: MAWS selects informative tokens with high mutual similarity to the class token in both class-token and token contexts without extra parameters.This selection guides which tokens from each layer are added to the last transformer layer.
- Results: FFVT achieves state-of-the-art performance on four fine-grained benchmarks.The benchmarks include widely used FGVC datasets and challenging small-scale ultra-fine-grained datasets.
- Results: The proposed method improves the backbone by 5.42%, 4.67% and 0.80% on CottonCultivar80, SoyCultivarLocal and CUB, respectively.These are reported ablation-study improvements.
2 Related Works
Related FGVC methods localize or attend to discriminative regions, while transformer-based approaches offer a pure attention mechanism but remain relatively underexplored for FGVC. FFVT differs from TransFG by fusing multilevel features and using a simpler token-selection strategy.
- FGVC Methods: FGVC methods are broadly divided into localization-based and attention-based approaches.Localization methods classify detected foreground or discriminative regions, whereas attention methods use self-attention to localize them.
- Localization-Based Methods: Localization-based methods reduce manual annotation through weak supervision but may still rely on detection branches to identify discriminative regions.The related work describes weakly supervised object detection and instance segmentation alternatives.
- Attention-Based Methods: Attention-based methods avoid extra region annotations but often suffer from high computational cost.They use mechanisms such as diversified, two-level, or progressive attention to collect discriminative information.
- Vision Transformers: ViT achieves state-of-the-art image-classification performance with pure transformers, but relatively few studies apply vision transformers to FGVC.This motivates extending transformer architectures to fine-grained categorization.
- Comparison with TransFG: Compared with TransFG, FFVT fuses local and multilevel information, while TransFG sends selected tokens directly to the final transformer layer.FFVT’s MAWS uses attention from one transformer layer, unlike TransFG’s use of all layers and matrix multiplication.
3 Methods
FFVT extends ViT for FGVC by fusing selected tokens from multiple transformer layers before the final layer. Its MAWS module selects informative tokens using mutual attention without extra parameters.
- ViT For Image Recognition: ViT splits images into fixed-size patches, adds position embeddings and a class token, then processes them through Transformer Encoder blocks for classification.Patch embeddings are projected into a latent space, while the final class token is passed to an MLP classification head.
- FFVT Architecture: FFVT fuses low-level, middle-level and high-level tokens because ViT may not capture enough local information for FGVC.The fused representation is intended to enrich local information and feature representation capability.
- Feature Fusion Module: Before the last transformer layer, FFVT replaces non-class-token inputs with tokens selected from previous layers, enabling the final class token to interact with multilevel features.The selected tokens are combined with the classification token and fed into the final transformer layer.
- Mutual Attention Weight Selection Module: MAWS introduces no extra learning parameters and is described as simpler and more efficient than the matrix multiplication used in TransFG.The method selects K important tokens after averaging multi-head attention scores.
- Mutual Attention Weight Selection Module: MAWS selects important tokens using attention scores between the class token and other tokens in both attention contexts.The method averages attention scores across heads and collects token indexes according to mutual attention values.
- Mutual Attention Weight Selection Module: Unlike single attention weight selection, MAWS addresses noise from tokens that receive high class-token attention while aggregating information from noisy patches.The selected tokens are required to be similar to the classification token in the classification-token context and in the tokens' own contexts.
4 Experiments
FFVT is evaluated on four fine-grained benchmarks, including challenging ultra-fine-grained datasets, and achieves strong accuracy with evidence from ablations supporting feature fusion and MAWS.
- Benchmark comparisons: FFVT achieves 91.6% accuracy on CUB, 0.1% below TransFG, while using less computation cost and GPU memory.The comparison is affected by TransFG using a batch size twice as large on CUB.
- Benchmark comparisons: 91.5% accuracy on Stanford Dogs makes FFVT the best listed method, exceeding TransFG by 0.9%.
- Ultra-fine-grained benchmarks: 57.92% accuracy on CottonCultivar80 is 4.17% above the second-best method, while SoyCultivarLocal reaches 44.17% and beats all listed methods.These datasets contain super-subtle inter-class differences and only three training images per category.
- Ablation studies: MAWS improves the backbone by +6.67%, +4.84% and +0.80% on CottonCultivar80, SoyCultivarLocal and CUB, respectively.The results indicate that MAWS better exploits attention information by selecting informative tokens.
- Hyper-parameter analysis: FFVT performs best with K=12 tokens per layer on SoyCultivarLocal, increasing accuracy from 43.17% at K=10 to 44.17%.Accuracy decreases to 42.5% at K=14, while all tested K values exceed the ViT backbone accuracy of 39.33%.
5 Conclusion
The paper proposes FFVT to enrich transformer classification with local and multi-level information, using MAWS to select discriminative tokens. Experiments report state-of-the-art performance across four FGVC benchmarks and effectiveness across normal-scale, small-scale, and ultra-fine-grained settings.
- FFVT enriches the classification token with local, low-level and middle-level information through feature fusion.
- MAWS selects discriminative tokens using self-attention scores to support feature aggregation.
- FFVT achieves state-of-the-art performance on four benchmarks and improves standard ViT across different fine-grained settings.
- The reported results show particular effectiveness on challenging datasets involving subtle differences and discriminative information.