Source-linked AI summary
Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification
Haowei Zhu, Wenjing Ke, Dong Li, Ji Liu, Lu Tian, Yi Shan
TL;DR
Fine-grained recognition must distinguish objects with subtle visual differences, but self-attention alone does not specifically reinforce local discriminative clues or regulate attention across image pairs. The paper proposes DCAL, combining global-local and pair-wise cross-attention with self-attention; it performs on par with state-of-the-art methods and consistently improves self-attention baselines across FGVC and Re-ID.
Problem
Fine-grained recognition seeks subtle feature embeddings for distinguishing visually similar bird species, person identities, and other object categories.
Method
DCAL combines GLCA for global-image and local-region interactions with PWCA, which regularizes attention by using another training image as a distractor.
Results
DCAL performs on par with state-of-the-art methods and consistently improves multiple self-attention baselines across FGVC and Re-ID benchmarks.
Takeaways & Limitations
The cross-attention design is easy to implement and compatible with different vision Transformer baselines.
Takeaways & Limitations
GLCA increases computation relative to self-attention, adding 9% Params and 2% FLOPs for DeiT-Tiny on CUB and 8% Params and 3% FLOPs for ViT-Base on VeRi-776.
Abstract
from arXiv · showhide
Recently, self-attention mechanisms have shown impressive performance in various NLP and CV tasks, which can help capture sequential characteristics and derive global information. In this work, we explore how to extend self-attention modules to better learn subtle feature embeddings for recognizing fine-grained objects, e.g., different bird species or person identities. To this end, we propose a dual cross-attention learning (DCAL) algorithm to coordinate with self-attention learning. First, we propose global-local cross-attention (GLCA) to enhance the interactions between global images and local high-response regions, which can help reinforce the spatial-wise discriminative clues for recognition. Second, we propose pair-wise cross-attention (PWCA) to establish the interactions between image pairs. PWCA can regularize the attention learning of an image by treating another image as distractor and will be removed during inference. We observe that DCAL can reduce misleading attentions and diffuse the attention response to discover more complementary parts for recognition. We conduct extensive evaluations on fine-grained visual categorization and object re-identification. Experiments demonstrate that DCAL performs on par with state-of-the-art methods and consistently improves multiple self-attention baselines, e.g., surpassing DeiT-Tiny and ViT-Base by 2.8% and 2.4% mAP on MSMT17, respectively.
1. Introduction
The paper extends self-attention with dual cross-attention modules for fine-grained recognition, combining global-local interactions with pair-wise attention regularization. DCAL improves self-attention baselines across FGVC and Re-ID benchmarks.
- Fine-grained recognition requires subtle feature embeddings because visual differences among sub-classes are small.
- GLCA enhances interactions between global images and local high-response regions to reinforce spatial-wise discriminative clues.
- PWCA regularizes an image’s attention learning by treating another image as a distractor and diffusing attention toward complementary regions.
- PWCA is used only during training, so it introduces no extra computation cost during inference.
- 2.8% and 2.4% mAP improvements over DeiT-Tiny and ViT-Base, respectively, are reported on MSMT17.
- DCAL performs on par with state-of-the-art methods and consistently improves multiple self-attention baselines across FGVC and Re-ID.
2. Related Work
The related work frames self-attention as a global-dependency mechanism adapted from NLP to vision, while highlighting the challenges of fine-grained categorization and re-identification. Existing approaches use CNNs, metric learning, local parts, and self-attention-based token selection or feature fusion.
- Self-attention relates positions within a sequence and draws global dependencies, originating in NLP before adoption in vision tasks.
- 2.2. Fine-Grained Visual Categorization: Fine-grained visual categorization identifies highly confused categories with fine differences.
- 2.2. Fine-Grained Visual Categorization: Prior FGVC methods mine multi-level features, use multi-granularity training, localize discriminative parts, or model pair-wise feature interactions.
- 2.2. Fine-Grained Visual Categorization: Transformer-based FGVC methods use multi-level feature fusion or patch-token selection, whereas the proposed approach adds two cross-attention modules beyond self-attention.
- Object re-identification distinguishes person or vehicle identities with subtle inter-class differences, commonly using CNNs, metric learning, and discriminative part-level features.
3. Proposed Approach
The proposed approach extends self-attention with global-local and pair-wise cross-attention to learn discriminative embeddings for fine-grained recognition. It selects high-response local regions for global interaction and uses paired images during training to regularize attention.
- Overview: DCAL stacks self-attention, global-local cross-attention, and pair-wise cross-attention modules in one network.The architecture uses L self-attention, M global-local cross-attention, and T pair-wise cross-attention modules.
- Revisit Self-Attention: Self-attention maps queries to weighted combinations of value vectors using scaled query-key similarities and softmax normalization.Queries, keys, and values are derived from the same input embedding through separate linear transformations; class and patch embeddings interact through attention.
- Global-Local Cross-Attention: GLCA selects high-response local queries using attention rollout, then attends to global key-value pairs to reinforce spatially discriminative clues.Attention rollout incorporates residual connections and identifies regions with high accumulated responses from the class embedding.
- Pair-Wise Cross-Attention: PWCA concatenates key-value matrices from two training images, making each target query attend to both its own and the paired image’s features.The jointly normalized attention scores introduce distractor information that increases training difficulty and reduces overfitting to sample-specific features.
- Pair-Wise Cross-Attention: PWCA is used only during training and removed during inference, so it adds no extra inference computation cost.The method uses T = 12 PWCA blocks during training.
4. Experiments
Experiments evaluate DCAL across fine-grained categorization and object re-identification benchmarks, comparing Transformer baselines, components, attention behavior, and computational costs. DCAL achieves competitive results, improves multiple baselines, and alters attention toward complementary discriminative regions.
- Experimental setting: Experiments cover three FGVC benchmarks and four Re-ID benchmarks using DeiT and ViT self-attention baselines.The evaluated architectures include DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Base, and R50-ViT-Base.
- Fine-grained visual categorization: DCAL reaches 92.0%, 95.3% and 93.3% top-1 accuracy on CUB-200-2011, Stanford Cars and FGVC-Aircraft with R50-ViT-Base.It also improves DeiT-Tiny by 2.2% and R50-ViT-Base by 1.3% on Stanford Cars.
- Object re-identification: 80.2%, 64.0%, 87.5%, 80.1% mAP is achieved on VeRi-776, MSMT17, Market1501 and DukeMTMC with ViT-Base.On MSMT17, DCAL surpasses DeiT-Tiny by 2.8% and ViT-Base by 2.4% mAP.
- Qualitative analysis: GLCA reduces misleading attention, while PWCA diffuses attention responses toward more complementary object parts.PWCA assigns dominant attention to the target image rather than the distractor image.
- Limitations: GLCA increases computational cost by 9% Params and 2% FLOPs for DeiT-Tiny on CUB, and by 8% Params and 3% FLOPs for ViT-Base on VeRi-776.Joint cross-attention training also takes longer than the self-attention baseline: 3.8 versus 2.1 hours on CUB.
5. Conclusion
The paper introduces GLCA and PWCA to improve fine-grained feature learning, combining global-local and pair-wise interactions with self-attention. Across seven benchmarks, DCAL is effective on FGVC and Re-ID and compatible with vision Transformer baselines.
- GLCA models interactions between global images and local regions to reinforce spatial-wise discriminative clues.
- PWCA establishes interactions between image pairs and serves as a regularization strategy to alleviate overfitting.
- DCAL is easy to implement and compatible with different vision Transformer baselines.
- Experiments on seven benchmarks demonstrate DCAL's effectiveness on fine-grained visual categorization and object re-identification.
A. Overview
The supplementary material provides additional experiments and analyses covering inference architectures, local-query selection, attention visualizations, and more Transformer baselines.
- The supplementary material presents more experimental results and analysis.
- It tests different inference architectures.
- It provides an ablation study on the ratio used for local-query selection.
- It shows additional visualizations of generated attention maps across different benchmarks.
- It evaluates the method on more Transformer baselines.
B. Different Inference Architectures
The default inference architecture removes PWCA and uses SA with GLCA, combining their outputs differently for FGVC and Re-ID. Two alternative architectures use only SA or only GLCA for inference.
- The default inference architecture removes all PWCA modules and retains only SA and GLCA modules.
- FGVC: For FGVC, classifiers' SA and GLCA class probabilities are added for prediction.
- Re-ID: For Re-ID, the final SA and GLCA class tokens are concatenated as the prediction feature.
- The study also tests SA-only and GLCA-only inference architectures.
C. Ablation Study on Effect of R
Different ratios for selecting high-response regions as local queries produce similar performance. The method therefore uses task-specific default ratios for FGVC and Re-ID.
- Different local-query selection ratios obtain similar performance.
- FGVC: The default ratio is R = 10% for all FGVC benchmarks.
- Re-ID: The default ratio is R = 30% for all Re-ID benchmarks.
D. More Visualization Results
Visualization comparisons across FGVC and Re-ID show that DCAL reduces misleading attention and spreads responses across complementary recognition parts.
- DCAL reduces misleading attentions in attention maps across multiple FGVC and Re-ID benchmarks.The visualizations compare self-attention with the proposed cross-attention method.
- DCAL diffuses attention responses to discover more complementary parts for recognition.
E. More Transformer Baselines
DCAL improves performance across additional CaiT and Swin Transformer baselines on CUB and MSMT. For Swin Transformer, only PWCA is applied because windowed attention already provides locality.
- 0.9 percentage points: DCAL improves Swin-T from 84.9% to 85.8% top-1 accuracy on CUB.
- 1.0 percentage point: DCAL improves Swin-T from 55.7% to 56.7% mAP on MSMT.
- For Swin Transformer, DCAL applies only PWCA because locality is already incorporated through windows.