Source-linked AI summary
Identity-Aware Textual-Visual Matching with Latent Co-attention
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, Xiaogang Wang
TL;DR
Textual-visual matching needs to measure correspondence between sentence descriptions and images, but existing methods often underuse identity-level annotations. The paper proposes a two-stage identity-aware CNN-LSTM framework using CMCE embedding and latent co-attention refinement, and reports large-margin gains over state-of-the-art methods on three datasets.
Problem
Most existing textual-visual matching methods do not effectively utilize identity-level annotations, despite their availability in benchmark datasets.
Method
A stage-1 CNN-LSTM uses CMCE with feature buffers for identity-aware embedding and screening, while a stage-2 CNN-LSTM uses spatial and latent semantic co-attention for refinement.
Results
The framework outperforms state-of-the-art approaches by large margins on three datasets with identity-level annotations.
Takeaways & Limitations
Identity-level annotations can support more discriminative cross-modal representations, while latent co-attention aligns image regions and sentence concepts across wording variations.
Takeaways & Limitations
CMCE-generated textual and visual features are not tightly coupled, so stage-2 refinement is essential for accurate matching results.
Abstract
from arXiv · showhide
Textual-visual matching aims at measuring similarities between sentence descriptions and images. Most existing methods tackle this problem without effectively utilizing identity-level annotations. In this paper, we propose an identity-aware two-stage framework for the textual-visual matching problem. Our stage-1 CNN-LSTM network learns to embed cross-modal features with a novel Cross-Modal Cross-Entropy (CMCE) loss. The stage-1 network is able to efficiently screen easy incorrect matchings and also provide initial training point for the stage-2 training. The stage-2 CNN-LSTM network refines the matching results with a latent co-attention mechanism. The spatial attention relates each word with corresponding image regions while the latent semantic attention aligns different sentence structures to make the matching results more robust to sentence structure variations. Extensive experiments on three datasets with identity-level annotations show that our framework outperforms state-of-the-art approaches by large margins.
1. Introduction
Textual-visual matching is challenging because language-image relations are nonlinear and images with similar descriptions can vary substantially, while identity annotations are often unused. The paper addresses this gap with a two-stage identity-aware framework combining CMCE-based embedding and latent co-attention refinement.
- Textual-visual matching connects natural-language descriptions with images for applications including text-image embedding, zero-shot learning, and visual question answering.
- The task is difficult because language-image relations are highly nonlinear and visual appearances can vary substantially, including subtle differences among images with similar descriptions.
- Most existing textual-visual matching methods ignore identity-level annotations available in benchmark datasets.
- The proposed two-stage framework uses a CNN-LSTM with CMCE loss for identity-aware representation learning, then latent co-attention to align image regions and latent semantic concepts.
- Stage 1 screens easy incorrect matchings and initializes stage-2 training, while stage 2 addresses the incomplete coupling of textual and visual features produced by CMCE.
- Latent co-attention combines word-specific spatial attention with semantic alignment across sentence structures to reduce sensitivity to wording variations.
2. Related Work
Related work developed identity-aware visual matching and image-language methods, but identity annotations remained underused in textual-visual matching. The paper proposes a two-stage approach to address these limitations and reports large-margin improvements on three datasets.
- Identity-aware visual matching commonly uses either simultaneous identity classification or pairwise and triplet distance losses, each with stated limitations.
- Image-language similarity methods support applications including image captioning, visual question answering, and text-image embedding.
- Identity-level annotations are widely used in person re-identification and face recognition, but textual-visual matching methods seldom exploit them.
- Prior work applied identity annotations to CUB, Oxford-102 Flowers, and person re-identification image-description matching, but faced problems shared with existing visual matching methods.
- The proposed two-stage framework targets efficient textual and visual feature learning and outperforms the cited state-of-the-art methods by large margins on three datasets.
3. Identity-Aware Textual-Visual Matching with Latent Co-attention
The framework uses identity-aware cross-modal learning in stage 1 and latent co-attention refinement in stage 2. CMCE compares sampled features against feature buffers, while spatial and semantic attention establish word-region correspondences and handle sentence-structure variation.
- Stage 1: The stage-1 CNN-LSTM maps images and descriptions into a joint embedding space, bringing same-identity features closer and different-identity features farther apart.It uses a specialized CMCE loss to exploit identity-level annotations.
- Stage 1: The stage-1 network covers hard negatives each training epoch and supports O(N) evaluation-time sampling of test samples.This contrasts with pairwise or triplet approaches whose sampling and feature calculation can scale quadratically with the number of identities.
- Stage 1: CMCE compares sampled features from one modality with all identities stored in the other modality’s feature buffer.Textual and visual buffers enable cross-modal affinity calculations beyond the current minibatch.
- Stage 2: Stage 1 efficiently screens easy incorrect matchings but compresses sentences into single vectors, leaving word-region correspondences insufficiently modeled.The paper therefore uses stage 2 to refine the initial matching results.
- Stage 2: Stage 2 uses spatial attention to relate each word to relevant image regions and latent semantic attention to align word-image features across sentence structures.A decoder LSTM uses the aligned features to produce the final matching confidence.
4. Experiments
Experiments evaluate the identity-aware two-stage framework on CUHK-PEDES, CUB, and Flower, using retrieval metrics, ablations, and qualitative examples. The results show strong performance gains from CMCE, identity annotations, latent co-attention, and stage-1 screening.
- Datasets: The framework is evaluated on CUHK-PEDES, CUB, and Flower datasets with identity-level annotations.CUHK-PEDES supports person search, while CUB and Flower provide multiple descriptions per image and non-overlapping train/test classes.
- CUHK-PEDES results: On CUHK-PEDES, the method outperforms state-of-the-art methods by large margins in top-1 and top-10 text-to-image retrieval accuracy.The reported evaluation focuses on text-to-image retrieval because image-to-text retrieval is not a practical setting for this dataset.
- CUHK-PEDES results: A 2.50% top-1 gain over GNA-RNN and a 21.55% top-1 accuracy demonstrate the effectiveness and efficiency of the CMCE-based stage-1 model.CMCE also reaches 21.55% top-1 accuracy versus 10.48% for Word CNN-RNN, while feature buffers improve efficiency with many identities.
- CUHK-PEDES results: The stage-2 CNN-LSTM with latent co-attention further improves top-1 and top-10 accuracies by 4.39% and 5.70%, respectively.Its co-attention aligns visual regions with latent semantic concepts and focuses training on hard samples after stage-1 screening.
- Ablation studies: Removing identity annotations reduces Stage-2 top-1 and top-10 accuracies by 2.47% and 5.71%, while replacing CMCE with triplet loss lowers stage-1 top-1 by 6.79%.Triplet loss also requires three times more training time, and removing latent semantic or spatial attention decreases retrieval performance.
- CUB and Flower results: On CUB and Flower, the method exceeds CNN-RNN by more than 3% in top-1 image-to-text accuracy and about 10% in text-to-image AP@50.Qualitative retrievals show correct identity matching across different images, including cases with subtle visual differences.
5. Conclusion
The paper concludes that its identity-aware two-stage framework combines CMCE-based embedding with latent co-attention for visual-semantic matching. Experiments on three datasets and ablations report large-margin gains over state-of-the-art approaches.
- The framework uses a stage-1 CNN-LSTM with CMCE loss to embed images and descriptions in a shared feature space while minimizing intra-identity distance.Stage 1 also screens incorrect matchings and supplies training and evaluation samples for stage 2.
- The stage-2 CNN-LSTM jointly learns spatial and latent semantic attention through an alignment decoder LSTM.This mechanism aligns words with image regions and reduces the impact of sentence-structure variations.
- The method was evaluated on three datasets with ablation studies examining the effect of each component.
- The framework outperformed state-of-the-art approaches by a large margin for identity-aware visual-textual matching.