PaperlayerView exact revision on arXiv

Source-linked AI summary

AXM-Net: Implicit Cross-Modal Feature Alignment for Person Re-identification

Ammarah Farooq, Muhammad Awais, Josef Kittler, Syed Safwan Khalid

arXiv:2101.08238v3cs.CVcs.LG

TL;DR

Cross-modal person Re-ID seeks to align person semantics across visual and textual representations, a challenge complicated by different modality structures and background information. AXM-Net learns implicit cross-modal alignment during feature extraction using a unified feature-learning block. It reports new SOTA performance on CUHK-PEDES and a wide margin on CrossRe-ID and CUHK-SYSU retrieval scenarios.

Abstract

from arXiv · show

Cross-modal person re-identification (Re-ID) is critical for modern video surveillance systems. The key challenge is to align cross-modality representations induced by the semantic information present for a person and ignore background information. This work presents a novel convolutional neural network (CNN) based architecture designed to learn semantically aligned cross-modal visual and textual representations. The underlying building block, named AXM-Block, is a unified multi-layer network that dynamically exploits the multi-scale knowledge from both modalities and re-calibrates each modality according to shared semantics. To complement the convolutional design, contextual attention is applied in the text branch to manipulate long-term dependencies. Moreover, we propose a unique design to enhance visual part-based feature coherence and locality information. Our framework is novel in its ability to implicitly learn aligned semantics between modalities during the feature learning stage. The unified feature learning effectively utilizes textual data as a super-annotation signal for visual representation learning and automatically rejects irrelevant information. The entire AXM-Net is trained end-to-end on CUHK-PEDES data. We report results on two tasks, person search and cross-modal Re-ID. The AXM-Net outperforms the current state-of-the-art (SOTA) methods and achieves 64.44\% Rank@1 on the CUHK-PEDES test set. It also outperforms its competitors by $>$10\% in cross-viewpoint text-to-image Re-ID scenarios on CrossRe-ID and CUHK-SYSU datasets.

Introduction

Cross-modal person Re-ID uses visual and textual information to support searching when an image query is unavailable. AXM-Net aims to align the modalities implicitly during feature learning while emphasizing person attributes and suppressing background information.

Introduction

  • Person search uses a natural-language description as the query, whereas person Re-ID uses an image query.Multimodal search can address cases in which an image of the person is unavailable.Source: Introduction · p. 1
  • Cross-modal alignment should associate image parts with textual phrases through shared person attributes while disregarding background noise.The paper identifies differing modality structures and the need to learn semantics rather than memorize training image-text identity pairs.Source: Introduction · p. 2
  • The alignment network combines multi-context intra-modal information with cross-modal attributes to boost informative concepts and suppress noisy or background information.AXM-Net uses multi-stream feature maps and an alignment network trained on semantics present in both modalities.Source: Introduction · p. 2
  • The proposed AXM-Block mutually learns visual and textual representations from person attributes, aligning semantics implicitly during feature learning.The approach is presented as the first to employ implicit semantic alignment across modalities at the feature-learning stage in Re-ID.Source: Introduction · pp. 1–2
  • AXM-Net outperforms current SOTA by 3% on CUHK-PEDES and by a wide margin on CrossRe-ID and CUHK-SYSU in all retrieval scenarios.The claimed advantage on CrossRe-ID and CUHK-SYSU applies across retrieval scenarios.Source: Introduction · p. 2
  • Visual part-based feature coherence enhances local spatial details while using context from other image parts.Textual contextual attention learns long-term associations among attributes that can recur across a description.Source: Introduction · p. 2

Related Work

Prior person-search and cross-modal Re-ID work explored global-local associations, auxiliary branches, and cross-modal matching. AXM-Net contrasts with approaches that encode modalities separately and align them afterward, instead learning latent alignment during feature extraction.

Related Work

  • Earlier person-search work progressed from global cross-modal features to global-local associations, feature-embedding improvements, and auxiliary branches using cues such as pose, attributes, segmentation, and body parts.Gao et al. (2021) used multi-scale visual features and cross-modal non-local attention; Zheng et al. (2020a) used global, phrase, and word-level matchings.Source: Related Work · p. 2
  • Prior methods generally extract visual and textual features with separate backbones before cross-modal alignment, often using auxiliary tasks; AXM-Net instead targets latent alignment during feature extraction.The related-work passage describes prior methods as mostly requiring auxiliary tasks to increase matching performance.Source: Related Work · p. 2

Cross-modal AXM-Net Framework

AXM-Net combines global visual, part-based visual, and textual features through unified cross-modal learning and modality-specific attention. Its training aligns the representations using identity and image-text matching losses.

Cross-modal AXM-Net Framework

Architecture of the AXM-Block

  • The AXM-Block pools multi-scale visual and textual maps, learns channel weights from their shared information, and sums the attended features.The block uses convolution filter scales r = 3, 5, 7, and 9.Source: Architecture of the AXM-Block · p. 3, Figure 3 · p. 4

Unified Feature Learning using AXM-Block

  • Stacked AXM-Blocks capture intra-modal local context and dynamically align coarser-to-finer semantics across modalities during feature learning.The text describes this cross-modal, cross-scale alignment as distinct from intra-modal multi-scale learning.Source: Unified Feature Learning using AXM-Block · p. 3
  • Text serves as a super-annotation signal without segmentation, landmark detection, or word/phrase matching, reducing visual-background effects.The passage attributes the reduced background effect to textual input having no background clutter.Source: Unified Feature Learning using AXM-Block · pp. 3–4

Visual Part-based Feature Coherence (VFC)

  • VFC applies shared strip-wise channel attention to link features with body-part locations, then fuses the attended strips into a visual feature.The shared network retains cross-part feature correspondence.Source: Visual Part-based Feature Coherence… · p. 4

Self-Attention for Text (TSA)

  • Text self-attention computes interactions among phrase-level features to model long-term dependencies and complement convolutional feature learning.It operates on feature maps from the final convolution block.Source: Self-Attention for Text (TSA) · p. 4

Objective Function

  • Training combines shared-classifier cross-entropy across three branches with pairwise margin-triplet and binary affinity losses on image-text matches.Affinity loss compares image-text similarity and distinguishes matching from non-matching pairs.Source: Objective Function · pp. 4–5

Experiments and the Results

AXM-Net sets a new CUHK-PEDES person-search Rank@1 result and outperforms the prior method across cross-modal Re-ID scenarios. Experiments also examine training, dataset protocols, and component choices.

Implementation Details

  • Training uses two stages, initially freezing the ImageNet-pretrained vision backbone; evaluation uses cosine similarity and reports Rank@1 and mAP.Text descriptions use 768-dimensional BERT embeddings, and the word-embedding layer remains fixed during training.Source: Implementation Details · p. 5

Datasets

  • CUHK-PEDES contains 13,003 identities, 40,206 images, and 80,440 descriptions; CrossRe-ID and CUHK-SYSU use separate evaluation protocols.CrossRe-ID has 824 identities; CUHK-SYSU uses 5,532 training and 2,099 test identities.Source: Datasets · p. 5

Comparison with State-of-the-Art Methods

  • 64.44% Rank@1 establishes AXM-Net’s new CUHK-PEDES state of the art using affinity and triplet losses together.With the joint identity loss alone, it exceeds 62% Rank@1.Source: Results on Person Search · p. 5, Table 2 · p. 6
  • AXM-Net outperforms the previous method in all retrieval scenarios on CrossRe-ID and CUHK-SYSU, including cross-pose text-to-image search.In T→V, descriptions and gallery images depict different poses; the authors connect Rank@1 gains to matching viewpoint-invariant semantic details.Source: Results on Cross-modal Re-ID · p. 5, Table 3 · p. 6

Component Analysis

  • Each component improves retrieval over the baseline; semantic alignment, VFC local attention, TSA textual dependencies, and richer language embeddings contribute to the reported gains.The component study uses joint cross-entropy loss on the CUHK-PEDES test set; the embedding comparison is reported separately.Source: Component Analysis · pp. 6–7
  • Removing the global visual branch causes a 4.58% performance drop, while single-stage training differs from the two-stage policy.The authors recommend first learning weakly aligned textual features to support best convergence.Source: Component Analysis · p. 7, Table 4 · p. 7
  • A shared MLP across image strips supports feature coherence and reduces parameters; global max pooling captures local cues, while tested feature dropping did not help.The feature-dropping tests used random batch-block or horizontal-strip dropping.Source: Component Analysis · p. 7, Table 5 · p. 7
  • Alignment at all network levels achieves the best performance; lower-level alignment adds some benefit despite less-defined semantic concepts.The authors attribute lower-block contextual information to the first strided 7×7 image convolution.Source: Component Analysis · p. 7, Table 6 · p. 7

Conclusion

AXM-Net combines implicit visual–textual semantic alignment with image-part feature coherence, without external cues for explicit alignment. The authors report new state-of-the-art performance on CUHK-PEDES and potential for more challenging cross-modal person Re-ID applications.

Conclusion

  • AXM-Net defines new state-of-the-art performance on the CUHK-PEDES benchmark.The conclusion also reports potential for more challenging cross-modal person Re-ID applications.Source: Conclusion · p. 7
  • AXM-Block implicitly aligns semantic features across visual and textual modalities through unified feature learning.Source: Conclusion · p. 7
  • The model reinforces feature coherence among image parts as a second design contribution.Source: Conclusion · p. 7
  • AXM-Net integrates cross-modal feature alignment and learning without external cues for explicit feature-embedding alignment.The authors describe it as the first such integrated framework in the Re-ID context.Source: Conclusion · p. 7
Loading 2101.08238v3…