PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification

Zefeng Ding, Changxing Ding, Zhiyin Shao, Dacheng Tao

arXiv:2107.12666v2cs.CV

TL;DR

Text-to-image ReID must bridge a substantial modality gap and large variation among descriptions. SSAN addresses this with automatically aligned part features, body-part relation modeling, Compound Ranking loss, and ICFG-PEDES, and it outperforms existing approaches by significant margins.

  • Problem

    Text-to-image ReID remains challenging because of the significant modality gap and large intra-class variance in textual descriptions.

  • Method

    SSAN automatically extracts aligned visual and textual part features, models body-part relationships, uses Compound Ranking loss, and introduces the ICFG-PEDES database.

  • Results

    SSAN outperforms existing approaches by large margins in experiments on ICFG-PEDES and CUHK-PEDES.

  • Takeaways & Limitations

    SSAN provides an efficient, self-aligned approach for text-to-image ReID, while ICFG-PEDES supports further research with identity-centric and fine-grained descriptions.

Abstract

from arXiv · show

Text-to-image person re-identification (ReID) aims to search for images containing a person of interest using textual descriptions. However, due to the significant modality gap and the large intra-class variance in textual descriptions, text-to-image ReID remains a challenging problem. Accordingly, in this paper, we propose a Semantically Self-Aligned Network (SSAN) to handle the above problems. First, we propose a novel method that automatically extracts semantically aligned part-level features from the two modalities. Second, we design a multi-view non-local network that captures the relationships between body parts, thereby establishing better correspondences between body parts and noun phrases. Third, we introduce a Compound Ranking (CR) loss that makes use of textual descriptions for other images of the same identity to provide extra supervision, thereby effectively reducing the intra-class variance in textual features. Finally, to expedite future research in text-to-image ReID, we build a new database named ICFG-PEDES. Extensive experiments demonstrate that SSAN outperforms state-of-the-art approaches by significant margins. Both the new ICFG-PEDES database and the SSAN code are available at https://github.com/zifyloo/SSAN.

I. INTRODUCTION

Text-to-image ReID is challenging because free-form descriptions vary substantially and describe body parts in arbitrary orders. SSAN addresses these issues with self-aligned part features, body-part relation modeling, compound ranking supervision, and the ICFG-PEDES database.

  • Text-to-image ReID searches for target-person images using natural-language descriptions, which contain more diverse and fine-grained visual information than predefined attributes.
  • Descriptions of the same image may vary dramatically, creating large intra-class variance in textual features.
  • Body parts may appear in arbitrary orders and be described with varying numbers of words, complicating semantically aligned part-feature extraction.
  • SSAN extracts aligned visual and textual part features without splitting descriptions or performing pair-specific cross-modal operations.It uses image body-part alignment as supervision and contextual language cues to infer correspondences.
  • A multi-view non-local network models relationships between body parts, while Compound Ranking loss uses descriptions of same-identity images as additional weak supervision.
  • ICFG-PEDES provides identity-centric, fine-grained descriptions, more challenging images, and 36% more images than CUHK-PEDES.Its captions average 58% more words than CUHK-PEDES, and the database is intended to expedite text-to-image ReID research.
  • Extensive experiments on ICFG-PEDES and CUHK-PEDES show that SSAN outperforms existing approaches by large margins and offers efficiency and ease-of-use advantages.

A. Image-text Retrieval

Image-text retrieval aligns visual and textual representations, while text-to-image ReID requires especially fine-grained region–word or region–phrase correspondence. SSAN builds on part-based visual representations while targeting efficient textual part-feature extraction.

  • Image-text retrieval methods project holistic images and descriptions into shared spaces or establish region–word correspondences using cross-modal and intra-modal relationships.
  • Text-to-image ReID is particularly challenging because its fine-grained alignment must connect visual regions with words or phrases.
  • Optimization-based prior methods use ranking, adversarial, or instance losses, but do not explicitly solve large intra-class variance in textual features.
  • Part-based image ReID uses fine-grained representations, commonly obtained by uniformly partitioning CNN feature maps because pedestrian body parts are roughly aligned after detection.
  • Alternative visual methods detect body parts with external tools or attention mechanisms, often requiring more complex architectures or training strategies.
  • SSAN adopts uniform visual partitioning and combines ResNet-50 visual extraction with Bi-LSTM processing of word embeddings.

B. Global Feature Extraction

SSAN projects global visual and textual representations into a common space using pooled features and a shared 1 × 1 convolution. Weight sharing is intended to tighten semantic alignment between modalities.

  • SSAN obtains global visual and textual features by applying Global Max Pooling to visual maps and Row-wise Max Pooling to textual representations.
  • A shared 1 × 1 Conv layer projects both pooled representations into a common feature space.The resulting global visual and textual features are represented as v_g and t_g in R^M.
  • Weight sharing on the global projection layer encourages tighter semantic alignment between visual and textual features than prior methods.
  • The SSAN architecture combines global and part branches, with each part branch containing Part-specific Feature Learning and Part Relation Learning modules.

C. Part-level Feature Extraction

SSAN extracts semantically aligned part-level visual and textual features without splitting descriptions or using external tools. Its WAM uses aligned visual body parts and contextual language cues to infer word–part correspondences.

  • SSAN adds part branches containing Part-specific Feature Learning and Part Relation Learning modules to extract aligned visual and textual representations.
  • External noun-phrase splitting can break context, making phrases such as “white line” ambiguous across clothing items and accessories.
  • WAM predicts each word’s probability of belonging to each visual body part using contextual textual features and aligned visual supervision.
  • Each part branch applies global max pooling and a shared 1 × 1 convolution to visual feature regions and recurrently processed textual features.
  • Shared projections encourage visual and textual part features to be discriminative and similar, allowing semantically aligned features without external tools.
  • Part-level visual and textual similarity is computed after concatenating the K corresponding part features.

2) Part Relation Learning:

MV-NLN captures relationships among body parts to address cases where phrases span multiple regions or describe interactions between parts. It processes visual and textual part features with shared parameters and compares their concatenated outputs.

  • Equal partitioning can be suboptimal because one phrase may cover multiple body parts, while descriptions may encode relationships such as “holding a bag.”
  • MV-NLN computes each visual part’s similarity with the other K−1 parts in a shared embedding space through multi-view projections.
  • Interaction strengths αki aggregate information from the other K−1 part features to produce relation-enhanced visual part representations.
  • The same MV-NLN process captures correlations in textual part features, sharing parameters across visual and textual modalities.
  • Cosine similarity compares concatenated visual and textual features produced by MV-NLN for an image–text pair.

IV. OPTIMIZATION

The optimization introduces Compound Ranking loss, combining exact and same-identity cross-image image-text pairs to address overfitting and variable descriptive power. Adaptive margins further accommodate variation in weak annotations.

  • Compound Ranking loss: The popular ranking loss uses only matching image-text pairs, which may create a risk of overfitting.The CR loss adds weakly supervised terms to address this training limitation.
  • Compound Ranking loss: Compound Ranking loss combines strong exact-match pairs with weak pairs using an image and another same-identity image’s description.This exposes each training image to more diverse textual descriptions as a data augmentation strategy.
  • Compound Ranking loss: Weak supervision is motivated by the observation that descriptions can approximately annotate other images of the same identity, but their descriptive power varies dramatically.The variation depends on text quality and appearance differences between images.
  • Adaptive margins: Adaptive margins are proposed because similarity for same-identity images varies, making a fixed margin potentially suboptimal.The adaptive strategy adjusts α2 for weak supervision terms.
  • Feature optimization: The optimization applies CR loss and ID loss to global, PFL-part, and PRL-part features, with weights 1, 0.5, and 0.5.ID loss is imposed on each of K part features, whereas CR loss uses concatenated K part features.
  • Inference: At testing, the overall image-text similarity score is the sum of Sg, Sl, and Sn.These scores are combined into one pairwise similarity measure.

V. EXPERIMENTS

Experiments evaluate text-to-image ReID on CUHK-PEDES and the newly constructed ICFG-PEDES database using Rank-1, Rank-5, and Rank-10 accuracy. ICFG-PEDES addresses limited database availability and emphasizes identity-centric, fine-grained descriptions.

  • Experimental setting: Experiments use CUHK-PEDES and ICFG-PEDES, evaluating both databases with Rank-1, Rank-5, and Rank-10 accuracies.These metrics follow the evaluation protocol adopted in prior work.
  • CUHK-PEDES: CUHK-PEDES contains 40,206 images and 80,412 descriptions covering 13,003 identities, with two captions per image.Descriptions average 23.5 words, and the official split includes training, validation, and test identities.
  • Motivation for ICFG-PEDES: A single large-scale database makes reliable verification of text-to-image ReID effectiveness difficult.This motivates constructing an additional benchmark.
  • ICFG-PEDES: ICFG-PEDES contains 54,522 images from 4,102 identities, with one caption per image averaging 37.2 words.Its descriptions are more identity-centric and fine-grained than those in CUHK-PEDES.

A. Implementation Details

Implementation uses standardized image preprocessing, established visual backbones, and fixed training settings. The ablation baseline removes all part branches and CR loss, retaining only global features.

  • Preprocessing and backbones: Images are resized to 384 × 128 pixels and randomly horizontally flipped for augmentation.VGG-16 and ResNet-50, both ImageNet-pretrained, serve as visual backbones.
  • Text processing: The text vocabulary is built by counting unique words in the training set.This follows the setup used by previous methods.
  • Training settings: Training uses Adam with batch size 64 for 60 epochs, an initial learning rate of 0.001, and empirically set values 512 and 0.1 for the specified parameters.The passage identifies these as the training and selected hyperparameter settings.
  • Baseline: The ablation baseline removes all part branches and CR loss, extracting only global features and optimizing with cross-entropy and ordinary ranking losses.This baseline isolates the contribution of SSAN’s part branches and CR loss.
  • Ablation design: Ablations analyze PFL, PRL, and CR loss on both databases, with results summarized in Table I.The study evaluates each key component of SSAN.

1) Effectiveness of PFL:

Ablation results show that PFL provides the largest reported component gains, while PRL and CR loss add further improvements. Adaptive margins outperform fixed margins on both databases.

  • Effectiveness of PFL: 4.58% and 3.56% Rank-1 gains result from adding PFL to the baseline on CUHK-PEDES and ICFG-PEDES, respectively.These results support PFL’s effectiveness for automatically learning semantically aligned part-level features.
  • Effectiveness of PFL: PFL outperforms SCAN and ViTAA by considerable margins, while WAM-based ViTAA* significantly outperforms ViTAA in Rank-1 accuracy.WAM predicts correspondences for all words using contextual cues, rather than nouns only.
  • Effectiveness of PRL: 1.33% and 0.95% Rank-1 improvements result from adding PRL to PFL on CUHK-PEDES and ICFG-PEDES, respectively.Removing MV-NLN layers except the final 1 × 1 Conv yields limited improvements, attributing gains to part-relation modeling.
  • Effectiveness of CR loss: 1.62% and 1.21% Rank-1 gains result from adding CR loss to the baseline on CUHK-PEDES and ICFG-PEDES, respectively.Adding CR loss to PFL and MV-NLN further improves Rank-1 by 0.78% and 0.7% on the two databases.
  • Adaptive margins: Adaptive margins consistently outperform fixed margins for CR-loss weak supervision terms on both databases.They are more flexible to variation in the descriptive power of rough annotations.

1) Performance Comparisons on CUHK-PEDES:

SSAN consistently outperforms prior text-to-image ReID methods on standard, new-dataset, and cross-domain evaluations, while avoiding manual or external part-text extraction.

  • The baseline with weight sharing outperforms ViTAA, whereas removing weight sharing yields 53.12% Rank-1 accuracy with ResNet-50.
  • SSAN outperforms ViTAA by 5.4% in Rank-1 accuracy on CUHK-PEDES with the same input image size and backbones.
  • SSAN automatically extracts semantically aligned part features, unlike methods that typically rely on external tools for part-level textual features.
  • SSAN beats SCAN by 4.18% and ViTAA* by 3.25% in Rank-1 accuracy on ICFG-PEDES, with all methods using ResNet-50.
  • SSAN outperforms SCAN by 4.3%, 6.9%, 11.2%, and 12.3% in Rank-1 accuracy across four source-only cross-domain transfer tasks.
  • SSAN exceeds MAN by 6.0%, 6.0%, 2.1%, and 12.3% in Rank-1 accuracy across the four source-and-target cross-domain transfer tasks.

D. Qualitative Results

Qualitative results indicate that WAM aligns words with appropriate body parts, including objects whose positions vary across descriptions, using visual supervision and textual context.

  • WAM uses relatively well-aligned visual body parts as supervision and contextual cues in descriptions to infer word-part correspondences.
  • The qualitative evaluation covers WAM predictions for fixed body parts and objects with flexible positions.
  • WAM correctly predicts words corresponding to fixed body parts, such as shirt, jumpsuit, and dress, across appropriate part regions.
  • WAM assigns “bag” to different body-part regions depending on whether the description places it over the shoulder or in the hand.
Loading 2107.12666v2…