PaperlayerView exact revision on arXiv

Source-linked AI summary

Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search

Shuanglin Yan, Hao Tang, Liyan Zhang, Jinhui Tang

arXiv:2208.14365v2cs.CV

TL;DR

TBPS must match text descriptions with pedestrian images despite unreliable explicit local parts and image-specific background or environmental information. MANet suppresses those image-specific factors and learns global and implicit local alignment through modality-shared semantic centers. It outperforms existing methods across the reported CUHK-PEDES and ICFG-PEDES evaluations.

  • Problem

    TBPS requires cross-modal alignment, but explicit local parts can be ambiguous or noisy and image-specific information creates inequality between image and text modalities.

  • Method

    MANet suppresses image background and environmental factors, then combines global alignment with implicit local alignment using shared semantic topic centers.

  • Results

    MANet significantly outperforms existing methods on two databases, achieving new state-of-the-art performance on CUHK-PEDES and surpassing LGUR across all reported ICFG-PEDES metrics.

  • Takeaways & Limitations

    Global and implicit local alignment provide the paper’s integrated approach to learning image-text representations while addressing image-specific information in TBPS.

Abstract

from arXiv · show

Text-based person search (TBPS) is a challenging task that aims to search pedestrian images with the same identity from an image gallery given a query text. In recent years, TBPS has made remarkable progress and state-of-the-art methods achieve superior performance by learning local fine-grained correspondence between images and texts. However, most existing methods rely on explicitly generated local parts to model fine-grained correspondence between modalities, which is unreliable due to the lack of contextual information or the potential introduction of noise. Moreover, existing methods seldom consider the information inequality problem between modalities caused by image-specific information. To address these limitations, we propose an efficient joint Multi-level Alignment Network (MANet) for TBPS, which can learn aligned image/text feature representations between modalities at multiple levels, and realize fast and effective person search. Specifically, we first design an image-specific information suppression module, which suppresses image background and environmental factors by relation-guided localization and channel attention filtration respectively. This module effectively alleviates the information inequality problem and realizes the alignment of information volume between images and texts. Secondly, we propose an implicit local alignment module to adaptively aggregate all pixel/word features of image/text to a set of modality-shared semantic topic centers and implicitly learn the local fine-grained correspondence between modalities without additional supervision and cross-modal interactions. And a global alignment is introduced as a supplement to the local perspective. The cooperation of global and local alignment modules enables better semantic alignment between modalities. Extensive experiments on multiple databases demonstrate the effectiveness and superiority of our MANet.

I. INTRODUCTION

TBPS must align heterogeneous image and text modalities, but explicit local parts can lose context or introduce noise, while image-specific information creates an information imbalance. MANet addresses these issues through image-specific information suppression and implicit multi-level alignment.

  • TBPS retrieves pedestrian images from a gallery using text descriptions, offering a more accessible query modality than image- or video-based re-identification.
  • Existing TBPS methods increasingly model local correspondence, but explicitly generated parts may lose contextual information, introduce noise, and make cross-modal interactions unreliable.Hard splitting can create ambiguous parts, while external models may add domain-gap noise and computational cost.
  • Image background and environmental factors such as illumination and weather are absent from text descriptions, creating information inequality and potentially damaging image semantics.Environmental factors can also produce large intra-class variations.
  • MANet combines Image-Specific Information Suppression with Implicit Local Alignment and Global Alignment to learn semantically aligned image-text features at multiple levels.The suppression module targets image-specific information, while the alignment modules address local and global correspondence.
  • Implicit local alignment aggregates image pixels and text words around shared semantic centers, avoiding explicit part generation, additional supervision, and direct cross-modal interactions.The shared centers let both modalities produce locally aligned features from complete contextual information.

B. Cross-modal Retrieval

The paper situates TBPS within cross-modal retrieval and implements MANet with visual and textual encoders, suppression, and global and local alignment branches. Its design targets robust modality-shared representations while retaining both global identity information and local detail.

  • B. Cross-modal Retrieval: Cross-modal retrieval methods seek effective modality-shared visual and textual embeddings in a joint space, while local-matching approaches target fine-grained correspondence.
  • B. Cross-modal Retrieval: MANet extracts visual and textual features, suppresses image background and environmental factors, and sends the processed representations to global and local alignment branches.The implementation uses CNN and RNN-based feature extraction before alignment.
  • A. Visual and Textual Representations: The visual representation uses a ResNet-50 backbone to produce an image feature map, followed by global and local branches for image alignment.The global branch produces one image feature, while the local branch produces K local visual features.
  • A. Visual and Textual Representations: The local branch assigns image pixels and text words to shared semantic centers and aggregates them into corresponding local features.
  • A. Visual and Textual Representations: Text representations are formed from W2V or BERT word embeddings processed by a bi-LSTM to obtain contextual-aware word features and global text features.Word representations are stacked into a caption-level feature representation before global and local alignment.

B. Image-Specific Information Suppression

The image-specific information suppression module reduces modality inequality by suppressing background, environmental, and other redundant image information while retaining identity-relevant features.

  • Relation-Guided Localization: RGL computes relations between each spatial position and all image positions to guide attention toward modality-shared pedestrian regions.Its relation vectors provide detailed associations beyond scalar Euclidean or cosine distances.
  • Relation-Guided Localization: Cross-modal alignment supervision assigns higher weights to human body regions and lower weights to non-pedestrian regions, suppressing background noise.The enhanced feature map is subsequently aligned with the corresponding text feature.
  • Channel Attention Filtration: CAF addresses environmental factors that widen the modality gap and increase intra-class variation among images.These factors include lighting conditions, weather, and viewpoint differences.
  • Channel Attention Filtration: CAF uses instance normalization and channel attention to filter redundant noise while restoring identity-relevant information removed by normalization.An identity-relevant content consistency loss is introduced to retain effective identity information after filtration.
  • Module Outcome: After localization and filtration, image and text features have aligned information volume before global and local semantic alignment.The resulting image feature map is denoted ˆF, while the text feature map is denoted E.

C. Implicit Local Alignment

Implicit local alignment aggregates image pixels and text words into shared semantic topic centers, producing modality-aligned local features without explicitly splitting local parts.

  • Semantic Topic Centers: ILA assumes shared semantic topic centers whose combinations can express pedestrian semantics across image and text modalities.The centers are learned jointly with the network in a shared semantic space.
  • Cross-Modal Assignment: Image pixel and text word features are adaptively assigned to the same centers to learn fine-grained local correspondence.The assignment operation is described for visual features and applied analogously to textual features.
  • Local Feature Construction: Aggregated local image and text features are produced for all semantic topic centers and concatenated into modality-aligned local representations.The resulting local features are vl and tl, each containing K center-specific feature vectors.

D. Global Alignment

Global alignment complements local alignment by extracting comprehensive image and text representations in a shared semantic embedding space.

  • Global Visual Feature: Global visual features are obtained by projecting image features into a joint semantic space and applying global max-pooling.Global max-pooling is used to mine salient information and filter noise from image regions.
  • Global Text Feature: Global text features are obtained by global max-pooling over text features followed by projection into the joint semantic space.The resulting global image and text features are denoted vg and tg.
  • Batch Representation: The model generates global image and text feature sets for a batch of N paired samples, with both sets represented in dimension dg.These features support global image-text similarity alignment.

E. Training Loss and Inference

MANet is trained with identity, cross-modality ranking, and content-consistency losses, then ranks gallery images using combined global and local cosine similarities.

  • Training Objective: Training combines identification loss, cross-modality triplet ranking loss, and identity-relevant content consistency loss in an end-to-end objective.The total loss balances the individual terms through hyperparameters.
  • Identification Loss: Identification loss applies cross-entropy to global and local image-text features to support identity-level matching.The classification uses one-hot identity labels for paired images and texts.
  • Cross-Modality Ranking Loss: Cross-modality ranking loss supervises modality-shared global and local features using bidirectional dual-constrained ranking.Descriptions of other images with the same identity provide weak supervision for text intra-class variance.
  • Content Consistency Loss: Content consistency loss keeps identity information consistent before and after CAF while preserving feature discrimination.It pulls corresponding pre- and post-CAF features together and pushes hardest negative relationships apart.
  • Inference: During inference, cosine similarities from global and local features are summed and used to rank gallery images for each text query.The combined score is S = Sg + Sl.

IV. EXPERIMENTS

The experiments evaluate MANet on two TBPS databases, comparing it with state-of-the-art methods and testing individual components through ablation studies.

  • IV. EXPERIMENTS: Experiments evaluate MANet on two TBPS databases using comparisons with state-of-the-art methods and ablation studies.The section also introduces the databases and implementation and training details.

1) Datasets:

The experiments use CUHK-PEDES and ICFG-PEDES, with specified identity, image-text, vocabulary, preprocessing, optimization, and evaluation settings.

  • Datasets:: CUHK-PEDES contains 40,206 images, 80,412 text descriptions, and 13,003 persons, with two descriptions per image.Its vocabulary contains 9408 unique words, and the official split protocol is followed.
  • Datasets:: ICFG-PEDES contains 54,522 images of 4,102 persons, with one text description per image averaging 37 words.It includes 34,674 training image-text pairs and 19,848 testing pairs; evaluation uses R@1, R@5, and R@10.
  • Datasets:: Images use a modified ImageNet-pretrained ResNet-50, while texts use Bi-LSTM with unified length L = 100.Images are resized to 384×128, and horizontal flipping is used for augmentation.
  • Datasets:: Training uses Adam for 70 epochs with batch size 64, warm-up for 10 epochs, scheduled learning-rate decay, and specified loss margins and weighting factors.The implementation is trained on a single RTX3090 24G GPU.

B. Comparison with State-of-the-art

MANet consistently outperforms reported state-of-the-art methods on CUHK-PEDES and ICFG-PEDES, while ablations show benefits from global and implicit local alignment and image-specific information suppression.

  • B. Comparison with State-of-the-art: MANet consistently outperforms state-of-the-art methods on CUHK-PEDES and ICFG-PEDES.The comparisons cover both public databases and multiple evaluation settings.
  • B. Comparison with State-of-the-art: 65.64% R@1 on CUHK-PEDES surpasses LGUR by 0.39% without explicit image splitting or cross-modal interactions.MANet also exceeds SSAN’s 61.37% R@1 by 2.55% in the W2V comparison.
  • B. Comparison with State-of-the-art: 59.44%, 76.80%, and 82.75% R@1, R@5, and R@10 with BERT surpass LGUR by 2.02%, 1.83%, and 1.30% on ICFG-PEDES.With W2V, MANet achieves 57.73%, 75.42%, and 81.72% in R@1, R@5, and R@10.
  • B. Comparison with State-of-the-art: The ablation study evaluates global alignment, implicit local alignment, image-specific information suppression, and the number of semantic topic centers.Baseline uses max-pooling followed by embedding into a common space for global alignment.
  • B. Comparison with State-of-the-art: Adding global alignment improves Baseline by 2.35%, 2.23%, and 1.68% in R@1, R@5, and R@10, while adding implicit local alignment improves them by 3.88%, 2.93%, and 2.47%.RGL and CAF further improve R@1 by 1.12% and 1.22%, and their joint deployment achieves the best performance.

2) Effectiveness of Image-Specific Information Suppression:

The image-specific information suppression and implicit local alignment analyses show how MANet handles environmental noise, feature aggregation, center initialization, and local-feature granularity.

  • 2) Effectiveness of Image-Specific Information Suppression:: RGL and CAF address image background and environmental factors, while comparisons with CBAM, Non-Local, and RGA show performance degradation for alternative attention methods.The analysis attributes the degradation to irregular image regions and limitations of regular local context or generic attention aggregation.
  • 2) Effectiveness of Image-Specific Information Suppression:: Replacing CAF with instance normalization reduces R@1 by 1.66% because discriminative identity information is removed with environmental factors.CAF restores useful identity information through channel attention and improves R@1 by nearly 0.5% with content consistency loss.
  • 3) Different Aggregation Strategy:: NetVLAD causes a 2.83% performance drop over ILA because residual-based aggregation introduces too much uncertainty.ILA instead uses relations between pixel or word features and centers as weights for aggregating features.
  • 3) Different Aggregation Strategy:: Semantic topic centers are initialized with several schemes, and the center parameter matrix has limited dependence on initialization.The experiments include normal, uniform, Kaiming, Xavier, zero, one, identity, and constant initialization schemes.
  • 3) Different Aggregation Strategy:: The number K of semantic topic centers controls local-feature granularity: too few centers provide insufficient granularity, with K=1 representing a global feature.The study selects K from 1 to 32 using Figure 5.

5) Number of Topic Centers:

MANet combines image-specific information suppression with implicit local and global alignment to improve text-based person search while keeping the main local modules computationally lightweight. Qualitative retrieval and response-map visualizations show improved alignment and more accurate results, while added cost is concentrated in GA and RGL.

  • Cost Complexity: The main computational overhead comes from GA and RGL, while CAF and ILA require very low Params and FLOPs.The paper attributes the higher cost relative to Baseline primarily to GA's fully connected layer and RGL's attention mechanism.
  • Cost Complexity: Adding the local branch causes only a small increase in inference time because similarity calculation adds one concatenated local vector.The comparison concerns inference across all test queries on CUHK-PEDES.
  • Qualitative Results: The RGL module highlights human body regions and suppresses background regions in image response maps.The visualization indicates that relation-guided localization focuses responses on pedestrian regions.
  • Qualitative Results: MANet produces more accurate top-7 retrieval results than Baseline for the same text queries, sometimes placing the correct image within the top-3 when Baseline fails.The retrieved images generally contain the semantic attributes described by the query, although some top-7 results remain incorrect.
  • Method: MANet adaptively aggregates image and text features into shared semantic topic centers for implicit local alignment without extra supervision or cross-modal interactions.The shared centers preserve complete modality context when generating local features, avoiding ambiguity and noise from explicit local-part generation.
Loading 2208.14365v2…