PaperlayerView exact revision on arXiv

Source-linked AI summary

TIPCB: A Simple but Effective Part-based Convolutional Baseline for Text-based Person Search

Yuhao Chen, Guoqing Zhang, Yujiang Lu, Zhenxing Wang, Yuhui Zheng, Ruili Wang

arXiv:2105.11628v1cs.CV

TL;DR

Text-based person search must retrieve person images from descriptions despite a large image-text feature gap, while many local-alignment methods are difficult to deploy. TIPCB uses a simple end-to-end dual-path local alignment network and progressive multi-stage matching, achieving state-of-the-art performance in experiments. Its supported scope is CUHK-PEDES, and the paper notes that some compared methods produce false matches for shared attributes such as a black backpack.

  • Problem

    Text-based person search is challenging because image and text modalities have a significant feature gap, while existing local-matching methods are often impractical for realistic scenarios due to complexity.

  • Method

    TIPCB is an end-to-end framework combining dual-path local alignment with horizontally segmented images, adaptively aligned text, and multi-stage matching across three feature levels.

  • Results

    TIPCB achieves state-of-the-art performance in experiments on the CUHK-PEDES dataset.

  • Takeaways & Limitations

    The framework is designed to facilitate practical text-based person search without additional models or complex evaluation strategies.

  • Takeaways & Limitations

    The paper illustrates a remaining matching ambiguity: different persons can share attributes such as a black backpack, producing incorrect matches.

Abstract

from arXiv · show

Text-based person search is a sub-task in the field of image retrieval, which aims to retrieve target person images according to a given textual description. The significant feature gap between two modalities makes this task very challenging. Many existing methods attempt to utilize local alignment to address this problem in the fine-grained level. However, most relevant methods introduce additional models or complicated training and evaluation strategies, which are hard to use in realistic scenarios. In order to facilitate the practical application, we propose a simple but effective end-to-end learning framework for text-based person search named TIPCB (i.e., Text-Image Part-based Convolutional Baseline). Firstly, a novel dual-path local alignment network structure is proposed to extract visual and textual local representations, in which images are segmented horizontally and texts are aligned adaptively. Then, we propose a multi-stage cross-modal matching strategy, which eliminates the modality gap from three feature levels, including low level, local level and global level. Extensive experiments are conducted on the widely-used benchmark dataset (CUHK-PEDES) and verify that our method outperforms the state-of-the-art methods by 3.69%, 2.95% and 2.31% in terms of Top-1, Top-5 and Top-10. Our code has been released in https://github.com/OrangeYHChen/TIPCB.

1. Introduction

Text-based person search retrieves person images from a large gallery using textual descriptions, but substantial visual-textual differences and local ambiguity make it challenging. TIPCB addresses practical deployment concerns with an end-to-end local-alignment and multi-stage matching framework.

  • Problem and motivation: Text-based person search ranks gallery person images by similarity to a textual description, supporting retrieval when target images are unavailable.The paper highlights applications such as searching for suspects from eyewitness descriptions.
  • Problem and motivation: Visual representations are disrupted by occlusion, background clutter, and pose or viewpoint variation, while image-text domains have significant feature differences.Inter-modal feature variance is described as much larger than intra-modal variance.
  • Existing approaches: Local feature matching is necessary because key information can be hidden in image and text details that global representations may miss.The paper contrasts local details with global representation extraction and irrelevant-region noise.
  • Practicality gap: Most existing local-matching methods are impractical because additional models, multi-task strategies, or complex similarity measures increase computation and use-phase time.These approaches may also prevent end-to-end learning or require repeated local-similarity calculations.
  • TIPCB framework: TIPCB introduces a dual-path local alignment network that extracts visual and textual local representations, with horizontal image stripes and adaptive textual alignment.Visual parts use a PCB-style strategy, while textual branches align representations to corresponding visual locals.
  • TIPCB framework: TIPCB progressively reduces the modality gap through low-level, local-level, and global-level matching in an end-to-end trainable framework.Experiments on CUHK-PEDES report state-of-the-art performance for the proposed framework.

2. Related Work

Related work spans local representation learning for person re-identification and global or local cross-modal matching for text-based person search. Existing text-based methods improve alignment but often rely on extra models, complex similarity strategies, or global representations that miss local detail.

  • Person Re-identification: Person re-identification methods learn detailed and distinctive local representations, including independently processed horizontal image stripes and multi-granularity parts.PCB applies horizontal feature-map segmentation, while MGN uses multi-granularity representation learning.
  • Person Re-identification: Some re-identification methods use spatial attention, pose estimation, or human semantic segmentation to align parts and mine detailed information.These auxiliary components are presented as approaches for improving local representation robustness.
  • Text-based Person Search: Text-based person search methods include global matching models that learn unified image-text spaces but may miss distinctive local details or mix noise.The paper identifies this as a limitation of methods focusing only on global representations.
  • Text-based Person Search: Local-matching methods use attribute recognition, attribute segmentation, pose information, or attention to bridge image-text representations.Examples include auxiliary segmentation and multi-granularity attention mechanisms.
  • Text-based Person Search: Despite stronger local alignment, many methods incur substantial computation through additional models or multi-task learning and remain time-consuming during use.Complex similarity measures can require multiple local representations and repeated similarity calculations.
  • Text-based Person Search: TIPCB instead uses an end-to-end trainable dual-path network to learn local aligned representations simply and effectively.This design is presented as a response to the complexity of prior local-matching approaches.

3. Proposed Method

TIPCB uses dual CNN paths to learn visual and textual representations, with horizontal image partitioning and adaptive textual alignment. Multi-stage cross-modal projection matching operates across low-, local-, and global-level representations to reduce modality differences and support final similarity evaluation.

  • Dual-path local alignment: TIPCB combines visual and textual CNN branches in a dual-path local alignment structure.The visual branch uses PCB after the backbone, while the textual branch uses a multi-branch residual network after pretrained BERT.
  • Multi-stage cross-modal matching: Cross-modal projection matching is applied to low-level, local-level, and global-level representations to reduce the modality gap.Low-level matching reduces the gap early, local-level matching supports image-text alignment, and global-level matching improves compatibility of evaluation representations.
  • Dual-path local alignment: Images are partitioned into K horizontal stripes, and K textual residual branches adaptively learn representations corresponding to those visual regions.Each textual branch contains P residual bottlenecks and is designed to match visual local representations.
  • Multi-stage cross-modal matching: During testing, only the global-level representation is used to measure image-text similarity.Training nevertheless applies matching objectives at all three representation levels.
  • Textual representation learning: The textual branch uses pretrained BERT word embeddings followed by a multi-branch textual CNN with shallow residual processing.BERT weights are frozen to provide semantic embeddings, while the CNN further processes them; shallow processing avoids information loss associated with downsampling.
  • Textual representation learning: Skip connections transmit lower-layer information to higher layers, restraining network degradation and speeding model training.The bottleneck convolutions use unit strides to keep textual information uncompressed.

4. Experiment

Experiments evaluate TIPCB on CUHK-PEDES using standard top-k retrieval accuracy and compare it with global- and local-matching methods. Ablations examine local representations, region granularity, textual bottlenecks, downsampling, and CMPM losses across feature levels.

  • Experimental Settings: CUHK-PEDES contains 40,206 person images from 13,003 identities, with about two textual descriptions per image and descriptions averaging over 23 words.The benchmark uses 34,054 training images, 3,078 validation images, and 3,074 test images.
  • Experimental Settings: Performance is evaluated with Top-1, Top-5, and Top-10 accuracy, where a search succeeds when a matching image appears among the top-k ranked gallery results.Gallery images are ranked by image-text similarity for each query description.
  • Comparison with State-of-the-Art Methods: Local-alignment methods generally achieve better performance than global-matching methods, underscoring the importance of fine-grained image-text alignment.The comparison includes global-matching methods such as GNA-RNN and local-matching methods such as GLA, MIA, and NAFS.
  • Effects of Textual CNN Design: The textual CNN performs best with three bottlenecks per branch, whereas one bottleneck provides insufficient capacity; two to three bottlenecks perform better overall.Top-1 accuracy first increases and then decreases as the number of bottlenecks grows.
  • Effects of Textual CNN Design: Downsampling more often significantly decreases Top-1 accuracy because it causes obvious textual information loss.The ablation evaluates different numbers of downsampling operations in the textual branches.

5. Conclusion

TIPCB is a simple end-to-end framework for practical text-based person search, combining dual-path local alignment with multi-stage cross-modal matching. It learns visual and textual local representations and matches them across three feature levels to reduce the modality gap.

  • TIPCB is a simple but effective end-to-end learning framework for text-based person search.It is designed to facilitate practical application.
  • Unlike existing local-matching methods, TIPCB avoids additional models and complex evaluation strategies through its end-to-end structure.
  • The framework uses an end-to-end trainable dual-path network to learn visual and textual local representations.Images are segmented horizontally, while texts are aligned adaptively.
  • TIPCB applies a multi-stage cross-modal matching strategy across low-level, local-level, and global-level features.The strategy eliminates the modality gap step by step.
  • The outstanding experimental results verify the superiority of the proposed TIPCB method.
Loading 2105.11628v1…