PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold

Zijie Wang, Aichun Zhu, Jingyi Xue, Xili Wan, Chao Liu, Tian Wang, Yifeng Li

arXiv:2209.06209v1cs.CVcs.IRcs.MM

TL;DR

Text-based person retrieval must bridge the heterogeneous gap between visual and textual data, while many common-manifold approaches map one modality without seeing the other's distribution. LBUL looks at both modality distributions before mapping into C3M, and the authors report state-of-the-art performance on two retrieval datasets.

  • Problem

    Common-manifold methods often map one modality without access to the opposite modality's distribution, making cross-modal consensus depend on model experience.

    Source: §1 · pp. 1–2

  • Method

    LBUL uses looking and leaping steps to consider both visual and textual distributions before embedding samples into a Consistent Cross-modal Common Manifold.

    Source: §1 · p. 2

  • Results

    LBUL outperforms existing methods and achieves state-of-the-art performance on CUHK-PEDES and RSTPReid.

    Source: §1 · p. 3

  • Takeaways & Limitations

    LBUL produces a consistent common representation for each sample so cross-modal similarity can be properly measured.

    Source: §1 · p. 2

Abstract

from arXiv · show

The core problem of text-based person retrieval is how to bridge the heterogeneous gap between multi-modal data. Many previous approaches contrive to learning a latent common manifold mapping paradigm following a \textbf{cross-modal distribution consensus prediction (CDCP)} manner. When mapping features from distribution of one certain modality into the common manifold, feature distribution of the opposite modality is completely invisible. That is to say, how to achieve a cross-modal distribution consensus so as to embed and align the multi-modal features in a constructed cross-modal common manifold all depends on the experience of the model itself, instead of the actual situation. With such methods, it is inevitable that the multi-modal data can not be well aligned in the common manifold, which finally leads to a sub-optimal retrieval performance. To overcome this \textbf{CDCP dilemma}, we propose a novel algorithm termed LBUL to learn a Consistent Cross-modal Common Manifold (C$^{3}$M) for text-based person retrieval. The core idea of our method, just as a Chinese saying goes, is to `\textit{san si er hou xing}', namely, to \textbf{Look Before yoU Leap (LBUL)}. The common manifold mapping mechanism of LBUL contains a looking step and a leaping step. Compared to CDCP-based methods, LBUL considers distribution characteristics of both the visual and textual modalities before embedding data from one certain modality into C$^{3}$M to achieve a more solid cross-modal distribution consensus, and hence achieve a superior retrieval accuracy. We evaluate our proposed method on two text-based person retrieval datasets CUHK-PEDES and RSTPReid. Experimental results demonstrate that the proposed LBUL outperforms previous methods and achieves the state-of-the-art performance.

1 INTRODUCTION

Text-based person retrieval must bridge the heterogeneous visual–textual gap, but common-manifold methods often map one modality without seeing the other's distribution. LBUL considers both modalities before mapping into C3M and reports state-of-the-art retrieval performance.

  • Text-based person retrieval searches a large image database for the pedestrian matching a textual description.Text queries are described as easier to access than image-based queries in realistic applications.Source: §1 · p. 1
  • CDCP-based methods map one modality without seeing the opposite modality's distribution, leaving consensus to model experience and risking misaligned manifolds.The introduction calls this limitation the CDCP dilemma.Source: §1 · p. 2
  • LBUL's looking and leaping steps use both modalities' distribution characteristics to build consistent representations in C3M.USEM forms unified features; XProj projects into the opposite sub-manifold, and LASM maps the paired representations into C3M.Source: §1 · p. 2
  • LBUL significantly outperforms existing methods and achieves state-of-the-art performance on CUHK-PEDES and RSTPReid.Source: §1 · p. 3

2 RELATED WORKS

Prior person re-identification work spans handcrafted descriptors, metric learning, and deep learning, alongside emerging unsupervised approaches. Text-based person retrieval research includes varied alignment and matching methods, many of which follow CDCP-style common-manifold mapping.

2.1 Person Re-identification

  • Person re-identification work spans handcrafted descriptors, metric learning, and deep learning, while unsupervised methods use handcrafted features, pseudo-label clustering, or transfer learning.Source: §2.1 · p. 3

2.2 Text-based Person Retrieval

  • Text-based retrieval studies use approaches such as patch-word matching, pose attention, multi-granularity alignment, and modality-adversarial matching; most follow CDCP.The related-work section positions LBUL as a way to construct a more consistent common manifold.Source: §2.2 · p. 3

3 METHODOLOGY

LBUL frames retrieval as matching text descriptions to gallery images, then prepares visual and textual representations for a shared cross-modal manifold. Its pipeline maps features through a look-then-leap mechanism, scores cross-modal similarity, and optimizes in two stages.

3.1 Problem Formulation

  • The task uses a textual description to identify the most relevant pedestrian image from a large-scale gallery, with a dataset of paired images and descriptions.Source: §3.1 · p. 3

3.2 Feature Representation Preparation

  • Visual features combine a global representation with horizontal local strips, while textual features combine a whole-sentence representation with phrase features; self-attention distills global and local information into unified features.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.3 Look Before You Leap

3.4 Similarity for Inference

  • Inference combines C3M cosine similarity with auxiliary global similarity and fine-grained similarity computed using cross-modal attention.Source: §3.4 · p. 5

3.5 Optimization

  • Two-stage training first uses margin ranking and ID losses for global and local features, then applies projected-to-target ranking, ID, and common-manifold pair losses.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

LBUL is evaluated on CUHK-PEDES and RSTPReid, with ablations of its mapping components and training strategy. The authors report state-of-the-art retrieval performance and improvements over CDCP-based paradigms.

4.1 Experimental Setup

  • Experiments use CUHK-PEDES and RSTPReid, reporting rank-1, rank-5, and rank-10 retrieval accuracy.A search succeeds when an image of the corresponding person appears among the top-k results.Source: §4.1.1 · p. 6

4.2 Ablation Analysis

  • LBUL-based mapping outperforms CDCP-based mapping with a shared block on both datasets, supporting the authors’ claim of more consistent common representations.The comparison is reported in Table 1.Source: Table 1 · p. 6, §4.2.1 · p. 6
  • The model using USEM performs better than the tested substitutions, supporting the module’s effectiveness.The comparison is reported in Table 2.Source: §4.2.2 · p. 6, Table 2 · p. 7

4.3 Comparison with SOTA on Text-based Person Retrieval

  • 61.95%, 81.16%, and 87.19% rank-1/5/10 on CUHK-PEDES and 43.35%, 66.85%, and 76.50% on RSTPReid: LBUL reports state-of-the-art performance without BERT.The reported comparisons are in Tables 3 and 4.Source: Table 3 · p. 7, Table 4 · p. 7, §4.3 · p. 8

4.2 Ablation Analysis

  • The proposed LASM feature aggregation outperforms its basic variants, while concatenation and addition yield similar accuracies, with concatenation slightly better.The comparison is reported in Table 2.Source: §4.2.3 · p. 7, Table 2 · p. 7
  • Removing Distribution Shifting reduces retrieval accuracy by 1.71%, 1.04%, and 0.87% on CUHK-PEDES and 2.10%, 1.70%, and 0.95% on RSTPReid.The authors identify Distribution Shifting as a key component in XProj.Source: §4.2.4 · p. 7
  • One-stage training loses 8.45%, 6.11%, and 4.51% on CUHK-PEDES and 8.50%, 6.70%, and 5.55% on RSTPReid; performance stabilizes after stage two starts at epoch 15.The authors report a slight decline when the second-stage starting epoch becomes too large.Source: §4.2.5 · p. 8, Figure 3 · p. 8

5 CONCLUSION

LBUL learns a Consistent Cross-modal Common Manifold for text-based person retrieval and outperforms previous methods on CUHK-PEDES and RSTPReid, achieving state-of-the-art performance. Its two-step mapping considers both modalities’ distribution characteristics before embedding data into the manifold.

5 CONCLUSION

  • LBUL outperforms previous methods and achieves state-of-the-art performance on CUHK-PEDES and RSTPReid.The passage reports evaluation on both text-based person retrieval datasets.Source: §5 · p. 8
  • LBUL learns a Consistent Cross-modal Common Manifold (C3M) for text-based person retrieval.Source: §5 · p. 8
  • Its common-manifold mapping uses looking and leaping steps, considering both modalities’ distributions before embedding data from either one.Source: §5 · p. 8
Loading 2209.06209v1…