Source-linked AI summary
ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
Zhe Wang, Zhiyuan Fang, Jun Wang, Yezhou Yang
TL;DR
Person search by natural language needs robust visual-textual associations that exploit information from both modalities. ViTAA addresses this with attribute-specific alignment and achieves state-of-the-art results on CUHK-PEDES while linking visual cues with textual phrases across related tasks.
Problem
Robust visual-textual associations are increasingly critical, creating a demand for representation learning that fully exploits both modalities.
Method
ViTAA learns attribute-aware representations by aligning visual and textual features, using natural language parsing, semantic segmentation, and cross-modal contrastive alignment.
Results
ViTAA achieves state-of-the-art results on the challenging CUHK-PEDES benchmark and links specific visual cues with corresponding words or phrases across person-search tasks.
Takeaways & Limitations
Attribute-specific alignment provides a framework for exploiting common attribute information across visual and textual modalities and learning fine-grained correspondence.
Abstract
from arXiv · showhide
Person search by natural language aims at retrieving a specific person in a large-scale image pool that matches the given textual descriptions. While most of the current methods treat the task as a holistic visual and textual feature matching one, we approach it from an attribute-aligning perspective that allows grounding specific attribute phrases to the corresponding visual regions. We achieve success as well as the performance boosting by a robust feature learning that the referred identity can be accurately bundled by multiple attribute visual cues. To be concrete, our Visual-Textual Attribute Alignment model (dubbed as ViTAA) learns to disentangle the feature space of a person into subspaces corresponding to attributes using a light auxiliary attribute segmentation computing branch. It then aligns these visual features with the textual attributes parsed from the sentences by using a novel contrastive learning loss. Upon that, we validate our ViTAA framework through extensive experiments on tasks of person search by natural language and by attribute-phrase queries, on which our system achieves state-of-the-art performances. Code will be publicly available upon publication.
1 Introduction
ViTAA addresses ambiguity in natural-language person search by decomposing visual and textual inputs into attribute components and aligning them at fine granularity. It combines attribute-aware feature learning, cross-modal contrastive alignment, and surrogate positive sampling, achieving strong results and interpretable attribute correspondence.
- Motivation: Natural-language person search is challenging because complex descriptions must be matched against variable, low-quality surveillance imagery.The task involves complex syntax and inconsistent interpretations of surveillance images.
- Motivation: Global image-text matching can confuse people with similar overall appearance, motivating attribute-level decomposition and fine-grained matching.ViTAA instead decomposes both modalities into attribute components.
- Approach: ViTAA learns attribute-aware visual and textual representations using semantic segmentation and natural-language parsing.Image local branches receive segmentation supervision, while textual descriptions are parsed into attribute-related phrases.
- Findings: Attribute-specific features help resolve ambiguity from pose, viewpoint, occlusion, and clutter while supporting interpretable attribute retrieval.The model is evaluated for both natural-language and attribute-based person search, with qualitative analysis of fine-grained correspondence.
- Approach: Its cross-modal contrastive alignment adaptively highlights informative samples, while unsupervised sampling mines surrogate positives across different identities.These components enrich training pairs and support alignment learning.
2 Related Work
Prior work spans image-based person re-identification, text-based person search, and general visual-semantic embedding. Existing approaches address complex language and cross-modal matching through attention, joint objectives, dense features, and hierarchical embeddings.
- Person Search: Person search is divided into image-query re-identification and text-query retrieval, with textual search improving flexibility when image queries are unavailable.Text-based search must handle complex syntax and inconsistent interpretations of low-quality surveillance images.
- Person Search: Existing text-based person-search methods use attention, cross-modal objectives, dense visual features, and hierarchical matching to connect descriptions with people.These approaches target relations between visual and textual representations at global, local, or hierarchical levels.
- Visual-Semantic Embedding: Visual-semantic embedding learns a joint feature space for visual inputs and corresponding textual annotations across captioning, retrieval, and question-answering tasks.This framework underlies multiple vision-language applications.
3 Our Approach
ViTAA learns global and attribute-level visual-textual representations, using parsing and auxiliary segmentation to align corresponding person attributes. Its contrastive alignment further exploits informative positive and negative pairs, including cross-identity surrogate positives.
- 3 Our Approach: ViTAA extracts one global and multiple attribute-level visual representations, then matches each visual component with its corresponding textual phrase alongside global representations.The image stream produces v0 and {v1, ..., vNatt}; the language stream produces t0 and {t1, ..., tNatt}.
- 3.1 The Image Stream: Auxiliary segmentation supervises each local image branch to predict an attribute category, diversifying local features while remaining removable during inference.Annotations come from an off-the-shelf human parsing network, and the segmentation layer is removed during inference to reduce computational cost.
- 3.2 The Language Stream: The language stream parses noun phrases into attribute categories and encodes sentence-level and phrase-level embeddings, including accessory as an attribute type.A dictionary of category anchors assigns noun phrases by highest cosine similarity; accessory is included because it can provide a crucial matching clue.
- 3.3 Visual-Textual Alignment Learning: Cross-modal alignment uses contrastive triplets at both global and attribute levels, with cosine similarity encouraging positive matches and margin-separated negative matches.The least required positive-negative similarity margin is set to 0.2 in practice.
- 3.3 Visual-Textual Alignment Learning: The logistic alignment loss produces continuous gradients that assign higher weights to more informative samples, addressing redundant negative pairs in pair-based training.The method motivates logistic loss because its derivative depends on the input rather than remaining constant.
- 3.3 Visual-Textual Alignment Learning: K-reciprocal sampling enlarges positive-pair mining across identities by treating samples sharing an attribute, such as similar shoes, as surrogate positives.Missing attribute information in either modality can be ignored during sampling.
4 Experiment
Experiments evaluate ViTAA on CUHK-PEDES person search, component ablations, visual attribute segmentation, surrogate-positive sampling, qualitative retrieval, and attribute retrieval. ViTAA reports state-of-the-art CUHK-PEDES performance and supports fine-grained visual-textual attribute correspondence.
- Experimental Setting: CUHK-PEDES contains 40,206 images of 13,003 identities, each paired with two human-annotated sentences, and uses Recall@K for retrieval evaluation.The standard metrics are R@1, R@5, and R@10.
- Comparisons with the State-of-The-Arts: ViTAA consistently leads all CUHK-PEDES R@1–R@10 metrics, outperforming GALM by 1.85%, 0.39%, 0.55%, and another reported margin.The authors describe the results as new state-of-the-art performance and note the challenge of improving R@1.
- Ablation Study: Combining segmentation, global-alignment, and attribute-alignment losses yields the best performance over the ID-loss baseline.Segmentation loss alone provides only marginal improvement, while the alignment components are reported as complementary.
- Ablation Study: The auxiliary segmentation layer produces accurate pixel-wise attribute labels under varied human poses, while local branches learn corresponding attribute feature maps.The visualization is used to support distilled person-parsing knowledge for cross-modal alignment.
- Ablation Study: K=8 achieves the best R@1 and R@10 in surrogate-positive sampling, while performance declines for larger K values.Surrogate positives use visual attributes shared by different person identities as valuable positive samples.
- Extension: Attribute Retrieval: ViTAA successfully associates textual attribute phrases with visual attributes on Market-1501 and DukeMTMC upper-body clothing retrieval.A pretrained model transferred from CUHK-PEDES without further finetuning performs strongly on almost all tested sub-attributes.
5 Conclusion
ViTAA addresses natural-language person search through attribute-specific alignment across visual and textual modalities. It achieves state-of-the-art CUHK-PEDES results and shows promise for advancing person search by natural language.
- 5 Conclusion: ViTAA aligns common attribute information across visual and textual modalities and associates visual attribute features with corresponding textual phrases.The model approaches person search through attribute-specific alignment learning.
- 5 Conclusion: ViTAA achieves state-of-the-art results on the challenging CUHK-PEDES benchmark.The conclusion presents this as evidence of the model’s promising potential for person search by natural language.