PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

Kai Niu, Yan Huang, Wanli Ouyang, Liang Wang

arXiv:1906.09610v1cs.CV

TL;DR

Description-based person Re-id must compare heterogeneous images and descriptions even though all samples share the pedestrian category, making fine-grained discrimination difficult. The paper proposes MIA, which hierarchically aligns global contexts, global-local relations, and visual parts with noun phrases, using step-by-step training. It reports state-of-the-art performance on CUHK-PEDES, including improvements over prior methods and together training.

  • Problem

    Description-based person Re-id lacks direct cross-modal similarity measurement and is further complicated because all images belong to the pedestrian category.

  • Method

    MIA hierarchically combines global-global, global-local, and local-local alignments, with step-by-step training for the multiple granularities.

  • Results

    MIA achieves state-of-the-art performance on CUHK-PEDES, with 1.3% higher R@1 than together training and 15.85% improvement in R@1 over the best Dual-Path model using VGG-16.

  • Takeaways & Limitations

    Hierarchical multi-granularity alignment provides a comprehensive cross-modal similarity evaluation by combining global contexts, relation-guided filtering, and fine-grained matching.

Abstract

from arXiv · show

Description-based person re-identification (Re-id) is an important task in video surveillance that requires discriminative cross-modal representations to distinguish different people. It is difficult to directly measure the similarity between images and descriptions due to the modality heterogeneity (the cross-modal problem). And all samples belonging to a single category (the fine-grained problem) makes this task even harder than the conventional image-description matching task. In this paper, we propose a Multi-granularity Image-text Alignments (MIA) model to alleviate the cross-modal fine-grained problem for better similarity evaluation in description-based person Re-id. Specifically, three different granularities, i.e., global-global, global-local and local-local alignments are carried out hierarchically. Firstly, the global-global alignment in the Global Contrast (GC) module is for matching the global contexts of images and descriptions. Secondly, the global-local alignment employs the potential relations between local components and global contexts to highlight the distinguishable components while eliminating the uninvolved ones adaptively in the Relation-guided Global-local Alignment (RGA) module. Thirdly, as for the local-local alignment, we match visual human parts with noun phrases in the Bi-directional Fine-grained Matching (BFM) module. The whole network combining multiple granularities can be end-to-end trained without complex pre-processing. To address the difficulties in training the combination of multiple granularities, an effective step training strategy is proposed to train these granularities step-by-step. Extensive experiments and analysis have shown that our method obtains the state-of-the-art performance on the CUHK-PEDES dataset and outperforms the previous methods by a significant margin.

I. INTRODUCTION

Description-based person Re-id is difficult because images and descriptions are heterogeneous while all images share the pedestrian category. MIA addresses this fine-grained cross-modal problem through hierarchical global-global, global-local, and local-local alignments plus step-by-step training.

  • Problem: Description-based person Re-id is harder than conventional image-description matching because all samples belong to the same pedestrian category.
  • Challenges: Existing methods face unavailable body-part or attribute annotations, costly external-cue preprocessing, and ambiguities in image-part and text matching.Pedestrian datasets lack the annotations needed for some pose-, region-, or attribute-based preprocessing; simple word partitioning and text-to-part retrieval can also be ambiguous.
  • Method: Global-local relations highlight distinguishable components and eliminate uninvolved components from cross-modal representations.For example, image attributes such as legs and shoes that are not mentioned in a description should contribute less to the visual representation.
  • Training: The model is end-to-end trainable, but global contexts and local components are trained hierarchically and step-by-step to address their different semantic levels and optimization difficulties.The paper reports that simultaneous training can be harmed by local-component ambiguities and mismatched objectives.
  • Method: MIA hierarchically combines global-global matching, global-local relation-guided filtering, and local-local matching between visual human parts and noun phrases.The GC module matches global contexts, RGA filters uninvolved components, and BFM performs visual-part–noun-phrase matching.
  • Results: The method obtains state-of-the-art performance on CUHK-PEDES and significantly outperforms previous methods.

II. RELATED WORK

Prior work spans global image-text alignment, fine-grained matching, and person re-identification using visual cues. This paper distinguishes itself by addressing cross-modal fine-grained matching through multi-granularity alignments and visual-part–noun-phrase matching.

  • Global image-text methods align image and sentence features, while later approaches improve matching through selective multimodal or hard-negative mechanisms.
  • Fine-grained difficulty is central in description-based person Re-id because all images belong to the pedestrian category.
  • Person re-identification methods commonly exploit visual cues such as pose, body parts, background regions, or masks for discriminative representations.
  • The proposed approach is presented as an early effort to match visual human parts with noun phrases and use multi-granularity image-description alignments.

III. PROPOSED APPROACH

The proposed framework combines global and local representation extraction with a hierarchical multi-granularity alignment model. Its modules align global contexts, filter local components using cross-modal relations, and compute intermediate similarities.

  • The framework has global and local representation extraction alongside a multi-granularity image-text alignment model.
  • CNN features are split into global context and image-part paths, while textual descriptions are encoded through sentence-level recurrent representations.
  • The GC module aligns global visual and textual context representations to obtain a fundamental global-global similarity.
  • The RGA module uses cross-modal relation alignments to filter uninvolved attributes and produce better aggregated representations with an intermediate global-local similarity.
  • The framework contains three modules corresponding to global-global, global-local, and local-local alignment granularities.

2) Relation-guided Global-local Alignment (RGA):

RGA improves local representations before fine-grained matching by modeling relations between image parts, noun phrases, and global contexts. Bidirectional attention selectively aggregates components into relation-guided representations.

  • Image parts are obtained from vertically partitioned CNN feature maps, while noun phrases are extracted dynamically using syntactic analysis and part-of-speech tagging.
  • Direct local-component matching is considered inappropriate because modality heterogeneity, imperfect partitioning, and insufficiently trained local representations create ambiguities.
  • RGA has image-guided phrase alignment and sentence-guided part alignment, using cross-modal relations to selectively aggregate local features.
  • In the I →T direction, each image part is related to the global textual context, and these relations supervise a relation-guided visual representation.
  • The resulting relation-guided representations support intermediate global-local similarity in both alignment directions.

3) Bi-directional Fine-grained Matching (BFM):

BFM performs bidirectional fine-grained matching between visual human parts and textual noun phrases. It uses adapted component representations and attention-based aggregation to obtain discriminative local-local similarities.

  • BFM matches visual human parts with textual noun phrases to obtain distinguishable fine-grained cross-modal similarities.
  • Two BFM-specific MLPs adapt visual and textual component representations for appropriate parts-phrases matching.
  • The module uses two opposite directions: noun-phrase-related P →N matching and part-related N →P matching.
  • In P →N matching, each noun phrase evaluates similarity with all image parts before attention combines the relevant visual features.
  • The opposite N →P direction produces combined textual representations associated with individual image parts.

4) Similarity Fusion:

The model fuses five similarities spanning global-global, global-local, and local-local alignments into a final cross-modal similarity.

  • Five similarities are organized into three categories: sG is global-global, sI and sT are global-local, and sP and sN are local-local.
  • Two hyper-parameters adjust the proportions of the RGA and BFM similarities in the final fusion.

B. Learning Procedure

Training combines identity and matching objectives through a three-step procedure that first learns global contexts, then fine-grained representations, and finally BFM components.

  • 1) Objectives: Identity classification treats pedestrian IDs as categories, while matching uses cross-modal retrieval objectives for paired and mismatched image-description samples.
  • 1) Objectives: The identity objective provides a loose constraint for initialization, whereas the matching objective is stricter and better suited to fine-tuning accurate cross-modal relations.
  • 2) Training Strategy: The step training strategy has three stages corresponding to the GC, RGA, and BFM modules.
  • 2) Training Strategy: Step one trains global representations with only the identity objective, leaving the pretrained visual CNN unfine-tuned while learning the textual path and global visual layer.
  • 2) Training Strategy: Step two adds the matching objective and fine-tunes parameters for fine-grained component representations under trained global contexts.
  • 2) Training Strategy: Step three fixes other parameters and trains only the two BFM MLPs for visual parts and noun phrases.
  • Discussion: The identity objective is restricted to global contexts because local components are not tightly related to person identity and may be shared across people.

IV. EXPERIMENTS AND ANALYSIS

Experiments evaluate MIA on CUHK-PEDES using recall-at-K retrieval metrics and specified visual, part-based, textual, and optimization settings.

  • Dataset and Metrics: CUHK-PEDES contains 40,206 images from 13,003 pedestrians, with two hand-annotated descriptions per image.
  • Dataset and Metrics: Performance is measured by R@1, R@5, R@10, and their summation, where success means retrieving any corresponding-person image within the top K.
  • Implementation Details: The experiments use pretrained VGG-16 and ResNet-50 visual CNNs, resize images to 384×128, and apply random mirroring for augmentation.
  • Implementation Details: Visual feature maps are divided into six equal vertical parts for image-part representations.
  • Implementation Details: The four visual and textual MLPs map part and noun-phrase features to 1024-dimensional representations using two linear layers with ReLU.
  • Implementation Details: Cosine similarity and Adam optimization are used, with batch size 96 and staged learning-rate schedules.

C. Evaluation of Ablation Models

Ablations show that combining multiple granularities improves retrieval, with RGA generally stronger than BFM and the full MIA model achieving the best reported combinations.

  • C. Evaluation of Ablation Models: The ablation study labels Context, Relation, and Component as global-global, global-local, and local-local alignments, respectively.
  • 1) Granularities and Their Combination: GC+RGA and GC+BFM both outperform GC, demonstrating improved cross-modal similarity evaluation from multi-granularity alignments.
  • 1) Granularities and Their Combination: RGA is more effective than BFM because trained global contexts provide a more appropriate optimization reference than direct matching of potentially ambiguous local components.
  • 1) Granularities and Their Combination: Adding BFM to GC+RGA worsens performance relative to GC+RGA, indicating that joint BFM training can produce an incorrect optimization direction.
  • 1) Granularities and Their Combination: MIA(R) outperforms MIA(G) because relation-guided filtering highlights significant components and suppresses uninvolved ones instead of weighting all global components equally.
  • 1) Granularities and Their Combination: The full MIA model outperforms MIA(G), MIA(R), and MIA(L), supporting comprehensive similarity evaluation through combined granularities.
  • Visualization Analysis: Fine-grained matching correctly associates explicit noun phrases with image parts, but ambiguous phrases such as “Somebody” cannot be localized accurately.

2) Step Training Strategy Analysis:

The step training strategy addresses optimization difficulties caused by combining multiple granularities and improves the final similarity over together training. It progressively strengthens fine-grained feature extraction while preserving global similarity quality.

  • Directly training BFM with other modules harms global-context similarity because ambiguous local components can induce incorrect optimization directions.
  • Step training alleviates ambiguous local-component extraction and prevents the global-context similarity sG from decreasing.
  • Fine-grained similarity sL improves progressively as GC, RGA, and BFM are introduced step-by-step with appropriate objectives.
  • Compared with GC + RGA, MIA further improves R@1 by 0.8% through backbone pretraining followed by BFM-specific adaptation.

3) Objective Analysis:

The objective analysis compares training objectives, initialization strategies, noun-phrase selection, and state-of-the-art methods. Identity-aware objectives and noun phrases support retrieval, while MIA substantially outperforms prior methods across visual backbones.

  • Combining identity and matching objectives in step-2 produces better retrieval results than using the matching objective alone.
  • Identity-objective initialization in step-1 performs better than matching-objective initialization because it matches the person-identification testing protocol.
  • Jointly combining step-1 and step-2 training performs much worse than the proposed strategy, validating identity-only initialization for step-1.
  • MIA performs better with noun phrases than with all description words because unmatched words introduce cross-modal noise.
  • With ResNet-50, MIA exceeds the best Dual-Path model by 8.70% and GLIA by 9.52% in R@1.

E. Effect of the Hyper-parameters

The paper examines similarity-balancing hyper-parameters and reports failure cases that remain difficult for MIA. It concludes by identifying incomplete coverage and fuzzy descriptions as practical error sources and suggesting future remedies.

  • Effect of the Hyper-parameters: Increasing λ1 improves performance, while increasing λ2 helps until around 0.7 before performance decreases slightly.
  • Failure Cases Analysis: MIA failure cases include incomplete coverage, where descriptions omit key distinguishing attributes present in candidate images.
  • Failure Cases Analysis: Fuzzy attributes, such as an ambiguous gray-versus-blue shirt color, can place the ground-truth image outside the top-10 results.
  • Failure Cases Analysis: Future improvements include attribute-relevant image partitioning, constraints linking image parts to noun phrases, data cleaning, and image-quality enhancement.
  • Conclusion: MIA combines global-global, global-local, and local-local alignments with step training in an end-to-end model for description-based person re-identification.
Loading 1906.09610v1…