Source-linked AI summary
Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association
Dapeng Chen, Hongsheng Li, Xihui Liu, Yantao Shen, Zejian Yuan, Xiaogang Wang
TL;DR
Person re-identification needs discriminative visual features, but existing auxiliary supervision does not fully exploit compact semantic language descriptions. The paper uses global and local image-language associations during training, achieving state-of-the-art re-identification performance and outperforming conventional joint-embedding methods. A limitation is that descriptions can ambiguously identify people when visual attributes are shared.
Problem
Person re-identification requires discriminative visual features, motivating language supervision that captures compact semantic aspects beyond pixel-level image data.
Method
The method combines global discriminative association between image and description features with local reconstructive association between image regions and noun phrases.
Results
The method achieves state-of-the-art performance on three public person re-identification datasets and outperforms conventional image-text joint embedding methods.
Takeaways & Limitations
Language descriptions can supervise visual feature learning through complementary global and local semantic associations without requiring auxiliary information during testing.
Takeaways & Limitations
Descriptions may ambiguously identify people when different individuals share similar visual attributes, making image and language features unequally informative.
Abstract
from arXiv · showhide
Person re-identification is an important task that requires learning discriminative visual features for distinguishing different person identities. Diverse auxiliary information has been utilized to improve the visual feature learning. In this paper, we propose to exploit natural language description as additional training supervisions for effective visual features. Compared with other auxiliary information, language can describe a specific person from more compact and semantic visual aspects, thus is complementary to the pixel-level image data. Our method not only learns better global visual feature with the supervision of the overall description but also enforces semantic consistencies between local visual and linguistic features, which is achieved by building global and local image-language associations. The global image-language association is established according to the identity labels, while the local association is based upon the implicit correspondences between image regions and noun phrases. Extensive experiments demonstrate the effectiveness of employing language as training supervisions with the two association schemes. Our method achieves state-of-the-art performance without utilizing any auxiliary information during testing and shows better performance than other joint embedding methods for the image-language association.
1 Introduction
The paper uses natural-language descriptions as training supervision to improve person re-identification features, combining global and local image-language associations. It aims to learn visual representations that capture both identity discrimination and semantic correspondences between images and language.
- Global association compares whole-image and whole-description features to distinguish matched from unmatched image-language pairs.The association is learned using identity labels, encouraging higher relevance for descriptions of the same person.
- Local association models correspondences between image regions and noun phrases to enforce semantic consistency at a finer granularity.Together, the global and local schemes supervise person visual-feature learning.
- Language descriptions provide compact, semantic auxiliary supervision for learning more discriminative person re-identification features.The approach differs from prior cross-modal retrieval work by using language during training to improve visual representations.
- The method reports state-of-the-art person re-identification performance and outperforms conventional cross-modal embedding methods.The contribution summary also reports that ablations validate the effectiveness and complementarity of the two association schemes.
2 Related Work
Prior person re-identification work improves visual representations through feature extraction, metric learning, CNNs, and diverse auxiliary information. This paper instead uses language-guided image-text associations at multiple granularities to enhance visual representation learning.
- Earlier person re-identification methods focused on feature extraction or metric learning, while CNN advances combined both into robust identity-discriminative representations.
- Auxiliary signals such as pose, camera identity, depth, and infrared data have been used to align features or improve inter-image similarity estimation.
- Image-language association methods support applications including image captioning, visual question answering, and text-image retrieval.They can be modeled with generative or discriminative approaches.
- The proposed framework combines discriminative and generative-style association across global and local granularities to use language as supervision for visual representation learning.Its intermediate visual features connect with global description and local phrase features through distinct association mechanisms.
3 Our Approach
The approach improves person re-identification features by jointly learning visual discriminability and global and local associations between images and language. Global descriptions are matched by identity, while noun phrases attend to image regions and are reconstructed from aggregated visual features.
- 3.1 Visual and Linguistic Representation: The visual branch uses a ResNet-50 CNN to produce intermediate feature maps and a linearly projected feature from their average.The intermediate map is formed after a 1×1 convolution over the last residual block, while the objective feature is projected from the average map.
- 3.1 Visual and Linguistic Representation: The linguistic branch encodes whole descriptions and extracted noun phrases with LSTM-based global and local features.Interested phrases include adjective-containing noun phrases and multiple nouns joined by prepositions.
- 3.2 Global Discriminative Image-language Association: Global association scores image-description pairs as matching when they share an identity and nonmatching otherwise.The scheme projects a joint representation of the average visual feature and global text feature into a scalar relevance score and trains it with binary cross-entropy.
- 3.2 Global Discriminative Image-language Association: Unlike tuple-based cosine ranking, the global scheme uses person IDs and a projected joint representation to model more complex image-text correlations.This allows one description to correspond to different images of the same person rather than only to one image-text tuple.
- 3.3 Local Reconstructive Image-language Association: Local association uses phrase-conditioned attention to aggregate relevant image-map locations, then reconstructs the phrase with an LSTM.Neighboring feature maps are average-pooled before aggregation to reduce overfitting and merge spatially adjacent regions.
- 3 Our Approach: The method combines image and text ID losses with discriminative global and reconstructive local image-language association losses.The final objective jointly optimizes visual identity classification, text identity classification, global matching, and local phrase reconstruction.
4 Experiments
Experiments evaluate language-supervised global and local image-language associations through ablations, visualizations, retrieval, and comparisons with state-of-the-art person re-identification methods.
- Experimental Setup: Experiments use language-augmented standard person re-identification datasets, with ablations focused mainly on Market-1501 and CUHK-SYSU.The study also reports comparisons on Market-1501, CUHK03, and CUHK01.
- The Effect of Global Discriminative Association (GDA): 5.6% and 4.4% mAP gains on Market-1501 and CUHK-SYSU, respectively, show that global discriminative association improves the baseline visual representation.The gains are reported for basel.+GDA relative to the baseline in Table 2.
- The Effect of Local Reconstructive Association (LRA): 5.2% and 3.9% mAP gains on the two datasets show that local reconstructive association improves visual feature learning and complements global association.The proposed method also outperforms basel.+GDA, supporting the added value of LRA.
- The Effect of Local Reconstructive Association (LRA): Phrase-guided attention heat maps roughly capture image regions described by the corresponding noun phrases.The visualizations align attention weights for specific phrases with image regions.
- Results on Text-to-image Retrieval: The method achieves competitive text-to-image retrieval results on CUHK-PEDES despite not being specifically designed for retrieval.Retrieval ranks images by cross-modal relevance and is evaluated quantitatively and qualitatively.
5 Conclusions
The paper concludes that language descriptions can improve visual features for person re-identification through global and local image-language associations. These schemes improve feature learning, outperform existing image-text joint embedding methods, and achieve state-of-the-art performance on three public datasets.
- State-of-the-art performance is achieved on three public person re-identification datasets.
- Global association improves global visual features using overall language descriptions, while local association aligns local visual features with noun phrases through phrase reconstruction.
- Ablation studies show that the two image-language association schemes improve visual feature learning and outperform existing image-text joint embedding methods.