PaperlayerView exact revision on arXiv

Source-linked AI summary

Attribute Recognition by Joint Recurrent Learning of Context and Correlation

Jingya Wang, Xiatian Zhu, Shaogang Gong, Wei Li

arXiv:1709.08553v1cs.CV

TL;DR

Pedestrian attribute recognition must operate on poor-quality surveillance images with limited labelled data. JRL jointly models attribute context and sequential correlation through an end-to-end recurrent encoder-decoder, and experiments on PETA and RAP report superior performance and robustness across several task settings.

  • Problem

    Poor image quality, complex appearance variation, and small labelled datasets make pedestrian attribute recognition difficult.

  • Method

    JRL jointly learns inter-person image context and intra-person sequential attribute correlation in an end-to-end encoder-decoder recurrent model.

  • Results

    JRL outperforms a wide range of state-of-the-art pedestrian attribute and multi-label classification methods on PETA and RAP.

  • Takeaways & Limitations

    JRL is reported as more robust with small training sets and more scalable to applications with limited annotation budgets.

  • Takeaways & Limitations

    The approach assumes labelled pedestrian training images with image-level binary attribute annotations.

Abstract

from arXiv · show

Recognising semantic pedestrian attributes in surveillance images is a challenging task for computer vision, particularly when the imaging quality is poor with complex background clutter and uncontrolled viewing conditions, and the number of labelled training data is small. In this work, we formulate a Joint Recurrent Learning (JRL) model for exploring attribute context and correlation in order to improve attribute recognition given small sized training data with poor quality images. The JRL model learns jointly pedestrian attribute correlations in a pedestrian image and in particular their sequential ordering dependencies (latent high-order correlation) in an end-to-end encoder/decoder recurrent network. We demonstrate the performance advantage and robustness of the JRL model over a wide range of state-of-the-art deep models for pedestrian attribute recognition, multi-label image classification, and multi-person image annotation on two largest pedestrian attribute benchmarks PETA and RAP.

1. Introduction

Pedestrian attribute recognition is difficult because surveillance imagery is degraded, attributes vary across appearance and location, and labelled datasets are small. JRL addresses these constraints by jointly modelling inter-person context and sequential intra-person attribute correlations in an encoder-decoder recurrent framework.

  • Surveillance attributes support person re-identification and identification, while remaining robust to viewpoint and viewing-condition changes.
  • Poor resolution, motion blur, pose variation, illumination, background clutter, and occlusion make automatic attribute recognition difficult.
  • PETA and RAP contain only 9,500 and 33,268 training images, far fewer than ILSVRC’s 1.2 million and MS COCO’s 165,482 images.
  • Existing approaches often study attribute correlation and visual context independently, despite correlations offering inference constraints and context providing complementary scene information.
  • JRL jointly models inter-person image context and intra-person sequential attribute correlation to exploit higher-order dependencies under limited training data.
  • The encoder processes person-region sequences and similar exemplar images, while the decoder generates a variable-length ordered attribute sequence.
  • Across PETA and RAP, JRL is evaluated against seven contemporary models and reports superior performance.

2. Related Work

Related work spans pedestrian attribute recognition and sequential multi-label image classification. The paper distinguishes JRL by targeting poor-quality pedestrian imagery with small labelled datasets and a CNN-RNN-RNN design.

  • Pedestrian Attribute Recognition: Earlier pedestrian attribute methods commonly trained separate classifiers using hand-crafted colour and texture features, while later work incorporated inter-attribute correlation.
  • Multi-Label Image Classification: Pedestrian attribute recognition is a multi-label image classification problem because multiple attribute categories can coexist in one image.
  • Multi-Label Image Classification: Prior sequential multi-label methods use CNN-RNN designs, whereas JRL uses a CNN-RNN-RNN architecture.
  • Multi-Label Image Classification: Existing multi-label classification models assume large labelled datasets and sufficiently good image quality, assumptions that do not hold for surveillance attributes.

3. Joint Recurrent Learning of Attributes

The JRL model unifies intra-person visual context, inter-person similarity context, and sequential attribute correlation in an end-to-end recurrent encoder-decoder. It represents image regions as sequences, predicts attributes sequentially, and can use attention to focus on relevant regions during decoding.

  • Unified JRL framework: JRL jointly models image context and high-order attribute correlation in a recurrent encoder-decoder framework.The model addresses attribute recognition as sequential prediction rather than treating attributes independently.
  • Intra-person attribute context: The encoder processes a person image as a sequence of regions and propagates spatial context through recurrent states.Its final hidden representation summarizes the image while preserving dependencies among spatial regions.
  • Inter-person similarity context: Visually similar training exemplars provide inter-person context that compensates for ambiguous or corrupted appearance in the target image.Their context vectors are combined with the target representation, with maximum pooling reported as more effective than averaging.
  • Sequential attribute correlation: The decoder LSTM uses previous attribute predictions to model varying high-order dependencies across the ordered attribute sequence.Longer prediction sequences expose higher-order attribute correlations through recurrent feedback.
  • Recurrent attribute attention: Recurrent attribute attention produces a time-varying context representation by weighting encoded spatial regions for each predicted attribute.This provides an explicit sequence-to-sequence alignment between image regions and attribute predictions.
  • Attribute ordering: Because attribute labels are naturally unordered, JRL requires an explicitly selected attribute prediction order during training.The paper discusses random and frequency-based ordering strategies for converting multi-label annotations into sequences.

4. Experiments

Experiments evaluate JRL on PETA and RAP against conventional, deep, and recurrent competitors, then test robustness and individual context components. JRL achieves strong benchmark performance, degrades less under reduced training data, and benefits from intra-person context, inter-person similarity context, and sequential attribute correlation.

  • Datasets: PETA contains 19,000 images with 65 attributes, while RAP contains 41,585 images, providing two large pedestrian-attribute benchmarks.PETA uses 9,500 training, 1,900 verification, and 7,600 evaluation images.
  • Comparison to the State-Of-The-Arts: JRL is compared with seven contemporary models spanning conventional attribute recognition, deep attribute learning, and recurrent multi-person annotation.Competitors include SVM, MRFr2, ACN, DeepSAR, DeepMAR, and recurrent annotation models.
  • Comparison to the State-Of-The-Arts: JRL achieves the best accuracy on all four PETA metrics and on three of four RAP metrics, ranking second on RAP mPrcins at 78.11% versus ACN’s 80.12%.On RAP, JRL has higher mRclins and F1ins than ACN: 78.98% versus 72.26% and 78.58% versus 75.98%.
  • Robustness Against Training Label Sparsity: When training data falls from 100% to 25%, JRL mAPcls drops 3.64% on PETA and 3.55% on RAP, less than DeepMAR and SR CNN-RNN.The corresponding drops are 6.23% and 5.73% for DeepMAR, and 6.24% and 5.85% for SR CNN-RNN.
  • Further Analysis and Discussions: Adding intra-person attribute context improves mAPcls by 2.22% on PETA and 2.62% on RAP compared with removing the encoder.The improvement is similar under the other metrics.
  • Further Analysis and Discussions: Inter-person similarity context boosts mAPcls by 0.65% on PETA and 0.87% on RAP when auxiliary exemplar images are included.The comparison excludes exemplars during both training and inference in the reduced model.
  • Further Analysis and Discussions: Qualitative examples show JRL using sequential attribute ordering to recover unclear attributes, while wrong ordering can cause missed or incorrect predictions.JRL correctly predicts attributes such as “hair long” and “skirt” in examples where DeepMAR misses them, but ordering errors can cause mistakes.

5. Conclusion

The JRL model jointly learns attribute context and correlation for low-quality surveillance images with limited training data. It outperforms a wide range of state-of-the-art methods and is more robust when trained with small datasets.

  • JRL jointly models attribute context and correlation for pedestrian attribute recognition in low-quality surveillance images with small training datasets.
  • The model outperforms a wide range of state-of-the-art pedestrian attribute and multi-label classification methods.
  • Experiments demonstrate advantages from jointly learning high-order sequential inter-attribute correlation on the PETA and RAP benchmarks.
  • JRL is more robust than state-of-the-art deep models with small training datasets, supporting applications with limited annotation budgets.
Loading 1709.08553v1…