PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-label Image Recognition by Recurrently Discovering Attentional Regions

Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, Liang Lin

arXiv:1711.02816v1cs.CV

TL;DR

Multi-label recognition requires relating multiple semantic labels to image regions and modeling their dependencies, while common solutions rely on extracted region proposals. The paper introduces a recurrent memorized-attention module that alternates spatial transformation and LSTM-based prediction without proposals, and reports improved accuracy and efficiency on large-scale benchmarks.

  • Problem

    Multi-label image recognition is challenging because it requires associating multiple semantic labels with regions and capturing their dependencies in visually diverse images.

  • Method

    A recurrent memorized-attention module alternates a spatial transformer that locates attentional regions with an LSTM that predicts labeling scores and controls the transformer.

  • Results

    The model significantly improves state-of-the-art accuracy and efficiency on large-scale benchmarks including PASCAL VOC and COCO.

  • Takeaways & Limitations

    The framework provides proposal-free, end-to-end multi-label recognition while discovering semantic-aware regions and modeling their contextual dependencies.

  • Takeaways & Limitations

    The spatial transformer may repeatedly select the same salient region, making it difficult to retrieve all objects in an image.

Abstract

from arXiv · show

This paper proposes a novel deep architecture to address multi-label image recognition, a fundamental and practical task towards general visual understanding. Current solutions for this task usually rely on an extra step of extracting hypothesis regions (i.e., region proposals), resulting in redundant computation and sub-optimal performance. In this work, we achieve the interpretable and contextualized multi-label image classification by developing a recurrent memorized-attention module. This module consists of two alternately performed components: i) a spatial transformer layer to locate attentional regions from the convolutional feature maps in a region-proposal-free way and ii) an LSTM (Long-Short Term Memory) sub-network to sequentially predict semantic labeling scores on the located regions while capturing the global dependencies of these regions. The LSTM also output the parameters for computing the spatial transformer. On large-scale benchmarks of multi-label image classification (e.g., MS-COCO and PASCAL VOC 07), our approach demonstrates superior performances over other existing state-of-the-arts in both accuracy and efficiency.

1. Introduction

Multi-label recognition requires associating semantic labels with regions and capturing their dependencies. The paper proposes an end-to-end, proposal-free recurrent attention framework and reports superior accuracy and efficiency on large-scale benchmarks.

  • Motivation: Multi-label recognition is difficult because images contain diverse semantics and require associating labels with regions while capturing their dependencies.It also inherits challenges such as viewpoint, scale, occlusion, and illumination variation.
  • Motivation: Existing pipelines first extract hypothesis regions and then aggregate their features for multi-label classification.These regions may be produced from bottom-up image cues or extra detectors.
  • Approach: The proposed end-to-end framework discovers attentional regions corresponding to multiple semantic labels without an extra hypothesis-region extraction step.The regions are discovered over image scales and are intended to support contextualized classification.
  • Approach: The recurrent memorized-attention module combines a spatial transformer for locating regions with an LSTM that predicts labeling scores and outputs transformer parameters.This design captures global contextual dependencies among attentional regions.
  • Approach: Three spatial-transformer constraints are proposed to learn more meaningful and interpretable regions for multi-label classification.The paper presents these constraints as an additional contribution of the framework.
  • Results: Experiments on PASCAL VOC and Microsoft COCO demonstrate superiority over other leading methods in recognition accuracy and efficiency.The evaluations are conducted on large-scale multi-label image-classification benchmarks.

2. Related Works

Prior multi-label recognition methods range from handcrafted bag-of-words features to deep models and attention mechanisms. The paper positions its recurrent memorized-attention approach as a proposal-free alternative that jointly locates regions and predicts labels.

  • Earlier methods: Bag-of-words methods can perform well on simple benchmarks but may fail on complex scenes because they depend heavily on handcrafted low-level features.The paper contrasts these features with representations learned by deep models.
  • Deep models: Several deep-learning approaches classify multi-label images using off-the-shelf or fine-tuned features, ranking objectives, or related architectures.These methods represent earlier attempts to adapt deep learning to multi-label recognition.
  • Region information: Whole-image feature methods lack explicit spatial information about detected labels and are vulnerable to complex backgrounds.This motivates methods that focus on informative image regions.
  • Region information: Object-proposal methods address background influence by extracting regions and focusing feature computation on informative areas.The paper identifies proposal extraction as a common strategy for incorporating spatial information.
  • Proposed direction: The proposed method jointly locates attentional regions and predicts labels without extracting object proposals, enabling end-to-end training.Its recurrent memorized-attention module performs these operations together.
  • Attention models: Earlier recurrent attention models commonly use reinforcement learning and target low-resolution images with clean backgrounds.The paper distinguishes its setting from those attention applications.

3. Model

The model combines a spatial transformer and LSTM in a recurrent memorized-attention module to locate regions and predict multi-label scores iteratively. It operates on convolutional feature maps, uses memory across regions, and fuses regional predictions by category-wise max-pooling.

  • Architecture: The input image is processed by a VGG-16 ConvNet, whose last convolutional feature maps provide the features for recurrent attention.The model uses the conv5_3 feature maps and does not add object proposals.
  • 3.1. ST for Attentional Region Localization: The spatial transformer extracts attentional-region features from whole-image feature maps using a learned transformation matrix and bilinear interpolation.The transformation is constrained to cropping, translation, and scaling, with parameters estimated by a neural network for end-to-end learning.
  • 3.2. Recurrent Memorized-Attention Module: The recurrent memorized-attention module iteratively combines spatial transformation with LSTM computation to search discriminative regions and predict their label-score distributions.The LSTM receives each sampled feature map and updates its memory cell and hidden state while the localization network predicts the next transformation.
  • 3.2. Recurrent Memorized-Attention Module: The memory cell preserves information from previous regions, supporting recognition through category co-occurrence and promoting diversity and complementarity among attended regions.The paper describes these as two expected benefits of remembering earlier regions.
  • 3.2. Recurrent Memorized-Attention Module: After K + 1 iterations, the model produces K regional score vectors and applies category-wise max-pooling to obtain the final label distribution.The first iteration estimates a transformation matrix without producing a score because no attentional region exists initially.

4. Learning

The model trains classification and localization objectives together while constraining attentional regions to reduce redundancy, missed tiny objects, and spatial flipping.

  • 4.1. Loss for Classification: The classification objective uses Euclidean loss between predicted and normalized ground-truth label-probability vectors.Each label vector marks present classes, and normalization defines the target probability vector.
  • 4.2. Loss for Attentional Region Constraints: The model aggregates scores from attentional regions, so omitting an object can cause classification errors.The LSTM has not encountered an omitted object during prediction.
  • 4.2. Loss for Attentional Region Constraints: Three spatial-transformer constraints address redundant regions, oversized regions that neglect tiny objects, and mirrored regions.The constraints are respectively anchor, scale, and positive constraints.
  • 4.2. Loss for Attentional Region Constraints: The anchor constraint pushes later regions away from the image center toward uniformly selected anchor points.With K=5, four anchor points are generated at (0.5, 0.5), (0.5, −0.5), (−0.5, 0.5), and (−0.5, −0.5).
  • 4.2. Loss for Attentional Region Constraints: The localization loss is a weighted sum of the three constraint components and is jointly optimized with classification loss.The reported weights are λ1=0.01 and λ2=0.1, while γ=0.1 balances the overall objective.

5.1. Settings

The experiments use ImageNet-pretrained, target-dataset-fine-tuned CNN initialization, multi-scale ten-view testing, and standard multi-label metrics.

  • Implementation details: Training initializes the CNN on ImageNet, then fine-tunes it on the target multi-label dataset before adding the learned parameters to the proposed model.The implementation is based on Caffe.
  • Implementation details: Testing uses ten views across input sizes N=512 and N=640, combining five patches with their horizontally flipped versions.Features are cropped from shared VGG-16 ConvNet conv5_3 feature maps rather than recomputed independently for every patch.
  • Evaluation metrics: Overall F1, per-class F1, and mAP are identified as relatively important evaluation metrics.The metric definitions use correctly predicted, predicted, and ground-truth image counts for each label.

5.2. Comparison with State-of-the-art Methods

The method is evaluated against leading multi-label recognition approaches on PASCAL VOC 2007 and MS-COCO, achieving higher reported performance on both benchmarks.

  • 5.2. Comparison with State-of-the-art Methods: The evaluation compares the method with state-of-the-art approaches on PASCAL VOC 2007 and MS-COCO.VOC comparisons include FeV+LV-20-VD, HCP, RLSD, CNN-RNN, VeryDeep, and CNN-SVM; COCO comparisons include CNN-RNN, RLSD, and WARP.
  • 5.2.1 Performance on the VOC 2007 dataset: 91.9% mAP is achieved on PASCAL VOC 2007, exceeding the previous best HCP result of 90.9%.The method remains better than previous works when learned with a single scale of 512 or 640.
  • Dataset settings: The benchmarks span 9,963 VOC images from 20 categories and COCO’s 80 categories with about 3.5 object labels per image.COCO is described as more challenging because label counts vary considerably across images.
  • 5.2.2 Performance on the MS-COCO dataset: 67.4% per-class F1 and 72.0 overall F1 are achieved on MS-COCO, improving the previous best method by 5.4% and 4.2%, respectively.The single-scale model also beats the state-of-the-art approaches.

5.3. Ablation Study

The ablations compare learned attentional regions with object proposals and test localization constraints and multi-scale, multi-crop evaluation. Attentional regions perform better in the shared framework, all three constraints contribute, and combining crops and scales improves performance.

  • Attentional regions v.s. object proposals: Attentional regions lead to better classification performance than five object proposals in the same framework on PASCAL VOC 2007.The comparison uses single-crop evaluation at 512×512.
  • Analysis of the attentional region constraints: Removing all three attentional-region constraints causes a significant mAP drop, and removing any individual constraint also reduces performance.The reported results indicate that the anchor, scale, and positive constraints work cooperatively on VOC and COCO.
  • Multi-scale multi-view evaluation: Multi-crop fusion improves performance at a single scale, while fusing both scales produces a further improvement.The evaluation uses scales of 512×512 and 640×640, with ten crop features extracted per scale.

6. Conclusion

The paper concludes that a recurrent memorized-attention module addresses multi-label image recognition by combining spatial localization with recurrent label prediction. Experiments on PASCAL VOC and COCO report improved accuracy and efficiency over the state of the art.

  • Conclusion: The recurrent memorized-attention module combines a spatial transformer for localizing regions with an LSTM that predicts label scores and preserves past region information.The module is introduced within a deep neural network architecture for multi-label image recognition.
  • Conclusion: Experiments on large-scale benchmarks such as PASCAL VOC and COCO demonstrate improved accuracy and efficiency over the state of the art.This is the paper's reported overall experimental conclusion.
Loading 1711.02816v1…