PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Label Image Classification with Regional Latent Semantic Dependencies

Junjie Zhang, Qi Wu, Chunhua Shen, Jian Zhang, Jianfeng Lu

arXiv:1612.01082v3cs.CV

TL;DR

Multi-label classification remains challenging because global visual features have limited discrimination for small objects and visual concepts. RLSD localizes regions containing semantically dependent labels and models their dependencies with RNNs, achieving superior benchmark performance and approaching a bounding-box-guided upper bound without bounding-box annotations.

  • Problem

    Global CNN features and global label-dependency models remain limited for predicting small objects and visual concepts in multi-label images.

  • Method

    RLSD uses fully convolutional localization to find regions with multiple dependent labels, then applies regional RNNs to model their latent semantic dependencies.

  • Results

    RLSD consistently outperforms state-of-the-art methods across benchmark datasets, including 87.3% mAP on VOC PASCAL 2007 and 4% higher precision than CNN-RNN on NUS-WIDE.

  • Takeaways & Limitations

    Regional semantic dependencies improve overall multi-label performance, particularly for small objects and visual concepts, while RLSD approaches the bounding-box-guided upper bound without extra annotations.

Abstract

from arXiv · show

Deep convolution neural networks (CNN) have demonstrated advanced performance on single-label image classification, and various progress also have been made to apply CNN methods on multi-label image classification, which requires to annotate objects, attributes, scene categories etc. in a single shot. Recent state-of-the-art approaches to multi-label image classification exploit the label dependencies in an image, at global level, largely improving the labeling capacity. However, predicting small objects and visual concepts is still challenging due to the limited discrimination of the global visual features. In this paper, we propose a Regional Latent Semantic Dependencies model (RLSD) to address this problem. The utilized model includes a fully convolutional localization architecture to localize the regions that may contain multiple highly-dependent labels. The localized regions are further sent to the recurrent neural networks (RNN) to characterize the latent semantic dependencies at the regional level. Experimental results on several benchmark datasets show that our proposed model achieves the best performance compared to the state-of-the-art models, especially for predicting small objects occurred in the images. In addition, we set up an upper bound model (RLSD+ft-RPN) using bounding box coordinates during training, the experimental results also show that our RLSD can approach the upper bound without using the bounding-box annotations, which is more realistic in the real world.

I. INTRODUCTION

Multi-label classification must bridge visual content and multiple labels while modeling dependencies among them. RLSD addresses global-feature limitations by localizing semantically related regions and modeling their dependencies regionally, improving small-object prediction.

  • Multi-label classification assigns objects, attributes, actions, and scenes to images for applications including retrieval and semantic segmentation.
  • The central challenge is bridging the semantic gap between image visual content and multiple labels.
  • Independent regional classifiers struggle to model dependencies among labels, while global CNN-RNN methods remain limited for small objects and attributes.
  • RLSD captures latent semantic dependencies at the regional level by combining region-based features with RNN-based label co-occurrence modeling.
  • RLSD predicts small labels such as bottle, vase, and wine glass that Multi-CNN and CNN+LSTM miss in the illustrated image.
  • RLSD approaches the RLSD+ft-RPN upper bound without using bounding-box annotations during training.

II. RELATED WORKS

Prior work models multi-label prediction with handcrafted features, deep CNNs, semantic representations, graphical models, and global RNN dependencies. RLSD instead combines learned region localization with recurrent modeling of regional label dependencies.

  • Earlier multi-label methods combined handcrafted feature representations, classifiers, and context modeling in bag-of-words pipelines.
  • Deep approaches expanded CNNs to multi-label prediction using ranking objectives and related classification formulations.
  • Multi-modal methods learned shared latent semantic spaces between image content and labels using techniques such as CCA and KCCA.
  • Probabilistic graphical models represented dependencies through structures including trees, directed acyclic graphs, and chain rules.
  • RNNs capture high-order label dependencies, but prior CNN-RNN work modeled them at the global rather than regional level.
  • RLSD combines region proposals, a fully connected recognition network, and RNNs to identify semantically rich regions while modeling dependencies.

A. Localizing Multi-label Regions

RLSD generates regions designed to contain multiple semantically dependent labels rather than single objects. Its localization layer regresses and samples proposals, then uses differentiable bilinear interpolation to produce regional features.

  • Conventional object-proposal methods are unsuitable because they generally target regions containing a single object.
  • The localization layer receives convolutional features and outputs spatial regions of interest with fixed-sized representations.
  • Localization layer: Anchors at convolutional-map locations are regressed into proposal boxes with predicted offsets and confidence scores.
  • Training: Smooth L1 localization loss is used only when fine-tuning the localization layer in the RLSD+ft-RPN upper-bound model.
  • Proposal sampling: The model samples a minibatch of 256 proposals, selecting high-confidence positives and low-confidence negatives for the subsequent LSTM stage.
  • Localization layer: Compared with MCG, RLSD generates larger regions that can contain multiple objects, supporting regional dependency modeling and small-object prediction.
  • Feature extraction: Bilinear interpolation replaces ROI pooling so regional features remain compatible with the fully connected layer and gradients reach features and box coordinates.

3) Encoded by a Fully-Connected Network:

RLSD encodes each sampled region into a feature vector before regional recurrent prediction. The model also retains the whole image among proposals because some labels relate to global image content.

  • Encoded by a Fully-Connected Network:: Each sampled region is flattened and passed through two 4096-dimensional fully connected layers, producing a 4096-dimensional region vector.
  • Encoded by a Fully-Connected Network:: All encoded regions form a minibatch of regional feature vectors for subsequent processing.
  • Encoded by a Fully-Connected Network:: RLSD-generated boxes are generally larger than MCG proposals and may contain multiple objects, supporting richer label dependencies.
  • Encoded by a Fully-Connected Network:: The entire image is added to the proposals because some labels are related to the whole image.
  • Encoded by a Fully-Connected Network:: An MCG-based baseline replaces RLSD localization to test the effectiveness of the proposed multi-label region localization layer.

B. An LSTM-based Multi-Label Generator

The model uses an LSTM to generate label sequences for each region, capturing latent semantic dependencies among labels. Region features initialize the sequence, while predicted labels are recursively embedded as later inputs.

  • LSTMs generate a sequence of label probability distributions for each localized region.
  • The LSTM updates its memory from the current input, previous hidden state, and previous cell state.
  • Each hidden state is passed through Softmax to produce a probability distribution over all labels.
  • A region feature vector initializes the LSTM, after which label embeddings from previous predictions provide subsequent inputs.
  • Predictions from all regions and time steps form an M × T × L matrix, with zero-padding for shorter label sequences.

C. Max-pooling and Loss Function

RLSD fuses regional and temporal label predictions through max-pooling, producing one image-level prediction. The fused output is optimized with a multi-way softmax and squared loss.

  • Cross-region and time max-pooling combines predictions into one integrative image-level output.
  • For each label category, the fused value is the maximum prediction across regions and time steps.
  • Max-pooling is intended to make the model robust to noisy region proposals or time-step predictions.
  • The fusion output enters a multi-way softmax layer trained with squared loss against ground-truth probability vectors.
  • During testing, the model also uses localized regions to capture small objects such as wine glasses, bottles, and vases.

D. Initialization and Pre-Training

RLSD can train end-to-end, but initialization and pre-training are important for promising performance and rapid convergence. Pre-training targets both localization and regional sequence modeling.

  • RLSD supports end-to-end training from scratch, although proper initialization and pre-training are important for promising performance.
  • At test time, localized multi-label regions feed extracted features into a shared LSTM, including small objects within those regions.
  • The localization layer is pre-trained on Visual Genome region captions containing multiple objects and visual concepts.
  • The LSTM is first pre-trained on global images before initializing the regional LSTM.
  • This LSTM initialization helps the model converge quickly.

IV. EXPERIMENTS

The experiments evaluate RLSD on three benchmark datasets using precision, recall, and mAP, with comparisons against baselines and state-of-the-art methods. Bounding-box fine-tuning defines an upper-bound variant for assessing localization quality.

  • RLSD is evaluated on VOC PASCAL 2007, Microsoft COCO, and NUS-WIDE against state-of-the-art and baseline models.
  • RLSD+ft-RPN uses bounding-box coordinates to fine-tune localization and serves as an upper-bound model for testing RLSD’s generality.
  • The evaluation reports overall and per-class precision and recall, together with mean average precision.
  • The evaluation defines per-label counts for correctly labeled images, predicted images, and ground-truth images.

C. Baseline Models

The paper compares RLSD against baselines that omit label dependencies, model them globally, or use alternative region proposals. These comparisons isolate the contributions of regional semantic dependencies and the proposed localization mechanism.

  • Purpose: The baseline comparisons are designed to evaluate the effectiveness of RLSD and its components.The paper states that the baseline results indicate the significance of the model’s components.
  • Multi-label CNN: Multi-label CNN independently predicts labels without modeling label dependencies.It uses a standard CNN configuration with element-wise logistic loss.
  • CNN+LSTM: CNN+LSTM models label dependencies globally by feeding a global image representation into an LSTM.Its configuration otherwise matches the proposed RLSD model.
  • MCG-CNN+LSTM: MCG-CNN+LSTM tests localization by replacing the region proposal network with Multiscale Combinatorial Grouping proposals.The model sends the top-256 pre-extracted object proposals for each image into the region-based LSTM.

D. Results on the VOC PASCAL 2007

On PASCAL VOC 2007 and MS COCO, RLSD outperforms the reported baselines across key metrics and is particularly effective for small objects. Its performance remains close to the bounding-box-guided upper-bound model.

  • PASCAL VOC 2007: 87.3% mAP on PASCAL VOC 2007 exceeds CNN-RNN at 84% and CNN+LSTM at 83.5%.The reported gap over Multi-CNN is 4.3%, compared with 0.5% for CNN+LSTM.
  • PASCAL VOC 2007: RLSD reaches 87.3% versus 84.9% for MCG-CNN+LSTM and is only 1.2% below RLSD+ft-RPN.The comparison supports regional proposals containing multiple highly dependent labels rather than single-object proposals.
  • PASCAL VOC 2007: RLSD outperforms baselines on all reported PASCAL VOC metrics and approaches RLSD+ft-RPN, especially in precision.The reported metrics include per-class and overall precision and recall, plus F1 score.
  • MS COCO: The MS COCO evaluation uses 82,081 training images and 40,137 test images with 80 annotated object concepts.The dataset provides strong label dependencies, such as keyboard and computer co-occurring frequently.
  • MS COCO: On MS COCO, RLSD exceeds CNN-RNN in per-class precision, per-class recall, and overall precision, while achieving higher mAP and mAP@10.Its overall recall is lower because a threshold of 0.5 can result in fewer than k predicted labels.
  • Small-object prediction: RLSD is especially effective for visually small objects, including bird, fire hydrant, kite, sports ball, and bottle.Its recall is much higher than CNN+LSTM for labels whose corresponding bounding boxes are smaller.

F. Results on the NUS-WIDE

On NUS-WIDE, RLSD is evaluated against multiple prior methods and achieves higher precision than CNN-RNN, while this dataset lacks bounding-box annotations for an upper-bound comparison.

  • NUS-WIDE contains 269,648 images and 1,000 tags, manually labeled into 81 concepts after removing unannotated images.
  • RLSD outperforms the compared methods on NUS-WIDE and achieves 4% higher precision than CNN-RNN.The comparison includes metric learning, multi-edge graph, K nearest neighbor, softmax prediction, WARP, and CNN-RNN.
  • Because NUS-WIDE provides no bounding-box annotations, the dataset has no RLSD+ft-RPN upper-bound model.
  • Across benchmark datasets, RLSD is reported to achieve superior overall performance, especially for predicting small objects and visual concepts.
Loading 1612.01082v3…