Source-linked AI summary
Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification
Feng Zhu, Hongsheng Li, Wanli Ouyang, Nenghai Yu, Xiaogang Wang
TL;DR
Multi-label image classification methods often lack spatial label relations because spatial annotations are generally unavailable. The paper introduces an end-to-end SRN that learns semantic and spatial relations from label attention maps using image-level supervision, and reports significant improvements across three datasets.
Problem
Existing multi-label methods exploit semantic label relations but cannot model spatial label relations when spatial annotations are unavailable.
Method
SRN generates attention maps for all labels, applies learnable convolutions to capture semantic and spatial relations, and aggregates its outputs with a ResNet-101 main network.
Results
The method significantly outperforms state-of-the-art approaches across NUS-WIDE, MS-COCO, and WIDER-Attribute; on MS-COCO, ResNet-SRN reaches 77.1 mAP versus 76.1 for ResNet-SRN-att.
Takeaways & Limitations
The learned SRN models effectively capture semantic and spatial label relations for improving multi-label classification.
Takeaways & Limitations
Improvements are limited for classes with few training samples, as illustrated by the toaster class in COCO.
Abstract
from arXiv · showhide
Multi-label image classification is a fundamental but challenging task in computer vision. Great progress has been achieved by exploiting semantic relations between labels in recent years. However, conventional approaches are unable to model the underlying spatial relations between labels in multi-label images, because spatial annotations of the labels are generally not provided. In this paper, we propose a unified deep neural network that exploits both semantic and spatial relations between labels with only image-level supervisions. Given a multi-label image, our proposed Spatial Regularization Network (SRN) generates attention maps for all labels and captures the underlying relations between them via learnable convolutions. By aggregating the regularized classification results with original results by a ResNet-101 network, the classification performance can be consistently improved. The whole deep neural network is trained end-to-end with only image-level annotations, thus requires no additional efforts on image annotations. Extensive evaluations on 3 public datasets with different types of labels show that our approach significantly outperforms state-of-the-arts and has strong generalization capability. Analysis of the learned SRN model demonstrates that it can effectively capture both semantic and spatial relations of labels for improving classification performance.
1. Introduction
Multi-label image classification is challenging because images commonly contain multiple objects or attributes, while existing methods capture semantic but not spatial label relations. The paper proposes SRN, which learns both relation types from attention maps using only image-level supervision.
- 1. Introduction: Multi-label classification is practical and challenging because real-world images typically contain multiple labels, including objects or attributes.Applications include scene recognition, multi-object recognition, and human attribute recognition.
- 1. Introduction: Binary relevance independently trains one binary classifier for each label as a straightforward extension of single-label classification.
- 1. Introduction: Existing proposal-based approaches model different visual regions but ignore semantic label relations, while recent relational methods still cannot capture spatial relations without location annotations.
- 1. Introduction: SRN learns an attention map for each label and applies learnable convolutions across maps to capture semantic and spatial relations.The network is trained end-to-end with only image-level supervisions.
- 1. Introduction: Evaluations on NUS-WIDE, MS-COCO, and WIDER-Attribute report significant improvements over state-of-the-art approaches across different label types.The contribution also claims strong generalization capability.
2. Related Work
Related work extends single-label methods through independent classifiers, specialized losses, region proposals, and label-relation modeling. Attention mechanisms provide a way to focus on relevant image regions for spatially related labels.
- 2. Related Work: Independent classifiers, ranking losses, region proposals, and multi-instance learning are established strategies for adapting deep features to multi-label classification.
- 2. Related Work: Some methods model label dependencies with classifier chains, probabilistic graphical models, or learned label-relation graph structures.
- 2. Related Work: Attention mechanisms adaptively focus on image regions in vision tasks and are used here to capture spatial relations among labels.
3. Methodology
The unified framework combines a ResNet-101 main network with an SRN that learns label-specific attention maps and spatial regularizations from image-level supervision. It aggregates main-network and SRN confidences to exploit semantic and spatial label relations.
- Framework: The main network follows ResNet-101, while SRN takes its visual features and learns spatial regularizations between labels.The framework aggregates label confidences from the main network and SRN to produce final predictions.
- Framework: SRN uses successive attention and spatial-regularization subnetworks to learn label relations from image-level multi-label supervision.The attention subnetwork learns label attention maps, and the regularization subnetwork operates on those maps.
- Label Attention: Label-specific attention maps are spatially normalized and weighted by label confidence to encode local attention and global label visibility.The weighted maps are formed by element-wise multiplying attention maps with sigmoid-transformed confidence maps.
- Label Attention: Learned attention maps focus on regions associated with individual labels, such as faces for “sunglass” and legs for “longPants.”The attention estimator is trained using only image-level labels, without attention-map ground truth.
- Spatial Regularization: The spatial-regularization network decouples semantic and spatial relation learning across convolutional layers to avoid the excessive parameters of a naive large-kernel design.The first layers model semantic relations, while a later convolution explores spatial relations using large receptive fields.
4. Experiments
Experiments on three datasets with concept, object, and human-attribute labels show consistent gains from SRN, while analyses indicate it learns both individual-label locations and multi-label spatial patterns.
- Datasets: Three benchmarks cover concept labels in NUS-WIDE, object labels in MS-COCO, and human attributes in WIDER-Attribute.The evaluation uses datasets with different label types and reports comparisons against state-of-the-art and baseline methods.
- NUS-WIDE: On NUS-WIDE, ResNet-SRN outperforms all baseline methods by approximately 2 percent.ResNet-101-semantic does not improve performance there, whereas the SRN exploits spatial and semantic label relations.
- Evaluation protocol: Removing the fixed top-3 prediction constraint improves ResNet-SRN F1-C from 48.9 to 58.5.The authors argue that variable-length label output is more practical for real-world applications.
- MS-COCO: On MS-COCO, ResNet-SRN reaches 77.1 mAP, compared with 76.1 for ResNet-SRN-att.The weighted-attention variant also outperforms baseline models, and semantic-context gains are limited to 75.5 mAP from 75.2.
- WIDER-Attribute: On WIDER-Attribute, ResNet-SRN performs best and consistently improves over state-of-the-art and baseline methods using only cropped person patches.These results are reported despite competing methods using full-image context, bounding boxes, or event labels.
- Visualization and analysis: SRN neurons capture either individual-label locations or specific relative-position patterns among several labels.Correlations include vertical locations of “longHair” and “flag,” while another neuron responds to recurring positions of four human attributes.
- Visualization and analysis: In COCO, spatial regularization is more effective for classes with more co-existing labels, whereas toaster gains little because of limited training samples.This class-dependent pattern identifies label co-occurrence and sample availability as practical boundaries on improvement.
5. Conclusion
The paper improves multi-label image classification by learning spatial relations from label attention maps using only image-level supervision. Evaluations show strong performance and relation-capturing capability across multiple datasets.
- The method learns attention maps for all labels and captures semantic and spatial relations from weighted attention maps.
- Figure 7 compares per-class AP improvement with the average number of concurrent labels in true-positive images, sorting classes by AP improvement.
- The approach significantly outperforms state-of-the-art methods on NUS-WIDE, MS-COCO, and WIDER-Attribute datasets.