PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition

Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, Liang Lin

arXiv:1908.07325v1cs.CV

TL;DR

Multi-label image recognition lacks accurate semantic-region localization and complete modeling of region interactions and label co-occurrence. SSGRL uses category-guided semantic decoupling and graph-based semantic interaction, outperforming leading methods across PASCAL VOC, Microsoft-COCO, and Visual Genome benchmarks.

  • Problem

    Existing methods cannot accurately locate semantic regions or fully model their interactions and statistical label co-occurrence.

  • Method

    SSGRL uses category semantics to learn semantic-specific features and propagates messages through a graph constructed from statistical label co-occurrence.

  • Results

    SSGRL outperforms current state-of-the-art methods across the evaluated benchmarks, with reported mAP improvements on PASCAL VOC, Microsoft-COCO, and Visual Genome.

  • Takeaways & Limitations

    The framework learns category-related features and explores their interactions across multiple benchmark settings.

Abstract

from arXiv · show

Recognizing multiple labels of images is a practical and challenging task, and significant progress has been made by searching semantic-aware regions and modeling label dependency. However, current methods cannot locate the semantic regions accurately due to the lack of part-level supervision or semantic guidance. Moreover, they cannot fully explore the mutual interactions among the semantic regions and do not explicitly model the label co-occurrence. To address these issues, we propose a Semantic-Specific Graph Representation Learning (SSGRL) framework that consists of two crucial modules: 1) a semantic decoupling module that incorporates category semantics to guide learning semantic-specific representations and 2) a semantic interaction module that correlates these representations with a graph built on the statistical label co-occurrence and explores their interactions via a graph propagation mechanism. Extensive experiments on public benchmarks show that our SSGRL framework outperforms current state-of-the-art methods by a sizable margin, e.g. with an mAP improvement of 2.5%, 2.6%, 6.7%, and 3.1% on the PASCAL VOC 2007 & 2012, Microsoft-COCO and Visual Genome benchmarks, respectively. Our codes and models are available at https://github.com/HCPLab-SYSU/SSGRL.

1. Introduction

Multi-label image classification must identify multiple semantic objects while modeling relationships among their regions and labels. SSGRL addresses limitations in region localization and dependency modeling with semantic guidance, graph-based interactions, and benchmark evaluation.

  • 1. Introduction: Multi-label recognition must handle multiple semantic objects, visual variation, semantic-region discovery, and associations among regions.The task remains challenging under viewpoint, scale, illumination, and occlusion changes.
  • 1. Introduction: Existing localization methods search redundant category-agnostic proposals or require additional design constraints, while attention networks locate regions only roughly without supervision or guidance.These limitations hinder accurate semantic-region learning and end-to-end integration.
  • 1. Introduction: Sequential RNN/LSTM dependency modeling cannot fully exploit direct associations among every region or label pair and does not explicitly model statistical label co-occurrence.The paper identifies label co-occurrence as important for multi-label recognition.
  • 1. Introduction: SSGRL uses category semantics to learn semantic-specific features, then correlates them through a statistical label-co-occurrence graph and graph propagation.Its two modules are semantic decoupling and semantic interaction.
  • 1. Introduction: 95.0% and 94.8% mAP are reported on PASCAL VOC 2007 and 2012, while Microsoft-COCO and Visual Genome 500 reach 83.8% and 36.6%.With COCO pre-training and fused scale results, VOC 2012 reaches 95.4% mAP.

2. Related Works

Prior work combines object localization, attention, and dependency modeling, but faces redundancy, annotation, supervision, and limited dependency-coverage issues. SSGRL instead uses category semantics and an explicitly structured graph to model label interactions.

  • 2. Related Works: Object-proposal methods aggregate local features but incur redundant computation and prevent end-to-end deep-network training.Proposal generation assumes candidate regions contain possible foreground objects.
  • 2. Related Works: Attention-based methods enable end-to-end training with image-level annotations but lack sufficient supervision and guidance for precise semantic-region localization.The related work describes spatial transformers as an example of adaptive semantic-region search.
  • 2. Related Works: Graphical models, dependency networks, co-occurrence matrices, and CNN-RNNs model label dependencies, but prior approaches may use semantic redundancy and co-occurrence implicitly.These methods target pairwise correlations or contextual dependencies among labels.
  • 2. Related Works: SSGRL incorporates category semantics and directly correlates all label pairs in a structured graph, propagating information to explore mutual interactions.The graph is guided by statistical label co-occurrence.

3. SSGRL Framework

SSGRL learns category-specific image representations under semantic guidance, then models their interactions through a label-co-occurrence graph and graph propagation network.

  • Overall framework: The framework extracts feature maps with a fully convolutional network, then processes category-specific representations through semantic decoupling and semantic interaction modules.The final contextualized features are used to predict the label distribution.
  • Semantic decoupling: Semantic guided attention combines image features with each category’s semantic embedding to focus on corresponding semantic-aware regions.Category embeddings are extracted with pre-trained GloVe, and low-rank bilinear pooling fuses image and semantic features.
  • Semantic decoupling: Weighted average pooling produces one category-related feature vector for each category after attention coefficients are normalized across image locations.The process is repeated for all categories to obtain the complete set of category-related feature vectors.
  • Semantic interaction: The semantic interaction graph represents categories as nodes and statistical label co-occurrence probabilities as directed edge weights computed from training annotations.Graph construction requires no additional annotation beyond the dataset’s label annotations.
  • Semantic interaction: A gated recurrent graph update propagates messages among category nodes, encouraging propagation for highly correlated categories and suppressing it otherwise.After T propagation steps, contextualized hidden states are combined with input features for category-wise confidence prediction.
  • Optimization and implementation: The framework uses category-specific classification functions to convert concatenated graph features into a score vector and trains end-to-end with cross-entropy loss.The implementation uses a ResNet-101-based feature extractor and initializes it with ImageNet-pre-trained parameters.

4.1. Evaluation Metrics

The evaluation uses AP and mAP as primary measures, supplemented by precision, recall, and F1 metrics under fixed-top-3 and probability-threshold settings.

  • AP is measured for each category, while mAP averages AP across all categories.
  • The evaluation also reports overall and per-class precision, recall, and F1-measure metrics.
  • With top-3 prediction, the metrics compare the three highest-scoring labels for each image against ground-truth labels.
  • Under probability-threshold evaluation, a label is positive when its estimated probability exceeds 0.5.
  • mAP, OF1, and CF1 are identified as the most important metrics for comprehensive evaluation.

4.2. Comparison with State-of-the-art

SSGRL is evaluated against existing methods on Microsoft COCO, PASCAL VOC, and Visual Genome, using category-specific and aggregate comparisons where reported. It achieves higher performance across these benchmarks, including gains on difficult categories and large-scale label recognition.

  • Microsoft COCO: 83.8% mAP, 72.7% CF1, and 76.2% OF1 on Microsoft COCO exceed the previous best methods by 6.7%, 5.3%, and 3.3%, respectively.The previous best methods are RDAR and ResNet-SRN.
  • PASCAL VOC 2007: 93.4% mAP on PASCAL VOC 2007 improves over RCP’s 92.5% result, while COCO pre-training raises SSGRL’s mAP to 95.0%.The reported SSGRL results are generated by a single model, unlike some competing aggregated results.
  • PASCAL VOC 2012: 93.9% and 94.8% mAP on PASCAL VOC 2012 without and with COCO pre-training improve over the previous best by 1.7% and 2.6%.
  • Visual Genome 500: 3.1% higher mAP than the existing best method is achieved on the VG-500 dataset, reaching 36.6% across 500 frequent categories.Visual Genome contains substantially more categories than PASCAL VOC and Microsoft COCO; VG-500 retains the 500 most frequent.

4.3. Ablative study

The ablation study evaluates SSGRL against a ResNet-101 baseline and isolates the contributions of semantic decoupling, semantic interaction, and category-specific representations. Results show stronger gains for difficult categories and a measurable loss when graph propagation is removed.

  • Ablation setup: Table 5 compares the full framework with variants without semantic decoupling or semantic interaction using mAP on Microsoft-COCO.The reported variants are Ours, Ours w/o SD, Ours w/o SD-concat, and Ours w/o SI.
  • Category-level analysis: 24.7% and 32.5% AP improvements occur for toaster and hair drier, respectively, while easier categories such as giraffe and zebra improve only slightly.The framework benefits categories that the baseline recognizes less accurately.
  • Contribution of semantic decoupling: Semantic decoupling highlights category-specific regions when the corresponding objects are present, including skis, a snowboard, and a person’s leg.Figure 4 visualizes the input image, top-confidence semantic maps, and predicted label distribution.
  • Contribution of semantic interaction: 1.6% mAP is lost when semantic interaction is removed, reducing Ours w/o SI to 82.2% on Microsoft-COCO.Without graph propagation, each classifier directly uses its corresponding decoupled feature vector.

5. Conclusion

SSGRL learns semantic-related feature vectors with semantic-guided attention and models their interactions through graph propagation guided by statistical label co-occurrence. Experiments on multiple benchmarks demonstrate its effectiveness over existing leading methods.

  • Conclusion: SSGRL combines semantic-guided attention for semantic-related feature vectors with graph propagation to explore their interactions under statistical label co-occurrence.The framework is evaluated on Microsoft-COCO, Pascal VOC 2007 and 2012, and Visual Genome.
Loading 1908.07325v1…