PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition

Tianshui Chen, Liang Lin, Riquan Chen, Xiaolu Hui, Hefeng Wu

arXiv:2009.09450v1cs.CV

TL;DR

Multi-label recognition must model interactions among semantic regions and labels while handling limited samples for novel categories. KGGR integrates statistical label correlations with graph propagation in feature and semantic spaces, and it outperforms leading methods on both MLR and ML-FSL benchmarks. The framework therefore provides a knowledge-guided approach to contextualized representations and classifier learning across correlated categories.

  • Problem

    Existing methods do not fully capture mutual region-label interactions, explicitly integrate statistical label co-occurrences, or generalize to novel categories with limited samples.

  • Method

    KGGR constructs a statistical co-occurrence knowledge graph and uses graph propagation to exchange information among semantic-specific features and classifier weights.

  • Results

    KGGR demonstrates effectiveness over existing leading methods on both multi-label image recognition and multi-label few-shot learning tasks across public benchmarks.

  • Takeaways & Limitations

    Explicit category correlations guide contextualized feature learning and classifier training across correlated categories in both multi-label and few-shot settings.

  • Takeaways & Limitations

    Feature propagation initializes each node with the feature vector of its corresponding category, assuming category-specific node-feature alignment.

Abstract

from arXiv · show

Recognizing multiple labels of an image is a practical yet challenging task, and remarkable progress has been achieved by searching for semantic regions and exploiting label dependencies. However, current works utilize RNN/LSTM to implicitly capture sequential region/label dependencies, which cannot fully explore mutual interactions among the semantic regions/labels and do not explicitly integrate label co-occurrences. In addition, these works require large amounts of training samples for each category, and they are unable to generalize to novel categories with limited samples. To address these issues, we propose a knowledge-guided graph routing (KGGR) framework, which unifies prior knowledge of statistical label correlations with deep neural networks. The framework exploits prior knowledge to guide adaptive information propagation among different categories to facilitate multi-label analysis and reduce the dependency of training samples. Specifically, it first builds a structured knowledge graph to correlate different labels based on statistical label co-occurrence. Then, it introduces the label semantics to guide learning semantic-specific features to initialize the graph, and it exploits a graph propagation network to explore graph node interactions, enabling learning contextualized image feature representations. Moreover, we initialize each graph node with the classifier weights for the corresponding label and apply another propagation network to transfer node messages through the graph. In this way, it can facilitate exploiting the information of correlated labels to help train better classifiers. We conduct extensive experiments on the traditional multi-label image recognition (MLR) and multi-label few-shot learning (ML-FSL) tasks and show that our KGGR framework outperforms the current state-of-the-art methods by sizable margins on the public benchmarks.

1 INTRODUCTION

Multi-label image recognition must discover semantic regions and model their interplay, yet existing methods do not fully capture pairwise interactions or explicitly use statistical label correlations. KGGR addresses these gaps with knowledge-guided graph propagation across feature and semantic spaces and reports substantial improvements on both MLR and ML-FSL benchmarks.

  • Multi-label image analysis is challenging because images contain diverse categories whose regions and semantics interact.
  • RNN/LSTM-based methods model sequential region or label dependencies but cannot fully exploit direct pairwise interactions or explicitly integrate statistical category correlations.
  • Conventional deep convolutional approaches require many category-specific samples and cannot generalize to novel categories with limited samples.
  • KGGR represents semantic correlations with a structured knowledge graph and propagates information among category-specific features and classifier representations.Category semantics guide semantic-specific feature learning, while correlated categories provide information for classifier training.
  • KGGR extends the framework to both multi-label image recognition and multi-label few-shot learning, with substantial performance improvements on both tasks.

2 RELATED WORKS

Related work spans multi-label image recognition, few-shot learning, and knowledge representation learning. Existing approaches use localization, attention, dependency models, and few-shot techniques, motivating explicit knowledge integration for representation learning.

  • Multi-label image recognition: Multi-label classification methods commonly combine object proposals or localized regions with deep learning to aggregate local discriminative features.
  • Label dependencies: Graphical and dependency models capture label co-occurrence, while KGGR uses structured correlations and parallel graph networks for feature and classifier interactions.
  • Few-shot learning: Few-shot learning targets novel concepts from few examples through metric learning, meta-learning, and data synthesis techniques.
  • Knowledge representation learning: Deep neural networks can learn powerful representations but lack the human capability of exploiting accumulated prior knowledge and reasoning.

3 KGGR FRAMEWORK

KGGR encodes category correlations in a graph and uses two propagation networks to transfer information across feature and semantic spaces. This pipeline learns contextualized features while regularizing classifier-weight learning.

  • KGGR builds on a graph encoding prior knowledge of category correlations and applies two graph propagation networks in feature and semantic spaces.
  • A CNN generates image features, while category semantics guide semantic-specific feature learning before feature information is propagated through the graph.
  • Semantic-space propagation transfers classifier information across categories, helping train classifiers while feature-space propagation learns contextualized representations.

3.1 Knowledge Graph Construction

The knowledge graph represents categories as nodes and their statistical co-occurrences as edges. Edge probabilities are computed from training-set annotations without requiring additional annotation.

  • Each graph node represents a category, and each edge represents the co-occurrence relationship between two corresponding categories.
  • The framework computes category-pair probabilities from training-set annotations, so constructing the graph introduces no additional annotation.

3.2 Graph Feature Propagation

KGGR uses category semantics to extract category-specific image features, then propagates them through a knowledge-guided graph to learn contextualized representations. The graph’s gated updates encourage information exchange among correlated categories while preserving category-specific features.

  • 3.2 Graph Feature Propagation: The feature propagation module decouples an image into category-specific feature vectors using semantically guided attention before graph propagation.Each category receives a feature vector focused on its semantic regions.
  • 3.2 Graph Feature Propagation: Category semantics guide attention by combining image feature maps with pretrained GloVe embeddings for each category.The semantic embedding focuses attention on regions relevant to the corresponding category.
  • 3.2 Graph Feature Propagation: Attention coefficients are normalized across locations, and weighted average pooling produces one semantic-specific feature vector per category.The process repeats across all categories to obtain {f0, f1, . . . , fC−1}.
  • 3.2 Graph Feature Propagation: The graph initializes each node with its corresponding category feature and uses gated recurrent updates to propagate messages among nodes.The final hidden state combines information from the node’s own category with contextualized messages from other categories.
  • 3.2 Graph Feature Propagation: Propagation is strengthened between highly correlated categories and suppressed between weakly correlated categories.After Tf iterations, final hidden states encode category-specific and cross-category information, which is combined with the initial feature.

3.3 Graph Semantic Propagation

KGGR propagates classifier prototypes through the same category-correlation graph. This transfers information from correlated labels to refine classifiers, particularly when categories have few training samples.

  • 3.3 Graph Semantic Propagation: Classifier weights serve as prototype representations for their corresponding categories.The graph propagates these prototypes to exploit information from correlated categories during classifier training.
  • 3.3 Graph Semantic Propagation: Each graph node is initialized with its corresponding classifier weight before semantic propagation begins.The initial hidden state is set to winit_c.
  • 3.3 Graph Semantic Propagation: At each iteration, a node aggregates parameter messages from correlated nodes to refine its own classifier representation.Higher correlation encourages propagation, whereas lower correlation suppresses it.
  • 3.3 Graph Semantic Propagation: After Ts gated propagation iterations, an output network predicts the final classifier weights.The resulting weights incorporate information transferred across correlated categories.

3.4 Network Architecture

The implementation uses a ResNet-101 feature extractor and two graph networks with shared gated architectures. Both propagation networks run for three iterations with 2,048-dimensional hidden and output representations.

  • 3.4 Network Architecture: The feature extractor is based on ResNet-101, with the final average-pooling configuration replaced by a 2×2 layer with stride 2.The low-rank bilinear pooling dimensions are N=2,048, ds=300, d1=1,024, and d2=1,024.
  • 3.4 Network Architecture: The two gated graph networks share an architecture with 2,048-dimensional hidden states and three iterations each.Their output dimension is also 2,048.
  • 3.4 Network Architecture: Each output network is a 4,096-to-2,048 fully connected layer followed by a hyperbolic tangent function.This configuration applies to both feature and classifier propagation networks.

3.5 From MLR to ML-FSL

KGGR converts category features and propagated classifier weights into per-category scores for multi-label recognition. For few-shot learning, it uses a two-stage procedure that trains on base categories before adapting the graph networks to novel categories.

  • 3.5 From MLR to ML-FSL: Category features and propagated classifier weights are multiplied to produce one score per category.All category scores form the score vector s.
  • 3.5 From MLR to ML-FSL: The model predicts category probabilities with a sigmoid function and trains using cross entropy in an end-to-end fashion.The training set contains image annotations represented as binary category indicators.
  • 3.5 From MLR to ML-FSL: Multi-label few-shot learning separates Cb base categories with sufficient samples from Cn novel categories with limited samples.Novel categories may have only 1, 2, or 5 training samples.
  • 3.5 From MLR to ML-FSL: The few-shot adaptation uses two stages: base-category training first, followed by graph-network training on novel categories with the backbone and attention modules fixed.Novel-category correlations are computed from semantic similarity because limited samples prevent reliable statistical co-occurrence estimation.

4 EXPERIMENTS

The experiments evaluate KGGR on multi-label recognition benchmarks and few-shot settings using AP/mAP and precision, recall, and F1 metrics. KGGR improves over prior methods across Microsoft COCO, Pascal VOC, and VG-500, including larger-scale categories.

  • Evaluation Metrics: Experiments evaluate multi-label recognition using AP, mAP, precision, recall, and F1-measure metrics, including all-label and top-3 settings.For top-3 evaluation, labels below probability 0.5 are excluded; mAP, OF1, and CF1 are identified as the most important metrics.
  • Datasets: The benchmarks include Pascal VOC 2007 and 2012, Microsoft COCO, and Visual Genome 500, which covers 500 categories for larger-scale evaluation.Visual Genome 500 uses 400 base and 100 novel categories for few-shot evaluation, with K=1 or K=5 novel-category samples.
  • Multi-Label Recognition: On Microsoft COCO, KGGR achieves 84.3% mAP, improving the previous best result of 83.1% by 1.2%.The comparison covers mAP and precision, recall, and F1 metrics under all-label and top-3 settings.
  • Multi-Label Recognition: On Pascal VOC 2007, KGGR reaches 93.6% mAP without pretraining and 95.0% with COCO pretraining, outperforming aggregated competing results.The reported results come from a single model, whereas some competing methods aggregate multiple models or fuse results.
  • Multi-Label Recognition: On Pascal VOC 2012, KGGR obtains 93.9% and 95.0% mAP without and with COCO pretraining, while fusion raises performance to 95.6%.The improvements over the previous best method are 1.7% without pretraining and 2.8% with pretraining; the fused evaluation uses two models and multi-scale crops.
  • Large-Scale Categories: On VG-500, KGGR improves over ML-GCN by 4.8% mAP, 1.5% top-3 CF1, 1.8% top-3 OF1, 1.4% top-all CF1, and 1.0% top-all OF1.The reported KGGR scores are 37.4%, 32.5%, 47.2%, 19.4%, and 28.1%, respectively.

4.4 Results on Multi-Label Few-Shot Learning

The framework is evaluated for multi-label few-shot learning on Microsoft COCO and Visual Genome 500, where it outperforms prior methods in both 1-shot and 5-shot settings.

  • Evaluation setup: The evaluation reports mAP for both 1-shot and 5-shot settings on Microsoft COCO and Visual Genome 500.Tables 5 and 6 summarize these comparisons.
  • Microsoft COCO: 52.3% and 63.5% mAP are achieved on Microsoft COCO in the 1-shot and 5-shot settings, exceeding LaSO by 7.0% and 5.4%.Using GoogleNet-v3 instead of ResNet-101, the framework still achieves 49.4% and 61.0% mAP.
  • Visual Genome 500: Visual Genome 500 is introduced because Microsoft COCO has only 80 categories, whereas some few-shot benchmarks contain thousands.The authors describe the larger-category benchmark as more realistic.
  • Visual Genome 500: 20.7% and 26.1% mAP are achieved on Visual Genome 500 in the 1-shot and 5-shot settings, exceeding LaSO by 4.1% and 4.3%.Both methods use ResNet-101 for this comparison.

4.5 Ablative study

Ablations show that graph feature propagation is the key component, while semantic propagation and semantically guided attention provide additional gains in multi-label recognition and few-shot learning.

  • Overall framework: KGGR contains two graph propagation modules that interact to learn features and classifiers.The ablation study evaluates their separate contributions.
  • Graph feature propagation: Removing graph feature propagation decreases multi-label recognition mAP from 84.3% to 80.6% on Microsoft COCO.The module learns contextualized category-specific feature vectors through semantic attention and knowledge-embedded propagation.
  • Semantically guided attention: Removing semantically guided attention yields 80.9% mAP, only slightly above the baseline, indicating limited benefit without that component’s semantic-specific feature maps.The mechanism focuses feature maps on corresponding semantic regions.
  • Graph feature propagation: Without graph feature propagation, multi-label few-shot mAP falls from 52.3% to 44.1% for 1-shot and from 63.5% to 56.0% for 5-shot learning.These results are reported for Microsoft COCO.
  • Graph semantic propagation: Removing graph semantic propagation reduces Microsoft COCO mAP from 84.3% to 83.8%.Graph semantic propagation transfers information from correlated categories to learn classifiers.
  • Graph semantic propagation: Removing graph semantic propagation also reduces performance in the multi-label few-shot task.The passage states a similar phenomenon but does not provide the corresponding values.

5 CONCLUSION

KGGR integrates prior label-correlation knowledge with deep neural networks through graph propagation in feature and semantic spaces. It learns contextualized category features and transfers classifier information across correlated categories for multi-label recognition and few-shot learning.

  • Conclusion: KGGR integrates statistical category co-occurrence correlations with graph propagation in both feature and semantic spaces.The framework is applied to multi-label image recognition and multi-label few-shot learning.
  • Conclusion: Feature propagation learns contextualized representations for each category, while semantic propagation transfers classifier information across categories.The two mechanisms guide feature and classifier learning, respectively.
Loading 2009.09450v1…