PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Label Classification with Label Graph Superimposing

Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, Shilei Wen

arXiv:1911.09243v1cs.CV

TL;DR

Multi-label recognition still lacks clear strategies for modeling label correlations and using label-system structure during feature learning. KSSNet superimposes statistical and knowledge graphs and injects label embeddings into CNN features, improving mAP on MS-COCO and Charades.

  • Problem

    The best way to model label correlations and use mutually related labels to improve CNN feature learning remains unclear for multi-label classification.

  • Method

    KSSNet superimposes statistical and knowledge-prior label graphs, then connects GCN label embeddings to shallow, middle, and deep CNN layers.

  • Results

    KSSNet improves mAP over its plain CNN counterpart by 6.4% on MS-COCO and 12.0% on Charades, achieving new state-of-the-art performance.

  • Takeaways & Limitations

    Experiments support graph superimposing and label-feature correlation modeling as effective for label-relationship-aware image and video recognition.

  • Takeaways & Limitations

    The graph superimposing framework uses a mixing weight λ constrained to the interval [0, 1].

Abstract

from arXiv · show

Images or videos always contain multiple objects or actions. Multi-label recognition has been witnessed to achieve pretty performance attribute to the rapid development of deep learning technologies. Recently, graph convolution network (GCN) is leveraged to boost the performance of multi-label recognition. However, what is the best way for label correlation modeling and how feature learning can be improved with label system awareness are still unclear. In this paper, we propose a label graph superimposing framework to improve the conventional GCN+CNN framework developed for multi-label recognition in the following two aspects. Firstly, we model the label correlations by superimposing label graph built from statistical co-occurrence information into the graph constructed from knowledge priors of labels, and then multi-layer graph convolutions are applied on the final superimposed graph for label embedding abstraction. Secondly, we propose to leverage embedding of the whole label system for better representation learning. In detail, lateral connections between GCN and CNN are added at shallow, middle and deep layers to inject information of label system into backbone CNN for label-awareness in the feature learning process. Extensive experiments are carried out on MS-COCO and Charades datasets, showing that our proposed solution can greatly improve the recognition performance and achieves new state-of-the-art recognition performance.

Introduction

The introduction frames multi-label recognition around modeling label correlations and incorporating label-system information into feature learning, then presents KSSNet as a label-graph-superimposed deep convolutional solution. Experiments on MS-COCO and Charades report substantial mAP gains and new state-of-the-art performance.

  • Motivation: Multi-label recognition must model label co-occurrence priors and determine whether label-system information should influence CNN feature extraction before final label prediction.The introduction identifies both label relationships and the two-stage CNN-to-label mapping as central unresolved issues.
  • Related work: Existing probabilistic graph models can face scalability costs, while RNNs depend on label ordering and may miss global dependencies.These limitations motivate graph-based alternatives for label-correlation modeling.
  • Proposed method: The proposed KSSNet uses a label graph superimposing framework to address the two introduction questions for multi-label recognition.The framework is described as a label graph superimposed deep convolution network.
  • Results: 6.4% and 12.0% absolute mAP improvements are obtained on MS-COCO and Charades, respectively, over the plain CNN counterpart.The reported gains come with very limited computation cost overhead.
  • Results: 1.3% and 2.4% mAP improvements over current state-of-the-art solutions are reported on MS-COCO and Charades, respectively.The paper states that these results establish new state-of-the-art performance.

Related Work

Related work improves multi-label recognition by modeling label relationships, moving from probabilistic formulations toward neural and graph-based methods. Conventional classification extensions leave label correlation unexplored, motivating graph-based approaches such as tree, knowledge, and GCN label structures.

  • Multi-Label Classification: Replacing cross-entropy with multi-binary classification loss enables standard image or video classification frameworks to perform multi-label classification, but leaves label correlation unexplored.The passage identifies this straightforward extension as degrading recognition performance.
  • Label Relationship Modeling: Existing multi-label methods model label relationships using probabilistic graph models based on label co-occurrence, while neural-network solutions address their computational cost.The passage cites probabilistic and neural-network-based approaches as established directions.
  • Graph-Based Label Modeling: Graph-based approaches model label structure through maximum-spanning-tree label graphs and knowledge graphs describing label dependencies.These are presented as two typical label graph solutions.
  • Graph-Based Label Modeling: GCN was introduced for non-grid structured data modeling and has since been leveraged in computer vision tasks.The passage frames GCN as a graph-based technique used across multiple computer vision applications.

Approach

KSSNet models label correlations by superimposing statistical co-occurrence information with knowledge-prior relationships in a KS graph. It also injects GCN-derived label-system embeddings into CNN features at multiple stages through lateral connections for label-aware representation learning.

  • Label graph construction: KSSNet superimposes a statistical label graph with a knowledge-prior graph to model label correlations.The statistical graph reflects training-set label co-occurrence, while the knowledge graph uses human-created resources such as ConceptNet.
  • Label-aware feature learning: Lateral connections inject GCN label information into backbone 2D/3D CNNs at shallow, middle, and deep stages rather than only at final recognition.The operation computes correlations between CNN feature points and label embeddings, maps them through a 1×1 or 1×1×1 convolution, and adds the result to CNN features.
  • Label graph construction: The superimposed adjacency matrix is normalized and combines the statistical and knowledge graphs using a weight coefficient λ ∈[0, 1].A threshold τ is used to suppress redundant edges, and further adjacency adjustment addresses potential over-smoothing in deeper GCNs.
  • Graph convolution: GCN layers update each label embedding by mixing embeddings from neighboring nodes through learnable transformations and nonlinear activation.The initial label embeddings are extracted from semantic networks such as ConceptNet.
  • Label-aware feature learning: The lateral-connection design makes hidden CNN features aware of label relationships and provides additional gradients that regularize hidden label embeddings.This strategy is intended to produce label-system anchored representations and can overcome deeper-GCN over-smoothing to some extent.

Experiment

Experiments evaluate the proposed solution on image and video multi-label recognition tasks, alongside ablation studies of the graph construction method in KSSNet.

  • Experiment: Experiments assess the proposed solution on image and video multi-label recognition tasks.The section also conducts ablation studies evaluating the proposed graph construction method in KSSNet.
  • Experiment: Ablation studies evaluate the effectiveness of the proposed graph construction method in KSSNet.

Datasets and Evaluation Metrics

The study evaluates multi-label recognition on MS-COCO images and Charades videos using validation sets and dataset-specific recognition metrics, additionally reporting computational cost.

  • MS-COCO: MS-COCO contains about 82K training, 41K validation, and 41K test images with 80 object labels, averaging 2.9 labels per image.All methods are evaluated on the validation set because test-set ground-truth labels are unavailable.
  • Charades: Charades contains around 9.8K videos, with about 8K for training and 1.8K for validation, averaging 6.8 labels per video across 157 action labels.Videos average about 30 seconds, and each action label combines a noun and verb.
  • Evaluation metrics: MS-COCO evaluation reports CP, CR, CF1, OP, OR, OF1, and mAP, while Charades evaluation uses mAP.The metrics support comprehensive comparison across methods on both datasets.
  • Evaluation metrics: The evaluation also reports each model’s FLOPs to characterize computational cost.The supplied passage introduces FLOPs reporting alongside recognition metrics.

Implementation Details

KSSNet uses modality-specific CNN backbones alongside a four-layer GCN, with specified activation, embedding, graph-construction, and training settings. Image models use ResNet101, while video models use inflated Inception-I3D with 64-frame clips.

  • Image recognition: For image recognition, KSSNet uses ImageNet-pretrained ResNet101 as its backbone and a four-layer GCN with output channels 256, 512, 1024, and 2048.LeakyReLU with negative slope 0.2 is used in graph convolution layers, and three 2D lateral-connections operations connect the GCN and CNN.
  • Label embeddings and graph construction: Initial label embeddings come from 300-dimensional GloVe, with multiword labels represented by averaging the embeddings of their component words.The statistical matrix GS follows the strategy of Chen et al. (2019).
  • Training settings: The image-model hyperparameters are λ = 0.4, τ = 0.02, and η = 0.4.These values are specified for equations (2), (3), and (4), respectively.
  • Video recognition: For video recognition, KSSNet uses ImageNet-pretrained BN-Inception inflated according to I3D and fine-tunes on 64-frame input clips.Clips contain 64 one-frame snippets, use 224 × 224 spatial crops from 256 × 256 scaled videos, and set λ, η, and τ to 0.6, 0.4, and the passage's truncated value, respectively.

Comparison with Baselines

KSSNet outperforms state-of-the-art and backbone baselines on MS-COCO and Charades, with label-graph-based embeddings delivering substantial mAP gains at limited computational cost.

  • Experimental comparison: The comparisons are conducted on the MS-COCO validation set and Charades to evaluate the effectiveness of the proposed solution against state-of-the-art methods.The experiments cover both image and video multi-label recognition settings.
  • MS-COCO: On MS-COCO, KSSNet is compared with CNN-RNN, SRN, ResNet101, Multi-Evidence, and ML-GCN, whose quantitative validation-set results are reported in Table 1.ML-GCN is identified as the current state-of-the-art and uses a GCN+CNN framework based on a statistical label graph.
  • MS-COCO: 6.4% is the maximum absolute mAP improvement of KSSNet over its ResNet101 backbone, demonstrating the benefit of explicitly modeling label relationships.The passage attributes this gain to GCN label embeddings, which exploit label relationships that plain CNNs may fail to learn or ignore.
  • Charades: 12.0% higher mAP is achieved by KSSNet than the backbone I3D model on Charades, while computation increases only from 108 GFLOPs to 127 GFLOPs.This gain is larger than the 6.4% improvement reported on MS-COCO.

Ablation Studies

Ablations show that KSS graph construction is dataset-dependent, while increasing GCN depth consistently improves KSSNet performance. The best Charades hyperparameters are λ=0.6 and τ=0.03, yielding mAP 44.93.

  • Label graph ablation: On MS-COCO, the knowledge graph performs worse than the statistical graph and the proposed KS graph because it misses relationships among uncovered labels and includes trivial edges that cause over-smoothing.On Charades, the result reverses: KSSNet (knowledge graph) outperforms KSSNet (statistical graph) by a mAP of 0.4, attributed to complex labels and fewer training samples.
  • GCN depth ablation: KSSNet (3 layers) improves over KSSNet (2 layers) by 0.6% mAP on MS-COCO and 1.9% mAP on Charades.All variants use the KSS graph as the adjacency matrix; the backbone is ResNet101 for MS-COCO and Inception-I3d for Charades.
  • GCN depth ablation: KSSNet (4 layers) outperforms KSSNet (3 layers) by 0.2% and 1.1% in mAP on MS-COCO and Charades, respectively.The improvement comes with a small increase in computational cost and model size, and is attributed to additional LC operations and regularizing gradients.
  • Hyperparameter sensitivity: On Charades, varying λ from 0 to 1 produces mAP values of 41.05, 41.7, 43.44, 44.93, 44.83 and 41.57, with the maximum at λ=0.6.With λ fixed at 0.6, varying τ from 0.01 to 0.04 produces mAP values of 44.6, 44.62, 44.93 and 44.77, maximizing at τ=0.03.

Conclusion

The paper concludes that label relationships are crucial for multi-label recognition and introduces KSSNet, which superimposes knowledge and statistical graphs and injects GCN embeddings into CNN features.

  • Conclusion: KSSNet constructs a KS graph by superimposing a knowledge graph into a statistical graph for label correlation modeling.This models label relationships using both knowledge and statistical information.
  • Conclusion: Its LC operation injects GCN embeddings into CNN features to model label-feature correlations.The operation connects label embeddings with visual features.
  • Conclusion: The resulting model learns label-anchored feature representations for multi-label recognition.The conclusion identifies label-anchored representation learning as the benefit of LC.
Loading 1911.09243v1…