PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-based Dual Relation Graph for Multi-label Image Recognition

Jiawei Zhao, Ke Yan, Yifan Zhao, Xiaowei Guo, Feiyue Huang, Jia Li

arXiv:2110.04722v2cs.CV

TL;DR

Multi-label image recognition must handle varying object scales, inconsistent appearances, and confused inter-class relationships. TDRG combines structural and semantic relation graphs, and reports state-of-the-art results on MS-COCO and VOC benchmarks.

  • Problem

    Multi-label recognition faces varying object scales, inconsistent visual appearances, and confused inter-class relationships, while attention-based spatial methods do not explicitly model label dependencies.

    Source: §1 · p. 1

  • Method

    TDRG combines a cross-scale Transformer structural graph for long-range context with a dynamic semantic graph constrained by class-specific information and structural embeddings.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    TDRG achieves new state-of-the-art results on the MS-COCO and PASCAL VOC multi-label recognition benchmarks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The paper concludes that jointly learning complementary structural and semantic relations yields robust graph relations for multi-label recognition.

    Source: §5 · p. 8

Abstract

from arXiv · show

The simultaneous recognition of multiple objects in one image remains a challenging task, spanning multiple events in the recognition field such as various object scales, inconsistent appearances, and confused inter-class relationships. Recent research efforts mainly resort to the statistic label co-occurrences and linguistic word embedding to enhance the unclear semantics. Different from these researches, in this paper, we propose a novel Transformer-based Dual Relation learning framework, constructing complementary relationships by exploring two aspects of correlation, i.e., structural relation graph and semantic relation graph. The structural relation graph aims to capture long-range correlations from object context, by developing a cross-scale transformer-based architecture. The semantic graph dynamically models the semantic meanings of image objects with explicit semantic-aware constraints. In addition, we also incorporate the learnt structural relationship into the semantic graph, constructing a joint relation graph for robust representations. With the collaborative learning of these two effective relation graphs, our approach achieves new state-of-the-art on two popular multi-label recognition benchmarks, i.e., MS-COCO and VOC 2007 dataset.

1. Introduction

Multi-label recognition remains difficult because images contain objects at varying scales and with inconsistent appearances, while their inter-class relationships can be confused. The paper proposes complementary structural and semantic relation graphs to address these challenges, reporting state-of-the-art results on MS-COCO and VOC.

1. Introduction

  • Multi-label parsing is challenged by varying object scales, inconsistent visual appearances, and confused inter-class relationships.Source: §1 · p. 1
  • Attention-based spatial representations strengthen CNN features, but do not explicitly model the multi-label dependencies needed to understand label relationships.Source: §1 · p. 1
  • Prior label-wise graphs omit spatial interactions and long-range context, while high-level semantic features can be unstable and class-unspecific.Static co-occurrence relations can also disadvantage objects with less-frequent co-occurrences.Source: §1 · p. 2
  • TDRG combines a Transformer-based structural graph for cross-scale, position-wise context with a semantic graph using explicit class-specific constraints and structural embeddings.The learned structural embedding enters semantic relation construction through adjacent correlation construction and feature-wise complementarity.Source: §1 · p. 2
  • The proposed collaborative structural and semantic relation framework achieves state-of-the-art results on MS-COCO and PASCAL VOC.The structural graph captures position-wise context, while the semantic graph models class-wise dependencies.Source: §1 · p. 2

2. Related Work

Prior multi-label recognition work primarily models spatial information or co-occurrence dependencies. This paper situates its approach among attention-based spatial methods, recurrent and graph-based label-dependency methods, and Transformer work that had seen less use in multi-label recognition.

2. Related Work

  • Spatial-information methods use proposals or attention to locate objects and strengthen spatial representations, with proposal localization noted as unstable without extra annotations [14].Attention-based approaches include self-attention for label spatial relationships [46] and a proposal-free iterative pipeline [34].Source: §2 · p. 2
  • RNN methods model label dependencies sequentially [32] [24] [4] [40], while GCNs exploit pair-wise label relationships [8] [5] [33] [41] [42] [3].The passage notes that sequential RNN methods explore limited relationships between labels.Source: §2 · p. 2
  • Transformers use self-attention and had been applied to image recognition and object detection [11] [1], but were less explored for relationship modeling in multi-label recognition.Source: §2 · p. 3

3. Approach

The approach jointly models position-wise structural and class-wise semantic dependencies, then combines the two relation graphs through collaborative learning.

3.1. Structural Relation Graph

  • Cross-scale attention transformers capture long-term context across positions and scales, balancing small-object perception against the computation and noise of high-resolution features.Features from different scales are fused position-wise, then processed by weight-sharing transformer units and concatenated.Source: §3.1 · p. 4

3.2. Semantic Relation Graph

  • The joint relation graph combines semantic-aware and structural vectors to build learnable correlations that encode semantic classes and spatial context.The graph then models joint co-occurrence dependencies over the combined nodes using a graph convolutional network.Source: §3.2 · p. 5
  • Class-specific vectors form semantic-aware features under explicit classification constraints; top-k max-pooling and an auxiliary loss target more accurate activation maps with less noise.The vectors are formed by fusing class maps with reduced image features.Source: §3.2 · p. 5

3.3. Learning Objective

  • Collaborative learning supervises the semantic constraints and each graph's predictions, as well as the final fused prediction, with classification losses.The objective sums L_joint, L_sac, L_trans, and L_gcn.Source: §3.3 · p. 6

4. Experiments

The method achieves the best reported VOC 2007 mAP, outperforming ML-GCN by 1.0%, and reports a new performance leaderboard on MS-COCO. Experiments assess two benchmarks and ablate the structural and semantic relation components.

4.1. Datasets and Evaluation Metrics

  • The experiments evaluate MS-COCO on its validation set and VOC 2007 on its test set, using per-class and overall precision, recall, F1, and mAP.The paper identifies AP and mAP as the most comprehensive comparison metrics; precision, recall, and F1 use a threshold of 0.5.Source: §4.1 · p. 6

4.2. Implementation Details

  • Training uses an ImageNet-pretrained ResNet-101, 448 × 448 test inputs, and a 50-epoch schedule; the structural relation graph receives weight α = 0.7.Experiments use SGD and two NVIDIA Tesla V100 GPUs.Source: §4.2 · p. 6

4.3. Comparison with State-of-the-art

  • 1.0%: on VOC 2007, the approach achieves the best mAP and outperforms ML-GCN by 1.0%.Table 2 compares methods on VOC 2007 at 448 × 448 resolution; the paper reports its three MS-COCO results as a new leaderboard with a clear margin.Source: §4.3 · p. 7, Table 2 · p. 7

4.4. Performance Analysis

  • On MS-COCO, Transformer-based structural relation improves performance by 4.3%, and cross-scale attention adds 1.0%.Replacing multiplication with summation drops performance by 0.7%; replacing Transformer units with an MLP lowers R_structural mAP by 0.6%.Source: §4.4 · pp. 7–8, Table 4 · p. 7
  • Semantic-relation ablations on MS-COCO report gains of 3.9% from GCN, 1% from semantic-aware constraints, 0.6% from a learnable correlation matrix, and 0.5% from structural guidance.The final semantic relation uses top-k max-pooling with a 5% threshold.Source: §4.4 · p. 8, Table 5 · p. 8
  • Grad-CAM visualizations show improved localization of small objects and handling of visually similar labels with the structural relation module.Examples include fork and spoon localization, and distinguishing fork/knife/spoon and backpack/handbag or skateboard/snowboard labels.Source: §4.4 · p. 8, Figure 5 · p. 8

5. Conclusion

The paper concludes that its Transformer-based Dual Relation Graph combines long-term contextual structural relations with dynamic semantic co-occurrences and semantic-aware constraints. The joint approach achieves new state-of-the-art results on two multi-label recognition benchmarks.

5. Conclusion

  • TDRG combines Transformer-based, cross-scale structural relations with dynamic semantic co-occurrences and semantic-aware constraints, achieving new state-of-the-art results on two benchmarks.The structural graph builds position-wise relationships across scales; the semantic graph models dynamic co-occurrences.Source: §5 · p. 8

Related papers

Loading 2110.04722v2…