PaperlayerView exact revision on arXiv

Source-linked AI summary

Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation

Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, Liqiang Nie

arXiv:2203.09811v2cs.CVcs.AI

TL;DR

SGG remains limited by insufficient vision-language fusion and biased relationship predictions. The paper proposes Stacked Hybrid-Attention for the encoder and Group Collaborative Learning for the decoder. On VG and GQA, it reports a new state-of-the-art in the unbiased metric and nearly double performance over two baselines.

  • Problem

    Existing SGG approaches often fuse visual contents and semantic cues simply and predict biased relationships, limiting practical use.

    Source: §1 · pp. 1–2

  • Method

    Stacked Hybrid-Attention combines self-attention and cross-attention in the encoder, while Group Collaborative Learning trains classifiers over predicate groups with parallel optimization and knowledge distillation.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Results

    On VG and GQA, the paper reports a new state-of-the-art in the unbiased metric and nearly double performance compared with two typical baselines using model-agnostic GCL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents GCL as a model-agnostic debiasing method and reports a reasonable trade-off between improving tail predicates and preserving head-class discrimination.

    Source: §5 · p. 8

    GCL’s collaborative knowledge distillation uses a regular Kullback-Leibler Divergence loss, and the authors identify more efficient distillation strategies as a potential improvement.

    Source: §5.2 · p. 11

Abstract

from arXiv · show

Scene Graph Generation, which generally follows a regular encoder-decoder pipeline, aims to first encode the visual contents within the given image and then parse them into a compact summary graph. Existing SGG approaches generally not only neglect the insufficient modality fusion between vision and language, but also fail to provide informative predicates due to the biased relationship predictions, leading SGG far from practical. Towards this end, in this paper, we first present a novel Stacked Hybrid-Attention network, which facilitates the intra-modal refinement as well as the inter-modal interaction, to serve as the encoder. We then devise an innovative Group Collaborative Learning strategy to optimize the decoder. Particularly, based upon the observation that the recognition capability of one classifier is limited towards an extremely unbalanced dataset, we first deploy a group of classifiers that are expert in distinguishing different subsets of classes, and then cooperatively optimize them from two aspects to promote the unbiased SGG. Experiments conducted on VG and GQA datasets demonstrate that, we not only establish a new state-of-the-art in the unbiased metric, but also nearly double the performance compared with two baselines.

1. Introduction

The paper identifies insufficient vision-language fusion and biased relationship predictions as obstacles to practical scene graph generation. It proposes an attention-based encoder and collaborative classifier training to address these issues.

1. Introduction

  • The paper frames insufficient modality fusion and biased relationship predictions as problems that leave SGG far from practical applications.Long-tailed training data lets a few head predicates dominate, while prior debiasing approaches risk over-fitting tail classes and sacrificing head classes.Source: §1 · pp. 1–2
  • Stacked Hybrid-Attention uses self-attention for intra-modal refinement and cross-attention for inter-modal interaction, addressing simple visual-semantic feature fusion.The paper organizes the attention units into hybrid-attention layers and stacks them to form the encoder.Source: §1 · p. 1
  • Group Collaborative Learning divides predicates into relatively balanced groups, then cooperatively trains classifiers through parallel optimization and collaborative knowledge distillation.Median Re-Sampling provides each classifier with a relatively balanced training set; parallel optimization gathers gradients, while distillation transfers prior classifiers’ discriminating capability.Source: §1 · p. 2
  • On VG and GQA, the model establishes a new state-of-the-art in the unbiased metric and nearly doubles performance compared with two typical baselines when using model-agnostic GCL.The reported result is attributed to the Group Collaborative Learning strategy.Source: §1 · p. 2

2. Related Work

Prior SGG work has explored richer features and contextual encoders, but biased training data can still yield trivial, less informative relations. The paper positions its attention encoder and collaborative decoder training as responses to gaps in modality fusion and predicate bias.

2. Related Work

  • Earlier SGG approaches used message passing, LSTMs, graph neural networks, and self-attention for context, while predicted relations often remained trivial and less informative under biased data.Debiasing methods included re-sampling, re-weighting, disentangling unbiased representations, and tree-based predicate filtering, but could over-fit tail classes at the expense of head classes.Source: §2 · p. 2
  • Figure 2 places SHA in the object and relation encoders and GCL in the relation decoder within the common five-component SGG pipeline.The highlighted components are the two encoders and the relation decoder.Source: Figure 2 · p. 3
  • Few SGG approaches address insufficient fusion between object proposals and their class names, motivating SHA’s intra-modal refinement and inter-modal interaction.The passage contrasts this SGG gap with cross-attention work in vision-and-language tasks such as VQA and joint image-text pre-training.Source: §2 · p. 3
  • Unlike conventional teacher-student distillation, the described knowledge transfer uses earlier classifiers’ outputs as soft labels for subsequent classifiers.This establishes a layer-to-layer transfer between classifiers added to the same model.Source: §2 · p. 3

3. Methodology

The method retains a standard encoder–decoder SGG pipeline, modifying its encoders with SHA and its relation decoder with GCL. SHA combines intra-modal refinement with inter-modal interaction, while GCL groups predicates and cooperatively trains classifiers.

3.1. Problem Formulation

  • SGG forms a graph by predicting a predicate for each object pair and organizing the resulting object–predicate–object triplets.The formulation represents the graph as G = {(o_i, p_i→j, o_j)}.Source: §3.1 · p. 3

3.2. Overall Framework

  • The encoder–decoder framework refines object features from visual, spatial, and initial-label semantic features, then predicts pairwise predicates using final object features and their union feature.The object encoder can use feature-refinement modules such as BiLSTMs or GNNs; the relation encoder shares the object encoder’s architecture.Source: §3.2 · pp. 3–4

3.3. Encoder: Stacked Hybrid-Attention

  • SHA stacks layers with self-attention for intra-modal refinement and cross-attention for inter-modal interaction, then sums the final visual and semantic features.Each layer contains two parallel hybrid-attention cells.Source: §3.3 · p. 4

3.4. Decoder: Group Collaborative Learning

  • GCL sorts predicates by training frequency into groups whose maximum and minimum instance counts are bounded by threshold µ, then grows classifier spaces by adding groups.Balanced re-sampling retains instances for newly added predicates while under-sampling predicates in the previous classification space.Source: §3.4 · pp. 4–5
  • During training, PCO jointly optimizes classifiers and CKD transfers specialist predictions to later classifiers; only the final classifier predicts at evaluation.Together, these constraints aim to mitigate excessive penalties on tail classes and compensate for head-class under-fitting.Source: §3.4 · pp. 5–6

4. Experiments

Experiments evaluate SHA+GCL on VG150 and GQA200 across three SGG tasks, with comparisons against existing methods and ablations of its components. SHA+GCL performs strongly on unbiased metrics, while parameter analysis favors the µ=4, Top-Down configuration.

4.1. Experimental Settings

  • Experiments use VG150 and cleaned GQA200 across PredCls, SGCls, and SGDet, comparing SHA+GCL with SGG systems and model-agnostic debiasing methods.Evaluation uses mean Recall@K for unbiased SGG.Source: §4.1 · p. 6, §4.2 · p. 6

4.2. Compared Methods

  • SHA+GCL significantly outperforms baselines across all three tasks, while Motifs+GCL and VCTree+GCL nearly double mean Recall over their original models.The comparison covers VG150 and GQA200; SHA+GCL also gains over the GCL-equipped baselines.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

4.3. Ablation Study

  • SHA+GCL nearly doubles performance over w/o-GCL; PCO aids convergence, CKD preserves head discrimination, and removing either attention unit degrades results.Figure 5 shows gains on most predicate classes, with acceptable decay on head classes in Groups 1 and 2; CKD yields comparable tail predictions.Source: §4.3 · pp. 7–8, Table 3 · p. 8, Figure 5 · p. 8

4.4. Parameter Analysis

  • The analysis tests µ=3/4/5, yielding six/five/four groups, and Adjacent versus Top-Down matching; µ=4 with Top-Down performs best.Adjacent transfers knowledge from the nearest predecessor; Top-Down uses all predecessors.Source: Table 4 · p. 8, §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

The paper identifies insufficient modality fusion and biased relationship predictions as constraints on practical SGG, and proposes Stacked Hybrid-Attention and Group Collaborative Learning to address them.

5. Conclusion

  • The paper identifies insufficient modality fusion and biased relationship predictions as concerns restricting SGG's practical applications.Source: §5 · p. 8
  • The proposed Stacked Hybrid-Attention network and Group Collaborative Learning strategy address these two concerns.Source: §5 · p. 8
  • The paper reports a new state of the art on the unbiased metric and presents GCL as a model-agnostic debiasing method.Future work will explore more robust group-dividing methods and knowledge-distillation strategies.Source: §5 · p. 8

1. Introduction

The supplementary material presents further analyses, experiments, and visualizations, and discusses limitations and future work.

1. Introduction

  • The supplementary material presents additional analyses of the method.Source: §1 · p. 11
  • It also presents additional experiments and visualization results.Source: §1 · p. 11
  • The supplementary material discusses the method's limitations and future work.Source: §1 · p. 11

2. Parameter Statistics

The parameter comparison covers Motifs, VCTree, and SHA alongside versions enhanced with model-agnostic GCL; the reported additional parameter count is small relative to the baselines.

2. Parameter Statistics

  • The parameter comparison covers Motifs, VCTree, and SHA, each with and without model-agnostic GCL.Source: §2 · p. 11
  • About 2M parameters are added by GCL, compared with about 200M total trainable parameters in the original methods.Source: §2 · p. 11
  • The authors state that GCL's additional parameters could hardly influence the overall training procedure.Source: §2 · p. 11

3. Detailed Performance

SHA+GCL achieves the best unbiased mR@K performance across three SGG tasks on VG150 and GQA200. On VG150, the authors report specified mR@K precision thresholds, and GCL-equipped methods nearly double performance over their originals.

3. Detailed Performance

  • The experiments report Recall@K, unbiased Mean Recall@K, and their mean for PredCls, SGCls, and SGDet on VG150 and GQA200, using the same pre-trained detector for fair comparison.Source: §3 · p. 11
  • SHA+GCL achieves the best mR@K performance on all three tasks in both datasets; on VG150 it exceeds 40% on PredCls mR@50 and mR@100, and reaches 20% on SGCls and SGDet mR@100.Source: §3 · p. 11
  • In VG150 and GQA200, methods equipped with GCL nearly double performance compared with their original versions.Source: §3 · p. 11

4. Visualization Results

Visual examples show SHA+GCL producing more informative predicate predictions and capturing plausible relationships absent from the ground truth annotations.

4. Visualization Results

  • SHA+GCL favors specific predicates such as “riding” over common ones such as “on” in the PredCls examples.The examples include “person1-riding-elephant” and “train-pulling-car.”Source: §4 · p. 11
  • Figure 7 presents group-split as a promising alternative to the group-incremental configuration for the “conquer” step.The authors plan to explore more robust group-dividing methods and classifier configurations.Source: Figure 7 · p. 11
  • SHA+GCL also predicts reasonable relationships that are not annotated in the ground truth.Examples include “person1-watching-person2” and “sidewalk-beside-train.”Source: §4 · p. 11

5. Limitations and Future Work

The limitations discussion identifies classifier-configuration and knowledge-distillation choices as directions for further improvement of SHA+GCL.

5. Limitations and Future Work

  • The section frames its limitations discussion as a basis for potential improvements to SHA+GCL.It covers alternative classifier configurations and stronger collaborative knowledge distillation.Source: §5 · p. 11, §5.1 · p. 11, §5.2 · p. 11

5.1. More Configurations Could be Further Explored

  • The current “conquer” step uses group-incremental mainly for simplicity and efficiency, while group-split is another configuration to explore.The authors plan to investigate more robust group-dividing methods and classifier configurations.Source: §5.1 · p. 11

5.2. “Strong Constraint” Could be Further Enhanced

  • The “cooperate” step uses a regular Kullback-Leibler Divergence loss, and more efficient distillation strategies could strengthen the “Strong Constraint.”The authors identify new knowledge-distillation strategies as a direction for improving GCL.Source: §5.2 · p. 11

Related papers

Loading 2203.09811v2…