PaperlayerView exact revision on arXiv

Source-linked AI summary

Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation

Rongjie Li, Songyang Zhang, Bo Wan, Xuming He

arXiv:2104.00308v2cs.CVcs.AI

TL;DR

Scene graph generation remains difficult because long-tailed distributions, scarce annotations, and predicate variation bias relationship predictions. The paper addresses this with a confidence-aware bipartite graph network using adaptive message propagation and bi-level resampling, achieving superior or comparable performance across three scene graph datasets.

  • Problem

    Long-tailed class distributions, scarce annotations, and large predicate variation make scene graph generation biased toward head categories and difficult for tail categories.

  • Method

    The method combines a confidence-aware bipartite graph neural network with adaptive message propagation, multi-stage refinement, and bi-level data resampling.

  • Results

    The method achieves competitive or state-of-the-art performance on Visual Genome, Open Images V4, and Open Images V6.

  • Takeaways & Limitations

    The results support the effectiveness and generality of the approach for unbiased scene graph generation across several challenging benchmarks.

Abstract

from arXiv · show

Scene graph generation is an important visual understanding task with a broad range of vision applications. Despite recent tremendous progress, it remains challenging due to the intrinsic long-tailed class distribution and large intra-class variation. To address these issues, we introduce a novel confidence-aware bipartite graph neural network with adaptive message propagation mechanism for unbiased scene graph generation. In addition, we propose an efficient bi-level data resampling strategy to alleviate the imbalanced data distribution problem in training our graph network. Our approach achieves superior or competitive performance over previous methods on several challenging datasets, including Visual Genome, Open Images V4/V6, demonstrating its effectiveness and generality.

1. Introduction

Scene graph generation is a fundamental visual understanding task, but long-tailed data, scarce annotations, and large predicate variation cause biased relationship prediction. The paper proposes confidence-aware bipartite message passing with adaptive propagation and bi-level resampling, achieving competitive or state-of-the-art benchmark performance.

  • Scene graph generation detects objects and their relationships as structured triplets, supporting applications such as visual question answering, image captioning, and image retrieval.
  • Intrinsic long-tailed distributions cause models to favor head categories, while scarce annotations and few-shot tail categories degrade tail-category performance.
  • Complex predicate semantics and large intra-class variation make non-head categories difficult to represent and classify.
  • Fully connected context graphs can propagate noisy information, whereas sparse structures may limit model flexibility.
  • Bi-level resampling combines image-level over-sampling and instance-level under-sampling to improve the trade-off between head and tail categories.
  • The method achieves competitive or state-of-the-art performance across Visual Genome, Open Images V4, and Open Images V6, supported by empirical and ablation studies.
  • The proposed bipartite graph neural network uses adaptive message propagation to improve context modeling and reduce error propagation.

2. Related Works

Prior scene graph generation methods model context through sequential, fully connected, or sparse structures and use varied message-passing mechanisms. This work unifies connectivity inference and message passing in a bipartite graph framework, while applying bi-level sampling to long-tailed prediction.

  • Existing scene graph generation methods use graph-based context modeling for node and edge prediction, focusing on graph structure or message propagation.
  • Prior context structures include sequential models, fully connected graphs, and sparse graphs tailored to tasks or proposal information.
  • Message passing commonly occurs between entity proposals or between entities and predicates to build scene graph representations.
  • BGNN jointly considers message passing and network connectivity, with earlier scene graph models representable as special cases of the framework.
  • Long-tail recognition methods include dataset resampling, loss re-weighting, knowledge transfer, and two-stage learning.
  • Naive re-balancing is difficult for scene graph generation, motivating bi-level sampling that combines image-level over-sampling with instance-level under-sampling.
  • Scene graph long-tail approaches also include semantic loss designs, graph-based context encoding, external knowledge, and causal unbiased inference.

3. Our Approach

The approach combines a bipartite graph network with confidence-aware adaptive message propagation and bi-level resampling for unbiased scene graph generation. It models entity–predicate context while controlling noisy information flow and balancing head and tail categories.

  • Overview: Scene graph generation parses an image into object entities and their visual relationships, represented as a structured graph.The method follows a hypothesize-and-classify pipeline that generates entity and predicate proposals before predicting their categories.
  • Bipartite Graph Neural Network: BGNN explicitly models interactions between entity and predicate representations through a directed bipartite graph.The two node groups are connected to represent different information-flow directions between entities and predicates.
  • Overview: The model uses a modular pipeline comprising proposal generation, multi-stage BGNN context encoding, and scene graph prediction.The proposal network generates entity and relationship proposals, the BGNN performs adaptive iterative refinement, and the predictor decodes the graph.
  • Bipartite Graph Neural Network: Relationship confidence estimation supplies predicate confidence scores that control adaptive message propagation through entity-to-predicate and predicate-to-entity updates.The confidence-aware mechanism is designed to reduce noise from false-positive predicate proposals, using hard or soft gating according to confidence.
  • Learning with Bi-level Data Sampling: Bi-level resampling combines image-level oversampling with instance-level predicate undersampling to improve the trade-off between head and tail categories.The strategy computes image repeat factors from category frequency and dropout rates for predicate instances.

4. Experiments

Experiments evaluate BGNN across Visual Genome and Open Images using recall, mean recall, and weighted relationship metrics, including long-tail group breakdowns. Results show strong benchmark performance, with ablations supporting adaptive propagation, multi-stage refinement, and bi-level sampling.

  • Evaluation protocol: The evaluation covers PredCls, SGCls, and SGGen, using R@K, mR@K, and head/body/tail mean-recall breakdowns on Visual Genome.Open Images evaluation additionally uses mR@50, R@50, wmAPrel, wmAPphr, and scorewtd.
  • Visual Genome results: BGNN achieves state-of-the-art mR@50/100 across all three Visual Genome subtasks on the X-101-FPN backbone.It exceeds Unbiased by 5.0 and 4.2 on PredCls, while showing comparable R@50/100 performance in SGCls and SGGen.
  • Visual Genome results: BGNN significantly outperforms prior methods on the Visual Genome SGGen tail group and obtains the highest mean recall across all groups.On VGG16, it also surpasses PCPL by a large margin on SGGen while remaining comparable on SGCls and PredCls.
  • Ablation study: 12.6 mR@100 is reached after adding bipartite propagation, confidence-aware propagation, and multi-stage refinement to the baseline.The intermediate values are 10.5 with plain propagation and 11.7 after adding confidence awareness.
  • Ablation study: BLS improves especially on body and tail categories while maintaining head-category performance, and three message-passing iterations with three stages provide the best refinement setting.The stage-based performance saturates at Nt=3 when Ni=3.
  • Open Images results: On Open Images V4 and V6, the method achieves competitive weighted scores and state-of-the-art or significantly improved mean recall.V4 has nine predicate categories, whereas V6 has 31, enabling broader evaluation of long-tail scene graph generation.

5. Conclusion

The conclusion presents BGNN as a bipartite graph neural network for unbiased scene graph generation, combining confidence-aware message passing with bi-level resampling. Across three scene graph datasets, it reports superior or comparable performance to prior state-of-the-art methods.

  • Conclusion: BGNN combines confidence-aware message passing for scene-context encoding with bi-level resampling to mitigate imbalanced training data.These are identified as the method’s two key components.
  • Conclusion: Across three scene graph datasets, BGNN achieves superior or comparable performance to prior state-of-the-art approaches.The conclusion states this result without restricting it to a single evaluation task.

A. Relation Confidence Estimation Module

The relation confidence estimation module is introduced as an important component and is trained with predicate-category and predicate-validity supervision. Its multi-task objective uses multi-category and binary confidence predictions.

  • Module design: RCE predicts both multi-category confidence scores and binary confidence scores for predicate proposals.Predicate class labels and validity-versus-background labels supervise the module.
  • Training: The RCE module uses a multi-task loss distinct from the cross-entropy loss used for final predicate classification.The described supervision includes one-hot predicate labels and binary labels for positive predicate proposals.

A.2. Performance

The RCE module is evaluated for removing negative predicate proposals using AUC on the Visual Genome validation set. It outperforms entity prediction scores and Graph-RCNN’s relation proposal network in that comparison.

  • Performance: 0.839 AUC is achieved by RCE on the Visual Genome validation set, versus 0.629 for entity prediction scores and 0.671 for Graph-RCNN’s relation proposal network.The comparison evaluates effectiveness at removing negative predicate proposals.

B. Model Comparison w/ Resampling

The study compares BGNN and bi-level resampling against recent methods under matched resampling conditions. Results and supplementary visual analyses assess whether the approach remains effective across scene graph generation benchmarks.

  • BGNN is compared with reimplemented GPS-Net and MSDN after adding the proposed bi-level resampling strategy.The comparison targets the SGGen task.
  • BGNN still outperforms other approaches under the same resampling strategy.
  • Bi-level resampling improves all three evaluated methods, supporting its effectiveness beyond BGNN alone.
  • Table 6 reports performance comparisons between state-of-the-art methods and the proposed bi-level sampling strategy.
  • Supplementary analyses include long-tail partition details, per-class PredCls comparisons, and visualized predictions against GPS-Net.

C.1. Long-tail Categories Groups Partition

The supplementary analysis partitions predicate categories by training frequency and examines performance across head, body, and tail groups. It reports stronger effects from reducing context-modeling noise on tail categories and improved tail performance on Open Images.

  • Visual Genome: Visual Genome categories are divided into head, body, and tail groups using training-instance thresholds of over 10k, 0.5k–10k, and under 0.5k.
  • Visual Genome: Reducing context-modeling noise improves the baseline by a large margin, especially for tail categories with few data points.
  • Open Images: Open Images V6 categories are partitioned into head, body, and tail groups using thresholds of over 12k, 0.2k–12k, and under 0.2k training instances.
  • Open Images: On Open Images, the method significantly improves tail-category performance while achieving comparable overall performance with GPS-Net and Causal.

C.2. Per-class Performance Comparison with the Other Models

Per-class PredCls evaluation compares the proposed model with Causal and GPS-Net across all 50 Visual Genome categories. The reported gain is concentrated in low-frequency categories.

  • Recall@100 is compared for each PredCls category against the two state-of-the-art methods, Causal and GPS-Net.
  • The evaluation covers all 50 Visual Genome categories rather than only the top-35 frequency categories.
  • The model achieves a significant performance gain on low-frequency categories, supporting the effectiveness of BGNN.

C.3. Visualization of Model Prediction

The visualization analysis examines scene-graph predictions and long-tail performance comparisons across Visual Genome and Open Images. It highlights stronger performance for body and tail categories and compares per-class Recall@100 with prior methods.

  • Qualitative visualization: Visual Genome scene-graph predictions show significant improvement for body and tail category groups compared with GPS-Net.
  • Qualitative visualization: Confidence-aware message propagation is associated with better context modeling of visual representations for low-frequency categories.
  • Visual Genome: Figure 7 presents the long-tail category partition and upper-bound comparison on Visual Genome.
  • Open Images: Figure 8 presents long-tail partitions and per-class performance comparisons on Open Images V4 and V6 against Causal and GPS-Net.
  • PredCls comparison: Figure 9 compares Recall@100 on PredCls across all categories with Causal and GPS-Net, indicating where resampling is applied.
Loading 2104.00308v2…