PaperlayerView exact revision on arXiv

Source-linked AI summary

GPS-Net: Graph Property Sensing Network for Scene Graph Generation

Xin Lin, Changxing Ding, Jinquan Zeng, Dacheng Tao

arXiv:2003.12962v1cs.CV

TL;DR

SGG must model directed relationships, unequal node priority, and long-tailed relationship frequencies. GPS-Net addresses these properties with direction-aware message passing, node-priority-sensitive loss, and adaptive relationship-frequency modeling, achieving top-level performance across three datasets, including a 2.4% gain on OpenImages.

  • Problem

    SGG involves complex relationship characterization, directed edges, unequal node importance, and imbalanced training data, while these graph properties remain under-explored.

  • Method

    GPS-Net uses DMP for direction-aware, node-specific contextual message passing, NPS-loss for node priority, and ARM for long-tailed relationship distributions.

  • Results

    GPS-Net achieves top-level performance across VG, OI, and VRD, outperforming RelDN by 2.4% on OpenImages’ overall metric scorewtd.

  • Takeaways & Limitations

    The experiments and ablations validate GPS-Net’s three-property design for SGG across multiple datasets and evaluation settings.

  • Takeaways & Limitations

    DMP treats edge direction as unclear during message passing because the relationship between two nodes is still unknown.

Abstract

from arXiv · show

Scene graph generation (SGG) aims to detect objects in an image along with their pairwise relationships. There are three key properties of scene graph that have been underexplored in recent works: namely, the edge direction information, the difference in priority between nodes, and the long-tailed distribution of relationships. Accordingly, in this paper, we propose a Graph Property Sensing Network (GPS-Net) that fully explores these three properties for SGG. First, we propose a novel message passing module that augments the node feature with node-specific contextual information and encodes the edge direction information via a tri-linear model. Second, we introduce a node priority sensitive loss to reflect the difference in priority between nodes during training. This is achieved by designing a mapping function that adjusts the focusing parameter in the focal loss. Third, since the frequency of relationships is affected by the long-tailed distribution problem, we mitigate this issue by first softening the distribution and then enabling it to be adjusted for each subject-object pair according to their visual appearance. Systematic experiments demonstrate the effectiveness of the proposed techniques. Moreover, GPS-Net achieves state-of-the-art performance on three popular databases: VG, OI, and VRD by significant gains under various settings and metrics. The code and models are available at \url{https://github.com/taksau/GPS-Net}.

1. Introduction

GPS-Net targets three underexplored scene-graph properties—edge direction, node priority, and long-tailed relationship frequencies—through specialized modules and achieves top-level performance across VG, OI, and VRD.

  • Motivation: SGG represents objects as labeled nodes and relationships as directed predicate edges, but relationship complexity and imbalanced training data make it challenging.Edge direction identifies subject and object and affects relationship classes and contextual information.
  • DMP: DMP uses a tri-linear Tucker-based model to encode edge direction while producing node-specific contextual information for message passing.It augments the attention map with its transpose to account for uncertain edge directions.
  • NPS-loss: NPS-loss adjusts focal-loss focusing parameters according to node frequency in triplets, directing more training attention toward high-priority nodes.The differentiable design is intended to be easier to optimize and deploy than a non-differentiable local-sensitive loss.
  • ARM: ARM softens the long-tailed relationship-frequency distribution with log-softmax and adaptively modifies it for each subject-object pair using visual appearance.This addresses errors such as confusing sitting on with the more frequent has.
  • Evaluation: GPS-Net combines DMP, NPS-loss, and long-tail relationship handling, with systematic evaluation reporting consistently top-level performance on VG, OI, and VRD.The paper presents these three components as its core innovations and evaluates them across three popular SGG databases.

2. Related Work

Prior SGG work models global or object-specific context and often overlooks intrinsic scene-graph properties. GPS-Net instead emphasizes those properties while remaining a one-stage method that is easy to optimize and deploy.

  • Visual Context Modeling: Visual context methods model either global context through feature rescaling or recurrent networks, or object-specific context through pixel-level self-attention.The related-work discussion identifies SENet, PSANet, Neural Motif, NLNet, and CCNet as examples.
  • Visual Context Modeling: First-order linear models may fail to learn object-specific context, motivating GPS-Net’s direction-aware message passing with a tri-linear model.The proposed module generates node-specific context while incorporating edge direction.
  • Scene Graph Generation: Existing one-stage SGG methods generally focus on detection and relationship representation while largely ignoring edge direction and node priority.Two-stage methods add training stages to refine results, whereas GPS-Net is presented as a one-stage alternative.
  • GPS-Net Framework: GPS-Net’s framework combines Faster R-CNN proposals with DMP, NPS-loss, and ARM to address contextual modeling, node priority, and relationship-frequency imbalance.The framework caption identifies these three modules and their respective roles.
  • Scene Graph Generation: Prior approaches use non-differentiable loss functions for some scene-graph properties, while GPS-Net is described as easier to optimize and deploy to existing models.The comparison specifically contrasts the proposed approach with methods using non-differentiable losses.

3. Approach

GPS-Net addresses direction, node priority, and relationship-frequency properties in scene graphs through direction-aware message passing, priority-sensitive loss, and adaptive relationship reasoning.

  • 3.1. Direction-aware Message Passing: DMP combines direction-aware context modeling with a transformer layer to produce node-specific contextual features for message passing.The module is designed to address the limitations of earlier message-passing modules that ignore edge direction and node-specific context.
  • 3.1. Direction-aware Message Passing: The DMP contextual coefficient uses a tri-linear Tucker-decomposition model that jointly couples node and union-box features through Hadamard products.This model expands the contextual receptive field and captures higher-order interactions among the feature types.
  • 3.1. Direction-aware Message Passing: DMP represents both forward and backward contextual coefficients because the relationship direction between two nodes is unknown during message passing.The two directional coefficients are stacked as a two-element vector before producing the module output.
  • 3.2. Node Priority Sensitive Loss: NPS-loss adjusts focal-loss focusing parameters from each node’s triplet frequency, making the differentiable training objective sensitive to node priority.A nonlinear mapping is used because a linear transformation exaggerates differences between high- and middle-priority nodes while narrowing differences between middle- and low-priority nodes.
  • 3.3. Adaptive Reasoning Module: ARM softens relationship-frequency priors with log-softmax and adapts them to each node pair using visual appearance.The original frequency vector is determined by the object classes of the two nodes, while the adaptive mechanism modifies the prior for the pair.

4. Experiments

Experiments evaluate GPS-Net on VG, OI, and VRD using established protocols and compare it with state-of-the-art methods. Results show consistent gains across datasets, metrics, and ablations, including improved handling of class imbalance.

  • Evaluation Settings: GPS-Net is evaluated on Visual Genome, OpenImages, and Visual Relationship Detection using established dataset-specific protocols and metrics.VG uses SGDET, SGCLS, and PREDCLS; OI uses R@50, wmAPrel, and wmAPphr; VRD reports R@50 and R@100 across relationship, predicate, and phrase detection.
  • Comparisons with State-of-the-Art Methods: 1.8% average gain over KERN at R@50 and R@100 across VG protocols, with additional gains over CMAT, VCTREE, and RelDN.On VG, GPS-Net improves over KERN by 1.9%, 2.7%, and 1.2% at R@100 for SGDET, SGCLS, and PREDCLS, respectively.
  • Comparisons with State-of-the-Art Methods: 2.4% higher scorewtd than RelDN on OI, including APrel gaps of 24.5% for wears and 20.6% for hits.The comparison uses the same object detector and reports performance under severe class imbalance.
  • Comparisons with State-of-the-Art Methods: GPS-Net consistently achieves superior performance on VRD with both backbone models used for fair comparison.The evaluation follows the VRD metrics for relationship, predicate, and phrase detection.
  • Ablation Studies: Adding DMP, NPS-loss, and ARM sequentially improves the baseline, while DMP stacking and NPS-loss further improve performance across tested settings.NPS-loss performs best with µ = 4 and outperforms focal loss; DMP stacking consistently improves DMP across metrics.
  • Ablation Studies: DMP outperforms GCMP and S-GCMP because it encodes edge direction and provides node-specific contextual information during message passing.Qualitative comparisons likewise show diverse attention contexts for DMP rather than the similar contexts produced by the other modules.

5. Conclusion

GPS-Net addresses three scene-graph properties—edge direction, node priority, and long-tailed relationship frequencies—through dedicated mechanisms and validates them across three datasets.

  • GPS-Net encodes edge direction through node-specific contextual information in DMP, represents node priority with NPS-loss, and improves relationship-frequency usage through ARM.
  • The method is evaluated with comparative experiments and ablation studies on VG, OI, and VRD.

Appendix

The appendix adds mean-recall comparisons and qualitative analyses of generated scene graphs and message-passing attention maps.

  • Section A: Section A reports detailed Mean Recall comparisons on VG and Recall results across different numbers of relationship predictions per object pair on VRD.
  • Section B: Section B compares GPS-Net qualitatively with MOTIFS under SGDET and visualizes attention maps from different message-passing modules.

A.1. Mean Recall for Scene Graph on VG

On VG, GPS-Net achieves the best performance under all protocols and evaluates recall across multiple predictions per object pair because several predicates may be plausible.

  • 2.3% is GPS-Net’s average improvement over VCTREE-HL across the three VG protocols.
  • The appendix reports recall for k=1 and k=70 because each object pair may have several plausible predicates, making top-1-only evaluation potentially unreasonable.

B.1. Generated Scene Graph

Qualitative comparisons associate GPS-Net’s different modules with improved predictions for low-priority nodes, high-priority nodes, and low-frequency relationships.

  • GPS-Net predicts low-priority nodes and high-frequency relationships better than MOTIFS, attributed to DMP’s edge-direction encoding and node-specific context.
  • GPS-Net makes fewer mistakes for high-priority nodes than MOTIFS, a gain attributed to NPS-loss.
  • ARM improves predictions for low-frequency relationships such as walking on and wearing.

B.2. Attention Maps of Different MP Modules

Figure 7 compares attention maps from GCMP, S-GCMP, and DMP against object regions and the ground-truth relationship matrix. DMP produces node-specific context and an attention map consistent with the relationship structure.

  • Attention-map comparison: DMP produces diverse attention patterns across nodes, unlike the similar context generated by GCMP and S-GCMP.In the attention maps, elements within each column are diverse for DMP but similar for the other modules.
  • Attention-map comparison: DMP’s attention map closely matches the ground-truth relationship matrix.The comparison uses object regions and a relationship matrix in which yellow indicates a relationship and purple indicates the opposite.
  • Interpretation: The qualitative comparison identifies DMP as a key component supporting GPS-Net’s scene graph generation performance.
Loading 2003.12962v1…