PaperlayerView exact revision on arXiv

Source-linked AI summary

SGTR: End-to-end Scene Graph Generation with Transformer

Rongjie Li, Songyang Zhang, Xuming He

arXiv:2112.12970v3cs.CVcs.LG

TL;DR

SGG must model compositional relationships while avoiding the complexity and restrictive assumptions of prior designs. SGTR addresses this by constructing a bipartite graph end to end with entity-aware predicates and directed edge assembly, achieving competitive benchmark performance and efficient inference, though small entities remain a limitation.

  • Problem

    SGG remains difficult because compositional visual relationships make compact representation, localization, and classification challenging, while prior designs incur complexity or restrictive assumptions.

  • Method

    SGTR uses a transformer-based end-to-end bipartite graph framework that generates entity and predicate nodes, then assembles directed edges using entity-aware predicates.

  • Results

    SGTR outperforms or matches prior state-of-the-art methods on Visual Genome and OpenImages-V6 while maintaining efficient inference.

  • Takeaways & Limitations

    The bipartite formulation and entity-aware predicate structure provide a strong end-to-end Transformer baseline for scene graph generation.

  • Takeaways & Limitations

    SGTR performs sub-optimally on relationships involving small entities because its DETR detector is weaker on small entities than Faster R-CNN.

Abstract

from arXiv · show

Scene Graph Generation (SGG) remains a challenging visual understanding task due to its compositional property. Most previous works adopt a bottom-up two-stage or a point-based one-stage approach, which often suffers from high time complexity or sub-optimal designs. In this work, we propose a novel SGG method to address the aforementioned issues, formulating the task as a bipartite graph construction problem. To solve the problem, we develop a transformer-based end-to-end framework that first generates the entity and predicate proposal set, followed by inferring directed edges to form the relation triplets. In particular, we develop a new entity-aware predicate representation based on a structural predicate generator that leverages the compositional property of relationships. Moreover, we design a graph assembling module to infer the connectivity of the bipartite scene graph based on our entity-aware structure, enabling us to generate the scene graph in an end-to-end manner. Extensive experimental results show that our design is able to achieve the state-of-the-art or comparable performance on two challenging benchmarks, surpassing most of the existing approaches and enjoying higher efficiency in inference. We hope our model can serve as a strong baseline for the Transformer-based scene graph generation. Code is available: https://github.com/Scarecrow0/SGTR

1. Introduction

SGG models relationships between entities as scene-graph structure, but existing two-stage and one-stage designs face computational, noise, or modeling limitations. SGTR formulates SGG as bipartite graph construction with entity-aware predicates and end-to-end graph assembly.

  • Motivation: Scene graph generation represents visual relationships as structured connections between entity nodes and predicate relations.A relationship is represented as a subject entity, predicate, and object entity, enabling a compact scene representation.
  • Limitations of Prior Work: Two-stage methods generate O(N^2) predicate proposals, increasing computation and contextual noise.They first detect N entity proposals and then predict predicates for entity combinations.
  • Limitations of Prior Work: One-stage methods reduce relation proposals but assume nonoverlapping interaction regions, restricting modeling of complex scenes.This assumption limits their applicability when interaction regions overlap.
  • SGTR Approach: SGTR formulates SGG as bipartite graph construction, jointly generating entity and predicate nodes before connecting them with directed edges.The formulation supports a rich hypothesis space for inferring visual relations.
  • SGTR Approach: SGTR uses an entity-aware predicate representation to encode subject and object associations and produce fewer high-quality predicate proposals.Its transformer-based framework includes entity and predicate generators plus a graph assembling module.
  • Results: SGTR achieves state-of-the-art or comparable performance across reported metrics while providing more efficient inference.Experiments evaluate the method on Visual Genome and OpenImages-V6 against prior SGG methods.

3. Preliminary

SGG parses an image into entity nodes and predicate-labeled relations. SGTR models this process as generating two node groups and connecting them through directed edges in a differentiable bipartite graph.

  • Problem Setting: SGG parses an image into entity nodes with categories and boxes, plus predicate-labeled edges between subject-object entity pairs.The resulting scene graph is G_scene = {V_e, E_r}.
  • Bipartite Formulation: SGTR represents relationship generation as a bipartite graph with entity and predicate node groups connected by directed edge sets.The graph separates entity representations from predicate representations while retaining directional associations.
  • Bipartite Formulation: The differentiable function F_sgg maps an image to a bipartite graph through node generation followed by directed edge connection.This formulation enables end-to-end training.
  • Node Generation: Entity and predicate node generators extract their respective proposals, while the predicate generator augments predicates with entity information.The predicate generator uses three parallel sub-decoders for this augmentation.
  • Graph Assembly: A graph assembling module constructs the bipartite scene graph from the generated entity and predicate proposals.This module performs the directed edge connection step.

4. Our Approach

SGTR constructs scene graphs end to end by generating entity and entity-aware predicate nodes, then assembling their directed bipartite connections. Its structural predicate generator incorporates entity information into predicate queries, while graph assembling refines associations into final triplets.

  • Overall Architecture: SGTR uses four submodules: a backbone, transformer-based entity node generator, structural predicate node generator, and bipartite graph assembling module.The backbone produces scene features; subsequent modules generate entity proposals, decode entity-aware predicate nodes, and connect them into the final graph.
  • Entity Node Generator: The entity node generator applies a DETR decoder to learnable queries, producing entity locations, class scores, and feature representations.Each entity box is represented by normalized center coordinates, width, and height.
  • Predicate Node Generator: The predicate node generator combines predicate query initialization, predicate-specific image encoding, and structural decoding to produce entity-aware predicate nodes.A lightweight predicate encoder extracts predicate-specific features, while the structural decoder decodes predicate nodes from image and entity features.
  • Predicate Query Initialization: Predicate queries are decomposed into subject indicators, object indicators, and predicate representations, allowing entity information to be incorporated through cross-attention.The resulting structural query encodes potential predicate-entity associations for later graph-edge prediction.
  • Structural Predicate Node Decoder: The structural decoder independently refines predicate and entity-indicator components, then fuses them to improve entity-predicate associations across decoder layers.Predicate representations attend to image features, while entity indicators attend to entity proposal features from the entity generator.
  • Bipartite Graph Assembling: Graph assembling matches subject and object indicators to entity nodes, retains top-K links, and outputs the final relationship triplets.The module computes correspondence scores from localization and classification distances before selecting links and forming the scene graph.

5. Experiments

Experiments evaluate SGTR on OpenImages-V6 and Visual Genome through configuration studies, ablations, state-of-the-art comparisons, and qualitative analysis. SGTR achieves strong benchmark performance while maintaining efficient inference, though small-entity recognition remains a limitation.

  • Experiments Configuration: SGTR is evaluated on OpenImages-V6 using weighted mAP metrics and on Visual Genome using recall and mean-recall metrics.Visual Genome evaluation also reports mean recall across head, body, and tail category groups.
  • Experiments Configuration: The model uses ResNet-101 and DETR, with a three-layer predicate encoder, six-layer predicate decoder, 256-dimensional hidden states, and 150 predicate queries.The entity detector is pretrained before joint training with the predicate node generator.
  • Ablation Study: Removing the structural predicate design reduces Visual Genome R@100 and mR@100 by 1.9 and 1.4, respectively.The ablation compares holistic predicate queries with the proposed structural form.
  • Comparisons with State-of-the-Art Methods: On OpenImages-V6, SGTR improves over BGNN by 2.28 and increases wmAPrel and wmAPphr by 5.83 and 7.36, respectively.These comparisons use the same ResNet-101 backbone.
  • Comparisons with State-of-the-Art Methods: SGTR achieves comparable inference time to one-stage methods using the same backbone, but performs sub-optimally on Visual Genome relationships involving small entities.The authors attribute this limitation to weaker DETR detection of small entities than Faster R-CNN.
  • Qualitative Results: Attention visualizations show predicate decoders focusing on contextual regions around entities while entity sub-decoders attend to entity regions.The authors associate this division of attention with more effective compositional relationship modeling and improved prediction accuracy.

6. Conclusions

SGTR formulates scene graph generation as end-to-end bipartite graph construction with entity and predicate nodes connected by directed edges. Its entity-aware predicate representation and graph assembling module support competitive performance on Visual Genome and OpenImages-V6, while SGG may also be misused for surveillance.

  • Conclusions: SGTR is an end-to-end CNN-Transformer approach for scene graph generation.The method is presented as a transformer-based framework for constructing scene graphs.
  • Conclusions: The method generates entity and predicate nodes, models predicates with entity-aware structural decoding, and connects them through graph assembling.These steps form the proposed bipartite graph construction process.
  • Conclusions: SGTR outperforms or matches previous state-of-the-art methods on Visual Genome and OpenImages-V6.The conclusion reports this result across both benchmarks.
  • Conclusions: SGG could serve as a base module for surveillance abuse and private-information collection.The paper identifies this as a possible negative societal impact.

Overview of Appendixes

The supplementary material provides additional experiments, implementation details, and technical descriptions of graph assembling and the loss function.

  • Document Information: The document is dated 31 March 2022 and identified as arXiv:2112.12970v3.This metadata appears in the supplementary material.
  • Additional Experiments: The supplementary material adds experiments on overall recall, stronger long-tail learning strategies, and qualitative results.These analyses are described in its first section.
  • Technical Details: It documents implementation details and provides the graph assembling mechanism and loss function.These technical descriptions are presented in later supplementary sections.

1. More Experimental Results

The experiments analyze SGTR’s detector behavior, long-tail strategies, zero-shot retrieval, hyperparameters, and qualitative predictions. Results highlight balanced entity detection, improved rare-category retrieval, and limitations on small entities.

  • Overall Recall Analysis: More than half of Visual Genome relationships contain small entities, where BGNN outperforms SGTR by a large margin.The analysis attributes this gap to DETR’s limitation in detecting small entities.
  • Influence of Object Detector: DETR performs similarly to Faster R-CNN for tail predicates, and SGTR outperforms a DETR-equipped BGNN baseline.The comparison supports gains from SGTR’s design beyond the detector choice.
  • Long-tail Learning Strategy: DisAlign increases mR@100 by 1.6 with a 0.4 drop in R@100, while decoupled retraining gains 6.4 mR@100 at a 3.6 R@100 decrease.These results expose a trade-off between overall recall and mean recall.
  • Long-tail Learning Strategy: SGTR’s entity detection remains relatively balanced across long-tailed entity frequencies, reducing the benefit of rebalancing the entity classifier.The reported analysis links this balance to the transformer-based detector.
  • Zero-shot Relationship Retrieval: SGTR gains 2.6 on zR@100 over VCTree-TDE, indicating stronger zero-shot relationship retrieval.The authors describe this result as evidence of generalization to unseen relationships.
  • Qualitative Results: Graph assembling refines rough entity indicators into accurate entity results, improving the quality of generated scene graphs.Qualitative visualizations compare entity indicators before assembling with entity nodes after assembling.
  • Qualitative Results: SGTR retrieves more relationships from less frequent semantic categories than BGNN in qualitative comparisons.The rare-category predictions are marked in the comparison visualization.

2. Implementation Details

SGTR uses a two-phase training process: entity-detector pre-training followed by joint training that continues refining the detector. Previous methods are reimplemented under matched backbone settings for comparison.

  • Training Procedure: Training consists of entity-detector pre-training followed by joint SGTR training.The implementation is based on PyTorch 1.8 and cvpods.
  • Entity Detector Pre-training: The entity detector follows DETR configuration with AdamW, a 1e-5 learning rate, batch size 16, and 100 epochs on four TITAN V GPUs.Images are resized with shortest sides between 480 and 600 pixels and longest sides capped at 1000 pixels.
  • Entity Detector Pre-training: Transformer settings such as attention heads and dropout are kept the same as DETR.
  • Joint Learning: Joint training refines detector parameters instead of freezing the detector, which empirically improves entity detection.The joint phase uses the same optimizer, learning rate, and batch-size configuration as pre-training.
  • Comparison Settings: Prior two-stage methods are reimplemented with ResNet-101 instead of ResNeXt-101 FPN, and one-stage HOI methods are also evaluated on SGG.All models are trained until convergence using reported or released-code hyperparameters.

3. Correspondence Matrix for Assembling

The correspondence matrix assembles predicate and entity predictions by combining spatial and category matching distances. Spatial matching uses box overlap and center distance, while category matching uses classification similarity.

  • Matrix Construction: The correspondence matrix pairs each of Nr predicate predictions with Ne entity predictions.The matrix is formed from semantic outputs including entity boxes, classifications, and predicate entity indicators.
  • Distance Function: The matching distance combines spatial distance dloc with category distance dcls.The spatial term contains GIoU-based and center-distance components.
  • Spatial Matching: Spatial matching uses clipped GIoU between entity and indicator boxes plus L1 distance between their centers.The center coordinates are normalized before center-distance computation.
  • Category Matching: Category matching uses cosine distance between the classification distributions of the entity predictions.

4. Matching Cost and Loss Function

SGTR supervises relationship predictions with set matching between predicted and ground-truth triplets. The matching cost combines predicate and assembled-entity terms, while training uses detector, predicate, indicator, and classification losses.

  • Matching Cost: Set matching pairs relationship predictions with ground-truth triplets by minimizing a cost matrix C between Nr predictions and Ngt relationships.The relationship target set contains subject, predicate, and object components.
  • Matching Cost: The matching cost combines predicate cost Cp and entity cost Ce.Predicate cost uses predicate classification and predicted relationship location, while entity cost uses assembled subject/object predictions.
  • Predicate Cost: Predicate cost compares predicted predicate distributions and entity-pair center coordinates with ground-truth relationships.Ground-truth predicate labels are one-hot vectors, and relationship centers are represented by four coordinates.
  • Entity Cost: Entity cost compares assembled subject/object boxes and category distributions with ground-truth entities.The subject and object roles are handled separately.
  • Loss Function: The total loss combines the entity-detector loss with predicate-node-generator losses.The detector loss follows DETR, while predicate-generator loss includes entity-indicator and predicate-subdecoder terms.
  • Loss Function: Entity-indicator supervision includes localization and classification terms for subject and object representations.Localization uses L1 and GIoU losses, while classification uses cross entropy.

5. Social Impacts

SGG may serve as a base module for surveillance abuse.

  • SGG may serve as a base module for surveillance abuse.
Loading 2112.12970v3…