Source-linked AI summary
LinkNet: Relational Embedding for Scene Graph
Sanghyun Woo, Dahun Kim, Donghyeon Cho, In So Kweon
TL;DR
Scene graph generation requires reasoning about relationships among image objects, not only detecting objects individually. LinkNet addresses this with relational embedding that jointly models object inter-dependencies, augmented by global context and geometric layout encoding. It achieves state-of-the-art results on Visual Genome, with ablations supporting the effectiveness of the design.
Problem
Scene graph generation must capture object relationships, but reasoning about inter-object dependencies remains challenging despite progress in isolated object detection.
Method
LinkNet uses relational embedding to jointly model dependencies among all object instances, then adds global context encoding and geometric layout encoding.
Results
LinkNet achieves state-of-the-art scene graph generation results on the Visual Genome benchmark.
Takeaways & Limitations
Relational embedding is crucial for performance, while global context and geometric layout modules further boost scene graph generation results.
Abstract
from arXiv · showhide
Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits the main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.
1 Introduction
Scene graph generation targets structured representations that jointly capture image objects and their relationships, addressing the challenge of reasoning about inter-object interactions. LinkNet models object inter-dependencies with relational, contextual, and geometric modules and achieves state-of-the-art Visual Genome results.
- Scene graphs jointly represent objects and their relationships, supporting structured understanding for image search, question answering, and 3D scene synthesis.
- Relational embedding explicitly models inter-dependency among all object instances instead of reasoning about objects in isolation, benefiting relationship classification.
- Global context encoding and geometrical layout encoding further improve LinkNet’s scene graph generation network.
- LinkNet achieves state-of-the-art scene graph generation results on Visual Genome, with extensive ablations demonstrating the proposed network’s effectiveness.
2 Related Work
Prior work addresses relationship recognition through graph reasoning, message passing, associative embedding, staged inference, and relational reasoning. LinkNet instead uses relational embedding to reason explicitly about relations within nodes and edges without external knowledge sources.
- Relational reasoning methods aggregate information across graph elements or input positions to model dependencies between elements.
- Unlike a related graph approach, LinkNet does not require external knowledge sources for training.
- Scene graph generation approaches include graph refinement, associative node-edge embedding, joint region captioning, and staged inference.
- LinkNet uses self-attention within relational embedding to reason about relations among nodes and edges across multiple prediction stages.
3.1 Problem Definition
Scene graph generation constructs a topological scene representation whose nodes encode objects and whose directed edges encode relationships. The task jointly involves object classification, bounding-box prediction, and pairwise relationship classification.
- A scene graph encodes object instances, object categories, and relationships between objects as a topological representation.
- Each node is represented by a bounding box and object class, while each directed edge defines a relationship predicate between two nodes.
- Scene graph generation classifies objects, predicts their bounding-box coordinates, and classifies pairwise relationship predicates.
3.2 LinkNet
LinkNet starts from object proposals and features produced by Faster R-CNN, then applies three modules for scene graph generation. Relational embedding targets relationship classification, while global context and geometric layout encoding provide additional information.
- Faster R-CNN supplies initial object bounding-box proposals for LinkNet from an input image.
- For each proposal, the detector provides ROI-aligned features and an object-label distribution used by the scene graph network.
- LinkNet contains relational embedding, global context encoding, and geometric layout encoding modules.
- Relational embedding models inter-dependency among object instances and improves relationship classification, while the other modules add contextual and geometric information.
3.3 Object Classification
LinkNet classifies proposed objects using features that combine local appearance, predicted labels, image-level context, and relational information from all object regions.
- Object representation: Each region proposal uses ROI-aligned features, embedded predicted object-label distributions, and image-level context to form its representation.The object feature concatenates f RoI_i, K0l_i, and c.
- Object-relational embedding: The object-relational embedding attends to features from all N object regions rather than processing each object in isolation.Object proposals are stacked into O0, from which a relational embedding matrix is computed.
- Object-relational embedding: Object-level instances are the primitive elements of LinkNet’s relational reasoning, unlike prior approaches operating on pixels or words.
- Object-relational embedding: Residual relational embedding stages transform object features into relation-aware representations and ultimately into object-label distributions O4.The element-wise summation supports residual learning, and the resulting features pass through another relational embedding process.
- Object classification: The final object-classification output is optimized as a Cobj-way classification probability over the proposed regions.
- Global context encoding: Global context is learned through auxiliary multi-label object classification after average pooling RPN image features into c ∈R512.This context feature is concatenated with initial object features to support scene graph generation.
3.4 Relationship Classification
LinkNet classifies directed object-pair relationships by combining subject, object, union-region, and relative-geometry features in a relational embedding pipeline.
- Edge-relational embedding: Relationship classification evaluates labels for each possible directed object pair, including a background class.For N objects, the candidate edges number N^2−N.
- Edge-relational embedding: Subject and object edge features are combined with union-region representations and relative geometry before mapping to relationship logits.The low-rank outer product combines the edge and union representations, while K2 embeds geometry into 128 dimensions.
- Evaluation: The Visual Genome evaluation distinguishes perfect-detector Predicate Classification and Scene Graph Classification from detector-based Scene Graph Detection.The table caption states that the first two tasks use the same perfect detector, while Scene Graph Detection uses a customized pre-trained detector.
- Relationship classification: The relationship classifier optimizes the resulting G2 representation for Crel-way predicate prediction conditioned on the image, boxes, and object classes.
- Geometric layout encoding: Relative location and scale are encoded from object and subject coordinates, widths, and heights, then concatenated with subject-object features.The geometry encoding is motivated by correlations between relative geometry and geometric, possessive, and semantic predicates.
3.5 Loss
LinkNet is trained end-to-end while sequentially predicting boxes, object categories, and relationship categories, with its loss terms equally weighted by default.
- End-to-end training: The network jointly trains bounding-box, object-category, and relationship-category prediction in an end-to-end sequence.
- Loss weighting: λ1 and λ2 are set to 1 by default, so all loss terms receive equal weight.
4 Experiments
Experiments on Visual Genome evaluate LinkNet quantitatively, through ablations, relational-embedding visualizations, and qualitative scene-graph detection examples. Results support the relational embedding module and show additional benefits from geometric and global context encoding.
- Quantitative Evaluation: LinkNet achieves new state-of-the-art results on Visual Genome scene graph generation, outperforming a strong baseline on scene graph and predicate classification.Classification tasks use the same perfect detector across methods, whereas detection depends on a customized pre-trained detector.
- Ablation Study: Two relational embedding modules achieve the best hyperparameter result, while too many modules introduce optimization difficulty.Multiple modules can support multi-hop communication, but excessive modules can make optimization difficult.
- Ablation Study: The design study combines one-hot object class representations with contextualized representations when constructing the edge-relational embedding input E0.Object label probabilities are converted into one-hot vectors with argmax before concatenation.
- Ablation Study: Removing proposed modules lowers performance, while jointly using relational embedding, geometric layout, and global context encoding gives the strongest ablation result.The relational embedding module is described as crucial, with the other modules further boosting performance.
- Ablation Study: Geometric layout encoding significantly increases recall for geometrically related predicates such as using, carrying, and riding.The analysis selects the top-10 predicates with the highest recall increase in scene graph classification.
- Ablation Study: Softmax row-wise operation performs better than sigmoid, indicating that competition along rows helps form more informative relational embeddings.The comparison is between the softmax and sigmoid operations in the relational embedding matrix.
- Ablation Study: Dot-product relation computation performs slightly better than Euclidean distance, while the introduced modules continue improving performance with Euclidean distance.This suggests relational embedding behavior is important and is less sensitive to the relation computation method.
- Qualitative Evaluation: Relational embedding weights represent inter-dependency among object instances consistently with ground-truth relationships, including some missing annotations.The visualization uses a binary triangular ground-truth relation matrix and a trained intermediate relational embedding matrix.
5 Conclusion
The paper addresses image-to-scene-graph generation by modeling global object interactions with relational embeddings. Adding global context and geometric layout encoding produces LinkNet, whose ablations and visualizations support its efficacy and state-of-the-art performance.
- Conclusion: LinkNet combines relational embedding with global context and geometric layout encoding to improve scene graph generation over a basic Faster R-CNN system.The paper reports extensive ablations and relational-embedding visualizations supporting the approach.