Source-linked AI summary
Pixels to Graphs by Associative Embedding
Alejandro Newell, Jia Deng
TL;DR
Scene graph prediction must capture object relations rather than only detect isolated objects, while prior systems often decompose the task into staged or pairwise processing. The paper trains a convolutional network end-to-end to detect all graph elements and connect them with associative embeddings and unordered output supervision. On Visual Genome, it reports state-of-the-art scene-graph performance, including nearly threefold improvement over prior work on the most difficult setting.
Problem
Prior image representations and staged relationship pipelines do not fully capture object interactions or jointly reason over the complete scene graph.
Method
A convolutional network detects vertices and edges directly from raw images, using unique associative embeddings and unordered output slots to assemble complete graphs.
Results
The method achieves state-of-the-art results on Visual Genome, improving prior work by nearly a factor of three on the most difficult task setting.
Takeaways & Limitations
Associative embeddings and unordered outputs provide the flexibility needed to train a single network for direct graph prediction from pixels.
Takeaways & Limitations
Performance depends partly on output resolution: lower resolution increases overlapping detections, while higher resolution improves localization.
Abstract
from arXiv · showhide
Graphs are a useful abstraction of image content. Not only can graphs represent details about individual objects in a scene but they can capture the interactions between pairs of objects. We present a method for training a convolutional neural network such that it takes in an input image and produces a full graph definition. This is done end-to-end in a single stage with the use of associative embeddings. The network learns to simultaneously identify all of the elements that make up a graph and piece them together. We benchmark on the Visual Genome dataset, and demonstrate state-of-the-art performance on the challenging task of scene graph generation.
1 Introduction
The paper targets direct scene-graph prediction from pixels, addressing the limits of object-only and staged approaches. It combines associative embeddings with unordered supervision to detect and connect graph elements jointly.
- Motivation: Scene graphs augment object identities and locations with relations between objects, providing a richer representation of image semantics.They represent objects as vertices and relationships as edges.
- Motivation: Existing pipelines often separate object detection from relationship prediction, restricting later reasoning and visual features.They may propose objects first and then analyze object pairs in isolation.
- Approach: The proposed network predicts a complete graph directly from a raw image while reasoning jointly over the entire scene.It does not explicitly rely on external Region Proposal Networks for an initial detection pool.
- Approach: Vertices and edges are treated as visual entities to detect, with edges representing object-object relations localized in image space.The network learns visual properties indicating where objects and relationships are likely to exist.
- Approach: Associative embeddings are repurposed so each vertex has a unique identifier and each edge refers to its source and destination vertices.Matching edge embeddings to vertex embeddings reconstructs the final graph.
- Approach: Unordered supervision handles overlapping detections by allowing correct loss computation without enforcing a fixed mapping to output slots.This addresses multiple vertices or edges grounded at the same image location.
- Results: The method achieves state-of-the-art Visual Genome results, improving prior work by nearly a factor of three on the most difficult task setting.The paper applies the system to semantic graph generation from objects and relations.
2 Related Work
Related work frames scene understanding as detecting and grounding relationships between objects. Prior methods use varied visual, language, recurrent, and proposal-based strategies, whereas this paper adapts associative embeddings to arbitrary directed graphs.
- Relationship detection: Relationship-detection tasks identify object relations and ground them in images with accurate object bounding boxes.The related-work scope includes visual relationship detection and scene graph generation.
- Relationship detection: Existing approaches reason over object pairs using vision-language models, message-passing RNNs, triplet proposals, and other diverse strategies.The task’s open-ended nature has produced multiple formulations.
- Relationship detection: Most prior approaches rely on Region Proposal Network boxes, while this method detects directly from the image and can optionally use proposals as additional input.The paper also contrasts pairwise processing with whole-image prediction.
- Associative embeddings: Associative embeddings were previously used to group body-joint detections into individual people by assigning shared embeddings within each person.That formulation represents clusters of nodes rather than arbitrary graph structure.
- Associative embeddings: This paper instead treats edges as detectable image elements and uses embeddings to express arbitrary graphs of vertices and directed edges.The change is motivated by the need to connect graph nodes rather than merely cluster them.
3 Pixels →Graph
The method constructs a pixel-grounded directed graph by detecting objects and relationships, predicting their properties, and connecting detections with associative embeddings. It handles unordered overlapping detections through multiple output slots.
- Graph representation: The system represents objects as vertices and object-object relations as directed edges grounded in image pixels.Vertices include object classes and bounding boxes; edges specify relationship predicates and source-target direction.
- Detecting graph elements: The network is trained end-to-end to define the complete graph while reasoning over the full image and all graph components.This avoids focusing exclusively on isolated object pairs and does not require an external Region Proposal Network.
- Detecting graph elements: A network produces per-pixel features that identify likely vertex and edge locations and encode their properties.Two heatmaps indicate vertex and edge likelihoods, while features at top activations feed property-specific fully connected networks.
- Connecting elements: Unique vertex embeddings let edge detections refer to their source and destination vertices and enable final graph assembly.Training pulls references toward the corresponding vertex embedding and pushes embeddings for different vertices apart.
- Overlapping detections: Separate object and relationship slots allow multiple detections to be predicted at the same pixel location.A single pixel feature produces predictions for s_o objects and s_r relationships, rather than only one of each.
4 Implementation details
The implementation uses a stacked hourglass network with defined input and output resolutions, augmentation, and separate support for prior detections. Sparse supervision reduces penalties for plausible but unannotated relationships.
- Network configuration: The model processes 512x512 images with a 64x64 output resolution and feature length f = 256.Training uses a stacked hourglass architecture with 512 features at its two lowest resolutions.
- Training procedure: Training applies random translation and scaling while updating annotations to ignore objects and relationships cropped from the image.Classification, bounding-box regression, and associative-embedding losses are weighted equally.
- Overlapping detections: The implementation sets s_o = 3 object slots and s_r = 6 relationship slots, covering annotations in nearly all cases.Only a small fraction of cases exceed these slot capacities.
- Incorporating prior detections: Prior object detections can be incorporated as two-channel inputs containing box-center activations and binary box masks.When many boxes are provided, the representation separates them by object class or bounding-box anchors.
- Sparse supervision: Sparse supervision samples negatives instead of exhaustively supervising heatmaps, reducing penalties for unannotated detections.This addresses the difficulty of exhaustively annotating the many plausible relationships in real-world scenes.
5 Experiments
The experiments evaluate scene graph generation on Visual Genome across three task settings and analyze predicate and slot behavior. The method improves over prior work, including when predicting directly from raw images without proposal boxes.
- Dataset: 108,077 images with object detections and object-object relationships make Visual Genome a challenging scene graph benchmark.The evaluation uses the preprocessed dataset with 150 object classes and 50 relationship predicates, following the prior-work split.
- Tasks and metric: Scene graph evaluation measures Recall@k for subject-predicate-object tuples whose classes match and whose object boxes overlap ground truth by at least 0.5 IoU.The benchmark includes SGGen, SGCls, and PredCls settings with different amounts of ground-truth object information.
- Results: Significant improvements over prior work occur on SGGen, SGCls, and PredCls, including promising performance without Region Proposal Network boxes.The system can produce detections from the raw image, is trained from scratch, and does not rely on pretraining on other datasets.
- Predicate analysis: Predicate predictions are dominated by frequent classes such as “on” and “wearing,” reflecting the distribution of Visual Genome annotations.The dataset contains several orders of magnitude more examples for “on” than for most other predicates.
- Predicate analysis: Poor performance on the least frequent predicate classes has little effect on final performance because those relationships have few instances.Table 2 reports performance per relationship predicate, with the top ten and bottom ten predicates shown separately.
- Slot analysis: Relationship detections use six available slots, which the network divides into predicate subsets for most classes but distributes uniformly for wearing/wears.Predicates behind, has, in, of, and on exclusively occupy three of the six slots.
6 Conclusion
The paper addresses the complexity of predicting information-rich scene graphs by supervising a single network to reason over the graph. Associative embeddings and unordered output slots provide the flexibility needed for training, and Visual Genome results demonstrate the approach’s effectiveness.
- Conclusion: Associative embeddings and unordered output slots enable a single network to abstract graph reasoning during scene graph prediction.The conclusion presents these components as the flexibility needed to make training the task possible.
- Conclusion: Results on Visual Genome clearly demonstrate the effectiveness of the proposed approach.