Source-linked AI summary
Neural Motifs: Scene Graph Parsing with Global Context
Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi
TL;DR
Scene graph parsing requires structured predictions of objects and their relationships, but the role of recurring graph structure needs quantitative characterization. The paper analyzes motifs in Visual Genome, introduces a frequency-based relation baseline and MOTIFNET for global context, and reports improvements over prior methods and the new baseline. These results show that object-conditioned and higher-order structural regularities are useful for scene graph prediction.
Problem
The paper investigates how recurring motifs and object–relation dependencies can be modeled when producing structured scene graphs from images.
Method
The paper analyzes Visual Genome motifs, predicts frequent relations for detected object-label pairs, and introduces MOTIFNET, which stages predictions using recurrent global context.
Results
The baseline improves prior state-of-the-art by 1.4 mean recall points, while MOTIFNET adds 2.9 mean points, a 7.1% relative gain, over the baseline.
Takeaways & Limitations
Scene graph models benefit from capturing asymmetric object–relation dependencies and higher-order global interactions, not only visual cues.
Takeaways & Limitations
Detector failures can cascade into missed relations; failure to detect one house produced five false-negative relations in an example.
Abstract
from arXiv · showhide
We investigate the problem of producing structured graph representations of visual scenes. Our work analyzes the role of motifs: regularly appearing substructures in scene graphs. We present new quantitative insights on such repeated structures in the Visual Genome dataset. Our analysis shows that object labels are highly predictive of relation labels but not vice-versa. We also find that there are recurring patterns even in larger subgraphs: more than 50% of graphs contain motifs involving at least two relations. Our analysis motivates a new baseline: given object detections, predict the most frequent relation between object pairs with the given labels, as seen in the training set. This baseline improves on the previous state-of-the-art by an average of 3.6% relative improvement across evaluation settings. We then introduce Stacked Motif Networks, a new architecture designed to capture higher order motifs in scene graphs that further improves over our strong baseline by an average 7.1% relative gain. Our code is available at github.com/rowanz/neural-motifs.
1. Introduction
The paper studies scene graph parsing through recurring structural motifs in Visual Genome. It shows that object labels strongly constrain relations, motivating frequency-based baselines and a globally contextual Stacked Motif Network.
- Motivation: Scene graph parsing produces semantic graphs of objects and relationships from real-world images.These representations support natural-language image tasks and broaden potential computer-vision applications.
- Motivation: Visual Genome contains annotated scene graphs for 100k COCO images, with over 1M object instances and 600k relations.The dataset enables analysis of recurring structural regularities, or motifs.
- Scene graph motifs: Object categories make relation distributions highly skewed, while relation labels provide less information about object categories.This asymmetric dependence motivates conditioning relation prediction on detected objects.
- Baselines: The frequency-based baseline improves prior state-of-the-art by 1.4 mean recall points, or 3.6% relative.It predicts the most frequent training-set relation for each detected object-label pair.
- Model: MOTIFNET stages box prediction, object classification, and relationship prediction so earlier global context informs later local predictors.The architecture represents context with recurrent sequential models such as LSTMs.
- Results: MOTIFNET improves over the new baseline by 2.9 mean points, a 7.1% relative improvement, across Visual Genome evaluation settings.The experiments also report a 14.0-point average improvement from pretraining the detector on Visual Genome.
2. Formal definition
A scene graph is a structured representation of an image’s semantic content, combining localized boxes, object labels, and binary relationships.
- Definition: A scene graph G represents the semantic content of an image.
- Components: The graph includes bounding boxes B, object labels O, and binary relationships R between objects.Each box is paired with a class label, while relationships connect object pairs.
- Dataset composition: Visual Genome contains 25.2% part entities, while 90.9% of relations are geometric or possessive.The table organizes object and relation types by super-type.
3. Scene graph analysis
The analysis finds strong local and higher-order regularities in Visual Genome scene graphs. Relation types are highly predictable from object categories, motivating context-aware relation prediction and motif modeling.
- Analysis goals: The analysis measures correlations between relation and object types and recurrence of higher-order graph structures.These findings motivate both the proposed baselines and the global-context model.
- Prevalent relations: Geometric and possessive relations dominate Visual Genome, while semantic relations comprise only 8.7% of relation instances.Geometric, possessive, and semantic edges account for 50.9%, 40.9%, and 8.7%, respectively.
- Prevalent relations: Clothing and parts are linked almost exclusively by possessive relations, whereas furniture and buildings are linked almost exclusively by geometric relations.Semantic relations are mostly headed by people and commonly connect to artifacts, vehicles, and locations.
- Predictability: Given an object pair, the five most frequent relations contain the correct label 97% of the time, while the single most frequent relation is correct 70% of the time.This asymmetry indicates that object identities provide strong information for edge prediction.
- Higher-order motifs: The study mines repeated object-relation-object combinations iteratively to identify higher-order motifs in scene graphs.Previously identified motifs are replaced with atomic symbols before mining larger combinations.
4. Model
MOTIFNET factorizes scene-graph prediction into bounding regions, object labels, and relations while explicitly retaining dependencies among these stages. It uses recurrent global context to condition object and relation predictions on the full set of regions, labels, and spatial features.
- Model factorization: MOTIFNET decomposes graph prediction into bounding-region, object-label, and relation factors without imposing independence assumptions.Predicted object labels may depend on one another, and relation labels may depend on predicted object labels.
- Bounding regions: Faster R-CNN proposes region boxes and produces features and noncontextualized object-label probabilities for each proposal.The detector retains background as a possible label, so it has not yet committed to a fixed set of object boxes.
- Object context: A bidirectional LSTM aggregates the ordered proposal regions, features, and label probabilities into contextualized representations for object prediction.Its hidden states allow all proposal regions to contribute information about potential object identities.
- Object decoding: An LSTM sequentially decodes each region’s category label while conditioning on previously decoded labels, then passes the object commitments to the relation model.The relation stage uses the committed object labels rather than the object decoder’s hidden states.
- Relation prediction: For relation prediction, additional bidirectional LSTMs contextualize regions and labels, and each possible edge combines head, tail, and union-box information.The edge model computes probabilities for the quadratic set of possible relations, including the background label.
- Global context: The architecture therefore propagates global context between region detection, object labeling, and edge classification stages.Figure 5 summarizes these staged predictions and the context passed between them.
5. Experimental Setup
The experiments use Visual Genome scene-graph tasks with standardized detection, ordering, training, and recall protocols. Models are adapted from Faster R-CNN and MOTIFNET uses contextual recurrent processing for objects and relations.
- Model architecture: Bounding regions are ordered left-to-right by default, with confidence, size, and random orderings also evaluated.Left-to-right ordering is intended to encourage predictions between nearby objects; confidence ordering lets easier regions provide context for harder ones.
- Training and adaptation: Training jointly minimizes cross-entropy losses for object and predicate prediction, using sampled annotated and negative relationships.MOTIFNET is trained on ground-truth boxes and later fine-tuned on noisy detector proposals to reduce the train–test mismatch.
- Dataset and evaluation: Visual Genome evaluation covers PREDCLS, SGCLS, and SGDET with 150 object classes and 50 relation classes.PREDCLS predicts edges from ground-truth boxes and labels; SGCLS predicts labels and edges from ground-truth boxes; SGDET predicts boxes, labels, and edges.
- Dataset and evaluation: Scene-graph recall counts a relation as matched when predicted boxes sufficiently overlap the ground truth and object and relation labels agree.The protocol uses IoU ≥ 0.5 and prohibits multiple edge labels for a given directed box pair.
- Baselines: The FREQ baseline predicts predicate probabilities from training-set relationship frequencies conditioned on detected object labels.FREQ-OVERLAP additionally uses the observation that nearly all annotated relationships involve overlapping boxes, classifying non-overlapping pairs as background.
6. Results
Results compare MOTIFNET and frequency baselines with prior models under standardized scene-graph evaluation. The strongest baseline and MOTIFNET both improve substantially over earlier systems, with MOTIFNET providing an additional gain.
- Main comparisons: The evaluation compares MOTIFNET and frequency baselines with VRD, ASSOC EMBED, MESSAGE PASSING, and MESSAGE PASSING+.MESSAGE PASSING+ reuses the updated detector, edge model, and NMS settings for a stronger comparison.
- Main comparisons: 1.4 mean recall points separate the best frequency baseline, FREQ+OVERLAP, from prior state-of-the-art.The reported mean averages the three evaluation modes over R@50 and R@100; R@20 is also reported separately.
- Main comparisons: 2.9 additional mean points, or a 7.1% relative gain, separate MOTIFNET from the new strong baseline.The improvement is reported across the evaluation settings and follows the frequency-baseline improvement over previous work.
- Task-specific results: 5.5 and 6.5 mean points are MOTIFNET’s improvements over MESSAGE PASSING+ in detection and predicate classification, respectively.The reported gains are attributed primarily to improvements in detection and predicate classification.
- Ablations: 1.2 mean points is the improvement of full MOTIFNET over MOTIFNET-NOCONTEXT, with the largest gains at R@20.The ablation indicates that contextual information contributes beyond non-contextualized head and tail label embeddings.
7. Qualitative Results
Qualitative results suggest that MOTIFNET can induce graph motifs from detection context, while revealing errors caused by predicate ambiguity and detector failures. Improvements are concentrated partly in relationship ordering at low recall thresholds.
- Qualitative evidence: MOTIFNET’s qualitative examples suggest it can induce graph motifs from detection context.The authors report that visual inspection appears stronger than the quantitative results because some predicted edges seem correct despite absent ground-truth annotations.
- Failure cases: Predicate ambiguity, such as “wearing” versus “wears,” is one common failure case.This ambiguity is illustrated by the skateboarder carrying a surfboard example.
- Relationship ordering: 51% of images have fewer than 50 relationship candidates and 78% have fewer than 100, limiting the need for ordering at higher thresholds.The larger low-threshold improvement therefore suggests that the models mainly improve relationship ordering rather than classification.
- Failure cases: Detector failures can cascade into missing relations for the affected object.Failing to detect “house” in one example caused five false-negative relations.
8. Related Work
The paper relates scene-graph parsing to contextual, structured, sequential, and prior-based approaches in vision and language. It emphasizes staged global context and recurrent sequence modeling as alternatives or complements to prior graph-propagation methods.
- Context modeling: The approach is related to graphical models that combine object co-occurrence and other contextual information.It differs by staging context incorporation so that large conditioning sets can provide meaningful global context.
- Context modeling: Object layouts and commonsense priors are presented as relevant sources of context for visual prediction.The paper suggests that object layouts provide sufficient context for Visual Genome scene-graph parsing, paralleling findings in image captioning.
- Structured and sequential models: Unlike language-grounding work, this task requires reasoning over all possible graph structures rather than structures determined by language.The paper places its use of sequential models alongside captioning, visual question answering, and other vision-language tasks.
- Structured and sequential models: Graph linearization and recurrent models have succeeded in several vision-and-language tasks, motivating their use for memorizing long scene-graph sequences.The paper connects this strategy to generating captions, parsing language, and generating text from abstract meaning graphs.
- Scene-graph methods: The method differs from graph-propagation approaches that reportedly converge quickly and bottleneck information exchange through edges.Its focus is global context exchange rather than the propagation mechanism used in those prior systems.
9. Conclusion
The paper finds that motifs are prevalent in Visual Genome and important to model. It introduces interaction-based baselines and MOTIFNET, which achieves additional gains over those baselines.
- Motifs are prevalent in the Visual Genome dataset and therefore important to model.
- The introduced baselines improve over prior state-of-the-art models by modeling intra-graph interactions while mostly ignoring visual cues.
- MOTIFNET captures higher-order structure and global interactions in scene graphs, achieving additional significant gains over the strong baselines.
Supplemental
The supplement explains that scene-graph results vary substantially with preprocessing, graph constraints, and task definitions. It therefore compares methods cautiously across configurations and reports that the proposed model and baselines are at least competitive.
- Evaluation variation includes custom dataset handling, preprocessing, label sets, and whether graph constraints are omitted.
- Removing graph constraints significantly increases reported performance because systems may output multiple labels for a head-tail pair.
- Predicate detection and phrase detection are easier than predicate classification and scene graph detection, respectively.
- MSDN-FREQ outperforms MSDN on all evaluation settings on the alternative split, suggesting robustness across data settings.
- The reported variation requires extremely cautious interpretation when comparing methods across configurations.
- The model and baselines are at least competitive with other approaches on different task configurations.