Source-linked AI summary
Energy-Based Learning for Scene Graph Generation
Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, Leonid Sigal
TL;DR
Cross-entropy training treats scene-graph objects and relationships independently, overlooking dependencies and biasing relation predictions. The paper introduces a model-agnostic energy-based framework with message passing to incorporate graph structure, improving benchmark performance and learning in zero- and few-shot settings.
Problem
Cross-entropy training factorizes scene-graph likelihood into independent object and relationship terms, ignoring correlations, exclusions, and relation-frequency imbalance.
Method
A model-agnostic energy-based loss uses graph message passing to model the joint conditional density of scene graphs without requiring partition-function estimation.
Results
The framework consistently improves mean Recall across three tasks and models, with up to 21% improvement on Visual Genome and 27% on GQA, while also improving zero-shot recall.
Takeaways & Limitations
The structure-aware framework supports efficient learning when labels are scarce and can be applied to existing generation models without assumptions about their underlying architecture.
Abstract
from arXiv · showhide
Traditional scene graph generation methods are trained using cross-entropy losses that treat objects and relationships as independent entities. Such a formulation, however, ignores the structure in the output space, in an inherently structured prediction problem. In this work, we introduce a novel energy-based learning framework for generating scene graphs. The proposed formulation allows for efficiently incorporating the structure of scene graphs in the output space. This additional constraint in the learning framework acts as an inductive bias and allows models to learn efficiently from a small number of labels. We use the proposed energy-based framework to train existing state-of-the-art models and obtain a significant performance improvement, of up to 21% and 27%, on the Visual Genome and GQA benchmark datasets, respectively. Furthermore, we showcase the learning efficiency of the proposed framework by demonstrating superior performance in the zero- and few-shot settings where data is scarce.
1. Introduction
The paper identifies independent cross-entropy training as a source of structurally inconsistent and biased scene graphs, then introduces energy-based learning to incorporate graph structure directly. Experiments show improved performance, generalization, and data efficiency across established models and benchmarks.
- Problem: Cross-entropy training treats objects and relationships independently, ignoring correlations and exclusions in the scene-graph output space.
- Problem: Independent relation losses can assign similar penalties to contextually different errors and incentivize frequent, generic relations.For example, <man, riding, wave> is inconsistent with other graph relations, while <man, on, rock> replaces the more informative <man, standing on, rock>.
- Method: The proposed energy-based framework directly incorporates scene-graph structure and learns a joint conditional density through graph message passing.This changes the objective from maximizing summed individual likelihoods to maximizing the joint likelihood of objects and relations.
- Method: The framework is model-agnostic and can train off-the-shelf scene-graph generation models using an energy-based loss.The experiments apply it to multiple state-of-the-art models and compare them with cross-entropy-trained counterparts.
- Results: Experiments also demonstrate improved zero-shot relation retrieval and stronger relative performance for few-shot relation triplets.
- Results: Energy-based training produces more granular relations and excludes improbable relations in generated scene graphs.The proposed model predicts <man, standing on, rock> instead of <man, on, rock>, and <man, in front of, wave> instead of <man, riding, wave>.
- Results: The framework improves performance by up to 21% on Visual Genome and 27% on GQA compared with cross-entropy counterparts.
2. Related Work
Prior scene-graph work improved context aggregation or addressed cross-entropy shortcomings with handcrafted losses, while this paper learns an approximate joint distribution using energy-based modeling. The paper positions this as incorporating output structure into scene-graph learning.
- Scene Graph Generation: Early scene-graph methods focused on context aggregation modules to learn better representations.
- Scene Graph Generation: Recent methods identified cross-entropy shortcomings and proposed handcrafted loss formulations to improve performance.
- Scene Graph Generation: This work generalizes that direction by learning an approximate joint distribution over scene graphs and images with an energy-based formulation.
- Energy Based Modeling: Energy-based modeling has been used increasingly for generative tasks but remains relatively unexplored for discriminative tasks.
- Energy Based Modeling: The paper formulates discriminative scene-graph generation with an energy-based model.
3. Approach
The approach replaces independent cross-entropy training with an energy-based framework that incorporates scene-graph structure into discriminative learning. It computes energies for image–scene-graph configurations using graph-based state refinement and trains the generator through energy-based optimization.
- Motivation: Standard cross-entropy sums independent object and relationship losses, ignoring dependencies and structure in the scene-graph output space.This formulation can favor dominant relations and produce biased predictions.
- Energy-based learning: The proposed discriminative energy formulation learns relative energies of scene-graph configurations without estimating the partition function or computing expectations.A carefully crafted loss permits parameterizing the energy function with an arbitrary neural-network architecture.
- Energy-based learning: The framework defines images as inputs and scene graphs SG=(O,R) as outputs, with object and relationship labels represented by node and edge tensors.The number of objects is n, while d and d′ denote the numbers of possible object and relation labels.
- Energy model architecture: The architecture refines image-graph and scene-graph states with graph neural networks, pools each graph, concatenates their representations, and predicts a scalar joint energy with an MLP.Image-graph nodes use object-region features, while scene-graph processing uses an edged graph neural network.
- Optimization: The energy model compares predicted and ground-truth configurations, then uses SGLD to approximately find a lower-energy scene-graph configuration through iterative noisy gradient steps.Differentiation through this optimization path guides learning of the model producing the initial prediction.
- Energy model architecture: The edged message-passing variant aggregates neighboring node and edge messages, mixes their contributions with α, and preserves edge direction in relationship updates.Direction awareness distinguishes relations such as <cat, has, tail> from <tail, on, cat>.
4. Experiments
The experiments evaluate energy-based training across two datasets, multiple scene graph generators, and relationship-retrieval settings, including zero-shot, few-shot, and sentence-to-graph retrieval.
- Datasets and models: Experiments cover Visual Genome and GQA, using multiple state-of-the-art scene graph generation models.Visual Genome includes VCTree, Neural Motifs, Iterative Message Passing, and VCTree-TDE; GQA uses Transformers.
- Evaluation: Mean Recall@K evaluates relationship retrieval in Predicate Classification, Scene Graph Classification, and Scene Graph Detection.Mean Recall@K is preferred over regular Recall@K because relation-frequency imbalance can bias reporting.
- Evaluation: Zero-shot Recall measures retrieval of subject-predicate-object triplets absent from training across PredCls, SGCls, and SGDet.This evaluates generalization to unseen relation triplets.
- Evaluation: Few-shot Recall groups relation triplets by training frequency, including 1−5-shot and 6−10-shot ranges.The grouped ranges replace separate 1-shot, 2-shot, and similar splits.
- Evaluation: Sentence-to-Graph Retrieval uses detected scene graphs as image representations and reports Recall@20/100 on 1k/5k galleries.Captions are queries, and retrieval relies solely on the detected scene graph rather than other visual information.
- Implementation: Implementation uses a frozen Faster R-CNN detector, identical optimization setups for baseline and proposed training, and dataset-specific frequency-bias choices.Frequency bias is used for Visual Genome but omitted on GQA because of high memory requirements.
5. Experimental Results
Across benchmark, zero-shot, few-shot, retrieval, qualitative, and ablation evaluations, energy-based training consistently improves scene graph generation over cross-entropy baselines. The gains include stronger performance with scarce data and more coherent, informative predictions, while additional optimization steps improve recall at added computational cost.
- Quantitative Results: 12.7%, 22.3%, and 5.6% relative improvements in mR@20 are obtained for VCTree on PredCls, SGCls, and SGDet, respectively, versus cross-entropy training.The reported improvements are on Visual Genome.
- Quantitative Results: Energy-based models show larger improvements for relations with fewer training annotations than their cross-entropy counterparts.This supports improved performance on underrepresented relations.
- Zero-Shot Recall: Consistent zero-shot recall improvements are observed across models, attributed to structure-aware learning that supports global scene graph reasoning.The evaluated metrics are zsR@20 and zsR@50.
- Few-shot Recall: Few-shot experiments with VCTree show a significant performance boost when less training data is available, indicating data-efficient scene graph learning.The comparison uses Few-shot Recall@20 against standard cross-entropy training.
- Sentence-to-Graph Retrieval: Sentence-to-graph retrieval recall improves by 5%-23% across VCTree and Motif compared with corresponding baseline scene graphs.The experiments use galleries of 1000 and 5000 images.
- Ablation Studies: Increasing optimization steps almost consistently raises mean recall, but adds computational overhead and training time.Removing image information from the energy model causes a performance drop.
- Qualitative Results: Qualitative examples show more informative relations, including mounted on, parked on, walking on, and standing on, instead of generic on predictions.Energy-based training also produces spatially informative relations and more accurate zero-shot predictions.
6. Conclusion
The paper presents a model-agnostic energy-based framework that incorporates scene-graph structure into training. Experiments show improved performance over cross-entropy training, including zero-shot and few-shot relationship settings.
- The framework enables structure-aware learning for scene graph generation models.It is designed to train models without assumptions about the underlying generation architecture.
- The method improves performance compared with traditional cross-entropy-based training on Visual Genome and GQA.
- Experiments demonstrate the framework’s generality and efficiency in zero-shot and few-shot relationship settings.
- The proposed training framework can be applied easily to any scene graph generation model.