Source-linked AI summary
Graphical Contrastive Losses for Scene Graph Parsing
Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, Bryan Catanzaro
TL;DR
Scene graph parsers using independent predicate classification struggle with same-class entity confusion and nearby relationship pairings. The paper introduces structure-aware Graphical Contrastive Losses with targeted hard negatives and evaluates them in RelDN, achieving state-of-the-art results across three datasets. Gains are smaller on Visual Genome because possessive relationships create fewer entity-confusion cases and Recall@k is less strict than mAP.
Problem
Two-stage scene graph parsers trained with cross-entropy struggle to distinguish related from unrelated entities in same-class and closely spaced relationship configurations.
Method
The paper adds three structure-aware Graphical Contrastive Losses with hard negatives targeting generic, same-class entity, and same-predicate confusion, and uses them in RelDN.
Results
State-of-the-art results are reported on OpenImages, Visual Genome, and Visual Relationship Detection; on OpenImages Private, RelDN scores 0.328 and surpasses the winning model by 4.7% (16.5% relative).
Takeaways & Limitations
Contrastive supervision using scene-graph structure improves relationship detection by contrasting positive edges against hard negative edges for each node.
Takeaways & Limitations
On Visual Genome, the losses yield smaller gains because possessive relationships involve fewer entity-confusion cases and Recall@k is less strict than mAP.
Abstract
from arXiv · showhide
Most scene graph parsers use a two-stage pipeline to detect visual relationships: the first stage detects entities, and the second predicts the predicate for each entity pair using a softmax distribution. We find that such pipelines, trained with only a cross entropy loss over predicate classes, suffer from two common errors. The first, Entity Instance Confusion, occurs when the model confuses multiple instances of the same type of entity (e.g. multiple cups). The second, Proximal Relationship Ambiguity, arises when multiple subject-predicate-object triplets appear in close proximity with the same predicate, and the model struggles to infer the correct subject-object pairings (e.g. mis-pairing musicians and their instruments). We propose a set of contrastive loss formulations that specifically target these types of errors within the scene graph parsing problem, collectively termed the Graphical Contrastive Losses. These losses explicitly force the model to disambiguate related and unrelated instances through margin constraints specific to each type of confusion. We further construct a relationship detector, called RelDN, using the aforementioned pipeline to demonstrate the efficacy of our proposed losses. Our model outperforms the winning method of the OpenImages Relationship Detection Challenge by 4.7\% (16.5\% relative) on the test set. We also show improved results over the best previous methods on the Visual Genome and Visual Relationship Detection datasets.
1. Introduction
Scene graph parsing detects entities and predicts predicates between entity pairs, but two-stage models often confuse same-class instances and nearby relationship pairings. The paper addresses these failures with Graphical Contrastive Losses and RelDN, reporting gains over prior methods.
- Scene graph parsing infers localized entities and predicate edges as subject-predicate-object triplets.
- Two recurring errors are Entity Instance Confusion among same-class entities and Proximal Relationship Ambiguity among nearby subject-object pairings.
- Fine-grained visual cues, such as hand coverage of a glass or a musician’s pose and finger placement, determine relationships like “hold” and “play.”
- Graphical Contrastive Losses add hard-negative supervision through margin-based triplet formulations designed for the two confusion types.
- RelDN achieves 0.328 on the Private set of the OpenImages Relationship Detection Challenge, exceeding the winning model by 4.7% (16.5% relative).
- The paper reports state-of-the-art performance on Visual Genome and Visual Relationship Detection datasets.
2. Related Work
Prior scene graph methods commonly classify each entity pair independently, overlooking relationships among graph edges. The paper instead uses structure-aware hard negatives and contrastive margins to target specific confusion types.
- Most prior approaches detect entities first, then independently classify predicates from features of each entity pair.
- Figure 2 illustrates baseline errors involving a man and one of three similar wine glasses, and a guitar player incorrectly paired with a drum.
- Independent pairwise prediction ignores intrinsic graph structure and contributes to Entity Instance Confusion and Proximal Relationship Ambiguity.
- Associative Embedding also uses contrastive learning, but this work defines different hard-negative sets for specific scene graph parsing errors.
- Graph-structured hard-negative contrastive margins target entity-instance and proximal-relationship confusion while adding supervision to N-way cross-entropy.
3. Graphical Contrastive Losses
Graphical Contrastive Losses add margin-based supervision tailored to entity-instance and proximal-relationship confusions. Three loss variants contrast pairings generally, within entity classes, or within predicate classes.
- The losses use an affinity Φ(s, o), interpreted as the probability that a subject-object pair has a relationship.Φ(s, o) is defined from the predicate distribution by excluding the no-relationship class.
- Class Agnostic Loss: Class Agnostic loss contrasts positive and negative subject-object pairs without using class information.It maximizes the lowest-scoring positive affinity against the highest-scoring negative affinity.
- Entity Class Aware Loss: Entity Class Aware loss targets entity instance confusion by restricting contrastive sets to instances sharing an entity class.This focuses the margin between same-class instances that could otherwise be confused.
- Predicate Class Aware Loss: Predicate Class Aware loss targets proximal relationship ambiguity by grouping instances according to their associated predicate classes.It penalizes incorrect pairings among nearby instances joined by the same predicate, such as assigning a musician to the wrong instrument.
- Overall Objective: The final objective combines the contrastive losses with L0, the cross-entropy loss over predicate classes.The loss components are defined over sampled entities and pairings from individual images.
- Sampling Complexity: Class-agnostic sampling has practical complexity O(NK), and the class-aware losses retain O(NK) complexity after filtering negatives by class.The implementation uses N = 512 and K = 64 per batch.
4. RelDN
RelDN extends the two-stage relationship-detection pipeline with dedicated predicate features and fused semantic, visual, and spatial modules. Its predicate branch is designed to emphasize relationship-indicating regions rather than only large entities.
- Architecture: RelDN first proposes likely subject-object pairs, then classifies each candidate into a predicate using extracted relationship features.The model follows the paper’s two-stage relationship-detection pipeline.
- Predicate Features: The predicate branch uses a CNN with the same structure as the entity-detector CNN to extract predicate-focused features.The entity detector branch is called conv body det, while the predicate branch is conv body rel.
- Module Fusion: RelDN computes semantic, visual, and spatial feature logits for each proposal, adds them elementwise, and applies softmax normalization.The logits are denoted fvis, fspt, and fsem for the three modules.
- Feature Visualization: Feature visualizations show salience shifting from large entities toward small regions that strongly indicate predicates.The comparison uses ground-truth relationships, entity-detector features, and predicate-backbone features.
- Module Fusion: Skip connections from subject-only and object-only ROI features improve predicate prediction by 6.4.They reflect cases where a relationship can be inferred from the appearance of only one entity.
5. Implementation Details
Implementation freezes the independently trained entity detector, fine-tunes a predicate CNN and second-stage relationship model, and ranks exhaustive relationship proposals using detector and predicate probabilities.
- Training: The entity detector CNN is trained independently and then frozen, while the predicate CNN is initialized from its weights and fine-tuned with the second stage.This choice is made for simplicity rather than end-to-end detector fine-tuning.
- Sampling: Training samples positive and negative pairs independently for each loss, with class-aware negatives grouped by entity or predicate classes.The loss weights are λ1 = 1.0, λ2 = 0.5, and λ3 = 0.1.
- Inference: At test time, RelDN takes up to 100 detector outputs and exhaustively forms relationship proposals from all entity pairs.Proposals are ranked by pdet(s)·ppred(pred)·pdet(o).
- Backbones: RelDN uses ResNeXt-101-FPN for OpenImages and VGG-16 for Visual Genome and Visual Relationship Detection.These backbones match the architectures used by previous state-of-the-art methods.
6. Experiments
Experiments show that Graphical Contrastive Losses improve RelDN across targeted confusion cases, module ablations, and external benchmarks, with gains varying by dataset and evaluation metric.
- Loss Combinations: The full loss combination L0 + L1 + L2 + L3 consistently outperforms L0 alone, especially for predicates affected by the targeted confusions.APrel on “holds” rises from 41.84 to 43.09, “plays” from 36.04 to 41.04, and “interacts with” from 40.43 to 44.16.
- Targeted Error Subset: On 100 selected images, the model with Graphical Contrastive Loss exceeds the L0-only model by 1.4 overall and by 4.1 on APrel for “holds”.The subset contains five predicates selected for entity instance confusion or proximal relationship ambiguity.
- Margin Thresholds: Margin values m = 0.1 and m = 0.2 achieve the best performance, while m = 1.0 is the largest possible margin because affinity scores range from 0 to 1.The thresholds α1, α2, and α3 are varied together as a common margin m.
- Model Analysis: Adding visual features gives a 4.7 gain over the semantic module, while separate subject-object connections and spatial features provide further predicate-specific improvements.The spatial module yields the largest gaps on “at”, “on”, and “inside of”, including +2.4 for “inside of”.
- External Comparisons: RelDN surpasses the OpenImages Challenge winner Seiji by 4.7% on the Private set and 2.9% on the full set.Using Seiji’s entity detector still yields gaps of 1.4% and 0.8%; RelDN also outperforms prior methods on Visual Genome and is competitive or stronger on VRD depending on pretraining.
- Evaluation Considerations: Visual Genome gains are smaller than on OpenImages because possessive relationships involve fewer entity-confusion cases and Recall@k is less strict than mAP.Recall@100 can remain 100% when the ground-truth target appears among the top 100 predictions, regardless of ranking.
7. Conclusion
The paper addresses Entity Instance Confusion and Proximal Relationship Ambiguity in scene graph parsing with Graphical Contrastive Losses that use scene-graph semantics to contrast positive relationships against hard negatives. Added to a model using the same pipeline, these losses achieve state-of-the-art results on three datasets.
- Graphical Contrastive Losses target Entity Instance Confusion and Proximal Relationship Ambiguity in scene graph parsing.
- The losses exploit semantic properties of scene graphs to contrast positive relationships against hard negatives.
- Three loss types address the two issues from complementary aspects.
- The authors demonstrate efficacy by adding the losses to a model built with the same pipeline.
- The resulting model achieves state-of-the-art results on three datasets.