Source-linked AI summary
CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation
Jing Yu, Yuan Chai, Yujing Wang, Yue Hu, Qi Wu
TL;DR
Long-tailed relationship labels bias SGG toward head classes, while conventional debiasing work has overlooked correlations among biased classes. CogTree builds a hierarchy from biased predictions and applies a coarse-to-fine loss; the paper reports improved mR@K across three models and state-of-the-art performance against existing debiasing methods.
Problem
Long-tailed relationship data biases SGG toward head classes, and conventional debiasing approaches overlook correlations among biased classes.
Source: §1 · pp. 1–2Method
CogTree organizes relationships into a hierarchy derived from biased SGG predictions and trains with a loss that distinguishes them coarse-to-fine.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3Results
On mR@K, CogTree improves all three evaluated models, with the strongest baseline VCTree gaining 3.6%∼13.6%; it also outperforms TDE on VCTree in mR@K and most R@K metrics.
Source: §4.1 · p. 5Takeaways & Limitations
The CogTree loss is model-agnostic and consistently boosts performance across various SGG models.
Source: §5 · p. 6
Abstract
from arXiv · showhide
Scene graphs are semantic abstraction of images that encourage visual understanding and reasoning. However, the performance of Scene Graph Generation (SGG) is unsatisfactory when faced with biased data in real-world scenarios. Conventional debiasing research mainly studies from the view of balancing data distribution or learning unbiased models and representations, ignoring the correlations among the biased classes. In this work, we analyze this problem from a novel cognition perspective: automatically building a hierarchical cognitive structure from the biased predictions and navigating that hierarchy to locate the relationships, making the tail relationships receive more attention in a coarse-to-fine mode. To this end, we propose a novel debiasing Cognition Tree (CogTree) loss for unbiased SGG. We first build a cognitive structure CogTree to organize the relationships based on the prediction of a biased SGG model. The CogTree distinguishes remarkably different relationships at first and then focuses on a small portion of easily confused ones. Then, we propose a debiasing loss specially for this cognitive structure, which supports coarse-to-fine distinction for the correct relationships. The loss is model-agnostic and consistently boosting the performance of several state-of-the-art models. The code is available at: https://github.com/CYVincent/Scene-Graph-Transformer-CogTree.
1 Introduction
SGG turns images into relationship-rich scene graphs that support visual reasoning, but long-tailed relationship data biases predictions toward head classes. CogTree addresses this by organizing relationships hierarchically and training with a coarse-to-fine loss.
1 Introduction
- SGG abstracts images into object-relationship graphs that support downstream question answering, visual understanding, and explainable reasoning.Scene graphs provide relational information for reasoning beyond independent objects.Source: §1 · p. 1
- Long-tailed training data biases relationship prediction toward head classes, often collapsing distinct tail relationships into vague or visually similar labels.Examples include predicting “on” instead of “looking at” or “walking on,” and confusing “standing on,” “sitting on,” and “lying on.”Source: §1 · p. 1
- Prior debiasing work reweights losses or disentangles representations, while the paper frames concept organization—not only learning strategy or representation—as a neglected distinction.The proposed motivation is hierarchical reasoning: distinguish broad relationship concepts first, then finer differences among easily confused relationships.Source: §1 · pp. 1–2
- CogTree builds a hierarchy from biased SGG predictions and trains with a tree-based loss that distinguishes relationships coarse-to-fine.The approach progressively addresses inter-concept and intra-concept distinctions rather than requiring all relationships to be distinguished at once.Source: §1 · p. 2
- The model-agnostic loss alleviates SGG bias and achieves state-of-the-art performance compared with existing debiasing methods.The paper reports extensive experiments across several representative SGG models.Source: §1 · p. 2
2 Related Work
Prior SGG work developed contextual and sequential models, while unbiased relationship prediction remained difficult under biased data. Existing debiasing approaches include data pruning and causal interventions; CogTree instead uses the relationships’ hierarchical organization.
2 Related Work
- SGG methods progressed from independent object and relationship networks to context-aware message passing, recurrent architectures, and contrastive learning.These methods incorporate global visual context to improve object and relationship representations.Source: §2 · p. 2
- For biased SGG, mean Recall was proposed as an unbiased metric [Chen et al., 2019b; Tang et al., 2019], while other approaches prune predominant relationships [Liang et al., 2019] or use counterfactual surgeries [Tang et al., 2020].The passage distinguishes the metric proposal from subsequent debiasing solutions.Source: §2 · p. 2
- CogTree differs from prior balancing, reweighting, curriculum, and representation-separation methods by using relationships’ inherent hierarchy for coarse-to-fine reweighting.The authors describe this as enabling more discriminative representation learning.Source: §2 · p. 2
3 Methodology
The methodology builds a relationship hierarchy from biased SGG predictions, trains models with a CogTree-based debiasing loss, and applies that loss to three representative SGG models.
3 Methodology
- MOTIFS, VCTree, and SG-Transformer are case-studied as representative models to which the model-agnostic CogTree loss can be applied.SG-Transformer uses transformer architectures for both object and relationship encoding.Source: §3 · p. 2, §3.3 · p. 4
3.1 Automatic CogTree Building
- CogTree induces concepts from a biased model’s predictions, builds concept-centered subtrees, then aggregates them into four layers from a virtual root to fine-grained relationships.The layers distinguish concepts, coarse versus fine relationships, and discrepancies among easily confused relationships.Source: §3.1 · pp. 2–3, Figure 2 · p. 3
3.2 Debiasing CogTree Loss
- The loss combines tree-based class-balanced classification across CogTree layers with class-balanced softmax cross-entropy over biased predicted probabilities.TCB first addresses noise between concepts, then refines distinctions within a concept; training starts from scratch and prediction remains a flat multiclass classification.Source: §3.2 · p. 4
3.3 Scene Graph Generation Models
- MOTIFS, VCTree, and SG-Transformer provide the three model architectures used to study CogTree loss, with SG-Transformer using transformer encoders for objects and relationships.The relationship encoder combines relationship embeddings with object embeddings before decoding.Source: §3.3 · p. 4, Figure 4 · p. 4
4 Experiments
Experiments evaluate CogTree on Visual Genome and report improvements in balanced recall, including comparisons across models, ablations, qualitative examples, and parameter settings. The qualitative and tail-class analyses also examine how these gains relate to finer-grained predictions and conventional recall.
4 Experiments
- Evaluation uses the Visual Genome split with 150 object classes and 50 relationship classes, reporting R@K and class-averaged mR@K across PredCls, SGCls, and SGDet.The authors do not treat conventional R@K as the main evaluation because of its bias.Source: §4 · p. 4
4.1 State-of-the-Art Comparison
- CogTree improves every baseline on all mR@K metrics, with VCTree boosted by 3.6%∼13.6%.It also outperforms conventional debiasing methods and TDE; mR@20 shows greater improvement than mR@50/100.Source: §4.1 · p. 5, Table 1 · p. 5
- CogTree significantly improves R@K for the tail 45 classes on predicate classification, despite the overall R@K drop noted alongside rising mR@K.The authors interpret the overall drop as reflecting CogTree's preference for fine-grained tail labels over vague head labels.Source: §4.1 · p. 5, Table 2 · p. 5
4.2 Ablation Study
- Tree-based and flat classification losses provide complementary benefits, while violating either CogTree's same-concept grouping or coarse-to-fine layering significantly reduces performance.Both loss terms benefit from weighting, and AVERAGE outperforms MAX and SUM for non-leaf nodes.Source: Table 3 · p. 5, §4.2 · p. 6
4.3 Qualitative Analysis
- In PredCls examples, CogTree predicts finer relationships such as “parked on” and “in front of” and distinguishes similar relations that the baseline confuses.The baseline predicts “walking on” as “standing on” and “in front of” as “behind” in the cited example.Source: §4.3 · p. 6, Figure 6 · p. 6
4.4 Parameter Analysis
- λ = 1 performs best on most metrics, performance drops slightly across [0.7, 1.3], and the highest mR@K uses 3 O2O and 2 R2O blocks.These settings are used in the full model.Source: §4.4 · p. 6, Table 4 · p. 6
5 Conclusion
The conclusion presents CogTree as a model-agnostic loss for unbiased scene graph generation with highly biased data. It organizes relationships into a coarse-to-fine tree from biased predictions, and identifies commonsense knowledge as future work.
5 Conclusion
- CogTree organizes relationships from biased SGG predictions into a multilevel tree representing coarse-to-fine relationships.The tree structure is used to organize otherwise independent relationships.Source: §5 · p. 6
- The CogTree loss supports hierarchical distinction of correct relationships while progressively eliminating irrelevant ones.Source: §5 · p. 6
- The loss is model-agnostic and consistently boosts performance across various SGG models.Source: §5 · p. 6
- Incorporating commonsense knowledge to optimize the CogTree structure is identified as future work.Source: §5 · p. 6