PaperlayerView exact revision on arXiv

Source-linked AI summary

Unbiased Scene Graph Generation from Biased Training

Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, Hanwang Zhang

arXiv:2002.11949v4cs.CVcs.LG

TL;DR

SGG struggles with biased relationship predictions, and conventional debiasing cannot reliably separate harmful long-tailed bias from useful contextual priors. The paper builds a causal graph and uses counterfactual Total Direct Effect inference to remove harmful bias, reporting better unbiased results with a diagnosis toolkit on Visual Genome and prevailing models.

  • Problem

    SGG relationship predictions are biased by long-tailed annotations, while useful contextual priors make it difficult for conventional debiasing methods to distinguish harmful from beneficial bias.

  • Method

    The method performs biased SGG training with a causal graph, then uses counterfactual Total Direct Effect inference to separate visual content effects from context bias.

  • Results

    The method significantly improves most predicates, and its unbiased results are considerably better than biased counterparts when evaluated with the Scene Graph Diagnosis toolkit.

  • Takeaways & Limitations

    The causal graph provides a general roadmap for applying TDE-based unbiased prediction across SGG models.

  • Takeaways & Limitations

    The causal graph is presented as broadly applicable, imposing no constraints on detailed implementation, but the supplied discussion does not state a narrower limitation.

Abstract

from arXiv · show

Today's scene graph generation (SGG) task is still far from practical, mainly due to the severe training bias, e.g., collapsing diverse "human walk on / sit on / lay on beach" into "human on beach". Given such SGG, the down-stream tasks such as VQA can hardly infer better scene structures than merely a bag of objects. However, debiasing in SGG is not trivial because traditional debiasing methods cannot distinguish between the good and bad bias, e.g., good context prior (e.g., "person read book" rather than "eat") and bad long-tailed bias (e.g., "near" dominating "behind / in front of"). In this paper, we present a novel SGG framework based on causal inference but not the conventional likelihood. We first build a causal graph for SGG, and perform traditional biased training with the graph. Then, we propose to draw the counterfactual causality from the trained graph to infer the effect from the bad bias, which should be removed. In particular, we use Total Direct Effect (TDE) as the proposed final predicate score for unbiased SGG. Note that our framework is agnostic to any SGG model and thus can be widely applied in the community who seeks unbiased predictions. By using the proposed Scene Graph Diagnosis toolkit on the SGG benchmark Visual Genome and several prevailing models, we observed significant improvements over the previous state-of-the-art methods.

1. Introduction

SGG is undermined by biased relationship prediction: long-tailed annotations make models favor trivial or coarse predicates, while useful contextual priors complicate debiasing. The paper proposes counterfactual TDE inference to remove harmful context effects while retaining visual relationship cues, and evaluates it with a broader diagnosis toolkit.

  • Problem: Biased relationship prediction leaves scene graphs with trivial, less informative relations despite accurate object detection.Examples include predictions such as near, on, and has, which provide limited information for downstream reasoning.
  • Problem: Long-tailed annotations cause frequent predicates to dominate rarer, more fine-grained relationships during inference.The paper illustrates this with a model trained on on far more often than standing on.
  • Motivation: Existing bias is not uniformly harmful because contextual priors can filter unreasonable relationship candidates.The paper distinguishes useful priors, such as person read book rather than eat, from harmful long-tailed effects.
  • Method: The proposed method compares factual and counterfactual scenes to remove context bias while preserving the main visual effect of the relationship.The counterfactual changes the content features while keeping context unchanged, and TDE operationalizes this comparison.
  • Results: TDE significantly improves most predicates, with improvements no longer following a long-tailed distribution.The authors interpret this pattern as evidence that gains come from the proposed method rather than better exploitation of context bias.
  • Evaluation: The Scene Graph Diagnosis toolkit combines relationship retrieval, mean Recall, and Sentence-to-Graph Retrieval for broader SGG evaluation.It is used on Visual Genome and prevailing baselines to assess bias and unbiased prediction.

2. Related Work

Prior SGG work emphasizes feature extraction and uses several debiasing strategies, but the paper frames TDE as a causal alternative that separates bias through counterfactual graph surgery without additional bias-modeling layers.

  • Scene Graph Generation: Existing SGG methods largely focus on improving feature extraction networks, while prior work introduced bias-sensitive evaluation through mean Recall.The cited approaches did not resolve the underlying biased SGG problem.
  • Debiasing SGG: Prior debiasing strategies include data augmentation or resampling, unbiased learning objectives, and disentangling biased from unbiased representations.These categories cover changes to data, training losses or curricula, and learned representations.
  • Causal Inference: TDE belongs to representation-disentangling approaches but separates bias through counterfactual causal-graph surgeries without training additional layers.The method directly operates on existing models rather than explicitly modeling bias with new layers.

3. Biased Training Models in Causal Graph

The framework represents SGG as a general causal graph in which image content, object features, object classes, and visual context contribute to predicate classification. Conventional biased training learns these dependencies using model-specific feature extraction, classification, fusion, and auxiliary losses.

  • Causal graph formulation: The causal graph models SGG as a directed acyclic graph whose variables interact through directed causal links.The graph is designed to capture how variables obtain their values before counterfactual analysis is applied.
  • Causal graph formulation: The graph is broadly applicable because it imposes no constraints on detailed model implementations.The paper instantiates it with VTransE, MOTIFS, and VCTree.
  • Model components: Object features are extracted from subject-object pairs, while object classes provide pairwise one-hot representations for predicate prediction.Different models use Bi-LSTMs, Bi-TreeLSTMs, or fully connected layers for object processing and feature merging.
  • Predicate classification: The predicate classifier receives pairwise object features, object-class language priors, and visual context from the union region of the two objects.The union-region branch applies RoIAlign to the combined bounding boxes before contextual processing.
  • Predicate classification: Final predicate logits combine the three input branches through fusion functions, with SUM and GATE tested in the framework.Auxiliary cross-entropy losses additionally predict logits from each branch to prevent one link, especially the object-class branch, from dominating.

4. Unbiased Prediction by Causal Effects

After biased training, the framework uses interventions and counterfactuals to separate visual content from context-specific bias. It replaces conventional predicate prediction with Total Direct Effect, requiring no additional parameters and applying across models.

  • Intervention and counterfactuals: Causal inference permits direct manipulation of graph variables after training instead of treating the entire SGG model as a black box.An intervention cuts incoming links to a variable and assigns it a specified value, enabling counterfactual prediction.
  • Intervention and counterfactuals: The pairwise object feature X is the control variable, with its intervened value set to the training-set mean feature or a zero vector.The intervention preserves the original object label z in the counterfactual world.
  • Total Direct Effect: Total Direct Effect computes unbiased prediction as the difference between the observed outcome and a counterfactual outcome that retains the original object labels.This removes context-specific bias while preserving the mediator associated with the observed object features.
  • Total Direct Effect: Unlike Total Direct Effect, Total Effect changes all descendants of X and removes only general dataset-level bias rather than mediator-specific bias.The distinction motivates selecting TDE for the paper’s unbiased predicate prediction.
  • Overall unbiased SGG: The final unbiased logits replace one-time prediction with observational and counterfactual passes corresponding to the TDE calculation.The procedure evaluates the original graph and an imaginary alternate before subtracting the latter from the former.
  • Overall unbiased SGG: TDE introduces no additional parameters and can be applied to a variety of SGG models.Its model-agnostic design supports use with existing trained architectures.

5. Experiments

Experiments evaluate SGG on Visual Genome using relationship retrieval, zero-shot retrieval, and sentence-to-graph retrieval, alongside implementation and ablation comparisons. TDE generally improves discrimination and retrieval over biased baselines and conventional debiasing methods, although some trade-offs remain.

  • Dataset and evaluation: Visual Genome experiments use 150 object categories and 50 predicate categories from the widely adopted split, with a 5k validation set sampled from training data.The full dataset contains 108k images, 75k object categories, and 37k predicate categories, but the evaluation follows the frequent-category split.
  • Dataset and evaluation: The diagnosis toolkit evaluates Relationship Retrieval, Zero-Shot Relationship Retrieval, and Sentence-to-Graph Retrieval using mean Recall@K and graph-level retrieval metrics.Relationship Retrieval includes PredCls, SGCls, and SGDet; S2GR evaluates Recall@20/100 and median ranking on galleries of 1,000 and 5,000.
  • Ablation studies: Conventional debiasing methods provide limited gains: Reweight reaches high mR@K in RR but scores 0.0/0.0 in ZSRR SGDet, while Focal barely works.The ablations also compare Resample, causal graph surgeries, TE, and NIE against TDE.
  • Quantitative studies: TDE improves RR and ZSRR over the baseline across causal alternatives, and even its worst result exceeds previous state-of-the-art methods by a large margin on RR mR@K.X2Y and X2Y-Tr provide limited improvements, TE is similar to TDE, and NIE is worse than baseline.
  • Qualitative studies: Qualitative examples show TDE replacing trivial or biased predicates such as on and holding with more discriminative relationships such as looking at.In S2GR examples, baseline graphs collapse walking and standing into on, losing detailed human actions and worsening retrieval.

6. Conclusions

The paper presents a general causal framework for unbiased SGG from biased training and evaluates it with a diagnosis toolkit. The conclusion reports that TDE-based predictions are considerably better than their biased counterparts.

  • Conclusions: The paper presents a general framework for unbiased SGG from biased training and identifies it as the first work addressing SGG’s serious bias issue.The framework is designed to separate harmful bias from useful context bias.
  • Conclusions: The method calculates Total Direct Effect with a causal graph to remove harmful bias while retaining good context bias.The causal graph is intended as a roadmap for training any SGG model.
  • Conclusions: Using the Scene Graph Diagnosis toolkit, the proposed unbiased SGG results are considerably better than their biased counterparts.

Abstract

The supplementary document reviews causal-effect analysis, details simplified network structures, adds quantitative studies, and provides qualitative studies.

  • Supplementary organization: The supplementary document includes a comprehensive review of causal-effect analysis in causal inference.
  • Supplementary organization: It also provides additional quantitative and qualitative studies alongside details of simplified network structures.

A. Review of Causal Effect Analysis

This section reviews causal-effect analysis using the paper’s proposed causal graph and original notation. It omits some formal concepts while preserving understanding of the framework.

  • Causal-effect analysis: The review explains causal-effect analysis through the causal graph proposed in Section 3 and retains the original paper’s notation.It points readers to standard causal-inference and mediation-analysis references for additional background.
  • Causal-effect analysis: A mediator is a descendant of X located on the path between X and an output variable Y.The paper illustrates the concept with a carcinogenesis example involving smoking.

A.2. Total, Direct and Indirect Effects

Total Effect captures the overall effect of a variable, while causal decomposition separates direct effects from mediator-driven indirect effects. The paper emphasizes evaluating the effect of interest under the real environment.

  • Total Effect: Total Effect measures X's overall effect on Y without counterfactual intervention on mediator Z.Under intervention X = ¯x, Z takes its value calculated from the causal graph for ¯x.
  • Effect Decomposition: Total Effect decomposes into Direct Effect along X →Y and Indirect Effect along X →Z →Y.The appropriate decomposition depends on whether the main effect or mediator side effect is the target.
  • Direct and Indirect Effects: TDE with NIE/PIE separates the effect of X in the real situation from mediator effects under a pure or natural environment.NIE/PIE changes Z from ¯z to z while keeping X at its unactivated value ¯x.
  • Direct and Indirect Effects: TIE with NDE/PDE instead isolates mediator effects in the real case and removes pure or natural direct effects.This decomposition is appropriate when the indirect effect of the mediator is the target.
  • Effect Decomposition: The paper recommends placing the effect of interest under the real environment to obtain case-specific results.For the two decompositions, this means using TDE or TIE rather than the corresponding pure or natural effect.

B.1. Scene Graph Generation

The implementation extracts multiscale visual features and encodes image and text scene graphs through shared bilinear attention. Training uses triplet loss with L1 distance and SGD.

  • Feature Extraction: The feature extractor uses ResNeXt-101-FPN with four scales, assigning each bounding box to a feature map according to its area.ROIAlign then extracts the feature for the selected bounding box.
  • Visual Context: The visual context module applies ROIAlign across all four feature maps before projecting context into R4096.The multiscale features provide complementary contextual information for each subject-object union box.
  • Sentence-to-Graph Retrieval: Sentence-to-Graph Retrieval represents image and text scene graphs with entities, relationships, and one-hot predicate categories.Image and text graphs use different embedding layers because their dictionaries differ.
  • Sentence-to-Graph Retrieval: A shared Bilinear Attention Network encodes entity-relation interactions for both image and text graphs into a common representation.Normalized scene-graph connectivity supplies the attention map, and the final graph feature has dimension 1024.
  • Training: The model is trained for 30 epochs with SGD, batch size 12, and learning-rate decay at epochs 10 and 25.The initial learning rate is 12 × 10^-2 and is reduced by a factor of 10 at each decay point.

C. Quantitative Studies

Quantitative studies compare baseline and TDE relationship retrieval using conventional Recall@K and mean Recall@K, with TDE producing finer-grained predicate predictions despite some conventional-recall decreases.

  • Relationship Retrieval: TDE shows a performance drop on conventional Recall@K, attributed to more fine-grained predicate classification.The evaluation includes both conventional Recall@K and mean Recall@K.
  • Predicate Classification: TDE makes predicate improvements less long-tailed than conventional debiasing methods on Predicate Classification.The comparison covers three models, two fusion functions, and baseline versus TDE.
  • Predicate Classification: TDE can improve frequent predicates such as behind and above rather than uniformly suppressing frequent predicates.These predicates are described as subclasses of near, showing that the gains do not simply follow distribution reweighting.

D. Qualitative Studies

Qualitative evaluations contrast baseline and TDE scene graphs for relationship retrieval and sentence-to-graph retrieval. TDE distinguishes entities and recovers discriminative predicates, but can overemphasize action predicates.

  • Relationship Retrieval: TDE produces relationship predictions that are more sensitive to different entities than the baseline in qualitative SGCls examples.The baseline often assigns the same relationship to multiple entities, such as placing the same sign on every pole.
  • Relationship Retrieval: TDE can overemphasize action predicates, producing holding for a pole-sign pair where on is described as more natural.This is identified as a problem of TDE in the qualitative comparison.
  • Sentence-to-Graph Retrieval: In sentence-to-graph retrieval, the baseline misses eating and consequently supplies only spatial relationships for the query caption.The reported example uses subgraphs from SGDet because the complete detected graphs contain many trivial objects and predicates.
  • Evaluation Design: The qualitative studies examine top-10 relationship retrieval, zero-shot relationship retrieval, and sentence-to-graph retrieval.The figures compare baseline and TDE outputs using colored boxes and relationship labels.
Loading 2002.11949v4…