PaperlayerView exact revision on arXiv

Source-linked AI summary

A Causal Adjustment Module for Debiasing Scene Graph Generation

Li Liu, Shuzhou Sun, Shuaifeng Zhi, Fan Shi, Zhen Liu, Janne Heikkilä, Yongxiang Liu

arXiv:2503.17862v1cs.CVcs.AI

TL;DR

SGG debiasing has often focused on long-tailed relationships, while skewed object and object-pair distributions also contribute to bias. The paper models these distributions with a co-occurrence mediator and uses CAModule for triplet-level adjustment. It reports state-of-the-art mean recall and substantial improvement in zero-shot recall across benchmarks and SGG backbones.

  • Problem

    SGG debiasing often attributes bias to long-tailed relationships, while skewed object and object-pair distributions are also underlying sources of bias.

    Source: §1 · pp. 1–2

  • Method

    MCCM models object, co-occurrence, object-pair, and relationship distributions, and CAModule uses these variables to produce triplet-level adjustment factors.

    Source: §1 · p. 2

  • Results

    CAModule achieves state-of-the-art mean recall rates and substantial improvement on zero-shot recall across SGG backbones and benchmarks.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The method composes zero-shot relationships, with object-pair inference rules further supporting recognition of such relationships.

    Source: §1 · p. 2

    Zero-shot relationship composition presupposes that its object pair appears in training, so it does not directly cover test-time zero-shot pairs absent from training.

    Source: §3.4 · p. 9

Abstract

from arXiv · show

While recent debiasing methods for Scene Graph Generation (SGG) have shown impressive performance, these efforts often attribute model bias solely to the long-tail distribution of relationships, overlooking the more profound causes stemming from skewed object and object pair distributions. In this paper, we employ causal inference techniques to model the causality among these observed skewed distributions. Our insight lies in the ability of causal inference to capture the unobservable causal effects between complex distributions, which is crucial for tracing the roots of model bias. Specifically, we introduce the Mediator-based Causal Chain Model (MCCM), which, in addition to modeling causality among objects, object pairs, and relationships, incorporates mediator variables, i.e., cooccurrence distribution, for complementing the causality. Following this, we propose the Causal Adjustment Module (CAModule) to estimate the modeled causal structure, using variables from MCCM as inputs to produce a set of adjustment factors aimed at correcting biased model predictions. Moreover, our method enables the composition of zero-shot relationships, thereby enhancing the model's ability to recognize such relationships. Experiments conducted across various SGG backbones and popular benchmarks demonstrate that CAModule achieves state-of-the-art mean recall rates, with significant improvements also observed on the challenging zero-shot recall rate metric.

1 INTRODUCTION

The introduction argues that SGG bias reflects skewed object and object-pair distributions as well as long-tailed relationships, motivating a causal model with co-occurrence as a mediator. CAModule uses this structure for triplet-level adjustment, and the paper reports strong mean-recall and zero-shot-recognition results.

1 INTRODUCTION

  • Prior debiasing methods focus on relationship long tails, while skewed object and object-pair distributions also contribute to SGG bias.Objects influence pair frequencies, and object pairs generate relationships, so imbalance can propagate through this chain.Source: §1 · pp. 1–2
  • MCCM models objects, co-occurrence, object pairs, and relationships, inserting co-occurrence as a mediator between objects and object pairs.The mediator addresses cases where object abundance alone does not predict which pairs commonly co-occur, such as frequent “man” and “tree” objects but less frequent <man, tree> pairs.Source: §1 · p. 2
  • The proposed inference rules expand object-pair distributions with inferred zero-shot pairs, supporting composition of additional zero-shot relationships.The paper notes that relationship composition alone does not cover test-time zero-shot relationships whose object pairs were absent during training.Source: §1 · p. 2
  • CAModule estimates MCCM’s causal structure and produces triplet-level adjustment factors to correct biased SGG predictions.The factors are learned in the causal module rather than set as inverse category frequencies.Source: §1 · p. 2
  • CAModule achieves state-of-the-art mean recall rates across SGG backbones and benchmarks, with substantial improvement on the challenging zero-shot recall rate metric.The introduction attributes zero-shot recognition gains to composing relationships and proposes object-pair inference rules to extend that capability.Source: §1 · p. 2

2 RELATED WORKS

Related work surveys SGG debiasing strategies and causal approaches, then distinguishes this paper’s modeling scope and adjustment granularity. It emphasizes object, co-occurrence, and object-pair distributions alongside relationship imbalance.

2.1 Scene Graph Generation

  • SGG debiasing methods include resampling, reweighting, output adjustment, and hybrid strategies, each addressing imbalance through a different stage or combination of strategies.Resampling changes data composition; reweighting changes training influence; adjustment modifies outputs; hybrid methods combine approaches.Source: §2.1 · p. 3
  • Existing SGG debiasing approaches face trade-offs, while this work models object and object-pair imbalance and uses co-occurrence mediation with triplet-level adjustment.The paper contrasts its factors with relationship-level adjustment, which can cause cross-relationship interference.Source: §2.1 · p. 3

2.2 Causal Inference

  • Prior causal SGG methods target biases such as bad causal relationships, semantic confusion, dual imbalance, or spurious context-predicate correlations.The survey describes TDE, TsCM, DSDI, and common-sense causal reasoning approaches.Source: §2.2 · p. 4
  • This work models object, co-occurrence, object-pair, and relationship distributions, and reports gains on both mR@K and zero-shot zR@K.The authors attribute zero-shot gains to composing object-pair and relationship combinations.Source: §2.2 · p. 4

3 METHODS

This section presents a causal approach to scene-graph bias, modeling skewed object, co-occurrence, object-pair, and relationship distributions with MCCM and using CAModule to adjust triplet predictions. It also describes mediator-based object-pair modeling and attribute-based inference for zero-shot relationship recognition.

3.1 Overview

  • CAModule replaces two-pass TDE with one-inference, triplet-level logit adjustment because object and relationship imbalance makes same-relation triplets behave differently.The overview also reports a notable zR@K improvement, tentatively attributed to MCCM’s ability to combine potential zero-shot relationships.Source: §3.1 · p. 5

3.2 Modeling Structural Causal Model

  • The method adopts SCM for intervention analysis and estimates ATE across object, subject, and relationship jointly, enabling triplet-specific counterfactual adjustment.Unlike relationship-level methods that assign one factor to all triplets sharing a relation, its counterfactual comparison intervenes on object-pair distribution states and requires one model inference.Source: §3.1 · p. 5, §3.2 · pp. 6–7
  • MCCM inserts co-occurrence C between object O and pair P, correcting unrealistic pair priors and downweighting implausible pairs before relationship prediction.The paper reports that standard CCM can overestimate pairs such as <man, tree> because it relies heavily on individual object frequencies, whereas co-occurrence can favor realistic pairs such as <man, shirt>.Source: §3.2 · pp. 6–7

3.3 Causal Adjustment Module

  • CAModule uses a transformer encoder on observed O, C, P, and R distributions to model O → C → R and O → P → R and output adjustment factors.For each batch, it extracts sub-distributions corresponding to the relationship instances rather than processing the complete distributions.Source: §3.3 · pp. 7–8
  • CAModule fuses O with C and P with R through transformer blocks, then applies implicit causal adjustment factors to baseline logits without modifying the baseline model.The module is model-agnostic and uses the baseline model’s usual cross-entropy supervision.Source: §3.3 · pp. 8–9
  • The MCCM models O → C → P, using cooccurrence C to constrain object-pair distributions and capture indirect effects that can help predict rare relationships.This mediator distinguishes the approach from direct O → P modeling and enables adjustment across causal pathways.Source: §3.3 · p. 9

3.4 Optimize Object Pair Distribution P

  • CAModule achieved state-of-the-art mR@K and substantially improved zR@K by recombining object pairs with relationships, while C downweights implausible triplets.This recombination can generate zero-shot relationships when their object pairs occur in training; it does not cover zero-shot pairs absent from training.Source: §3.4 · p. 9
  • To infer zero-shot object pairs, the method substitutes objects with similar attributes in observed triplets, measuring similarity with GloVe features, Euclidean distance, and cosine similarity.Experiments indicate optimal performance at α = 0.7; lower β recalls more zero-shot pairs but may admit implausible pairs, while higher β favors plausibility but misses pairs.Source: §3.4 · pp. 9–10

3.5 Compared to Current Debiasing Method

  • Unlike methods focused on relationships, CAModule adjusts O, C, P, and R at the triplet level in an implicit space, targeting within-relation tails and zero-shot generalization.The paper contrasts this with explicit adjustment parameters such as reciprocal category frequencies; it also notes resampling and reweighting drawbacks and says GloVe adds no external knowledge beyond common SGG embeddings.Source: §3.5 · p. 10

4 EXPERIMENTS

Experiments compare CAModule with debiasing baselines across datasets, SGG backbones, and evaluation modes. They report strong mR@K, R@K, and MR@K results, alongside ablations of zero-shot pair optimization, similarity choices, and computational cost.

4.1 Experiment Setup

  • Experiments span VG150, GQA, and OI V6, using MotifsNet, VCTree, and Transformer across PredCls, SGCls, and SGDet modes.The evaluation also reports mR@K, R@K, and MR@K, with additional OI V6 metrics.Source: §4.1 · pp. 10–11

4.2 Comparison with State-of-the-art

  • 2.7%, 8.6%, 6.8%, and 9.3% higher mR@100 than TransRwt, PPDL, RTPB, and NICEST, respectively, for VCTree PredCls on VG150; the method also leads on VG150, GQA, and OI V6 mR results.The comparison covers state-of-the-art mR@K results on VG150 and GQA, with similar results on OI V6.Source: Table 1 · p. 11, §4.2 · pp. 11–12, Table 4 · p. 12
  • 8.0%, 8.9%, and 4.4% higher R@100 than TransRwt, PPDL, and TBE, respectively, in MotifsNet SGCls; MR@K also exceeds debiasing baselines and biased backbones.For MotifsNet SGCls, CAModule’s R@K decrease versus the biased backbone is 1.7%, compared with larger drops for TransRwt, PPDL, and TBE.Source: §4.2 · pp. 12–13, Table 2 · p. 12, Table 3 · p. 12

4.3 Ablations and Hyper-parameters

  • 6.3%, 3.9%, and 6.2% higher zero-shot recall than CAModule−P, and 12.7%, 7.5%, and 8.0% above the original backbones, respectively, across three backbones in PredCls.Inferred plausible zero-shot object pairs are weighted by attribute similarity and added to object-pair distribution P.Source: Table 5 · p. 13, §4.3 · p. 14

4.2 Comparison with State-of-the-art

  • Qualitative comparisons show finer-grained tail, spatial, and semantic relationship predictions, including specific relations where baselines predict broader ones.The paper attributes these changes to tail-friendly logit adjustments based on object, object-pair, co-occurrence, and relationship distributions, and to language-model features for semantic associations.Source: §4.2 · p. 13

4.3 Ablations and Hyper-parameters

  • Vanilla logit adjustment has almost no effect on SGG bias; combining Euclidean distance and cosine similarity improves performance over either alone, with cosine similarity stronger.The similarity ablations report gains over unoptimized P of 2.1% for Euclidean distance and 3.4% for cosine similarity in MotifsNet PredCls.Source: §4.3 · pp. 13–15, Table 7 · p. 14, Table 6 · p. 14
  • α = 0.7 yields optimal performance; β = 0.4 gives the strongest zero-shot capability, while looser or stricter thresholds perform worse.Low β can admit implausible pairs, whereas high β can miss zero-shot pairs.Source: §4.3 · pp. 15–16, Figure 8 · p. 15, Figure 9 · p. 15
  • 3% to 5% additional inference and training time accompanies CAModule integration, and the authors describe this overhead as negligible.They identify more efficient causal-relationship modules as future work.Source: §4.3 · p. 16, Table 8 · p. 16

5 CONCLUSION

CAModule uses MCCM to model causality among objects, object pairs, and relationships, including mediator variables, and produces triplet-level adjustment factors. The authors report state-of-the-art mean recall and substantial progress on zero-shot recall, while identifying zero-shot recognition as a capability for further refinement.

5 CONCLUSION

  • MCCM models causality among objects, object pairs, and relationships, incorporating mediator variables to enrich the modeled causality.CAModule uses MCCM variables to produce triplet-level adjustment factors aimed at correcting biased predictions.Source: §5 · p. 16
  • MCCM composes zero-shot relationships, and optimizing the object pair distribution further enhances recognition of them.The authors plan to refine this capability, which they consider important for complete debiasing in SGG.Source: §5 · p. 16
  • CAModule achieves state-of-the-art performance on mean recall and makes substantial progress on zero-shot recall.The authors say zero-shot recognition still needs refinement toward complete debiasing.Source: §5 · p. 16

Related papers

Loading 2503.17862v1…