PaperlayerView exact revision on arXiv

Source-linked AI summary

Recovering the Unbiased Scene Graphs from the Biased Ones

Meng-Jiun Chiou, Henghui Ding, Hanshu Yan, Changhu Wang, Roger Zimmermann, Jiashi Feng

arXiv:2107.02112v1cs.CVcs.MM

TL;DR

In scene graph generation, unequal missing labels across predicates contribute to long-tailed predictions, but existing debiasing methods do not address this reporting bias. The paper treats SGG as positive-unlabeled learning and uses label frequencies to recover unbiased probabilities, with DLFE estimating frequencies from augmented training examples. Experiments report more reliable estimates, long-tail alleviation, and state-of-the-art debiasing performance on VG.

  • Problem

    In SGG, unequal missing-label rates across predicates contribute to the long tail, and existing debiasing methods do not solve this reporting-bias problem.

    Source: §1 · p. 1

  • Method

    The paper treats SGG as positive-unlabeled learning, corrects biased probabilities using per-class label frequencies, and estimates those frequencies with DLFE during augmented training.

    Source: §1 · p. 2

  • Results

    DLFE yields more reliable label-frequency estimates than a naive traditional-estimator variant, alleviates the long tail, and achieves state-of-the-art debiasing performance on VG.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Qualitative examples show DLFE producing more descriptive inconspicuous relations and more balanced scene graphs.

    Source: §4.5 · pp. 8–9

    The probability recovery relies on the SCAR assumption that positive examples are selected for labeling at random regardless of x.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Given input images, scene graph generation (SGG) aims to produce comprehensive, graphical representations describing visual relationships among salient objects. Recently, more efforts have been paid to the long tail problem in SGG; however, the imbalance in the fraction of missing labels of different classes, or reporting bias, exacerbating the long tail is rarely considered and cannot be solved by the existing debiasing methods. In this paper we show that, due to the missing labels, SGG can be viewed as a "Learning from Positive and Unlabeled data" (PU learning) problem, where the reporting bias can be removed by recovering the unbiased probabilities from the biased ones by utilizing label frequencies, i.e., the per-class fraction of labeled, positive examples in all the positive examples. To obtain accurate label frequency estimates, we propose Dynamic Label Frequency Estimation (DLFE) to take advantage of training-time data augmentation and average over multiple training iterations to introduce more valid examples. Extensive experiments show that DLFE is more effective in estimating label frequencies than a naive variant of the traditional estimate, and DLFE significantly alleviates the long tail and achieves state-of-the-art debiasing performance on the VG dataset. We also show qualitatively that SGG models with DLFE produce prominently more balanced and unbiased scene graphs.

1 INTRODUCTION

Scene graph generation faces long-tailed predictions partly because inconspicuous relations are less likely to be labeled, making their probabilities more underestimated. The paper treats this reporting bias as a positive-unlabeled learning problem, recovers unbiased probabilities using label frequencies, and estimates those frequencies dynamically during training.

1 INTRODUCTION

  • In VG, head predicates occur far more often than tail predicates, and models favoring heads produce coarse-grained, less-informative scene graphs.The training examples for “on” outnumber those for “painted on” by about 830×; MOTIFS reaches 74.3 Recall@20 for “on” and 0.0 for “painted on.”Source: §1 · p. 1
  • Unlike long tails driven mainly by class-prior imbalance, SGG’s long tail is significantly affected by unequal missing labels: conspicuous predicates are more likely to be annotated than inconspicuous ones.Treating unlabeled pairs as negatives can underestimate probabilities, with greater underestimation for classes with lower label frequencies.Source: §1 · p. 1
  • The paper views SGG as positive-unlabeled learning and recovers unbiased relation probabilities by dividing biased probabilities by estimated per-class label frequencies.This correction is intended to offset reporting bias and raise confidence in under-estimated inconspicuous predicates.Source: §1 · p. 2
  • DLFE estimates label frequencies during training by averaging per-batch biased probabilities over augmented examples, increasing valid examples for estimation, especially in SGDet.This addresses the traditional post-training estimator’s inability to estimate some classes in SGDet; a trained MOTIFS model had no valid examples for 9 of 50 predicates.Source: §1 · p. 2
  • DLFE provides more reliable label-frequency estimates than a naive traditional-estimator variant, alleviates the long tail, and achieves state-of-the-art debiasing performance with more informative scene graphs.The estimates use training-time data augmentation and averages across iterations to introduce more valid examples.Source: §1 · p. 2

2 RELATED WORK

The related work situates the paper among scene-graph methods and debiasing approaches, distinguishing its focus on reporting bias from prior work on other sources of bias. It also frames SGG as positive-unlabeled learning and contrasts the paper’s inference-time correction with a prior pseudo-labeling approach.

2.1 Scene Graph Generation (SGG)

  • SGG covers predicting relationships with ground-truth boxes and labels, predicting relationships and object labels with boxes, or predicting all three from images alone.These settings are Predicate Classification (PredCls), Scene Graph Classification (SGCls), and Scene Graph Detection (SGDet), respectively.Source: §2.1 · p. 2
  • SGG systems commonly generate proposals, refine object contexts for classification, and use object and box features to predict relationships; prior debiasing methods address context, class relatedness, or missing-label bias.Less-biased mean-recall metrics helped draw attention to class imbalance in SGG.Source: §2.1 · p. 2
  • The paper targets reporting bias through unbalanced label frequencies, rather than class relatedness, context co-occurrence bias, or missing-label bias.The authors characterize their approach as model-agnostic debiasing.Source: §2.1 · p. 2

2.2 Positive Unlabeled (PU) learning

  • SGG is framed as positive-unlabeled learning because unlabeled examples may be negative or belong to one or more classes; the paper adopts inference of a biased model for convenience and flexibility.Treating every unlabeled example as negative can introduce missing-label bias and unbalanced predictions.Source: §2.2 · p. 2
  • Unlike Chen et al.’s three-stage pseudo-labeling approach, the paper says its method adapts to different SGG models with minimal modification and has superior debiasing performance.The cited approach generates pseudo-labels for unlabeled examples and then trains a less-biased model with those additional positives.Source: §2.2 · p. 3

3 METHODOLOGY

The method treats scene graph generation with missing labels as a positive-unlabeled learning problem and uses estimated per-class label frequencies to recover unbiased probabilities. DLFE estimates those frequencies during training using augmented data and multiple iterations.

3.1 Model Components

  • The SGG pipeline combines proposal generation, object classification, and relationship prediction, with graph-based context refinement used for objects and relations.The described components use a pretrained detector, message passing for object contexts, and pairwise features decoded into predicate probabilities.Source: §3 · p. 3, §3.1 · p. 3

3.2 Recovering the Unbiased Scene Graphs

  • Under a positive-unlabeled formulation, label frequency is the fraction of positive examples that are labeled, and correcting for it recovers unbiased probabilities.Treating unlabeled pairs as negative can produce missing-label and reporting bias; the method assumes labeled positive examples are selected at random regardless of features.Source: §3.2 · p. 4

3.3 Dynamic Label Frequency Estimation

  • DLFE estimates frequencies during training from per-batch biased probabilities, using horizontal flips and multiple iterations to add valid examples, especially for tail classes.It updates per-class moving averages with momentum and updates a class only when the batch contains a valid example; Train-Est can lack valid examples for some SGDet classes.Source: §3.3 · pp. 4–5
  • 4× or more valid examples for inconspicuous classes make DLFE’s frequency estimates possible for all classes without post-training estimation.Figure 3 compares per-class valid-example ratios for Train-Est and DLFE on VG150 SGDet; DLFE values are averaged over epochs.Source: Figure 3 · p. 4, §3.3 · p. 5

4 EXPERIMENTS

The experiments compare DLFE with Train-Est and other debiasing methods across SGG settings and backbones. They report stronger debiasing results and qualitative examples of more balanced predictions.

4.1 Evaluation Settings

  • Evaluation covers PredCls, SGCls, and SGDet, using per-class and non-graph-constraint mean recall to assess debiasing across head, middle, and tail predicates.The evaluation uses VG150 and reports recall at K=20, 50, or 100.Source: §4.1 · p. 5

4.2 Implementation Details

  • Experiments use MOTIFS and VCTree with a frozen Faster R-CNN detector and a shared training setup.The detector uses ResNeXt-101-FPN; models are trained with SGD, and all hyperparameters are tuned on the validation set.Source: §4.2 · p. 5

4.3 Comparing DLFE to Train-Est

  • DLFE estimates are more consistent across SGG settings and yield higher, more consistent per-class SGDet recall gains than Train-Est.Train-Est barely improves recall, especially for tail classes lacking enough valid examples.Source: Figure 4 · p. 5, Figure 5 · p. 5, §4.3 · p. 7

4.4 Comparing to other Debiasing Methods

  • DLFE significantly improves ng-mR for both MOTIFS and VCTree and outperforms existing debiasing methods.The comparison is on VG150; Table 1 reports ng-mR@K.Source: Table 1 · p. 6, §4.4 · p. 7
  • DLFE improves MOTIFS graph-constraint mR across all three settings, with state-of-the-art results; VCTree also attains new state-of-the-art results in PredCls, SGCls, and SGDet.The PredCls result for VCTree is specifically mR@20.Source: Table 2 · p. 6, §4.4 · p. 8
  • Only DLFE significantly improves tail-class recall; Reweighting also visibly improves tail classes, but by a smaller amount.Figure 6 compares non-graph-constraint per-class Recall@20 against the MOTIFS backbone-only model.Source: Figure 6 · p. 7, §4.4 · p. 8

4.5 Qualitative Results

  • Qualitative examples show more descriptive ground-truth matches and bias corrections, while DLFE raises confidence in other reasonable predicates.Examples include correcting a fork relation and increasing confidence in predicates such as mounted on and part of.Source: Figure 7 · p. 7, §4.5 · pp. 8–9, Figure 8 · p. 8

5 CONCLUSIONS

The conclusion frames SGG’s long tail as a reporting-bias problem caused by unbalanced missing labels. It presents PU-based probability recovery and DLFE as effective on VG.

5 CONCLUSIONS

  • The paper addresses SGG’s long tail at its proposed source, unbalanced missing labels, rather than only its long-tailed class distribution.It treats SGG as a PU-learning problem and recovers unbiased probabilities from biased ones.Source: §5 · p. 9
  • DLFE estimates label frequencies using training-time data augmentation and moving averages, introducing more valid samples, especially in SGDet.This supports recovery of unbiased probabilities.Source: §5 · p. 9
  • On VG, DLFE estimates label frequencies more effectively than a naive traditional estimator and achieves state-of-the-art debiasing performance with more balanced scene graphs.Source: §5 · p. 9

A DERIVING THE TRAIN-EST ESTIMATOR

The section derives a label-frequency estimate from biased probabilities for annotated examples. Averaging those probabilities over training or validation examples provides the estimate.

A DERIVING THE TRAIN-EST ESTIMATOR

B MORE RESULTS OF TRAIN-EST AND DLFE

Additional non-graph-constraint mean-recall results compare DLFE with Train-Est across two backbones and three SGG settings.

B MORE RESULTS OF TRAIN-EST AND DLFE

C RESULTS IN HEAD/MIDDLE/TAIL RECALLS

Across PredCls, SGCls, and SGDet, DLFE substantially improves tail recall while keeping head and middle recall on par with other debiasing methods. The gains hold for both MOTIFS and VCTree backbones.

C RESULTS IN HEAD/MIDDLE/TAIL RECALLS

  • In all three SGG tasks, MOTIFS-DLFE and VCTree-DLFE outperform other debiasing methods by a large margin on tail recall while remaining on par on head and middle recall.The comparisons include Reweighting, TDE, PCPL, and STL.Source: C RESULTS IN HEAD/MIDDLE/TAIL RECALLS · p. 11
  • The head, middle, and tail comparisons cover both MOTIFS and VCTree backbones across PredCls, SGCls, and SGDet.The bar plots show the two backbones side by side, with one row per task.Source: C RESULTS IN HEAD/MIDDLE/TAIL RECALLS · p. 11
  • Across recalls with and without graph constraint, DLFE improves middle and tail recalls more significantly, at a head-recall cost similar to other debiasing approaches.The reported task-specific results are in Tables 5, 6, and 7 for PredCls, SGCls, and SGDet, respectively.Source: C RESULTS IN HEAD/MIDDLE/TAIL RECALLS · p. 11

D MORE RESULTS IN RECALLS

DLFE achieves state-of-the-art mean recall among debiasing methods with the same backbone across PredCls, SGCls, and SGDet. The paper cautions that plain recall favors head classes and mean recall does not account for multiple relations per object pair.

D MORE RESULTS IN RECALLS

  • Plain recall favors head classes, while mean recall treats classes equally but still does not account for multiple relations between an object pair.The paper notes that debiasing methods may show drops in conventional recall when predictions are more descriptive than the annotated ground truth.Source: D MORE RESULTS IN RECALLS · p. 11
  • The comparison excludes because its reported numbers were selective and incomplete and its method was not fairly compared using the same backbone.The cited example of an unfair comparison is against.Source: D MORE RESULTS IN RECALLS · p. 11
  • Across all models and backbones, either MOTIFS-DLFE or VCTree-DLFE attains the highest mR scores, while DLFE achieves state-of-the-art mR@K among debiasing methods with the same backbone.Source: D MORE RESULTS IN RECALLS · p. 11

E ADDITIONAL QUALITATIVE RESULTS

Qualitative examples show DLFE shifting confidence toward more descriptive predicates, including fine-grained alternatives to broad predictions. The accompanying figures and tables cover confidence distributions and recall comparisons across SGG tasks.

E ADDITIONAL QUALITATIVE RESULTS

  • For table-window, bird-pole, and car-street examples, MOTIFS-DLFE raises confidence in more descriptive predicates; for one example, it raises mounted on and attached to despite keeping on as the top prediction.The parked on prediction for car-street is more descriptive but is absent from ground truth because of missing labels.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 11
  • Table 4 compares Train-Est and DLFE on non-graph-constraint mean recall across PredCls, SGCls, and SGDet.The caption identifies the comparison's scope but supplies no numerical outcomes.Source: Table 4 · p. 12
  • Figure 9 plots head, middle, and tail classes for MOTIFS and VCTree, with separate rows for PredCls, SGCls, and SGDet.The plots are evaluated on VG150.Source: Figure 9 · p. 12
  • Tables 5–7 report head, middle, and tail recalls with and without graph constraint for PredCls, SGCls, and SGDet, while Tables 8–10 report recall and mean recall for those tasks.The supplied captions do not provide table values.Source: Table 5 · p. 13, Table 6 · p. 13, Table 7 · p. 14
  • Figure 10 compares predicate confidence distributions from MOTIFS and MOTIFS with DLFE, coloring correct predicates green, incorrect ones red, and acceptable ones tangerine.Acceptable predicates are non-ground-truth predictions judged reasonable.Source: Figure 10 · p. 17
Loading 2107.02112v1…