PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Grained Scene Graph Generation with Data Transfer

Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua

arXiv:2203.11654v2cs.CVcs.AI

TL;DR

Long-tailed and semantically ambiguous annotations lead SGG models toward frequent, uninformative predicates. IETrans automatically enhances training data through internal and external transfer; experiments report macro-performance gains and improved large-scale predicate coverage.

  • Problem

    Long-tailed annotations and semantic ambiguity leave SGG models prone to predicting frequent, uninformative predicates, limiting downstream usefulness.

    Source: §1 · pp. 1–2

  • Method

    IETrans creates an enhanced training set by transferring general predicates to informative ones and relabeling missed relations from NA samples.

    Source: §1 · p. 2, §3.3 · p. 8

  • Results

    Across four baselines, IETrans boosts macro performance and achieves SOTA F@K; Neural Motif doubles mR@100 and reaches the highest F@100 on VG-50 predicate classification.

    Source: §1 · p. 3

  • Takeaways & Limitations

    On VG-1800, IETrans correctly predicts 467 predicate categories, compared with fewer than 70 for every other baseline.

    Source: §1 · p. 3

Abstract

from arXiv · show

Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the enhanced dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch.

1 Introduction

Long-tailed and ambiguous predicate annotations limit SGG; IETrans enhances training data through internal and external transfer, with reported gains across baselines and on large-scale SGG.

1 Introduction

  • In Visual Genome, over 90% of predicate classes have less than 10 samples; annotators prefer general predicates for simplicity, and predictions can collapse to them even when informative predicates have enough training samples.Source: §1 · p. 2
  • IETrans automatically transfers general predicates to informative ones and relabels missed relations from NA data to create an enhanced training set.Internal transfer uses triplet-level changes; external transfer draws on missed positive samples that current methods treat as negative.Source: §1 · pp. 2–3
  • On VG-1800, IETrans correctly predicts 467 categories, while every other baseline predicts fewer than 70.The benchmark contains 1,807 predicate classes and has more than five test samples per class.Source: §1 · p. 3
  • IETrans is plug-and-play, and the authors introduce VG-1800 as a benchmark for evaluating large-scale SGG.The paper describes the benchmark as providing reliable and stable evaluation.Source: §1 · p. 3

2 Related Works

Related work traces SGG from visual relation detection to context-aware generation, then reviews informative-predicate methods and large-scale benchmarks.

2.1 Scene Graph Generation

  • SGG developed from independently detected visual relations to context-aware dual-graph generation, followed by representation-refinement and vision-language approaches [36] [34].CPT [36] is noted for few-shot ability, while PEVL [34] is reported to outperform models trained from scratch.Source: §2.1 · pp. 3–4

2.2 Informative Scene Graph Generation

  • Prior work addresses trivial-predicate collapse with mean recall, causal inference, semantic relationships, or resampling, whereas this paper motivates triplet-level transfer.The passage notes that an informative alternative depends on the subject and object classes, so predicate-level adjustment can be less precise.Source: §2.2 · p. 4

2.3 Large-scale Scene Graph Generation

  • Large-scale SGG benchmarks include VG80K and a cleaned 2,000-class split, but their test sets have annotation noise or sparse categories; the authors re-split Visual Genome into 1,807 classes.The new split ensures more than five test samples per predicate class; the section also notes that semantic ambiguity limits data extension and motivates internal transfer.Source: §2.3 · pp. 4–5

3 Method

IETrans builds an enhanced dataset by transferring examples from general predicates to informative ones and relabeling missed relations. It combines the two transfers and retrains a model from scratch on the resulting data.

3 Method

  • IETrans aims to add tail-class training samples and make general predicates more informative by automatically enhancing the dataset.The framework combines internal transfer from general predicates and external relabeling of missed relations.Source: §3 · p. 5

3.1 Problem Definition

  • SGG represents detected objects and their predicate-labeled relations; inference detects objects, refines object and relation features, then classifies them.Source: §3.1 · pp. 5–6

3.2 Internal Data Transfer

  • The method averages a pretrained model’s prediction scores for each training-set triplet class and treats predicates scored above the ground truth as candidate confusion sources.Manual pair enumeration is impractical because relations depend on the subject–object classes, while existing knowledge bases do not match visual relation hierarchies.Source: §3.2 · pp. 6–7
  • Attraction factors identify general-to-informative transfer directions, account for triplet instance counts, and filter candidates; the top kI% samples are transferred, with conflicts assigned to the target having the highest attraction factor.The method notes that transferring to triplet types supported by only one or two samples may introduce annotation noise and hurt performance.Source: §3.2 · pp. 7–8

3.3 External Data Transfer

  • External transfer treats unannotated object pairs as possible missed relations, restricting candidates to overlapping boxes and existing relational triplet types.NA samples include both truly negative samples and missed positive samples; borrowing zero-shot relational triplets is left for future work.Source: §3.3 · p. 8
  • For each selected NA pair, the method assigns a predicate label, ranks pairs by ascending NA score, and transfers the top kE% as likely missed positives.Source: §3.3 · p. 8

3.4 Integration

  • Internal transfer uses annotated data and external transfer uses unannotated data, so their outputs can be merged before retraining a new model from scratch.The retrained model makes inferences on the test set.Source: §3.4 · pp. 8–9

4 Experiments

The experiments test IETrans across baseline models, large-scale SGG, transfer settings, and the enhanced dataset. IETrans improves macro-oriented performance across these settings, while the large-scale results show a trade-off between Acc and mAcc.

4.1 Generalizability with Different Baseline Models

  • IETrans boosts mR@K across four diverse models; GPS-Net’s mR@50/100 doubles and F@50/100 improves by over 9 points across all three tasks.The models include Motif, VCTree, GPS-Net, and Transformer; results also show competitive F@K performance.Source: §4.1 · pp. 10–11, Table 1 · p. 10

4.2 Expansibility to Large-Scale SGG

  • 19.12% vs. 4.37% top-10 mAcc: IETrans (kI = 10%) improves this metric while maintaining comparable Acc on VG-1800.IETrans (kI = 90%) achieves the best F-Acc, while varying kI traces an Acc–mAcc curve above the baseline points.Source: Table 2 · p. 11, §4.2 · p. 12, Figure 4 · p. 12
  • Over 3 times the second-highest baseline’s F-Acc, IETrans (kI = 90%) also predicts 467 predicate classes correctly, versus fewer than 70 for every other baseline.The comparison is on VG-1800; all baselines lie below the IETrans Acc–mAcc curve.Source: Table 2 · p. 11, §4.2 · p. 12, Figure 4 · p. 12

4.3 Ablation Studies

  • Internal transfer substantially improves mAcc, with a minor Acc drop; external transfer adds further benefit, and combining both maximizes their advantages.Top-10 F-Acc rises with kI through 80% before declining above 80%; external-transfer performance improves significantly with the last 10% of ranked data.Source: §4.3 · pp. 13–14, Figure 6 · p. 13, Figure 7 · p. 13

4.4 Analysis of Enhanced Dataset

  • IETrans is less accurate when transferring relations on VG-1800, but its enhanced dataset supplies samples for non-head classes.The analysis presents instance-level examples of changed relations and compares original versus enhanced predicate-frequency distributions.Source: §4.4 · p. 14, Figure 8 · p. 14

5 Conclusion

The paper presents IETrans as a data-transfer method for generating an enhanced SGG dataset. It concludes with experiments supporting the method and proposes extending it to other large-scale visual recognition problems.

5 Conclusion

  • IETrans combines internal relabeling of general predicates as informative ones with external completion of missed annotations to generate an enhanced SGG dataset.The authors propose exploring similar challenges in image classification and semantic segmentation in future work.Source: §5 · p. 14

A VG-1800 Dataset

VG-1800 is intended to provide reliable evaluation for large-scale scene graph generation.

A VG-1800 Dataset

A.1 Dataset Construction

The dataset construction filters Visual Genome predicate categories and creates train, validation, and test splits with minimum per-category counts.

A.1 Dataset Construction

  • VG-1800 filters unreasonable predicate categories and splits Visual Genome into 70% training and 30% test, with 5,000 training images reserved for validation.Each predicate category has at least five test samples and at least one training sample; retained categories have over five samples.Source: §A.1 · p. 17

A.2 Dataset Statistics

VG-1800 provides a large predicate-class benchmark with broader tail-class test coverage than VG8K-LT. Its dataset scale and split sizes are also reported.

A.2 Dataset Statistics

  • VG-1800 contains 70,098 object categories, 1,807 predicate categories, 272,084 distinct relation triplets, and 66,289/4,995/32,893 train/validation/test images.The dataset averages 19.5 objects and 16.0 relations per image.Source: §A.2 · p. 17
  • VG-1800 has 1,807 predicate classes with at least 5 test samples, compared with 526 for VG8K-LT.The authors also describe VG-1800 as cleaner than VG8K and more stable for evaluating many tail classes than VG8K-LT.Source: Figure 10 · p. 18, §A.2 · p. 18

B Implementation Details

The implementation removes resampling, reweighting, and frequency bias for fair comparison and informative predictions, with separate settings for VG-50 and VG-1800.

B.1 VG-50

  • On VG-50, reproduced Motif, Transformer, VCTree, and GPS-Net baselines without resampling, reweighting, or frequency bias.Removing frequency bias may raise mR@K and F@K while lowering R@K relative to the models’ original-paper results.Source: §B.1 · pp. 18–19
  • Transformer experiments used a batch size of 16 on two GPUs, with lower learning rates for PREDCLS and SGDET and an additional reduction for SGCLS.The learning rates were 0.08 for PREDCLS and SGDET, and 0.016 for SGCLS.Source: §B.1 · p. 19

B.2 VG-1800

  • On VG-1800, the authors retained the VG-50 backbone and training schedule, removed frequency-bias items, and set kI to 90% and kE to 100%.Frequency bias was removed to reduce memory use given the increase in subject-predicate-object combinations.Source: §B.2 · p. 19

C Supplementary Experiments

Internal and external transfer respond differently to their transfer percentages: more internal transfer is not always beneficial, while the final external samples can improve mAcc.

C.1 Influence of kI abd kE

  • On VG-1800, internal transfer raises mAcc through kI ≤80%, but the final 20% lowers it; Acc decreases linearly as kI increases.The authors suggest that noise in the last 20% may reduce training-data quality.Source: §C.1 · p. 19
  • External transfer has almost no effect on Acc or mAcc through kE ≤90%, while kE = 100% significantly boosts mAcc.The authors suggest the final 10% provides valuable tail-class training samples that models fail to distinguish from NA samples.Source: §C.1 · p. 19

C.2 Adaptive Threshold.

The section tests adaptive thresholds for internal transfer and reports results for external transfer and VG-1800 SGCLS. Adaptive thresholds underperform fixed percentages, while SGCLS results show both gains and a remaining task gap.

C.2 Adaptive Threshold

  • Adaptive internal transfer relabels a general triplet when its informative-predicate score exceeds the mean score for real examples plus k times the standard error.The rule uses prediction scores for the corresponding informative predicate and its real relational-triplet instances.Source: §C.2 · p. 20
  • The adaptive-threshold model performs worse overall than fixed percentages, with the authors suggesting scarce informative triplets can yield near-zero averages and over-transfer.They leave a more intelligent adaptive threshold for future work.Source: §C.2 · p. 20
  • For external transfer, missed-annotation samples with the highest NA scores bring the greatest performance benefits, despite unreliable NA scores..Source: §C.2 · p. 20

SGCLS Results on VG-1800

  • On VG-1800 SGCLS, IETrans (kI = 10%) improves top-1 mAcc and Non-Zero by over 5 times versus Motif, with less than 1 point lost in top-1 Acc.IETrans achieves the best F-Acc across top-1, top-5, and top-10 versus other baselines; top-1 F-Acc at kI = 90% is 2.62% for SGCLS versus 4.70% for PREDCLS.Source: Table 4 · p. 21, §C.2 · p. 21

D Discovered Visual Hierarchy Analysis

Human evaluation finds most sampled transferred relation pairs reasonable, while the discovered examples show predicates becoming more specific. The examples are ranked by how many samples were transferred.

D Discovered Visual Hierarchy Analysis

Related papers

Loading 2203.11654v2…