Source-linked AI summary
PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation
Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, Xian-Sheng Hua
TL;DR
Long-tailed predicate distributions limit unbiased scene graph generation, and frequency-based re-balancing can mishandle strongly correlated predicates. PCPL adaptively uses predicate correlations to set loss weights and adds graph-based context encoding; on VG150, it improves tail classes, preserves head performance, and significantly outperforms prior state-of-the-art methods in mean recall.
Problem
Long-tailed predicate annotations and correlations between predicate classes make unbiased scene graph generation difficult, while conventional re-balancing relies on frequencies without directly using relatedness.
Method
PCPL dynamically perceives predicate correlations through a learnable class graph to assign relatedness-aware loss weights, with a graph encoder for context features.
Results
PCPL performs better on tail classes while preserving head-class performance and significantly outperforms previous state-of-the-art methods in mean recall on VG150.
Takeaways & Limitations
Predicate relatedness can guide adaptive loss weighting for reducing long-tailed bias in scene graph generation.
Takeaways & Limitations
The paper’s motivating assumption is that predicates strongly correlated with multiple classes can benefit from learning those correlated classes.
Abstract
from arXiv · showhide
Today, scene graph generation(SGG) task is largely limited in realistic scenarios, mainly due to the extremely long-tailed bias of predicate annotation distribution. Thus, tackling the class imbalance trouble of SGG is critical and challenging. In this paper, we first discover that when predicate labels have strong correlation with each other, prevalent re-balancing strategies(e.g., re-sampling and re-weighting) will give rise to either over-fitting the tail data(e.g., bench sitting on sidewalk rather than on), or still suffering the adverse effect from the original uneven distribution(e.g., aggregating varied parked on/standing on/sitting on into on). We argue the principal reason is that re-balancing strategies are sensitive to the frequencies of predicates yet blind to their relatedness, which may play a more important role to promote the learning of predicate features. Therefore, we propose a novel Predicate-Correlation Perception Learning(PCPL for short) scheme to adaptively seek out appropriate loss weights by directly perceiving and utilizing the correlation among predicate classes. Moreover, our PCPL framework is further equipped with a graph encoder module to better extract context features. Extensive experiments on the benchmark VG150 dataset show that the proposed PCPL performs markedly better on tail classes while well-preserving the performance on head ones, which significantly outperforms previous state-of-the-art methods.
1 INTRODUCTION
Scene graph generation represents objects and their semantic relationships for downstream vision-and-language tasks, but long-tailed predicate distributions make unbiased relationship prediction difficult. PCPL addresses this imbalance by using predicate correlations to assign adaptive loss weights, alongside graph-based context encoding.
- Scene graph generation detects objects and recognizes semantic relationships, supporting image generation, retrieval, visual question answering, and captioning.
- Frequency-based re-sampling and re-weighting improve mean recall but manually tune sampling or classifier costs without accounting for predicate relatedness.
- PCPL perceives correlations among predicate classes and inversely adapts their loss weights, improving tail predicates while preserving head-predicate performance.
- A graph encoding module extracts global context through stacked graph encoders to strengthen relationship representations.
2 RELATED WORK
Prior SGG methods improve relationship prediction through contextual visual features, semantic or statistical information, and class-imbalance strategies. PCPL differs by adapting training costs from predicate relatedness rather than sample frequency alone.
- Relationship prediction methods enhance visual reasoning with contextual message passing, recurrent global-context encoding, semantic labels, and statistical correlations.
- Mean recall@K was introduced to expose class imbalance, while later methods addressed long-tailed bias through causal inference or balanced visual-feature extraction.
- Figure 2 compares constrained and unconstrained recall@100 improvements for weakly versus strongly correlated predicate groups.
- Traditional re-balancing uses re-sampling or re-weighting based mainly on class frequencies, including inverse-frequency, effective-number, and focal-loss schemes.
- PCPL adaptively assigns training costs from predicate relatedness instead of relying solely on sample frequencies.
3 METHODS
The method addresses predicate imbalance by measuring predicate relatedness and using it to adapt classification loss weights, while also encoding global scene context with graph-based attention. Experiments and observations indicate that frequency-based re-balancing can fail for correlated predicates, whereas PCPL aims to improve tail performance without sacrificing head classes.
- 3.2 An observation of re-balancing: Re-balancing improves weakly correlated tail predicates but can overfit strongly correlated tail predicates and reduce the primary predicate’s constrained R@100.The comparison uses matched-frequency predicate groups and evaluates constrained and unconstrained R@100 against cross-entropy.
- 3.3 Learning Process: PCPL adaptively assigns predicate loss weights from class relatedness, giving smaller weights to predicates correlated with multiple classes and higher weights to more independent predicates.Predicate relatedness is estimated from distances between learnable predicate class centers and converted into correlation factors used as classification loss weights.
- 3.3 Learning Process: The class-correlation graph is updated during feature-network optimization, enabling PCPL to track changing predicate relatedness throughout training.Predicate centers are updated first, followed by graph edges and global class correlations.
- 3.3 Learning Process: PCPL can use the dynamic graph to identify and discard noisy samples, making training more stable to some extent.A sample is treated as noisy when the graph-based Ddrop criterion is positive for any class, with λ set to 2.
- 3.4 Context Encoding: The graph encoding module contextualizes region features by propagating messages through stacked self-attention and feed-forward encoders.Each encoder attends over neighboring nodes, and repeated encoders produce final contextual representations for image regions.
4 EXPERIMENTS
Experiments on VG150 evaluate PCPL across SGG tasks, metrics, debiasing baselines, predicate classes, and component ablations. PCPL improves mean recall while preserving head-predicate performance, with graph encoding and noisy-label dropping providing additional benefits.
- Compared methods: 22.1% and 32.9% mean constrained mR@K, with relative improvements of 31.5% and 24.2% over previous state-of-the-art methods across the three VG150 tasks.PCPL achieves higher constrained mR@K than the compared methods on PredCls, SGCls, and SGGen.
- Compared methods: PCPL improves mR@50/100 over re-weighting while obtaining comparable R@50/100 with SMN+TDE, indicating a more balanced debiasing result.Class-balanced loss and re-weighting improve mean recall but substantially reduce recall, whereas PCPL avoids that trade-off in the reported comparison.
- Compared methods: 83%, 93%, 97.5%, and 90.2% recall@100 on four head predicates represent improvements of 29.9%, 13.8%, 8.6%, and 8.1% over re-weighting.PCPL also improves tail predicates such as walking in, mounted on, and painted on.
- Ablation Study: MinMaxNorm outperforms SoftmaxNorm and ScalingNorm because it magnifies correlation-factor differences instead of producing an overly smooth distribution.The normalization comparison concerns how global predicate correlations affect adaptive loss weights.
- Ablation Study: PCPL performance rises over the GE baseline on all three tasks, while GE itself outperforms IMP+, FREQ, and SMN.These ablations support the contributions of predicate-correlation learning and graph-based context encoding.
- Ablation Study: Noisy-label dropping provides a further performance improvement when added to the model.The ablation attributes this additional gain to the noisy-label dropping schema.
5 CONCLUSION
PCPL addresses SGG’s long-tailed bias by modeling correlations among predicate classes and adaptively assigning loss weights, with graph encoding for context features. On VG150, it improves tail performance while preserving head performance and significantly outperforms prior state-of-the-art methods.
- The framework adaptively assigns predicate loss weights by directly perceiving and utilizing correlations among predicate classes to address SGG’s long-tailed bias.
- PCPL significantly outperforms previous state-of-the-art methods on VG150 in mean recall while improving tail classes and preserving head performance.