Learning of Visual Relations: The Devil is in the Tails

1. Introduction

• Scene graph generation must jointly handle long-tailed entity recognition and visual relation recognition, where relations are subject-predicate-object tuples.A popular subject such as “ball” can occur in a rare tuple such as “robot-kicking-ball.”Source: §1 · p. 1
• Complexity-focused SGG methods may overfit popular predicates, while infrequent predicates can provide more informative depictions of scene content.The paper notes that Recall@k favors highly populated classes.Source: §1 · p. 2
• Visual Genome has skewed entity and predicate distributions of different magnitudes, and the paper reports that prior SGG methods had not considered them jointly.The proposed DT2 uses a simpler architecture with a more sophisticated training procedure.Source: §1 · p. 2
• Alternating Class-Balanced Sampling (ACBS) captures the interplay between long-tailed entity and predicate distributions within DT2.The paper presents ACBS as a novel sampling strategy.Source: §1 · p. 2
• DT2-ACBS significantly outperforms state-of-the-art methods with more complex architectures on all Visual Genome scene graph generation tasks.DT2 also uses a model with 10× fewer parameters and significantly outperforms prior architectures under mRecall@K.Source: §1 · p. 2

2.1. Scene graph generation

• SGG approaches often target Recall@K with complex architectures, while underrepresented classes perform worse; prior methods addressed dataset bias but considered at most one long-tailed distribution and did not disentangle sampling gains from architectural complexity.Examples include causal inference, balanced visual features, predicate correlations, and confidence-based gating with resampling.Source: §2.1 · p. 2

2.2. Long-tailed recognition

• Long-tailed recognition approaches include data resampling, cost-sensitive losses, and transfer learning.Resampling changes class representation; cost-sensitive methods weight errors by frequency or difficulty; transfer methods share information from head to tail classes.Source: §2.2 · pp. 2–3
• Decoupled recognition methods use different sampling strategies across representation and classifier learning, motivating different distributions for training entity and predicate components.The cited approaches include decoupled stages and a two-stream model with mixed sampling.Source: §2.2 · p. 3

3.1. Definitions

• Visual relation inference is decomposed into bounding-box prediction, entity classification given boxes, and predicate classification given boxes and entity labels.The scene graph is represented as entities and their relations.Source: §3.1 · p. 3
• The paper distinguishes Scene Graph Detection, Predicate Classification, and Scene Graph Classification by which box and entity-category information is supplied.PredCls receives both boxes and entity classes; SGCls receives boxes only.Source: §3.1 · p. 3

3.2. Long-tailed visual relations

• Visual Genome’s most frequent entity class is 35× more populated than its least frequent, while the predicate ratio is 12,000× (5,000× excluding the least frequent predicate).The predicate imbalance exceeds the square of the entity-class ratio, 1,225×.Source: §3.2 · p. 3
• Long-tailed entity frequencies combine with predicate rarity and entity-pair combinations, producing relation distributions more skewed than either source alone.Rare predicates can occur even with popular entities, and relations can remain rare even when their entities and predicate are frequent.Source: §3.2 · p. 3
• Recall@K averages over ground-truth triplets and is dominated by frequent relation classes, so it does not penalize ignoring infrequent classes.The paper links this concern to the possibility of systems detecting only a few common, less informative relations.Source: §3.2 · pp. 3–4
• The paper proposes testing whether a simpler model with sampling designed for long-tailed visual relations can outperform more complex models that ignore this property.The authors note that adopting mRecall@K addresses evaluation imbalance but does not itself make the learning algorithm address imbalance.Source: §3.2 · p. 4
• mRecall@K averages recall within predicate classes and then across classes, addressing class imbalance at evaluation but not in the learning algorithm.The paper argues that learning should explicitly address the imbalance.Source: §3.2 · p. 4

4.1. Notations

• A relation is represented as a subject-predicate-object tuple; entities include bounding boxes and class labels, and softmax outputs are probability vectors.Bounding boxes may come from ground truth or a detector, depending on whether the task is SGCls or SGDet.Source: §4.1 · p. 4

4.2. Model architecture

• The entity encoder extracts semantic and appearance features, which are paired with bounding-box embeddings and passed to a predicate classifier.The semantic branch predicts entity classes, while the appearance branch captures visual cues needed to distinguish relationships between the same entity classes.Source: §4.2 · pp. 4–5

4.3. Training

• Entity and predicate classification use cross-entropy losses so the network learns from both entity labels and relation predicates.The losses compare predicted entity and predicate probabilities with their respective ground-truth one-hot labels.Source: §4.3 · p. 5

4.4. Sampling strategies

• Standard random sampling trains the feature extractor and classifier together, then class-balanced sampling relearns only the classifier weights with the feature extractor fixed.Restricting balanced-sampling fine-tuning to classifier weights addresses the risk of overfitting to resampled tail classes.Source: §4.4 · p. 5

4.5. Sampling for visual relationships

• Entity and predicate distributions differ, so visual-relation learning requires distinct class-balanced sampling strategies for the two classifier steps.Predicate-balanced sampling alone is not class-balanced for entities and can overfit the entity-classification parameters.Source: §4.5 · pp. 5–6
• ACBS alternates entity- and predicate-balanced sampling, using a teacher classifier and distillation to preserve entity predictions across steps.The P-step updates predicate and teacher parameters; the E-step updates the student entity classifier, encouraging it to mimic the teacher.Source: §4.5 · p. 6

5.1. Dataset

• VG150 uses 150 object classes and 50 predicate classes, and predicates with fewer than five instances are ignored during balanced sampling.The dataset's predicate distribution remains highly long-tailed.Source: §5.1 · p. 6

5.2. Comparison to SOTA

• DT2-ACBS improves mR@100 over existing methods by 1.9% on PredCls, 7.9% on SGCls, and 12.7% on SGDet.The PredCls gain is reported against methods addressing predicate long-tail bias; the SGCls and SGDet passages describe margins over existing methods.Source: Table 1 · p. 7, §5.2 · p. 7
• DT2-ACBS improves middle- and tail-class performance, while remaining comparable on head classes for SGCls and SGDet.Some apparent head-class degradation reflects predictions of finer-grained predicates such as “standing on” or “sitting on” instead of “on”; the metric counts these as incorrect.Source: §5.2 · pp. 7–8, Table 2 · p. 7

5.3. Ablations on sampling strategies

• Two-stage training outperforms single-stage training, and separate entity and predicate class-balanced sampling outperforms using predicate-balanced sampling for both classifiers.ACBS further improves SGCls by distilling knowledge between the P-step and E-step.Source: §5.3 · p. 8

5.4. Qualitative results

• Qualitative examples show predictions spanning frequent and tail entity and predicate classes, including finer-grained predicates that may differ from the ground truth.The paper gives “walking on,” a subclass of “on,” as an example of a reasonable but non-matching predicate prediction.Source: §5.4 · p. 8, Figure 6 · p. 8

6. Conclusions

• On VG150, DT2-ACBS significantly outperforms state-of-the-art methods with more complex architectures.The conclusion attributes the approach to a simple model and alternating sampling strategy for long-tailed visual relations.Source: §6 · p. 8

A. Model complexity

• DT2 has 10× fewer trainable parameters than most recent approaches; SGCls model sizes are 224 MB for DT2, versus 1.68 GB for MOTIFS, 1.65 GB for VCTree, and 2.1 GB for TDE-MOTIFS.The authors state that the lower complexity is by design, to emphasize accounting for long tails during training.Source: A. Model complexity · p. 11

B. Additional results

• The appendix section presents additional results for DT2-ACBS.Source: B. Additional results · p. 11

• DT2-ACBS predicts predicates from head classes such as has and wearing as well as the less-populated class walking on.Source: §B.1 · p. 11
• Some metric-counted errors remain reasonable predictions, including belonging to/of and wears/wearing.Source: §B.1 · p. 11
• In SGCls examples, it correctly predicts head entities boy, horse, and head, plus tail entities racket and sock.Source: §B.1 · p. 11

B.2. Ablations on appearance branch

• Removing the appearance branch leaves entity accuracy similar but sharply reduces PredCls and SGCls performance, including when ground-truth entity labels are used.Source: §B.2 · p. 11, Table 4 · p. 11

B.3. E-step as teacher

• Using the E-step as teacher weakens results because its update to entity weights can conflict with the predicate-weight update; ACBS distillation avoids that incompatibility.Source: §B.3 · p. 11, Table 5 · p. 11

B.4. Recall values

• Recall@K is biased toward dominated classes such as on and is not suitable for long-tailed visual relations; Table 6 provides Recall and mRecall values for reference.Source: §B.4 · p. 11

C. Implementation Details

• DT2-ACBS first trains θ, φ, and ψ with SRS, then learns classifiers with ACBS.Source: C. Implementation Details · p. 11
• Both stages use Adam with initial learning rate 10^-3, decayed by 0.5 every 5 epochs; batch sizes are 256 in stage one and 2 objects and 5 predicates per class in stage two.Source: C. Implementation Details · p. 11
• Validation sets α, β, and τ_s to 0.2, 1, and 10, respectively, and evaluation filters subject-object pairs without relationships using the protocol of.Source: C. Implementation Details · p. 11
• Figure 7 marks correct and incorrect predictions by color in PredCls and SGCls examples, with ground truth shown in brackets.Source: Figure 7 · p. 12
• Table 6 reports Recall and mRecall values for scene graph generation tasks.Source: Table 6 · p. 12