Source-linked AI summary
Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
Ruichi Yu, Ang Li, Vlad I. Morariu, Larry S. Davis
TL;DR
Visual relationship detection must learn a large space of subject–predicate–object combinations from limited, long-tail data. The paper mines predicate statistics from annotations and public text, then distills them through a teacher–student framework into an end-to-end model. On VRD and Visual Genome, linguistic knowledge distillation improves accuracy and generalization, including zero-shot relationships.
Problem
Visual relationship prediction has a huge semantic space and limited labeled data, especially for long-tail combinations and subject–object pairs unseen in training.
Method
The model combines visual and semantic representations with internal annotation statistics and external public-text statistics, distilling the resulting linguistic knowledge into an end-to-end student network.
Results
Experiments on VRD and Visual Genome show significant improvements in accuracy and generalization, with the strongest gains reported for unseen relationships.
Takeaways & Limitations
The results support using linguistic knowledge alongside visual representations to improve prediction and generalization in visual relationship detection.
Takeaways & Limitations
External linguistic knowledge can be very noisy because scene-graph parsing may make mistakes, especially on broad-domain text.
Abstract
from arXiv · showhide
Understanding visual relationships involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the (subj,obj) pair (both semantically and spatially) to predict the predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships, but complicates learning since the semantic space of visual relationships is huge and the training data is limited, especially for the long-tail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a (subj,obj) pair. Then, we distill the knowledge into a deep model to achieve better generalization. Our experimental results on the Visual Relationship Detection (VRD) and Visual Genome datasets suggest that with this linguistic knowledge distillation, our model outperforms the state-of-the-art methods significantly, especially when predicting unseen relationships (e.g., recall improved from 8.45% to 19.17% on VRD zero-shot testing set).
1. Introduction
Visual relationship detection jointly models subjects, predicates, and objects, but its large relationship space and long-tail data make learning difficult. The paper addresses this by distilling internal and external linguistic knowledge into an end-to-end visual model.
- Visual relationships are represented as subject–predicate–object tuples describing observable interactions between object pairs.
- 7.11% Recall@100 was reported for visual prediction using only visual cues, motivating joint predicate modeling with subject and object information.The cited baseline predicts predicates without object-category or relative-location knowledge.
- The large variety and long-tail distribution of relationship tuples leaves insufficient labeled images for many relationships.Collecting and annotating enough examples for all relationships is described as costly and difficult.
- Internal linguistic knowledge estimates P(pred|subj, obj) from training annotations, while external knowledge from Wikipedia covers unseen pairs but can be noisy.External statistics capture words commonly used to describe subject–object relationships, including pairs absent from training data.
- The teacher–student framework trains the student to match both one-hot ground-truth labels and the teacher’s soft predicate beliefs.The teacher combines the visual pipeline with a model scoring solutions using linguistic knowledge.
- Visual Genome experiments found minimal gains from training-set size but large gains from linguistic knowledge, while evaluations covered VRD and Visual Genome.The authors argue that readily available knowledge sources are valuable because visual data is expensive to annotate.
2. Related Work
Prior work decomposed visual relationship prediction and used object interactions or linguistic cues, while knowledge distillation transferred auxiliary information into neural networks. This paper jointly models objects and relationships and explicitly distills linguistic correlations, targeting long-tail and zero-shot cases.
- Knowledge distillation research transferred predicted distributions or massive knowledge sources, including logic rules, into deep neural networks.
- Visual relationship research used object co-occurrence, spatial relationships, and decomposed object and predicate prediction for large-scale detection.
- Phrase localization work fused appearance, size, bounding-box, and linguistic features, achieving results comparable with an earlier VRD method.
- The proposed model jointly represents objects and relationships while using linguistic knowledge to model predicate–subject/object correlations, especially for long-tail relationships.
- The teacher performs better with sufficient training data, whereas the student generalizes to cases with few or no training examples.The framework combines both networks because they have complementary advantages.
3. Our Approach
The approach combines visual, semantic, spatial, and linguistic representations in an end-to-end teacher-student framework to regularize predicate prediction. Internal annotations and external Wikipedia text provide predicate distributions conditioned on subject-object pairs, helping address long-tail and noisy knowledge challenges.
- Semantic and Spatial Representations: The model concatenates union-region visual features, semantic object representations, and spatial features to predict predicates end-to-end.Semantic representations use word embeddings, while spatial features describe the detected objects’ locations and sizes.
- Knowledge Distillation: The teacher-student objective trains the student to fit ground-truth labels while mimicking the teacher’s linguistic-knowledge-regularized predicate distribution.The loss balances the ground-truth and teacher predictions through α.
- Linguistic Knowledge: Linguistic knowledge encodes P(pred|subj, obj), capturing predicates humans commonly use for fixed subject-object pairs.The conditional probability is used to penalize unlikely predicates and reward likely ones in the teacher network.
- Knowledge Distillation: Unlike post-processing approaches, distillation lets the data-driven student absorb linguistic knowledge during training and improves generalization, especially for zero-shot relationships.The teacher is constructed from the student, so its knowledge-regularized predictions also improve as the student improves.
- Linguistic Knowledge Collection: Internal annotation statistics are insufficient for many subject-predicate-object combinations because the relationship space is large and long-tailed.The method therefore supplements training statistics with external knowledge mined from Wikipedia, including subject-object pairs unseen in training.
- Linguistic Knowledge Collection: Wikipedia-derived linguistic knowledge can be noisy because scene-graph parsing errors arise when processing text from a broad domain.The student’s labeled-data supervision helps correct errors from noisy external probabilities, while weighted internal and external statistics support seen and unseen relationships.
4. Experiments
Experiments on VRD and Visual Genome evaluate predicate, phrase, and relationship detection, including zero-shot generalization. Across settings, linguistic-knowledge distillation improves performance, with external knowledge especially benefiting unseen relationships.
- Evaluation setup: The evaluation covers predicate detection, phrase detection, and relationship detection on VRD and Visual Genome, including zero-shot test sets.Zero-shot sets contain relationships absent from training data.
- Predicate prediction: End-to-end soft-max training raises VRD predicate recall from 7.11% to 34.82% for R@100/50, k=1 over the compared visual baseline.The comparison uses the same BB-Union visual representation as [19].
- Predicate prediction: 51.34% vs. 47.87% for R@100/50, k=1 shows that the linguistic-only baseline outperforms [19] on the entire VRD test set.On zero-shot data, however, linguistic-only prediction performs poorly when subject-object pairs are absent from extracted training statistics.
- Predicate prediction: 37.15% to 42.98% for R@100/50, k=1 on the entire VRD test set shows the gain from distillation with U+W features.Improvements occur across feature settings, and student networks absorbing linguistic knowledge generalize better than data-driven and knowledge-only baselines.
- Phrase and relationship detection: Combining teacher and student predictions performs best, while the method significantly outperforms state-of-the-art methods on phrase and relationship detection, including zero-shot evaluation.For phrase detection, an off-the-shelf detector may explain weaker R@100 performance than; relationship detection outperforms both compared methods.
- External knowledge and generalization: 19.17% recall on the VRD zero-shot set results when Wikipedia knowledge is distilled alongside internal knowledge, while maintaining good entire-test-set performance.On Visual Genome augmentation experiments, adding data yields marginal improvements, whereas linguistic distillation remains beneficial and teacher-student combination performs best.
5. Conclusion
The paper presents a teacher-student framework that distills internal and external linguistic knowledge into a deep neural network for visual relationship detection, improving accuracy and generalization on VRD and Visual Genome.
- The proposed framework distills linguistic knowledge into a deep neural network for visual relationship detection.
- The model incorporates rich visual relationship representations and absorbs internal knowledge from training annotations and external knowledge from public Internet text.
- Experiments on the VRD and Visual Genome datasets show significant improvements in accuracy and generalization capability.