Source-linked AI summary
Large-Scale Visual Relationship Understanding
Ji Zhang, Yannis Kalantidis, Marcus Rohrbach, Manohar Paluri, Ahmed Elgammal, Mohamed Elhoseiny
TL;DR
Visual relationship detection must scale beyond small, balanced vocabularies because object and relation triples are numerous, semantically connected, and highly imbalanced. The paper addresses this with separate visual and semantic embeddings trained to preserve both discrimination and semantic affinity, achieving strong results across large- and small-scale benchmarks. Its main scope is classification and detection under the evaluated datasets and embedding-based training setup.
Problem
Visual relationship detection must handle more than 80,000 visual entities whose subject–relation–object triples are highly imbalanced and semantically interconnected.
Method
The model maps visual features and semantic word representations into separate object and relation embedding spaces, matching corresponding pairs while preserving semantic similarity.
Results
The approach outperforms strong baselines on Visual Genome’s 53,000+ object and 29,000+ relation benchmark and achieves state-of-the-art performance on VRD and a Visual Genome scene graph dataset.
Takeaways & Limitations
Multi-level subject–object feature integration and semantic correlation preservation are crucial for relationship embeddings that remain discriminative at large scale.
Takeaways & Limitations
Fixed-margin triplet loss can stop learning once negatives exceed the margin, producing embeddings that are insufficiently discriminative for classification.
Abstract
from arXiv · showhide
Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of <subject, relation, object> triples. In real-world scenarios with large numbers of objects and relations, some are seen very commonly while others are barely seen. We develop a new relationship detection model that embeds objects and relations into two vector spaces where both discriminative capability and semantic affinity are preserved. We learn both a visual and a semantic module that map features from the two modalities into a shared space, where matched pairs of features have to discriminate against those unmatched, but also maintain close distances to semantically similar ones. Benefiting from that, our model can achieve superior performance even when the visual entity categories scale up to more than 80,000, with extremely skewed class distribution. We demonstrate the efficacy of our model on a large and imbalanced benchmark based of Visual Genome that comprises 53,000+ objects and 29,000+ relations, a scale at which no previous work has ever been evaluated at. We show superiority of our model over carefully designed baselines on the original Visual Genome dataset with 80,000+ categories. We also show state-of-the-art performance on the VRD dataset and the scene graph dataset which is a subset of Visual Genome with 200 categories.
Introduction
Visual relationship recognition must handle open-ended, highly imbalanced object and relationship categories while exploiting semantic connections between related visual entities. The proposed model targets more than 80,000 visual entities and is evaluated on a Visual Genome benchmark with over 53,000 objects and 29,000 relations.
- Scale and imbalance: Open-vocabulary visual understanding involves many more relationship triples than object categories, with long-tailed frequencies that make rare entities and triples difficult to learn.A triple may be infrequent because its subject, object, relation, or their combination is rare.
- Semantic structure: Conditional semantics matter because relationships sharing subjects and objects can differ substantially, while different objects can participate in similar relationship types.For example, riding a horse resembles riding an elephant more than walking with a horse.
- Proposed scale: More than 80,000 visual entities are studied using continuous output spaces for objects and relations instead of discrete labels.The approach is evaluated on a Visual Genome benchmark containing 53,000+ objects and 29,000+ relations.
- Evaluation: The model outperforms competitive baselines on the large Visual Genome benchmark and achieves state-of-the-art performance on VRD and a Visual Genome scene graph dataset.The reported evaluations cover both the large imbalanced setting and smaller established benchmarks.
Related Work
Prior visual relationship methods largely use small vocabularies or treat complete relationship triplets as indivisible classes. This work instead targets large-scale open-vocabulary recognition by separately embedding relationship components while preserving their semantic connections and discriminative power.
- Visual Relationship Detection: Most earlier visual relationship approaches operate on small vocabularies, such as 100 object and 70 relation categories in VRD or frequent Visual Genome subsets.These settings limit evaluation to restricted category inventories.
- Visual Relationship Detection: Some prior methods use semantic knowledge or distillation, while ViP-CNN formulates recognition over limited classes and cannot scale to open-vocabulary scenarios.The paper positions its approach as directly targeting large category scales and infrequent classes.
- Open-vocabulary recognition: Plummer et al. learn relationship triplets as whole classes, whereas this work embeds subjects, relations, and objects separately and connects them through visual fusion and semantic preservation.Separate embedding spaces avoid treating every complete triplet as an indivisible class.
- Semantically Guided Visual Recognition: Unlike similarity-oriented semantic embedding work, this paper seeks semantic connections without sacrificing discriminative capability for detection.Visual relationships also retain the structured subject–relation–object form.
- Terminology: The paper uses “relation” for a predicate and “relationship” for a subject–relation–object tuple.These are the paper’s terminology conventions for the remainder of the work.
Method
The model maps visual and semantic representations of subjects, relations, and objects into shared embedding spaces, then predicts labels through nearest-neighbor matching. Its architecture separates object and relation spaces, uses relationship-aware semantic representations, and combines discriminative losses with visual consistency constraints.
- Visual Module: The visual module produces separate embeddings for subjects, relations, and objects, while relation features are conditioned on subject and object features.The model outputs visual embeddings x_s, x_p, and x_o; relation recognition depends on subject and object information, whereas object recognition is independent of relations.
- Prediction: At test time, the model embeds all object and relation labels and uses nearest-neighbor search to select relationship predictions.During training, semantic embeddings include ground-truth and negative labels; during testing, the full label sets are searched for the top-k predictions.
- Module Architecture: Subject and object branches share weights, while the relation branch remains independent in both the visual and semantic modules.The figure caption specifies shared subject/object weights, and the semantic module applies the same sharing pattern while preserving a separate relation branch.
- Semantic Module: Relationship-level co-occurrence embeddings model P(P|S, O), P(S|P, O), and P(O|S, P) from training-set relationship contexts.These skip-gram representations provide semantic initialization based on relationship co-occurrences and are designed to capture associations among triplet components.
- Training Loss: Visual consistency separates same-class and different-class visual samples while preserving slackness to avoid collapsing embeddings to class centers.The loss requires the minimum same-class similarity to exceed any different-class similarity by margin m; the final design uses triplet-softmax for semantic embeddings and triplet loss for visual embeddings.
Experiments
Experiments evaluate the model on VRD, VG200, and the large, imbalanced VG80k benchmark, including comparisons, ablations, and qualitative analyses. Results show strong relationship recognition across standard and large-scale settings, with different losses trading off general and tail-class performance.
- Datasets and evaluation: The evaluation covers VRD, VG200, and VG80k, including 53,304 object and 29,086 relation categories after annotation cleaning.VRD evaluation uses top-50 and top-100 recall; VG200 uses predicate classification, scene graph classification, and scene graph detection; VG80k uses recognition metrics with ground-truth boxes.
- VRD: On VRD, the model outperforms methods using the same proposals and remains competitive with the overall best method, without message-passing post-processing.The comparison separates methods by proposal source and reports results with both flexible and specified candidate counts.
- VG200: On VG200, larger performance gaps appear in scene graph and predicate classification, indicating stronger relationship recognition when localization is controlled.Predicate classification uses ground-truth subject and object boxes and labels, while scene graph classification uses ground-truth boxes.
- VG80k: The model’s loss performs best across all VG80k instances, while triplet loss performs better than softmax for tail classes with at most 1,024 occurrences.Triplet loss benefits infrequent classes through transferred knowledge, whereas softmax is more discriminative in the general case.
- Ablation study: Both early and late visual feature concatenations improve performance, with their combination performing best; late concatenation alone is stronger than early concatenation alone.The study attributes this pattern to relations being conditioned on subjects and objects.
- Qualitative results: Qualitative results cover interactive, positional, and attributive relationships across frequent and infrequent classes, with some errors remaining semantically related to the ground truth.Examples include predicting “light on pole” for “lamp on pole” and “scooter on sidewalk” for “motorcycle on sidewalk.”
Conclusions
The paper studies visual relationship detection at unprecedented scale and proposes a model designed to generalize across long-tailed class distributions. It validates the approach across multiple datasets and tasks, finding superiority over strong baselines and state-of-the-art methods.
- The model targets visual relationship detection at unprecedented scale and is designed to generalize better across long-tail class distributions.
- Integrating subject and object features at multiple levels is crucial for learning effective relation embeddings.
- The proposed loss embeds visual and semantic features into a shared space while preserving semantic correlations without sacrificing discriminative ability.
- Experiments across multiple datasets and both classification and detection tasks demonstrate superiority over strong baselines and state-of-the-art methods.
- Future work includes integrating relationship proposals to enable end-to-end training.