Source-linked AI summary
Visual Translation Embedding Network for Visual Relation Detection
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, Tat-Seng Chua
TL;DR
Visual relation detection remains difficult because subject-predicate-object triplets have challenging combinatorial complexity and diverse visual appearances. VTransE addresses this with translation embedding, differentiable feature extraction, and simultaneous end-to-end object and relation prediction. It improves over strong visual baselines across two datasets and is competitive with a multimodal language-prior method, while zero-shot learning remains a key limitation.
Problem
Existing vision-language efforts often leave relationships between objects poorly modeled, while the combinatorial complexity of relation triplets makes visual relation localization and prediction difficult.
Method
VTransE maps objects and predicates into a low-dimensional relation space, uses differentiable feature extraction for object-relation knowledge transfer, and detects objects and relations simultaneously.
Results
VTransE shows significant improvement over state-of-the-art visual relation models on Visual Relationship and Visual Genome, outperforming a multimodal language-prior method in detection and retrieval while slightly trailing it in zero-shot learning.
Takeaways & Limitations
Visual translation embedding provides a visual-only approach that competes with multimodal relation detection while avoiding explicit language priors.
Takeaways & Limitations
Zero-shot relation detection remains a key limitation: VTransE suffers an 88% R@100 performance drop, partly because its generic transformation may miss relation-specific visual deformations.
Abstract
from arXiv · showhide
Visual relations, such as "person ride bike" and "bike next to car", offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate $\approx$ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-to-end relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu's multi-modal model with language priors.
1. Introduction
The paper targets visual relation detection as an intermediate representation of object interactions, addressing the difficulty of modeling diverse subject-predicate-object triplets. VTransE combines translation embedding with end-to-end object and relation prediction, achieving strong results on two relation datasets.
- Existing vision-language models often connect visual and language models directly, leaving object relationships poorly modeled and limiting generalization across specialized tasks.
- Visual relations represent subject-predicate-object triplets that connect objects in semantic contexts such as what, where, and how they interact.
- VTransE is an end-to-end convolutional network that detects objects and predicts relations simultaneously, described as the first end-to-end relation detection network.
- VTransE models relations in a low-dimensional space where subject + predicate ≈ object, learning predicate translations despite diverse subject and object appearances.
- VTransE explicitly transfers knowledge between object detection and predicate prediction using classeme, location, and RoI visual features with differentiable bilinear interpolation.
- On Visual Relationship and Visual Genome, VTransE substantially improves over state-of-the-art visual relation models and outperforms a multimodal method in detection and retrieval while falling slightly short in zero-shot learning.
2. Related Work
Related work frames visual relation detection as a scalable alternative to treating every relation triplet as a separate class. VTransE extends translation embedding within an object-detection architecture and adds differentiable feature extraction for object-relation knowledge transfer.
- Visual relation detection grounds object interactions in images, supporting scene understanding for VQA, captioning, and complex query retrieval.
- Joint relation models suffer from long-tailed triplet distributions, so VTransE follows separate subject, object, and predicate modeling while addressing predicate appearance variance.
- VTransE maps subjects and objects into a low-dimensional relation space and represents predicates as translation vectors between them.
- The architecture builds on an object detection module, then applies a feature extraction layer and translation embedding for relation prediction in a single end-to-end system.
- Unlike non-smooth RoI pooling, the proposed feature extraction layer uses bilinear interpolation to enable knowledge transfer between object detection and relation prediction.
3. Our Approach: VTransE Network
VTransE jointly detects objects and predicts visual relations by embedding object features into a low-dimensional translation space. Its feature extraction and bilinear interpolation layers enable reciprocal object-relation learning through end-to-end optimization.
- Architecture: VTransE builds on an object detection module and adds feature extraction plus translation embedding for simultaneous relation prediction.The architecture combines object detection and relation prediction in one end-to-end network.
- Visual Translation Embedding: Translation embedding represents valid relations by mapping subject and object features into a low-dimensional space with subject + predicate ≈ object.VTransE learns predicate translation vectors and subject/object projection matrices.
- Relation Learning: VTransE uses a softmax prediction loss because visual relation validity depends on specific object instances and incomplete annotations make negative sampling problematic.The loss rewards deterministically accurate predicates rather than uncertain object compositions; with complete annotations, softmax and negative-sampling losses are interchangeable.
- Feature Extraction: The feature extraction layer combines classeme, location, and visual features, with learnable scaling weights controlling their contributions across relations.These features encode object class probabilities, relative box geometry, and convolutional appearance.
- Architecture Details: Bilinear interpolation replaces RoI pooling so relation gradients can propagate through feature coordinates to refine object bounding boxes.Because the grid positions depend linearly on the input box, gradients can be back-propagated to box coordinates.
- Optimization and Inference: Training uses a multi-task loss combining object detection and relation detection, while inference outputs detected objects and relation scores for every object pair.The implementation uses image-centric stochastic-gradient training and samples candidate regions for object detection.
4. Experiments
The experiments evaluate VTransE on predicate prediction, relation detection, phrase detection, retrieval, and zero-shot learning across VRD and Visual Genome. Results show benefits from translation embedding, feature fusion, end-to-end knowledge transfer, and separate relation modeling, while zero-shot performance remains a key limitation.
- Experimental questions: The evaluation asks whether relation embedding, feature choices and knowledge transfer are effective, and how VTransE compares with state-of-the-art visual relation models.The study evaluates these questions using multiple relation prediction and detection tasks.
- Datasets and metrics: VRD contains 5,000 images, 100 object categories, 70 predicates, 37,993 relation annotations, and 6,672 unique relations; VG contains 99,658 images, 200 categories, 100 predicates, and 19,237 unique relations.VRD uses 4,000 training and 1,000 test images; VG uses 73,801 training and 25,857 testing images.
- Datasets and metrics: Recall@50 and Recall@100 measure whether true relations appear among the top K predictions, while retrieval uses Recall@5 and median rank.Precision and average precision are considered inappropriate because relation annotations are incomplete.
- Evaluations of Translation Embedding: VTransE outperforms JointBox in predicate prediction, with translation vectors capturing relation meanings in a low-dimensional relation space rather than mainly modeling joint object co-occurrence.The reported qualitative examples show more diverse predicate generalization, such as extending park on beyond cars to planes and buses.
- Evaluations of Features: Fusing classeme, location, and visual features with a learned scaling layer performs best across relation types; classeme generally beats visual features, while location features are better for spatial relations.All features perform relatively poorly for prepositions, whose spatial and visual cues vary across examples.
- Evaluations of Features: End-to-end VTransE training improves object detection, because relation-prediction errors can back-propagate through the feature layer to the object detector.Qualitative examples show bounding-box refinement, recovery from dislocation, and corrections of semantically similar misdetections.
- Comparison with State-of-The-Arts: Separate models such as VTransE significantly outperform joint models, and on VRD VTransE is relatively better than Lu’s-VLK by 20% in phrase detection, 2% in relation detection, and 230% in retrieval.The comparison attributes joint-model weakness to large classification spaces and insufficient samples for infrequent relations.
- Comparison with State-of-The-Arts: In zero-shot relation detection, VTransE suffers an 88% R@100 performance drop, revealing a key limitation of its generic transformation from feature space to relation space.The paper gives difficulty distinguishing person-lying on-table from person-sit next to-table as an example.
5. Conclusions
VTransE performs simultaneous object detection and relation prediction in an end-to-end, fully convolutional architecture. The paper identifies higher-order relations, zero-shot relation learning, and VQA based on relation reasoning as future directions.
- VTransE combines object detection and relation prediction in an end-to-end, fully convolutional architecture.
- The architecture includes an object detection module, a differentiable feature extraction layer, and a visual translation embedding layer for predicate classification.
- Future work includes modeling higher-order relations such as person-throw-ball-to-dog and tackling zero-shot relation learning.
- The authors also plan to apply VTransE in a VQA system based on relation reasoning.
6. Supplementary Material
The supplementary material lists verb, spatial, prepositional, and comparative relation categories, and presents qualitative retrieval results and a demonstration interface. It reports that incomplete VRD annotation can make correct retrievals appear incorrect.
- VRD relation predicates include verb, spatial, prepositional, and comparative categories.Examples include ride, above, with, and taller than.
- Visual Genome relation predicates are likewise organized into verb, spatial, prepositional, and comparative categories.The listed comparative examples are small than and tall than.
- Figure 10 presents additional qualitative relation-retrieval results from VRD.
- Most of the top 10 retrieval results are correct, but none is labeled as the corresponding query relation.The paper attributes the low retrieval performance in Tables 3 and 4 to incomplete VRD annotation.
- The demonstration interface shows only detections scoring about the top 20% of each predicate type for visualization clarity.