Source-linked AI summary
Detecting Visual Relationships with Deep Relational Networks
Bo Dai, Yuqi Zhang, Dahua Lin
TL;DR
Visual relationship detection remains challenging because visual-phrase classes are numerous and imbalanced, while predicate classes contain highly diverse appearances. The paper introduces an integrated framework centered on DR-Net, which jointly models appearance, spatial configuration, and statistical dependencies. On two large datasets, it substantially outperforms prior methods, although object-detector recall limits box-based tasks.
Problem
Visual relationship detection is difficult because visual-phrase classification produces many imbalanced classes, while predicate-only classification has high intra-class diversity.
Method
The framework represents relationships as subject-predicate-object triplets and jointly infers their labels using appearance, spatial configuration, and statistical dependency through DR-Net.
Results
The proposed framework outperforms classification-based methods and CRFs on two large datasets, with gains across relationship-detection tasks.
Takeaways & Limitations
Statistical relations are significant for visual relationship detection, and DR-Net models complex relations within a deep neural network.
Takeaways & Limitations
CRF-based relational inference can require expensive or intractable normalizing-constant computation, especially in graphs with cycles.
Abstract
from arXiv · showhide
Relationships among objects play a crucial role in image understanding. Despite the great success of deep learning techniques in recognizing individual objects, reasoning about the relationships among objects remains a challenging task. Previous methods often treat this as a classification problem, considering each type of relationship (e.g. "ride") or each distinct visual phrase (e.g. "person-ride-horse") as a category. Such approaches are faced with significant difficulties caused by the high diversity of visual appearance for each kind of relationships or the large number of distinct visual phrases. We propose an integrated framework to tackle this problem. At the heart of this framework is the Deep Relational Network, a novel formulation designed specifically for exploiting the statistical dependencies between objects and their relationships. On two large datasets, the proposed method achieves substantial improvement over state-of-the-art.
1. Introduction
Visual relationship detection is difficult because relationships carry essential image-understanding information but do not fit cleanly into conventional classification schemes. The paper proposes DR-Net and an integrated framework that jointly exploits spatial configuration and statistical dependency.
- Motivation: Relationships among objects provide information beyond recognizing individual objects and can support richer image understanding.The paper cites image captioning as an application that benefits from relationships in addition to objects.
- Limitations of prior formulations: Treating each visual phrase as a class creates over 75K classes in Visual Genome, with samples ranging from a handful to over 10K.This produces a large, highly imbalanced class space.
- Limitations of prior formulations: Classifying predicates alone reduces the number of classes but creates substantial intra-class visual diversity, including unrelated examples such as “mountain-near-river” and “person-near-dog”.The authors report that even deep networks struggle with this level of diversity.
- Proposed approach: The proposed framework represents predictions as (subject, predicate, object) triplets and jointly infers their labels using spatial configuration and statistical dependency.The paper states that these relations are often more reliable than visual appearance.
- Contributions: DR-Net embeds statistical inference into a deep neural network through iteration unrolling, enabling discriminative parameter learning for relational modeling.The framework reportedly outperforms classification-based methods and CRFs on two large datasets.
- Contributions: The work contributes both DR-Net and an integrated visual relationship detection framework that brings state-of-the-art performance to a new level.
2. Related Work
Prior work used visual relationships mainly as auxiliary cues or treated relationship combinations as classification categories. This paper instead targets generic relationship detection with a unified, end-to-end relational framework.
- Earlier work: Earlier studies commonly focused on positional relations or actions, extracting relationships with heuristics or handcrafted features for other vision tasks.These tasks included object recognition, image classification, retrieval, scene understanding, generation, and text grounding.
- Earlier work: The paper differs by providing a method dedicated to generic visual relationship detection across positions, actions, functionals, and comparisons.
- Relationship-detection paradigms: Visual-phrase methods face difficulties because the number of object-category and predicate combinations can become very large.Recent alternatives recognize relationship predicates and object categories separately.
- Closest related method: Compared with Lu et al., the proposed method models spatial configurations and statistical dependencies through DR-Net rather than simply fusing them as features.The full framework integrates representation learning and relational modeling in one end-to-end network.
- Closest related method: Recall@50 for relationship predicate recognition rises from 47.9% to 80.8% and from 53.5% to 88.3% on two large datasets.
3. Visual Relationship Detection
The framework detects object pairs and jointly predicts triplets using appearance, spatial configuration, and statistical relations. DR-Net provides the central relational inference mechanism within this pipeline.
- Problem formulation: Each visual relationship is represented as a triplet (s, r, o), and the task is to locate relationships and infer their component categories.Here s, r, and o denote subject, predicate, and object categories.
- Overall Pipeline: The pipeline detects candidate objects, filters meaningless pairs, and performs joint recognition on retained pairs.Pair filtering uses spatial configurations and object categories to reduce later computational cost.
- Joint Recognition: Appearance features are extracted for individual objects and enclosing boxes to capture object categories and relationship-related visual cues.
- Joint Recognition: Dual spatial masks encode subject and object geometry as 32 × 32 binary masks, compressed into a 64-dimensional vector.The reported top-1 recalls for mask sizes 8, 16, 32, 64, and 128 are 0.47, 0.48, 0.50, 0.51, and 0.51.
- Joint Recognition: DR-Net incorporates statistical relational modeling into a deep neural network to exploit dependencies among triplet components.The authors report that these relations resolve ambiguities from visual or spatial cues and improve prediction accuracy.
- Integrated Prediction: The integrated prediction module concatenates appearance and spatial features, compresses them, and feeds them with individual-object features into DR-Net for joint inference.
4. Deep Relational Network
The Deep Relational Network embeds statistical inference for visual relationships into a feed-forward neural network, jointly refining subject, predicate, and object predictions. This formulation models dependencies among these variables directly rather than treating them as merely fused features.
- Deep Relational Network: The model jointly predicts the subject, predicate, and object by exploiting spatial configuration and statistical dependencies among them.These relations are emphasized as often more reliable than visual appearance when visual cues are ambiguous.
- From CRF to DR-Net: Unlike CRFs, DR-Net avoids explicit statistical inference by embedding the inference procedure into a deep neural network through iteration unrolling.CRF inference can require expensive or intractable computation of the normalizing constant, while approximate inference may create a gap between training and inference objectives.
- Deep Relational Network: DR-Net unrolls statistical inference into a sequence of neural-network computing layers called inference units.The iterative updates take current probability vectors for the subject, predicate, and object as inputs and output updated versions.
- Discussion: DR-Net treats object categories and relationship predicates as distinct but statistically related domains rather than as features combined by a fusion layer.Its parameters can be learned discriminatively within an integrated network spanning representation learning and relational modeling.
- From CRF to DR-Net: Each inference update uses current posterior vectors for the relational variables, allowing the network to refine estimates over multiple steps.The formulation can be viewed as a special recurrent neural network that repeatedly receives the observed subject, pair, and object features.
5. Experiments
Experiments evaluate the proposed framework on VRD and sVG across three detection settings, compare it with prior methods, and analyze architectural components and inference-unit choices. The method substantially outperforms VR while remaining limited by object-detector recall on localization tasks.
- Experiment Settings: Experiments use VRD and sVG, reporting Recall@50 and Recall@100 across predicate recognition, union box detection, and two boxes detection.Recall is used because relationship annotations are incomplete; two boxes detection requires both subject and object IoUs to exceed 0.5.
- Comparative Results: The proposed method outperforms VP, Joint-CNN, and VR across all three tasks on both datasets.VP struggles with the large imbalanced class space, Joint-CNN struggles to learn common representations, and VR performs better but remains unsatisfactory.
- Comparative Results: About 30% object recall at Recall@50 limits union box and two boxes detection despite the proposed method’s gains.The authors attribute this weakness primarily to object-detector limitations and suggest a more sophisticated detector is needed.
- Ablation Results: Appearance and spatial information are complementary, while DR-Net exploits statistical dependencies more effectively than CRF.Prediction perplexity is 2.64 with DR-Net versus 3.08 without it.
- Architectural Choices: Recall increases with more inference units, reaching 73% from 56%; without weight sharing it saturates at 8 units, versus 12 with sharing.Without weight sharing, the network generally performs relatively better with the same number of inference units.
6. Conclusion
The framework combines appearance, spatial configuration, and statistical relations through DR-Net for visual relationship detection. It outperforms the state of the art on Visual Genome and also produces promising scene graph generation results.
- The framework integrates appearance, spatial configurations, and statistical relations between objects and relationship predicates.Its central component, DR-Net, extends deep neural networks to model complex relations.
- On Visual Genome, the method outperforms the state of the art by a remarkable margin.
- The method also yields promising results for scene graph generation, representing a higher level of image understanding.