Source-linked AI summary
Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
Xiaodan Liang, Lisa Lee, Eric P. Xing
TL;DR
Visual relationship and attribute detection remains difficult because methods often miss global object interdependencies and cover only limited relationship types. VRL addresses this with a semantic action graph and adaptive, history-aware sequential reinforcement learning, achieving stronger detection results and generalization to unseen types on VRD and Visual Genome.
Problem
Existing methods often ignore global context among object instances and recognize only a limited set of relationship types, despite the large-scale scope of visual relationship and attribute detection.
Method
VRL sequentially discovers relationships and attributes using a language-prior semantic action graph, variation-structured action sets, ambiguity-aware object mining, and phrase-history embeddings in the state.
Results
VRL outperforms state-of-the-art methods for relationship and attribute detection on VRD and Visual Genome and generalizes to unseen types.
Takeaways & Limitations
Shared graph correlations and sequential global-context reasoning let VRL handle many relationship and attribute types beyond individually trained detectors.
Abstract
from arXiv · showhide
Despite progress in visual perception tasks such as image classification and detection, computers still struggle to understand the interdependency of objects in the scene as a whole, e.g., relations between objects or their attributes. Existing methods often ignore global context cues capturing the interactions among different object instances, and can only recognize a handful of types by exhaustively training individual detectors for all possible relationships. To capture such global interdependency, we propose a deep Variation-structured Reinforcement Learning (VRL) framework to sequentially discover object relationships and attributes in the whole image. First, a directed semantic action graph is built using language priors to provide a rich and compact representation of semantic correlations between object categories, predicates, and attributes. Next, we use a variation-structured traversal over the action graph to construct a small, adaptive action set for each step based on the current state and historical actions. In particular, an ambiguity-aware object mining scheme is used to resolve semantic ambiguity among object categories that the object detector fails to distinguish. We then make sequential predictions using a deep RL framework, incorporating global context cues and semantic embeddings of previously extracted phrases in the state vector. Our experiments on the Visual Relationship Detection (VRD) dataset and the large-scale Visual Genome dataset validate the superiority of VRL, which can achieve significantly better detection results on datasets involving thousands of relationship and attribute types. We also demonstrate that VRL is able to predict unseen types embedded in our action graph by learning correlations on shared graph nodes.
1. Introduction
VRL targets holistic scene understanding by sequentially discovering relationships and attributes while modeling global semantic dependencies. It combines a directed semantic action graph, adaptive traversal, ambiguity-aware object mining, and history-aware deep reinforcement learning, outperforming prior methods on VRD and Visual Genome.
- Motivation: Holistic scene understanding requires recognizing object interactions, relationships, and attributes because these details determine a scene’s interpretation.The paper illustrates this with descriptions such as a man standing on a skateboard and a woman wearing a wetsuit and kneeling on a surfboard.
- Challenges: Visual relationship and attribute detection is difficult because Visual Genome contains 13,894 relationship types and requires interpreting localized objects jointly.Global context is also needed to resolve ambiguities such as whether a woman is wearing a wetsuit or a shirt.
- Approach: VRL builds a directed semantic action graph from language priors so shared graph nodes transfer knowledge from frequent relationships and attributes to rare ones.The paper gives “person-riding-bicycle” helping predict the rarer “child-riding-elephant” as an example.
- Approach: Variation-structured traversal dynamically creates small action sets for attributes, predicates, and next-object categories based on the current state and historical actions.Ambiguity-aware object mining selects an appropriate category for objects that detectors cannot distinguish using global scene context.
- Approach: VRL encodes semantic embeddings of previously extracted phrases in its state vector to incorporate global context cues during sequential prediction.The framework sequentially detects subject-predicate-object relationships and subject-attribute descriptions.
- Results: Experiments on VRD and Visual Genome show that VRL outperforms state-of-the-art methods for relationship and attribute detection and generalizes to unseen types.The paper attributes this evaluation to extensive experiments on both datasets.
2. Related Works
Prior work addressed visual relationships, scene graphs, and deep reinforcement learning, but existing relationship detectors generally covered limited predefined types or missed structured semantic correlations. VRL instead formulates relationship and attribute detection as sequential decision-making using deep reinforcement learning.
- Visual relationship and attribute detection: Most existing visual relationship methods detect only a handful of frequent, predefined types by training individual detectors for each relationship.Word embeddings broadened coverage, but one cited model still ignored structured correlations between objects and relationships.
- Visual relationship and attribute detection: Scene-graph methods provide structured descriptions of objects, attributes, and relationships in images.The related-work discussion contrasts this structured representation with approaches focused on individual relationship detectors.
- Deep reinforcement learning: Deep reinforcement learning has been applied to decision-making tasks including games, robotic manipulation, indoor navigation, and object proposal generation.VRL extends this line of work to visual relationship and attribute detection as a sequential decision-making problem.
3. Deep Variation-structured Reinforcement Learning
VRL formulates visual relationship and attribute detection as sequential decision-making over a directed semantic graph. It combines adaptive graph traversal, ambiguity-aware object mining, breadth-first search, and image-plus-history state features.
- Semantic action graph: VRL organizes object categories, attributes, and predicates in a directed semantic graph that provides a compact representation of semantic correlations.The graph connects nouns to attributes and subject–predicate–object phrases, enabling shared graph nodes to support learning across types.
- Variation-structured action space: The variation-structured traversal scheme dynamically constructs small action sets for attributes, predicates, and candidate object categories at each step.These adaptive sets replace learning over the entire action space and depend on the current state and historical actions.
- Sequential traversal: VRL performs breadth-first search by predicting all relationships and attributes for the current subject before moving to the next subject instance.Object mining terminates dynamically through a terminal trigger or when no eligible neighboring instances remain.
- Ambiguity-aware object mining: Ambiguity-aware object mining uses neighboring instances and candidate categories to address cases where the detector cannot distinguish semantically ambiguous object labels.The candidate object-category set includes categories whose confidence is within 0.1 of the most confident category, together with a terminal trigger.
- State representation: The state vector concatenates subject and object features, whole-image features, and semantic embeddings of the last two relationship and attribute phrases.Whole-image features provide global context, while history embeddings encode previously traversed search paths and scene context.
4. Experiments
Experiments evaluate VRL on VRD and Visual Genome for relationship and attribute detection, including zero-shot generalization. Results compare state-of-the-art variants and show benefits from language priors, variation-structured traversal, history embeddings, and ambiguity-aware object mining.
- Datasets and Evaluation: VRL is evaluated on VRD and Visual Genome using Recall@100 and Recall@50 across relationship phrase, relationship, and attribute detection tasks.VRD contains 5,000 images, while Visual Genome supports large-scale evaluation and zero-shot testing on relationship types absent from training.
- Comparison with State-of-the-art Models: Language priors improve detection, with VRL and Lu et al. significantly outperforming baselines that rely only on visual cues.VRL further improves over Lu et al., combining a directed semantic action graph with sequential reinforcement-learning reasoning.
- Discussion: 13.34% versus 6.23% relationship-detection Recall@100 and 26.43% versus 12.47% attribute-detection Recall@100 show VRL outperforming standard RL.Variation-structured traversal reduces predicate actions from 347 to 15 on average, shrinking the action space during sequential search.
- Discussion: Removing history phrase embeddings reduces recall by over 4%, while replacing them with historical action vectors also lowers recall.The results support semantic phrase embeddings as a compact representation of previously traversed search paths and scene context.
- Discussion: Ambiguity-aware object mining improves recall by considering multiple semantically ambiguous object categories instead of only the top-1 category.The method resolves vague predictions such as “man” to “skier” and “stick” to “bat” using scene context from extracted relationships and attributes.
- Comparison with State-of-the-art Models: VRL outperforms state-of-the-art methods for relationship and attribute detection and achieves over 5% improvement over Lu et al. on both datasets in zero-shot learning.The comparison attributes this advantage to learning semantic correlations on shared nodes in a large action graph rather than relying only on word embeddings.
5. Conclusion and Future Work
VRL sequentially detects visual relationships and attributes by traversing a directed semantic action graph with global interdependency. Experiments on VRD and Visual Genome show advantages over baselines, while future work considers larger graphs and unsupervised learning.
- VRL sequentially discovers relationship and attribute instances through variation-structured traversal on a directed semantic action graph.
- The framework incorporates global interdependency to facilitate predictions in local image regions.
- Experiments on VRD and Visual Genome demonstrate VRL's power and efficiency over baselines.
- Future work includes building a larger directed action graph from natural-language sentences and generalizing VRL to unsupervised learning from unlabeled images.