PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Compose Dynamic Tree Structures for Visual Contexts

Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, Wei Liu

arXiv:1812.01880v1cs.CV

TL;DR

High-level visual reasoning needs object-level context, yet fixed chains and dense graphs inadequately capture selective hierarchical and task-dependent relationships. VCTREE learns a task-specific maximum-spanning-tree structure, encodes it with TreeLSTM, and jointly explores structures with reinforcement learning and trains end tasks; it reports consistent gains across scene graph generation and visual question answering while discovering interpretable contexts.

  • Problem

    Object-level visual reasoning requires useful context, but chains are oversimplified and fully connected graphs lack relational discrimination and can saturate message passing.

  • Method

    VCTREE scores object pairs for task-dependent validity, extracts a maximum spanning tree, converts it to a binary tree, encodes context with TreeLSTM, and uses hybrid supervised and REINFORCE learning.

  • Results

    VCTREE shows consistent performance gains in Visual Genome scene graph generation and VQA2.0 visual question answering compared with models with or without visual contexts.

  • Takeaways & Limitations

    VCTREE learns non-trivial, interpretable visual contexts that improve reasoning across scene graph generation and visual question answering.

  • Takeaways & Limitations

    The pretrained object-correlation model is imperfect because it lacks coherent graph-level constraints or a question prior, so it requires further fine-tuning.

Abstract

from arXiv · show

We propose to compose dynamic tree structures that place the objects in an image into a visual context, helping visual reasoning tasks such as scene graph generation and visual Q&A. Our visual context tree model, dubbed VCTree, has two key advantages over existing structured object representations including chains and fully-connected graphs: 1) The efficient and expressive binary tree encodes the inherent parallel/hierarchical relationships among objects, e.g., "clothes" and "pants" are usually co-occur and belong to "person"; 2) the dynamic structure varies from image to image and task to task, allowing more content-/task-specific message passing among objects. To construct a VCTree, we design a score function that calculates the task-dependent validity between each object pair, and the tree is the binary version of the maximum spanning tree from the score matrix. Then, visual contexts are encoded by bidirectional TreeLSTM and decoded by task-specific models. We develop a hybrid learning procedure which integrates end-task supervised learning and the tree structure reinforcement learning, where the former's evaluation result serves as a self-critic for the latter's structure exploration. Experimental results on two benchmarks, which require reasoning over contexts: Visual Genome for scene graph generation and VQA2.0 for visual Q&A, show that VCTree outperforms state-of-the-art results while discovering interpretable visual context structures.

1. Introduction

Visual context helps high-level visual reasoning, but common chain and fully connected graph structures impose oversimplified or indiscriminate relationships. VCTREE addresses this by learning efficient, dynamic tree structures and training them jointly with end tasks.

  • Motivation: Visual context links objects through coherent configurations and supports reasoning when image content is partially observed.Object-level context has been effective in image captioning and visual question answering.
  • Structure: The proposed structure is designed to encode hierarchical and parallel object relationships more efficiently than chains and fully connected graphs.Its binary representation supports efficient TreeLSTM encoding.
  • Limitations of prior structures: Chains capture limited spatial or co-occurrence information, while fully connected graphs do not distinguish hierarchical from parallel relations and may saturate message passing.These limitations motivate a more selective object-level structure.
  • VCTREE: VCTREE learns a task-specific score matrix, extracts a maximum spanning tree, and converts it into a binary tree for efficient TreeLSTM context encoding.The structure prioritizes objects that are more informative for the image content or question.
  • Learning: VCTREE combines supervised learning for context encoding and downstream tasks with REINFORCE-based exploration of tree structures.The supervised task evaluation, using Recall for SGG or Accuracy for VQA, serves as a self-critic.

2. Related Work

Prior work models visual context implicitly through image features or explicitly through object relationships and message passing. Scene graph generation and visual question answering provide important settings for evaluating such contextual reasoning.

  • Visual Context Structures: Implicit context models encode surrounding pixels through multi-scale feature maps, dilated convolutions, or feature pyramids.These approaches support dense prediction and object detection by combining contextual and detailed visual features.
  • Visual Context Structures: Explicit context models incorporate contextual cues at the object level for visual reasoning.The supplied related-work passage introduces this category but does not enumerate its full range of methods.
  • Visual Reasoning Tasks: Figure 2 summarizes VCTREE as a framework that extracts proposal features, constructs a dynamic tree, encodes context, and decodes task-specific outputs.The figure distinguishes supervised training for later stages from REINFORCE with a self-critic for tree construction.
  • Visual Reasoning Tasks: Scene graph generation treats each image as a whole and uses message passing to refine local predictions with global context.This makes SGG a setting for comparing different contextual models.

3. Approach

VCTree learns task-dependent object validity scores, builds a maximum spanning tree, encodes it with bidirectional TreeLSTM, and decodes task-specific predictions for SGG and VQA.

  • VCTREE Construction: VCTree represents each detected object with visual and spatial features, then learns a task-specific score matrix for pairwise contextual validity.Object features concatenate RoIAlign and bounding-box features; the score combines object correlation with task dependency.
  • Hybrid Learning: Hybrid learning combines supervised end-task training with reinforcement learning for tree exploration, using task evaluation as reward and a greedy self-critic baseline to reduce variance.The reward can be Recall@100 for SGG or Accuracy for VQA, and the self-critic uses the reward of the greedily constructed tree.
  • VCTREE Construction: The model constructs a maximum spanning tree from the symmetric score matrix using Prim’s algorithm, connecting each remaining node through the highest-validity choice.During exploration, construction decisions can instead be sampled in proportion to their validity scores.
  • TreeLSTM Context Encoding: A bidirectional TreeLSTM encodes the constructed tree into object-level visual context, using top-down and bottom-up directions with explicit left-right branch distinctions.Missing branches are padded with zero vectors.
  • Scene Graph Generation Model: For scene graph generation, VCTree applies TreeLSTM encoding to object and relation contexts before decoding object labels and pairwise predicates.Pairwise prediction fuses relation context, union-box RoIAlign features, and bounding-box pair features through element-wise products.
  • Visual Question Answering Model: For VQA, separate multimodal attention features capture visual and contextual information, which are concatenated and filtered by a question-guided gate before answer prediction.The gate adapts the relative use of visual and contextual features to the question.

4. Experiments on Scene Graph Generation

On Visual Genome, VCTREE is evaluated for scene graph generation using standard protocols and compared with alternative context structures. It performs best among the tested structures, with hybrid learning further improving results and qualitative evidence of hierarchical and parallel context.

  • 4.1. Settings: Visual Genome contains 108,077 images, and the adopted split retains the top-150 object and top-50 predicate categories.The data are divided 70%/30% into training and test sets, with 5,000 training images used for validation.
  • 4.1. Settings: SGGen, SGCls, and PredCls evaluate progressively different amounts of object localization and category information while predicting scene relationships.The protocols use Recall@K because Visual Genome annotations are incomplete and biased.
  • 4.3. Ablation Studies: VCTREE outperforms Chain, Overlap, Multi-Branch, and supervised VCTREE-SL structures, with hybrid VCTREE-HL providing further improvement.The ablation compares fixed and learned tree-construction policies, including a fully trained hybrid-learning variant.
  • 4.4. Comparisons with State-of-the-Arts: VCTREE achieves the best performances against previous state-of-the-art methods, while Overlap and Multi-Branch trees outperform other non-tree context models.The comparisons include models without visual contexts, implicit context models, and explicit context models with varied structures.
  • 4.4. Comparisons with State-of-the-Arts: Larger VCTREE-HL gains for PredCls under mR@100 indicate learned context beyond class-distribution bias, unlike the weaker MOTIFS result relative to FREQ.Mean recall is measured across predicate categories, and MOTIFS is reported as worse than its FREQ baseline under mR@100.
  • 4.4. Comparisons with State-of-the-Arts: For the “street” category, hierarchical node categories are long-tailed while parallel relations are more uniformly distributed, supporting VCTREE’s intended two-type context representation.Figure 6 summarizes left-branch hierarchical and right-branch parallel nodes.

5. Experiments on Visual Q&A

On VQA2.0, VCTREE is tested across answer types, balanced pairs, and standard test sets. The hybrid model outperforms alternative context structures and learns dynamic, interpretable question-dependent trees.

  • 5.1. Dataset: VQA2.0 provides 443,757 training and 214,354 validation question-image pairs, with balanced answers across Yes/No, Number, and Other types.The evaluation also includes a balanced-pair subset requiring different perfect answers for the same question across images.
  • 5.3. Ablation Studies: VCTREE-HL outperforms all evaluated context models across the three VQA2.0 answer types.The comparison includes five tree-construction policies and a fully connected graph using message passing.
  • 5.3. Ablation Studies: VCTREE-HL’s absolute gains over other structures are larger on the balanced-pair subset than on the original validation set.This pattern is reported as evidence that the learned context structures alleviate question-answer bias in VQA.
  • 5.4. Comparisons with State-of-the-Arts: VCTREE-HL achieves the best overall single-model performance on both VQA2.0 test-dev and test-standard.The reported methods use the same Faster-RCNN features for fair comparison.
  • 5.4. Comparisons with State-of-the-Arts: Question-conditioned trees place different objects at the root, such as “tree” for snow questions and “man” for sport questions.These examples illustrate dynamic structures with interpretability.

6. Conclusions

VCTREE captures task-specific visual contexts with dynamic tree structures for SGG and VQA, using bidirectional TreeLSTM encoding and hybrid structure learning. The paper reports consistent gains across both tasks and identifies dynamic forests as future work.

  • 6. Conclusions: VCTREE captures task-specific visual contexts for scene graph generation and visual Q&A through a dynamic tree structure.The model is designed to support both high-level vision tasks.
  • 6. Conclusions: The paper reports consistent performance gains in SGG on Visual Genome and VQA on VQA2.0 compared with models with or without visual contexts.The authors also tested VCTREE against category and question-answer biases.
  • 6. Conclusions: The authors identify a dynamic forest as a future direction for visual context structure.
  • 6. Conclusions: Binary TreeLSTM encodes VCTREEs and Overlap Trees bottom-up, while top-down TreeLSTM propagates information from each node’s parent.The implementation uses N-ary TreeLSTM for binary trees and traditional LSTM-style processing in the top-down direction.
  • 6. Conclusions: Child-Mean TreeLSTM handles Multi-Branch Trees by normalizing the hidden states of their children.The implementation replaces Child-Sum with Child-Mean for better normalization.
  • 6. Conclusions: Traditional top-down LSTM is used for efficiency because separate matrices for left- and right-branch parents did not significantly improve end-task results.Each node has at most one parent in these tree structures.

B.1. Mean Recall for Scene Graph

Mean recall evaluates performance across all 50 predicate categories. VCTREE-HL achieves the best performance among the ablation structures, while MOTIFS shows mixed behavior relative to FREQ across recall cutoffs.

  • B.1. Mean Recall for Scene Graph: VCTREE-HL shows the best mean-recall performance among all ablative structures.Mean recall is reported across all 50 predicate categories.
  • B.1. Mean Recall for Scene Graph: MOTIFS has lower mR@100 than FREQ in SGCls and PredCls, indicating weaker prediction of infrequent predicate categories at that cutoff.
  • B.1. Mean Recall for Scene Graph: MOTIFS has higher mR@20 and mR@50 than FREQ in SGCls and PredCls, indicating better separation of foreground from background relationships.

B.2. Predicate Recall Analysis

Predicate-level analysis compares PredCls Recall@100 for MOTIFS and VCTREE-HL across the top 35 categories ranked by frequency. VCTREE-HL improves especially on less frequent, semantically meaningful predicates.

  • B.2. Predicate Recall Analysis: VCTREE-HL significantly improves PredCls Recall@100 on less frequent but more semantically meaningful predicates.The comparison ranks the top-35 predicate categories by frequency and evaluates each independently.
  • B.2. Predicate Recall Analysis: Figure 8 compares MOTIFS and VCTREE-HL for each top-35 predicate category independently.

C. Qualitative Analysis

Qualitative analysis examines VCTREE-HL’s misclassified scene graphs and identifies cases where its predicted predicates are judged more appropriate than the annotated ground truth. It also highlights errors involving isolated objects.

  • C. Qualitative Analysis: VCTREE-HL predicts “in front of” and “behind” where the original annotation uses “near” in one example.
  • C. Qualitative Analysis: In two examples, VCTREE-HL produces more appropriate predicates than the ground truth labels “man in snow” and “window near building.”The qualitative analysis characterizes those annotations as improper.
  • C. Qualitative Analysis: Objects isolated from the scene graph when considering only R@20 predicates are easier to be misclassified.

C.2. Visual Question Answering

VCTree visualizations show dynamic, interpretable tree structures for VQA2.0 that adapt object contextual cues to different questions.

  • Question-specific VCTree structures let image objects incorporate different contextual cues for different VQA questions.
  • For a collar question, the model focuses first on the collar-like object rather than the dog.
  • For a sports question, the model focuses on the sportsman rather than the playground.
Loading 1812.01880v1…