PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Assemble Neural Module Tree Networks for Visual Grounding

Daqing Liu, Hanwang Zhang, Feng Wu, Zheng-Jun Zha

arXiv:1812.03299v3cs.CV

TL;DR

Visual grounding needs to handle compositional language, but existing approaches use monolithic or coarse reasoning. NMTREE assembles neural modules along dependency trees and accumulates grounding evidence bottom-up, with Gumbel-Softmax enabling end-to-end assembly training. It outperforms state-of-the-art methods on most benchmark splits and provides detailed, human-evaluated explanations.

  • Problem

    Existing visual grounding methods simplify compositional language into monolithic sentence scores or coarse subject-predicate-object compositions, limiting detailed reasoning.

  • Method

    NMTREE assembles Single, Sum, and Comp modules along a dependency parsing tree, accumulates grounding scores bottom-up, and trains discrete assembly end-to-end with Gumbel-Softmax.

  • Results

    NMTREE achieves new state-of-the-art performance on most test splits and grounding tasks across RefCOCO, RefCOCO+, and RefCOCOg, with transparent and explainable qualitative results.

  • Takeaways & Limitations

    Fine-grained dependency-tree reasoning can provide explainable visual grounding with superior performance without additional module-layout annotations.

Abstract

from arXiv · show

Visual grounding, a task to ground (i.e., localize) natural language in images, essentially requires composite visual reasoning. However, existing methods over-simplify the composite nature of language into a monolithic sentence embedding or a coarse composition of subject-predicate-object triplet. In this paper, we propose to ground natural language in an intuitive, explainable, and composite fashion as it should be. In particular, we develop a novel modular network called Neural Module Tree network (NMTree) that regularizes the visual grounding along the dependency parsing tree of the sentence, where each node is a neural module that calculates visual attention according to its linguistic feature, and the grounding score is accumulated in a bottom-up direction where as needed. NMTree disentangles the visual grounding from the composite reasoning, allowing the former to only focus on primitive and easy-to-generalize patterns. To reduce the impact of parsing errors, we train the modules and their assembly end-to-end by using the Gumbel-Softmax approximation and its straight-through gradient estimator, accounting for the discrete nature of module assembly. Overall, the proposed NMTree consistently outperforms the state-of-the-arts on several benchmarks. Qualitative results show explainable grounding score calculation in great detail.

1. Introduction

Visual grounding must resolve compositional language, but prior methods use holistic or coarse scores. NMTREE follows dependency trees with modular, bottom-up evidence accumulation and end-to-end assembly, achieving strong benchmark performance and explainability.

  • Motivation: Complex expressions require sequentially identifying a referent, applying contextual constraints, and accumulating evidence to localize the target.The example distinguishes two umbrellas and two girls before using “in pink boots” to identify the correct umbrella.
  • Motivation: Existing methods generally score an entire sentence monolithically or compose only subject, predicate, and object phrases.Word-level attention does not eliminate the coarseness of this reasoning.
  • NMTREE: NMTREE transforms a dependency parsing tree into modules whose grounding scores accumulate bottom-up, providing detailed visual explanations.Single handles leaves and the root, while Sum and Comp handle internal nodes and compose previously grounded evidence.
  • NMTREE: NMTREE uses Gumbel-Softmax training to address discrete module assembly and reduce the impact of dependency-parsing errors.The assembly is limited to runtime choices between Sum and Comp, with Sum requiring no visual grounding.
  • Results: NMTREE achieves new state-of-the-art performance on most test splits and grounding tasks across RefCOCO, RefCOCO+, and RefCOCOg, while qualitative results indicate transparency and explainability.The evaluation covers three challenging visual grounding benchmarks.

2. Related Work

Visual grounding localizes an image region from a natural-language expression, with linguistic structure helping distinguish targets from similar objects. Prior module networks simplify that structure, while finer-grained alternatives generally require additional layout annotations.

  • Visual Grounding: Visual grounding localizes a region given a natural-language expression and uses linguistic information to distinguish the target from same-category objects.The task differs from object detection because language identifies the intended instance among visually similar objects.
  • Existing Methods: Earlier grounding methods use CNN-LSTM or joint-embedding models, but the paper places its approach in the joint-embedding category while addressing neglected linguistic structure.These methods localize the region with the highest probability conditioned on the expression.
  • Module Networks: Prior module networks oversimplify language structure, whereas fine-grained module networks in visual question answering rely on additional annotations for sentence-to-module layout parsing.NMTREE instead trains its module layout from scratch with Gumbel-Softmax.

3. NMTREE Model

NMTREE converts a dependency parsing tree into an assembled neural module tree and performs bottom-up visual grounding, accumulating node-level scores at the root. Its modules separate primitive grounding from composite reasoning, while Gumbel-Softmax enables end-to-end training despite discrete assembly.

  • 3.1. Problem Formulation: Visual grounding retrieves the image region with the highest score for a language expression, requiring a scoring function that comprehends language composition.The image is represented by RoI features, and the target region maximizes S(x_i, L).
  • 3.1. Problem Formulation: Existing methods use either holistic sentence scores or subject-relationship-object triplets, which oversimplify longer compositional expressions.The paper contrasts monolithic sentence representations with coarse triplet decomposition.
  • 3.2. Sentence to NMTREE: NMTREE transforms a sentence into a dependency parsing tree, encodes nodes with a bidirectional Tree LSTM, and assembles neural modules from node representations.Unnecessary syntax nodes and edges are removed; each node retains linguistic information such as its word, POS tag, and dependency relation.
  • 3.3. NMTREE Modules: Single modules handle leaves and the root, while Sum and Comp modules handle internal nodes and support bottom-up grounding along the tree.Sum passes summed child scores upward, whereas Comp performs composite visual reasoning using contextual region features.
  • 3.2. Sentence to NMTREE: Sum modules tend to represent visible concepts, while Comp modules tend to represent relational concepts such as verbs and prepositions.This module assignment pattern provides an interpretable view of how linguistic components participate in grounding.
  • 3.3. NMTREE Modules: NMTREE exposes node-level grounding scores as attention maps, enabling detailed inspection of its intermediate reasoning process.The paper illustrates this transparency on a very long expression containing 22 tokens.
  • 3.4. NMTREE Training: Gumbel-Softmax approximates discrete module assembly during training, using stochastic forward selection and a temperature-controlled continuous relaxation for backpropagation.The temperature approaches argmax as it tends to zero, and the same noise sample is reused between forward and backward passes.

4. Experiments

Experiments evaluate NMTREE through dataset benchmarks, ablations, comparisons with prior grounding models, and qualitative and human assessments of explainability.

  • Experimental Settings: Experiments use RefCOCO, RefCOCO+, and RefCOCOg, three datasets collected from MS-COCO images.RefCOCO includes 142,210 referring expressions for 19,994 images; its testA and testB splits emphasize multiple people and multiple objects, respectively.
  • Experimental Settings: Evaluation reports Top-1 accuracy for detection, counting a prediction correct when its bounding-box IoU is at least 0.5.Top-1 accuracy is the fraction of correctly grounded expressions.
  • Ablation Studies: Ablations compare NMTREE against Chain, variants without Comp or Sum, and a hand-crafted linguistic assembly rule.These variants test language-structure use, module necessity, and learned versus fixed module selection.
  • Ablation Studies: NMTREE automatically identifies nodes requiring composite reasoning and uses visual cues during module assembly through Gumbel-Softmax training.The ablation discussion contrasts learned assembly with fixed rule-based decisions.
  • Comparison with State-of-the-Arts: Across three benchmarks, NMTREE achieves new state-of-the-art performance on most test splits and grounding tasks, while qualitative and human evaluations indicate transparency and explainability.The comparison discusses holistic, triplet-composition, parser-based, and other grounding models; it also notes a performance–explainability trade-off.
  • Qualitative Analysis: Qualitative visualizations show module assignments, intermediate attention maps, and final results, while attention maps become sharper along the tree.Visual concept words are usually assigned Sum modules, relationship words Comp modules, and human evaluators rated NMTREE reasoning more clearly than AccumAttn.

5. Conclusion

The conclusion presents NMTREE as an end-to-end visual grounding model that accumulates confidence along dependency trees using an annotation-free assembly of three neural modules.

  • 5. Conclusion: NMTREE localizes target regions by accumulating grounding confidence scores along a sentence’s dependency parsing tree.The model is described as a novel end-to-end approach for visual grounding.
  • 5. Conclusion: Its three neural modules are assembled during training without additional annotations.The conclusion characterizes the resulting language composition as more fine-grained and explainable than previous methods.
  • 5. Conclusion: Extensive experiments on three benchmarks demonstrate superior performance alongside more fine-grained and explainable language-composition reasoning.

Supplementary Material for “Learning to Assemble Neural Module Tree

The supplementary material passage only identifies the work as concerning neural module tree networks for visual grounding.

  • Supplementary Material: The supplied supplementary passage contains the phrase “Networks for Visual Grounding” but no substantive supplementary result or method description.

A. Implementation of Tree LSTM

The supplementary implementation section states that Tree LSTM transitions propagate child cell and hidden vectors using gated trainable transformations.

  • A. Implementation of Tree LSTM: The simplified Tree LSTM implementation is expanded into transition equations for each node and its children.
  • A. Implementation of Tree LSTM: For node t and child j, c↑_tj and h↑_tj denote the child’s cell and hidden vectors.
  • A. Implementation of Tree LSTM: The transitions use element-wise multiplication, sigmoid gating, and trainable parameters W, U, and b.

B. More Qualitative Results

The qualitative analysis visualizes NMTree’s internal reasoning, including Comp-module composition, tree structures, module assembly, intermediate attention maps, and final grounding results. Results are shown with ground-truth boxes, detected boxes, and detected masks, including failure cases.

  • B. More Qualitative Results: Figures 9–11 visualize tree structures, module assembly, intermediate attention maps, and final results across qualitative grounding examples.The examples use ground-truth bounding boxes, detected bounding boxes, and detected masks.
  • B. More Qualitative Results: Figure 8 visualizes compositional reasoning inside Comp modules through contextual and output attention maps.Colors identify the current node, its children, and its parent within a partial tree structure.
  • B. More Qualitative Results: Figure 9 compares predicted and ground-truth bounding boxes in qualitative results with ground-truth boxes.The original image shows the ground-truth box in green and the result in red.
  • B. More Qualitative Results: Figure 10 compares predicted and ground-truth bounding boxes when using detected boxes.The original image marks the ground truth in green and the result in red.
  • B. More Qualitative Results: Figure 11 presents qualitative results with detected masks, where blue marks maximum-score regions and green marks ground truth.Word colors indicate the Single, Comp, and Sum modules.
Loading 1812.03299v3…