PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Referring Expressions in Images by Variational Context

Hanwang Zhang, Yulei Niu, Shih-Fu Chang

arXiv:1712.01892v2cs.CV

TL;DR

Referring expression grounding must model complex visual context, but prior multiple-instance approaches simplify exponentially many context configurations. Variational Context uses reciprocal variational Bayesian inference with cue-specific language-vision embeddings, achieving consistent improvements across supervised and unsupervised benchmarks while retaining limitations in unsupervised relation discovery.

  • Problem

    Referring expression grounding requires modeling objects, attributes, and relationships, while prior multiple instance learning simplifies context configurations from O(2^N) to O(N).

  • Method

    Variational Context uses a variational Bayesian lower bound to model reciprocal referent-context inference with cue-specific language-vision modules trained end-to-end.

  • Results

    The model consistently outperforms previous methods across four benchmark datasets in both supervised and unsupervised settings.

  • Takeaways & Limitations

    Reciprocal context modeling supports referring-expression grounding in both annotated and unannotated referent settings.

  • Takeaways & Limitations

    Unsupervised context estimation can select objects outside the expression and confuse relations such as “right” and “left.”

Abstract

from arXiv · show

We focus on grounding (i.e., localizing or linking) referring expressions in images, e.g., "largest elephant standing behind baby elephant". This is a general yet challenging vision-language task since it does not only require the localization of objects, but also the multimodal comprehension of context --- visual attributes (e.g., "largest", "baby") and relationships (e.g., "behind") that help to distinguish the referent from other objects, especially those of the same category. Due to the exponential complexity involved in modeling the context associated with multiple image regions, existing work oversimplifies this task to pairwise region modeling by multiple instance learning. In this paper, we propose a variational Bayesian method, called Variational Context, to solve the problem of complex context modeling in referring expression grounding. Our model exploits the reciprocal relation between the referent and context, i.e., either of them influences the estimation of the posterior distribution of the other, and thereby the search space of context can be greatly reduced, resulting in better localization of referent. We develop a novel cue-specific language-vision embedding network that learns this reciprocity model end-to-end. We also extend the model to the unsupervised setting where no annotation for the referent is available. Extensive experiments on various benchmarks show consistent improvement over state-of-the-art methods in both supervised and unsupervised settings.

1. Introduction

Referring expression grounding requires modeling visual objects, attributes, and relationships to distinguish a referent from other same-category objects. Variational Context addresses this complex context with reciprocal variational modeling and improves grounding in supervised and unsupervised settings.

  • Motivation: Referring expression grounding must jointly interpret objects, attributes, and relationships to distinguish the referent from competing objects.For example, “largest elephant standing behind baby elephant” cannot be resolved by localizing “elephant” alone.
  • Prior limitations: External parsing followed by visual relation detection is impractical because detectors generalize poorly across unrestricted language and scene compositions.
  • Prior limitations: Multiple instance learning reduces context configurations from O(2^N) to O(N) by maximizing the joint likelihood of region pairs.This simplification addresses weak supervision but oversimplifies the possible context associated with a referent.
  • Proposed approach: Variational Context approximates combinatorial context configurations through reciprocal inference between the referent and hidden context.It estimates a coarse context for each candidate region, then uses that context to refine referent localization through three cue-specific multimodal modules.
  • Scope and results: The model supports both supervised grounding with referent annotations and unsupervised grounding without instance-level annotations.
  • Scope and results: The model consistently outperforms previous methods across four benchmark datasets in both supervised and unsupervised settings.The paper also qualitatively demonstrates grounding contextual words to corresponding image regions.

2. Related Work

Prior referring-expression work uses holistic or pairwise context models, while variational Bayesian modeling treats unannotated context as a hidden variable and retains a richer marginalization framework.

  • Referring expression grounding: Referring expression grounding differs from phrase grounding because context distinguishes the referent from other usually same-category objects.
  • Referring expression grounding: Earlier methods often represent context holistically through the entire image or visual feature differences between regions.
  • Referring expression grounding: Explicit referent-context pair models use multiple instance learning because context annotations are unavailable.
  • Variational Bayesian model: Variational Context formulates unannotated context as a hidden variable and approximates the resulting intractable log-sum likelihood with variational Bayesian inference.This connects referring-expression grounding to variational autoencoder-style posterior approximation.
  • Variational Bayesian model: MIL approximations can force every referent-context pair to explain the data, whereas the variational formulation explicitly retains a KL-divergence regularization term.

3. Variational Context

Variational Context models context as a hidden region subset and uses a variational lower bound to make reciprocal referent-context inference tractable. A deterministic, cue-specific architecture enables end-to-end supervised and unsupervised training.

  • 3.1. Problem Formulation: Grounding is formulated as retrieving the referent region x* from image regions X for a language expression L.The objective maximizes the conditional likelihood of the referent region.
  • 3.1. Problem Formulation: Context z may be a multinomial subset of regions rather than one region, producing an O(2^|X|) search space and generally intractable marginalization.
  • 3.1. Problem Formulation: The variational lower bound decomposes grounding into context posterior qφ(z|x,L), referent posterior pθ(x|z,L), and context prior pω(z|L).These modules respectively estimate context, localize the referent given context, and regularize context toward the expression.
  • 3.1. Problem Formulation: KL regularization keeps estimated context close to the expression’s generic visual context and prevents overfitting to inconsistent region features.
  • 3.2. Training and Test: A deterministic differentiable context encoder replaces high-variance sampling over exponentially many configurations with an efficient biased representation.The paper relates this design to re-parameterization-style stochasticity during training.
  • 3.2. Training and Test: Supervised training uses a softmax loss that raises the ground-truth referent score relative to other candidate regions.
  • 3.2. Training and Test: The architecture combines region and language feature extraction with three grounding modules and supports end-to-end supervised or unsupervised losses.

4. Model Architecture

The model builds cue-specific language-vision features and combines single-region, pairwise, referent, and context-consistency scores in an end-to-end variational-context architecture.

  • Architecture: The architecture integrates five modules end-to-end using deterministic context estimation.The overall model is described as end-to-end differentiable because its context is deterministic.
  • RoI Features: Each RoI feature concatenates visual and spatial information, optionally augmented with visual differences between same-category objects.Visual differences support attributes such as “largest” or “baby elephant.”
  • Cue-Specific Language Features: Cue-specific language features separately represent context, referent, and generic cues using single-RoI and pairwise associations.Context and referent features include single and pairwise components, whereas the generic feature uses only single-RoI association.
  • Cue-Specific Language Features: Darker word-weight visualizations show that cue-specific features dynamically emphasize different words across referring expressions.The figure provides qualitative examples of these cue-dependent weights.
  • Cue-Specific Language Features: Context-cue weights favor pairwise relationships, while referent-cue weights rely more on individual descriptions and still use pairwise information.The paper reports that c1 is nearly uniform, c2 is highly skewed, and r1 is more skewed than r2.
  • Score Functions: The score functions estimate context, ground the referent given that context, and regularize context consistency with the expression.Context estimation sums single and pairwise vision-language associations; referent grounding scores candidates given context, while regularization compares generic consistency against context estimation.

5. Experiment

Experiments evaluate VC across four referring-expression benchmarks, supervised and unsupervised settings, detected objects, qualitative context grounding, and ablations. VC achieves state-of-the-art supervised performance, improves context modeling over CMN, and remains effective unsupervised, although unsupervised relation discovery remains difficult.

  • Settings and Metrics: Four benchmarks—RefCLEF, RefCOCO, RefCOCO+, and RefCOCOg—are used, with supervised and unsupervised grounding evaluations.RefCOCO, RefCOCO+, and RefCOCOg include annotated regions; SSD-detected bounding boxes are also evaluated across the four datasets.
  • Supervised Grounding: VC achieves state-of-the-art supervised performance across RefCOCO, RefCOCO+, RefCOCOg, and RefCLEF.The authors attribute the improvement to variational Bayesian context modeling.
  • Supervised Grounding: VC provides more reasonable context localizations than CMN, including cases where VC's referent grounding fails.The comparison highlights better comprehension of language and image-region context, while multiple-human cases remain favorable to CMN.
  • Supervised Grounding: VC considerably outperforms CMN as the number of bounding boxes increases, especially on same-category Test A cases involving people.The authors use this pattern to argue that MIL is ineffective for context modeling when many image regions are present.
  • Unsupervised Grounding: VC outperforms GroundR on unsupervised RefCLEF grounding, supporting the usefulness of context when referent and context annotations are unavailable.Unsupervised grounding is also evaluated on RefCOCO, RefCOCO+, and RefCOCOg.
  • Unsupervised Grounding: Context regularization improves VC by over 2% in most cases and by over 5% on RefCOCO+ (det) and RefCOCOg (det).The authors report that these gains exceed those in supervised settings, where improvement is under 3%.
  • Unsupervised Grounding: Cue-specific language attention is generally ineffective for unsupervised short expressions, but removing language structure degrades performance on longer RefCOCOg expressions.Without attention, the language feature is the average of word embeddings and does not encode structural composition.
  • Unsupervised Grounding: Unsupervised context estimation still produces out-of-scope objects and reverses relations such as “right” and “left.”Nearly identical word attentions in supervised and unsupervised settings indicate that relation modeling remains less successful without supervision.

6. Conclusions

The paper addresses complex context modeling in referring expression grounding with Variational Context, using reciprocity between referent and context to reduce context complexity. Experiments on four benchmarks support its effectiveness in supervised and unsupervised settings.

  • 6. Conclusions: Variational Context models complex context through reciprocal inference between the referent and context.The variational lower bound is interpreted as allowing either the referent or context to help localize the other.
  • 6. Conclusions: A cue-specific language-vision embedding network implements the reciprocity model and is trained end-to-end.
  • 6. Conclusions: Experiments on RefCLEF, RefCOCO, RefCOCO+, and RefCOCOg validate the approach in supervised and unsupervised grounding.
  • 6. Conclusions: The supplementary material includes parsing examples from prior work.

7. Supplementary Material

The supplementary material describes tractable variational training, examines cue-specific language features, and discusses parser-based alternatives for unsupervised grounding. It also reports additional qualitative results and identifies weaker unsupervised language features as a limitation.

  • 7. Supplementary Material: The log-sum objective is transformed into a sum-log form using the concavity of the log function for tractable training.
  • 7. Supplementary Material: Figure 9 presents further cue-specific language features on RefCOCOg.
  • 7. Supplementary Material: Unsupervised VC has weaker language features than supervised VC, motivating parser-based alternatives.The paper adopts a parser jointly trained on referring expression grounding because conventional parsers are suboptimal for the task.
  • 7. Supplementary Material: The parser assigns word-level attention weights to subject, relation, and object features used in the model.
  • 7. Supplementary Material: Additional qualitative supervised and unsupervised grounding results are shown for RefCOCO, RefCOCO+, and RefCOCOg.
Loading 1712.01892v2…