PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding

Zhou Yu, Jun Yu, Chenchao Xiang, Zhou Zhao, Qi Tian, Dacheng Tao

arXiv:1805.03508v1cs.CV

TL;DR

Visual grounding research has paid less attention to proposal generation, leaving the properties of a good generator uncertain. DDPN combines proposal diversity and discrimination, and its baseline performs significantly better across four benchmark datasets.

  • Problem

    Proposal generation has been less thoroughly investigated in visual grounding, and what makes a good proposal generator remains uncertain.

    Source: §1 · p. 2

  • Method

    DDPN generates diverse, discriminative proposals using a class-aware detector trained on Visual Genome, and supports a grounding baseline with contextual KLD and box-regression losses.

    Source: §1 · p. 2, §2 · p. 2, §5 · p. 6

  • Results

    Across four benchmark datasets, results are significantly better; with VGG-16, ReferItGame improves by 16.1 points over the compared model.

    Source: §5 · p. 6, §4.5 · p. 6

  • Takeaways & Limitations

    The authors attribute the improvements primarily to DDPN’s high-quality proposals and the grounding model’s loss functions.

    Source: §4.5 · p. 6

    The studied models are baselines, leaving room for stronger DDPN backbones or more powerful multimodal fusion such as bilinear pooling.

    Source: §5 · p. 6

Abstract

from arXiv · show

Visual grounding aims to localize an object in an image referred to by a textual query phrase. Various visual grounding approaches have been proposed, and the problem can be modularized into a general framework: proposal generation, multi-modal feature representation, and proposal ranking. Of these three modules, most existing approaches focus on the latter two, with the importance of proposal generation generally neglected. In this paper, we rethink the problem of what properties make a good proposal generator. We introduce the diversity and discrimination simultaneously when generating proposals, and in doing so propose Diversified and Discriminative Proposal Networks model (DDPN). Based on the proposals generated by DDPN, we propose a high performance baseline model for visual grounding and evaluate it on four benchmark datasets. Experimental results demonstrate that our model delivers significant improvements on all the tested data-sets (e.g., 18.8\% improvement on ReferItGame and 8.2\% improvement on Flickr30k Entities over the existing state-of-the-arts respectively)

1 Introduction

Visual grounding must match an open-ended query to an object, but proposal generation has received less attention than feature fusion and ranking. The paper proposes DDPN to generate diverse, discriminative proposals and builds a baseline that improves on four benchmark datasets.

  • Visual grounding localizes the object referred to by a textual query, unlike object detection, which targets predefined object classes.Source: §1 · p. 1
  • Proposal generation has been less thoroughly investigated, and what makes a good visual-grounding proposal generator remains uncertain.Existing approaches commonly use class-agnostic selective search or RPNs [Ren et al., 2015].Source: §1 · p. 2
  • DDPN aims to generate a relatively small set of proposals covering many different objects by combining diversity and discrimination.The authors identify proposal-set size as a trade-off: too few proposals limit recall, while too many make accurate ranking harder.Source: §1 · p. 2
  • The DDPN-based baseline uses feature concatenation, KLD loss with soft labels for proposal context, and smoothed L1 regression to refine boxes.Source: §1 · p. 2
  • The proposed model delivers significant improvements on all four tested benchmark datasets.The study also evaluates its models through extensive ablation studies.Source: §1 · p. 2

2 The Diversified and Discriminative Proposal Networks (DDPN)

The paper argues that useful proposals should cover varied object classes across images while remaining discriminative within each image. DDPN approximates both properties with a class-aware detector trained to recognize a large set of classes.

  • Proposal diversity should span object classes across images, while discrimination should make each image’s proposals accurately represent its true semantics.Source: §2 · p. 2
  • DDPN trains class-aware Faster R-CNN on Visual Genome to detect a relatively large class set, approximating diversity while retaining discrimination.The proposed scheme relaxes the diversity constraint to more than 1k object classes.Source: §2 · p. 2
  • The processed Visual Genome training set contains 1,600 object classes and 400 attribute classes, with an additional loss for attribute prediction.Source: §2 · pp. 2–3
  • Unlike Anderson et al. [Anderson et al., 2017], who use Faster R-CNN for bottom-up attention, this paper uses it to learn proposals for visual grounding.Source: §2 · p. 3

3 Baseline Model for Visual Grounding

The baseline uses DDPN proposals and image-query features to rank and refine the referred object. Its learning objective combines soft-label ranking with bounding-box regression.

3 Baseline Model for Visual Grounding

  • The model uses DDPN to extract the top-N proposals, then represents each with a visual feature and five-dimensional spatial feature.The visual feature is the last fully connected layer after RoI pooling; spatial values encode normalized box coordinates and area.Source: §3 · p. 3
  • A one-layer LSTM encodes the embedded query, using its last-word feature as the query representation.The embedding layer maps words into a latent space before the sequence is passed to the LSTM.Source: §3 · p. 3
  • The baseline fuses query and visual features through concatenation and a fully connected layer rather than a complex fusion model.Source: §3 · p. 3

3.1 Learning Objective

  • KLD ranking uses normalized soft labels based on proposal IoU, assigning each proposal its IoU when it exceeds threshold η and zero otherwise.The target distribution is L1-normalized, and KLD brings the predicted score distribution closer to it.Source: §3.1 · p. 3
  • Smoothed L1 regression penalizes differences between refined proposal coordinates and ground-truth boxes across proposals.The regression layer is described as domain adaptation aligning Visual Genome source semantics with the target grounding dataset.Source: §3.1 · p. 4
  • The overall objective combines ranking and regression losses, with γ balancing their contributions.Source: §3.1 · p. 4

4 Experiments

The approach is evaluated on four commonly used visual-grounding benchmarks.

  • The evaluation uses Flickr30K Entities, ReferItGame, RefCOCO, and RefCOCO+.Source: §4 · p. 4

4.1 Implementation Details

The implementation trains DDPN with two CNN backbones and then trains the visual-grounding model with fixed hyperparameters and a specified testing procedure.

  • DDPN is trained with VGG-16 and ResNet-101 backbones for up to 30 epochs, taking two GPUs 2–3 weeks.Source: §4.1 · p. 4
  • The model uses 100 proposals and an IoU-score threshold η of 0.5; the LSTM feature dimension is 1024, and the fused-feature dimension is 512.Source: §4.1 · p. 4
  • The grounding model is trained with Adam for up to 10k iterations, and testing selects the highest-scoring proposal's refined box.Source: §4.1 · p. 4

4.2 Datasets

The experiments use three dataset groupings with distinct sizes, query characteristics, and splits.

Flickr30K Entities

  • Flickr30K Entities contains 32k images, 275k boxes, and 360k query phrases, with a 30k/1k/1k train/validation/test split.Phrases referring to multiple boxes use their union as ground truth.Source: Flickr30K Entities · p. 4

ReferItGame

  • ReferItGame contains over 99k boxes and 130k query phrases across 20k images, split into 9k training, 1k validation, and 10k test images.Source: ReferItGame · p. 4

RefCOCO and RefCOCO+

  • RefCOCO and RefCOCO+ contain 142k queries and 50k boxes from 20k COCO images; RefCOCO+ excludes location words and is harder to interpret.Their splits are train, validation, testA, and testB; testA has multiple people, while testB contains objects in other categories.Source: RefCOCO and RefCOCO+ · p. 4

4.3 Experimental Setup

The experiments define accuracy by bounding-box overlap and compare the approach with existing visual-grounding methods.

Evaluation Metric

Compared Methods

4.4 Ablation Studies

The ablations assess proposal diversity and discrimination alongside visual-grounding losses. DDPN improves proposal-generator scores and grounding performance, while the loss variants also improve accuracy.

4.4 Ablation Studies

  • The ablations evaluate DDPN and the visual-grounding model’s loss functions; Figure 4 visualizes the top 20 scored proposals from four generators.Source: §4.4 · p. 4, Figure 4 · p. 5

Effects of Diversity and Discrimination in Proposal Generation

  • SDIS is the share of samples with at least one proposal covering ground truth, while SDIV is the reciprocal of average covered proposals per ground-truth object.A good proposal generator should score highly on both measures.Source: Effects of Diversity and… · p. 5
  • DDPN significantly outperforms other proposal generators on discrimination, diversity, and visual-grounding performance.The comparison is evaluated on Flickr30k Entities; the table reports all three measures.Source: Effects of Diversity and… · p. 5, Table 1 · p. 5
  • RPN fine-tuning on Flickr30k Entities improves discrimination but reduces diversity, indicating a cost to generalizability.Source: Effects of Diversity and… · p. 5

Effects of Different Losses on Visual Grounding

4.5 Comparisons with State-of-the-Art

With VGG-16, the model improves on the reported benchmarks; replacing it with ResNet-101 further improves all results. The comparison reports accuracy on RefCOCO, RefCOCO+, and Flickr30k Entities, as well as ReferItGame.

  • The reported comparisons include method accuracies on RefCOCO and RefCOCO+ in Table 3 and Flickr30k Entities in Table 4.Source: Table 3 · p. 6, Table 4 · p. 6
  • With VGG-16, our model achieves absolute improvements of 3.2 points on RefCOCO (testA), 6.3 points on RefCOCO+ (testA), 4.9 points on Flickr30k Entities and 16.1 points on ReferItGame.Source: §4.5 · p. 6, Table 3 · p. 6, Table 4 · p. 6
  • Replacing VGG-16 with ResNet-101 improves all results by 3–4 points.The comparison reports that DDPN representation capacity significantly influences visual grounding performance.Source: §4.5 · p. 6

4.6 Qualitative Results

Qualitative examples on Flickr30k Entities and ReferItGame show that the model handles small, fine-grained objects and that bounding-box regression can refine inaccurate proposals. Complex queries and confused visual objects remain challenging.

  • The model handles small, fine-grained objects in the visualized Flickr30k Entities and ReferItGame examples.Source: §4.6 · p. 6
  • Bounding-box regression helps refine results by rectifying some inaccurate proposals.Source: §4.6 · p. 6
  • The approach still has limitations with complex queries or confused visual objects.Source: §4.6 · p. 6

5 Conclusions and Future Work

The paper proposes DDPN for high-quality visual grounding proposals and builds a baseline using those proposals and two novel losses. Experiments show significantly better results across four benchmark datasets, while the baseline leaves room for improvement.

  • DDPN generates high-quality proposals, and the paper’s baseline uses them with KLD loss and regression loss.The KLD loss captures proposal contextual information; regression refines proposals.Source: §5 · p. 6
  • Experiments on four benchmark datasets achieve significantly better results on all datasets.Source: §5 · p. 6
  • The baseline leaves room for improvement through a more advanced DDPN backbone or more powerful multimodal fusion such as bilinear pooling.Source: §5 · p. 6
Loading 1805.03508v1…