PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning

Li Yang, Yan Xu, Chunfeng Yuan, Wei Liu, Bing Li, Weiming Hu

arXiv:2205.00272v2cs.CV

TL;DR

Visual grounding methods based on predefined proposals or anchors may not fully exploit textual attributes and visual context. This paper uses text-conditioned feature refinement, language-guided context aggregation and multi-stage cross-modal reasoning, achieving state-of-the-art performance across five datasets. The model is trained only on limited-size visual-grounding corpora, which may limit generalization to more open language expressions.

  • Problem

    Predefined proposals or anchors may inadequately capture detailed visual concepts and contexts specified by language expressions, limiting visual grounding.

  • Method

    A transformer-based framework uses visual-linguistic verification, language-guided context aggregation and a multi-stage cross-modal decoder to form discriminative features and localize targets.

  • Results

    The method achieves state-of-the-art performance across five widely used visual-grounding datasets, outperforming prior methods in reported comparisons.

  • Takeaways & Limitations

    Text-conditioned feature refinement and iterative cross-modal reasoning provide an effective framework for accurate target localization.

  • Takeaways & Limitations

    The model is trained on limited-size visual-grounding corpora and may not generalize well to more open language expressions.

Abstract

from arXiv · show

Visual grounding is a task to locate the target indicated by a natural language expression. Existing methods extend the generic object detection framework to this problem. They base the visual grounding on the features from pre-generated proposals or anchors, and fuse these features with the text embeddings to locate the target mentioned by the text. However, modeling the visual features from these predefined locations may fail to fully exploit the visual context and attribute information in the text query, which limits their performance. In this paper, we propose a transformer-based framework for accurate visual grounding by establishing text-conditioned discriminative features and performing multi-stage cross-modal reasoning. Specifically, we develop a visual-linguistic verification module to focus the visual features on regions relevant to the textual descriptions while suppressing the unrelated areas. A language-guided feature encoder is also devised to aggregate the visual contexts of the target object to improve the object's distinctiveness. To retrieve the target from the encoded visual features, we further propose a multi-stage cross-modal decoder to iteratively speculate on the correlations between the image and text for accurate target localization. Extensive experiments on five widely used datasets validate the efficacy of our proposed components and demonstrate state-of-the-art performance. Our code is public at https://github.com/yangli18/VLTVG.

1. Introduction

Visual grounding must exploit textual attributes and relations while producing discriminative visual features, because proposal- and anchor-based methods can miss detailed context. The proposed framework verifies visual-linguistic relevance, aggregates language-guided context, and iteratively reasons across modalities for localization.

  • Visual grounding localizes an object or region from a natural-language expression containing categories, attributes, or visual relations.
  • Existing methods rank detected proposals or dense anchor locations, making performance dependent on pre-generated regions or predefined anchors.
  • Region and point features from predefined locations may inadequately capture detailed visual concepts and contexts expressed in language.
  • The framework establishes discriminative features through visual-linguistic verification and language-guided context aggregation before localization.
  • A multi-stage cross-modal decoder iteratively queries visual and linguistic information to reduce ambiguity and retrieve the referred object more accurately.
  • The method is benchmarked on five datasets, with extensive experiments and ablations validating its components and reporting improvements over prior state-of-the-art methods.

2. Related Work

Related work extends visual grounding from detector-based proposal ranking toward transformer-based multimodal modeling. The proposed approach uses transformer and attention mechanisms to model visual-linguistic correlations for image-based grounding.

  • Two-stage methods use off-the-shelf detectors to generate region proposals and match them with language, but depend heavily on detector or proposal-generator quality.
  • One-stage methods fuse language embeddings into detectors to generate dense detections and select the highest-confidence referred-object prediction.
  • Transformers have demonstrated broad modeling capability in vision and language, motivating their use as a flexible alternative for visual grounding.
  • The proposed architecture extracts image and language features separately, then applies verification, context encoding, and iterative decoding to infer the target location.
  • Unlike proposal-level multimodal methods, this work models visual-linguistic correlations pixel-wisely to focus feature encoding on semantic information for object localization.

3. Method

The framework directly retrieves the referred object's feature representation by combining text-conditioned visual discrimination, language-guided context aggregation, and iterative cross-modal decoding. These components focus visual features on relevant regions, gather contextual information, and progressively refine target localization.

  • Overall Architecture: The framework extracts visual and linguistic features separately, then applies verification, context encoding, and multi-stage decoding for direct object localization.Visual features come from a CNN and transformer encoder, while language expressions are encoded with BERT.
  • Visual-Linguistic Verification: Visual-linguistic verification computes fine-grained correlations between visual features and textual embeddings to emphasize language-relevant regions.Visual features serve as queries, while textual embeddings provide keys and values for constructing a spatially aligned semantic map.
  • Visual-Linguistic Verification: Verification scores modulate visual features pixel-wise, suppressing regions irrelevant to the referring expression and facilitating later recognition and localization.The scores represent each visual feature's semantic relevance to the linguistic expression.
  • Language-guided Context Encoder: The language-guided context encoder gathers visual context under textual guidance to model interaction relations and relative positions needed for target discrimination.Language-guided self-attention produces context representations that are fused with visual features and verification scores.
  • Multi-stage Cross-modal Decoder: The multi-stage cross-modal decoder iteratively queries linguistic and visual information, progressively refining target features before predicting bounding boxes.Each stage uses a target query, gathers semantic descriptions and corresponding visual features, updates the query, and predicts a box; all stages receive supervision.
  • Training Loss: Training directly regresses predicted boxes against the ground-truth box across stages using GIoU and L1 losses.This avoids positive/negative sample assignment used by ranking-based methods, with λ_giou and λ_L1 balancing the losses.

4.1. Datasets

The evaluation uses three MS COCO-based visual grounding benchmarks with distinct expression and split structures. RefCOCOg additionally contains generally longer referring expressions and two evaluation conventions.

  • RefCOCO: RefCOCO contains 19,994 images, 142,210 referring expressions, and 50,000 referred objects across train, validation, testA, and testB splits.The split sizes are 120,624, 10,834, 5,657, and 5,095 expressions, respectively.
  • RefCOCOg: RefCOCOg contains 25,799 images, 95,010 referring expressions, and 49,822 object instances, with generally longer expressions than the other two datasets.Experiments follow both RefCOCOg-google and RefCOCOg-umd splitting conventions.

4.2. Implementation Details

The implementation fixes image and language input dimensions, uses AdamW with separate learning rates for the network and feature-extraction branches, and evaluates localization by IoU accuracy.

  • Input Processing: Images are resized or padded to 640 × 640, while language expressions are capped at 40 tokens and processed with BERT using [CLS] and [SEP].During inference, the longer image edge is resized to 640 and the shorter edge is padded.
  • Optimization and Architecture: Training uses AdamW with batch size 64, learning rates of 10^-4 for the network and 10^-5 for feature-extraction branches, and ResNet-50 or ResNet-101 backbones.The visual branch uses six transformer encoder layers initialized from DETR, and the text branch is initialized with BERT.
  • Evaluation: A predicted bounding box is counted as correct when its IoU with the ground-truth box exceeds 0.5.Table 1 compares the method with other state-of-the-art methods on RefCOCO, RefCOCO+, and RefCOCOg.

4.3. Comparisons with State-of-the-art Methods

The method consistently outperforms prior state-of-the-art approaches across five visual-grounding datasets and multiple model settings. Gains are reported against both two-stage and one-stage methods, including TransVG.

  • Comparisons with two-stage methods: Up to 4.45%, 5.94% and 5.49% absolute improvements are achieved over Ref-NMS on RefCOCO, RefCOCO+ and RefCOCOg, respectively.The method outperforms other methods in all splits of the three datasets.
  • Comparisons with one-stage methods: 84.77% and 80.49% accuracies on RefCOCO val and testB improve on the previous one-stage best by 5.10 and 6.34 percentage points.These results are reported for the val and testB splits, respectively.
  • Comparison with TransVG: The method exceeds TransVG by 2.14%–4.52% on RefCOCO, 8.23%–9.37% on RefCOCO+ and 5.96%–7.37% on RefCOCOg.The comparison is reported across the dataset splits in Table 1.
  • Additional datasets: The method surpasses prior one-stage and two-stage methods on ReferItGame and Flickr30k Entities across different backbones.Improvements over TransVG are smaller on Flickr30k Entities than on the other datasets.

4.4. Ablation Study

Ablations show that iterative decoding, language-guided context aggregation and visual-linguistic verification each improve grounding accuracy. The proposed components add relatively little model size and computational complexity, while decoder performance saturates near six stages.

  • Component ablations: Adding more cross-modal decoder stages improves the baseline accuracy by 2.38 points through iterative reasoning for target localization.The baseline uses a single-stage decoder and achieves 63.64% accuracy.
  • Component ablations: Introducing the language-guided context encoder further boosts accuracy by 2.42 points by gathering visual contexts for grounding.The ablation is conducted on RefCOCOg val-g, which contains longer language expressions.
  • Efficiency: The full model’s proposed components add 8.81M parameters and 0.69 GFLOPS, increasing model size and complexity by 6.14% and 1.68%.These figures compare the proposed variants with the baseline.
  • Decoder-stage analysis: Accuracy rises from 65.22% to 67.57% as decoder stages increase, reaching saturation near N = 6.The paper therefore employs six decoder stages because gains become small when N ≥6.
  • Feature-learning design: Replacing the verification module and context encoder with common transformer encoders reduces performance by 2.07 percentage points.This comparison evaluates visual-linguistic feature-learning designs.

4.5. Visualization

Visualizations illustrate how language conditions verification, context aggregation and iterative decoding. The model emphasizes description-relevant regions and gathers surrounding context to support localization.

  • Verification and decoding: Verification scores are generally higher on objects or regions related to the language expression, reducing distraction in the modulated visual features.The multi-stage decoder then uses these focused regions during localization.
  • Context aggregation: The language-guided context encoder focuses on context around the related horse when given a language expression referring to a girl.The gathered visual context helps identify the referred girl.

5. Conclusions and Limitations

The framework establishes discriminative visual features and performs iterative cross-modal reasoning for visual grounding. Its limitation is that training uses visual grounding datasets with limited corpus sizes, which may restrict generalization to more open language expressions.

  • Conclusions: The framework combines visual-linguistic verification, language-guided context encoding, and multi-stage cross-modal decoding for target localization.Verification focuses encoding on text-relevant regions, context encoding gathers informative visual context, and decoding iteratively reasons over visual and linguistic information.
  • Conclusions: Extensive experiments on public datasets report state-of-the-art performance.
  • Limitations: The model is trained only on visual grounding datasets with limited corpus sizes.The authors state that this may prevent strong generalization to more open language expressions.
  • Limitations: The authors plan to extend the method to larger-scale grounding datasets to improve generalization ability.
Loading 2205.00272v2…