PaperlayerView exact revision on arXiv

Source-linked AI summary

TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer

Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, Wanli Ouyang

arXiv:2206.06619v1cs.CV

TL;DR

Visual grounding methods often use manually designed fusion mechanisms, which can overfit specific data distributions, or standalone fusion Transformers that are difficult to optimize with limited grounding data. TransVG uses Transformer layers and direct box regression, and TransVG++ reuses ViT layers for language-conditioned fusion; the authors report improvements over five benchmarks.

  • Problem

    Manually designed fusion can overfit specific scenarios, and standalone fusion Transformers are difficult to optimize when trained from scratch on limited visual-grounding data.

    Source: §1 · pp. 1–2

  • Method

    TransVG uses Transformer layers for multimodal reasoning and directly regresses boxes, while TransVG++ uses a language-conditioned ViT for vision encoding and intermediate-layer fusion.

    Source: §1 · pp. 2–3

  • Results

    TransVG++ improves over TransVG by 4.99%, 8.04%, 8.32%, 3.97%, and 2.39% absolute on RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that TransVG++ achieves state-of-the-art records across five prevalent visual-grounding datasets.

    Source: §1 · p. 2

    The language branch accounts for most model parameters, a concern for deployment on devices with limited resources.

    Source: §4.5 · p. 13

Abstract

from arXiv · show

In this work, we explore neat yet effective Transformer-based frameworks for visual grounding. The previous methods generally address the core problem of visual grounding, i.e., multi-modal fusion and reasoning, with manually-designed mechanisms. Such heuristic designs are not only complicated but also make models easily overfit specific data distributions. To avoid this, we first propose TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We empirically show that complicated fusion modules can be replaced by a simple stack of Transformer encoder layers with higher performance. However, the core fusion Transformer in TransVG is stand-alone against uni-modal encoders, and thus should be trained from scratch on limited visual grounding data, which makes it hard to be optimized and leads to sub-optimal performance. To this end, we further introduce TransVG++ to make two-fold improvements. For one thing, we upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding. For another, we devise Language Conditioned Vision Transformer that removes external fusion modules and reuses the uni-modal ViT for vision-language fusion at the intermediate layers. We conduct extensive experiments on five prevalent datasets, and report a series of state-of-the-art records.

1 INTRODUCTION

Visual grounding methods often rely on manually designed fusion or candidate-based prediction, which can overfit specific scenarios and depend on proposal or anchor generation. TransVG uses Transformer layers and direct box regression, while TransVG++ reuses a ViT-based vision encoder for language-conditioned fusion; the authors report improved results across five benchmarks.

1 INTRODUCTION

  • Existing fusion mechanisms can overfit specific scenarios and limit vision-language interaction, while candidate-based predictions depend on proposal or anchor generation and target assignment.Two-stage methods use region proposals, while one-stage methods use predefined anchor boxes.Source: §1 · p. 1
  • TransVG replaces manually designed fusion modules with Transformer encoder layers for multimodal reasoning and directly regresses referred-box coordinates.Its attention modules establish intra- and inter-modality correspondences without assuming language-expression structure or visual layout.Source: §1 · pp. 2–3
  • TransVG++ removes the standalone fusion Transformer by integrating language information into intermediate layers of a ViT-based vision encoder.Language prompter and language adapter strategies convert ViT layers into language-conditioned visual encoder layers.Source: §1 · pp. 2–3
  • TransVG++ improves over TransVG by 4.99%, 8.04%, 8.32%, 3.97%, and 2.39% absolute on RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities, respectively.The authors report that these results set a series of state-of-the-art records.Source: §1 · p. 2
  • TransVG++ upgrades the hybrid CNN-and-Transformer architecture to a purely Transformer-based framework and reports better efficiency in model size and computation costs than TransVG.The authors describe it as the first fully Transformer-based visual grounding framework without introducing inductive bias.Source: §1 · p. 3

2 RELATED WORK

Prior visual-grounding work is organized around two-stage region matching and one-stage dense fusion, while Transformer research spans vision and vision-language tasks. The paper positions its approach as adapting a plain ViT for multimodal grounding rather than relying on a separate fusion block.

2.1 Visual Grounding

  • Two-stage methods generate region proposals and select a language-matched region, whereas one-stage methods fuse language densely with visual features and predict boxes in a sliding-window manner.Prior work adds modular attention, scene graphs, multi-round fusion, and other mechanisms to improve relationship modeling.Source: §2.1 · p. 3
  • The paper explores removing the external fusion module and reusing a plain ViT for vision-language fusion, aiming for efficiency and access to advances in pretrained vision models.The authors describe the unified encoder as both lightweight and better-performing for the same computation budget.Source: §2.1 · p. 3

2.2 Transformer

  • Vision Transformers have been applied to classification and detection, and this work adapts a uni-modal ViT for multimodal visual grounding while seeking to preserve its vision-encoding capability.The related work notes that plain ViT can serve as a backbone for object detectors.Source: §2.2 · p. 4
  • Unlike vision-language pretraining, which learns general representations from large-scale data, this work explores Transformer-based grounding and removes standalone fusion layers used with ViT backbones.The cited pretraining literature includes image-text matching, word-region alignment, and masked prediction tasks.Source: §2.2 · p. 4

3 OUR APPROACH

TransVG replaces manually designed fusion with Transformer-based multimodal reasoning and direct box regression. Its preliminary framework combines vision and language features through a shared Transformer and predicts the referred region from a learned [REG] token.

3 OUR APPROACH

  • The approach replaces a manually designed multimodal fusion module with a stack of Transformer encoder layers.The paper frames attention as a mechanism for establishing correspondences between vision and language inputs without presupposing expression structure.Source: §3 · p. 4

3.1 Background: Transformer

  • Transformer encoder layers use multi-head self-attention and a feed-forward network, with layer normalization placed differently in post- and pre-normalization variants.Each sub-layer is wrapped in a residual structure.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Preliminary Version: TransVG Framework

  • TransVG directly regresses a referred region’s coordinates from the [REG] token after Transformer-based vision-language reasoning.The framework combines vision and language tokens in a common semantic space, where self-attention models intra- and inter-modality context.Source: §3.2 · pp. 5–6
  • The TransVG vision branch uses a ResNet feature map projected to 256 channels, flattened into tokens, and supplemented with sine position embeddings.Its language branch follows BERTBASE, using learned position embeddings and [CLS] and [SEP] tokens.Source: §3.2 · pp. 5–6
  • Separate linear projections place vision and language features in a shared 256-dimensional space before they enter the vision-language Transformer with a learned [REG] token.The joint input retains positional and modality information through learned position embeddings at each Transformer layer.Source: §3.2 · p. 6

Q K V

TransVG++ replaces the stand-alone fusion Transformer with a ViT-based encoder that incorporates language at intermediate layers. The section also describes two language-integration strategies and a scale-aware box-regression objective.

3.3 Advanced Version: TransVG++ Framework

  • TransVG++ replaces the stand-alone fusion module with LViT, which performs vision encoding and multimodal fusion while processing raw image patches.LViT converts a uni-modal ViT with minimal adaptation and negligible extra computation costs.Source: §3.3 · p. 6
  • LViT converts the last encoder layer in each of four groups into a language-conditioned layer, using either a language prompter or a language adapter.The strategies integrate language features into vision tokens at intermediate layers.Source: §3.3 · pp. 6–7
  • The language prompter appends generated prompt tokens to the [REG] and vision tokens for self-attention, then drops the prompt tokens.The prompt-generating FFN is randomly initialized; other components use corresponding pretrained ViT parameters.Source: §3.3 · p. 7
  • The language adapter adds cross-attention between self-attention and the FFN, using vision outputs as queries and language tokens as keys and values.Its resulting language-adapt tokens are added elementwise to vision tokens; the adapter module is randomly initialized.Source: §3.3 · p. 7
  • The language adapter achieves slightly better accuracy than the language prompter, while both strategies add negligible parameters and computation costs.Both strategies improve performance against the preliminary version with stand-alone fusion Transformers.Source: §3.3 · p. 7

3.4 Training Objective

  • The method directly predicts four box coordinates, normalizes ground-truth coordinates by image scale, and combines smooth L1 and GIoU losses.The paper motivates GIoU as addressing the scale sensitivity of smooth L1 loss.Source: §3.4 · p. 7

4 EXPERIMENTS

The experiments evaluate TransVG and TransVG++ through dataset benchmarking, component ablations, comparisons with state-of-the-art methods, and analyses of model costs and qualitative outputs.

4 EXPERIMENTS

  • The experiments are designed to validate the merits of the TransVG and TransVG++ frameworks.The paper reports evaluation across five prevalent datasets.Source: §4 · p. 7
  • Ablation studies examine components of both frameworks, followed by comparisons of their main results with other state-of-the-art methods.These are presented in Sections 4.3 and 4.4, respectively.Source: §4 · p. 7
  • The experiments also analyze TransVG++ model size and computation costs and present qualitative results for validation and analysis.The paper identifies resource-limited devices as a motivation for future directions.Source: §4 · p. 7

4.1 Datasets and Evaluation Metric

Evaluation covers five visual-grounding datasets and uses accuracy under a Jaccard-overlap threshold. The dataset descriptions specify their splits and, for RefCOCO+, a language-expression constraint.

4.1 Datasets and Evaluation Metric

  • A prediction counts as correct only when its Jaccard overlap with the ground-truth region exceeds 0.5.The frameworks are evaluated in terms of accuracy.Source: §4.1 · p. 7
  • RefCOCO contains 19,994 images, 50,000 referred objects, and 142,210 expressions, split into train, validation, testA, and testB sets.The split contains 120,624 training, 10,834 validation, 5,657 testA, and 5,095 testB expressions.Source: §4.1 · p. 7
  • RefCOCO+ [17] has 19,992 images, 49,856 referred objects, and 141,564 referring expressions; compared with RefCOCO, it excludes expressions indicating absolute position, such as ’left’ and ’right’.Its official split comprises train, validation, testA, and testB sets, containing 120,191, 10,758, 5,726, and 4,889 expressions, respectively.Source: §4.1 · p. 7
  • RefCOCOg has two reported split protocols, while ReferItGame uses its validation set for ablations and test set for comparisons.RefCOCOg results cover google validation and umd validation and test splits.Source: §4.1 · pp. 7–8
  • Flickr30K Entities adds short region-phrase correspondence annotations to Flickr30K and uses 29,783 training, 1,000 validation, and 1,000 test images.The dataset contains 31,783 images and 427K referred entities.Source: §4.1 · p. 8

4.2 Experimental Setup

The setup uses resized and padded 640 × 640 images and length-limited language queries. TransVG++ uses ViT patch tokens with resized position embeddings and trains for fewer epochs than TransVG.

Inputs

  • Images are resized with aspect ratio preserved; the longer edge becomes 640 and the shorter edge is padded to 640.Language queries are capped at 40 tokens for RefCOCOg and 20 for other datasets, with padding masked from the output.Source: §4.2 · p. 8

Network Architecture

  • TransVG++ divides 640 × 640 images into 16 × 16 patches and resizes ViT position embeddings to 40 × 40.Visual tokens are formed by flattening patches and projecting them through a fully connected layer; learnable position embeddings are added.Source: §4.2 · p. 8

Training and Inference Details

  • TransVG++ is trained for 60 epochs, compared with 90 epochs for RefCOCO, RefCOCOg, and ReferItGame in TransVG.TransVG training lasts 60 epochs on Flickr30K Entities and 180 on RefCOCO+; TransVG++ drops its learning rate after epoch 45.Source: §4.2 · p. 8
  • Both frameworks directly output box coordinates, so inference requires no post-processing.Source: §4.2 · p. 8

4.3 Ablative Experiments

The ablations examine TransVG’s [REG] token and branch Transformers, then TransVG++ fusion strategies and language-conditioned layer configurations. The framework comparison tests the effects of replacing the vision backbone and moving fusion into the vision encoder.

4.3.1 Ablation Studies on TransVG

  • The learnable [REG] embedding achieves 72.50% validation and 69.76% test accuracy on ReferItGame, the best among the tested initial-state designs.The alternative initial states are generated from vision or language tokens; the comparison evaluates accuracy on the validation and test sets.Source: §4.3.1 · pp. 8–9, Table 1 · p. 9
  • On ReferItGame, the full vision-and-language Transformer branches reach 69.76% accuracy, versus 64.24% without either branch Transformer.Adding only the vision Transformer reaches 68.48%, while adding only the language Transformer reaches 66.78%.Source: §4.3.1 · p. 9, Table 2 · p. 9

4.3.2 Ablation Studies on TransVG++

  • 73.17% on ReferItGame and 70.60% on RefCOCOg-google: MHCA language adapters outperform the preliminary fusion strategy by 2.17% and 3.36%, respectively.The comparison evaluates fusion strategies on the validation sets of both datasets.Source: §4.3.2 · pp. 9–10, Table 3 · p. 9
  • The default placement of language adapters at the 3rd, 6th, 9th, and 12th vision encoder layers ranks first among the tested configurations.The alternatives place adapters in the first four or last four layers; the evaluation uses RefCOCOg-google.Source: §4.3.2 · p. 10, Table 4 · p. 10
  • Increasing language-conditioned vision encoder layers from 2 to 4 raises RefCOCOg-google accuracy from 70.13% to 70.60%, with no shown benefit beyond four.The study uses a ViT-tiny backbone.Source: §4.3.2 · p. 10, Table 5 · p. 10

4.3.3 Improvements from TransVG to TransVG++

  • 70.60% accuracy on RefCOCOg-google: TransVG++ with language adapters improves over TransVG’s 66.35% baseline.The default configuration adds 0.57M parameters and reduces fusion computation from 7.38G FLOPs in model (b) to 0.92G FLOPs.Source: Table 6 · p. 10, §4.3.3 · pp. 10–11
  • Replacing ResNet-50 with ViT-tiny raises RefCOCOg-google accuracy from 66.35% to 67.24%, while reducing vision-feature parameters from 30.18M to 5.53M.With equalized vision-token counts, the dilated ResNet variant reaches 65.96%; the authors attribute the difference to the ViT backbone and fully Transformer-based architecture.Source: §4.3.3 · p. 10, Table 6 · p. 10
  • Using language adapters with ResNet-50 reaches 68.72% RefCOCOg-google accuracy, 2.37% above the baseline and with fewer parameters.This configuration converts the six vision encoder layers into language-conditioned layers rather than using a separate fusion block.Source: §4.3.3 · p. 10, Table 6 · p. 10

4.4 Main Results and Comparisons

TransVG and TransVG++ are evaluated against two-stage, one-stage, and Transformer-based methods across visual-grounding benchmarks. TransVG++ achieves the best grounding accuracy on the RefCOCO-family benchmarks, ReferItGame, and Flickr30K Entities, while the comparisons also report strong results for TransVG.

4.4.1 RefCOCO/RefCOCO+/RefCOCOg

  • On RefCOCO’s testB set, TransVG with ResNet-101 beats the strongest two-stage competitor, CM-A-E, by 5.62% accuracy.TransVG also outperforms all one-stage methods across the reported RefCOCO, RefCOCO+, and RefCOCOg subsets and splits.Source: §4.4.1 · p. 11
  • The authors report that TransVG’s simple stack of Transformer encoder layers can replace more complicated multi-modal fusion modules.Their explanation contrasts homogeneous intra- and inter-modality context reasoning with ReSC’s isolated language sub-query construction and visual-feature modulation.Source: §4.4.1 · p. 11
  • TransVG++ achieves the best grounding accuracy among compared methods on RefCOCO, RefCOCO+, and RefCOCOg, including with ViT-tiny’s 5.53M parameters.The authors also report that TransVG++ outperforms Refformer on all evaluated subsets; Refformer uses additional segmentation annotations.Source: §4.4.1 · pp. 11–12

4.4.2 ReferItGame

  • TransVG with ResNet-101 reaches 70.73% ReferItGame accuracy, and TransVG++ with ViT-base improves on it by 3.97%.The ResNet-101 TransVG result exceeds the strongest one-stage competitor by 6.13% and the strongest two-stage competitor by 7.73%.Source: §4.4.2 · p. 12, Table 8 · p. 12

4.4.3 Flickr30K Entities

  • 81.49% top-1 accuracy on Flickr30K Entities: TransVG++ with ViT-base exceeds the previous state-of-the-art record by 4.75% absolute.Source: §4.4.3 · p. 12, Table 8 · p. 12
  • TransVG++ reaches 79.10% top-1 accuracy on Flickr30K Entities with ResNet-101, compared with 81.49% using ViT-base.Source: §4.4.3 · p. 12

4.5 Discussion on Model Size and Computation Costs

TransVG++ adds few language-adapter parameters and FLOPs relative to the model, while the language branch accounts for much of the parameter count and the vision branch dominates computation. The authors identify multimodal efficiency for resource-limited devices as an open problem.

4.5 Discussion on Model Size and Computation Costs

  • Language-branch parameters make up 94.4%, 81.6%, and 53.3% of TransVG++ (tiny), (small), and (base), respectively; the tiny model’s adapter adds 0.57M parameters, or 0.49% of the total.The reported component breakdown compares models using ViT-tiny, small, and base vision backbones.Source: §4.5 · p. 12, Table 9 · p. 13
  • 97.7% of TransVG++ (base) computation comes from LViT, 42.47 times the language branch’s computation.FLOPs were measured with 640×640 images and padded query length 40.Source: §4.5 · pp. 12–13, Table 10 · p. 13
  • 0.92G FLOPs from the language adapters in TransVG++ (tiny) equal 4.99% of the language conditioned vision branch and 3.64% of the whole model.The FLOPs comparison uses 640×640 images and padded query length 40.Source: §4.5 · pp. 12–13, Table 10 · p. 13
  • Multimodal methods for improving parameter and computation efficiency remain an open direction for models intended for resource-limited devices.The authors mention smaller BERT variants, distillation, token pruning, and sub-window attention as possible strategies, while leaving multimodal efficiency for future investigation.Source: §4.5 · p. 13

4.6 Qualitative Result

Five RefCOCOg test examples illustrate the frameworks’ predicted boxes and [REG]-token attention. The authors report interpretable attention and clearer focus from TransVG++ in several examples.

4.6 Qualitative Result

  • Across five RefCOCOg test examples, the visualizations show predicted boxes and [REG]-token attention over vision tokens from the final fusion block.The attention is described as interpretable and corresponding to the referred object’s shape and location.Source: §4.6 · p. 13
  • For expressions with complicated relationships, both TransVG and TransVG++ can successfully model the query, as illustrated by one example.The paper’s example is “A woman see the sea water in back position.”Source: §4.6 · p. 13
  • Compared with TransVG, TransVG++ produces more focused, clearer attention and more correct predicted coordinates in examples (c), (d), and (e).Source: §4.6 · p. 13

5 CONCLUSION

The paper presents TransVG and its advanced version, TransVG++, for visual grounding with Transformers. TransVG++ removes stand-alone fusion modules by integrating language information into the ViT backbone, and the authors characterize it as simple, efficient, and accurate.

5 CONCLUSION

  • TransVG uses a stack of Transformer encoders for multimodal fusion and reasoning instead of complex manually designed fusion modules.Source: §5 · p. 13
  • TransVG++ upgrades to a purely Transformer-based architecture and integrates language referring information into the ViT backbone, removing stand-alone fusion modules.Source: §5 · p. 13
  • The authors characterize TransVG++ as a simple, efficient, and accurate visual-grounding framework with potential for future investigation.Source: §5 · p. 13

Related papers

Loading 2206.06619v1…