PaperlayerView exact revision on arXiv

Source-linked AI summary

TransVG: End-to-End Visual Grounding with Transformers

Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, Houqiang Li

arXiv:2104.08541v4cs.CV

TL;DR

Visual grounding must connect a language query to its referred image region, while prior approaches rely on manually structured fusion and indirect candidate prediction. TransVG uses transformer encoders for homogeneous multimodal reasoning and directly regresses box coordinates, setting state-of-the-art records across multiple datasets.

  • Problem

    Prior visual-grounding methods use manually designed fusion mechanisms that can overfit specific scenarios, limit visual-linguistic interaction, and localize indirectly through proposals or anchors.

  • Method

    TransVG uses transformer encoder layers to establish intra- and inter-modality correspondence and directly predicts four box coordinates.

  • Results

    TransVG sets state-of-the-art records on multiple datasets, including 70.73% on ReferItGame, 79.10% on Flickr30K, and 78.35% on RefCOCO.

  • Takeaways & Limitations

    A simple stack of multimodal transformer layers can replace complex fusion modules for visual grounding while supporting direct localization.

Abstract

from arXiv · show

In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-the-art methods, including two-stage or one-stage ones, rely on a complex module with manually-designed mechanisms to perform the query reasoning and multi-modal fusion. However, the involvement of certain mechanisms in fusion module design, such as query decomposition and image scene graph, makes the models easily overfit to datasets with specific scenarios, and limits the plenitudinous interaction between the visual-linguistic context. To avoid this caveat, we propose to establish the multi-modal correspondence by leveraging transformers, and empirically show that the complex fusion modules e.g., modular attention network, dynamic graph, and multi-modal tree) can be replaced by a simple stack of transformer encoder layers with higher performance. Moreover, we re-formulate the visual grounding as a direct coordinates regression problem and avoid making predictions out of a set of candidates i.e., region proposals or anchor boxes). Extensive experiments are conducted on five widely used datasets, and a series of state-of-the-art records are set by our TransVG. We build the benchmark of transformer-based visual grounding framework and make the code available at \url{https://github.com/djiajunustc/TransVG}.

1. Introduction

Visual grounding locates an image region referred to by language, but prior pipelines use manually structured fusion and indirect candidate-based localization. TransVG replaces these with transformer reasoning and direct box-coordinate regression, achieving strong benchmark results.

  • Visual grounding predicts the location of a region referred to by a language expression in an image.
  • Two-stage methods select among sparse region proposals, while one-stage methods score boxes associated with dense anchors.
  • Manually designed fusion structures can overfit query-specific scenarios and limit interaction between visual and linguistic contexts.
  • TransVG uses transformer encoder layers for intra- and inter-modality correspondence and directly outputs four box coordinates instead of selecting candidate boxes.
  • 70.73%, 79.10%, and 78.35% accuracy on ReferItGame, Flickr30K, and RefCOCO correspond to absolute gains of 6.13%, 5.80%, and 6.05%.
  • The paper presents transformer-based visual grounding as a neater framework with improved results across several prevalent benchmarks.

2. Related Work

Related work spans proposal-based and dense one-stage grounding, while transformers have separately advanced language, vision, and vision-language representation learning. TransVG applies transformer encoders specifically to homogeneous multimodal reasoning for visual grounding with limited task data.

  • Two-stage Methods: Two-stage grounding generates region proposals and then selects the best language-matching region.
  • One-stage Methods: One-stage grounding densely fuses linguistic context with visual features and predicts boxes in a sliding-window manner.
  • Transformer: Transformer attention aggregates whole-sequence information with adaptive weights and processes long sequences in parallel.
  • Transformer in Vision Tasks: Vision transformers have been applied to detection, classification, and low-level image processing tasks.
  • Transformer in Vision-Language Tasks: Vision-language pre-training uses transformer encoders for joint image-text representation learning from object proposals and text.
  • Transformer in Vision-Language Tasks: Unlike vision-language pre-training, TransVG develops a visual-grounding framework for homogeneous multimodal reasoning with a small amount of grounding data.

3. Transformers for Visual Grounding

TransVG uses transformer encoders to perform visual, linguistic, and cross-modal reasoning, then directly regresses the referred box coordinates. Its design combines separate visual and linguistic branches, a joint fusion module with a learnable [REG] token, and scale-aware training losses.

  • 3.1. Preliminary: TransVG uses only transformer encoder layers, whose attention supports intra- and inter-modality correspondence without a predefined fusion mechanism.The encoder layers contain multi-head self-attention and feed-forward sublayers with residual connections and layer normalization.
  • 3.2. TransVG Architecture: The visual branch applies a ResNet backbone, reduces the feature-map channels to 256, flattens spatial locations into tokens, and uses a six-layer visual transformer.Sine spatial position encodings preserve the original two-dimensional token positions during visual reasoning.
  • 3.2. TransVG Architecture: The linguistic branch tokenizes the expression with [CLS] and [SEP] markers, then uses a BERT-style transformer to produce linguistic embeddings.The linguistic transformer has 12 encoder layers and output channel dimension 768.
  • 3.2. TransVG Architecture: TransVG projects visual and linguistic embeddings to a common channel dimension, prepends a learnable [REG] token, and jointly processes them with a six-layer visual-linguistic transformer.Attention allows tokens to attend across modalities, and the [REG] state aggregates visual-linguistic context for box prediction.
  • 3.3. Training Objective: Instead of selecting region proposals or anchor boxes, TransVG directly predicts a four-dimensional box vector and trains it with normalized coordinates, smooth L1 loss, and GIoU loss.The GIoU term addresses scale sensitivity, with λ balancing the two losses.

4. Experiments

TransVG is evaluated against established visual-grounding methods on five benchmarks and through ablations of its [REG] token and branch transformers. It reports strong benchmark performance while showing benefits from learnable token initialization and intra-modality transformers.

  • Benchmark Evaluation: Experiments use top-1 accuracy, counting predictions correct when their Jaccard overlap with the ground-truth box exceeds 0.5.The evaluation covers ReferItGame, Flickr30K Entities, RefCOCO, RefCOCO+, and RefCOCOg.
  • Benchmark Evaluation: The reported results support replacing complicated multi-modal fusion modules with a simple stack of transformer encoder layers.The authors connect this finding to unified intra- and inter-modality context reasoning in TransVG.
  • Benchmark Evaluation: TransVG achieves 79.10% top-1 accuracy on Flickr30K Entities, a 5.80% absolute improvement over the previous state-of-the-art record.The comparison uses a ResNet-101 backbone.
  • Benchmark Evaluation: TransVG reaches 78.35% on the RefCOCO testB set, improving 6.05% absolutely over the previous state-of-the-art result.It also achieves the best performance on all RefCOCO and RefCOCOg subsets and splits reported in the passage.
  • Ablation Study: A learnable [REG] token embedding achieves 80.32% top-1 accuracy on the RefCOCO validation set, outperforming the alternative initialization designs.The authors characterize this initialization as more equitable and flexible than deriving the token from one modality.
  • Ablation Study: On ReferItGame, accuracy rises from 64.24% without branch transformers to 69.76% with both visual and linguistic transformers.Using only the visual transformer reaches 68.48%, while using only the linguistic transformer reaches 66.78%.
  • Qualitative Results: Qualitative examples show TransVG handling complicated relationships and producing [REG] attention corresponding to the referred object’s shape and location.The examples come from the RefCOCOg test set.
  • Qualitative Results: Across visual-linguistic transformer layers, [REG] attention shifts from broad image context to discriminative regions and finally toward the referred object.The final-layer attention more accurately reflects object shape and supports correct coordinate regression.

5. Conclusion

TransVG uses transformer encoders instead of complex manually designed fusion modules for visual grounding. Experiments show effective fusion and reasoning, with new state-of-the-art records across multiple datasets.

  • TransVG performs visual grounding with a simple stack of transformer encoders for multi-modal fusion and reasoning.
  • Its multi-modal transformer layers perform step-by-step fusion and reasoning.
  • TransVG sets new state-of-the-art records on multiple datasets.
Loading 2104.08541v4…