PaperlayerView exact revision on arXiv

Source-linked AI summary

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

arXiv:2109.11797v3cs.CVcs.CL

TL;DR

VL-PTMs have a pre-training–fine-tuning objective gap that leaves visual grounding dependent on large amounts of labeled data. CPT closes this gap by using color-based visual and textual prompts to recast grounding as fill-in-the-blank prediction, and it outperforms fine-tuning in reported zero- and few-shot evaluations.

  • Problem

    A significant objective gap between masked-language pre-training and classification-based fine-tuning creates a need for large labeled datasets for downstream visual grounding.

  • Method

    CPT marks image regions and query text with color-based co-referential prompts, reformulating visual grounding as fill-in-the-blank prediction with a reused masked-language-modeling head.

  • Results

    CPT outperforms fine-tuning in zero- and few-shot settings, including 17.3% absolute accuracy improvement and 73.8% relative standard deviation reduction on average with one shot in RefCOCO.

  • Takeaways & Limitations

    CPT provides a cross-modal prompt-tuning approach for zero-shot, few-shot, and fully supervised visual grounding, with reported gains in sample efficiency and training stability.

  • Takeaways & Limitations

    The color-based framework can suffer from color disturbance, and its visual prompt configuration may be sub-optimal because real-world color appearances differ from standard RGB.

Abstract

from arXiv · show

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/CPT.

1 INTRODUCTION

VL-PTMs face a mismatch between masked-language pre-training and classification-based fine-tuning, which increases labeled-data demands for visual grounding. CPT addresses this mismatch by reformulating grounding as color-based cross-modal fill-in-the-blank prediction and achieves strong few-shot and zero-shot results.

  • Motivation: VL-PTM pre-training recovers masked tokens, whereas downstream fine-tuning classifies unmasked representations with task-specific parameters, creating an objective gap.This gap hinders downstream adaptation and typically requires large amounts of labeled data for visual grounding.
  • CPT framework: CPT marks image regions and query text with color-based co-referential prompts, then recovers the corresponding color text from a masked query token.The framework uses visual blocks or segmentation masks, a color-based textual template, and the reused masked-language-modeling head.
  • CPT framework: CPT introduces a principled search method for high-quality cross-modal prompt configurations, including visual appearances and color texts.
  • Results: 17.3% absolute accuracy improvement and 73.8% relative standard deviation reduction were achieved on average with one shot in RefCOCO using colored blocks.The comparison is against vanilla fine-tuning in the reported evaluation setting.
  • Results: CPT supports strong few-shot and even zero-shot visual grounding, and also applies to position-input tasks such as visual relation detection.

2 PRELIMINARY

Visual grounding is formulated as referring expression comprehension: given an image and referring expression, the task is to locate the corresponding target region. Vanilla fine-tuning detects region proposals and classifies or ranks them using task-specific parameters.

  • Referring expression comprehension: Referring expression comprehension locates the image region corresponding to a query text expression.
  • Vanilla fine-tuning: Vanilla REC commonly detects region proposals and classifies or ranks them to select the target region.The input combines image-region tokens, textual query tokens, and special tokens.
  • Vanilla fine-tuning: Fine-tuning optimizes the target-region representation against negative regions with classification or ranking loss and introduces task-specific parameters.The resulting approach requires many labeled instances to stimulate visual grounding.

3 CROSS-MODAL PROMPT TUNING (CPT)

CPT adapts vision-language models by aligning visual and textual color markers with the masked-language-modeling objective, supporting grounding and visual relation detection with limited labels.

  • 3.1 OVERVIEW: CPT reformulates visual grounding as fill-in-the-blank prediction, using co-referential signals from image regions and text to reduce the pre-training–tuning gap.The framework contains visual and textual sub-prompts and reuses the pre-trained masked language modeling head without adding parameters.
  • 3.2 VISUAL SUB-PROMPT: Each image region proposal receives a unique color marker, implemented as colored blocks or segmentation masks that preserve the model architecture and parameters.Solid blocks and segmentation masks outperform bounding boxes in the reported experiments.
  • 3.3 TEXTUAL SUB-PROMPT: The textual sub-prompt converts a query into a masked template, prompting the model to select the color corresponding to the appropriate image region.The model scores candidate color embeddings using the reused MLM head.
  • 3.4 TRAINING AND INFERENCE: CPT supports zero-shot grounding and can be tuned with few or full labeled datasets using an entropy-based objective.Region batching addresses cases where proposal counts exceed the available color set and where overlapping colored blocks hinder grounding.
  • 3.4 TRAINING AND INFERENCE: CPT addresses two design challenges: selecting color configurations and handling more image regions than the limited pre-trained color vocabulary can represent.These challenges motivate cross-modal prompt search and image region batching.
  • 3.4 TRAINING AND INFERENCE: CPS jointly searches visual appearances and color texts using model decoding scores, improving over naive color choices based on frequent text and standard RGB values.A transparency parameter is applied so the raw content of colored regions remains available to the model.
  • 3.5 CPT FOR VISUAL RELATION DETECTION: For visual relation detection, CPT marks image regions, places an object pair in a query template, and predicts relation text from masked tokens.A variable token count accommodates relations spanning one to three tokens, with a special NA relation for absent relations.
  • 3.5 CPT FOR VISUAL RELATION DETECTION: Visual relation detection extends CPT beyond grounding by producing semantic labels from grounded object pairs for structured image representations and related cross-modal tasks.The method evaluates relation predictions through the task’s relation-text formulation rather than only locating regions.

4 EXPERIMENTS

Experiments evaluate CPT for visual grounding and visual relation detection across zero-shot, few-shot, and fully supervised settings. Results show strong gains over fine-tuning, effects from color selection and representation, and generalization beyond grounding.

  • Experimental Settings: Experiments cover zero-shot, few-shot, and fully supervised visual grounding settings across RefCOCO, RefCOCO+, and RefCOCOg.Few-shot settings use K = 1, 2, 4, 8, 16 training instances, while fully supervised settings use the full training set.
  • Main Results: CPT outperforms random and strong fine-tuning baselines by a large margin in zero-shot and few-shot visual grounding.CPT-Blk achieves 17.3% absolute accuracy improvement on average with one shot in RefCOCO evaluation.
  • Main Results: CPT-Seg outperforms CPT-Blk in zero-shot and few-shot settings, while requiring stronger annotation to train segmentation tools.Solid colors fitting object outlines are described as more natural visual sub-prompts than blocks.
  • Main Results: 73.8% relative standard deviation reduction is achieved by CPT-Blk on average with one shot in RefCOCO evaluation.The reported reduction indicates more stable few-shot training than fine-tuning in that setting.
  • Influence of Colors in CPT’s Visual Grounding: CPS-selected colors achieve higher mean accuracy and lower standard deviation than frequency-based colors across different shot settings.CPS jointly considers visual and textual semantics when adjusting and ranking colors.
  • Influence of Colors in CPT’s Visual Grounding: Individual color performance varies greatly, and optimal colors differ across shot settings; CPS colors are comparable or superior to baseline colors in most cases.Some CPS colors slightly underperform the baseline, potentially because a single textual template biases color adjustment.
  • Case Study: In an 8-shot RefCOCO case study, CPT distinguishes target objects among same-type distractors, but colors in image regions and text can still cause confusion.CPT typically produces reasonable predictions for hard candidates and can avoid outrageous few-shot fine-tuning predictions.
  • Experiments on Visual Relation Detection: With 32 shots for visual relation detection, CPT achieves mR@100 of 37.7%, outperforming fine-tuning by 13.6% absolute points.The experiment uses Visual Genome with 50 visual relations and evaluates zero-shot, few-shot, and fully supervised settings.

5 RELATED WORK

Prior work studies vision-language pre-training, textual prompt tuning, and visual grounding, but CPT extends prompt tuning across both image and text for explicit fine-grained grounding.

  • VL-PTMs use varied pre-training objectives and architectures, including masked language modeling and auto-regressive language modeling.
  • Textual prompt tuning has been applied to VL-PTMs, but these methods leave images unchanged and support only holistic implicit visual grounding.
  • CPT introduces cross-modal prompt tuning for both image and text, enabling explicit grounding of natural-language expressions to fine-grained image regions.
  • Visual grounding is commonly evaluated through referring expression comprehension, which locates an image region corresponding to a query expression.
  • Fully supervised grounding methods require large amounts of costly human-annotated data, while existing zero- and few-shot approaches still require training data for known object types.

6 CONCLUSION AND FUTURE WORK

The paper presents CPT as a cross-modal prompt-tuning framework for VL-PTMs and reports effectiveness across zero-shot, few-shot, and fully supervised visual grounding.

  • CPT is presented as the first cross-modal prompt-tuning framework for VL-PTMs, with a principled search approach for prompt configurations.
  • Experiments demonstrate CPT’s effectiveness in zero-shot, few-shot, and fully supervised visual grounding.
  • Future work will address color disturbance, improve CPT’s computation efficiency, and investigate other vision-language tasks.
  • The paper identifies exploring alternative prompt-tuning approaches for VL-PTMs as future work beyond its color-based framework.

7 ETHICS STATEMENT

The ethics statement addresses intellectual property, privacy, and potential bias in VL-PTMs used by CPT.

  • The adopted code and data are granted for research-purpose usage, raising an intellectual-property protection consideration.
  • The pre-training and tuning data were created by human annotators for research purposes and are stated not to cause privacy issues.
  • VL-PTMs may be biased toward some objects and attributes, motivating ongoing efforts to address this problem.

A.1 RESULTS OF LARGE SIZE VL-PTMS

Large VL-PTM experiments show that CPT generalizes across model sizes, improving and stabilizing zero- and few-shot performance while remaining comparable under full supervision.

  • CPT achieves significantly better and more stable performance than vanilla fine-tuning in zero-shot and few-shot settings for large VL-PTMs.
  • CPT produces comparable results to vanilla fine-tuning in fully supervised settings for large VL-PTMs.
  • The results show that CPT can generalize to VL-PTMs of different sizes.
  • Table 4 reports grounding accuracies across zero-shot, few-shot, and fully supervised settings using means and standard deviations over 5 random splits.

A.2 EFFECT OF COLOR TRANSPARENCY

Color transparency balances color-text alignment against visibility of raw image content. Moderate transparencies perform best, while the preferred transparency increases with more training shots.

  • Moderate color transparencies produce the best performance across different shot settings.Small transparency strengthens color-text and visual-appearance connections but reduces visibility of raw image regions; large transparency has the opposite trade-off.
  • The optimal transparency increases as the number of training shots grows.In low-shot settings, obvious colors support coarse-grained connections; with more shots, fine-grained region understanding becomes more important.
  • CPT-Seg outperforms CPT-Blk in zero-shot and few-shot settings but slightly underperforms it in fully supervised evaluation.The comparison is reported for the main RefCOCO results.
  • CPT-Seg performs better on large objects but worse on small objects than CPT-Blk in the fully supervised setting.The authors hypothesize that imperfect segmentation masks distort object outlines, with errors amplified for small objects during feature pooling.
  • CPT produces reasonable zero-shot predictions, while fully supervised training reduces disturbance from colors in raw images and text.A capable VL-PTM can distinguish colors of varying objects from predefined marker blocks more effectively after supervision.

C IMPLEMENTATION DETAILS

The implementation uses VinVL with region proposals and segmentation masks, trains visual grounding and relation detection with task-specific settings, and searches cross-modal color prompts. Baselines include fine-tuned VL-PTMs and visual grounding models.

  • Implementation: VinVLbase is used as the main backbone, with 768-dimensional hidden representations and 12 encoding layers.VinVL is selected because of its strong performance on many vision-language tasks.
  • Object Detection and Segmentation: Faster-RCNN region proposals and Mask-RCNN segmentation masks from MAttNet are used during CPT training and inference.Both detection and segmentation models are based on ResNet101; Mask-RCNN additionally uses a mask branch.
  • Visual Grounding: Visual grounding labels regions with IoU greater than 0.5 and selects the region with the largest decoding score at inference.Hyperparameters and models are selected by grid search on few-shot or fully supervised validation sets.
  • Visual Relation Detection: Visual relation detection selects hyperparameters by validation-set grid search and uses a learning rate of 3e-5 over 200 steps.The batch size is identical to the shot size.
  • Cross-modal Prompt Search: Cross-modal prompt search matches candidate color text and RGB sets by decoding scores, discarding candidates below 0.8.RGB candidates are searched within ±30 of standard RGB values with step size 5 in each channel.
  • Baselines: The experiments compare CPT with vanilla fine-tuning, other fine-tuned VL-PTMs, MAttNet, and a random baseline.The same MAttNet object proposals are used for all baselines and CPT-Blk for fair comparison.

D.2 DATASET DETAILS

The paper evaluates visual grounding on RefCOCO-family datasets and visual relation detection on Visual Genome, covering object references, object categories, and relation categories.

  • Visual Grounding Datasets: RefCOCO contains 142,210 referring expressions for 50,000 object instances across 19,994 images.Its train, validation, testA, and testB splits contain 120,624, 10,834, 5,657, and 5,095 expression-object pairs, respectively.
  • Visual Grounding Datasets: RefCOCO testA contains only people as target objects, whereas testB contains other object types.This split structure distinguishes target-object categories during evaluation.
  • Visual Relation Detection Datasets: Visual Genome contains 65,651 training images, 5,000 validation images, and 32,422 test images.Each image averages 10.3 objects and 4.8 labeled relation instances, with 150 object categories and 50 relation categories.

E DISCUSSION AND OUTLOOK

The discussion identifies color disturbance and computational inefficiency as CPT limitations, while outlining extensions to object classification and scene graph classification. These extensions could use open-world vocabularies rather than fixed label sets.

  • Limitations: Color disturbance can interfere with CPT because raw-image and text colors may disrupt its color-based semantic bridge.The limitation follows from CPT adding color-based sub-prompts in both modalities.
  • Limitations: CPT can be computationally inefficient because small image-region batches require multiple model evaluations per data instance.The authors use small batches to reduce color disturbance and account for the limited number of color candidates.
  • Outlook: CPT is proposed for object classification by coloring object proposals and prompting VL-PTMs to produce object labels.This extends the object-level visual and textual sub-prompts beyond visual grounding.
  • Outlook: CPT could unify scene graph classification by prompting VL-PTMs to produce object and predicate labels.The proposed framework decomposes textual sub-prompts for tasks involving different subtasks.
  • Outlook: The proposed extensions can produce object and predicate labels from open-world vocabularies instead of fixed label sets.The paper presents this as an advantage in addition to data efficiency.
Loading 2109.11797v3…