PaperlayerView exact revision on arXiv

Source-linked AI summary

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr

arXiv:2307.12980v1cs.CV

TL;DR

A systematic overview of prompt engineering on pretrained vision-language models was needed. This survey reviews prompting across VLM types and concludes that prompt engineering is versatile and applicable across them.

  • Problem

    The paper addresses the lack of a comprehensive survey of prompt engineering on pretrained vision-language models.

    Source: §1 · pp. 1–2

  • Method

    The survey organizes prompt engineering on pretrained VLMs, including hard and soft prompt methods and their applications across VLM types.

    Source: §1 · pp. 1–2, §8 · p. 16

  • Results

    The survey finds prompt engineering versatile and applicable across multimodal-to-text, image-text-matching, and text-to-image VLMs.

    Source: §8 · p. 16

  • Takeaways & Limitations

    The paper states that prompt techniques may deliver VLM performance gains without extensive labeled data and may reduce annotation burden and accelerate deployment.

    Source: §8 · p. 16

    Prompt tuning for popular generative multimodal-to-text pretrained models remains largely unexplored.

    Source: §7 · p. 15

Abstract

from arXiv · show

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.

1 INTRODUCTION

Prompt engineering adapts pretrained models through task-specific prompts, but a systematic overview of prompting for pretrained VLMs was lacking. This survey organizes methods and applications across three VLM types and discusses responsible-AI issues and research directions.

1 INTRODUCTION

  • Prompt engineering can adapt pretrained models with few labeled examples and make predictions without updating model parameters.The introduction contrasts this with approaches that require labeled data and task-specific training or fine-tuning.Source: §1 · p. 1
  • The survey addresses the lack of a systematic overview of prompt engineering on pretrained VLMs, despite growing interest in vision-language modeling.Prompt engineering had already been extensively studied in NLP.Source: §1 · p. 1
  • The survey classifies VLM prompting into hard prompts and soft prompts, with hard prompts spanning task instructions, in-context learning, retrieval, and chain-of-thought.Its primary scope retains the model architecture, rather than methods that add modules such as P-tuning or LoRA.Source: §1 · pp. 1–2
  • It covers multimodal-to-text generation, image-text matching, and text-to-image generation, and compares prompting in VLMs with language and vision models.The survey also summarizes model preliminaries, prompting methods, applications, responsible-AI considerations, and future research directions.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors build and release a project page that organizes and lists papers relevant to the survey topic.The page is described as facilitating the literature search.Source: §1 · p. 2

2 TAXONOMY

This section introduces terminology and notation used throughout the survey, including task types and prompting concepts.

2.1 Terminology

  • The survey distinguishes multimodal-to-text generation, image-text matching, and text-to-image generation by the direction and form of their outputs.The first generates text from multimodal input, the second aligns images with text, and the third generates images from text.Source: §2.1 · p. 2
  • Prompt terminology includes prompts and prompting methods, as well as in-context learning and chain-of-thought prompting.In-context learning uses instructions or demonstrations in context without additional training; chain-of-thought prompts for intermediate actions toward a multi-step solution.Source: §2.1 · p. 2

2.2 Notations

  • The paper's mathematical notation is listed in Table 1 and followed throughout unless otherwise specified.The section introduces terms and notations for use across the survey.Source: §2.2 · p. 2, §2 · p. 2

3.1 Preliminaries of Multimodal-to-Text Generation

Generation-based VLMs combine textual and visual representations through a fusion module to produce outputs. The survey describes both early encoder-decoder fusion and decoder-only fusion, including a text-based image representation as a special case.

3.1 Preliminaries of Multimodal-to-Text Generation

  • Generation-based VLMs use text features, visual features, and a fusion module to generate outputs from textual and visual information.The fusion module integrates the modalities into a joint representation and can receive manually designed task prompts.Source: §3.1 · p. 3
  • Text inputs may use tokenized words and special tokens, while images are represented as sequences of embedding vectors.The survey describes BERT-style [CLS] and [SEP] tokens and image tokens that mark modality or sequence boundaries.Source: §3.1 · p. 3
  • Visual features can come from detected object regions, whole-image encoders, or Vision Transformers.The survey notes that region-based features may overlook important image regions, while whole-image encoders provide broader context.Source: §3.1 · p. 3
  • Encoder-decoder fusion creates a joint representation from visual and textual information at an early stage before generation.The fused representation is passed to the generating module for downstream output.Source: §3.1 · p. 3
  • Decoder-only fusion combines visual and textual information during decoding, while PICa represents images as text and uses GPT-3 as its fusion module.BLIP-2 examines decoder-based OPT and encoder-decoder FlanT5 as alternative fusion modules.Source: §3.1 · pp. 3–4

3.2 Multimodal-Text Prompting Methods

The survey classifies prompting methods as hard, human-readable prompts and soft, continuous-vector prompts, then describes their main strategies and scope. Hard prompts include task instructions, in-context learning, retrieval-based prompting, and chain-of-thought prompting; soft prompts include prompt tuning and prefix token tuning.

3.2.1 Hard prompt

  • Task instruction prompting uses explicit task-related instructions to guide model behavior [37].The modified input is formed from the image and text through an instruction function.Source: §3.2.1 · p. 4
  • In-context learning supplies related examples or prompts as context, integrating that context with image and text inputs [7].The resulting input is intended to support coherent, contextually relevant responses [4].Source: §3.2.1 · p. 4
  • Retrieval-based prompting selects relevant prompts or context from a prompt pool or external knowledge base to guide generation or decision-making [16] [38] [39] [40].The retrieval method uses image and text inputs to obtain context.Source: §3.2.1 · p. 4
  • Chain-of-thought prompting iteratively builds prompts, using each response as input to the next step to maintain coherence [4] [9] [5].The prompt sequence progressively adds context or narrows the focus [4].Source: §3.2.1 · p. 4

3.2.2 Soft prompt

  • Soft prompts are continuous vectors fine-tuned with gradient-based methods, unlike manually crafted, interpretable hard-prompt text tokens [6] [41].The survey focuses on methods that do not modify the underlying model, excluding P-tuning [13] and LoRa [14].Source: §3.2.2 · p. 4, §3.2 · p. 4
  • Prompt tuning learns continuous vector input hints, refining them for task performance [6].Its objective minimizes the loss between model output and desired output, conditioned on previous outputs and the modified input.Source: §3.2.2 · p. 4
  • Prefix token tuning inserts task-specific vectors in all model layers, training them independently while freezing the remaining pretrained parameters [42].Prompt tuning and prefix token tuning are distinguished by how tokens are incorporated or attached to the model.Source: §3.2.2 · pp. 4–5, Figure 2 · p. 5

3.2 Multimodal-Text Prompting Methods

  • Prompting methods can be combined; their choice depends on the task, dataset availability, and desired control or customization.Hard prompts are manually crafted discrete text tokens, whereas soft prompts are learnable tensors that are not human-readable.Source: §3.2 · p. 4, §3.2.2 · p. 5

3.3 Advances in Prompting Techniques for VLM

The survey reviews prompting in encoder-decoder and decoder-based fusion models, including handcrafted instructions, prompt tuning, and visual-prefix approaches. Across these models, prompts support tasks ranging from captioning and question answering to multilingual generation, grounding, and multimodal reasoning.

Encoder-decoder fusion models

  • Encoder-decoder fusion models such as VL-T5, SimVLM [29], and OFA [28] use handcrafted instructions and prompt tuning.Their text prefixes specify tasks such as VQA and captioning; visual and location sentinel tokens support structured image-region references [28].Source: §3.3 · p. 5
  • OFA prompt tuning adds tunable embeddings at each layer and is reported as efficient and resilient against adversarial attacks [43].VL-T5 and OFA [28] also use region-oriented prompts for grounded captioning and question answering.Source: §3.3 · p. 5

Decoder-based fusion models

  • Decoder-based fusion models use visual prefixes: Frozen [17] trains a vision encoder with a frozen language model, while BLIP-2 [45] uses Q-Former query embeddings from a frozen image encoder.MAGMA [31] similarly adds an image-prefix encoder while keeping its language model frozen.Source: §3.3 · p. 5
  • Flamingo [1] uses few-shot formatting, retrieval-based example selection [16], and prompt ensembling, while Kosmos-1 [15] explores zero-shot, few-shot, and multimodal chain-of-thought prompting.Kosmos-2 [46] adds grounding and referring capabilities using text spans paired with bounding-box location tokens.Source: §3.3 · pp. 5–6

Other prompting approaches

  • PICa [16] converts images into textual descriptions and prompts GPT-3 with a few examples for visual question answering, without learning visual embeddings.GPT-4 [18] also uses task-specific prompts and in-context learning for complex tasks such as AP Art History.Source: §3.3 · p. 6

3.4 Understanding Prompting

The survey describes how prompt design and training choices shape prompting behavior in multimodal-to-text models. It reports limits on example scaling and prompt length, alongside robustness findings and possible bias from text-only prompts.

Dataset-specific Prefixes; Freezing the Language Model

  • A shared “vqa” prefix lets VL-T5 handle VQA and GQA without dataset-specific prefixes.The survey also notes that some models freeze language models to preserve language capabilities, while OFA [49] and KOSMOS-1 [15] use language-only tasks to address potential loss of language ability during fine-tuning.Source: §3.4 · p. 6

In-context Learning

  • More in-context examples improve performance up to a saturation point, while repeated content can reduce it; Frozen [17] also transfers concept binding in two-way but not five-way settings.Similar example-scaling conclusions are reported for Flamingo [1].Source: §3.4 · p. 6
  • Flamingo [1] and Kosmos-1 [15] report improved performance with individual text prompts rather than image-text pairs, but text-only prompts may introduce bias [1].Source: §3.4 · p. 6

Prompt Tuning

  • Prompt tuning is reported as more robust than fine-tuning across tasks, while longer prompts can help up to diminishing returns and excessive length may hurt performance.Prompt insertion at lower layers might also yield better performance [43].Source: §3.4 · p. 6

3.5 Application of Prompting

The survey describes prompting applications across question answering, commonsense reasoning, classification, captioning, and multimodal chat. The examples include few- and zero-shot settings, region-grounded tasks, and prompts that combine images with text.

Visual Question Answering

  • In-context prompts show results in few-shot and zero-shot VQA, while hard prompts with region tokens support grounded question answering [1] [16] [17] [15] [29] [30] [28].Prompting is also applied to web-page question answering, where models extract answers from page text and questions [15].Source: §3.5 · p. 6

Visual Commonsense Reasoning

  • Kosmos [15] uses zero-shot prompts to ask about object size and color relations in visual commonsense reasoning.Examples include comparing whether one item is larger than another and asking an object's color.Source: §3.5 · p. 6

Zero-shot Image Classification

  • Kosmos [15] prompts zero-shot image classification by completing “The photo of the” with a predicted class, optionally including class descriptions.Source: §3.5 · p. 7

Image Captioning

  • Prompts support few-shot captioning in Flamingo [1] and PaLI [30] and zero-shot captioning in BLIP-2 [32], MAGMA [31], SimVLM [29], and OFA [28].PaLI uses “Generate the alt text in EN” as a captioning prompt [30].Source: §3.5 · p. 7

Chatbot

  • Visual ChatGPT [12] uses a Prompt Manager to format inputs and outputs, translate visual information into language, and handle histories across visual foundation models.GPT-4 [18] accepts interlaced image and text prompts, while BiomedGPT [51] represents domain-specific migration of GPT.Source: §3.5 · p. 7

3.6 Responsible AI Considerations of Prompting

Language-based VLMs inherit risks from their underlying language and vision models, including bias, while researchers also study robustness and prompt-tuning defenses.

3.6 Responsible AI Considerations of Prompting

  • Language-based VLMs inherit risks from their underlying LLMs and vision models.The survey gives gender and racial bias when prompted with images as examples.Source: §3.6 · p. 7
  • Researchers examine VLM robustness to natural distribution shifts and adversarial attacks.Recent work also investigates prompt-tuning robustness to natural shifts and proposes multi-scale image features for robust prompt tuning.Source: §3.6 · p. 7
  • Prompting images can expose gender and racial biases inherited by language-based VLMs.The survey points to these biases as examples of inherited risks from underlying language and vision models.Source: §3.6 · p. 7

4 PROMPTING MODEL IN IMAGE-TEXT MATCHING

The survey reviews image-text matching VLMs and organizes prompting methods across text, visual, and joint branches, then surveys their applications and integrity concerns.

4.1 Preliminary of Image-Text Matching Models

  • Matching-based VLMs learn joint image-text representations by bringing paired embeddings closer and pushing non-pairs apart.These models use contrastive learning and can transfer to zero-shot benchmarks and fine-tuning tasks.Source: §4.1 · p. 7

4.2 Prompting Text Encoder of VLM

  • Text-encoder prompting spans hard prompts and learnable soft prompts, including global, group-specific, and instance-specific variants.Global prompts are shared across task inputs; instance-specific prompts adapt to individual samples, addressing some limits of grouped prompts on unseen classes or samples.Source: §4.2.2 · pp. 7–8, §4.2 · p. 7, §4.2.1 · p. 7

4.3 Prompting Image Encoder of VLM

  • Visual prompting uses learnable patches prepended to images or annotations applied directly to raw images.Examples include color cues, circles, and input-output image pairs used to guide attention or demonstrate a desired task.Source: §4.3 · p. 8

4.4 Unified Prompting on VLM

  • Unified prompting optimizes prompts in both branches, with coupled or decoupled designs.UPT reports unified prompting outperforming unimodal prompting; other approaches combine branch prompts for visual grounding or multimodal understanding.Source: §4.4 · pp. 8–9

4.5 Application of Prompting

  • Prompting supports image and text classification, including zero-shot, long-tailed, and multi-label image classification.For text classification, visual prompts can help leverage visual information.Source: §4.5 · p. 9
  • Detection and dense prediction applications include open-vocabulary object detection, visual relation detection, and semantic segmentation.Methods include learned detection prompts, region-text alignment, prompt-based relation predictions, and pixel-text matching.Source: §4.5 · p. 9
  • Prompt tuning is also applied to domain adaptation, continual learning, and generalization to unseen domains.Examples include prompts learned on unlabeled target data, prompt pools queried per instance, and domain prompts generated from domain similarity.Source: §4.5 · p. 9

4.6 Responsible AI Considerations of Prompting

  • Integrity research covers adversarial robustness, backdoor and poisoning attacks, and fairness and bias in matching-based VLMs.Reported responses include universal visual prompts, CleanCLIP fine-tuning, and approaches that calibrate prompted text or post-process outputs.Source: §4.6 · pp. 9–10

5 PROMPTING MODEL IN TEXT-IMAGE GENERA-

The survey reviews text-to-image prompting from diffusion-model foundations through prompt design, controllable generation, applications, and responsible-AI risks.

5.1 Preliminary of Text-Image Generation Models

  • Diffusion models add noise during training and learn a reverse process to generate images, with prompts supplying conditioning information.The survey describes prompts as fixed or learnable and textual or other-modal inputs within a generation pipeline that includes encoding, latent representation, and decoding.Source: §5.1 · p. 10

5.2 Understanding Prompting

  • Prompt semantics affect generated content: nouns introduce content, while adjectives, artist names, and lighting phrases alter outputs in different ways.Prompt modifiers can also diversify prompts through retrieval, subclasses, or language models, producing varied synthetic images.Source: §5.2 · p. 11
  • Personalized and controllable generation uses learned concept tokens, added conditions such as control maps, and prompt editing through cross-attention.DreamBooth uses unique identifier and subject pairs with a class-prior preservation loss; ControlNet training randomly replaces half of text prompts with empty strings.Source: §5.2 · p. 11

5.3.1 Generating Synthetic Training Data

  • Prompted diffusion models generate synthetic training data for segmentation, object detection, and image recognition.ImaginaryNet generates scene descriptions from class labels and reports improved object detectors, especially when real images and annotations are unavailable.Source: §5.3.1 · pp. 11–12

5.3.2 Generating Data in Target Domain

  • Prompt-based generation extends from images to video, 3D objects and scenes, and motion sequences.Examples include text-guided video generation and editing, DreamFusion and Magic3D text-to-3D methods, and fine-grained or time-varying prompts for motion generation.Source: §5.3.2 · p. 12
  • Complex applications include generating robotics, driving, and floorplan scenes, as well as prompt-centered stories and multimodal procedure plans.The procedure-planning approach captions generated image plans and feeds them back to an LLM to revise the initial plan.Source: §5.3.2 · p. 12, §5.3.3 · p. 13

5.4 Responsible AI Considerations of Prompting

  • Diffusion models are studied both as tools for adversarial examples or purification and as targets vulnerable to prompt attacks.DiffAttack manipulates latent space, DiffPure adds noise before reversing generation, and a five-character text perturbation is reported to shift Stable Diffusion output content.Source: §5.4 · p. 13
  • Text-to-image systems face backdoor, fairness, and privacy concerns, including prompt stealing from generated images.The survey describes training-time triggers that steer inference, bias mitigation during data processing, training, or post-processing, and membership-inference research on privacy risks.Source: §5.4 · p. 13

6 PROMPTING VLM VS. UNI-MODAL MODELS

The survey compares prompt engineering in language and pure vision models, where prompts support diverse tasks through adaptation or integration into model training and inference. It also reviews NLP prompting’s methods and ethical concerns.

6.1 Prompting in Natural Language Processing

  • NLP prompting spans question answering, classification, generation, and information extraction, while reasoning chains and example selection can affect performance and robustness.Complex example questions are reported to benefit robustness to format perturbation and distribution shift.Source: §6.1 · pp. 13–14
  • Prompt automation includes LLM-generated reasoning demonstrations, retrieval of relevant external text, and trainable soft prompts optimized by backpropagation.Retrieval-augmented language models append retrieved text to the input for knowledge-intensive tasks.Source: §6.1 · p. 14
  • For complex language tasks, approaches decompose problems into subtasks through follow-up questions, backward chaining, or task-specific language models.Subtask-specific language models improved performance on a line of textual multi-step reasoning tasks.Source: §6.1 · p. 14
  • NLP prompting research also examines adversarial probing and bias, including prompt-based attacks and generated adversarial examples for dialogue state trackers.The survey notes work reviewing fairness metrics for pretrained language models.Source: §6.1 · p. 14

6.2 Prompting on Pure Vision Models

  • Pure-vision prompting covers classification, segmentation, depth estimation, keypoints, denoising, detailing, and image enhancement, with prompts used for adaptation or in pretraining and inference.PromptCAL is described as discriminative to novel semantic information in generalized novel category discovery.Source: §6.2 · pp. 14–15
  • Visual prompting can adapt frozen vision models: a single prompt requires model access during training, while PGN generates image-dependent prompts from a learned token library.Visual prompting is presented as useful when access to hosted models is limited to inference APIs.Source: §6.2 · p. 14
  • Integrated prompting supports task-directed vision models: SAM maps spatial or text prompts to segmentation masks, while Painter frames dense prediction as image inpainting.Painter uses paired input/output images to indicate the task, including out-of-domain open-category object segmentation.Source: §6.2 · p. 14
  • Ethics and integrity work on pure-vision prompting includes testing adversarial robustness and differential privacy, but such studies remain relatively rare.The survey calls for more attention to this area.Source: §6.2 · p. 15

7 CHALLENGES AND OPPORTUNITIES

The survey identifies open questions in multimodal prompting, including the balance of hard and soft prompts, prompt mechanisms, and links between model types. It also highlights future opportunities in multimodal control and the need to address robustness, privacy, and bias.

Prompting Model in Multimodal-to-Text Generation and Image-Text Matching

  • Multimodal-to-text prompt research primarily uses hard prompts, while soft prompting is extensively studied with image-text matching models; soft prompt tuning for popular generative multimodal-to-text models remains largely unexplored.For multimodal-to-text models, the mechanisms behind in-context learning and instruction tuning, and the contributions of demonstration components, also remain unclear.Source: §7 · p. 15

Prompting Model in Image-Text Matching

  • Visual prompting for pretrained image-text matching encoders remains relatively unexplored, including which visual prompts matter and what semantic information they introduce.The survey points to dense objects, object hallucination, and adaptation to modern VLMs as challenging scenarios.Source: §7 · p. 15

Unified prompting

  • Unified prompting remains understudied, although prompts spanning modalities may provide referential information for visual grounding and cross-modal co-reference dialogues.The survey also identifies heterogeneity in formats, scales, and structures as a challenge when adding modalities such as audio and thermal.Source: §7 · p. 15

Prompting Model in Text-Image Generation

  • Future generation research could incorporate visual, video, object, or motion prompts into image, video, and 3D diffusion models, while using image-text matching for better generation alignment.The survey notes that inconsistencies in text-to-image control maps can lead to errors in generated videos and 3D objects.Source: §7 · p. 15

Responsible AI Considerations of Prompting

  • Responsible prompting remains an open concern: VLM adversarial robustness is not fully understood, and prompt adaptation should manage inherited bias and privacy risks from web-scale pretraining.The survey also calls for transparent, controllable generation and collaboration among society, research institutions, and government to manage risks.Source: §7 · p. 16

8 CONCLUSION

The survey reviews prompt engineering across pre-trained vision-language model types and finds it applicable to multimodal-to-text, image-text matching, and text-to-image generation. It notes limits on its coverage and identifies tracking, interpretability, and robustness as directions for continued work.

Conclusion

  • Prompt engineering applies across multimodal-to-text generation, image-text matching, and text-to-image generation models, with methods considered in light of each model type’s characteristics.The survey presents this versatility as a key finding.Source: §8 · p. 16
  • Prompt engineering may reduce labeled-data demands and accelerate real-world deployment of vision-language models, according to the survey’s stated implications.The survey links these possibilities to performance gains without extensive labeled data.Source: §8 · p. 16
  • The survey is not exhaustive because the field evolves rapidly and the range of existing approaches is broad; it focuses primarily on prompting pre-trained VLMs.It may not cover all recent advances in related areas.Source: §8 · p. 16
  • The authors plan to maintain a tracking platform and identify integration with emerging technologies, interpretability, and robustness as future research directions.They describe interpretability and robustness investigations as important for practical deployment and ethical use.Source: §8 · p. 16

Related papers

Loading 2307.12980v1…