PaperlayerView exact revision on arXiv

Source-linked AI summary

Review of Large Vision Models and Visual Prompt Engineering

Jiaqi Wang, Zhengliang Liu, Lin Zhao, Zihao Wu, Chong Ma, Sigang Yu, Haixing Dai, Qiushi Yang, Yiheng Liu, Songyao Zhang, Enze Shi, Yi Pan, Tuo Zhang, Dajiang Zhu, Xiang Li, Xi Jiang, Bao Ge, Yixuan Yuan, Dinggang Shen, Tianming Liu, Shu Zhang

arXiv:2307.00855v1cs.CVcs.AI

TL;DR

Large visual models require effective visual prompts for specific tasks, while prompt-based generalization and multimodal prompting remain active challenges. This review surveys visual prompt designs, large visual models, and prompt-engineering methods, concluding that prompts can enhance downstream performance and generalization while opening future research directions.

  • Problem

    Applying large visual models to specific tasks requires effective approaches for guiding learning and inference, and single-modality prompting can produce suboptimal performance.

  • Method

    The review assesses visual prompt engineering across ViT-based models and AGI models, including automatic, context-aware, and multimodal prompt methods.

  • Results

    The review finds that visual prompts have positive effects on downstream tasks and can enhance the emergent capabilities of large models.

  • Takeaways & Limitations

    Prompt design and modality selection are important across applications, with visual prompts offering potential to improve current models and support novel applications.

  • Takeaways & Limitations

    SAM’s practical efficacy remains questioned in medical image segmentation, camouflage object detection, and mirror and transparent object detection.

Abstract

from arXiv · show

Visual prompt engineering is a fundamental technology in the field of visual and image Artificial General Intelligence, serving as a key component for achieving zero-shot capabilities. As the development of large vision models progresses, the importance of prompt engineering becomes increasingly evident. Designing suitable prompts for specific visual tasks has emerged as a meaningful research direction. This review aims to summarize the methods employed in the computer vision domain for large vision models and visual prompt engineering, exploring the latest advancements in visual prompt engineering. We present influential large models in the visual domain and a range of prompt engineering methods employed on these models. It is our hope that this review provides a comprehensive and systematic description of prompt engineering methods based on large visual models, offering valuable insights for future researchers in their exploration of this field.

1 Introduction

The review frames visual prompt engineering as a way to guide large visual models toward desired outputs and support zero-shot generalization across visual tasks. It surveys visual prompt methods, influential models, applications, challenges, and future directions in computer vision.

  • Motivation: Large visual models build on pre-trained representations learned from massive image datasets to understand image content and extract semantic information.
  • Motivation: Visual prompt engineering designs and optimizes prompts to steer large visual models toward desired outputs for specific tasks.
  • Scope and focus: The review surveys prompt engineering across multimodal visual-language models, visual models, and general artificial intelligence models.
  • Scope and focus: It covers text, image, and text-image prompts, emphasizing that different tasks require distinct prompt characteristics and modalities.
  • Organization: The review examines influential models, downstream applications, prompt design, challenges, opportunities, and future research directions.

2 Background Knowledge

The background explains how prompts adapt pre-trained models to downstream tasks, then introduces Transformer-based visual and multimodal foundations, including CLIP, VPT, and SAM. These approaches use input-side prompts or prompt-based objectives to support efficient adaptation and broad task applicability.

  • 2.1 Prompts in Natural Language Processing: Prompt-based methods add context to inputs to adapt pre-trained models to downstream tasks with parameter-efficient tuning.
  • 2.1 Prompts in Natural Language Processing: Manual prompts are simple but task-specific, experience-dependent, potentially suboptimal, and inefficient for difficult tasks.
  • 2.1 Prompts in Natural Language Processing: Prompts narrow the gap between pre-training and downstream tasks while potentially approaching full parameter fine-tuning performance.
  • 2.2 Foundation Models: Transformers use multi-head self-attention and feed-forward networks, while ViT extends the architecture to computer vision and related models support diverse vision tasks.
  • 2.2 Foundation Models: Transformer tokenization supports multimodal interaction, enabling systems such as CLIP, DALL-E, Flamingo, and GPT-4 for image-text and generative tasks.
  • 2.2 Foundation Models: VPT learns task-specific continuous input vectors and a class head while keeping the pre-trained Transformer parameters fixed.
  • 2.2 Foundation Models: SAM uses prompt engineering and prompt segmentation as a pre-training objective to address broad downstream segmentation tasks on new data.

3 Visual Prompts Learning

Visual prompting adapts pretrained vision and vision-language models to task-specific requirements through manual, continuous, multimodal, visual, and position-guided prompts. These methods aim to improve transfer, efficiency, grounding, and generalization across diverse computer-vision tasks.

  • 3.1 Multi-Modal Models and Prompts: CLIP uses manually crafted prompts linking images and labels, but fixed prompt wording can strongly affect results.Its template is “a photo of a [class].”
  • 3.1 Multi-Modal Models and Prompts: CoOP replaces fixed templates with trainable continuous prompt vectors optimized for downstream task characteristics.A Meta-Net can condition prompts on image-encoder outputs to improve adaptation and generalization.
  • 3.1 Multi-Modal Models and Prompts: DenseCLIP transfers CLIP to dense prediction by using context-aware prompts that convert image-text matching into pixel-text matching.The prompts adapt to task and input context to capture fine-grained image-text semantics.
  • 3.1 Multi-Modal Models and Prompts: MaPLe jointly uses continuous prompts across text and image modalities to exploit their interaction during model training.The approach addresses suboptimal performance from relying on prompts in only one modality.
  • 3.1 Multi-Modal Models and Prompts: Position-Guided Text Prompting reformulates visual grounding as fill-in-the-blank prediction or regression of object-associated blocks.Adding PTP to several visual-language pretraining frameworks improved cross-modal learning and benchmark performance.
  • 3.2 Visual Prompts: Visual prompts include clicks, bounding boxes, scribbles, and support images that guide segmentation or transfer information to related target images.These inputs provide spatial or example-based guidance for identifying object boundaries.
  • 3.2 Visual Prompts: VPT guides fine-tuning of a frozen pretrained backbone by optimizing a small number of trainable input-space prompt parameters.AdaptFormer and Convpass instead add lightweight task-specific modules or convolutional bypasses to improve efficiency.
  • 3.2 Visual Prompts: ViPT uses prompt tuning to adapt pretrained knowledge to multimodal tracking with limited downstream data, while DAM-VP assigns optimized prompts to homogeneous dataset subsets under distribution shifts.Both approaches tailor adaptation without discarding the benefits of pretrained representations.

4 Visual Prompts in AGI

Visual prompts provide an interface for adapting large vision models and foundation models to downstream tasks, including detection, segmentation, multimodal fusion, and model-combination pipelines. The review highlights strong zero-shot transfer in several settings while noting reduced robustness in ambiguous or low-contrast scenarios.

  • 4 Visual Prompts in AGI: Prompt learning lets large computer-vision models adapt to downstream tasks while reducing training demands and resource use.SAM exemplifies zero-shot transfer through appropriately designed prompts.
  • 4 Visual Prompts in AGI: Several universal models use zero-shot transfer, making prompt learning a central mechanism for model generalization.The review names OneFormer, SegGPT, SEEM, and Uni-Perceiver v2 as examples.
  • 4 Visual Prompts in AGI: The review organizes prompt construction around object detection, interactive models, multimodal fusion, and combinations of multiple models.These methods are presented as tools for applying large-model generalization to downstream tasks.
  • 4.1 Object Detection: Point and box prompts are effective across practical scenarios, and SAM shows robust zero-shot performance in natural images, remote sensing, and medical imaging.Generalization may fall short in complex settings with ambiguous semantics or low contrast.
  • 4.1 Object Detection: For object counting, bounding boxes and a 32-by-32 point grid generate masks, which are matched to reference features using cosine similarity above a threshold.Recognized objects are accumulated to obtain the total count.
  • 4.1 Object Detection: Remote-sensing segmentation uses rotated bounding-box guidance because overhead imagery contains objects with arbitrary orientations.The corresponding bounding-box area serves as the mask prompt and supports efficient annotation.
  • 4.1 Object Detection: SAM-Adapter injects specialized domain knowledge and generates task-specific prompts early, including prompts at each Transformer layer.This design targets improved generalization across downstream tasks.
  • 4.2 Multi-modal Fusion: Multimodal fusion combines text, image, and interactive prompts: Text2Seg chains text-conditioned boxes, SAM masks, CLIP Surgery heatmaps, and point prompts.SAMText similarly converts detected text boxes into prompts, while Caption Anything combines SAM with ChatGPT for interactive image captioning.

5 Future Directions and Implications

Future work centers on adapting large vision models to downstream tasks through prompt-based methods while addressing domain breadth, interactive environments, and visual complexity. Applications across medicine, agriculture, and other domains illustrate the practical reach of visual prompts.

  • 5 Future Directions and Implications: Future progress should broaden domain knowledge through diverse datasets, interdisciplinary methods, and collaboration among experts.The review identifies domain-specific knowledge as a constraint on general artificial intelligence performance.
  • 5 Future Directions and Implications: Large vision models require effective adaptation to downstream tasks, with prompt fine-tuning, reinforcement learning, adapter modules, and knowledge distillation offering potential solutions.Prompt fine-tuning guides models with task-specific examples; reinforcement learning uses feedback, adapters modify small functional components, and distillation transfers knowledge to smaller models.
  • 5 Future Directions and Implications: Future visual-model development is more challenging than NLP because computer vision lacks clear interactive environments and faces high-cost, risky human-agent interactions.Virtual environments also raise concerns about transferring trained agents to real-world scenarios.
  • 5 Future Directions and Implications: Visual prompts support applications including medical imaging, agriculture, image editing, object detection, and audio-visual localization.Examples include disease detection and lesion quantification in medicine, plus crop monitoring, pest detection, and yield estimation in agriculture.
  • 5 Future Directions and Implications: Visual prompts provide annotated data, context, and guidance that enhance machine interpretation of visual data across domains.Their integration with large visual models is associated with improved medical diagnosis, image manipulation, and broader application opportunities.

6 Conclusion

The conclusion presents the review as a comprehensive assessment of visual prompt engineering in computer vision, covering ViT-based prompt design, AGI-model applications, and prompt effects on downstream tasks. It emphasizes prompt engineering’s significance across scenarios and its potential to improve models, enable applications, and motivate future research.

  • 6 Conclusion: The review assesses visual prompt engineering methods in computer vision, including prompt design based on ViT architectures and applications using AGI models.It also examines prompt effects on downstream tasks from a model-centric perspective.
  • 6 Conclusion: Prompt engineering is discussed as significant across varied computer-vision scenarios and domains, with reported performance implications and substantial future potential.The article frames prompt engineering as a pivotal area for computer vision research.
  • 6 Conclusion: The paper identifies future research avenues in which prompt engineering may improve current models and enable novel computer-vision applications.The conclusion describes these prospects as relevant to the growing use of prompts across computer-vision fields.
Loading 2307.00855v1…