PaperlayerView exact revision on arXiv

Source-linked AI summary

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, Hongsheng Li

arXiv:2403.20271v3cs.CV

TL;DR

Whole-image language prompts may not convey which specific regions users intend, and existing visual-prompting methods have limitations in flexibility and image-level perception. Draw-and-Understand equips pre-trained MLLMs with a visual prompt encoder and multi-domain instruction data, and evaluates them with MDVP-Bench. The authors report consistent outperformance on that benchmark while retaining image-level perception.

  • Problem

    Whole-image language interactions can fail to capture users’ intent when they need to identify specific image regions, and existing methods have limitations in flexibility, multi-object reference, and image-level perception.

    Source: §1 · p. 2

  • Method

    Draw-and-Understand adds a visual prompt encoder to pre-trained MLLMs and pairs the architecture with multi-domain instruction data and MDVP-Bench.

    Source: §1 · pp. 2–3

  • Results

    VP-MLLMs consistently outperform other visual-prompting methods on MDVP-Bench, while the paper reports retained image-level visual perception.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Users can interact in their native language and use actions such as clicking and drawing to ask about regions of interest.

    Source: §1 · p. 3

Abstract

from arXiv · show

In this paper, we present the Draw-and-Understand framework, exploring how to integrate visual prompting understanding capabilities into Multimodal Large Language Models (MLLMs). Visual prompts allow users to interact through multi-modal instructions, enhancing the models' interactivity and fine-grained image comprehension. In this framework, we propose a general architecture adaptable to different pre-trained MLLMs, enabling it to recognize various types of visual prompts (such as points, bounding boxes, and free-form shapes) alongside language understanding. Additionally, we introduce MDVP-Instruct-Data, a multi-domain dataset featuring 1.2 million image-visual prompt-text triplets, including natural images, document images, scene text images, mobile/web screenshots, and remote sensing images. Building on this dataset, we introduce MDVP-Bench, a challenging benchmark designed to evaluate a model's ability to understand visual prompting instructions. The experimental results demonstrate that our framework can be easily and effectively applied to various MLLMs, such as SPHINX-X and LLaVA. After training with MDVP-Instruct-Data and image-level instruction datasets, our models exhibit impressive multimodal interaction capabilities and pixel-level understanding, while maintaining their image-level visual perception performance.

1 INTRODUCTION

Whole-image language interactions can miss users’ intended regions, while existing visual-prompting methods have limitations in flexibility, scalability, multi-object reference, and image-level perception. Draw-and-Understand combines a flexible visual-prompt architecture with multi-domain instruction data and a benchmark for evaluating visual-prompt understanding.

1 INTRODUCTION

  • Existing methods can be architecture-specific, depend on fixed references, fail to handle multiple objects, or neglect image-level perception.ROI-based approaches may require external segmentation or ground-truth masks and training from scratch; these constraints can limit flexibility, scalability, and practical applicability.Source: §1 · p. 2
  • Draw-and-Understand adds a visual prompt encoder to a general VP-MLLM architecture, supporting multiple prompt formats and simultaneous prompts while retaining vision-language understanding.The encoder embeds prompt coordinates; the framework is designed to adapt to mainstream pre-trained MLLMs.Source: §1 · p. 2
  • MDVP-Instruct-Data contains approximately 0.8 million images and 1.2 million image-point-text and image-region-text query-answer triplets.The dataset includes object attributes and relationships with surrounding entities and backgrounds.Source: §1 · p. 2
  • Figure 2 highlights OCR and complex-reasoning performance alongside pixel-level object perception and analysis of arbitrarily shaped regions of interest.The caption gives “a girl holding a pen” as an example of pixel-level perception.Source: Figure 2 · p. 3
  • VP-MLLMs consistently outperform other visual-prompting methods on MDVP-Bench, which evaluates point- and region-level captioning, inter-relationship analysis, and complex reasoning.The framework also supports native-language interaction through clicking and drawing to ask about regions of interest.Source: §1 · p. 3

2 MDVP-INSTRUCT-DATA

MDVP-Instruct-Data combines multiple approaches to create visual-prompt instruction examples across image domains. MDVP-Bench is derived from this data through manual filtering and evaluates open-ended visual-prompt understanding tasks.

2 MDVP-INSTRUCT-DATA

  • MDVP-Instruct-Data combines dynamic multi-target captioning with instruction-conversation data reconstructed from grounding QA and generated with GPT-4V.The captioning data repurposes ground-truth boxes as visual prompts and uses natural-language descriptions as answers; conversation data treats grounding boxes as prompt inputs.Source: §2 · pp. 3–4
  • Figure 3 illustrates the dataset’s image-domain distribution and a GPT-assisted example of point- and region-based instruction-following data.It depicts the breadth of image sources alongside an example of the prompt formats used.Source: Figure 3 · p. 4
  • GPT-4V-assisted construction covers natural images, OCR content, documents, webpage and mobile screenshots, remote sensing, and multi-panel images.Its assigned tasks include brief and detailed captioning, inter-relationship analysis, and complex-reasoning generation; Set-of-Marks prompting and object categories help associate generated data with referenced objects.Source: §2 · p. 4
  • MDVP-Bench is created by manually filtering and refining a subset of MDVP-Instruct-Data, covering concise descriptions, elaborate narratives, region interconnections, and complex reasoning.For open-ended evaluation, GPT-4V scores images and textual questions using SoM prompting and LLaVA-bench guidelines, on a 1–10 scale.Source: §2 · pp. 4–5

3 VISUAL PROMPTING MLLM

VP-MLLMs integrate image, visual-prompt, and language representations for joint processing by an LLM. Their training aligns prompt features before instruction fine-tuning, with augmentation to support free-form inputs.

3 VISUAL PROMPTING MLLM

  • The VP-MLLM section introduces an approach for integrating visual-prompt understanding into pre-trained MLLMs and a training strategy for alignment and fine-tuning.The strategy is organized into two stages.Source: §3 · p. 5

3.1 ARCHITECTURE

  • A visual prompt encoder adds prompt representations to the image and language tokens that are concatenated and fed to the LLM.The architecture retains the vision encoder, tokenizer, and LLM used by existing MLLMs.Source: §3.1 · p. 5
  • The encoder embeds point centers and box corners with positional encodings and learnable corner embeddings, then maps prompt types to a shared dimension.A fixed token allotment accommodates a dynamic number of prompts; learned vectors distinguish valid from invalid tokens.Source: §3.1 · p. 6

3.2 TRAINING STRATEGY FOR VP-MLLM

  • Stage 1 freezes the pre-trained vision encoder and LLM while training visual-prompt features to align with image and text features.Segmentation pixels provide point prompts and labels; detection boxes provide box prompts and labels across natural, remote-sensing, document, OCR, and screenshot data.Source: §3.2 · p. 6
  • Stage 2 freezes the vision and visual-prompt encoders, then fine-tunes the visual-prompt projector and LLM on instruction data.The tasks include detailed captioning, inter-relationship analysis, and complex reasoning, alongside image-level instruction data.Source: §3.2 · p. 6
  • Noise-based alignment-stage augmentation simulates free-form prompts, which are converted to bounding boxes at inference.Box noise can extend beyond or partly cover a target, while multiple sampled pixels within a target mask are used to train point prompts.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate visual-prompt models on referring classification, region-level captioning, and broader image- and pixel-level benchmarks, then test key design choices through ablations.

4.1 IMPLEMENTATION DETAILS

  • VP-SPHINX and VP-LLaVA use 7B, 8B, and 13B model sizes and are evaluated zero-shot without dataset-specific fine-tuning.Evaluation uses SPHINX-X and LLaVA-Next-LLaMA3 as the base MLLMs.Source: §4.1 · pp. 6–7

4.2 REFERRING CLASSIFICATION

  • VP-SPHINX and VP-LLaVA outperform state-of-the-art methods on LVIS/PACO Semantic Similarity and S-IOU, and beat ChatSpot by more than 10% on regional COCO-Text OCR.They also show strong object localization and category identification in closed-set binary-choice classification.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 REFERRING REGION-LEVEL CAPTIONING

  • 88.19% accuracy for VP-SPHINX-13B and an 86.67% score for VP-LLaVA-8B; the latter surpasses Osprey-7B by 9.13% on detailed RefCOCOg descriptions.VP-MLLMs also outperform Kosmos-2 and Osprey on brief RefCOCOg region descriptions.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 COMPREHENSIVE ASSESSMENT

  • Across MME, SEED-Bench, POPE, and OCRBench, VP-MLLMs lead other visual-prompt methods but remain slightly below the original MLLMs.Box prompts perform better with fewer prompts per image, while point prompts perform significantly better when images involve more prompts.Source: §4.4 · pp. 8–9
  • VP-MLLMs significantly outperform Osprey and Ferret across multiple domains on MDVP-Bench.The assessment combines different tasks; VP-MLLMs are also competitive with Ferret-v2 on Ferret-Bench referring description and reasoning.Source: §4.4 · pp. 8–9, Table 3 · p. 9

4.5 ABLATION STUDY

  • The visual prompt encoder outperforms alpha-blended prompts and text-specified coordinates, while two-stage training improves all evaluated metrics within the same training duration.GPT-4V-constructed data improves classification and complex reasoning and dialogue; the complete dataset achieves the highest performance.Source: Table 4 · p. 10, §4.5 · p. 10

5 CONCLUSION

Draw-and-Understand equips existing MLLMs with flexible visual prompting while preserving image-level perception. The paper also introduces training and evaluation datasets for visual interaction.

5 CONCLUSION

  • VP-MLLMs support points, boxes, and free-form shapes while preserving original image-level visual perception.Source: §5 · p. 10
  • MDVP comprises 1.2 million image-point-text and image-region-text triplets for training, plus MDVP-Bench for evaluation.Source: §5 · p. 10
  • The authors report superior performance across various visual prompting tasks as evidence of VP-MLLM efficacy and robustness.Source: §5 · p. 10

APPENDIX

  • The appendix identifies separate statistics tables for Stage 1 and Stage 2 training data; the Stage 1 data also sources the GPT-4V-constructed dataset.Source: Table 5 · p. 19, Table 6 · p. 19

A IMPLEMENTATION DETAILS

The implementation retains each pre-trained MLLM’s structure and adapts its image and visual-prompt inputs into a sequence for the LLM.

A.1 MODEL SETTINGS

  • VP-MLLM construction inherits the complete structure of each pre-trained MLLM, with MoV and LLaMA2-7B/13B for VP-SPHINX-X and CLIP-ViT-L-14 and LLaMA3-8B for VP-LLaVA-8B.Source: §A.1 · pp. 19–20
  • Each base MLLM’s Any Resolution strategy processes images through sub-image splitting, padding, and aspect-ratio selection to produce flattened image-token lists.Source: §A.1 · p. 20
  • The visual prompt encoder yields visual-prompt tokens, which are concatenated with image and text tokens for the LLM, with learnable start and end markers around the prompt tokens.Source: §A.1 · p. 20

A.2 TRAINING DATA

Training combines pixel- and region-level prompt tasks with instruction tuning across varied image domains and task types. The related-work discussion notes flexibility and complex reasoning challenges in earlier visual-prompting approaches.

A.2 TRAINING DATA

  • Stage 1 trains the visual prompt encoder and projection layer on detection and segmentation annotations, using boxes, sampled mask points, or pixels as prompts to elicit category labels.Source: §A.2 · p. 20
  • Stage 1 spans natural images, documents, scene text, remote sensing, and mobile or web interfaces, covering categories and elements such as layouts, signage, regions, icons, and search bars.Source: §A.2 · p. 20
  • Stage 2 fine-tuning targets instruction-following and pixel-level understanding, including detailed captioning, relationship analysis, complex reasoning, and question answering.Source: §A.2 · p. 20
  • Training-data construction combines images, perturbed ground-truth bounding boxes, and question-answer text, with prompts drawn from templates or enhanced by GPT-4V.Source: §A.2 · p. 21
  • The appendix identifies Figure 6 as depicting training-data preprocessing and construction and Table 7 as listing VP-MLLM hyperparameters.Source: Figure 6 · p. 21, Table 7 · p. 21

B RELATED WORKS

  • Earlier visual-prompting approaches include region-specific comprehension, color cues, and mask-based instructions, but face flexibility and complex-reasoning challenges, including multi-target referencing and reliance on predefined masks.Source: B RELATED WORKS · p. 22

C MORE EXPERIMENTS

VP-SPHINX-13B achieved the highest reported scores on three VCR evaluation methods. The evaluation tests visual commonsense reasoning with multiple-choice answers and rationales.

C.1 REGION-LEVEL REASONING

  • 88.92%, 90.23%, and 80.65%: VP-SPHINX-13B achieved the highest performance scores across three VCR evaluation methods.Source: §C.1 · p. 22
  • VCR evaluates high-level visual commonsense reasoning through image-based multiple-choice questions that require selecting an answer and providing a rationale.Source: §C.1 · p. 22
  • The models were fine-tuned on the VCR training set, and accuracy was the evaluation metric.Source: §C.1 · p. 22

D MORE DETAILS OF MDVP-INSTRUCT-DATA CONSTRUCTION

This section describes constructing MDVP-Instruct-Data by reconstructing open-source grounding datasets and using GPT-4V to generate instruction data across domains.

D MORE DETAILS OF MDVP-INSTRUCT-DATA CONSTRUCTION

D.1 EXAMPLES OF DYNAMIC MULTI-TARGET CAPTIONING DATA

The appendix illustrates dynamic multi-target captioning by reversing phrase grounding, and describes prompts designed for GPT-4V data generation.

D.1 EXAMPLES OF DYNAMIC MULTI-TARGET CAPTIONING DATA

  • The dynamic multi-target captioning example reverses phrase grounding: the model receives bounding boxes as visual prompts and generates natural-language descriptions.The original task maps referring phrases in a ground-truth sentence to image boxes; Figure 7 illustrates the reversed task with two Flickr30k images.Source: §D.1 · p. 23, Figure 7 · p. 23

D.2 DETAILED PROMPTS FOR GPT4V

  • GPT-4V prompts assign roles and embed category-specific information, using a universal template alongside domain-tailored variants.Tables 9–14 describe the templates, and Table 15 gives an example response.Source: §D.2 · p. 24

E MORE QUALITATIVE RESULTS

The appendix presents additional visual examples of VP-MLLM capabilities across multiple domains and supplementary GPT-4V prompt materials.

E MORE QUALITATIVE RESULTS

  • VP-MLLM examples span natural scenes, mobile and web interfaces, natural-setting OCR, document layouts, and multi-panel images.The paper reports that the model responds accurately to queries in these complex situations.Source: E MORE QUALITATIVE RESULTS · p. 24
  • The supplementary materials include a public GPT-4V prompt template and an example response during data generation.The public template has slight domain-specific variations.Source: Table 9 · p. 25, Table 15 · p. 29
  • Tables 10–14 provide role or role-and-format prompt templates for natural images, screenshots, documents, OCR spotting, and multi-panel images.Source: Table 10 · p. 26, Table 11 · p. 27, Table 12 · p. 27
  • Figure 8 provides additional visual examples of VP-SPHINX-13B across multiple domains.Source: Figure 8 · p. 30

Related papers

Loading 2403.20271v3…