Source-linked AI summary
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
TL;DR
Fine-grained visual grounding remains difficult for GPT-4V and other LMMs. The paper introduces Set-of-Mark prompting, which overlays segmented image regions with interpretable marks, and reports strong performance across fine-grained tasks, including a zero-shot RefCOCOg comparison. The method also reveals scope boundaries involving mark placement and model-specific interpretation.
Problem
GPT-4V’s fine-grained visual grounding is limited, while coordinate-based textual outputs can hurt language fluency and underuse vision models’ spatial understanding.
Method
SoM partitions images into semantically meaningful regions and overlays them with alphanumeric marks, masks, or boxes before prompting an LMM.
Results
Across a wide range of grounding-intensive vision tasks, SoM improves GPT-4V; GPT-4V with SoM outperforms a fully fine-tuned referring segmentation model on RefCOCOg in zero-shot evaluation.
Takeaways & Limitations
SoM demonstrates that symbolic visual marks can support GPT-4V’s fine-grained multimodal perception, cognition, and reasoning without task-specific fine-tuning.
Takeaways & Limitations
Mark placement is not universally optimal at region centers, and other tested open-source LMMs could hardly interpret the marks or ground on them.
Abstract
from arXiv · showhide
We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V. As illustrated in Fig. 1 (right), we employ off-the-shelf interactive segmentation models, such as SEEM/SAM, to partition an image into regions at different levels of granularity, and overlay these regions with a set of marks e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can answer the questions that require visual grounding. We perform a comprehensive empirical study to validate the effectiveness of SoM on a wide range of fine-grained vision and multimodal tasks. For example, our experiments show that GPT-4V with SoM in zero-shot setting outperforms the state-of-the-art fully-finetuned referring expression comprehension and segmentation model on RefCOCOg. Code for SoM prompting is made public at: https://github.com/microsoft/SoM.
1 Introduction
GPT-4V has strong general multimodal capabilities but struggles with fine-grained visual grounding. Set-of-Mark prompting addresses this by adding interpretable marks to segmented image regions, improving grounding across vision tasks.
- GPT-4V’s fine-grained visual grounding remains challenging despite its strong vision-language understanding capabilities.
- SoM partitions images into meaningful regions and overlays them with numeric, alphabetic, mask, or box marks.The marked image is then supplied to GPT-4V as a visual prompt.
- SoM enables GPT-4V to associate visual content with corresponding marks without leaking mark information through the conversation.Examples include grounding the right laptop, lamp, and left chair to their marked regions.
- SoM is presented as a simple visual prompting technique that empirically unleashes GPT-4V’s visual grounding ability.
- The authors introduce evaluation benchmarks using segmented regions with overlaid marks to test grounding in GPT-4V and other LMMs.
- GPT-4V with SoM outperforms a state-of-the-art fully fine-tuned referring segmentation model on RefCOCOg in a zero-shot setting.
2 Set-of-Mark Prompting
SoM augments images with semantically meaningful, marked regions that GPT-4V can use for visual grounding. Its pipeline combines flexible segmentation, conflict-aware mark placement, and either plain or interleaved textual prompts.
- 2.1 Problem Definition: SoM augments an input image with visual marks while leaving the LMM’s other inputs unchanged.The method is designed to equip LMMs with location-by-location visual understanding.
- 2.1 Problem Definition: The prompting strategy requires semantically meaningful regions aligned with textual outputs and auxiliary image information that is interpretable and speakable.
- 2.2 Image Partition: The image-partition toolkit combines segmentation models chosen for precise layouts, open-vocabulary recognition, and object-part granularity.MaskDINO supports strong performance, SEEM supports open vocabulary, and SAM/Semantic-SAM support richer granularity.
- 2.2 Image Partition: Users can customize partitioning through automatic or interactive modes, selectable mask granularity, and interactive or user-drawn prompts.The toolkit represents the image as K binary region masks.
- 2.3 Set-of-Mark Generation: SoM selects mark types and locations, using compact alphanumeric marks alongside boxes or mask boundaries while avoiding conflicts with image contents.
- 2.3 Set-of-Mark Generation: The mark allocation algorithm prioritizes smaller regions, excludes overlaps, and uses a distance transform to place marks away from boundaries.This addresses conflicts caused by placing every mark at a region center.
- 2.4 Interleaved Prompt: SoM produces unique region-mark pairs that can support either ordinary plain-text prompts or interleaved prompts containing symbolic mark references.Plain prompts can allow automatic grounding, while interleaved prompts explicitly reference regions through their marks.
3 SoM Prompting for Vision
SoM prompting applies marked image regions to GPT-4V across multiple visual grounding tasks, including segmentation, referring selection, phrase allocation, and video object segmentation. The benchmark uses task-specific region proposals and a limited subset of dataset images or videos.
- Open-Vocabulary Image Segmentation: SoM supports open-vocabulary image segmentation by asking GPT-4V to identify categories for marked regions.
- Referring Segmentation: Referring segmentation selects the top-matched candidate region for a given referring expression.
- Referring Segmentation: Numeric region marks let GPT-4V map descriptions such as “laptop behind the beer bottle” to specific region IDs.
- Phrase Grounding: Phrase grounding allocates corresponding regions to all noun phrases in a complete sentence.
- Video Object Segmentation: SoM can also ground visual objects across video frames when GPT-4V receives a query image and a second image.
- Benchmarks: The evaluation uses task-specific partition tools and small subsets of validation data because GPT-4V access is quota-limited.
4 Experiments
The experiments evaluate zero-shot SoM prompting across multiple grounding and segmentation tasks, comparing GPT-4V with baselines, specialist models, and open-source LMMs. Results show strong performance, while ablations and qualitative observations identify benefits from mark design and limitations from region extraction and annotation quality.
- Experimental Setup: All reported results for SoM are zero-shot, using new chat windows per instance to avoid context leakage.The study uses a divide-and-conquer strategy because GPT-4V was accessed through the ChatGPT interface.
- Experimental Setup: The benchmark evaluates a subset of validation data because GPT-4V quota prevents full validation-set evaluation.Images receive task-specific region proposals with overlaid marks.
- Image Segmentation: GPT-4V with SoM achieves zero-shot segmentation performance close to fine-tuned MaskDINO and much higher than zero-shot OpenSeeD on ADE20K.GPT-4V receives ground-truth masks with numeric IDs and selects class labels for each region.
- Referring Tasks: GPT-4V with SoM outperforms PolyFormer by a large margin on RefCOCOg referring segmentation and beats specialist and open-source LMM baselines for referring comprehension.Direct coordinate prediction performs poorly at 25.7, while SoM enables mask- and box-based referring outputs.
- Phrase Grounding: SoM achieves zero-shot phrase-grounding performance comparable with SOTA GLIPv2 and Grounding DINO on Flickr30K.The pipeline uses Grounding DINO proposals, SAM masks, and boxes, masks, and numbers overlaid on images.
- Video Object Segmentation: 78.8 J&F is the best tracking performance achieved by GPT-4V with SoM compared with other generalist vision models on DAVIS2017.Mask proposals from MaskDINO are overlaid with numeric labels, and the first frame serves as the reference.
- Ablation Study: Adding boxes to numbers and masks significantly improves phrase-grounding performance on Flickr30K.The ablation compares marks consisting of numbers and masks with marks consisting of numbers, masks, and boxes.
- Ablation Study: Ground-truth masks improve RefCOCOg referring segmentation by 14.5 mIoU, while remaining failures are often caused by ambiguous or incorrect annotations.This isolates region-selection errors from errors attributable to GPT-4V.
5 Related Work
Related work frames SoM as visual prompting for LMMs, building on prompting methods for language models, interactive segmentation, and multimodal systems. Unlike prior hand-drawn or example-based approaches, the paper emphasizes systematic region marking and quantitative grounding evaluation.
- LLMs and Prompting: Prompt engineering has been used to elicit capabilities from LLMs through in-context learning, chain-of-thought, and tree-of-thought methods.SoM similarly prompts models to inspect image regions location-by-location but does not require in-context examples.
- Visual and Multimodal Prompting: Interactive segmentation research uses spatial inputs such as clicks, points, boxes, and texts to guide or refine masks.SAM and related models unify several prompting modes, while SEEM extends visual prompting across images.
- LMMs and Prompting: LMM research connects vision models with language models to support multimodal perception and reasoning, including fine-grained vision and video tasks.Recent open-source LMMs have limited capacity for some emerging capabilities, motivating attention to GPT-4V prompting.
- LMMs and Prompting: Prior GPT-4V prompting showed referring capability with hand-drawn circles or arrows but lacked a comprehensive quantitative study.The paper positions SoM as a systematic visual prompting method for evaluating grounding.
6 Discussion
The discussion examines why SoM works especially well with GPT-4V, connects visual and language prompting, and identifies potential for scaling fine-grained multimodal data.
- The mystery in GPT-4V: SoM’s effectiveness remains mysterious because LLaVa-1.5 and MiniGPT-v2 can hardly interpret the marks or ground themselves on them.The authors hypothesize that GPT-4V’s scale and training-data curation may contribute to its extraordinary grounding capability.
- Connecting visual and LLMs prompting: SoM helps connect visual prompting with language-model prompting despite the language bottleneck that makes visual prompts difficult to express precisely in text.The authors describe it as a practical way to inherit the strongest LMM’s capabilities while improving grounding.
- Scaling data via Set-of-Mark Prompting with GPT-4V: SoM with GPT-4V could scale multimodal data containing both fine-grained spatial information and detailed language descriptions.This addresses the continuing difficulty of adding semantic labels to regions despite progress in scaling fine-grained spatial annotations.
7 Conclusion
The paper concludes that overlaying symbolic marks on image regions can unleash GPT-4V’s visual grounding across fine-grained vision tasks. It reports superior performance to fully fine-tuned specialist models and other open-source LMMs, alongside broad qualitative capabilities.
- 7 Conclusion: SoM-prompted GPT-4V is reported as superior to fully-finetuned specialist models and other open-sourced LMMs across a wide range of fine-grained vision tasks.The conclusion also reports fine-grained multimodal perception, cognition, and reasoning capabilities across the board.
A Use Cases
The use cases compare GPT-4V with and without SoM across CAPTCHA solving, dish analysis, food suggestions, tool instructions, gaming, and navigation. These examples illustrate more grounded answers, detailed region references, and practical assistance in varied scenarios.
- General visual grounding: SoM enables GPT-4V to ground reasoning on detailed image contents and make clear cross-image object references.The comparison is made against GPT-4V without SoM.
- CAPTCHA solving: In CAPTCHA solving, SoM helps GPT-4V find the correct squares with corresponding marks, unlike the unprompted model’s incorrect answer.
- Dish analysis: For dish images, SoM lets GPT-4V identify ingredients and associate them with image regions, whereas the original image does not produce a grounded answer.
- Food suggestions: SoM-prompted GPT-4V gives precise suggestions in a food-related case where the original model fails and hallucinates foods such as soft drinks.
- Tool usage instruction: SoM supports thorough controller instructions by teaching the function of each button, including information about buttons that are not labeled.
- Gaming and navigation: SoM-prompted GPT-4V provides reasonable suggestions for achieving goals in gaming and may help embodied navigation locate functionally relevant regions.