Source-linked AI summary
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Aleksandar Shtedritski, Christian Rupprecht, Andrea Vedaldi
TL;DR
The paper addresses the limited use of VLM prompting for discriminative vision tasks beyond classification by engineering inputs in image space. It uses visual markings, especially red circles, to guide CLIP while preserving context, achieving strong zero-shot localization results and revealing undesirable biases. The method is not appropriate for sensitive-data analysis because marking can elicit harmful behaviors.
Problem
VLM prompting has focused largely on text, although visual inputs can better express geometric properties such as location.
Method
The paper overlays humanlike visual markers, especially red circles, on candidate regions and uses CLIP to select regions matching textual descriptions.
Results
The method achieves state-of-the-art zero-shot referring expression comprehension, outperforms cropping in some prediction tasks, and supports finer-grained keypoint localization.
Takeaways & Limitations
Visual prompt engineering extracts useful zero-shot behavior from CLIP while preserving contextual information that cropping loses.
Takeaways & Limitations
The authors caution that marking should not be used to analyze sensitive data because red circles can elicit unwanted behaviors and negative connotations.
Abstract
from arXiv · showhide
Large-scale Vision-Language Models, such as CLIP, learn powerful image-text representations that have found numerous applications, from zero-shot classification to text-to-image generation. Despite that, their capabilities for solving novel discriminative tasks via prompting fall behind those of large language models, such as GPT-3. Here we explore the idea of visual prompt engineering for solving computer vision tasks beyond classification by editing in image space instead of text. In particular, we discover an emergent ability of CLIP, where, by simply drawing a red circle around an object, we can direct the model's attention to that region, while also maintaining global information. We show the power of this simple approach by achieving state-of-the-art in zero-shot referring expressions comprehension and strong performance in keypoint localization tasks. Finally, we draw attention to some potential ethical concerns of large language-vision models.
1. Introduction
The paper introduces visual prompt engineering for VLMs, using image-space markings to steer CLIP toward regions while preserving context. It demonstrates useful zero-shot capabilities alongside emergent behaviors linked to rare training examples, large model capacity, and ethical risks.
- Visual prompt engineering manipulates the visual modality rather than textual prompts, which is useful for expressing geometric properties such as location.
- Marking achieves state-of-the-art zero-shot referring expression comprehension and outperforms cropping-based prompt engineering in some prediction tasks.
- Drawing a plain red circle steers CLIP to analyze the enclosed image region while retaining global information for object, part, and region localization.
- Red circles are rare in YFCC15M, yet only larger VLMs exhibit the associated behavior reliably, consistent with learning from rare examples through high capacity.
- The paper identifies undesirable behavior: adding a red circle increases the probability that CLIP characterizes a person as a criminal or missing person.
2. Related work
Related work situates the paper among prompting methods for VLMs, referring expression comprehension, visual reasoning, and bias assessment. Existing approaches include learnable-token prompting, pixel-space optimization, cropping, coloring, and specialized models.
- VLM prompting commonly prepends learnable tokens to text, vision, or both inputs, while other methods optimize pixel-space augmentations or use visual prompting for inpainting.
- Referring expression comprehension localizes an object matching a textual description, typically by scoring object proposals or predicting a bounding box.
- Unsupervised referring expression comprehension became feasible with large pretrained models such as CLIP, with prior methods using cropping, colored proposals, or pseudo-descriptions.
- CLIP has supported visual reasoning tasks including classification, OCR, semantic segmentation, and open-vocabulary segmentation.
- Prior VLM bias studies measure misclassification across racial groups or fairness in retrieval, whereas this work examines bias induced by adding a visual marker.
3. Method
The method recasts zero-shot VLM inference as compatibility-based question answering, engineering textual and visual prompts for task-specific questions and answers. It compares cropping with marking, then applies these prompts to keypoint matching, localization, and referring-expression comprehension.
- Prompt engineering: VLMs score image-text compatibility, so prompt engineering transforms an input image-text pair into a question-answer compatibility score.The transformed image and text encode the question-answer pair while preserving the VLM's compatibility-scoring interface.
- Prompt engineering: Text prompts encode semantic questions, while visual prompts encode geometric answers such as image locations.For keypoint localization, the keypoint name is expressed textually and its 2D location visually.
- Visual prompting via marking: Visual marking overlays a circle, box, or arrow on the desired region, preserving contextual information that cropping removes.The paper uses marking as an alternative to cropping for indicating a location in the image.
- Naming keypoints: Keypoint matching scores every name-location pair using cropped or marked images and decodes the cost matrix into a permutation with optimal transport.The Sinkhorn-Knopp algorithm efficiently solves the temperature-weighted optimization used to normalize the matching matrix.
- Keypoint localization: Keypoint localization searches a salient-region regular grid, selecting the location with the highest compatibility score for the named keypoint.Unlike naming keypoints, this task does not assume prior knowledge of possible keypoint locations.
- Referring expression comprehension: Referring-expression comprehension extracts object proposals as candidate answers and uses referring expressions as questions, with average-question subtraction reducing responses to universally salient hypotheses.The score adjustment downweights proposals such as faces that respond strongly to many questions.
4. Experiments
The experiments evaluate visual marking across keypoint naming, keypoint localization, referring expression comprehension, marker design, model scale, and potential bias. Red-circle marking outperforms crop-based or other-marker baselines while retaining a documented ethical risk in sensitive applications.
- Naming Keypoints: Red-circle visual marking significantly outperforms random and crop-based baselines for naming keypoints, achieving almost twice the accuracy.The Sinkhorn-Knopp normalization further improves matching, especially for ambiguous nearby points such as mouth and nose.
- Marker Ablations: Red circles perform best among tested marker shapes, sizes, and colors, while crosses perform worst.Red remains the best color despite being common in natural images; the authors attribute this to human-centric annotations in training data.
- Training-Data Analysis: The marker detector finds human-annotated images containing red circles in YFCC15M, supporting the hypothesis that CLIP encountered similar annotations during pretraining.The study trains a binary classifier, filters a 6M-image subset, and manually examines the top 10k images.
- Model Scaling: Keypoint-matching performance improves with pretraining-dataset size and vision-encoder size, although LAION-2B produces worse results than the compared WIT-400M setting.The authors suspect filtering differences may have removed annotation examples from LAION-2B.
- Model Scaling: Larger VLMs exhibit red-circle behavior more reliably, and the improvement over smaller models is much larger for zero-shot keypoint matching than for ImageNet classification.The reported improvements are 250% for zero-shot keypoint matching and 20% for zero-shot ImageNet-1K classification.
- Referring Expression Comprehension: For referring expression comprehension, red-circle marking achieves state-of-the-art results on most zero-shot baselines and outperforms crop-based and explicitly trained alternatives on most benchmarks.The evaluation uses an ensemble of RN50x16 and ViT-L/14@336 CLIP backbones to score MAttNet bounding-box proposals.
5. Conclusions
Visual prompt engineering via marking extracts useful zero-shot behavior from CLIP, achieving strong referring-expression performance while revealing both rare training cues and undesirable associations.
- Visual marking achieves state-of-the-art zero-shot referring expression comprehension and significantly outperforms traditional image cropping.
- The behavior appears to emerge from rare marking examples in the VLMs’ training data, requiring very large models trained on very large datasets.
- Adding a red circle can increase CLIP’s belief that an image has a negative connotation.
Appendix
The supplementary material provides additional dataset details, implementation information, ablations, and qualitative and quantitative evaluations.
- The supplementary material expands the paper with dataset details, implementation details, ablations, and further qualitative and quantitative evaluations.
6. Datasets
The appendix adds named keypoint annotations to Spair71k and resolves ambiguous or redundant animal keypoints through consistent selection rules.
- The authors manually name animal keypoints in Spair71k, whose original annotations lack keypoint names.
- For nostrils, the right nostril is labeled nose and the left nostril is omitted.
- For tails, the point not attached to the body is labeled tail, while the attached point is omitted.
- For ears, the point not attached to the head is labeled ear, while the attached point is omitted.
- For birds, only the foot annotation is retained because foot, ankle, and knee points are often ambiguous and close together.
- Different names are explicitly assigned to potentially ambiguous keypoints so the questions and answers in Section 3.3 have distinct roles.
7. Discovered annotations
Among discovered YFCC-15M annotations, circles and red markers are common, with red circles forming a substantial subset.
- 44% of discovered YFCC-15M annotations contain red circles.
- 73% of the annotations are circles, while 65% are red.
8. Additional implementation details
The method uses selected CLIP backbones and red-circle annotations, with blur and grayscale variants ensembled for referring-expression detection. Keypoint localization uses image-relative circle dimensions and fixed query sampling.
- Backbone: RN50x16 and ViT-L/14@336 form the selected backbone ensemble for the method.This combination was chosen because it led to the highest performance among evaluated backbone combinations.
- Annotations: The annotation ensemble contains three images per annotation: a red circle, a red circle with outside blur, and a red circle with outside grayscale.The augmentations were inspired by annotated examples found in YFCC15M, and adding them improved overall results.
- Annotations: A thin red circle performs best on average among tested marker shapes, sizes, and colours.The implementation uses the red circle together with blurred-outside and gray-scaled-outside augmentations.
- Keypoint localization: For keypoint localization, circles use radius r = 0.06H and thickness t = 0.01H, where H is the image’s shorter side.With H = 336px, these settings become r = 20px and t = 3px.
- Keypoint localization: Keypoint localization samples M = 30, producing 900 query locations before applying the pseudo mask.The task uses templates naming bird parts for CUB and animal parts for the other setting.
9. Qualitative evaluations
The qualitative evaluation covers naming keypoints, keypoint localization, and referring-expressions comprehension through the figures cited for these tasks. The reported Referring Expressions table is organized by dataset split and backbone method.
- Keypoint localization: Qualitative keypoint-localization results are presented in Figures 11 and 12.The evaluation includes localization examples across the reported keypoint task settings.
- Referring expressions comprehension: Qualitative referring-expressions comprehension results are presented in Figure 13, while the accompanying table lists RefCOCO, RefCOCO+, and RefCOCOg splits by method and backbone.The table includes validation and test splits where listed.
Red Circle
The red-circle design is evaluated across backbone choices, annotation variants, and qualitative task examples. The figures and ablation tables specify marker encodings, dataset settings, and comparison conditions.
- Referring expressions: Figure 9 shows four referring-expression annotation types: the original box, red circle, red circle with outside blur, and red circle with outside grayscale.The experiments use an ensemble of the last three types unless otherwise stated.
- Keypoint tasks: Table 8 lists SPair71k keypoint part names and numbers, including inconsistent original annotation numbering for parts such as the horse’s tail.The horse tail is part 10, whereas it is part 9 for other animal classes.
- Referring expressions: Table 9 ablates mean subtraction against negative queries and ensembling across red-circle, blur-outside, and gray-outside markers.The ablation uses RN50×16 and ViT-L/14@336px with the optimal circle size from Table 10.
- Marker ablation: Table 10 compares marker sizes, shapes, and colours without ensembling or mean subtraction, isolating the effectiveness of the markers themselves.A unit size equals 0.5% of the image’s larger side, or 1 pixel for a 224-pixel image.
- Marker ablation: Figure 10 varies circle diameter and thickness on CUB for text-to-image matching; the best annotation has radius 12px and thickness 4px.Scatter-plot dot colours identify red, green, blue, purple, yellow, and cyan annotations.
- Keypoint localization: Figure 11 and Figure 12 show SPair71k keypoint localization, marking correct predictions in green and wrong predictions with red dashed coding under PCK with α = 0.1.The red circle marks the model’s queried location and the blue dot marks ground truth.
- Referring expressions: Figure 13 presents RefCOCOg qualitative results with correct predictions on the left and wrong predictions on the right.Its last right-hand row includes an example where the ground-truth bounding box is wrong.
- Naming keypoints: Figures 14 and 15 provide normalized cost matrices for naming keypoints on CUB and SPair71k, respectively.The figures document qualitative keypoint-naming evaluations for the two datasets.