Source-linked AI summary
Visual Prompting via Image Inpainting
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, Alexei A. Efros
TL;DR
The paper asks whether a pretrained visual model can adapt to novel image-to-image tasks without task-specific fine-tuning or model modification. It constructs visual prompts from task examples and a query, then uses masked inpainting models trained on academic figures to fill the output. The approach works across varied downstream tasks, although it is not competitive with supervised task-specific models and depends on the Computer Vision Figures dataset.
Problem
The paper asks whether one general visual model can perform a wide range of user-specified tasks without fine-tuning or weight modification.
Method
The method concatenates input-output task examples and a query into a grid-like visual prompt, then uses masked inpainting to reconstruct the missing output; models are trained on unlabeled figures.
Results
The authors demonstrate visual prompting across varied image-to-image tasks, with training on the Figures dataset improving most models and MAE-VQGAN outperforming other models for detection and segmentation.
Takeaways & Limitations
Visual prompting provides a test-time way to apply one pretrained inpainting model to multiple image-to-image tasks using task examples.
Takeaways & Limitations
The approach is not competitive with supervised task-specific models and requires training on the Computer Vision Figures dataset; performance is worse for unaligned examples.
Abstract
from arXiv · showhide
How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by prompting in NLP, this paper investigates visual prompting: given input-output image example(s) of a new task at test time and a new input image, the goal is to automatically produce the output image, consistent with the given examples. We show that posing this problem as simple image inpainting - literally just filling in a hole in a concatenated visual prompt image - turns out to be surprisingly effective, provided that the inpainting algorithm has been trained on the right data. We train masked auto-encoders on a new dataset that we curated - 88k unlabeled figures from academic papers sources on Arxiv. We apply visual prompting to these pretrained models and demonstrate results on various downstream image-to-image tasks, including foreground segmentation, single object detection, colorization, edge detection, etc.
1 Introduction
The paper asks whether visual models can perform user-specified image-to-image tasks without task-specific fine-tuning, and answers by recasting task examples and a query as a grid-like inpainting problem. It trains inpainting models on unlabeled academic figures and shows this approach can support varied computer-vision tasks.
- Motivation: The approach targets a single general model that performs user-specified vision tasks without fine-tuning or weight modification.
- Visual prompting: Visual prompting supplies input-output examples and a query image, then asks an inpainting model to fill the masked output consistently with those examples.The examples define the task, while the query specifies a new input.
- Visual prompting: The method requires tasks to be expressible as image-to-image translations, covering a large subset of vision problems.It concatenates examples and the query into one image with a hole rather than imposing an explicit left-to-right analogy.
- Training data: The authors curate 88 thousand unlabeled figures from computer-vision papers and train inpainting models to reconstruct randomly masked patches using other patches from the same figure.Many figures contain image grids pairing inputs with segmentation masks, stylized versions, or edges.
- Contributions: The contributions include a general visual-prompting approach, an unlabeled dataset for learning grid structures, and evidence that additional generic image data further improves results.
2 Related Work
The related work connects visual prompting to natural-image inpainting, self-supervised hole-filling, few-shot learning, image analogies, and NLP prompting. The paper differs by completing grid-like prompts that combine multiple images and by avoiding task-specific architectures or base-class assumptions.
- Natural Image Inpainting: Unlike standard inpainting, this work completes grid-like visual prompts that require reasoning across multiple images within one prompt image.
- Hole-filling as a Pretext task: Hole-filling has been used as a self-supervised pretext task in which a model predicts a randomly removed image region from its context.
- Few-Shot Learning: Unlike few-shot learning, the approach does not assume a large labeled base-class dataset or a task-specific architecture; it uses one or two task examples in a visual prompt.
- Image Analogies: Image analogies infer a transformation from an input-output pair and apply it to a query, whereas visual prompting concatenates task examples and the query into a single image with a hole.
- Prompting in NLP: NLP prompting reformulates tasks as completion from predefined prompts, motivating visual prompting without fine-tuning in computer vision.
3 Visual Prompting via Image Inpainting
This section presents visual prompting as image inpainting: task examples and a query are arranged in a grid-like prompt, then the missing output is synthesized. The approach combines MAE-VQGAN with manually designed prompts, ensembling, and a figures dataset tailored to this visual structure.
- 3.1 Inpainting using MAE-VQGAN: The inpainting function receives an image and binary mask, then synthesizes the masked regions to produce a completed image.The model reconstructs the image from non-masked patches.
- 3.1 Inpainting using MAE-VQGAN: MAE-VQGAN masks image patches and predicts distributions over VQGAN visual tokens, which are decoded into the completed image.It uses a ViT-based masked autoencoder, a fixed ImageNet-pretrained VQGAN codebook, and cross entropy training.
- 3.2 Prompting Inpainting Models: Visual prompts concatenate one or more input-output examples with a new query image, leaving a masked region for the predicted output.The completion must be consistent with the task defined by the examples, and the query portion of the inpainted image is extracted as the result.
- 3.2 Prompting Inpainting Models: The prompt-construction function is manually engineered, typically arranging examples and the query in an (n + 1) × 2 image grid.Different valid constructions can exist for a task, so the work also averages predictions from multiple prompts.
- 3.3 The Computer Vision Figures Dataset: The training data addresses the domain gap between ordinary image datasets and the unnatural grid-like prompts produced by stitching images from different distributions.The authors therefore collect computer-vision figures and infographics from academic articles.
- 3.3 The Computer Vision Figures Dataset: Training randomly samples crops from the curated figures without additional parsing, allowing the model to learn visual grid structures without labels or task descriptions.The figures are unlabeled and contain image grids with corresponding task results such as segmentation masks, stylized versions, and edges.
- 3.3 The Computer Vision Figures Dataset: The Computer Vision Figures dataset contains 88,645 images collected from computer-vision Arxiv sources selected for their grid-like structure.The cited collection covers cs.CV papers from 2010 to 2022; the dataset passage also describes removing unrelated source images.
4 Experiments and Results
The experiments evaluate visual prompting across downstream and synthetic tasks, showing that Figures-dataset pretraining is important and that model choice and prompt design materially affect performance.
- Prompt construction: Visual prompting represents tasks as grid inpainting, with an example input-output pair in the first row and a query image in the bottom-left cell.The model completes the remaining cell to produce the task output.
- Downstream computer vision tasks: Training on the Figures dataset improves results for most models across downstream tasks, while MAE-VQGAN leads detection and segmentation and produces sharper images than MAE.VQGAN struggles with accurate outputs, likely because of sequential decoding, while BEiT is outperformed by MAE, likely because its training is less sample efficient.
- Synthetic data study: Synthetic change-prediction experiments show that models improve with Figures-dataset training but struggle as task combinations become more complex.The synthetic tasks vary shape color, shape, size, or combinations of two changes; evaluation uses color-aware mIOU.
- Dataset effect: Combining Figures with ImageNet performs best, whereas MAE-VQGAN trained only on ImageNet achieves a consistently low 5 points mIOU.This indicates that additional unlabeled natural-image data can benefit the model when combined with Figures training.
- Prompt design and extrapolation: More examples, alternative prompt layouts, and prompt ensembling can improve or stabilize results, while style/content extrapolation remains difficult for new or complex content.The model generates plausible completions under layout and mask-format changes, and attention patterns change with prompt layout.
- Limitations: Visual prompting is a proof of concept rather than a competitor to supervised task-specific models and requires pretraining on the Computer Vision Figures dataset.The authors note that additional natural-image data may provide further benefit.
5 Discussion
The method’s strong performance remains unexplained, despite evidence that its training data matters. The authors suggest that task simplicity and sophisticated long-range co-occurrence learning in inpainting models may contribute.
- The authors do not yet have a good explanation for why the simple method performs well across many visual tasks.
- The specific training data appears important, but it does not fully explain the surprising extent of generalization.
- The discussion raises two possibilities: image-to-image tasks may be simpler than assumed, while inpainting models learn sophisticated long-range co-occurrences.
Supplementary Material
The supplementary material adds visual prompting examples and directs readers to further experimental details and the dataset datasheet.
- Additional visual prompting results are provided in Figure 12.
- The supplement includes more information about the experimental study and the Computer Vision Figures Dataset datasheet.
6 Experiments
Experiments compare visual prompting across inpainting models and tasks, showing smoother segmentation from MAE-VQGAN and sensitivity to prompt examples and postprocessing.
- Qualitative model comparison: MAE-VQGAN produces smoother and more accurate foreground segmentation results than the other compared inpainting models.
- Single Object Detection: Single Object Detection converts raw MAE-VQGAN outputs into bounding boxes using rounding and morphological postprocessing.
- Input-output example effect: Different input-output examples can produce slightly different synthesis results, but meaningful examples still yield plausible outputs.The query is fixed while the examples vary.
- Quantitative evaluation: Visual prompting results on Pascal 5i report mean and standard deviation of mIOU for Foreground Segmentation and Single Object Detection.
7 The Computer Vision Figures Dataset Datasheet
The Computer Vision Figures dataset was created to support visual prompting by providing unlabeled, grid-like figures resembling visual prompts. It contains Arxiv computer-vision figures filtered for natural-image content and accompanied by documented splits, formats, and provenance.
- Purpose: The dataset was proposed to enable visual prompting with grid-like figures stitching together images from different distributions.Its design addresses the mismatch between standard datasets and visual-prompt images.
- Composition: 88,645 unlabeled image instances were collected from Arxiv computer-vision papers and partitioned into 90% training and 10% validation.Each instance is a single Arxiv paper figure.
- Composition: 84% of manually reviewed images were grid-like, while around 40% contained no embedded annotation.The statistics came from manually labeling 100 randomly selected figures.
- Collection: The collection covers Arxiv cs.CV figures from 2010–2022 and retains figures containing at least one natural image.A classifier trained from 2,000 manually tagged images selected informative figure-like sources from 23,302 papers.
- Quality: Manual review found 96% of figures clean of unintended graphs or charts, while potential overlap and imprecise embedded annotations remain acknowledged dataset issues.Overlap checks against Pascal-5i found no duplicates in one 100-image CLIP-based check.
- Data format: Images are PNGs up to 1024 × 1024 with Arxiv paper identifiers and train/validation indicators, and the dataset contains no instance-level labels.The dataset also releases links to source papers and figures.
- Splits: The dataset was randomly split into train and validation, but only the training partition was used for unlabeled-figure pretraining in this paper.The validation partition was reserved for possible future tuning or generative-model evaluation.
Any other comments?
The dataset was automatically extracted from Arxiv sources collected under normal access protocols and selectively filtered for figure-like computer-vision images. Collection involved student researchers without compensation beyond their regular positions.
- Acquisition: The data was directly observable and automatically extracted from Arxiv sources.Sources were downloaded according to Arxiv’s official bulk-data access guidelines.
- Sampling: The sampling strategy restricted the dataset to 2010–2022 cs.CV figures containing at least one natural image.A classifier trained on 2,000 manually tagged images filtered out unrelated graphs and charts.
- Personnel: Students participated in data collection, and none received compensation beyond their regular position-based compensation.
Any other comments?
Preprocessing combined manual tagging, classifier-based filtering, and image preparation, while raw-source links and preprocessing software were intended for release.
- Preprocessing: Manual tagging of 2,000 images trained a binary classifier to retain figure-like images with natural content and remove unrelated graphs or charts.The resulting collection came from 23,302 papers and was manually checked for cleanliness.
- Release: Links to raw figure source files and software for downloading, extracting, and preprocessing images were planned for public release.
Any other comments?
The dataset was intended for unsupervised visual-model pretraining and may support additional generative and retrieval applications. It was planned for public distribution with source links and code under stated licensing arrangements.
- Uses: The dataset has been used for unsupervised learning algorithms, including pretraining inpainting models.
- Uses: Potential future uses include generative modeling, image retrieval, and explainability.
- Risks: The authors do not anticipate negative biases, potential harms, or offensive content in the dataset.This expectation is attributed partly to Arxiv moderation.
- Distribution: The dataset was planned for public availability after the review period, distributed through a CSV of Arxiv links and optionally a fair-use tarball.The CSV file and accompanying code were to use the MIT license.
- Licensing: Distribution relies on Arxiv’s non-exclusive license to distribute submitted papers and sources, with source-link publication described as compliant.
Any other comments?
The dataset is planned to be hosted and maintained on GitHub, with contributions submitted through standard open-source workflows. The passages also state that no dataset updates are currently planned, while contact information will be posted after review.
- The dataset will be hosted and maintained by the authors on GitHub.
- The authors will post contact information after the review period.
- The passages raise questions about errata, retention limits, older-version support, and validation or distribution of contributions without supplying corresponding answers.
- There are no plans to update the dataset at this time.
- Contributions can be submitted as pull requests to the relevant GitHub repository using standard open-source tools.