Source-linked AI summary
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, Tiejun Huang
TL;DR
Computer vision lacks a clear general interface for in-context learning because tasks use different output representations. Painter represents task outputs and prompts as images, trains with masked image modeling, and achieves competitive performance across seven tasks while improving over generalist models on challenging tasks.
Problem
Vision tasks vary in output representations, making general-purpose prompts for in-context and out-of-domain learning unclear.
Method
Painter redefines core vision-task outputs and task prompts as images, then trains on stitched input/output pairs with masked image modeling.
Results
Painter achieves competitive performance against task-specific models across seven representative vision tasks and significantly improves over recent generalist models on challenging tasks.
Takeaways & Limitations
Visual signals can serve as task context for in-context inference across diverse in-domain and out-of-domain vision tasks.
Takeaways & Limitations
Panoptic segmentation remains difficult relative to specialized models, and the visual-signal interface is not natural for modeling language signals.
Abstract
from arXiv · showhide
In-context learning, as a new paradigm in NLP, allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. But in computer vision, the difficulties for in-context learning lie in that tasks vary significantly in the output representations, thus it is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. In this work, we present Painter, a generalist model which addresses these obstacles with an "image"-centric solution, that is, to redefine the output of core vision tasks as images, and specify task prompts as also images. With this idea, our training process is extremely simple, which performs standard masked image modeling on the stitch of input and output image pairs. This makes the model capable of performing tasks conditioned on visible image patches. Thus, during inference, we can adopt a pair of input and output images from the same task as the input condition, to indicate which task to perform. Without bells and whistles, our generalist Painter can achieve competitive performance compared to well-established task-specific models, on seven representative vision tasks ranging from high-level visual understanding to low-level image processing. In addition, Painter significantly outperforms recent generalist models on several challenging tasks.
1. Introduction
Painter addresses the difficulty of visual in-context learning by representing both vision-task outputs and task prompts as images. It trains with masked image modeling and uses paired examples to condition inference across diverse tasks.
- Vision in-context learning is difficult because tasks use varied output representations, task-specific losses, and architecture designs.
- Painter treats images as a natural interface for visual perception and reformulates dense-prediction tasks as image inpainting.Given an input image, the model inpaints the desired missing output image.
- The model encodes outputs for depth, keypoints, segmentation, and restoration as 3-channel output images, while task prompts are image pairs.
- Training stitches input/output image pairs from the same task and applies masked image modeling to reconstruct masked output pixels conditioned on the input image.
- At inference, a paired example from the target task indicates which task to perform, supporting flexible in-domain and out-of-domain visual tasks.
- Painter achieves competitive performance against task-specific models across seven vision tasks and state-of-the-art NYUv2 depth estimation, while improving over other generalist models on challenging tasks.
2. Related Work
Related work has explored unified Transformer-based and generalist vision models, but Painter emphasizes continuous image-like outputs to support visual in-context learning.
- Unified Modeling: Transformers have enabled shared modeling modules across language, vision, speech, and multimodal domains when inputs are represented as token sequences.
- Vision Generalist: Vision generalists have unified tasks through Transformer architectures and discrete output spaces, including autoregressive object detection.
- Vision Generalist: Painter represents selected vision-task outputs as continuous images to reduce discretization quantization error and enable masked-image-modeling-based visual in-context learning.
- In-Context Learning: Prior in-context learning work primarily used language sequences as the general interface, including for visual-linguistic tasks.
- In-Context Learning: A concurrent vision approach demonstrated in-context capability for several tasks but predicted outputs in a discrete space.
3. Approach
Painter unifies diverse vision tasks by encoding their outputs as spatially aligned 3-channel images, then trains a masked image modeling framework for visual in-context inference.
- 3.1. Redefining Output Spaces as “Images”: Painter reformulates depth, segmentation, keypoint detection, and restoration as image inpainting by redefining their output spaces as images.
- 3.1. Redefining Output Spaces as “Images”: Each redefined output is an H × W × 3 image whose pixels preserve spatial correspondence with the input while encoding task-specific values in RGB space.
- 3.1. Redefining Output Spaces as “Images”: Depth values are mapped from [0, 10] meters to [0, 255], and inference averages the three output channels before applying the inverse transformation.
- 3.1. Redefining Output Spaces as “Images”: Semantic segmentation represents category labels through three RGB channels using a base and margin, then discretizes predicted pixels to recover categories.
- 3.1. Redefining Output Spaces as “Images”: Keypoint detection separates 17-category classification from class-agnostic localization using colored squares and Gaussian heatmaps.
- 3.1. Redefining Output Spaces as “Images”: Panoptic segmentation combines separately predicted semantic and instance segmentation results, while restoration tasks already use RGB input and output spaces.
- 3.2. A Masked Image Modeling Framework: Training concatenates two augmented same-task image pairs, masks the task-output image, and reconstructs its missing pixels with a standard MIM pipeline.
- 3.3. In-Context Inference: In-context inference concatenates a same-task input/output pair with a new input and masked output, allowing visual examples to specify the task without language instructions.
4. Experiments
Painter is evaluated across seven vision tasks spanning high-level understanding and low-level image processing, with comparisons against generalist and specialized models, training settings, prompt strategies, and out-of-domain generalization.
- Settings: 464 indoor scenes from NYUv2, 25K ADE20K images, COCO training and validation data, and SIDD, LoL, and deraining benchmarks support evaluation across the seven tasks.The experiments report RMSE, absolute mean relative error, and δ thresholds for depth; mIoU for semantic segmentation; OKS-based AP for keypoints; and task-specific restoration benchmarks.
- System-level comparison: Painter achieves competitive performance against task-specific models across seven representative tasks and sets new records for NYUv2 depth estimation.The comparison covers high-level visual understanding and low-level image processing without task-specific design.
- System-level comparison: 7.3 AP separates Painter from Pix2Seq v2 on COCO keypoint detection, while several other tasks achieve performance comparable to well-designed specialized models.The cited comparison also notes remaining room for improvement against specialized models, partly because Painter uses 448 × 448 inputs versus 1024 × 1024 in an example specialized model.
- Joint training vs. separate training: Joint training generally outperforms separate training on most tasks, although it performs slightly worse on keypoint detection.The result suggests cross-task benefits within the unified in-context framework while leaving possible task conflicts unresolved.
- Prompt tuning: Searched and learned prompts perform better than random prompts, while random prompts still work relatively well, indicating prompt optimization benefits alongside model robustness.The learned-prompt approach freezes the model and optimizes learnable prompt tensors using training loss; learned prompt images are visualized separately.
- Generalization: Painter performs trained tasks on inputs whose categories are unseen during training and largely outperforms a concurrent method on the FSS-1000 few-shot segmentation benchmark.Examples include open-vocabulary keypoint detection, object segmentation, and instance segmentation; the quantitative comparison is reported in Table 4.
5. Discussion and Conclusion
Painter explores in-context visual learning by defining context as visual signals and reports competitive performance across seven diverse tasks. The authors identify remaining limitations in panoptic segmentation and language-signal modeling.
- Painter defines visual context for in-context learning and reports competitive performance across seven representative vision tasks.The approach is presented as a vision-centric solution for learning and completing tasks in context.
- Panoptic segmentation remains a difficult task where Painter has substantial room for improvement relative to specialized models.
- Visual signals provide the general interface, but modeling discrete language signals as continuous ones remains an open direction.
- Painter is positioned as an early attempt to let models learn and complete vision tasks in context, with potential for out-of-domain tasks.
A. Additional Implementation Details
The implementation details encode vision-task outputs as images and use task-specific color representations with lightweight post-processing. Additional materials describe dataset statistics, pseudo-code, and the ablation-study organization.
- The appendix provides PyTorch-style pseudo-code for color generation, semantic segmentation post-processing, keypoint detection, and instance-mask labeling.
- Painter’s task outputs use image representations, including RGB color encodings for semantic categories and three-channel outputs for keypoint detection.Semantic segmentation uses black for background and ignored areas; keypoint outputs use heatmaps and category channels.
- Keypoint post-processing converts the three-channel output image into a 17-channel heatmap and then obtains final keypoint locations.The red channel contains class-agnostic heatmaps, while green and blue encode keypoint categories.
- Panoptic segmentation combines semantic predictions with class-agnostic instance masks, then assigns semantic classes by majority voting and removes duplicate predictions with Matrix NMS.
- The supplementary materials include image-restoration dataset statistics and an ADE-20K ablation table organized by patch merging, encoder, head type, and loss function.
B. Additional Results
Additional experiments examine training components and prompt representations. They report gains from the lightweight head, stronger performance from a larger encoder, and task-dependent effects from prompt selection.
- The ablation experiments use a shorter 3k-iteration schedule on ADE-20K semantic segmentation while keeping other hyper-parameters unchanged.
- ViT-L outperforms ViT-B by very large margins in the encoder ablation.The authors suggest generalist models may require more capacity because they use more data with less task-specific prior.
- Painter’s supplementary visualizations include learned prompt pairs whose patterns differ across tasks and can be fed to the model to enable applications.
- The light three-layer head achieves clear gains over a linear-only baseline on ADE-20K semantic segmentation.The head combines linear and convolutional layers with features sampled from transformer blocks.
- Smooth-ℓ1 is Painter’s default regression loss because it achieves the best performance and is more stable during training.
C. Additional Visualization
Painter performs in-context inference on visual tasks that were not included during training, using different prompt images to guide the task.
- Painter visualizes in-context inference for keypoint detection on potato, object segmentation on bee, and instance segmentation on tomato despite not being trained on these tasks.