Source-linked AI summary
Unleashing Text-to-Image Diffusion Models for Visual Perception
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, Jiwen Lu
TL;DR
Visual perception needs transferable semantic knowledge, but text-to-image diffusion models use a generation-oriented pipeline that differs from conventional perception pre-training. VPD repurposes a pre-trained text-to-image diffusion model as a prompted backbone with adapted text features and cross-attention guidance. Across three perception tasks, it achieves competitive performance and faster convergence, while its computational cost remains relatively high.
Problem
The paper asks how to transfer visual knowledge from text-to-image diffusion models to downstream visual perception despite their pipeline and architectural differences from conventional visual pre-training.
Method
VPD uses a pre-trained diffusion UNet as a backbone, prompts it with task-specific text, refines text features with an adapter, and uses cross-attention maps for guidance.
Results
Across semantic segmentation, referring image segmentation, and depth estimation, VPD achieves competitive performance and faster adaptation, including 73.3% oIoU on RefCOCO and 0.254 RMSE on NYUv2.
Takeaways & Limitations
The experiments support using generative text-to-image pre-training for visual perception and suggest a step toward unifying visual generation and perception.
Takeaways & Limitations
VPD currently has relatively high computational cost because the generative model prioritizes synthesis quality over complexity.
Abstract
from arXiv · showhide
Diffusion models (DMs) have become the new trend of generative models and have demonstrated a powerful ability of conditional synthesis. Among those, text-to-image diffusion models pre-trained on large-scale image-text pairs are highly controllable by customizable prompts. Unlike the unconditional generative models that focus on low-level attributes and details, text-to-image diffusion models contain more high-level knowledge thanks to the vision-language pre-training. In this paper, we propose VPD (Visual Perception with a pre-trained Diffusion model), a new framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Instead of using the pre-trained denoising autoencoder in a diffusion-based pipeline, we simply use it as a backbone and aim to study how to take full advantage of the learned knowledge. Specifically, we prompt the denoising decoder with proper textual inputs and refine the text features with an adapter, leading to a better alignment to the pre-trained stage and making the visual contents interact with the text prompts. We also propose to utilize the cross-attention maps between the visual features and the text features to provide explicit guidance. Compared with other pre-training methods, we show that vision-language pre-trained diffusion models can be faster adapted to downstream visual perception tasks using the proposed VPD. Extensive experiments on semantic segmentation, referring image segmentation and depth estimation demonstrates the effectiveness of our method. Notably, VPD attains 0.254 RMSE on NYUv2 depth estimation and 73.3% oIoU on RefCOCO-val referring image segmentation, establishing new records on these two benchmarks. Code is available at https://github.com/wl-zhao/VPD
1. Introduction
The paper asks whether visual knowledge learned by large text-to-image diffusion models can support visual perception, despite differences between generative diffusion pipelines and conventional perception models. VPD adapts a pre-trained diffusion model for multiple perception tasks and achieves strong benchmark results with rapid adaptation.
- Motivation: Text-to-image diffusion models learn low-level image knowledge and high-level visual-linguistic relations from large-scale image-text pairs.Their generation process captures textures, edges, structures, and semantic relations despite differing from conventional visual pre-training.
- VPD Framework: VPD uses a pre-trained diffusion model as a backbone, directly processes natural images, and performs a single prompted denoising step to extract semantic information.Features from multiple UNet decoder hierarchies form visual representations for downstream tasks.
- Evaluation: VPD evaluates semantic segmentation, referring image segmentation, and depth estimation, spanning high-level concepts, vision-language modeling, and low-level structural understanding.These tasks cover complementary visual perception requirements.
- Results: 73.3% oIoU on RefCOCO and 0.254 RMSE on NYUv2 establish new state-of-the-art results on referring image segmentation and depth estimation.With a lightweight Semantic FPN decoder, VPD also achieves 54.6% mIoU on ADE20K.
- Results: 44.7% mIoU on ADE20K after only 4K training iterations outperforms existing pre-training methods.The result supports rapid adaptation on a challenging semantic segmentation benchmark.
2. Related Work
Related work covers diffusion models, diffusion-based perception, and conventional visual pre-training. VPD instead uses a text-to-image denoising autoencoder as a backbone to transfer its learned visual-text knowledge to perception tasks.
- Diffusion Models: Diffusion models generate images through progressive denoising of a learned reverse process, while latent diffusion performs this process in a lower-resolution latent space.Cross-attention provides a generic mechanism for adding conditions such as text.
- VPD: VPD studies how to extract high-level knowledge from text-to-image diffusion models while leveraging their visual-language correspondence through implicit and explicit mechanisms.The framework is presented as an alternative way to connect generative pre-training with visual perception.
- Diffusion for Perception: Text-to-image diffusion models exhibit visual-text correlations that motivate exploiting their pre-trained knowledge for downstream visual perception.This direction differs from prior frameworks that reformulate perception as progressive denoising.
- Visual Pre-training: Visual pre-training commonly uses supervised classification, contrastive learning, or masked image modeling to learn transferable representations for downstream perception.These approaches support perception through a pre-training and fine-tuning paradigm.
3. Method
VPD adapts a pre-trained text-to-image diffusion model for visual perception by using it as a backbone, prompting it with task labels, and exploiting vision-language interactions. The framework combines refined text features, hierarchical visual features, and cross-attention guidance with lightweight task-specific prediction heads.
- Framework: The prediction model connects task labels to natural language by conditioning visual feature extraction on text features derived from task-specific category names.This decomposes prediction into text-feature extraction, hierarchical feature extraction, and a task-specific prediction head.
- Text prompting: A two-layer MLP text adapter refines CLIP features while a small learnable scale factor helps preserve pre-trained text-encoder knowledge and mitigate the downstream domain gap.Text inputs use the template “a photo of a [CLS]”, and the feature from the [EOS] token is used.
- Framework: VPD uses a pre-trained diffusion model as a backbone for visual perception rather than reformulating perception as progressive denoising.Only a single UNet is used for the backbone, requiring a single forward pass instead of multiple timestep evaluations.
- Semantic guidance: Cross-attention maps provide explicit semantic guidance by being averaged across channels, concatenated with hierarchical feature maps, and passed to the prediction head.The lowest-resolution cross-attention maps are omitted by default because they are less accurate.
- Task applications: VPD applies a similar architecture to semantic segmentation, referring image segmentation, and depth estimation, with task-specific conditioning inputs.The conditioning set contains dataset class names for semantic segmentation, while other tasks use different procedures.
4. Experiments
VPD is evaluated across semantic segmentation, referring image segmentation, and depth estimation, including accelerated training settings and component analyses. It achieves strong downstream performance, adapts quickly, benefits from cross-attention guidance and stronger diffusion pre-training, but remains computationally costly.
- VPD is evaluated on semantic segmentation, referring image segmentation, and depth estimation, covering both high-level and low-level visual perception.
- Semantic Segmentation: 53.7 mIoUss and 54.6 mIoUms are achieved on semantic segmentation, outperforming ConvNeXt-XL with comparable computational complexity.VPD uses a lightweight Semantic FPN, 512×512 crops, and 80K iterations, whereas other methods use UPerNet and more than 120K iterations.
- Semantic Segmentation: VPD surpasses all baseline methods at 8K iterations with VPDA32 and yields better results at 4K iterations with VPDA64.The results indicate faster downstream adaptation, while additional cross-attention guidance can expedite convergence further.
- Referring Image Segmentation: VPD achieves better overall IoU consistently than previous methods across RefCOCO, RefCOCO+, and G-Ref, including a one-epoch training schedule.The evaluation reports overall IoU on validation sets under both default and fast schedules.
- Depth Estimation: 0.254 RMSE on NYUv2 establishes the state of the art, while one-epoch training reaches 0.349 RMSE versus 0.381 for SwinV2-L.Lower RMSE is better, and the comparison demonstrates faster convergence against masked-image-modeling pre-training.
- Analysis: Ablations show that prompts, the text adapter, and cross-attention guidance each improve performance, while combining upsampling and downsampling maps works best.The text adapter improves mIoU from 42.0 to 42.9 at 4K and from 47.1 to 48.0 at 8K; cross-attention adds 0.7 mIoU at 8K.
- Analysis: Longer Stable-Diffusion pre-training improves downstream performance, indicating that VPD benefits from stronger text-to-image diffusion models.
- Limitations: VPD currently has relatively high computational cost because diffusion generative models prioritize synthesis quality over complexity.
5. Conclusion
VPD transfers high-level knowledge from pre-trained text-to-image diffusion models to downstream visual perception tasks. Across extensive experiments, it achieves competitive performance and faster convergence than methods using varied visual pre-training paradigms.
- VPD transfers high-level knowledge from a pre-trained text-to-image diffusion model to downstream visual perception tasks.
- The framework uses designs that encourage visual-language alignment and implicitly and explicitly prompt the pre-trained model.
- VPD achieves competitive performance and faster convergence compared with methods using various visual pre-training paradigms.