Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

1 Introduction

• Learned prompts are task-specific, generalize poorly beyond base classes, and often require task-specific data, retuning, or manual repository search and composition.The introduction argues for learning prompt structure across tasks without the double optimization loop used by classical meta-learning.Source: §1 · pp. 1–2
• DMP learns a diffusion model over previously learned prompts and synthesizes task-specific prompts from natural-language descriptions without revisiting source-task data or losses.The paper also gives a bound on expected task loss in terms of repository empirical loss and diffusion denoising error.Source: §1 · pp. 2–3
• Test-time steering interpolates each predicted clean prompt with a training-selected, highest-accuracy repository prompt without retraining or using target examples or novel test classes.The selected prompt serves as an anchor during DDIM sampling.Source: §1 · p. 2
• DMP supports concept composition, guided sampling, novel subject generation, editing, and negative prompting without explicit training.A single model can produce prompts for subject personalization and slider attributes.Source: §1 · p. 3
• DMP extends beyond prompt embeddings to sparse multi-view retrieval, synthesizing missing-view embeddings for retrieval without training on gallery objects.The retrieval gallery combines observed and synthesized embeddings.Source: §1 · p. 3
• DMP achieves up to 2.0% accuracy gains in cross-task and composite classification and reduces storage and inference costs by over 90% versus prompt retrieval.It also improves retrieval by +3.44 mAP over direct DINOv2 retrieval.Source: §1 · p. 3

Foundation models

• Vision-language and text-to-image models use prompts for tasks including open-set recognition and personalization (Gal et al., 2022).Prompt Sliders learn concepts or attributes in textual space.Source: §2 · p. 3

Prompt learning

• Prompt-learning methods tune textual, visual, or multimodal prompts for CLIP, often refining prompts for a single task or example (Zhou et al., 2022a; Jia et al., 2022; Khattak et al., 2023).DMP instead generates across tasks without task-specific data or model training, using precomputed prompts.Source: §2 · p. 4

Meta learning

• Meta-learning typically amortizes adaptation with a double optimization loop, while diffusion-based prompt refinement has also been trained for a specific example and classification task.DMP is presented as a generalist method that samples prompts across tasks and downstream functionalities.Source: §2 · p. 4

3 Diffusion Meta-Prompting

• DMP trains a conditional diffusion generator on a repository of learned prompts, synthesizing prompts from task descriptions without retraining for each task.The repository contains prompts already learned with downstream task losses; standard MSE training suffices to synthesize prompts of low downstream risk.Source: §3 · p. 4
• DMP trains by noising repository prompts and teaching a text-conditioned U-Net to predict noise, then samples from a noise seed using the task description.The model’s meta-parameters amortize prompt production across tasks, enabling one-shot synthesis rather than per-task optimization.Source: §3 · pp. 4–5
• With enough repository prompts and sufficiently small denoising loss, DMP is guaranteed to meet prompt learning’s performance.The guarantee bounds the distance between task losses for traditionally learned prompts and DMP-sampled prompts.Source: §3 · p. 5
• DMP uses a 1D Stable Diffusion variant, operating directly on prompt embeddings except for unusually long prompts (d ≥ 2048), which use a lower-dimensional VAE.The VAE has fewer than 1M parameters and a latent dimension of 128.Source: §3 · p. 5
• For task composition, DMP combines task-conditioned noise predictions using temperature weights, producing one prompt for multiple tasks with less complexity than running the downstream model once per task.The composition rule follows Liu et al. (2022b).Source: §3 · p. 5
• Test-time steering interpolates DMP’s denoised prompt with a repository anchor selected by training accuracy, stabilizing sampling without retraining or access to novel test data.The generated prompt is pulled toward a repository region associated with strong training performance, not copied from the anchor.Source: §3 · pp. 5–6

4.1 DMP for classification

• DMPClass uses CoOp prompts from ImageNet and 10 fine-grained datasets, and the experiments assess generalization rather than state-of-the-art prompt-classifier performance.The authors note that modeling additional method-specific parameters would require larger diffusion models and more compute than used here.Source: §4.1 · p. 6
• Across 55 dataset pairs, DMP gains +2.0% for Base, +1.9% for New, and +1.9% for All over ProMetaR, and +4.3%, +5.7%, and +4.4% over CoOp, respectively.For EuroSAT and Flowers, DMP exceeds BPL’s zero-shot baseline by +8.5/+4.4% and CoOp by +25.6/18.4% for All/New classes, respectively; it also beats trained BPL by +2.7% on New classes.Source: §4.1 · p. 7
• Test-time steering improves composite-classification accuracy by an average of +2.0 points over unsteered DMP sampling.Gains are +1.7 points on All, +2.1 on Base, and +2.2 on New; Table 4 identifies the steering and CoOp gain columns.Source: §4.1 · p. 7, Table 4 · p. 7
• For hierarchical classification, DMP gains 2.4–9.4% over BPL and 5.8–11.7% over CoOp, despite no hierarchical-classification prompt training.The experiment uses ImageNet, SUN, and OOD ImageNet variants; Figure 2 compares DMP, CoOp, and BPL.Source: §4.1 · p. 8, Figure 2 · p. 8

4.2 DMPVariation

• DMPVariation learns prompt variations from Textual Inversion prompts and can sample variations for novel subjects when conditioned on their Textual Inversion prompt.During training, the model denoises 40 variations of a conditional prompt input.Source: §4.2 · p. 8
• On novel prompts, DMPVariation improves CLIP-Text similarity by 1.4% and CSD style by +11.4% versus Textual Inversion, while producing more instruction-compliant images.The reported metrics do not capture the full qualitative gap; Figure 3 shows improved adaptation across contexts and variation prompts for objects despite training on CelebA faces.Source: Table 6 · p. 7, §4.2 · pp. 8–9
• For sparse multi-view retrieval, DMP improves mAP by +3.44, NDCG by +1.83, and Top-1 accuracy by +0.30 over direct DINOv2 embeddings, with comparable MRR.DMP synthesizes embeddings for missing views of gallery objects; Table 7 describes the 5,447-object gallery and disjoint training objects.Source: Table 7 · p. 7, §4.2 · p. 9
• DMP reduces storage needs by 91% and improves inference speed by 92% compared with repository-based retrieval.Table 5 compares DMP and Stylus for the tasks considered in the paper.Source: Table 5 · p. 7, §4.2 · p. 9

5 Ablation Studies

• Accuracy varies by at most 0.27 points under moderate-to-large perturbations of the steering anchor.The perturbation results are averaged over dataset pairs and repeated noise samples; Table 8 defines the fixed-anchor and resampled conditions.Source: §5 · p. 9, Table 8 · p. 10
• Replacing the training-selected anchor with a fixed random training prompt lowers All/Base/New accuracy from 63.1/69.4/70.8 to 62.4/69.1/70.2.Because both anchors remain fixed during sampling, the comparison indicates that the training-selected anchor provides task-relevant guidance beyond stabilization alone.Source: §5 · p. 9
• Resampling the perturbation direction at every denoising step reduces accuracy by 1.8/1.9/2.2 points below the unperturbed anchor on All/Base/New splits.The results indicate that a consistent steering target also contributes to sampling stability.Source: §5 · p. 10
• With steering, sampled-prompt distances remain 0.025–0.049 and accuracy stays at 93.0–93.2% on Caltech101 and 64.5–64.9% on ImageNet as class names are replaced.Table 9 measures condition and prompt distances from the original condition while replacing Caltech101 class names with ImageNet names.Source: §5 · p. 10, Table 9 · p. 10

6 Conclusion

• DMP synthesizes prompts from natural language and generalizes across recognition, retrieval, and generation beyond the tasks, label sets, views, and concepts seen during training.It avoids the overhead of storing and searching large prompt repositories.Source: §6 · p. 10
• DMP improves cross-task and composite classification in challenging generalization regimes and supports compositional control, semantic negation, and better prompt compliance for subject variations.Source: §6 · p. 10
• The theoretical bound links expected downstream loss of DMP-sampled prompts to repository quality and diffusion denoising error.It offers a principled account of conditions under which meta-prompt generation succeeds.Source: §6 · p. 10

Broader Impact

• DMP uses pretrained models that have been shown to contain harmful biases, which prompts may elicit.Source: Broader Impact · p. 11
• Personalizing celebrities could be misused to propagate harmful, unlawful, or unethical information.Source: Broader Impact · p. 11
• The paper suggests inspecting prompt embeddings with nearest-neighbor tokens before image generation and using watermarking to help identify generated content.Source: Broader Impact · p. 11

A.1 Theoretical Analysis

• For each concept, the setup assumes a textual description, a downstream loss for candidate prompts applied to a frozen foundation model, and a repository of exemplar prompts.The exemplars come from existing prompt-learning techniques.Source: §A.1 · p. 15
• The analysis targets a bound on expected downstream loss when prompts are sampled from the pretrained generator.Source: §A.1 · p. 15
• When the generator closely matches the repository distribution and that repository has low expected task loss, the sampled prompts are expected to have low task loss as well.Source: §A.1 · p. 15

A.1.1 Distributional discrepancy bound

• The distributional discrepancy proposition assumes task loss is bounded in [0, Lmax] and compares repository and generator prompt distributions for a fixed condition.Source: §A.1.1 · p. 15
• The proof uses the bounded loss to relate the expected-loss difference to total variation, then invokes Pinsker’s inequality to obtain the stated bound.Source: §A.1.1 · p. 15
• The bound decomposes expected downstream loss into repository-distribution loss and KL divergence, with the latter controlled by denoiser training quality.Source: §A.1.1 · p. 15

• A small denoising loss implies a small KL divergence between the learned and data distributions of clean prompts, under standard DDPM assumptions and mild regularity conditions.Source: §A.1.2 · p. 16
• The simplified DDPM denoising objective is equivalent, up to constants and time-discretization effects, to score matching that estimates ∇x log pdata(x).Source: §A.1.2 · p. 16
• The bound is qualitative; its constant depends on the variance schedule, discretization, and model parameterization, while the o(1) term vanishes with finer discretization.Source: §A.1.2 · p. 16

• The finite-repository analysis uses n sampled prompts per condition y to form an empirical distribution from the repository.Source: §A.1.3 · p. 16
• The population expectation is replaced by an expectation under the repository’s empirical distribution.Source: §A.1.3 · p. 16
• Hoeffding’s inequality or McDiarmid’s inequality supplies a probability-at-least-1−δ sampling bound over repository draws.Source: §A.1.3 · p. 16

• With denoising loss at most ε and n i.i.d. repository prompts for condition y, Proposition 2 bounds DMP’s performance guarantee with probability at least 1−δ.Source: §A.1.4 · p. 16
• The diffusion approximation contributes a discretization error represented by an o(1) term.Source: §A.1.4 · p. 16
• The expected task-loss bound decomposes into empirical repository loss, a denoising-risk-controlled approximation term, and a sampling term that vanishes as repository size n grows.Source: §A.1.4 · p. 17

A.2 Relation to Prior Methods

• Unlike task- and model-specific prompt learning, DMP learns from a prompt repository and generates prompts for multiple tasks without task-specific data or losses.The paper also reports that DMP can outperform the prompt-learning method used to create its repository on unseen classes of the same dataset.Source: §A.2 · p. 17, Table 10 · p. 18

A.3 DMP for Classification

• For composite classification, DMP improved across prompt-learning backbones, with unseen-class gains up to +1.2 points for MaPLe and +0.6 points for TAC.Gains were largest for CoOp; for MaPLe and TAC, DMP synthesized prompts while leaving their additional learned components fixed.Source: §A.3 · p. 18, Table 18 · p. 20
• +0.4% to 4.8% average accuracy gains on unseen classes were reported across prompting methods, and DMP outperformed CoOp on 11/11 datasets for new classes.Table 19 reports averages over three seeds and 11 datasets; DMP prompts had slightly lower base-class performance, while overall accuracy across seen and unseen classes increased by up to +3.0%.Source: §A.3 · pp. 19–20, Table 19 · p. 20
• +11.4% MTA and +5% HCA on SUN were reported for DMPCoOp, while DMPTAC gained +10% on SUN and +9% on ImageNet-Sketch for hierarchical classification.DMPCoOp was consistently more accurate across both cross-task metrics on all considered datasets; TAC cross-task gains were larger than its cross-domain gains.Source: Table 14 · p. 19, §A.3 · p. 20, Table 21 · p. 22
• DMP-generated prompts showed 70.7% nearest-neighbor label agreement with real prompts, while sampled-prompt accuracy varied by 0.7% versus 4.7% across baseline initialization seeds.Across five seeds on Caltech101–ImageNet, accuracy standard deviation was 0.09 percentage points, per-class accuracy correlation was 0.999, and error-set overlap was 98.66%; prompts disagreed on about 1.08% of predictions. Moderate steering gave the strongest average composite accuracy, while accuracy decreased as CFG scales increased.Source: Table 11 · p. 18, §A.3 · p. 21, Table 20 · p. 21

A.4 Multi-task DMP

• DMPMulti uses one shared CLIP-ViT L/16 text encoder to synthesize prompts for both personalized subjects and prompt sliders.On identity prompts, DMPMulti achieved higher identity scores, comparable prompt compliance, and lower similarity to training images than Stylus and Textual Inversion; the 20-sample variant had 17% higher prompt compliance than the 40-sample variant. Generated slider prompts also preserved subject identity more robustly than Prompt Sliders at higher scales.Source: §A.4 · pp. 22–23

A.5 DMP for Variations

• DMPVariation generated variation prompts for objects including birds and statues despite training only on CelebA-face identities, and its prompts generalized across contexts while preserving subject identity.Across 66 unseen identities, generated prompts shared an average of 2.7 top-5 and 5.6 top-10 nearest-neighbor tokens with the conditioned Textual Inversion prompts; DMP also enabled identity composition.Source: §A.5 · pp. 23–24

A.6 Ablation Studies

• The class-name sensitivity ablation specifies the original Caltech101 condition and replacement ImageNet class names used at each replacement level.This passage describes the ablation setup and its corresponding class-name conditions, not the ablation results.Source: §A.6 · pp. 25–26

A.6.1 Ablation Study on Alternative methods for Meta-Prompting.

• Diffusion models the continuous prompt distribution more effectively than the tested Transformer and GPT-2 alternatives, while also supporting capabilities challenging or infeasible for autoregressive methods.The Transformer tends to overfit base classes, while discretizing prompt embeddings for GPT-2 loses variation across seeds; diffusion additionally supports classifier-free guidance, negative prompting, inversion, editing, and composition.Source: §A.6.1 · p. 26

A.6.2 Ablation on the number of prompts per concept

• +1.2% to +3.0% gains on new classes are reported as the number of prompts per class increases from 5 to 40.DMPCoOp gains +1.2% with 5 prompts, increasing to +1.6%, +2.5%, and +3.0% with 10, 20, and 40 prompts, respectively. The authors relate this trend to their theoretical guarantee that higher prompt counts reduce downstream risk and improve generalization.Source: §A.6.2 · p. 27, Table 27 · p. 32

A.6.3 Ablation on the text inputs to DMPCoOp model

• Using dataset names as DMPCoOp text conditions beats baseline CoOp prompts by 0.3% on average but trails class-name conditioning by 0.7%.The authors conclude that class-specific names produce prompts with more robust generalization than a single dataset name.Source: §A.6.3 · p. 27

A.6.4 Ablation on the quality of training data

• +1.9% over baseline is retained by DMPCoOp trained with 10% noisy prompts, whose average H.M is 76.7 versus 76.5 for clean-prompt DMPCoOp.The study reports robustness to noise affecting 10% to 40% of training prompts and suggests that a small amount of noise may help generalization.Source: §A.6.4 · p. 28, Table 28 · p. 33

A.7 Ablation on DMPMulti

• DMPMulti performs comparably to separately trained identity and slider models, while Textual Inversion identity composition often distorts or repeats identities.DMPMulti produces high-quality identity compositions in approximately 5 seconds, only 1 second slower than standard Stable Diffusion; the Textual Inversion comparison takes 4 to 8 seconds per image when composing multiple identities.Source: §A.7 · p. 29, §A.7.1 · p. 29, Figure 14 · p. 30

A.8 DMP for Personalization

• DMP personalization supports novel identity generation, negative prompting, identity composition, and fine-grained face interpolation, with generated images comparable in quality to those using original Textual Inversion prompts.DMPVariation can generate intermediate embeddings from existing prompt repositories for on-the-fly prompt generation; DMP also generalizes across models sharing the CLIP text encoder without retraining.Source: §A.8 · pp. 30–31

A.9 Limitations and Future Work

• DMP’s effectiveness depends on the quality and expressiveness of its source prompt-learning method, and its text prompts offer limited fine-grained control.The method is sensitive to noise seeds, although classification accuracy varied by only 0.7% standard deviation from the mean in the reported experiments. Future directions include joint training with downstream foundation models, distilling LoRA adapters into prompts, and synthesizing LoRA weights with Meta-LoRA.Source: §A.9 · pp. 31–32

A.10 Implementation details

• DMP synthesizes only text prompts attached to the CLIP input, replacing the baseline prompt at inference while keeping its other weights fixed.The paper notes that deeper-layer weights for methods such as MaPLe, CoPrompt, and TAC are too large to synthesize with DMP, whereas text prompts have 2048 or fewer parameters (256 in latent space). Reported results average three random seeds.Source: §A.10 · p. 32