Source-linked AI summary
You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models
Andrea Morales-Garzón, Salvador López-Joya, Miguel López-Pérez, Maria J. Martin-Bautista
TL;DR
Prompt ensembling remains difficult under domain shift because prompt quality and the value of disagreement signals are unclear in specialized domains. This paper evaluates ZPE in agrifood vision-language models and introduces PID, finding that ZPE substantially benefits out-of-distribution settings and prompt disagreement improves failure detection under severe shift.
Problem
ZPE’s behavior under domain shift remains underexplored, including whether its prompt-disagreement signal remains meaningful far from the training distribution.
Method
The paper evaluates ZPE across agrifood domains and introduces PID, which repurposes prompt disagreement within ZPE as an epistemic-uncertainty signal for detecting unreliable outputs.
Results
ZPE provides marginal benefit in-distribution but substantial benefit out-of-distribution, while domain-specific prompt pools outperform larger generic pools and PID improves failure detection under severe shift.
Takeaways & Limitations
ZPE can act as an unsupervised domain-alignment detector, while prompt disagreement provides a reliability signal for selective prediction under out-of-distribution conditions.
Takeaways & Limitations
PID’s broader applicability remains unresolved, and its temperature parameters are fitted on validation data rather than in a fully zero-shot setting.
Abstract
from arXiv · showhide
Vision-language models enable zero-shot classification through natural language prompts, but performance is sensitive to prompt formulation, especially in specialized domains. Zero-shot Prompt Ensembling (ZPE) addresses this by weighting prompts by discriminative signal, yet its behavior under domain shift remains unexplored. We evaluate ZPE in the agrifood domain using CLIP and SigLIP across four datasets and four prompt pools, spanning in-distribution (ID) food and out-of-distribution agricultural benchmarks. ZPE provides limited benefit under ID conditions but substantially improves performance and calibration under domain shift, where domain-specific pools of 51-52 prompts consistently outperform generic pools of 247-426. Lexical analysis shows that ZPE acts as an unsupervised domain-alignment detector without label access. We further introduce PID (Prompt-based Inconsistency Detection), which repurposes prompt disagreement as epistemic uncertainty, improving failure detection under severe domain shift where standard confidence measures collapse.
1 Introduction
This study examines Zero-shot Prompt Ensembling (ZPE) for agrifood vision-language classification under in-distribution and domain-shifted conditions. It finds that ZPE is marginally beneficial in-distribution but substantially more useful out-of-distribution, and introduces PID to detect unreliable predictions from prompt disagreement.
- Motivation: ZPE assigns greater weight to prompts with stronger discriminative signal, addressing performance degradation from poorly aligned or noisy prompt templates.The weighting is estimated without an external validation set.
- Findings: ZPE provides limited benefit under ID conditions but becomes substantially more useful when OOD domain shift is severe.The study evaluates this regime-dependent behavior across agrifood domains, including food classification and agriculture-related tasks.
- Contributions: PID repurposes prompt disagreement within the ZPE ensemble as a quality-gated epistemic uncertainty signal for detecting unreliable OOD predictions.It improves reliability under domain shift without modifying the prediction pipeline or adding computational cost.
- Findings: OOD performance disparities are especially pronounced in agriculture and bean disease benchmarks, contrasting with robust results in food classification.These disparities highlight sensitivity to domain-specific vocabulary and fine-grained visual variation.
- Contributions: Lexical analysis associates concrete visual terms with higher ZPE scores, whereas abstract or weakly visual words correspond to lower-quality prompts.This analysis provides a lexical perspective on prompt quality.
2 Related Work
Related work shows that zero-shot vision-language performance depends strongly on prompt design, while calibration and ensemble disagreement provide complementary perspectives on prediction reliability. Agrifood benchmarks further expose challenges from fine-grained recognition, limited supervision, and distribution shift.
- Prompt engineering for VLMs: Prompt formulations can produce different zero-shot predictions, motivating methods that replace class-name prompts with automatically generated discriminative descriptions.Classification by description uses features such as stripes, claws, and size to improve accuracy and interpretability.
- Calibration in zero-shot models: Modern architectures, including CLIP, are often better calibrated, and calibration can improve with model scale under distribution shift.This work instead studies prompt disagreement within ZPE rather than applying explicit calibration.
- Agrifood image classification: Agrifood applications remain difficult because fine-grained recognition, domain shift, and limited supervision challenge otherwise strong CLIP and BLIP-2 baselines.Prior studies report significant performance degradation when agricultural expertise is required.
- Agrifood image classification: Food-101 provides a standard 101-category fine-grained food benchmark, whereas Beans offers a three-class plant-disease stress test where zero-shot models often perform near chance.These datasets capture substantial intra-class variability and robustness under distribution shift.
- Epistemic uncertainty in ensembles: Ensemble disagreement serves as a practical proxy for epistemic uncertainty, with more diverse predictions associated with higher uncertainty, especially under distribution shift.The proxy reflects insufficient or mismatched training signal.
3 Methodology
The methodology evaluates CLIP and SigLIP zero-shot predictions with transductive Zero-shot Prompt Ensembling (ZPE), which scores prompts and converts their quality into ensemble weights. It also introduces Prompt-based Inconsistency Detection (PID), using prompt disagreement to support selective prediction and diagnose domain shift.
- VLM encoding: Each image and class-conditioned text prompt is encoded into D-dimensional representations, whose similarity produces a prompt-specific class logit.The image encoder is v = fθ(x), while the text encoder is tp,c = gϕ(tmplp(c)).
- VLM encoding: CLIP uses contrastive loss with α ≈100, whereas SigLIP uses sigmoid loss with α≈117 and removes dependence on batch normalisation.These are the two evaluated vision-language model families.
- Zero-shot Prompt Ensembling: ZPE assigns greater weight to prompts with stronger discriminative signal by scoring each prompt across all N test images before individual predictions are made.This makes ZPE a transductive procedure over a pool of P prompts.
- Zero-shot Prompt Ensembling: ZPE scores prompts using either the mean maximum logit or logits centred by the test-set mean, then converts scores into weights with a softmax temperature τ.The normalized variant removes each prompt’s unconditional logit bias, while τ is selected from 0.1 to 10.0 using held-out validation accuracy.
- Prompt-based Inconsistency Detection: PID repurposes prompt disagreement as a per-sample failure detector, adding confidence κ(x) for selective prediction without changing predicted classes or probabilities.High κ indicates prompt agreement and likely correctness; low κ supports abstention or escalation.
- Prompt-based Inconsistency Detection: The τa ablation diagnoses domain shift: a low-τa peak indicates useful quality-gating, whereas a monotonically increasing curve indicates that collective disagreement is most informative.τa and c are fitted on a held-out validation split, with τa selected by correctness-prediction AUROC and c by Dirichlet-Categorical NLL.
4 Experimental setup
The experiments span four food and agricultural benchmarks, six vision-language models across SigLIP and CLIP, and four general or domain-specific prompt pools. Class-name preprocessing and prompt-pool design address tokenization and domain coverage across the evaluated tasks.
- Preprocessing: Replacing underscores with spaces in class names is critical for Beans because naive underscore names reduce accuracy by ∼30 percentage points through text-encoder tokenization mismatch.All class names are preprocessed using this replacement.
- Datasets: The evaluation covers Food-101, Food-11, Agricultural crops, and Beans, spanning food recognition, coarse-grained crop classification, and bean leaf disease classification.Food-101 has 101 categories and 1,000 test images per class; Food-11 has 11 groups and 1,000 samples per class; Agricultural crops has 30 categories and 829 images.
- Models: Six models compare SigLIP and CLIP across three scales each, enabling evaluation across architectures and model capacities.The models are SigLIP base, SO/400M, and large, plus CLIP B/32, B/16, and L.
- Prompt pools: The study uses four prompt pools: two general pools and two domain-specific pools designed to add visual angles absent from generic templates.Pool-247 contains 247 templates, Pool-426 adds 179 ChatGPT-generated prompts, Pool-Food has 51 prompts, and Pool-Agri has 52 prompts.
5 Results
Results show that ZPE is most useful under domain shift: domain-specific prompt behavior improves performance and calibration, while prompt disagreement provides complementary uncertainty signals when confidence collapses. Lexical analyses further indicate that ZPE detects prompt quality and domain alignment without labels.
- Backbone and dataset results: CLIP-L/14 and SigLIP-400M are the strongest backbones in their respective families, with SigLIP SO400M consistently outperforming CLIP ViT-L/14 across prompt pools and datasets.Subsequent performance analysis therefore reports only CLIP-L/14 and SigLIP-400M.
- Backbone and dataset results: ID datasets show high, similar accuracy and minimal calibration error across ensemble strategies, whereas OOD Agriculture exhibits greater variation, especially in calibration metrics.The challenging Beans dataset is identified as a particularly difficult case, although the supplied passage truncates before its detailed results.
- Prompt quality and domain alignment: ZPE identifies mismatched prompts without labels: the fundus-image prompt scores 6.4, 2.4, 4.9, and 0.8, alongside accuracies 0.48, 0.60, 0.60, and 0.37, on Food-101, Food-11, Agriculture, and Beans.Prompt scores reflect how much each prompt changes the ensemble prediction relative to the ensemble mean.
- Prompt quality and domain alignment: Food and Agriculture prompt scores are strongly correlated (ρ = 0.765), while domain-specific top prompts differ lexically in definite versus indefinite constructions.Food-specific prompts favor forms such as “aerial photo of the {},” whereas agriculture-specific prompts favor “aerial photo of a {}.”
- PID uncertainty evaluation: On Beans, PID reaches AUROC = 0.646 versus max-conf AUROC = 0.275 for CLIP-B/16 with Pool-247, a +0.37 gap, while max-conf spans AUROC 0.27–0.72 across models.On ID Food-101 and Food-11, max-conf remains a strong baseline with AUROC 0.88–0.94; PID is presented as complementary and regime-dependent.
- PID uncertainty evaluation: The τa curve diagnoses domain shift: peaked curves indicate useful quality-gating and some domain grounding, whereas monotonically increasing curves indicate that collective disagreement is more reliable.Food-101 peaks around τ ∗ a ≈0.2–0.5, while a plateau at τa = 1.0 indicates no quality-gating is needed.
6 Conclusion
The conclusion frames ZPE as an unsupervised domain-alignment detector and motivates PID as quality-gated epistemic uncertainty from prompt disagreement. It also identifies practical diagnostics for domain shift and outlines limitations and future directions for specialized agrifood VLMs.
- Contributions: ZPE assigns higher scores to prompts whose vocabulary matches the visual domain, explaining why small domain-specific pools outperform large generic ones without labels.This motivates PID, which repurposes prompt disagreement to predict model failures.
- PID: PID estimates errors reliably under mild shift and provides tighter error bounds under severe domain shift.It measures epistemic uncertainty from disagreement while quality-gating which prompts contribute.
- PID: The τa ablation curve diagnostically reveals whether a model is domain-grounded or operating under full domain shift.Peaked curves indicate beneficial quality-gating, whereas monotonically increasing curves indicate full domain shift.
- Practical implications: The study provides practical guidance for prompt-pool design and for understanding how quality-gating affects performance, calibration, and selective prediction.These conclusions extend the account of when and why prompt ensembling works in specialized visual domains.
- Future work: Future work should explain why semantically weak prompts can score highly, including possible indirect visual cues or representation-space regularization effects.Spanish prompt consistency is another direction requiring multilingual VLMs and Spanish prompt pools.
- Limitations: Validation-fitted a and c values depart from fully zero-shot evaluation, while CLIP’s near-random Beans performance limits conclusions about PID on larger out-of-domain benchmarks.The Beans result is attributed likely to small size, class imbalance, and a large pretraining domain gap; data-free temperature selection is proposed as future work.