Source-linked AI summary
Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap
Earl Ranario, Jared Smith, Lars Lundqvist, Urmil Jatin Chandarana
TL;DR
VLMs perform poorly on zero-shot agricultural classification, and the paper investigates whether the gap reflects weak visual representations or poor alignment with domain knowledge. Using a broad benchmark, linear probes, reference descriptions, and rubric-grounded test-time verification, it finds that visual features are largely present and that structured reasoning nearly doubles judged F1, while verifier confidence is not a reliable uncertainty measure.
Problem
Zero-shot VLM performance is poor across agricultural disease, pest, damage, quality, and species classification, leaving it unclear whether the limiting factor is visual representation or mapping visual features to diagnoses.
Method
The paper builds a 116-dataset agricultural benchmark, probes frozen VLM vision towers, tests oracle reference descriptions, and uses rubric-guided candidate generation with a PPT verifier.
Results
Rubric-grounded verification nearly doubles judged F1 over the unaided lower bound and reaches 0.71 disease F1 for Gemma 4 E4B-it, above its 0.60 upper bound on that task.
Takeaways & Limitations
VLM vision encoders already contain substantial agricultural structure, while rubric-grounded generation helps recover latent knowledge without supplying an oracle description.
Takeaways & Limitations
The oracle upper bound measures answering after receiving exactly the needed visual information, not reasoning about a novel description.
Abstract
from arXiv · showhide
Vision-language models (VLMs) show promise for agricultural classification, but zero-shot performance on disease, pest, damage, quality, and species identification remains poor, and it is unclear whether this reflects weak visual features or a failure to connect them to domain knowledge. We build a benchmark of 116 datasets, 834 classes, and 8,324 images spanning these tasks to isolate where the gap arises. Linear probing shows VLM vision encoders already encode agricultural features nearly as separable as a self-supervised DINOv3 baseline, ruling out weak visual representations as the primary bottleneck. Conditioning each model on an oracle reference description (an upper bound on its parametric knowledge) closes most of the gap left by an unaided lower bound, showing VLMs already know more about agriculture than they show. To close this gap without an oracle description at inference time, we structure test-time reasoning around a fixed, per-task diagnostic rubric: the model generates $K$ candidate responses and a Probabilistic Pivot Tournament (PPT) verifier, scored pairwise against the rubric, selects the best one. This nearly doubles judged F1 over the lower bound and matches or exceeds the upper bound on several tasks, notably pushing Gemma 4 E4B-it's disease F1 to 0.71, above its own upper bound of 0.60. However, the verifier's letter-scale confidence score has the opposite of its intended effect: filtering to its most confident predictions does not improve accuracy and correlates negatively with correctness across every model and pool size tested, so the score cannot serve as a measure of predictive uncertainty, and most of the observed gain likely comes from rubric-grounded generation rather than pairwise verification.
1 Introduction
Agricultural VLMs remain unreliable in zero-shot classification, prompting the paper to test whether the bottleneck lies in visual features or in connecting those features to domain knowledge. The authors address this with a cross-task benchmark and diagnostic pipeline separating vision from alignment.
- Zero-shot agricultural classification remains difficult because disease, pest, and species identification depend on complex domain-specific patterns across locations and management practices.
- The paper asks whether poor VLM performance reflects insufficient visual feature extraction or failure to map visual features to agricultural diagnoses.
- The benchmark contains 116 datasets, 834 classes, and 8,324 images spanning disease, pest, damage, and weed species classification.
- Linear probing finds VLM vision towers encode agricultural features with F1 separability comparable to DINOv3, indicating sufficient task-relevant visual information.
- Reference descriptions close much of the performance gap, supporting the hypothesis that VLMs possess agricultural knowledge but fail to reliably express it unaided.
2 Setup and prior analysis
The study evaluates agricultural VLMs with a broad benchmark and diagnostic analyses to determine whether failures arise from visual representations or from connecting visual features to agricultural knowledge.
- 2.1 Datasets and models: The benchmark contains 116 datasets, 834 classes, and 8,324 images spanning disease, pest, damage, quality, and species identification across crops and locations.The evaluation pool is drawn from the AgML collection and is reused across experiments.
- 2.2 Linear probing setup: Linear probing freezes each VLM vision encoder, pools patch features into image embeddings, and trains a single linear classifier to measure task-relevant visual separability.The restricted probe is intended to reflect frozen representation quality rather than classifier capacity; performance uses macro-averaged F1.
- 2.3 Vision towers encode sufficient information: DINOv3 reaches F1-macro separability of 0.67, 0.60, 0.68, and 0.82 across disease, pest/damage, quality, and species, while Gemma 4 26B-A4B-it reaches 0.68, 0.60, 0.69, and 0.78.Among VLM encoders, Gemma 4 models are strongest, indicating that the vision encoder is not the primary bottleneck.
- 2.4 Lower and upper bound of expected performance: Reference descriptions define an upper-bound condition approximating parametric knowledge, while image-only classification defines the lower bound for unaided visual reasoning.Both conditions use the same open-ended classification question and differ only in whether a class description is prepended.
- 2.5 Descriptions help ground visual features: Every evaluated model improves substantially with reference descriptions, with Gemma 4 E4B-it gaining Δ=0.15 and Gemma 4 26B-A4B-it gaining Δ=0.11.Gemma 4 E4B-it begins with the lowest unaided baseline but matches or exceeds the larger Gemma model after descriptions are provided.
- 2.5 Descriptions help ground visual features: The upper bound measures answering when the needed visual information is supplied, not reasoning about a novel description, so it approximates a ceiling on parametric knowledge.Its interpretation depends on descriptions synthesized specifically to contain the visual detail needed for class identification.
3 Proposed solution and results
The method structures test-time reasoning with fixed task-specific diagnostic rubrics, candidate generation, and PPT pairwise verification. Verifier performance approaches or exceeds oracle-description upper bounds on several tasks, but its confidence score is not reliable.
- 3.1 Verifier methodology: A fixed, task-specific diagnostic rubric structures reasoning for disease/pest and damage, species/weed identification, and quality/ripeness.The rubric is appended to the base classification question and contains no class names or ground-truth information.
- 3.1 Verifier methodology: For each image, the model independently generates K rubric-based candidates in one fill-in-the-blank turn, then PPT ranks them and reports the top candidate.The verifier evaluates candidate pairs against the same task-specific rubric in a single holistic pass.
- 3.2 Verifier and rubric results: 0.71 judged F1macro on disease for Gemma 4 E4B-it at K = 8 exceeded its own 0.60 upper bound.At K = 4, the same model reached 0.69.
- 3.2 Verifier and rubric results: The verifier matched or exceeded the upper bound on several tasks, while increasing K from 4 to 8 changed each task’s score by at most 0.02.The caption reports minimal gains from additional candidates within each model.
- 3.2 Verifier and rubric results: The verifier’s letter-scale win rate correlated negatively with correctness across every tested model and pool size, so it cannot measure predictive uncertainty.Filtering to higher-scoring predictions reduced accuracy rather than improving it.
4 Conclusion
The paper identifies alignment rather than vision as the main bottleneck in zero-shot agricultural classification. Rubric-grounded generation recovers latent knowledge, while the verifier score should not be used to filter predictions.
- 4 Conclusion: Linear probes and reference descriptions indicate that VLM vision encoders contain agricultural information, but models fail to expose it unaided.The conclusion contrasts visual separability with the remaining alignment gap.
- 4 Conclusion: Rubric-grounded candidate generation nearly doubled judged F1 over the unaided lower bound and matched or exceeded the oracle upper bound on several tasks.The conclusion attributes the improvement more plausibly to single-pass rubric-grounded generation than pairwise verification.
- 4 Conclusion: The verifier’s letter-scale score collapsed toward the top of its scale and correlated negatively with correctness, making it unsuitable as a confidence signal.The paper calls for more discriminative verification scores in future work.
A Manual CoT tested as a standalone condition
Manual CoT tested the diagnostic rubric without class-specific information or candidate verification. It produced only a small, inconsistent gain over the lower bound, with no task-specific effect reliably distinguishable from zero.
- A Manual CoT tested as a standalone condition: ∆= 0.008 overall, Manual CoT produced only a modest gain over the lower bound.The condition answered rubric criteria in order before producing a final label.
- A Manual CoT tested as a standalone condition: ∆= −0.002 for species identification, where the structured criteria chain slightly reduced performance.Every task-level confidence interval spanned zero, so no individual effect could be distinguished from no effect.
- A Manual CoT tested as a standalone condition: Manual CoT alone was not a reliable lever for closing the gap to the upper bound.The results indicate that fixed reasoning structure without explicit verification offers no guarantee of a positive effect.
B Verification prompt structure
The verifier compares two rubric-filled candidate trajectories as an agricultural scientist reviewer. It evaluates their diagnoses against the task rubric and visual evidence, then emits one score token per trajectory.
- B Verification prompt structure: The verifier receives a task description, evaluation rubric, and two candidate trajectories for pairwise comparison.
- B Verification prompt structure: Each comparison asks the reviewer to explain what each trajectory diagnoses and how well it matches the visual evidence and criteria.The prompt requires the specific difference between candidates and checks visually similar but distinct conditions when relevant.
- B Verification prompt structure: The verifier appends one score token for each trajectory after generating its comparison analysis.
C Uncertainty analysis
The verifier’s letter-based score fails as a confidence signal. Accuracy decreases as the retention threshold rises, while scores concentrate near the scale’s ceiling, with model-specific distribution patterns.
- C Uncertainty analysis: 92% of Gemma 4 E4B-it scores and 80% of Gemma 4 26B-A4B-it scores fall within letters A–C.The remaining seventeen letters account for only a small fraction of the score mass.
- C Uncertainty analysis: Gemma 4 26B-A4B-it shows a secondary L–M cluster at both K values, unlike Gemma 4 E4B-it’s near-ceiling concentration.The distributions suggest distinct response behaviors across the two models.
- C Uncertainty analysis: Accuracy declines almost monotonically as the win_rate threshold increases for both models at K = 4 and K = 8.This is opposite to the behavior expected from a useful confidence signal.
C.1 Judge leniency: symptom-shape descriptions vs. specific diagnoses
The verifier can credit plausible symptom descriptions even when the stated diagnosis is generic or causally incorrect. The examples distinguish specificity gaps from mismatches in causal category.
- C.1 Judge leniency: symptom-shape descriptions vs. specific diagnoses: The judge often credits a prediction when its observational narrative plausibly matches the reference class, despite an incorrect or generic stated diagnosis.The full response includes crop, symptom color, shape, texture, damage pattern, and final diagnosis.
- C.1 Judge leniency: symptom-shape descriptions vs. specific diagnoses: Some cases are specificity gaps: the diagnosis identifies the general symptom category but omits the pathogen-specific name needed to distinguish similar conditions.
- C.1 Judge leniency: symptom-shape descriptions vs. specific diagnoses: Other cases are causal-category mismatches, such as naming a fungal pathogen instead of an insect pest despite an accurate visual description.