PaperlayerView exact revision on arXiv

Source-linked AI summary

Fusing Perceptual Vision Experts with Multimodal Large Language Models for Explainable Plant Disease Diagnosis: From Benchmark Imagery to Real-World Robotic Field Validation

Ranjan Sapkota, Konstantinos I. Roumeliotis, Pengyao Xie, Nikolaos D. Tselikas, Lirong Xiang, Manoj Karkee

arXiv:2608.24934v1cs.CVcs.AI

TL;DR

Field plant disease diagnosis needs reliable fusion of conflicting visual evidence, but single classifiers can disagree under challenging field conditions. H2MAF fuses CNN experts with MLLM semantic arbitration, improving results mainly when expert conflicts are frequent while showing model-specific risk-calibration differences.

  • Problem

    Field conditions produce conflicting classifier evidence, leaving reliable fusion of perceptual signals insufficiently addressed for accurate plant disease diagnosis.

  • Method

    H2MAF combines decision-level fusion of two CNN experts with structured-JSON semantic arbitration by multimodal language models across benchmark and robotic field datasets.

  • Results

    +7.6 points on PlantDoc’s 41.7% CNN-conflict subset, while Cornell gains were marginal at 1.7–4.1% disagreement and Gemma’s risk tracking exceeded Qwen’s calibration.

  • Takeaways & Limitations

    MLLM arbitration is most useful when vision experts disagree sufficiently, but agricultural risk-assessment deployment requires model-specific calibration verification.

  • Takeaways & Limitations

    Temporal leakage from near-duplicate robot frames across splits likely inflates Cornell CNN accuracy, limiting the realism of those performance estimates.

Abstract

from arXiv · show

Accurate field plant disease diagnosis requires reliable fusion of uncertain and conflicting perceptual evidence. We present the Hybrid Hierarchical Multi-Agent Framework (H$^{2}$MAF), combining decision-level fusion of EfficientNet-B3 and ConvNeXt-Tiny with semantic arbitration by open-weight multimodal large language models (MLLMs), Gemma 4 E4B and Qwen3.5 4B, using structured JSON evidence to generate explainable diagnoses, risk levels, treatment urgency, and financial exposure. (H$^{2}$MAF) is evaluated on 14,364 images (1,370 test images) across PlantDoc (2,922 images, 27 classes) and two non-public, continuously captured Cornell robot-acquired field datasets: Stage 2 (20 GB; 4,215 images) and Stage 4 (40 GB; 7,227 images), covering Early Blight, Late Blight, and Septoria Leaf Spot under uncontrolled field conditions. On PlantDoc, Gemma improves accuracy from 63.9% to 68.5%, achieving +7.6 points on the 41.7% CNN-conflict subset. Cornell accuracies reach 99.3% and 98.9%, with only 1.7-4.1% disagreement, demonstrating conflict-dependent MLLM utility. The critical-risk error of gemma is 0.14-0.5 points, whereas Qwen overflags by 3.5-14.4 points. These results establish MLLM arbitration as a promising, yet calibration-dependent, approach for explainable agricultural AI and robotic field decision support. Github Link: https://github.com/Applied-AI-Research-Lab/Explainable-AI-Plant-Disease-Detection

1. Introduction

Plant disease diagnosis faces gaps in field robustness, translation from classification to action, and validation beyond curated benchmarks. H2MAF addresses these gaps through hierarchical CNN–MLLM fusion, explainable decision support, and evaluation on progressively realistic datasets.

  • Motivation: Plant diseases cause 20–40% of global crop yield losses annually, while field systems remain constrained by perceptual fragility, classification-to-action disconnect, and benchmark-to-real-world validation gaps.Controlled benchmarks can yield top-1 accuracies exceeding 99% for standard CNNs, but real-field imagery introduces clutter, overlapping leaves, and variable illumination.
  • Decision Support: The MLLMs receive the leaf image and JSON artifact to generate structured XAI reports containing diagnosis, arbitration reasoning, symptoms, risk level, business recommendation, and treatment window.This translates perceptual outputs into explainable agricultural decision support.
  • Framework: H2MAF closes all three gaps with a two-stage information-fusion architecture validated across three progressively realistic datasets, including two unpublished closed field datasets.The framework combines decision-level fusion with semantic-level fusion.
  • Framework: Two independently fine-tuned CNNs, EfficientNet-B3 and ConvNeXt-Tiny, produce disease-class probability distributions that are serialized with crop and disease context into structured JSON for MLLM reasoning.Context includes crop value, pathogen identity, weather conditions, and risk-aversion level.
  • Key Findings: +7.6 points at a 41.7% conflict rate demonstrates substantial MLLM arbitration gains on PlantDoc, whereas Cornell gains are marginal at 1.7–4.1% conflict rates.CNN experts already agree on 95.9–98.3% of Cornell images.
  • Key Findings: 0–17.6% accuracy for models overriding both CNNs supports constraining unconstrained MLLM autonomy, while Gemma shows 0.14–0.5-point Critical-risk error and Qwen shows 3.5–14.4-point over-flagging.The study also provides a reproducible open-source pipeline spanning CNN training, JSON generation, MLLM inference, and automated evaluation.

2. Related Work

Related work establishes strong CNN performance on controlled imagery but a substantial real-field gap, while MLLMs add visual-semantic reasoning and language-based explanations. H²MAF addresses these limitations by using MLLMs as structured-output arbitrators of CNN evidence and by connecting diagnoses to actionable economic context.

  • CNN-Based Plant Disease Classification: 99% accuracy on PlantVillage established deep learning’s early success, while later benchmarks exposed large performance drops on real-field images and motivated PlantDoc.ResNet, DenseNet, and EfficientNet variants also achieved near-perfect accuracy on controlled images.
  • Multimodal Large Language Models: MLLMs contribute pre-trained commonsense, domain knowledge, and language generation for context-aware explanations, but direct fine-grained disease classification remains unreliable for actionable recommendations.The proposed approach therefore positions MLLMs as arbitrators consuming pre-computed CNN evidence rather than replacing CNNs.
  • Transfer Learning: 97–99% accuracy from shallow fine-tuning of VGG-16 and AlexNet on PlantVillage established ImageNet transfer learning as the de facto protocol.Later work found deeper unfreezing and higher-resolution inputs consistently improve performance.
  • Model Ensembling and Arbitration: 3–8 percentage points is the reported advantage of plant-disease ensembles over single-model baselines, although standard ensembles rely on fixed aggregation rules.The framework instead uses an MLLM not trained on the target dataset and without direct access to class-probability statistics.
  • Structured Explainability and Decision Support: Above 96.8% coverage on every dataset was achieved after suppressing Qwen3.5 4B’s thinking mode, while Gemma 4 E4B reliably followed zero-shot JSON instructions.Structured output techniques address the parsing limitations of naive free-form prompting and support economic context such as crop value, pathogen identity, and risk aversion.

3. Datasets: From Internet Benchmark to Real-World Robotic Field Validation

The study validates one H²MAF pipeline across PlantDoc, an internet-sourced benchmark, and two independently acquired Cornell robotic field datasets. This design spans different acquisition conditions, scales, disease classes, and realism while exposing important limitations in comparing their accuracies.

  • PlantDoc: 2,922 PlantDoc images cover 27 classes spanning 13 plant species, with internet-sourced field photographs containing clutter, variable illumination, mixed content, and compression artefacts.The study excludes the two-image spider-mite class from testing while retaining 27 classes for the final dataset description.
  • Cornell datasets: 4,215 Stage 2 images represent Early Blight, Late Blight, and Septoria Leaf Spot under continuous robotic field capture, using a stratified 80/10/10 split.The evaluated Stage 2 test partition contains 403 images after corrupted or unreadable frames are filtered.
  • Cornell acquisition: Five stereo-camera and active-strobe imaging units continuously captured both sides of crop canopies at Cornell field sites during the 2025 growing season.Data collection occurred at MRS in Waynesville and MHCREC in Mills River, North Carolina, with active illumination reducing ambient-light distribution shifts.
  • Cornell datasets: 7,227 Stage 4 images form an independently acquired, later-season follow-up campaign, nearly doubling Stage 2 volume; 715 images are processed end-to-end from its nominal 723-image test partition.Stage 4 includes 3,295 Early Blight, 1,520 Late Blight, and 2,412 Septoria Leaf Spot images.
  • Limitations: 96–99.8% Cornell CNN accuracy versus 59–64% PlantDoc accuracy reflects an easier three-disease, narrow-host task and acquisition differences, so the gap is presented with an explicit caveat.The paper recommends session-level rather than frame-level splitting and localized lesion-patch cropping for future work.

4. Methodology

H²MAF uses two architecturally distinct CNN experts to generate structured evidence, which two MLLM agents semantically arbitrate into constrained, explainable outputs across all three datasets. The methodology combines dataset-specific business context, controlled prompting, and JSON validation to anchor and operationalize diagnosis.

  • H²MAF pipeline: Two CNN experts independently produce top-K class-probability distributions, which are combined with the original image and structured JSON evidence for two MLLM reasoning agents.The pipeline is applied identically across PlantDoc and both Cornell datasets.
  • CNN perceptual experts: Partial-unfreeze transfer learning freezes EfficientNet-B3 features[0–5] while unfreezing features[6–8], adaptive pooling, and the classification head across all datasets.This preserves low-level ImageNet representations while enabling task-specific adaptation.
  • CNN perceptual experts: Architecturally distinct EfficientNet-B3 and ConvNeXt-Tiny experts deliberately create differing confidence distributions that motivate MLLM arbitration on PlantDoc.The same experts reach near-unanimous agreement on the visually narrower Cornell datasets.
  • Evidence construction: Each test-image JSON artifact contains top-3 predictions and confidence scores from both CNNs plus dataset-specific business context, while withholding the ground-truth label from MLLMs.PlantDoc context includes crop value, risk aversion, and weather; Cornell context includes pathogen, crop value, risk aversion, and treatment urgency.
  • MLLM prompting: The user prompt consistently supplies the image, CNN prediction blocks, business context, and an instruction to output exactly seven JSON fields.Cornell prompts enumerate the three possible diagnoses, while Qwen3.5 uses enable_thinking=False to suppress chain-of-thought reasoning and force direct structured output.
  • Output validation: A two-pass parser extracts MLLM JSON directly or after removing Markdown fences; failures become parse errors, exclude images from accuracy, and count against coverage.The paper recommends a simplified-prompt retry loop as a third production fallback.

5. Experimental Setup

The experiments use fixed computational, training, generation, and evaluation protocols across three datasets. Macro F1 is primary, while cross-dataset comparisons emphasize relative quantities because dataset sizes and label taxonomies differ.

  • Compute and training: Four NVIDIA H100 NVL GPUs with 80 GB VRAM each support all experiments, while CNN fine-tuning runs for 50 epochs with validation Macro F1 checkpoint selection.PlantDoc models train for approximately 2 hours each and Cornell models for 2–4 hours each.
  • Hyperparameter selection: CNN hyperparameters follow established small-data architecture conventions without dataset-specific tuning, isolating dataset realism rather than per-dataset optimisation.ConvNeXt-Tiny uses the full fine-tune strategy recommended in its original paper for small-dataset transfer learning.
  • MLLM generation parameters: Both MLLMs use temperature=1.0, max_new_tokens=2048, and do_sample=True across all three datasets.top_p and top_k remain at each model’s defaults to avoid interfering with calibrated sampling distributions.
  • Evaluation metrics: Macro F1, the unweighted mean over target classes, is primary because it is invariant to class imbalance and reflects minority disease performance.Top-1 accuracy and Weighted F1 are also computed independently for each dataset.
  • Cross-dataset comparability: Absolute accuracy values are not directly comparable across datasets with 27 vs. 3 classes and test sets of 252, 403, and 715 images.Cross-dataset synthesis therefore focuses on MLLM accuracy gain over the best CNN, vision-expert agreement rate, and MLLM risk calibration.

6. Results · 6.1. PlantDoc Results

On PlantDoc, MLLM arbitration improves overall and conflict-image performance, especially for difficult classes and low-confidence cases, but independent overrides and damage cases expose calibration limits. Across Cornell Stage 2, near-unanimous CNN agreement leaves little reliable benefit for MLLM arbitration.

  • 6.1.1. Full Per-Class Classification Report: +8.3% to +25% Gemma gains concentrate in difficult PlantDoc classes, while structurally distinctive classes show no change.The reported gains are Tomato Septoria (+8.3%), Tomato Early Blight (+11.1%), Tomato Mosaic Virus (+20%), Apple Scab (+20%), and Corn Gray Leaf Spot (+25%).
  • 6.1.1. Full Per-Class Classification Report: 68.5% accuracy and 0.6843 Macro F1 make Gemma 4 E4B the strongest PlantDoc model, gaining 9.4 points over EfficientNet-B3 and 4.6 over ConvNeXt-Tiny.Qwen3.5 4B reaches 67.2%, with gains of 8.1 and 3.3 points over the respective CNN baselines.
  • 6.1.3. CNN Validation Performance: 13.3 points separate ConvNeXt-Tiny’s 0.7928 validation Macro F1 from EfficientNet-B3’s 0.6594, confirming distinct CNN capability profiles.The validation set contains 401 images.
  • 6.1.4. Vision Expert Agreement and Conflict Analysis: 41.7% of PlantDoc test images are CNN-conflict cases, where Gemma reaches 47.6% and Qwen 51.4%, versus 32.4% for EfficientNet-B3 and 43.8% for ConvNeXt-Tiny.The CNNs agree on 147 images (58.3%) and disagree on 105 of 252 test images.
  • 6.1.5. MLLM Override Behaviour: 13.6%–17.6% accuracy occurs when MLLMs override both CNNs with independent predictions, supporting arbitration between CNN signals rather than independent visual classification.When MLLMs align with both CNN experts, accuracy reaches 79.7–81.2%.
  • 6.1.6. Arbitration Benefit, Rescue, and Damage Cases: 5.3% rescue, 85% neutrality, and 13.9% damage show that PlantDoc gains mainly arise when MLLMs align with the correct expert in partially-correct CNN cases.The damage rate is identified as the principal limitation.
  • 6.1.7. Confidence-Calibrated Performance: +6.3 points is Gemma’s largest advantage over ConvNeXt-Tiny, concentrated on low-confidence images where CNN visual signal is weakest.The result is reported in the confidence-calibrated performance analysis.
  • 6.2.2. Vision Expert Agreement and MLLM Behaviour: 98.3% CNN agreement on Stage 2 leaves only 7 conflict images, where ConvNeXt-Tiny reaches 85.7% and Gemma 57.1%, while Gemma is neutral on 99.3% of images.On consensus images, all four models achieve 99.0–100% accuracy; Gemma improves on 0% and hurts on 0.7%.

6.3. Cornell Stage 4 Results (Real-World Field Data, 40 GB)

Qwen3.5 4B assigns 3.7× more Critical-risk labels than Gemma 4 E4B, while replication of this difference on Cornell datasets is examined in Section 6.4.

  • Risk-label behaviour: 3.7× more Critical-risk labels are assigned by Qwen3.5 4B than by Gemma 4 E4B.The passage identifies this as a striking behavioural difference between the two models.
  • Risk-label behaviour: Qwen3.5 4B assigns more Critical-risk labels than Gemma 4 E4B.The reported ratio quantifies the difference in model behaviour.
  • Follow-up analysis: Replication of the 3.7× Critical-risk-label difference on Cornell datasets is examined in Section 6.4.The passage points to Section 6.4 for systematic examination of whether the finding replicates.

6.2. Cornell Stage 2 Results (Real-World Field Data, 20 GB)

On the Cornell field datasets, CNN and MLLM performance was near-saturated, but Stage 4 exposed stronger conflict-arbitration gains for Gemma and substantial override-related damage from Qwen.

  • Stage 2 dataset: 403 images across 3 classes formed the Stage 2 held-out test set.Table 13 presents the primary quantitative comparison for this test set.
  • Stage 2 performance: 99.8% made ConvNeXt-Tiny the top Stage 2 performer, narrowly exceeding Gemma at 99.3%, while Qwen reached 97.7%.All models were essentially saturated on Early Blight and Late Blight.
  • Stage 4 dataset: 715 images across 3 classes comprised the Stage 4 held-out test set, nearly doubling Stage 2’s dataset volume.Stage 4 therefore provided a larger evaluation pool under the same real-world field setting.
  • Conflict arbitration: 82.8% made Gemma best on 29 Stage 4 conflict images, improving 13.8 points over ConvNeXt-Tiny at 69.0%.EfficientNet-B3 achieved 31.0%, and Qwen achieved 53.6%; the subset was larger than Stage 2’s n=7 conflict subset.
  • Override effects: 1.9% accuracy on Qwen’s 54 override cases accompanied a 7.6% override rate, helping explain its Stage 4 accuracy decline to 91.8%.Gemma was neutral on 99.3% of images and hurt on 0.7%, whereas Qwen hurt on 8.1%.
  • Risk calibration: 34.8% was Qwen’s Stage 4 Critical rate versus Gemma’s 20.6%, which closely tracked the 20.4% Late Blight prevalence.Qwen substantially over-flagged relative to the same ground truth.

6.4. Cross-Dataset Synthesis

Across PlantDoc and Cornell field datasets, MLLM arbitration gains depend on meaningful CNN conflict, while constrained candidate selection avoids severe override errors. Gemma also shows substantially better critical-risk calibration than Qwen across both Cornell datasets.

  • Cross-dataset performance: 68.5%: All-model Top-1 accuracy rises from PlantDoc’s 59.1–68.5% range to 91.8–99.8% on Cornell datasets, with Gemma and CNN experts stable across Cornell campaigns.Qwen varies from 97.7% on Stage 2 to 91.8% on Stage 4.
  • Conflict-dependent arbitration: +7.6 and +13.8 points: MLLM arbitration improves over the stronger CNN on PlantDoc and Stage 4 conflict subsets, respectively.The conflict subsets contain n=105 PlantDoc images and n=29 Stage 4 images; Stage 4’s conflict rate is 4.1% versus PlantDoc’s 41.7%.
  • Override behaviour: 0–1.9% accuracy: Simultaneously overriding both CNN experts remains severely unreliable across datasets, never exceeding 17.6% accuracy and reaching near-zero on Cornell.The result motivates constraining MLLMs to select among CNN-proposed candidates rather than generate independent diagnoses.
  • Risk calibration: 0.14 points: Gemma’s critical-risk calibration error stays within half a percentage point on Stage 2 and within 0.14 points on Stage 4, versus Qwen’s 3.5- and 14.4-point errors.Qwen’s Stage 2 raw accuracy is 97.7% versus Gemma’s 99.3%, indicating calibration differences are not explained by accuracy alone.
  • Explainability: 6 components: MLLM reports consistently provide diagnosis and confidence, CNN-referenced arbitration reasoning, visual symptoms, calibrated risk, treatment recommendation, and treatment window.The reports demonstrate explainability across all three datasets.

7. Discussion

The discussion finds that MLLM arbitration is most useful when CNN experts conflict, but unconstrained overrides and risk miscalibration limit deployment reliability. It recommends conflict-constrained arbitration, stronger validation, and operationally grounded deployment data.

  • Conflict-dependent utility: 41.7% CNN conflict on PlantDoc produced a +7.6-point arbitration gain, while Stage 2’s 1.7% conflict involved only 7 images and was statistically inconclusive.Stage 4 showed a +13.8-point gain on its smaller conflict subset despite 4.1% conflict.
  • Conflict-dependent utility: MLLMs use symptom semantics and CNN confidence scores to disambiguate overlapping diseases and often defer to the higher-confidence expert.On PlantDoc conflict cases, Gemma agreed with ConvNeXt-Tiny alone in 80 cases and achieved 50% accuracy versus EfficientNet-B3’s 32.4%.
  • Autonomy constraints: 7.6% of Stage 4 test images were Qwen overrides with only 1.9% accuracy, reinforcing that MLLM diagnoses generated independently of CNN evidence can substantially harm aggregate performance.PlantDoc override accuracy was 13.6–17.6%, and the findings motivate restricting MLLM autonomy to resolving pre-computed CNN signals.
  • Risk calibration: 0.14–0.5 percentage points of Gemma risk-calibration error contrasted with Qwen’s 3.5–14.4 points across two independent real-world datasets.Qwen’s error increased from 3.5 points on Stage 2 to 14.4 points on Stage 4, potentially triggering unnecessary interventions; generalization to other prevalence rates remains untested.
  • Operational implications: 124 PlantDoc intervention recommendations represented approximately e6.5M in estimated financial impact, but incorrect diagnoses can waste treatment costs or fail to trigger needed intervention.Gemma’s distribution flagged 25 Critical-risk images and 99 High-risk images; PlantDoc’s damage rate was 13.9%, while Qwen’s Stage 4 damage rate was 6.4%.
  • Limitations and future work: 96–99.8% Cornell CNN accuracy may be inflated by near-duplicate video frames across partitions, while zero-shot inference, static business tables, and closed datasets limit production readiness.The discussion recommends session-level splits, domain-specific fine-tuning, live market and sensor data, and replication with comparable proprietary field imagery.

8. Conclusion

H2MAF combines decision-level CNN fusion with MLLM semantic arbitration and was evaluated across benchmark and robot-acquired field datasets. Its benefits were strongest under CNN disagreement, while risk assessment depended on model calibration, motivating targeted future work.

  • H2MAF fuses two architecturally distinct CNN experts at the decision level, then combines their output with pretrained MLLM semantic knowledge at the reasoning level.The framework translates pixel-level evidence into explainable reasoning through a two-stage information-fusion pipeline.
  • 14,364 images and 1,370 held-out test images across PlantDoc and two continuously captured, robot-acquired field datasets supported the evaluation.The field datasets were Stage 2 (20 GB) and Stage 4 (40 GB), representing previously unpublished, context-specific agricultural data.
  • 68.5% accuracy with Gemma 4 E4B improved on the 63.9% best-CNN baseline, with a +7.6-point gain across the 41.7% CNN-conflict subset.On Cornell, CNN accuracy was already 96.8–99.8% and agreement was 95.9–98.3%, indicating that arbitration gains scale with conflict rate.
  • 0.14–0.5 percentage points separated Gemma’s Critical-risk rate from true urgent-disease prevalence, whereas Qwen over-flagged by 3.5–14.4 points.The replicated Cornell calibration analysis identifies a consistent, model-specific risk-assessment bias relevant to deployment selection.
  • Future work targets agricultural XAI fine-tuning, constrained decoding over the union of top-K CNN predictions, and session-level splitting of continuous field video.These directions address domain adaptation, override-collapse failure, and more conservative deployment-realistic accuracy estimation.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors used ChatGPT and Grammarly during manuscript preparation to improve grammar and sentence structure, then reviewed and edited the content and retained full responsibility.

  • Declaration of generative AI and AI-assisted technologies in the manuscript preparation process: The authors used ChatGPT and Grammarly to enhance grammatical accuracy and refine sentence structure, subsequently reviewing and editing the content as needed.They stated that they take full responsibility for the published article’s content.

Data Availability

The study releases its implementation code openly, while PlantDoc is publicly accessible and the Cornell field datasets remain closed but can be requested from the authors.

  • Code and datasets: All training, inference, artefact-generation, architecture, and evaluation code is publicly available on GitHub under the Apache-2.0 license.The repository includes CNN architectures, JSON artefact-generation scripts, and MLLM inference scripts.
  • Code and datasets: PlantDoc is publicly accessible through its respective repositories.
  • Code and datasets: The Cornell Stage 2 and Stage 4 datasets are closed, non-public data available from the authors upon reasonable request.They are shared by Cornell University’s Automation and Robotics Laboratory under a research collaboration agreement.
  • Code and datasets: The complete processing pipeline for the Cornell datasets is publicly available in the code repository and can be applied to comparable datasets.
Loading 2608.24934v1…