PaperlayerView exact revision on arXiv

Source-linked AI summary

Shortcut Learning in a Public Grape Disease Dataset: Annotation Granularity as a Modulator, Not a Cause

Pushuo Wang

arXiv:2608.20663v1cs.CV

TL;DR

Public agricultural disease datasets may appear usable from in-distribution metrics despite internally inconsistent annotations and hidden out-of-distribution shortcuts. This paper combines controlled model comparisons, cross-species negative controls, and counterfactual retraining to test annotation granularity. Granularity changes the shortcut’s magnitude but not its destination: shrinking coarse boxes reduces false positives, whereas coarsening another class does not create them.

  • Problem

    Public datasets are commonly judged by reported metrics, leaving annotation-scheme consistency and practical usability underexamined.

  • Method

    The paper compares model, resolution, and detection-paradigm controls, evaluates cross-species negatives, and performs single-variable counterfactual retraining with placebo and reverse manipulations.

  • Results

    Annotation granularity modulates shortcut magnitude: shrinking one class’s boxes cuts its cross-species false positives by 66%, while coarsening another class leaves false positives at zero.

  • Takeaways & Limitations

    Granularity can amplify or attenuate an existing shortcut but does not determine its destination, so the feature directing unfamiliar inputs remains unresolved.

  • Takeaways & Limitations

    What determines the shortcut’s destination remains an open question, and airborne lesion-level detection is addressed only through imaging optics rather than experiments.

Abstract

from arXiv · show

Public datasets for agricultural disease detection are usually judged fit for use from reported metrics, which say nothing about whether the annotation scheme is internally consistent. On one public grape disease dataset (3288 images, 11995 boxes, 6 classes), varying model capacity, input resolution and detection paradigm yields a test-set mAP50 range comparable to seed-to-seed noise, with the bottleneck at small objects across all five architectures. The finding lies on the data side: one class is annotated at whole-leaf level (median box area 43.16% of the image) while the other five are annotated at lesion level. On 5156 cross-species images containing no grape, 65.7% of the false-positive boxes fall into that one class, an over-representation of 13.41x relative to its share of the training annotations. Counterfactual retraining establishes a causal effect of granularity on the magnitude of the shortcut: shrinking only that class's boxes cuts its cross-species false positives by 66%, and a placebo control confirms the effect is specific to the manipulated class. A manipulation in the opposite direction, with criteria registered in advance, returns a negative result: coarsening the finest class to whole-leaf level (0.57% to 40.37%), matched in box count and share of annotations and with higher in-distribution AP, still leaves its cross-species false positives at zero boxes, while the unmanipulated original class holds 50.0% of them. Annotation granularity is therefore a modulator of this shortcut, not its cause: it can amplify or attenuate a sink that already exists, but cannot create one, and what fixes the destination remains open. We also give a granularity screening statistic requiring neither images nor training, and show airborne lesion-level detection to be optically out of reach. The failure mode is invisible to in-distribution evaluation.

1 Introduction

The paper asks whether high in-distribution detection metrics establish practical dataset usability, focusing on hidden shortcuts caused by dataset properties rather than model performance alone. It motivates objective early disease observation while distinguishing annotation-scheme defects from acquisition conditions and emphasizing that aggregate metrics can conceal failures.

  • Motivation: Symptom-based spraying decisions can lag infection completion, motivating early, objective, quantifiable disease observation.Visible symptoms may appear one or several weeks after infection, so symptom timing does not coincide with the effective control window.
  • Motivation: Low-altitude nadir observation can reveal upper-leaf disease signs hidden from people inspecting pergola and trellis canopies from below.This acquisition motivation concerns imaging feasibility; unmanned aerial vehicles are not experimentally evaluated in the paper.
  • Problem: The paper asks which intrinsic public-dataset properties determine whether a trained detection system is actually usable.The central concern is whether high in-distribution metrics can be treated as evidence of practical usability.
  • Related work: Shortcut learning combines normal in-distribution performance with sharp out-of-distribution degradation caused by superficial cues unrelated to the task.The paper examines this failure mode in object detection, where annotation-box scale provides an exploitable signal unavailable in classification.
  • Related work: This paper locates a shortcut in inconsistent annotation granularity between classes rather than image content, and uses counterfactual retraining to test its causal role.The boxes can individually enclose genuine disease signs, while one class is predominantly whole-leaf annotated and five are lesion-level annotated.
  • Evaluation: Aggregate metrics can mask scale-specific failure: raising input resolution improves AP_small significantly while aggregate mAP50-95 does not change.The introduction frames this as a broader reason to decompose detection performance by error type and object scale.

2 Data and experimental setup

The experiments use a six-class public grape disease dataset and a cross-species FieldPlant negative control, with controlled training and evaluation protocols. Dataset inspection reveals a class-specific, bimodal annotation-granularity inconsistency, while the setup includes documented controls and engineering limitations.

  • Dataset: The main dataset contains 3288 images, 11995 annotation boxes, and 6 classes, split into 2631/329/328 images.Images were platform-preprocessed, and the lesion-level target averages 3.65 boxes per image.
  • Dataset: The largest-to-smallest class ratio is 7.1:1, so all classes participate in training without resampling.The class distribution is presented in Table 1.
  • Annotation granularity: The mosaic virus disease class has a median box area 5.3 times the second-largest class and 76 times the smallest.Relative box area is computed as each annotation-box area divided by the whole image.
  • Annotation granularity: 58% of mosaic virus disease boxes exceed 30% of the image, while 28% fall below 10%, producing a bimodal distribution.The median is 43.2%, with the larger mode corresponding to whole-leaf boxes and the smaller mode to lesion-level boxes.
  • Annotation granularity: One image can mix whole-leaf mosaic-virus annotation with nine small powdery-mildew lesion boxes, directly illustrating non-uniform annotation standards.The example contains a 15.4% mosaic-virus box and small lesion-level boxes in the berry region.
  • Evaluation: The FieldPlant negative control provides 5156 cassava, maize, and tomato images that cannot contain grape mosaic virus, making grape detections objective false positives.Evaluation uses E1 weights at a confidence threshold of 0.25.

3 Performance ceiling and bottleneck

Across five configurations, aggregate test-set differences are comparable to seed-to-seed noise, while scale-stratified evaluation identifies small objects as the consistent bottleneck. Resolution improves small-object detection despite leaving aggregate mAP50-95 unchanged, and errors are dominated by missed detections and background false positives rather than class confusion.

  • 3.1 Five controls: A 1.54-point mAP50 range and 1.00-point mAP50-95 range spans substantial changes in parameter count, pixel area, and detection paradigm.The range covers 12.4-fold parameter-count, 6.25-fold pixel-area, and one detection-paradigm changes.
  • 3.1 Five controls: The 1.54-point between-configuration range is consistent with seed noise rather than a reliable ranking of the five groups.Single-configuration mAP50 standard deviation is 0.69 points, and the expected range of five draws is about 2.3 standard deviations.
  • 3.2 By scale: the bottleneck is small objects: AP on large objects is three to four times higher than AP on small objects across all five architectures, locating the bottleneck at target scale.Changing architecture does not remove the small-object gap.
  • 3.3 The effect of resolution is masked by aggregate metrics: The improvement in small-object AP is significant (p = 0.018), whereas aggregate mAP50-95 changes by −0.21 percentage points without significance (p = 0.731).AP_small was the prespecified primary endpoint because small objects were independently identified as the bottleneck.
  • 3.3 The effect of resolution is masked by aggregate metrics: AP_medium improves by +5.22 percentage points but remains nonsignificant (p = 0.075) because its seed-to-seed variation is large.Its standard deviation is 0.0245 for E1 and 0.0483 for E3, so more repetitions are needed for an equally strong conclusion.
  • 3.5 Class-level analysis: Only 4 of 1242 ground-truth instances are assigned another class, compared with 179 missed detections and 359 background false positives.The dominant error is localisation—whether the target is found—rather than classification once found.

4 Shortcut learning

The paper tests whether inconsistent annotation granularity creates a class-specific shortcut in grape disease detection. Counterfactual retraining shows granularity changes the shortcut’s magnitude and destination tendencies but is not sufficient to create the sink.

  • 4.4 Counterfactual retraining: 40.2% lower in-distribution AP removed the manipulated class’s anomalous first-place ranking after granularity normalisation.Its AP fell from 0.7609, ahead of the runner-up at 0.5668, to 0.4552, fourth of six; evaluation used each group’s own annotation scheme because the schemes barely overlap at IoU ≥ 0.5.
  • 4.4 Counterfactual retraining: False positives transfer to other classes when the original sink is removed, indicating redistribution among pre-existing receiving tendencies rather than granularity creating the shortcut.The next recipients were already prominent under the baseline, so their succession cannot be established as caused by granularity ranking.
  • 4.4 Counterfactual retraining: Annotation granularity accounts for roughly half of the shortcut magnitude, with 6.16× over-representation remaining after shrinking.The remaining source of the shortcut is unclear, and the destination is not established; the proposed image-level appearance explanation is future work.
  • 4.4 Counterfactual retraining: 66% fewer cross-species false positives followed shrinking only the mosaic class’s boxes, while the placebo class changed by only +4%.The class-specific response excludes a general reduction in false positives caused merely by perturbing annotation boxes.

5 Discussion

The discussion separates annotation inconsistency from annotation coarseness and shows that granularity modulates, but does not determine, shortcut behavior. It also limits airborne conclusions to optical feasibility and emphasizes dataset screening beyond in-distribution metrics.

  • 5.1 Airborne feasibility: 0.16 small-object AP cannot be extrapolated from close-range images to aerial viewpoints without verification.The paper therefore computes airborne feasibility directly from imaging optics.
  • 5.1 Airborne feasibility: At 30 m altitude, a 5 mm lesion occupies just 0.94 pixels, making airborne millimetre-scale lesion detection optically unattainable at feasible altitudes.The proposed division is canopy-level anomaly localisation from the air and lesion-level confirmation on the ground.
  • 5.3 Limitations: Transfer to aerial viewpoints is untested, and the paper's optical conclusions do not establish achievable detection accuracy on real aerial data.All experiments use ground-level close-range images.
  • 5.2 Data construction: R uses median relative box area to identify annotation-granularity departures, classifying medians ≥30% as whole-leaf and ≤10% as lesion level.The thresholds were set a priori, with no determination for intermediate values.
  • 5.2 Data construction: The main dataset is inconsistent rather than merely coarse: five classes are lesion-level, while one whole-leaf class reaches 43.16% median area and exceeds the across-class median by 15.2 times.By contrast, FieldPlant is uniformly coarse, with a coarsest-to-second-coarsest ratio of 1.06×.
  • 5.2 Data construction: 13.41-fold false-positive over-representation and counterfactual retraining show that granularity affects shortcut magnitude, while reverse manipulation leaves the shortcut destination unexplained.The screening statistic can identify internal inconsistency before training, but cannot predict which class becomes the sink.
  • 5.3 Data limitations: The public dataset contains almost no purely healthy images, while the recommendation to train on self-collected data lacks direct experimental evidence because only 10 such images were available.These recommendations address different levels: healthy samples concern whether output is emitted, whereas granularity concerns false-positive allocation.
  • 5.2 Data construction: Human visual sampling reveals composited images with two annotation granularities, defects absent from metrics and identified as the source of the shortcut.The paper also reports class errors and severe under-annotation from earlier pipeline validation.

6 Conclusions

The conclusions identify data properties, especially inconsistent annotation granularity, as more consequential than model-side choices for usability in this dataset. They establish granularity as a shortcut modulator, define practical screening and data-construction priorities, and delimit several open validation questions.

  • Core conclusions: The paper concludes that dataset properties, rather than model choice, determine usability under the studied data conditions.This overarching conclusion follows the paper's model, scale, annotation, and out-of-distribution analyses.
  • Model-side limits: Across five architectures, small objects remain the bottleneck; raising resolution improves AP_small by 1.35 percentage points (+8.1% relative, p = 0.018).Aggregate mAP50-95 masks this scale-specific improvement.
  • Annotation effects: Shrinking coarse boxes reduces class-specific out-of-distribution false positives by 66% and 84% and lowers total false positives from 9176 to 5971 to 3956.The result indicates that coarse annotation increases the model's overall propensity to emit output on unfamiliar input.
  • Annotation effects: A 13.41-fold false-positive over-representation and a 66% counterfactual reduction establish quantifiable shortcut learning linked causally to annotation granularity's magnitude.Removing the class transfers false positives to the next most coarsely annotated classes.
  • Annotation effects: Coarsening the finest class from 0.57% to 40.37% leaves its cross-species false positives at zero, while the original class retains 50.0%.Thus annotation granularity modulates an existing sink but is insufficient to create one; the destination remains open.
  • Future work: The paper recommends a uniform-granularity dataset with sufficient healthy samples as the precondition for subsequent work.Healthy negatives would provide a control for their independent contribution to false-positive output.
  • Future work: Future work includes measuring annotation costs for few-shot transfer, validating canopy-level aerial detection on real aerial data, and establishing fixed-viewpoint disease time series.Aerial validation is a distinct task requiring its own data and annotation standard.

Appendix Reproducibility

The appendix documents a reproducible computational setup, fixed evaluation procedures, counterfactual annotation checks, preregistered criteria, licensing constraints, and the author's use of generative AI.

  • Reproducibility: All experiments use fixed random seeds and are reproducible under identical software and hardware.The reported environment is Windows with an RTX 4060 Laptop 8 GB, PyTorch 2.13.0+cu132, and Python 3.11.
  • Reproducibility: Peak GPU memory use is 2.8 GB, below the available 8 GB limit.The appendix states that training and deployment do not depend on high-performance computing equipment.
  • Evaluation protocol: All in-distribution metrics use pycocotools on the same test set and confidence threshold, with per-class and scale-stratified metrics retained.The test set was not used during training or tuning.
  • Counterfactual validation: Counterfactual COCO conversion preserves image, filename, and category identifiers while replacing only boxes, and unchanged classes retain byte-identical AP.This self-check validates the annotation conversion used for counterfactual evaluation.
  • Registered analysis: Result 5 criteria were registered before training concluded and evaluated automatically through a 3×3 decision table.All nine decision-table combinations had pre-assigned interpretations.
  • Licensing: Ultralytics uses AGPL-3.0, whereas RF-DETR uses Apache 2.0 with fewer commercial constraints.The licensing difference informed subsequent technology selection.
  • Generative AI declaration: Generative AI assisted with prose organisation, language polishing, table formatting, layout, and scripts for data checks and statistical computations.The author states that research design, experiments, interpretation, and conclusions were completed independently.
Loading 2608.20663v1…