PaperlayerView exact revision on arXiv

Source-linked AI summary

AgroGround: Multi-Granularity Grounded Recognition in Agriculture

Abdulla Alshehhi, Zongyan Han, Rao Anwer

arXiv:2610.04425v1cs.CVcs.AI

TL;DR

Agricultural VQA datasets carry rich target labels but rarely connect them to image regions, limiting the available supervision for grounded recognition. AgroGround converts those labels into spatial supervision and trains a shared model on grounding, recognition-and-localization, and healthy-negative instructions. On the reported benchmark, mixed instructions improve recognition and joint accuracy, and the resulting model exceeds its annotation teacher in grounding F1 on the benchmark and PlantSeg.

  • Problem

    Agricultural VQA datasets rarely link their rich semantic labels to image regions, while manual spatial annotation is costly at scale.

    Source: §1 · p. 1

  • Method

    AgroGround converts labels from eight agricultural VQA datasets into spatial supervision and trains a shared vision-language model with grounding, recognition-and-localization, and healthy-negative instructions.

    Source: §1 · p. 2

  • Results

    72.6% recognition accuracy with recognition-and-localization instructions contrasts with 29.1% after grounding-only fine-tuning, while joint accuracy rises from 19.2% to 43.3%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting 2B model exceeds its annotation teacher in grounding F1 on the AgroGround benchmark and external PlantSeg test set.

    Source: §6 · p. 9

    The benchmark has a single annotator and 180 dense images, and seven of the eight training sources contain public evaluation sets.

    Source: §6 · p. 9

Abstract

from arXiv · show

Agricultural visual models are typically evaluated for either recognition or localization, but reliable diagnosis requires identifying what is present and localizing the evidence. Agricultural visual question answering (VQA) datasets carry rich semantic labels but rarely link them to image regions, and adding such annotations by hand is costly at scale. We introduce AgroGround, a large-scale dataset for grounded agricultural recognition: identifying plant diseases and other agricultural targets and localizing their image regions. An automated pipeline converts the labels of eight agricultural VQA datasets into annotations for disease lesions and whole objects, producing 794,850 instruction examples. Healthy images provide negative supervision for disease queries, teaching the model to return empty predictions. We fine-tune a shared vision-language model on known-target grounding instructions combined with instructions requiring both recognition and localization. We evaluate predicted identities, regions, joint correctness, and healthy-image abstention on 1,480 human-verified images disjoint from all training data. Grounding-only fine-tuning reduces recognition accuracy from 51.8\% to 29.1\%, while adding recognition-and-localization instructions raises it to 72.6\%. With images and annotations held fixed, combining the two formats raises joint accuracy from 19.2\% to 43.3\% at comparable grounding. Healthy negatives raise abstention on healthy images to 95.0\%, and reinforcement learning improves lesion-level grounding. The resulting 2B model exceeds its annotation teacher in grounding F1 on our benchmark and on the external PlantSeg test set. AgroGround establishes a benchmark for grounded agricultural recognition, measuring joint correctness of identity and localization along with abstention on healthy images. The code is available at https://github.com/AB-Abdulla/AgroGround.

1 INTRODUCTION

AgroGround addresses the gap between agricultural systems that recognize targets and those that localize them by building spatial supervision, a human-verified benchmark, and a shared model. Its results show that instruction format, healthy negatives, and grounded reinforcement learning relate to recognition, abstention, and lesion localization performance.

Motivation

  • Grounded recognition remains under-served: agricultural VQA provides semantic labels but rarely image regions, while recognition and localization are typically evaluated separately.Manual spatial annotation is costly to scale, and existing resources rarely pair semantic and regional labels at scale.Source: §1 · pp. 1–2

Main findings

  • 72.6% recognition accuracy with recognition-and-localization instructions contrasts with 29.1% after grounding-only fine-tuning; joint accuracy rises from 19.2% to 43.3%.The three capabilities are coupled: improving one can degrade another, motivating joint training and evaluation.Source: §1 · p. 2

Additional findings

  • Healthy negatives raise healthy-image abstention to 95.0%, while reinforcement learning raises lesion-level F1 from 0.392 to 0.439.After reinforcement learning, the 2B model exceeds its annotation teacher in grounding F1 on the benchmark and PlantSeg.Source: §1 · p. 2

Dataset and supervision

  • AgroGround converts labels from eight agricultural VQA datasets into 794,850 lesion- and object-level instruction examples, using healthy images as negative disease-query examples.The pipeline uses crop and target prompts for automatic boxes, then applies masks and geometry checks.Source: §1 · p. 2

Benchmark

  • The benchmark contains 1,480 human-verified images audited for training overlap and measures identity, localization, their joint correctness, and healthy-image abstention.The evaluation uses examples disjoint from training data.Source: §1 · p. 2

2 RELATED WORK

Prior agricultural resources provide semantic text or spatial annotations, but often lack language-conditioned disease grounding or broad paired recognition and localization. AgroGround converts VQA text supervision into spatial supervision and complements existing grounding benchmarks with training data and grounded recognition.

Agricultural vision–language data

  • Agricultural multimodal corpora carry text supervision but almost no box- or mask-level ground truth, while human-annotated localization sets are smaller and fixed-class.AgroGround converts text supervision into spatial supervision and compares with PlantSeg’s expert labels.Source: §2 · p. 2

Agricultural grounding resources

  • AgroGround complements AgroVG [Li et al., 2026a] and gRef-CW [Haghighat et al., 2026] with a training corpus, human-verified evaluation, disease-level targets, and grounded recognition.AgroVG is described as evaluation-only, while gRef-CW focuses on crop and weed targets.Source: §2 · p. 3

Automatic grounding supervision

  • AgroGround converts VQA answers rather than captions and uses a curated taxonomy, crop-aware prompts, and human-verified duplicate-audited evaluation.Its geometry filters address large-area false positives from GroundingDINO and SAM proposals.Source: §2 · p. 3

Grounded diagnosis and existence awareness

  • Related work studies diagnosis and localization jointly, but AgroGround reports that changing instruction format alone can restore recognition while answers remain unchanged.The paper also contrasts its target-absent recognition and abstention rewards with related existence-awareness and grounding methods.Source: §2 · p. 3

3 THE AGROGROUND DATASET AND BENCHMARK

AgroGround defines agricultural grounded recognition across target identification, localization granularity, and abstention, then builds automatic training supervision and a human-verified benchmark. Its evaluation scores recognition, regions, joint correctness, and abstention.

3.1 TASK DEFINITION

  • A model receives an image and instruction and returns labeled boxes or an empty list when the queried target is absent; instructions ask either for a named target or target choice among four candidates.Boxes represent individual lesions or affected leaves, fruits, and plants; the distinction matters because a whole-object box does not locate individual lesions.Source: §3.1 · p. 3

3.2 AUTOMATIC ANNOTATION OF TRAINING DATA

  • 794,850 instruction examples result from an automated pipeline that extracts localizable labels from eight VQA datasets, generates boxes, and assigns lesion- or object-level supervision.Of 614,765 readable groundable samples, 579,260 receive at least one valid box; images supporting both granularities contribute examples at both levels.Source: §3.2 · pp. 4–5

3.3 HUMAN-VERIFIED BENCHMARK

  • The final benchmark contains 1,480 human-verified images: 508 lesion-level, 467 object-level, and 505 healthy, with 3,463 human boxes.Candidates were screened for shared files and perceptual near-duplicates against training positives and negatives; benchmark ground truth was frozen before model comparison.Source: §3.3 · p. 5

3.4 EVALUATION PROTOCOLS AND METRICS

  • Evaluation scores instance precision, recall, and F1 with one-to-one matching at IoU 0.5, union-region IoU, healthy-image abstention, false abstention, recognition, and joint accuracy.Joint accuracy requires the correct disease and region IoU of at least 0.5; recognition is scored on 975 positives under a closed-set candidate protocol.Source: §3.4 · p. 5

4 LEARNING GROUNDED RECOGNITION

AgroGround trains a shared vision-language model to produce labeled boxes or empty outputs using instruction formats that target recognition and grounding, healthy-negative supervision, and grounded reinforcement learning. The training design assigns different supervision signals to these capabilities.

4 LEARNING GROUNDED RECOGNITION

  • A shared model uses mixed instruction formats for recognition, healthy negatives for abstention, and grounded reinforcement learning for instance coverage.The same output format serves both instruction formats and spatial granularities.Source: §4 · p. 5, Figure 2 · p. 6

4.1 MODEL AND OUTPUT FORMAT

  • The model outputs a text list of target-labeled boxes, or an empty list when the queried target is absent.A shared vision-language model takes an image and either known-target grounding or recognition-and-localization instructions.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 MIXING GROUNDING AND RECOGNITION INSTRUCTIONS

  • Known-target instructions can teach localization without recognition because they state the target; the mixed configuration therefore uses recognition-and-localization instructions for half of positive examples.Both formats share images and boxes, so the added format supervises target choice; same-crop distractors and randomized answer positions address shortcuts and order sensitivity.Source: §4.2 · p. 6

4.3 HEALTHY-NEGATIVE SUPERVISION

  • Healthy-negative supervision queries healthy images in both instruction formats with plausible same-crop diseases and an empty-list target.Positive examples alone never demonstrate an empty answer, leaving no abstention signal.Source: §4.3 · p. 6

4.4 REINFORCEMENT LEARNING WITH GROUNDED REWARDS

  • Grounded reinforcement learning scores localization, instance recall, recognition, abstention, and valid output format against automatically generated boxes.GRPO with a GSPO loss variant starts from the mixed-plus-healthy model; prompts oversample images with many boxes to focus learning on instance coverage.Source: §4.4 · pp. 6–7

5 EXPERIMENTS

Experiments compare training setups and model variants on agricultural recognition and grounding. Mixed-format training improves recognition and joint accuracy, while healthy negatives and reinforcement learning improve abstention and lesion grounding with trade-offs.

5.1 EXPERIMENTAL SETUP AND DATASET ANALYSIS

  • The setup fine-tunes 2B and 8B vision-language models; Table 1 contrasts the training corpus with the human-verified benchmark, and healthy images supply negative examples.Baselines include GroundingDINO, zero-shot 7B reasoning-grounding systems, and zero-shot 2B and 8B models.Source: §5.1 · p. 7, Table 1 · p. 7

5.2 MAIN RESULTS AND COMPARATIVE ANALYSIS

  • 0.477 grounding F1 after RL exceeds the annotation configuration by 0.085 on the human-verified benchmark.The full pipeline also exceeds the zero-shot 7B systems, with paired intervals excluding zero.Source: §5.2 · p. 7
  • Grounding-only training reduces recognition from 51.8% to 29.1%; 88.8% of its in-list picks select the first candidate.The position probe identifies prompt-order echoing; recognition-format training yields a uniform candidate-position distribution.Source: §5.2 · p. 7, Figure 3 · p. 8
  • 72.6% recognition and 43.3% joint accuracy follow adding recognition-and-localization instructions, versus 29.1% and 19.2% with grounding-only training.The mixed model preserves disease-given grounding F1 at 0.435 versus 0.426; the paired difference is not statistically resolved.Source: §5.2 · p. 8
  • 95.0% healthy-image abstention at 4.6% false abstention follows adding 12% negatives, with recognition, grounding F1, and joint accuracy unchanged or slightly improved.Doubling the negative share from 6% changes no metric beyond seed variance.Source: §5.2 · p. 8
  • RL raises grounding F1 from 0.440 to 0.477 and lesion-level F1 from 0.392 to 0.439, while recognition falls from 74.7% to 69.2%.Joint accuracy is unchanged within noise, and healthy-image abstention under the disease-given query also falls.Source: §5.2 · p. 9
  • +0.051 F1 on PlantSeg makes the RL model the best model there, compared with its initialization.The confidence interval is [+0.042, +0.059].Source: §5.2 · p. 9

5.3 ABLATIONS AND QUALITATIVE RESULTS

  • Ablations show reward re-weighting does not remove RL’s recognition cost, while a uniform rollout mix forfeits the grounding gain.Figure 4 compares predictions with human-verified references on two dense benchmark images; the numbers indicate box counts.Source: §5.3 · p. 9, Figure 4 · p. 9

6 DISCUSSION AND CONCLUSION

AgroGround combines a large grounding resource with a human-verified benchmark, while the authors report that instruction format, healthy negatives, and grounded reward affect model capabilities. The discussion also identifies limits in annotation, benchmark scope, image coverage, and model evaluation.

6 DISCUSSION AND CONCLUSION

  • The authors identify noisy training boxes and RL rewards, a single-annotator benchmark, and 180 dense images among the study’s limitations.Abstention is evaluated on mostly leaf-centric images, and fine-tuned models share one backbone family and are evaluated above their training resolution.Source: §6 · p. 9
  • The resulting 2B model exceeds its annotation teacher in grounding F1 on both the AgroGround benchmark and PlantSeg.The authors also report that instruction format determines which capabilities fine-tuning preserves, healthy negatives enable abstention, and grounded reward improves lesion coverage.Source: §6 · p. 9
  • Reported evaluation numbers are reproducible from released prediction files and the scoring script, with benchmark disjointness checked by file path and perceptual hash.The authors state that both surviving RL checkpoints are reported.Source: REPRODUCIBILITY STATEMENT · p. 10

A CORPUS DETAILS

  • 794,850 instruction examples span fine-region lesions and coarse-object targets, with samples assigned granularity from box geometry.Images may yield an example at each level; the groundable set contains 391,758 fine-region and 403,092 coarse-object examples.Source: A CORPUS DETAILS · p. 16

B NEAR-DUPLICATE AUDIT

  • Perceptual-hash auditing removed near-duplicate candidates, and the final benchmark has zero within-set near-repeats.The audit notes that hashing does not guarantee detection of heavily cropped or rotated derivatives.Source: B NEAR-DUPLICATE AUDIT · p. 16

C ANNOTATION CONVENTIONS

  • Annotators kept 66.7% of detector proposals, while 63.2% of final boxes had no pre-annotation within IoU 0.5 and were drawn by annotators.Across 975 positive images, the detector proposed 1,838 boxes; 30.7% were deleted.Source: C ANNOTATION CONVENTIONS · p. 17

D METRIC DEFINITIONS AND INPUT RESOLUTION

This section defines how recognition, localization, joint correctness, and abstention are scored, and clarifies the matching and resolution protocols used in evaluation.

D METRIC DEFINITIONS AND INPUT RESOLUTION

  • Greedy matching pairs predictions and ground truth by descending IoU, while precision, recall, and F1 are micro-averaged across positive images.Region IoU averages the IoU of the unions of predicted and ground-truth boxes across images; empty predictions score 0 and remain in the average.Source: D METRIC DEFINITIONS AND INPUT… · p. 17
  • Recognition and joint accuracy are measured on 975 positive images; healthy images are scored only for empty-output abstention.Boxes on healthy images are not counted as false positives, a choice the authors describe as generous to models that never abstain.Source: D METRIC DEFINITIONS AND INPUT… · pp. 17–18
  • Every fine-tuned model is evaluated above its training resolution: supervised fine-tuning caps images at 50,176 pixels, while evaluation uses native resolution.The authors say all models see the same images, making the comparison fair despite the resolution difference.Source: D METRIC DEFINITIONS AND INPUT… · p. 17
  • Greedy, Hungarian, and maximum-cardinality matching produce identical F1@0.5 to three decimals for every reported row.The section reports only IoU 0.5, whereas AgroVG uses maximum-cardinality matching and reports macro-aggregated Set-F1 at thresholds 0.50 and 0.75.Source: D METRIC DEFINITIONS AND INPUT… · p. 18
  • The sample-level validation split shares photographs with training and therefore measures fit rather than generalization.Its 79,464 examples were not used for model selection.Source: D METRIC DEFINITIONS AND INPUT… · p. 18

E METRIC ROBUSTNESS AND ADDITIONAL PROTOCOLS

Metric robustness results vary by image density, and additional controls probe localization, abstention, and baseline adaptations. These protocols clarify where reported model comparisons hold and what candidate-protocol baseline scores represent.

E METRIC ROBUSTNESS AND ADDITIONAL PROTOCOLS

  • On 180 dense images, the RL model leads every baseline in F1 (0.367 vs. 0.160 for the zero-shot 8B), while the zero-shot 8B leads on 795 non-dense images (0.643 vs. 0.603–0.629).At IoU 0.75, the taxonomy-prompt detector leads (0.375 vs. 0.342), indicating the reported gains are not in box tightness.Source: E METRIC ROBUSTNESS AND ADDITIONAL… · p. 18
  • Greedy, optimal-assignment, and maximum-cardinality matching give identical F1@0.5 to three decimals, while macro averaging compresses gaps between models.Table 7 rescores the disease-given protocol under alternative metric choices.Source: E METRIC ROBUSTNESS AND ADDITIONAL… · p. 18
  • On the 508 lesion-level images, fine-only joint accuracy is 15.9 / 29.5 / 30.9 / 32.1 / 35.6% for grounding-only, +recognition, +existence, RL, and 8B models, respectively.The RL model moves ahead of the supervised 2B models; the main-text region-IoU criterion remains the strictest joint criterion.Source: E METRIC ROBUSTNESS AND ADDITIONAL… · p. 19

F RECOGNITION AND ABSTENTION CONTROLS

  • On blank images, +recognition scores 34.1% against 25% chance, while existence-trained models mostly abstain; on shuffled images, accuracy remains near real-image levels.For hard negatives—absent-disease queries on diseased images—+existence abstains on 15.2% and RL on 7.2%, versus 0.3% for +recognition.Source: F RECOGNITION AND ABSTENTION CONTROLS · p. 19

G BASELINE ADAPTATIONS

  • VisionReasoner-7B scores 40.6% accuracy and 26.8% joint accuracy under the candidate protocol, but its 61.0% first-position share suggests the heuristic often captures a list restatement.GDINO’s adaptation runs one pass per candidate and selects the highest-confidence candidate, scoring 27.6%.Source: G BASELINE ADAPTATIONS · p. 19

H FULL RESULTS

The reported mixture configurations distinguish recognition and healthy-image supervision by name and training-example count.

H FULL RESULTS

  • “+recognition” denotes the mixed configuration and “+existence” denotes mixed plus healthy examples (12%); the 6% and 12% mixtures total 678,977 and 721,977 examples.These are configuration definitions and counts, not comparative performance results.Source: H FULL RESULTS · p. 19

I PER-SOURCE RESULTS

Per-source results distinguish teacher configurations by prompt and geometry filtering, while the RL details describe a five-term grounded reward and the selected training checkpoint.

I PER-SOURCE RESULTS

  • Taxonomy-prompt GroundingDINO reaches recall 0.411 at precision 0.380, while geometry filters raise precision to 0.584 and coarse F1 to 0.700 but cut recall to 0.295.Raw prompts yield coarse F1 0.608; taxonomy prompts gain on leaf-image sources and lose on whole-plant field sources.Source: I PER-SOURCE RESULTS · p. 19

J REWARD FUNCTION AND RL DETAILS

  • The main RL run uses GRPO with a five-term reward for format, localization, multi-instance recall, recognition, and two-sided abstention, trained on 18,750 prompts with dense-image oversampling.The rollout prompts exclude benchmark images; the reward scores sampled responses against automatically generated boxes.Source: J REWARD FUNCTION AND RL DETAILS · p. 20
  • Validation reward peaks at 0.879 at step 30, then falls to 0.867 at step 170 and 0.859 at step 292; both surviving checkpoints were evaluated.Step 170 is the main result because held-out validation reward favored it over step 292.Source: J REWARD FUNCTION AND RL DETAILS · pp. 20–21

K RL ABLATIONS AND CHECKPOINT CONSISTENCY

The RL ablations report a localization–recognition trade-off, while open-set naming remains weak and a manual audit finds issues in a substantial share of sampled pseudo-labeled images.

K RL ABLATIONS AND CHECKPOINT CONSISTENCY

  • Dense oversampling drives the localization gain, but every RL configuration incurs a recognition cost, and the final checkpoint is worse on F1, recall, recognition, and joint accuracy.The final checkpoint also has higher false abstention than the earlier checkpoint.Source: Table 11 · p. 21

L OPEN-SET NAMING

  • Strict open-set name matching is at most 2.1% across models, and the authors conclude that unprompted naming is not an acquired capability.Grounding F1 without the name ranges from 0.22 for zero-shot to 0.31–0.33 for RL models.Source: L OPEN-SET NAMING · p. 22

M PSEUDO-LABEL AUDIT

  • In a 100-image pseudo-label audit, 38 images had at least one issue, and detector confidence was only weakly related to box quality.The audit found missed lesions, oversized or duplicate boxes, and boxes on something other than the labeled condition; geometry filters were used instead of confidence filtering.Source: M PSEUDO-LABEL AUDIT · p. 22

N SEED VARIANCE, SCALING, EXPERT-LABEL BASELINE, AND 8B MODEL

The scaling and transfer experiments compare pseudo-labels with expert labels, assess an 8B model, and report results on PlantSeg; the section also presents qualitative examples and release terms.

O EXTERNAL TRANSFER TO PLANTSEG

N SEED VARIANCE, SCALING, EXPERT-LABEL BASELINE, AND 8B MODEL

  • At equal training size, expert and pseudo-label models are indistinguishable on the benchmark (F1 0.376 vs. 0.372); the full corpus beats the equal-size subset by +0.054 F1.The scale gain over a 10K one-epoch run is +0.026, concentrated at lesion level (fine F1 0.27 → 0.37); subset runs are single-seed and used three epochs versus one for larger runs.Source: N SEED VARIANCE, SCALING, EXPERT-LABEL… · p. 22, Table 13 · p. 23
  • Across three seeds, the +existence model has mean F1 0.436 ± 0.004 and joint accuracy 43.5 ± 1.2; the 6%-versus-12% negative-share difference is within seed variance.The three seeds' recognition accuracies are 74.7, 73.8, and 72.4.Source: N SEED VARIANCE, SCALING, EXPERT-LABEL… · p. 22, Table 12 · p. 23
  • The 8B model's disease-given F1 is 0.429, with a paired difference of −0.011 versus 2B that is not significant; its recognition is +3.5 points higher.It produces 1.88 boxes per lesion image versus 2.23 for the 2B model.Source: N SEED VARIANCE, SCALING, EXPERT-LABEL… · pp. 22–23

P QUALITATIVE EXAMPLES

  • In selected disease-given examples, the supervised model misses the flyspeck apple that RL localizes with three boxes, while both models under-enumerate a 25-lesion anthracnose leaf.On that anthracnose image, the supervised and RL models emit 7 and 6 boxes, respectively; the figure also contrasts their boxes with GroundingDINO and human ground truth.Source: Figure 7 · p. 26
  • In selected recognition examples, the grounding-only model echoes the first-listed candidate while localizing plausibly, whereas the recognition model names the target from the image.Two of the four examples name a species or pest rather than a disease, matching the composition of benchmark positives.Source: Figure 8 · p. 27

Q LICENSES AND RELEASE

  • AgroGround releases annotations partitioned by source license, alongside the taxonomy, construction and audit scripts, scoring code, and prediction files; PlantSeg-derived artifacts are not redistributed.The annotations are keyed to source image identifiers, and the PlantSeg-trained model is not released.Source: Q LICENSES AND RELEASE · p. 28, Table 15 · p. 28

R EXTENDED LIMITATIONS

The extended limitations identify noisy pseudo-labels and constrained evaluation scope, including benchmark coverage, contamination, and model-family limitations.

R EXTENDED LIMITATIONS

  • Pseudo-label noise limits fine-grained supervision, and the RL reward inherits it; dense-stratum human boxes also retain a mild detector correlation because detector pre-annotations seeded them.The annotator configuration has recall 0.295 against human boxes, while 36.8% of final boxes are accepted detector proposals kept essentially unchanged.Source: R EXTENDED LIMITATIONS · p. 28
  • The benchmark uses one trained annotator, covers 1,480 images with only 180 dense multi-lesion field images, and does not establish headline abstention rates for field photography.Seven of eight sources are public evaluation sets or contain evaluation splits, creating contamination; all fine-tuning uses Qwen3-VL, and open-set naming is absent in all models.Source: R EXTENDED LIMITATIONS · p. 29

Related papers

Loading 2610.04425v1…