PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating human-AI workflows for field research in viticulture

Niko Carvajal Janke, Daoyuan Jin, Shivranjani Baruah, Nicholas Gunner, Jacob Maus, Yu Jiang, Kaitlin M. Gold

arXiv:2610.07669v1cs.HCcs.AIcs.CY

TL;DR

The study asks whether longitudinal vineyard records and remote sensing can support red-leaf forecasting and virus-testing priorities under limited scouting and testing capacity. Researchers evaluated two live workflows using Aleks v1, and at 45% survey effort model-adaptive scouting achieved higher recall than adaptive scouting.

  • Problem

    The study asks whether longitudinal vineyard records and remote sensing can support site-specific red-leaf forecasting and whether higher scores associate with more frequent virus detection, amid limited scouting and testing capacity.

    Source: §1.3 · p. 3, §1.2 · p. 3

  • Method

    Researchers evaluated two live workflows using Aleks v1, a multi-agent system, for model development and sampling-plan design.

    Source: §1.3 · p. 3, §2.3 · p. 5

  • Results

    At 45% survey effort, model-adaptive scouting achieved 94.13% recall, compared with 85.85% for adaptive scouting.

    Source: §3.1.3 · p. 9

  • Takeaways & Limitations

    AI interactions should be assessed by methodological quality and how well their deliverables advance research objectives under project-specific constraints.

    Source: Conclusions · p. 16

    Internal 2024 performance estimates were not independent because imputation and SMOTE preceded the training/evaluation split.

    Source: §3.1.1 · p. 8, S2.4.1 – Developing a vine-scale… · p. 25

Abstract

from arXiv · show

We assessed the value of two live human-AI interactions in a precision disease control project in California vineyards. The project tested whether 2021-2024 commercial scouting records and remote-sensing measurements across 140 hectares could support 2025 red-leaf symptom forecasting for prioritized scouting and virus testing. In Workflow 1, Aleks v1, a multi-agent research system, developed forecasting models with iterative human refinement. We applied Aleks's 2024 vine-scale model to updated 2025 predictors and evaluated red-leaf forecasts against independent 2025 scouting. In retrospective simulations surveying 45% of all vine positions, adding model-informed row prioritization to adaptive scouting increased the encountered proportion of newly recorded red-leaf observations from 85.8% to 94.1%. Within-block scouting comparisons suggested the model mainly improved scouting allocation among blocks. Despite unreliable internal 2024 performance estimates from synthetic oversampling before train/test splitting, Aleks developed an informative vine-scale model in 145 minutes, increasing throughput and answering our research questions. In Workflow 2, we assessed whether higher model-score vines had more frequent virus detection, and whether Aleks could infer this sampling goal from a general prompt with data and literature. Aleks's plan prioritized balanced vineyard and model score coverage, while our plan prioritized field efficiency and high-model-score oversampling. Aleks's and our plans yielded 41/50 (82%) and 97/100 (97%) sampled vines. Aleks's plan omitted instructions for replacing missing vines, limiting implementation and operational value. Five of 137 sampled vines tested positive for grapevine red blotch virus (model score ROC AUC 0.735). These findings support assessing AI interactions by how well they advance field research objectives under live, project-specific constraints.

1 – Introduction

This introduction argues that live human-AI workflow evaluations can test whether AI deliverables remain useful under project-specific field constraints. It motivates site-specific decision support because virus symptoms and testing resources limit comprehensive vineyard surveys, and asks whether vineyard records and remote sensing can support forecasting and virus-testing priorities.

1.1 – Artificial intelligence systems for agricultural field research

  • Live workflow evaluations can assess whether AI task outputs advance research objectives beyond standalone task performance, under seasonal, site-specific, and commercial constraints.The proposed framework specifies each task’s objective, constraint, context and inputs, and criteria for an operationally meaningful deliverable.Source: §1.1 · p. 2
  • Precision-viticulture sensing and digital records support site-specific disease prediction, but integrating their outputs can create data-engineering bottlenecks amid narrow observation and sampling windows.These conditions motivate evaluating whether human-AI workflows can help teams explore modeling, validation, and sampling designs.Source: §1.1 · p. 2

1.2 – Mitigating grapevine viruses in California viticulture

  • Symptoms can overlap or be absent, so visual scouting cannot always identify virus infection; virus-specific testing is needed but its cost and labor can limit coverage.Testing timing and methods can also affect the reliability of negative results, motivating decision support for finite scouting and testing resources.Source: §1.2 · pp. 2–3

1.3 – Longitudinal vineyard records for site-specific decision support

  • The study asks whether longitudinal vineyard records and remote-sensing products can support site-specific red-leaf forecasting and whether higher model scores associate with more frequent virus detection.Spatially explicit records can inform virus management, while scouting labor and testing capacity constrain vineyard-wide coverage.Source: §1.3 · p. 3

2 – Methods

The methods evaluate two human-AI workflows using vineyard records and remote sensing: forecasting red-leaf observations for scouting and designing a virus-testing sample. Forecasts are assessed against independent scouting, while sampling-plan design, implementation, and testing outcomes are compared with a domain-scientist plan.

2.1 – Evaluating AI support through live research workflows

  • The study assessed AI support by whether each deliverable addressed a research bottleneck and intent, reviewing methods and scripts alongside operational outcomes.Workflow 1 used retrospective scouting simulations against independent full scouting records; Workflow 2 compared plan design, implementation, and virus-testing outcomes with a domain-scientist plan.Source: §2.1 · pp. 3–4

2.2 – Vineyard management records and dataset preparation

  • Records covered 296,489 vine positions across 110 commercial blocks and about 140 hectares, with scouting observations and repeated drone-derived measurements aligned to 10-meter, 3-meter, and vine-scale units.Vine-scale positions served as prediction units, with observations assigned within 1.5 m; one scouting record could contribute to multiple positions to account for GPS uncertainty. Aleks v1 used domain-scientist, data-analyst, and machine-learning-engineer agents communicating through shared memory.Source: §2.2 · pp. 4–5, §2.3 · p. 5

2.3 – AI system design and interaction

  • For model development, Aleks received the vineyard dataset and selected literature, and each run was limited to 20 iterations without a prescribed model type, predictors, imbalance treatment, or metric.For sampling-plan design, it received the model outputs, dataset, prior workflow materials, and literature, with a five-iteration limit; the prompt did not specify the intended score-group comparison or rarity of positive vines.Source: §2.3 · p. 5

2.4.1 – Model development and 2025 forecasting

  • The team selected vine-scale prediction units because scouting, sampling, and management decisions occurred at individual vines, then applied the preserved final model to updated 2025 inputs for all mapped positions.The final Auto-sklearn ensemble assigned continuous scores from 0 to 1, with scores of at least 0.50 classified as predicted positives.Source: §2.4.1 · p. 6

2.4.2 – Validation of 2025 red-leaf forecasts using independent scouting

  • Independent 2025 scouting records defined observed positives and negatives for both the newly recorded and all-record evaluations, and model ranking and threshold metrics were calculated across the management area and within the sampling vineyard.The team used ROC AUC, average precision, prevalence-normalized average-precision enrichment, block-adjusted ROC AUC, precision, and recall.Source: §2.4.2 · p. 6

2.4.3 – Whole-row scouting simulations

  • Five whole-row scouting strategies were simulated through 45% survey effort, comparing random, systematic, and adaptive routes with model-prioritized and model-adaptive row selection.Model prioritization ranked rows by the mean score of their highest-scoring 10% of vines; model-adaptive scouting combined this ranking with expansion around positive rows, and the strategies were compared with random routes.Source: §2.4.3 · pp. 6–7

2.5.1 – Sampling plan development

  • The sampling vineyard contained 12,792 candidate vines across 78 rows and two blocks; the domain-scientist plan targeted 100 vines, while Aleks’s plan targeted 50.Aleks selected 10 positions from each of five score quintiles with 15-meter minimum spacing, but gave no instructions for replacing empty or unsuitable positions. The domain-scientist plan selected 25 vines from each of four score and proximity subclasses, excluding positions with 2024 red-leaf records.Source: §2.5.1 · p. 7

2.5.2 – Field implementation and virus testing

  • The team collected petiole samples from 97 domain-scientist-plan vines and cane samples from those vines and 41 Aleks-plan positions, then compared plan characteristics, implementation, and virus-test outcomes.Cane tissue was pooled by vine and tested for GRBV by RT-PCR and GLRaV-3 by ELISA; subsequent score analyses used December cane results and accounted for the one vine selected by both plans.Source: §2.5.2 · pp. 7–8, §2.5.3 · p. 8

3 – Results

Aleks rapidly developed a vine-scale forecasting model, and retrospective model-informed scouting increased recall at 45% survey effort. In a separate sampling workflow, GRBV-positive vines had higher model scores, while the Aleks plan was less complete than the domain-scientist plan.

3.1.1 – Aleks designed red-leaf forecasting models at multiple prediction scales

  • After stakeholder review, Aleks selected vine-scale prediction units and completed 20 model-design iterations in 145 minutes across 296,489 vine positions.The final 16-predictor model’s reported 97% internal accuracy was not independent because imputation and SMOTE preceded the evaluation split; it was applied to updated 2025 inputs.Source: §3.1.1 · pp. 8–9

3.1.2 – Vine-scale model scores distinguished 2025 red-leaf observations

  • Across 140 ha, New in 2025 ROC AUC was 0.659 dataset-wide versus 0.516 block-adjusted; average precision was 0.65% against 0.35% prevalence.The model’s scores distinguished observed-positive from observed-negative positions more strongly across the management area than within blocks.Source: Table 1 · p. 9, §3.1.2 · p. 9

3.1.3 – Vineyard management data enabled scouting comparisons

  • At 45% survey effort, model-adaptive scouting achieved 94.13% recall versus 85.85% for adaptive and 45.02% for random scouting.Model-based strategies also exceeded every dataset-wide random route at 10%, 20%, and 45% effort; within-block gains were smaller and effort-dependent.Source: §3.1.3 · p. 9, Figure 2 · p. 10

3.2.1 – AI and domain-scientist plans established distinct priorities

  • The domain-scientist plan emphasized high-score, symptom-proximal, labor-efficient sampling; Aleks emphasized score coverage and dispersion, completing 41/50 targets (82%) versus 97/100 (97%).Aleks provided no replacement instructions for missing or young vines; median tested vines per sampled row were one under Aleks and 10 under the domain-scientist plan.Source: §3.2.1 · pp. 10–11, Table 2 · p. 12

3.2.2 – GRBV-positive vines had higher model scores in 2025

  • Among 137 unique sampled vines, five were GRBV-positive; positives had higher median model scores (0.525 versus 0.407) and ROC AUC was 0.735.Four positives had scores ≥0.50, and all five lacked recorded red-leaf symptoms; GRBV classifications agreed for all 97 paired petiole and cane samples, while no GLRaV-3 was detected in cane samples.Source: Table 3 · p. 12, §3.2.2 · p. 12

4 – Discussion

The case study found that vineyard records and remote sensing can support site-specific scouting decisions, while the value of AI deliverables depended on project objectives and field constraints. The discussion recommends live workflow evaluation that considers both methodological quality and operational value.

4.1 – Vineyard records enabled site-specific decision support modeling

  • Model forecasts were more useful for allocating scouting effort among blocks than for prioritizing rows within blocks, while adaptive row scouting achieved high recall without model scores.The authors note that performance at other sites or with other datasets remains to be evaluated.Source: §4.1 · p. 13
  • Existing vineyard records were not collected for vine-scale decision support, and future forecasting datasets could track vine-level diagnoses, management, and symptom histories.The proposed features could help distinguish local red-leaf patterns associated with disease, abiotic stress, or other damage.Source: §4.1 · p. 13
  • The tentative association between higher model scores and GRBV detection in 137 vines, including five positives, motivates testing whether score-guided sampling increases positive detections per test.Future validation should account for assay choice, sampled tissue, and sampling date because negative tests do not necessarily establish lack of infection.Source: §4.1 · p. 13

4.2 – The value of AI support differed between workflows

  • In Workflow 1, bounded AI modeling with iterative human review addressed a throughput bottleneck despite errors in the final model design.The authors report that the modeling work supported research questions and enabled the team to focus on dataset preparation and molecular sampling plans.Source: §4.2 · p. 14
  • Reviewing modeling scripts could address Aleks’s errors, but the required time and expertise would have reduced the value of AI support in this project.The authors suggest testing whether supplying relevant peer-reviewed literature helps Aleks avoid such modeling mistakes.Source: §4.2 · p. 14
  • For Workflow 2, Aleks favored balanced coverage across model-score quintiles and vineyard locations, whereas scientists prioritized higher-score vines; missing-vine instructions and dispersed selections limited implementation.The authors suggest that explicit downstream objectives and vineyard-specific vine-status information could improve future sampling designs.Source: §4.2 · p. 14

4.3 – Live workflow evaluations can guide the development of reliable AI support

  • Live workflow evaluation linked deliverable design to field implementation and project outcomes, assessing methodological quality alongside operational value.The authors emphasize that methodological oversights need not prevent useful support, while sound precedent alone does not ensure operational value.Source: §4.3 · p. 15
  • The evaluation approach balanced resources for assessing AI support against those needed to meet project objectives, using independent 2025 scouting and literature review for Workflow 1.The authors propose repeated evaluations across systems, interaction protocols, and contexts to identify approaches that reliably support field research.Source: §4.3 · p. 15

Conclusions

This case study evaluated AI support through live field-research workflows. Its findings support assessing both the methodological quality of AI deliverables and how well they advance research objectives under project-specific constraints.

Conclusions

  • Workflow 1 AI-supported model development addressed a throughput bottleneck and helped assess vineyard records and remote sensing for site-specific virus decision support.Workflow 2 compared virus detection rates across model scores, but Aleks’s sampling plan was only partially implementable and did not account for the potential rarity of virus-positive vines.Source: Conclusions · pp. 15–16
  • The findings support evaluating AI interactions by both deliverable methodological quality and how well deliverables advance research objectives under project-specific constraints.Repeated workflow evaluations across projects could help identify interaction protocols that reliably support agricultural field research.Source: Conclusions · p. 16

Author contributions (CRediT)

The author contributions span conceptualization, methods, analysis, investigation, data curation, resources, software, and writing.

Author contributions (CRediT)

  • Niko Carvajal Janke contributed to conceptualization, methodology, analysis, investigation, data curation, and writing.Source: Author contributions (CRediT) · p. 16
  • Daoyuan Jin contributed to methodology, software, investigation, analysis, and writing; Nicholas Gunner contributed to data curation and writing.Source: Author contributions (CRediT) · p. 16
  • Shivranjani Baruah, Jacob Maus, Yu Jiang, and Kaitlin M. Gold contributed to methods or resources, with conceptualization and writing contributions also reported for some authors.Source: Author contributions (CRediT) · p. 16

Funding

The work received support from NASA Acres, Cornell AgriTech, Cornell CALS, and USDA NIFA.

Funding

  • Support came from the NASA Acres Consortium, Cornell AgriTech, the Cornell CALS Moonshot Seed Grant Program, and USDA NIFA’s VitisGen3 project.The funders had no role in study design, data collection, analysis or interpretation, manuscript preparation, or the decision to submit.Source: Funding · p. 16

Supplemental Methods

The methods integrated vine positions, remote-sensing measurements, and scouting records into spatially aligned vine-scale and coarser datasets.

S2.2 – Vineyard management records and dataset preparation

  • Mapped vine positions served as a shared spatial reference for integrating remote-sensing measurements and scouting records across years.The resulting vine-scale records were also aggregated into coarser prediction units.Source: S2.2 – Vineyard management records and… · p. 23
  • Inverse-distance weighting estimated EVI, canopy area, canopy width, and vine presence at reference positions from up to 10 source points within 3 m.The weighting used power 2; unestimable measurements remained missing.Source: S2.2 – Vineyard management records and… · p. 23
  • Scouting observations within 1.5 m were linked to each vine position, with yearly red-leaf and girdled-vine records counted separately.A single observation could link to multiple vines; 46.4%–94.1% of red-leaf observations were linked within mapped blocks, depending on year.Source: S2.2 – Vineyard management records and… · p. 23
  • The 3 m × 3 m and 10 m × 10 m datasets summed scouting counts and averaged remote-sensing values within each prediction unit.Units without mapped vine positions were excluded.Source: S2.2 – Vineyard management records and… · p. 23

S2.3 – AI system configuration and prompting

Researchers supplied Aleks with task-specific prompts, project data, and publications, then ran its tasks within fixed iteration limits. The prompts specified different objectives and constraints across the two workflows.

S2.3 – AI system configuration and prompting

  • Researchers supplied Aleks with task-specific role and objective prompts, relevant publications, project datasets, prior records, and fixed iteration limits.Once started, each run proceeded without further human input, while shared memory passed earlier agent outputs to later stages.Source: S2.3 – AI system configuration and… · p. 24
  • The three Workflow 1 runs used a common analyst role prompt and 30 publications, while researchers reviewed model designs between runs.The retained task prompt asked for prediction of 2024 red-leaf disease from data before August 2024 but left model development to Aleks.Source: S2.3 – AI system configuration and… · p. 24
  • Workflow 2 supplied the sampling-plan task with the retained model script, run log, 2025 outputs, vineyard dataset, and 44 publications.The role prompt framed the task as designing rigorous, unbiased validation sampling.Source: S2.3 – AI system configuration and… · p. 24
  • The Workflow 2 prompt named the vineyard, sample size, and general validation goal but omitted the intended comparison of virus detection across model scores.The run had five iterations and no intermediate human review.Source: S2.3 – AI system configuration and… · p. 24

S2.4 – Workflow 1: Model development and evaluation

Workflow 1 developed a vine-scale model, applied it to updated 2025 predictors, evaluated forecasts against scouting, and compared whole-row scouting strategies retrospectively.

S2.4.1 – Developing a vine-scale red-leaf forecasting model using Aleks

  • The final vine-scale model classified 2024 red-leaf presence using 16 predictors, with predictors and outcomes prepared across 20 iterations.The model used SMOTE before the training/evaluation split, so the evaluation records included synthetic examples and were not independent.Source: S2.4.1 – Developing a vine-scale… · p. 25

S2.4.2 – Forecasting red-leaf observations for harvest 2025

  • For 2025 forecasts, researchers advanced the model’s year-specific inputs, applied the saved scaler and model without refitting, and produced scores and classes for 296,489 vine positions.Missing predictors were imputed using their medians across the 2025 dataset.Source: S2.4.2 – Forecasting red-leaf… · p. 26

S2.4.3 – Evaluating 2025 red-leaf observation forecasts

  • Forecast evaluation used completed 2025 scouting records for all positions and separately for positions without red-leaf records from 2021–2024.Metrics included ROC AUC, average precision, threshold-based precision and recall, and block-adjusted ROC AUC.Source: S2.4.3 – Evaluating 2025 red-leaf… · p. 26

S2.4.4 – Retrospective comparison of whole-row scouting strategies

  • Five whole-row strategies were compared across survey effort, including model-ranked rows and an adaptive strategy that expanded around red-leaf observations.Model row scores averaged the scores of each row’s highest-scoring 10% of vines; adaptive expansion added up to two neighboring rows per side within a block.Source: S2.4.4 – Retrospective comparison of… · p. 27
  • The simulations measured vine-position coverage, not travel, elapsed time, or labor, and route ranges reflected simulated variation rather than variation across vineyards or years.Strategies were compared with random routes and, in within-block comparisons, row selection was randomized while matching block-level row counts.Source: S2.4.4 – Retrospective comparison of… · pp. 27–28

S2.5 – Workflow 2: Sampling plan design and evaluation

Workflow 2 compared independently designed sampling plans, implemented field sampling and virus testing, and assessed model-score associations with virus detection.

S2.5.1 – Sampling plan development

  • Sampling was restricted to one Cabernet Sauvignon vineyard with a model-score gradient, and Aleks proposed four designs before selecting positions across five score quintiles.Its final positions differed from the proposed quintile allocation, and the selection script was not retained, preventing exact reproduction.Source: S2.5.1 – Sampling plan development · p. 28
  • The domain-scientist plan selected 25 vines in each of four subclasses defined by score threshold and proximity to 2024 red-leaf observations.It targeted 100 positions across 10 rows, with planned minimum spacing of 1.8 m.Source: S2.5.1 – Sampling plan development · pp. 28–29

S2.5.2 – Field implementation and virus testing

  • Field replacement sampling was specified for the domain-scientist plan, but Aleks’s plan gave no replacement instructions, leaving nine empty or young-vine positions unsampled.Three domain-scientist targets also remained unfilled after replacements.Source: S2.5.2 – Field implementation and… · p. 29

S2.5.3 – Sampling plan and model score evaluation

  • December cane results were used to assess GRBV detection by model-score group and calculate ROC AUC across five positive and 132 negative vines.The analysis also compared September petiole and December cane classifications for the 97 domain-scientist-plan vines.Source: S2.5.3 – Sampling plan and model score… · p. 29

Supplemental Figures and Tables

The supplement describes how 2025 vine-scale forecasts and whole-row scouting strategies were evaluated. It specifies the performance metrics, comparison designs, and predictor updates used in those evaluations.

Supplemental Figures and Tables

  • Figure S1 contrasts mean strategy-minus-random recall differences and recall exceedance across 5%–45% survey effort.Panels A and C use dataset-wide random selection; B and D use within-block random selections with matched row counts, which did not always preserve realized effort exactly.Source: Figure S1 · p. 30
  • Table S1 defines 2025 vine-scale performance metrics for all positions and positions newly recorded in 2025, including block-adjusted ROC AUC.For the full dataset, the 44 blocks in “All in 2025” and 42 in “New in 2025” contained all observed-positive positions and 38.94% and 39.59% of evaluated positions, respectively.Source: Table S1 · p. 31
  • The scouting simulations compare strategy recall with dataset-wide random selection and within-block-matched random selection across survey effort levels.Recall is the percentage of vine positions with newly recorded 2025 red-leaf observations within surveyed rows; the tables report results at 10%, 20%, and 45% effort.Source: Figure S1 · p. 30, Table S2 · p. 32, Table S3 · p. 33
  • Table S2 reports survey effort, recall, route comparisons, and Monte Carlo standard errors for five whole-row scouting strategies across the 140-ha management area.Random, systematic, and adaptive scouting results summarize 5,000 simulated routes; each model-based strategy has one route.Source: Table S2 · p. 32
  • Table S3 compares four strategies with within-block random selections matched to each route’s row counts at 10%, 20%, and 45% survey effort.Systematic and adaptive results summarize 5,000 routes, while deterministic model-based strategies have single-route values.Source: Table S3 · p. 33
  • For 2025 forecasting, year-specific predictors were advanced one year, while longitude, latitude, cultivar, and vine spacing remained unchanged.Model-development inputs covered 2021–2024 and were aligned with 2024 red-leaf outcomes.Source: Table S4 · p. 34
Loading 2610.07669v1…