Source-linked AI summary
Can Generative AI Automate Data Extraction for Meta-Analysis? A Case Study on Intercropping Research
Zehao Lu, Xingguo Xiong, Wopke van der Werf, Thijs L. van der Plas, Ioannis N. Athanasiadis
TL;DR
Intercropping meta-analysis is difficult because studies vary and extraction is labor-intensive, while the reliability of automated extraction needs assessment against curated records. The paper compares three LLM approaches across six models and finds direct zero-shot generally strongest, though extracted data often recover relationship direction without reproducing its magnitude.
Problem
Intercropping research lacks standardization, making data construction difficult, while the reliability of LLM extraction needs evaluation against human-curated records.
Source: §1 · pp. 1–2Method
The study compares direct zero-shot prompting, a staged workflow, and a multi-agent system across six LLMs against human-curated records and a reproduced meta-analysis.
Source: §1 · p. 2Results
Direct zero-shot extraction was generally strongest, but none of the approaches was fully accurate; extracted data often recovered the positive TND–LER relationship without matching its magnitude.
Source: §5.2 · p. 7, §6 · p. 8Takeaways & Limitations
The results support generative-AI extraction for qualitative exploration, but not as a substitute for human-curated data when precise quantitative conclusions are required.
Source: §5.2 · p. 7Automatically extracted data do not reliably reproduce the magnitude or statistical certainty of the downstream relationship.
Source: §5.2 · p. 7
Abstract
from arXiv · showhide
Meta-analysis is the synthesis of information from multiple sources to arrive at an overarching conclusion. There is a large need for meta-analysis in agricultural research to synthesize what is known and analyze overarching patterns. Extracting data from published literature is, however, labor-intensive, time-consuming, and tedious, and is impeded by a lack of standardization in research design, units of measurement, and terminology. These challenges are particularly evident in the domain of crop species mixtures, also called intercropping. With the growing capabilities of LLMs, many recent attempts have focused on building systems and tools to automate data collection, yet rigorous assessment against human-labeled ground truth is often missing. In this research, we evaluate three LLM-based approaches---direct zero-shot prompting, a staged workflow, and a multi-agent system---with six open-weight models to extract data from the intercropping literature. The results are evaluated against the manually curated ground truth and through a downstream statistical analysis. Overall, direct zero-shot prompting is the strongest and most consistent approach, achieving the highest mean similarity-adjusted F1 of 0.577, although none of the approaches is close to fully accurate. In the downstream analysis, most model--approach combinations recover the direction of the relationship between the predictor and outcome variables, but do not estimate its magnitude accurately.
1 Introduction
Intercropping evidence is difficult to synthesize because studies vary in design, terminology, conditions, and units, while information for each trial record is scattered across papers. This study compares LLM extraction approaches against human-curated records and tests whether they reproduce meta-analytic findings.
1 Introduction
- Intercropping benefits vary across crop combinations, pedoclimatic conditions, and management practices, so evidence from different environments must be synthesized.Intercropping is growing two or more crop species together to improve resource-use efficiency and ecological resilience.Source: §1 · p. 1
- Intercropping datasets are difficult to construct because research lacks standardization and reported metrics vary.A trial may report many measurements, with design, results, units, and exceptions scattered across text, tables, captions, and footnotes; manual curation takes substantial effort.Source: §1 · pp. 1–2
- Earlier NLP pipelines can propagate errors between stages and struggle to assemble complete experimental records from information spread across a document.LLMs offer broader-context extraction flexibility, but whether agentic designs improve extraction from complex agricultural studies remains open.Source: §1 · p. 2
- The study compares three LLM extraction approaches on multi-level trial structures and cross-table attribute binding, and evaluates their effects on reproducing published meta-analytic findings.It also formulates agricultural trial data structuring as an information-extraction task and establishes an annotated evaluation benchmark.Source: §1 · p. 2
2 Problem Statement
The task is to extract schema-defined experimental comparisons while retaining links among their values. Agricultural studies make this difficult because relevant relationships span treatments, crops, conditions, and distributed table content.
2.1 Structured Data Extraction
- Each record represents one experimental comparison with fixed fields for conditions, treatments, outcomes, and effects; extraction must preserve links among values from that comparison.The task assumes textual content extracted from a PDF; information available only in images is out of scope, and records may draw on different parts of a paper.Source: §2.1 · p. 2
2.2 Challenges Specific to Agricultural Studies
- Intercropping trials can span sites, seasons, and treatments, requiring mixtures to be matched with two sole-crop comparators under comparable management.Crop composition, spatial design, management, timing, location, and yields must remain linked despite nested table headers and differing units; broken links can distort downstream analysis.Source: §2.2 · p. 2
3 Data & Experiment
The study repeats the data-extraction stage of a reference intercropping meta-analysis using three LLM approaches, then reproduces its statistical analysis. Its setup uses mapped reference-study records, a 42-variable extraction subset, and schema variables central to the reference analysis.
3 Data & Experiment
- The experiment repeats the reference study’s extraction stage with three generative-AI approaches and reproduces its linear mixed-effects analysis on their outputs.The source papers were converted from PDF to Markdown for model input.Source: §3 · p. 3, §3.1 · p. 3
3.1 Data
- The reference ground truth contains 746 records from 100 papers, but extraction was run on the 90 papers reliably mapped to ground-truth studies and evaluated on 42 variables.The full ground-truth dataset contains 135 variables, including information outside the extraction task.Source: §3.1 · p. 3
3.2 Extraction Schema and Key Variables
- TND measures the difference between relay crops’ growing periods, while LER measures intercropping land-use efficiency relative to growing the crops separately.TND ranges from 0 for complete overlap to 1 for complete separation; LER above 1 indicates greater land-use efficiency under intercropping.Source: §3.2 · p. 3
3.2.1 Analysis in the Reference Study
- The reference analysis asks whether LER increases with TND and models their relationship using linear mixed-effects regression with random intercepts for experiments nested within studies.The model’s β1 is the change in LER per unit increase in TND.Source: §3.2.1 · p. 3
3.3.1 Extraction Approaches
- The study compares direct zero-shot prompting, a staged workflow, and a multi-agent system; the staged workflow separates evidence identification from record construction.The selected plan-and-execute agent design uses tools to identify and label evidence before producing and refining records.Source: §3.3.1 · p. 4
3.3.2 Language Model Choice
- Six open-weight models available through SURF were chosen in part to avoid sharing unpublished scientific work with commercial LLM providers.The tested models were QWEN3.6-27B-FP8, QWEN3.6-35B-A3B-FP8, QWEN3.5-122B-A10B-NVFP4, GEMMA-4-31B-IT-NVFP4, LLAMA-70B, and GPT-OSS-120B.Source: §3.3.2 · p. 4
4 Evaluation
The evaluation compares extracted records with human-curated ground truth and tests whether imperfect extractions reproduce the reference analysis. It assesses field-level errors, aggregates extraction quality, and fits the downstream regression using eligible records.
4 Evaluation
- The evaluation has two steps: compare extraction outputs with human-annotated ground truth, then reproduce the human researchers’ analysis using extracted data.The second step examines whether imperfect extractions yield similar conclusions and how errors affect the findings.Source: §4 · p. 4
4.1 Evaluation Against Ground Truth
- Field-level comparisons use data-type-specific rules reviewed with the senior author who helped curate the ground-truth dataset.The pipeline aligns records, applies comparison rules, and aggregates results.Source: §4.1 · p. 4
4.1.1 Error Types
- Mismatches are classified as incorrectness, hallucination, or incompleteness, measuring value disagreement, unsupported predictions, and omitted ground-truth values, respectively.Similarity, factual rate, and completeness rate correspond to these three error types.Source: §4.1.1 · p. 4
4.1.2 Evaluation Metrics
- The combined score uses field-level similarity, factual rate, and completeness rate, with a harmonic mean of similarity-adjusted factual rate and completeness.Comparison rules vary by field: they use normalized text overlap, numerical tolerances and unit conversion, date normalization, and Boolean or missing-value rules.Source: §4.1.2 · p. 5
4.1.3 Evaluation pipeline
- Records are greedily matched one-to-one by descending average field similarity, retaining the better original or swapped crop order and requiring no minimum similarity.Unmatched predicted fields count as unsupported extractions, unmatched ground-truth fields as omissions, and paper-level metrics are averaged across papers.Source: §4.1.3 · p. 5
4.2 Reproducing the Reference Analysis
- For each model and approach, the reproduced mixed-effects regression uses LER as the response, TND as the fixed-effect predictor, and experiment-within-study random intercepts.Records lacking usable LER or all four required yields, or lacking TND, are excluded; matching uses experimental-design information only, with ground-truth IDs used to recover grouping structure rather than outcome values.Source: §4.2 · p. 6
5 Results
Across the field-level and downstream evaluations, added workflow complexity did not consistently improve results. Automatically extracted data often recovered the relationship’s direction, but not its magnitude or statistical certainty.
5 Results
- The experiment applied the approaches to the same 90 papers and counted runs producing a parsable table with at least one record.The reported run outcomes distinguish non-empty, empty, and missing outputs.Source: §5 · p. 6, Table 1 · p. 6
5.1 Field-Level Extraction Results
- The direct approach achieved the highest similarity-adjusted F1 for all six models, while more complex approaches generally performed worse.The comparison includes papers with output from at least one approach; missing or empty outputs count as empty predictions.Source: §5.1 · pp. 6–7, Figure 2 · p. 6
- On the common-output subset, the direct approach led for four of six models, with the workflow and multi-agent system each slightly ahead for one model.The restricted comparison still provided no consistent evidence that increased method complexity improves extraction quality.Source: §5.1 · p. 7, Figure 3 · p. 7
5.2 Downstream Reproduction Results
- The direct approach produced positive TND coefficients for all six LLMs, and five estimates were significantly greater than zero.Qwen3.6-35B produced significant positive coefficients with all three approaches, but estimates exceeded the 90-study ground-truth coefficient; direct Qwen3.6-27B was numerically closest at β = 0.295, though its confidence interval included zero.Source: §5.2 · p. 7, Figure 4 · p. 7
- Automated extraction often recovered the relationship’s direction, particularly with the direct approach, but did not reliably reproduce its magnitude or statistical certainty.The authors support qualitative exploration, not replacing human-curated data when precise quantitative conclusions are required.Source: §5.2 · p. 7, Figure 4 · p. 7
6 Discussion & Conclusion
Direct zero-shot extraction was generally strongest, but no approach was fully accurate. The discussion highlights pipeline complexity, the time demands of evidence synthesis, and limits on estimating effects and transferring performance.
6 Discussion & Conclusion
- Direct zero-shot extraction was generally strongest, but none of the approaches was fully accurate.It recovered a positive TND–LER relationship but not the reference effect’s magnitude.Source: §6 · p. 8
6.1 Complexity Trap
- In eight papers, the workflow sometimes lost context and MAS sometimes carried initial errors forward, creating risks of omission, duplication, and misassociation.MAS produced eight records where ground truth had three, fifteen where it had two, and none for two other papers in the sample.Source: §6.1 · p. 8
6.2 Toward More Efficient Evidence Synthesis
- A small meta-analysis may take about six months, while one covering 50–150 studies may require two to three years of a PhD project.The authors suggest standardizing and streamlining synthesis as experience grows.Source: §6.2 · p. 8
- LLMs could reduce manual work, but outputs still need checking; direct extraction recovered a positive TND–LER relationship across six models, with effects generally larger than the reference and varying in precision.Many records and fields remained missing, incorrect, or wrongly associated.Source: §6.2 · p. 8
6.3 Relevance Beyond Intercropping
- Direct zero-shot prompting may transfer to other domains, but precision remains limited, and recovering relationship direction appears easier than estimating magnitude accurately.The authors suggest domain-specific post-training may be needed to improve performance.Source: §6.3 · p. 8
Generative AI usage
AI coding tools supported parts of the codebase and minor manuscript copy-editing; the authors say the code was manually verified.
Generative AI usage
- Claude, Cursor, and Codex co-wrote parts of the codebase and supported minor copy-editing; the authors manually verified all code.The authors take full responsibility for the methods and results presented.Source: Generative AI usage · p. 8
A Schema Definition
Table 2 defines the 42 fields in an extracted intercropping record.
A Schema Definition
- Table 2 provides definitions for the 42 fields in an extracted intercropping record.Source: Table 2 · p. 11
B Prompts and Agent Configurations
The appendices describe direct prompting, a two-step evidence-labeling workflow, and the MAS agents and tools.
B.1 Direct Zero-Shot Prompt
- The direct prompt replaces [SCHEMA] with the 42-field schema and [PAPER CONTENT] with the paper text, then requests structured output in that schema.Source: §B.1 · p. 12
- The direct prompt instructs the model to scan paragraphs, tables, captions, footnotes, and supplements for schema-field evidence.It also asks the model to gather reported year, treatment, fertilization, cropping system, species, and plot context for each anchor.Source: §B.1 · p. 12
- Direct-prompt record construction treats distinct table rows or factor combinations as separate records, preserves original units, and uses null for unreported fields.It instructs the model to remove duplicates only when year, treatment, cropping system, and species context all match.Source: §B.1 · p. 12
B.2.1 Step 1: Evidence Labelling
- The workflow’s first step retains schema-relevant text blocks and adds inline XML tags so the next step can construct records by experimental row.Source: §B.2.1 · p. 12
- The labelling prompt directs retention of complete results tables, captions, footnotes, and methods text containing schema-relevant values, while excluding unrelated sections.It directs the labeller to tag relevant numbers and labels without rewriting source text.Source: §B.2.1 · pp. 12–13
B.2.2 Step 2: Record Extraction
- Step two reuses the direct extraction prompt on the labeled text, while instructing the model to examine table rows and captions because missing tags do not prove information is absent.Source: §B.2.2 · p. 13
- The second-step instructions favor distinct records for supported table rows and use null only after relevant evidence has been examined.They advise preserving separate rows when year, treatment, or nitrogen level differs, unless the schema requires aggregation.Source: §B.2.2 · p. 13
B.3 Agents and Tools
- The MAS consists of a planning agent followed by four execution agents; only the labeller has a callable tool.The other agents operate on the document and intermediate outputs in their context.Source: §B.3 · p. 13