Source-linked AI summary
RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, Ranveer Chandra
TL;DR
The paper examines how RAG and fine-tuning can incorporate domain-specific knowledge into LLM applications, addressing the need for location-specific agricultural assistance. It develops and evaluates a pipeline that extracts and structures documents, generates Q&A data, uses RAG and fine-tuning, and measures their tradeoffs. The study reports complementary benefits from both approaches while identifying costs and evaluation challenges.
Problem
Agricultural AI applications lack specialized training data, while farmers need answers grounded in local conditions, varieties, livestock, and current information.
Method
The paper builds a pipeline that structures PDF content, generates and evaluates industry-specific Q&A pairs, uses RAG for answer generation, and fine-tunes multiple LLMs.
Results
RAG and fine-tuning both improve LLM performance in agricultural settings, with RAG effective for contextually relevant data and fine-tuning producing precise, succinct outputs.
Takeaways & Limitations
The study establishes agriculture-focused baselines and a reusable pipeline for adapting LLMs to industry-specific contexts across multiple models.
Takeaways & Limitations
Question-quality metrics remain difficult to define because natural language is complex and judgments about what constitutes a good question are subjective.
Abstract
from arXiv · showhide
There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.
1 Introduction
The paper addresses agriculture’s need for location-specific AI assistance by building an industry-specific LLM pipeline and evaluating RAG and fine-tuning across agricultural questions. It positions this work as a foundation for understanding model performance and adaptation in agriculture.
- Motivation: Agriculture has limited AI adoption partly because specialized training data is scarce and farmers need locally specific, current knowledge.General tools may not adequately answer questions involving local conditions, crop varieties, and livestock.
- Contribution: The paper proposes creating industry-specific AI copilots that generate high-quality questions and answers from curated agricultural documents.Documents are cleaned, structured, and used to generate, evaluate, and filter Q&A pairs.
- Approach: The agriculture pipeline combines Q&A generation, RAG as a knowledge source, and fine-tuning of several models, with quality assessed using proposed metrics.The study aims to produce geography-specific answers grounded in agricultural data.
- Evaluation: The study evaluates Llama2-13B, GPT-4, and Vicuna on benchmark agricultural datasets, including the effects of spatial shift and spatially scoped fine-tuning.GPT-4 consistently outperformed the other models, although its fine-tuning and inference costs require consideration.
- Evaluation: Table 1 compares GPT-4, Bing Chat, and expert answers across Arkansas, Connecticut, and Georgia for a planting-time question.GPT-4 gives the same answer across locations, Bing Chat adapts better, and expert answers remain more precise.
2 Methodology
The methodology builds a domain-specific Q&A and model-adaptation pipeline from curated documents, structured PDF extraction, RAG-based answer generation, and fine-tuning. Its central challenge is preserving document structure and location or topic signals needed for grounded agricultural questions.
- Pipeline: The pipeline acquires curated industry documents, extracts their content and structure, generates questions and answers, and fine-tunes models with the resulting pairs.Models are evaluated with and without RAG using GPT-4-based metrics.
- 2.1 Data Acquisition: Data acquisition uses high-quality repositories and web-scraping tools to collect relevant documents for the industry dataset.The stated sources include government agencies, scientific databases, and proprietary data when needed.
- 2.2 PDF Information Extraction: PDF extraction must recover textual, tabular, visual, and semantic information because PDF layouts mix heterogeneous elements and do not directly encode logical document structure.The pipeline therefore targets both content and structure rather than text alone.
- 2.2 PDF Information Extraction: Standard PDF text extraction can lose section markers and table or figure captions, hindering reasoning over document structure and sometimes discarding critical information.This limitation motivates structured extraction methods such as GROBID and conversion to JSON preserving document structure.
- 2.3 Question Generation: The question-generation process adds location and agronomic-topic tags extracted from each section to guide grounded questions across relevant topics and challenges.These supporting tags are added to the document content and structure before generation.
3 Dataset Overview
The study combines agricultural datasets from the USA, Brazil, and India to evaluate context-specific LLM responses. Sources include government and institutional documents, farmer questions, specialist answers, and advisory data.
- Geographic scope: The evaluation covers agriculture-related Q&A datasets from the USA, Brazil, and India.The study uses context-related questions and answers from three major crop-producing countries.
- United States: The USA dataset includes publicly available USDA, state-agency, and land-grant-institution documents on agricultural regulation, management, diseases, exports, assistance, and insurance.
- Brazil: The Brazil dataset contains producer- and farmer-oriented crop-cultivation questions paired with responses from Embrapa specialists.The source is the “500 Questions 500 Answers - Embrapa/SCT” dataset.
- Brazil: Brazilian questions are open-ended and often lack a single correct answer, making model-answer accuracy subjective to assess.
- India: The India dataset contains 100,000 farmer queries from 27 states, recorded between 2015 and 2022 with crop, season, query type, location, timestamp, and KVK response fields.The original questions and answers were in local state languages.
- India: Vikaspedia information was used to expand brief KVK answers into more detailed and clearer farmer responses.
4 Metrics
The paper develops metrics for evaluating generated questions and answers across relevance, coverage, fluency, coherence, groundedness, and completion. It emphasizes that question quality is under-measured and difficult to assess objectively.
- Metric framework: The proposed evaluation framework assesses both Q&A generation quality and the effectiveness of fine-tuning and RAG methodologies.
- Metric limitations: The framework identifies question-metric design as challenging because natural language is complex and “good” questions are subjective.Automated and human evaluation are commonly combined, but comprehensive question-quality metrics remain an open challenge.
- Examples: The study illustrates question relevance, global relevance, coverage, fluency, and answer coherence using score examples from agricultural benchmark datasets.
- Question evaluation: Question-quality metrics address relevance, global relevance, coverage, diversity, novelty, grammatical correctness, and fluency.The framework recognizes that question usefulness depends on context and that diversity and novelty are difficult to quantify.
- Question evaluation: GPT-4 rates generated questions from 1 to 5 according to whether they would be asked by a farmer given the context.
- Answer evaluation: Answer evaluation uses coherence, relevance, groundedness, and completion to compare generated answers with reference answers and context.Coherence and relevance use 1-to-5 scales, while completion counts answer tokens.
5 Experiments
The experiments examine Q&A quality, context configuration, metric-calculation models, generation strategy, retrieval, and fine-tuning. Together, they provide complementary evaluations of the agricultural LLM pipeline.
- Q&A Quality: The experiments evaluate Q&A quality generated by GPT-3, GPT-3.5, and GPT-4 using relevance, coverage, overlap, and diversity metrics.
- Context Study: The context study compares Q&A generation with no context, context, and external context setups.
- Model to Metrics Calculation: The model-to-metrics experiment compares GPT-3.5 and GPT-4 when calculating Q&A-quality metrics.
Succinctness ↑
The experiments include answer succinctness alongside generation, retrieval, and fine-tuning analyses. Succinctness is represented through examples of GPT-4-generated answers.
- Succinctness: Table 9 presents examples of succinctness scores for answers generated by GPT-4.
- Related experiments: The broader experiments also compare separate versus combined Q&A generation, retrieval capability, and fine-tuned versus base instruction-tuned models.
5.1 Q&A Quality
The study evaluates Q&A quality across models, context setups, and generation strategies using metrics for coverage, diversity, overlap, relevance, and fluency. Results show recurring tradeoffs between contextual breadth, prompt length, source overlap, and farmer-oriented relevance.
- Context Study: GPT-4 generally achieves the highest Coverage and Size Prompt across context setups, while models perform similarly on Diversity and Overlap.GPT-4 has a slight edge in Relevance and Fluency in the reported comparisons.
- Context Study: No context gives GPT-4 the best balance across average coverage, diversity, overlap, relevance, and fluency, but produces shorter answers.Context produces longer Q&As with slight decreases in most metrics, while external context produces the longest Q&As and slightly higher average relevance and fluency.
- Context Study: GPT-3 performs best overall without context, whereas external context slightly improves coverage and diversity compared with context alone.Context increases prompt size while reducing most other metrics, and external context produces prompts longer than no context but shorter than context.
- Evaluator Comparison: GPT-4 rates generated Q&A pairs as more grounded and fluent than GPT-3.5, but less diverse and relevant across contexts.These differences reflect contrasting model evaluations of the same generated content, whose pairs were generated using GPT-3.
- Combined vs Separated Generation: The only-questions method provides better Coverage, while combined generation generally improves Diversity, Overlap, and Relevance with similar Fluency.Higher Overlap for combined generation may partly result from its longer outputs, which can resemble the source text more closely.
5.2 Retrieval Ablation Study
The retrieval ablation study examines whether RAG recovers source passages used to create questions and how retrieval behavior changes with corpus size and the number of snippets retrieved. More retrieved snippets improve recovery consistency, while a larger corpus may increase collisions and reduce recall.
- Recall Setup: RAG recall measures how often retrieval recovers the original excerpt used to generate each question.The evaluation uses questions based on 573 Washington state benchmark documents and stored source passages.
- Top-k Ablation: Increasing top-k makes RAG recover the original excerpt more consistently by retrieving complementary chunks from split documents.The study therefore considers k = 3 snippets for the following analysis.
- Index-Size Ablation: As the supporting corpus grows, collisions among similar snippets may hinder relevant-information recovery and decrease recall.The study evaluates this corpus-size effect using recall as the index grows in number of documents.
5.3 Fine-tuning
This section evaluates base and fine-tuned language models on agricultural questions using guideline-based accuracy, succinctness, and full-correctness measures. GPT-4 performs strongly across settings, while RAG generally improves relevance and concision.
- Evaluation setup: The evaluation compares Llama2-13B-chat, Vicuna-13B-v1.5-16k, and GPT-4, using agriculture questions generated from Washington-state documents.The fine-tuning data contains approximately 2 million tokens from 573 documents, while evaluation uses 273 human-curated questions.
- Evaluation with Guideline: 86% accuracy is achieved by fine-tuned GPT-4 with RAG, compared with 81% without RAG.In the baseline setting, GPT-4 with RAG scores 80%, while Vicuna and Llama2-chat 13B score 79% and 75%, respectively.
- Qualitative comparison: Vicuna + RAG directly addresses both soil loss and air quality, whereas Llama2 fine-tuned + RAG omits soil loss.The comparison uses a reference answer identifying soil loss and air quality as the main concerns.
- Qualitative comparison: GPT-4 + RAG is clear and concise, while GPT-4 fine-tuned + RAG provides additional regional soil and emissions context.The GPT-4 fine-tuned model also aligns well with the reference answer, though its response is more in-depth.
- Succinctness: RAG generally increases response succinctness, with GPT-4 + RAG scoring 3.2 out of 5.Responses are evaluated on a scale from 1 (verbose) to 5 (succinct).
- Correctness: 56% of Vicuna answers are fully correct with RAG, up from 28% without RAG.Fine-tuned GPT-4 records 45% and 61% fully correct responses in the reported without-RAG and with-RAG conditions, respectively.
5.4 Knowledge Discovery
The knowledge-discovery experiment tests whether fine-tuned GPT-4 can use knowledge transferred across geographically similar questions. Fine-tuning substantially increases the share of new knowledge reflected in answers, with results varying by RAG usage.
- Experiment design: The experiment evaluates whether GPT-4 can learn knowledge based on similarities across different states.Questions shared across at least three states are selected using embedding cosine similarity.
- Experiment design: GPT-4 learns 47% of new knowledge without fine-tuning in the cross-state knowledge experiment.The study identifies 1,000 questions similar across at least three states and removes them from training.
- Results: Fine-tuning raises learned new knowledge to 72% with RAG and 74% without RAG.The reported upper bound reflects the maximum knowledge the model could learn from the presented data.
- Application: Fine-tuning and RAG are presented as ways to apply advisory knowledge from a similar region to a specific agricultural situation.The example concerns evaluating a pest or disease affecting corn when region-specific advisory information is unavailable.
6 Conclusion
The conclusion frames RAG and fine-tuning as complementary approaches with different benefits, costs, and data requirements. The study establishes agricultural LLM baselines and a document-to-Q&A pipeline while emphasizing application-specific selection and further investigation.
- Contributions: The study establishes agricultural performance baselines for Llama 2, GPT-3.5, and GPT-4 using RAG and fine-tuning.The comparison is intended to expose strengths and limitations of LLMs in agriculture.
- RAG tradeoffs: RAG offers low initial embedding costs and works well when retrieved data is contextually relevant, but it can enlarge prompts and produce verbose, harder-to-steer outputs.The conclusion identifies farm-data interpretation as an example of contextually relevant use.
- Fine-tuning tradeoffs: Fine-tuning produces precise, succinct outputs and can support domain-specific skills, but requires substantial initial work on new data.It also requires minimal input-token size when handling large datasets.
- Pipeline: The paper presents a pipeline combining structured document understanding, GPT-4 question generation, and RAG answer generation for industry-specific datasets.The generated questions are reported as specific to their source sections, and the answers use the full text.
- Evaluation: The proposed metrics evaluate generated-question quality against source documents and assess the quality of RAG-generated answers.These metrics extend evaluation beyond model accuracy alone.
- Scope and future work: Approach suitability depends on the application, dataset nature and size, and available development resources.The conclusion also identifies GPT-4 fine-tuning and inference costs as an important tradeoff.