PaperlayerView exact revision on arXiv

Source-linked AI summary

RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, Ranveer Chandra

arXiv:2401.08406v3cs.CLcs.LG

TL;DR

Agricultural questions can require local knowledge that general search tools may not provide, and the tradeoffs between RAG and fine-tuning are not well understood. The paper evaluates an agriculture-focused pipeline using Q&A generation, RAG, and fine-tuning, finding distinct benefits and costs for the two approaches.

  • Problem

    Agriculture-specific questions may require knowledge of local conditions, varieties, and current data that general search engines may not provide.

    Source: §1 · p. 2

  • Method

    The pipeline uses an agriculture dataset for Q&A generation, RAG, and fine-tuning, and evaluates the resulting models and stages.

    Source: §1 · p. 2

  • Results

    RAG is effective when data is contextually relevant, while fine-tuning offers precise, succinct outputs but has a high initial cost.

    Source: §6 · p. 29

  • Takeaways & Limitations

    The study establishes agricultural performance baselines for LLMs using RAG and/or fine-tuning and compares their strengths and limitations.

    Source: §6 · p. 29

    Common PDF extraction tools may recover text while losing document structure and figure or table captions needed to interpret content.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.

1 Introduction

The paper addresses the difficulty of adapting LLM copilots to agriculture, where useful answers require local context, by proposing a pipeline for generating industry-specific Q&A and evaluating RAG and fine-tuning. Its agriculture case study examines geography-specific responses and the performance of several LLMs.

Pipeline

  • The proposed pipeline collects and structures agricultural documents, generates and filters Q&A pairs, then uses the data for RAG and fine-tuning with proposed evaluation metrics.The agriculture study aims to generate geography-specific answers from its dataset.Source: §1 · p. 2

Evaluation

  • The evaluation covers Llama2-13B, GPT-4, and Vicuna on agriculture questions, including spatial shift and spatially scoped fine-tuning, with GPT-4 consistently outperforming the other models.The paper notes that GPT-4's fine-tuning and inference costs should be considered.Source: §1 · p. 2

RAG and fine-tuning

  • The study finds that RAG helps when data is contextually relevant, while fine-tuning teaches agriculture-specific skills and can yield more precise, succinct responses.The introduction flags the substantial initial work required to fine-tune on new data as a cost.Source: §1 · p. 2

Agricultural context

  • GPT-4 gives the same generic answer across three states, while Bing Chat adapts better and an agronomist provides more precise location-specific guidance.The introduction identifies limited specialized training data and the need to account for local conditions, crop varieties, and current information as barriers to agricultural AI adoption.Source: §1 · p. 2, Table 1 · p. 3

2 Methodology

The methodology describes a pipeline for building domain-specific copilot datasets through document extraction, grounded Q&A generation, and evaluation. It also specifies the fine-tuning setup: four epochs across seven nodes, each with eight A100 GPUs.

2.1 Data Acquisition

  • Geography-aligned, authoritative industry sources are gathered and scraped to provide curated material for generating grounded questions and answers.For agriculture, sources include government agencies, scientific repositories, and agronomist exam databases; Scrapy and BeautifulSoup were used to find and download documents.Source: §2.1 · p. 4

2.2 PDF Information Extraction

  • Because PDFs have complex layouts and their underlying structure may not match document logic, extraction must recover both content and structure.The authors use GROBID to convert scientific PDFs into TEI, then extract metadata, sections, tables, figure references, bibliography, and content into JSON; plain text extraction can lose section markers and captions.Source: §2.2 · p. 5

2.3 Question Generation

  • Guidance-based question generation uses document context plus extracted location and agronomic-topic tags to ground questions in source content.Prompts combine supporting context and section content, producing 5 to 15 questions per document section.Source: §2.3 · pp. 7, 9

2.4 Answer Generation

  • RAG retrieves relevant passages for each question and supplies them to an LLM to generate contextually grounded answers.The implementation embeds PDF text chunks, indexes them with FAISS, retrieves relevant chunks, and provides them to GPT-4 for answer generation and Q&A-pair formatting.Source: §2.4 · p. 9

2.5 Fine-tuning

  • The generated Q&A pairs support fine-tuning models to predict answer tokens from question prompts, with prompt tokens masked during training.The study also fine-tunes GPT-4 using LoRA, which tunes a reduced set of attention-module parameters to reduce memory and computational requirements.Source: §2 · p. 4, §2.5 · p. 10
  • Fine-tuning ran for 4 epochs with a batch size of 256, a base learning rate of 1e-4 that decayed during training, and seven nodes with eight A100 GPUs each over 1.5 days.The passage describes the approach as embedding domain-specific knowledge from new data without losing knowledge gained during base training.Source: §2.5 · p. 11

3 Dataset Overview

The study evaluates RAG and fine-tuned models on agricultural Q&A datasets from the USA, Brazil, and India, drawing on varied document and Q&A sources. These datasets differ in geographic coverage, format, and the challenges they pose for evaluation.

3 Dataset Overview

  • The study uses agricultural Q&A datasets from the USA, Brazil, and India, with source materials ranging from regulatory documents and scientific reports to exams and knowledge databases.The datasets provide context-related questions and answers for evaluating fine-tuned and RAG-enhanced models.Source: §3 · p. 11

3.1 USA

  • The USA corpus contains over 23k PDFs and 50M tokens across 44 states, with Washington’s 573 files and over 2M tokens used for benchmarking.Sources include USDA, state agencies, and Land-Grant Institution partners; documents cover regulatory, management, disease, export, assistance, insurance, and pricing information.Source: §3.1 · p. 11

3.2 Brazil

  • Brazil’s “500 Questions 500 Answers - Embrapa/SCT” pairs stakeholder questions about crop cultivation and management with responses from Embrapa specialists.Questions are open-ended and often lack a single correct answer, making model accuracy assessment subjective and requiring expertise to judge relevance and correctness.Source: §3.2 · p. 11

3.3 India

  • India’s datasets combine 100,000 farmer queries from 27 states with general agricultural information, and KVK answers were expanded using Vikaspedia content.Queries from 2015–2022 were originally in local state languages and translated into English; each record includes crop, query type, state, district, and other fields.Source: §3.3 · p. 12

4 Metrics

The section defines metrics for assessing generated Q&A pairs and fine-tuned models, emphasizing question quality alongside answer evaluation. It combines context-aware and context-independent question measures with separate assessments of answer quality and model outputs.

4 Metrics

  • The framework takes Q&A pairs with associated context as input and quantifies their quality, reflecting the premise that dataset quality affects model quality.Its metrics are intended to guide assessment of Q&A generation for fine-tuning and RAG.Source: §4 · p. 12

4.1 Question Evaluation

  • Because question quality is subjective and natural language complex, the authors address a gap in question-quality metrics with a suite for filtering and refining generated questions.They note that automated measures such as ROUGE and BLEU offer some insight, while human evaluation can assess subjective aspects such as relevance, novelty, and fluency.Source: §4 · p. 13, §4.1 · p. 13
  • GPT-4 rates farmer-perspective question relevance with context and global relevance without context on 1–5 scales; LLMs rate coverage on a 1–5 scale for answer extractability from context.Higher coverage scores indicate more reliable extraction and grounding in available information.Source: §4.1 · p. 15
  • Question overlap uses KL divergence, diversity uses WMD and average pairwise question similarity, detail counts tokens, and GPT-4 rates fluency and coherence.Lower KL divergence indicates greater overlap with source text; lower average similarity among questions indicates greater diversity.Source: §4.1 · p. 15

4.2 Answer Evaluation

  • AzureML Model Evaluation compares answers with ground truth using coherence, relevance, groundedness, and completion, with the first three assessing answer quality and completion counting tokens.Coherence and relevance are scored from 1 to 5; groundedness receives an integer score.Source: §4.2 · pp. 17, 19

4.3 Model Evaluation

  • GPT-4 evaluates model answers against about 270 agriculture Q&A ground-truth pairs, with five evaluation prompts per assessment to measure response variance.The evaluation includes guideline-based scoring from 0 to 1, succinctness grades from 1 to 5, and categorical correctness judgments.Source: §4.3 · p. 19

5 Experiments

The experiments evaluate agricultural Q&A generation and its use in RAG or fine-tuning through complementary studies of Q&A quality, context setup, and metric calculation.

Q&A Quality

  • GPT-3, GPT-3.5, and GPT-4 Q&A pairs are assessed for relevance, coverage, overlap, and diversity under different context setups.The study frames these measures as an evaluation of generated Q&A quality.Source: §5 · p. 19

Context Study

  • Generated Q&A pairs are compared across no-context, context, and external-context setups.The context study investigates how setup affects model performance in generating Q&A pairs.Source: §5 · p. 19

Model to Metrics Calculation

  • GPT-3.5 and GPT-4 are compared when used to calculate metrics for assessing Q&A-pair quality.Source: §5 · p. 19

Succinctness ↑

The section presents GPT-4 answer-succinctness examples and describes experiments on generation efficiency, retrieval, and fine-tuning.

Succinctness ↑

  • Table 9 presents examples of succinctness scores for GPT-4-generated answers.The supplied caption does not include score values.Source: Table 9 · p. 20
  • The combined-versus-separated generation experiment compares question-and-answer generation formats for token-use efficiency.Source: Succinctness ↑ · p. 20
  • The retrieval ablation evaluates RAG’s use of additional context, while the fine-tuning experiment compares fine-tuned models with base instruction-tuned models for learning new knowledge.Source: Succinctness ↑ · p. 20

5.1 Q&A Quality

The section evaluates generated Q&A quality across context setups and generation methods using LLM ratings and metrics including coverage, diversity, relevance, and fluency. Results show tradeoffs across models, contexts, and whether questions and answers are generated together or separately.

5.1.1 Context Study

  • Across context setups, GPT-4 generally has the highest coverage and prompt size, while model-level tradeoffs vary in the balance of coverage, relevance, fluency, and prompt length.For GPT-4, no context gives the best balance across average coverage, diversity, overlap, relevance, and fluency but shorter answers; external context preserves average coverage and slightly increases relevance and fluency, with the longest prompts. GPT-3 performs best overall without context, while GPT-3.5 has its highest average coverage and diversity with context, but slightly higher relevance and fluency with external context.Source: §5.1.1 · pp. 22–23, Table 13 · p. 22

5.1.2 Model to Metrics Calculation

  • Compared with GPT-3.5, GPT-4 rates Q&A pairs higher in coverage and fluency, but lower in diversity, overlap, and relevance across contexts.These differences reflect how the evaluator models rate the generated pairs, not differences in generation: the comparison used questions and answers generated with GPT-3.Source: §5.1.2 · pp. 23–24, Table 14 · p. 23
  • GPT-3.5 generates faster than GPT-4, so most Q&A-generation metrics were calculated with GPT-3.5 and model-evaluation metrics with GPT-4.Reported Azure OpenAI throughput is 240k–300k tokens per minute for gpt-35-turbo variants and 20k–80k for GPT-4 variants, depending on region and model.Source: §5.1.2 · p. 24

5.1.3 Combined vs Separated Generation

  • Separate generation uses fewer tokens and allows different models or approaches, such as RAG or fine-tuning, to answer the generated questions.This can be useful within language-model token limits and avoids relying solely on base language models.Source: §5.1.3 · p. 24
  • Generating only questions yields higher coverage, while combined generation scores higher on diversity, overlap, and farmer-perspective relevance; both methods perform similarly in fluency.The relevance scores were extremely similar when the farmer perspective was removed; the higher overlap for combined generation may reflect its longer output.Source: §5.1.3 · p. 24, Table 15 · p. 24

5.2 Retrieval Ablation Study

In this agricultural RAG ablation, recall declines as the supporting-document index grows but remains above 75% at sixfold scale. Retrieving three snippets recovers the original passage more than 80% of the time without considerably increasing prompt size.

5.2 Retrieval Ablation Study

  • Recall@top-3 remains above 75% when the FAISS index grows sixfold, from 573 to 3888 documents, despite declining as similar-snippet collisions increase.The evaluation uses the Washington benchmark dataset.Source: Table 17 · p. 25, §5.2 · p. 25
  • Recall measures how often retrieval returns the original excerpt used to generate each question, across questions based on 573 Washington benchmark documents.The study seeks higher recall because the LLM can filter unrelated retrieved information when the correct context is present.Source: §5.2 · p. 25
  • Using k = 3 snippets recovers the original passage more than 80% of the time without considerably increasing input prompt size.The authors note that document chunks can provide complementary information during answer generation.Source: §5.2 · p. 25

5.3 Fine-tuning

On Washington agriculture questions, fine-tuned GPT-4 achieved the highest reported guideline accuracy and fully-correct answer rates. RAG improved completeness and generally increased succinctness, while response quality varied by model and metric.

5.3 Fine-tuning

  • Fine-tuning used Washington questions drawn from 573 documents comprising approximately 2 million tokens, with answers generated by Llama2-13B-chat with RAG, and was evaluated on 273 human-curated questions.The comparisons included base Llama2-13B-chat, Vicuna-13B-v1.5-16k, and GPT-4, as well as agriculture-fine-tuned Llama2-13B and GPT-4.Source: §5.3 · pp. 25–26

5.3.1 Evaluation with Guideline

  • GPT-4 fine-tuning reached 86% accuracy with RAG and 81% without RAG; baseline GPT-4 with RAG scored 80%, versus 79% for Vicuna and 75% for Llama2-chat 13B.GPT-4 and Llama2-chat 13B had similar results without RAG; GPT-4 was the top performer across the reported scenarios.Source: §5.3.1 · p. 26, Table 18 · p. 26
  • In the blowing-dust example, Vicuna+RAG and GPT-4 variants best addressed the reference answer of soil loss and air quality, while Llama2 variants often omitted soil loss.GPT-4 alone was comprehensive but potentially too extensive; GPT-4+RAG was concise and direct, and both fine-tuned GPT-4 variants covered the key concerns.Source: §5.3.1 · p. 26

5.3.2 Succinctness

  • RAG generally made responses more succinct; GPT-4+RAG had the highest reported succinctness score at 3.2 out of 5, but fine-tuned GPT-4+RAG did not show this improvement.Whether less verbose answers are preferable depends on user preference and use case, particularly when model accuracy is similar.Source: §5.3.2 · pp. 26–27, Table 19 · p. 27

5.3.3 Correctness

  • RAG increased fully-correct answers for every model; Vicuna rose from 28% to 56%, while fine-tuned GPT-4 achieved 45% without RAG and 61% with RAG.GPT-4 and Llama2-chat 13B outperformed Vicuna, and fine-tuned GPT-4 had the highest fully-correct rates with and without RAG.Source: §5.3.3 · p. 27, Table 20 · p. 27

5.4 Knowledge Discovery

The study tests whether GPT-4 can learn new knowledge across similar US states and reports higher knowledge-learning rates with fine-tuning, with or without RAG. It presents cross-region advisory information as useful when region-specific information is unavailable.

5.4 Knowledge Discovery

  • The study selected 1,000 questions similar across at least three of the 50 US states, removed them from training, and tested fine-tuning with and without RAG.The questions were identified using cosine similarity between embeddings to assess whether GPT-4 could learn knowledge across states.Source: §5.4 · p. 27
  • When local pest or disease advisories are unavailable, information from a similar region could help a model learn and apply relevant knowledge to the farmer’s situation.The paper gives evaluating the impact of a pest or disease on corn as an example and proposes using fine-tuning and RAG to apply cross-region information.Source: §5.4 · pp. 27, 29
  • GPT-4 learned 47% of the new knowledge, compared with 72% and 74% for fine-tuned models depending on RAG use; these rates represent the stated upper bound.The study describes the upper bound as the maximum amount the model could have learned from the data presented to it.Source: §5.4 · pp. 27, 29, Table 22 · p. 29

6 Conclusion

The study establishes baselines for LLMs using RAG and fine-tuning in agriculture, highlighting their distinct costs and benefits. It also presents an industry-specific data-generation pipeline and identifies directions for future research.

6 Conclusion

  • The study establishes baselines for LLMs using RAG or fine-tuning and presents a pipeline intended to support applications across industries, beginning with agriculture.The authors describe the work as a step toward more efficient models, starting from the Q&A generation process.Source: §6 · p. 29
  • RAG is effective when data is contextually relevant and has low initial embedding costs, but can increase prompt size and produce verbose, harder-to-steer outputs.The conclusion gives interpreting farm data as an example of contextually relevant use.Source: §6 · p. 29
  • Fine-tuning can produce precise, succinct outputs and learn domain-specific skills, but requires substantial initial work and minimal input tokens.Examples include improving crop-yield predictions and optimizing irrigation schedules based on weather patterns.Source: §6 · p. 29
  • Structured document understanding, GPT-4 question generation, and RAG answer generation produced section-specific questions and comprehensive answers while enabling efficient token use.The authors propose evaluating question quality against source documents and measuring the quality of RAG-generated answers.Source: §6 · pp. 29–30
  • GPT-4 consistently outperformed other models, but its fine-tuning and inference costs remain important trade-offs.The suitability of RAG and fine-tuning depends on the application, dataset, and available development resources.Source: §6 · p. 30
  • Future work includes investigating fine-tuned knowledge, improving structured document extraction, and combining PDF information with images and captions for multimodal fine-tuning.The authors also identify further study of how to combine RAG and fine-tuning and improve industry-specific dataset-generation pipelines.Source: §6 · p. 30

Related papers

Loading 2401.08406v3…