PaperlayerView exact revision on arXiv

Source-linked AI summary

ShizishanGPT: An Agricultural Large Language Model Integrating Tools and Resources

Shuting Yang, Zehui Liu, Wolfgang Mayer

arXiv:2409.13537v1cs.CLcs.AI

TL;DR

Specialized agricultural knowledge remains a limitation for LLMs. ShizishanGPT integrates agricultural knowledge resources and tools in a question-answering system; the paper reports good performance and accuracy, with response quality further enhanced by external tools and retrieval.

  • Problem

    LLMs have limitations in specialized domain knowledge, particularly in agriculture and crop-related tasks.

    Source: §1 · p. 2

  • Method

    ShizishanGPT integrates an agricultural knowledge graph, a crop-literature vector base, and external gene-sequence and maize-phenotype tools into an agricultural QA system.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    ShizishanGPT scored 9.65 composite in manual scoring, and the conclusion reports good performance and accuracy on agricultural questions.

    Source: §4.2 · p. 12, §6 · p. 14

  • Takeaways & Limitations

    The study reports that external tools and the retrieval vector database further enhance the system’s response quality for agricultural questions.

    Source: §6 · p. 14

    The study may apply only to agriculture and specific crops and regions, and its GPT-4-generated dataset may have representativeness limits.

    Source: §5 · p. 13

Abstract

from arXiv · show

Recent developments in large language models (LLMs) have led to significant improvements in intelligent dialogue systems'ability to handle complex inquiries. However, current LLMs still exhibit limitations in specialized domain knowledge, particularly in technical fields such as agriculture. To address this problem, we propose ShizishanGPT, an intelligent question answering system for agriculture based on the Retrieval Augmented Generation (RAG) framework and agent architecture. ShizishanGPT consists of five key modules: including a generic GPT-4 based module for answering general questions; a search engine module that compensates for the problem that the large language model's own knowledge cannot be updated in a timely manner; an agricultural knowledge graph module for providing domain facts; a retrieval module which uses RAG to supplement domain knowledge; and an agricultural agent module, which invokes specialized models for crop phenotype prediction, gene expression analysis, and so on. We evaluated the ShizishanGPT using a dataset containing 100 agricultural questions specially designed for this study. The experimental results show that the tool significantly outperforms general LLMs as it provides more accurate and detailed answers due to its modular design and integration of different domain knowledge sources. Our source code, dataset, and model weights are publicly available at https://github.com/Zaiwen/CropGPT.

1 Introduction

Agricultural LLMs remain insufficiently developed for specialized crop knowledge and tasks. ShizishanGPT combines domain resources and tools to answer agricultural questions, including specialized predictions.

1 Introduction

  • ShizishanGPT targets the limited adaptation of language models to specialized agricultural and crop knowledge, including genetic tools and crop-related retrieval.The introduction contrasts its promoter-enrichment prediction capability with ChatGPT-4, which it says often fails in this agricultural context.Source: §1 · p. 2
  • The system combines an Agricultural Knowledge Graph, a crop-focused Vector Knowledge Base, and external gene-sequence and maize-phenotype prediction tools.The study evaluates the system on 100 agricultural question-answer pairs designed for this study.Source: §1 · p. 2
  • Figure 1 presents an example comparing ChatGPT and ShizishanGPT on predicting maize gene promoter enrichment values.Source: Figure 1 · p. 3

2 Related Works

Related work covers retrieval- and agent-based LLM enhancement, knowledge graphs, and external tools, including agricultural applications. ShizishanGPT combines RAG, ReAct, and agricultural knowledge resources.

2 Related Works

  • This section reviews RAG and agent frameworks, knowledge-graph-enhanced models, external tools, and agricultural knowledge graphs as approaches to improving complex-task accuracy and utility.Source: §2 · p. 3

2.1 RAG and Agent-Based Frameworks for LLM Enhancement

  • ShizishanGPT combines RAG, trained on more than a thousand scholarly articles, with ReAct’s structured reasoning and action sequences for complex agricultural tasks.The cited discussion presents this combination as supporting data-driven answers and agricultural decision support.Source: §2.1 · pp. 3–4

2.2 Knowledge Graph Enhanced Large Models

  • Prior work includes Shennong Big Model 1.0, which integrates agricultural knowledge graphs and vector databases to improve answer accuracy.The passage reports its training data included more than 10 million agricultural knowledge-mapping data and more than 20,000 agricultural books.Source: §2.2 · p. 4

2.3 External Tools Enhanced Large Models

  • Agricultural external tools include phenotype models, knowledge graphs, literature, and search; examples combine LLMs with vision models, retrieval, fine-tuning, or model libraries.HuggingGPT is described as coordinating and invoking hundreds of smaller models through GPT-4.Source: §2.3 · pp. 4–5

2.4 Knowledge Graphs in Agriculture

  • AgriKG extracts and connects entities and relationships from agricultural data, supporting entity retrieval and intelligent question answering.The study integrates AgriKG data with user queries in an LLM-based QA system.Source: §2.4 · p. 5

3 System Design

The system designs prompts around the user’s query: it extracts key information and action words, then uses them to guide retrieval and analysis with relevant agricultural resources.

3.1 Design of Prompt

  • The prompt mechanism dynamically adapts to query content to guide extraction and use of key information for agricultural data analysis.Source: §3.1 · pp. 5–6

Prompts Mechanisms:

  • The system extracts query keywords and action words, such as “corn,” “yield,” or “predict,” to construct structured, query-specific prompts.Source: Prompts Mechanisms · p. 6
  • For a maize DNA-sequence query in Huazhong, a prompt can specify growth-rate effects under regional climatic conditions and direct the system to relevant graphs, databases, and prediction models.Source: Prompts Mechanisms · p. 6

The Role of Prompts in System Architecture:

Prompts bridge user queries to the system’s database and help focus retrieval. The workflow selects tools, adapts to feedback, and can iteratively retry or decompose tasks.

The Role of Prompts in System Architecture:

3.2 Workflow of System

  • After a user submits a question, the system analyzes it and selects the most relevant functional-module problem description before calling tools.Source: §3.2 · p. 6
  • The language model adjusts its next plan step based on tool feedback, then summarizes the results with its own knowledge.Source: §3.2 · p. 6
  • Under ReAct, the agent can reanalyze a failed first tool choice, try another tool iteratively, or decompose a question into sub-problems.Source: §3.2 · p. 6

3.3 System Architecture

ShizishanGPT combines language models, external search, specialized literature retrieval, a knowledge graph, and crop-focused prediction models in an agricultural question-answering system.

3.3 System Architecture

  • The system combines a generic language model, real-time search, specialized-literature retrieval, AgriKG, and crop-focused prediction models for agricultural question answering.The modules address generic queries, current information, in-depth domain questions, structured crop knowledge, and crop prediction tasks.Source: §3.3 · p. 7
  • The system integrates specialized models for maize phenotype prediction and rice gene promoter enrichment value prediction.The description also identifies crop-focused prediction models as part of the architecture.Source: §3.3 · p. 7
  • A unified user interface manages the modules’ combined operation for natural-language queries.The authors describe the interface as supporting simplicity and efficiency of operation.Source: §3.3 · p. 7

3.4 Modules in the Architecture

The architecture assigns general agricultural questions to a language-model module and uses search to add current agriculture-related information.

Problem analysis module:

  • The generic module uses pretrained language models to parse and answer broad questions, such as agricultural basics, and checks whether other knowledge sources are needed.It determines the nature of the user query before selecting whether to supplement its response.Source: Problem analysis module · p. 8

Search module:

  • The search module retrieves and filters information from external search engines, then injects extracted content into the language-model prompt.It aggregates results from multiple sources, including web pages, articles, and reports, to summarize current agricultural information.Source: Search module · p. 8

Retrieval module:

The retrieval module supplies specialized literature for generation, while the knowledge-graph module retrieves structured agricultural facts through AgriKG.

Retrieval module:

  • The retrieval module uses RAG [21] to search over a thousand specialized agricultural documents and provide relevant literature as context for answer generation.The repository covers topics including crop breeding, disease control, and recent agricultural innovations.Source: Retrieval module · p. 8

Knowledge Graph module:

  • The knowledge-graph module uses AgriKG [4] to retrieve structured facts about crop classification, climate suitability, and pest control.It maps user-entered crop names and species to graph entities and relationships, supporting planting-strategy recommendations.Source: Knowledge Graph module · p. 8

External tool invocation module:

The external tool invocation module selects crop-related predictive models according to the user’s question and combines their outputs with knowledge from other modules.

External tool invocation module:

  • The module includes models for maize phenotype and gene-promoter prediction, crop growth and variety selection, pest and disease control, weather warnings, agricultural consulting, and market analysis.It automatically selects an appropriate model for the question and combines its results with knowledge from other modules to produce multi-faceted answers.Source: External tool invocation module · pp. 8–9

4 Experiments

The experiments evaluate ShizishanGPT on a 100-pair agricultural QA benchmark using machine and expert scoring, then test how removing system components affects performance.

4 Experiments

  • The benchmark contains 100 agricultural question-answer pairs drawn from representative questions, expanded with ChatGPT, manually screened, and answered using professional agricultural websites.Questions cover crop cultivation, agricultural technology, pest control, and agricultural processing, among other areas.Source: §4 · p. 9

4.1 Machine Scoring

  • Machine scoring compares generated answers with professional-website reference answers using BLEU, ROUGE, and GLEU; GLEU is described as assessing syntactic and semantic consistency.The paper notes that BLEU's phrase matching may miss contextual and semantic nuances, and that GLEU is more computationally expensive.Source: §4.1 · pp. 9–10

Scoring Standard:

  • The composite score weights BLEU, ROUGE, and GLEU at 0.4, 0.4, and 0.2, respectively; the ROUGE formula sets λ to 0.1 and both ω weights to 0.5.The paper describes the component weights as adjustable to reflect the importance of different evaluation indicators.Source: Scoring Standard · p. 10

Result:

  • ShizishanGPT outperformed the other models on BLEU and GLEU, while its ROUGE performance was similar to theirs.The paper interprets this as better similarity and semantic consistency relative to ChatGPT and the other models.Source: Result · pp. 10–11, Figure 3 · p. 11
  • 0.923 composite score, with BLEU 0.88, ROUGE 0.79, and GLEU 0.85, exceeded ChatGPT-4's 0.876 composite score.Table 1 also compares ShizishanGPT with Claude, WenXinYiYan, and ZhiPuAI.Source: Table 1 · p. 11, Result · p. 11

4.2 Manual Scoring

  • Manual evaluation used agriculture-background evaluators who rated answers without knowing their source, with at least three independent evaluators scoring each answer.Evaluators scored Accuracy, Professionalism, and Language Fluency, with a cross-checking mechanism to compare scorer consistency.Source: §4.2 · p. 12
  • 9.65 composite score, with 9.8 in Accuracy and 9.5 in Professionalism, made ShizishanGPT the top model in manual evaluation.The evaluation also scored Language Fluency; the reported weighting was 50% Accuracy, 30% Professionalism, and 20% Language Fluency.Source: Table 2 · p. 12, §4.2 · p. 12

4.3 Ablation Experiments

  • 9.21 total score after removing RAG, versus 9.65 for the full configuration; removing AgriKG slightly improved Accuracy and Professionalism, with a 9.50 total.The full configuration scored 9.8 in Accuracy, 9.5 in Professionalism, and 9.1 in Language Fluency.Source: Table 3 · p. 13, §4.3 · p. 13

5 Limitations

The authors identify limitations in both the dataset and the scope of the research. The dataset’s question quality and representativeness may be constrained, and the system may apply only to agriculture and specific crops or regions.

5 Limitations

  • The GPT-4-generated dataset may have question-quality biases, while manually selected seed questions may limit its representativeness.The authors attribute possible quality issues to GPT-4’s training-data biases and question-generation capabilities.Source: §5 · p. 13
  • The research may apply only to agriculture and to specific crops, regions, or scopes of questions.Source: §5 · p. 13

6 Conclusion

The study integrates a knowledge-graph-enhanced language model with external tools and retrieval to build an agricultural question-answering system. It reports good performance and accuracy, with tool and retrieval calls further enhancing response quality, and demonstrates support for agricultural production and decision-making.

6 Conclusion

  • The system combines a knowledge-graph-enhanced language model with external tools and a retrieval vector database for agricultural question answering.Source: §6 · p. 14
  • Calling external tools and the retrieval vector database further enhances the quality of the system’s agricultural answers.Source: §6 · p. 14
  • The study demonstrates agricultural knowledge querying, reasoning, and retrieval as support for production and decision-making.Source: §6 · p. 14

Related papers

Loading 2409.13537v1…