ChatAgri: Exploring Potentials of ChatGPT on Cross-linguistic Agricultural Text Classification

Authors

• The authors are Biao Zhao, Weiqiang Jin, Javier Del Ser, and Guang Yang.Zhao and Jin are noted as equal contributors.Source: Agricultural Text Classification · p. 2

Affiliations

• The listed affiliations span Xi’an Jiaotong University, TECNALIA, and Imperial College London.The addresses are in China, Spain, and the UK.Source: Agricultural Text Classification · p. 2

Keywords

• The keywords include agricultural text classification, very large pre-trained language models, GPT, and ChatGPT.Source: Agricultural Text Classification · p. 2

Motivation

• The study frames agricultural text classification as a way to organize online information and make it more readily indexable.Source: §1 · p. 2
• Existing PLM-based methods require substantial annotated data, transfer poorly across domains and languages, and need power-intensive deployment hardware.The passage notes that the largest T5 model has over 11 billion parameters.Source: §1 · p. 3

Contributions

• The authors present ChatAgri as a preliminary study of ChatGPT-based agricultural text classification and compare it with traditional machine learning, fine-tuned PLMs, and prompt-learning methods.Source: §1 · pp. 4–5
• ChatAgri achieves competitive performance on multilingual datasets compared with PLM-based fine-tuning approaches.The authors also report superior semantic understanding and an intelligent reasoning chain in case analyses.Source: §1 · p. 5
• Zero-shot experiments demonstrate ChatAgri’s potential relative to fine-tuned PLMs that depend on costly expert annotation.Source: §1 · p. 5
• ChatAgri’s multilingual experiments indicate cross-linguistic transferability across agricultural applications.The study evaluates data from insect pests, natural hazards, and agricultural market comments.Source: §1 · p. 5
• ChatAgri is described as relying on a network interface and minimum hardware requirements, unlike complex, power-intensive PLM-based methods.The authors characterize this as a prospect for lower-cost AI in smart agriculture.Source: §1 · p. 5

2.1. Agricultural Text Classification

• Earlier agricultural document classification used models such as decision trees, CNNs, LSTMs, and GRUs.Prior work also applied SVMs, Bi-LSTM with attention, and CNNs to agricultural data and tweets.Source: §2.1 · pp. 5–6
• Agricultural text-classification research includes BERT systems for labeling corpora, French plant-health bulletins, OCR, and product-review sentiment.Source: §2.1 · p. 6

2.2. Traditional Machine Learning methods, and PLM-based Fine-tuning, and Prompt-tuning

• Traditional agricultural methods included AETS-LSTM for export trends and a knowledge-based Jaccard-similarity system for rice pest and disease information.Source: §2.2 · p. 6
• PLM-based fine-tuning became mainstream after feature-engineering methods struggled with natural-language complexity, but fine-tuned models can require extensive annotation and generalize poorly.Prompt-tuning instead adapts tasks through constructed templates and can perform in few-shot or zero-shot settings.Source: §2.2 · p. 6

2.3. ChatGPT

• ChatGPT research describes a conversational model with large-scale parameters, chain-of-thought prompting, and reinforcement learning from human feedback.Prior work also examined multilingual translation, code generation, information extraction, and GPT-4.Source: §2.3 · p. 7

3.1. Methodology Overview

• ChatAgri is a preliminary study of applying ChatGPT to agricultural text classification, with a systematic experimental analysis.Source: §3.1 · p. 8
• The pipeline comprises prompt construction, black-box ChatGPT question-and-answer inference, and normalization or alignment of the response to predefined labels.Source: §3.1 · p. 8
• Prompt construction and answer alignment can be optimized, while ChatGPT’s inference procedure is treated as fixed; prompt quality, model version, and mapping priority can affect predictions.Source: §3.1 · p. 8
• The framework treats agricultural text classification as domain-specific and calls for corresponding customized strategies.Source: §3.1 · pp. 8–9

3.2. Prompt Question Construction

• The four strategies are manually defined prompts, ChatGPT-triggered prompts, zero-shot similarity-based prompts, and Chain-of-Thought prompts.The authors describe prompting engineering as requiring experience and trial and error.Source: §3.2 · p. 9

3.2.1. Manually Defined Prompts

• The manually defined prompts use [SENT] for the input text and [CATE] for the predefined categories, and the authors select a prompt using subset accuracy.They randomly sample 100 Twitter Natural Hazards examples by default; Figure 4 shows the prompt selected for subsequent experiments.Source: §3.2.1 · p. 9, Figure 4 · p. 9

3.2.2. ChatGPT Triggered Prompts

• ChatGPT-generated candidate templates are described as semantically consistent but different in format, and the best-performing candidate is selected for later comparisons.The request asks ChatGPT to provide five concise templates for agricultural text classification.Source: §3.2.2 · pp. 9–10, Figure 5 · p. 10

3.2.3. Zero-Shot Similarity Prompts

• For agricultural text classification, ChatGPT is treated as a similarity measurement for evaluating relationships between sentences, using either end-to-end direct QA-based similarity evaluation or progressive comparison QAs-based similarity evaluation.The progressive approach compares pairs of elements successively, similar to bubble sorting.Source: §3.2.3 · p. 10
• In a one-turn example, the direct similarity method assigns the target sentence to the category of sentence C.The prompt asks which candidate sentence is most similar to the target.Source: §3.2.3 · p. 10, Figure 6 · p. 10
• The progressive method uses two-turn Q&As, with the example assigning the target to sentence A’s category after the second-stage topic-similarity comparison.The authors describe this multi-stage similarity-comparison approach as, to their knowledge, the first of its kind for text classification.Source: §3.2.3 · p. 11, Figure 7 · p. 11

3.2.4. Chain-of-Thought Triggered Prompts

• For arithmetic, ChatGPT gives 334 for 4+32 · 5-2 but reaches the correct answer when prompted to calculate step by step.This observation motivates testing Chain-of-Thought prompts for agricultural classification against manually defined and ChatGPT-triggered prompts.Source: §3.2.4 · p. 11, Figure 8 · p. 11, Figure 9 · p. 12
• Chain-of-Thought prompts perform particularly well on datasets with many categories, but are not satisfactory on simpler datasets with two to three categories.The authors state that prompt effectiveness varies with dataset features, particularly the number of classification categories.Source: §3.2.4 · p. 12

3.3. ChatGPT Q&A Inference

• At each generation step, ChatGPT conditions on the prompt and previously generated tokens to produce a probability distribution over the next token.The generated sequence has length T.Source: §3.3 · p. 12
• The authors hypothesize that ChatGPT can act as an integrated zero-shot text-classification interface through interactive use.They create a fresh conversation thread for each prompt to prevent prior conversation history from affecting responses.Source: §3.3 · p. 12
• GPT-4 performs better than GPT-3.5, including in some complex semantic text-classification scenarios.The experiments evaluate both GPT-3.5 and GPT-4.Source: §3.3 · p. 12

3.4. Answer Alignment

• ChatGPT’s natural-language responses do not directly correspond to predefined labels, requiring answer alignment to convert them into categories for metrics such as accuracy and F1-score.The paper calls this additional mapping step answer alignment engineering.Source: §3.4 · pp. 12–13
• The rule-based strategy uses patterns based on token attributes, with spaCy’s Matcher to find matching tokens in ChatGPT’s responses.The authors derive and refine rules using expert experience and ChatGPT’s historical output habits.Source: §3.4 · p. 13
• The similarity-based strategy compares responses with category-specific pivot answers and selects the answer with the smallest Levenshtein edit distance.The authors note that character-based literal matching lacks semantic understanding, which PLM-based methods handle better.Source: §3.4 · p. 13
• When a category remains uncertain after rule-based parsing, the pipeline applies similarity matching; the authors report improved accuracy and recall for answer mapping.The similarity comparison uses each category’s answer examples.Source: §3.4 · p. 13, Figure 10 · p. 13

4. Experimental Setup

• The setup is intended to test which strategy factors influence ChatAgri’s final agricultural text-classification performance.The section introduces datasets, comparison baselines, evaluation metrics, and hyperparameters for experiments reported in Section 5.Source: §4 · p. 14

4.1. Datasets

• The five datasets span different classification topics and languages; Agri-News-Chinese adds seven categories and approximately 60,000 samples split 9:1 into train and test sets.The datasets include sentiment, plant-health, and disaster-related tasks; Natural-Hazards-Type is a smaller subset reorganized for disaster-type classification.Source: §4.1 · p. 14

4.2. Baselines

• Comparisons cover traditional machine learning, word-embedding deep learning, PLM fine-tuning, PLM prompt learning, and ChatGPT QA-based zero-shot learning.The setup includes SVM, Random Forest, TextCNN, TextRNN, BERT fine-tuning, and T5- and BART-based prompt tuning.Source: §4.2 · pp. 14–15

4.3. Evaluation Metrics

• Accuracy measures correctly predicted samples over all evaluated samples, while weighted-F1 accounts for category performance under varying weights.The paper describes F1 as considering precision and recall.Source: §4.3 · pp. 15–16

4.4. Hyperparameter Settings

• Final hyperparameters were selected based on development-set performance; baselines used GloVe and base BERT, T5, and BART, while prompt baselines shared a predefined template.Experiments used Python 3.7, PyTorch 1.9.0, and a single Nvidia GTX 1080Ti GPU.Source: §4.4 · p. 16

5.1. Methods Comparison

• ChatAgri matched or exceeded SOTA accuracy and weighted-F1, including about 3.0% accuracy over fine-tuned BERT and approximately 2.2% weighted-F1 over prompt-tuned BART on PestObserver-France.The reported comparison also gives ChatAgri gains of 0.6% in accuracy and weighted-F1 over both models on Natural-Hazards-Type.Source: Table 3 · p. 17, §5.1 · p. 17
• ChatAgri performed better on PestObserver-France and Agri-News-Chinese than on the English datasets; the authors speculate this may reflect broader language coverage in ChatGPT’s training corpus.The paper reports poorer performance on Amazon-Food-Comment and Natural-Hazards-Twitter, and presents the language-corpus explanation as speculation.Source: §5.1 · pp. 17–18

5.2. Improving ChatGPT with Advanced Prompting Strategies

• Chain-of-Thought Triggered-Prompts improved accuracy and weighted-F1 by an average of 2.7% on Agri-News-Chinese versus ChatAgri-base.It suits many-category tasks; Zero-Shot Similarity-Prompts helps the first three datasets but trails basic prompts on two, and similar pivots may confuse ChatGPT; Natural-Hazards-Type is near saturation.Source: §5.2 · pp. 18–19
• ChatGPT Triggered-Prompts outperformed manually defined prompts in most cases, with average accuracy gains of 2.1% on PestObserver-France and 1.1% on Agri-News-Chinese.The authors conclude that prompt engineering should incorporate ChatGPT’s own understanding and feedback.Source: §5.2 · p. 18, Table 4 · p. 19

5.3. Few-shot prompt-tuning and zero-shot ChatAgri

• ChatAgri-base exceeded prompt-tuning baselines on all test datasets, with approximately 10.5%, 15.1%, and 10.8% accuracy gains over 50-shot BART on three datasets.Separately, zero-shot BART achieved 57.3% accuracy versus 33.3% for average-probability guessing on Natural-Hazards-Twitter.Source: Table 5 · p. 20, §5.3 · p. 20

5.4. Potentials between ChatGPT and GPT-4

• GPT-4-based ChatAgriα gained about 2.9% accuracy and 3.1% weighted-F1 over GPT-3.5 on PestObserver-France; ChatAgriβ averaged 1.7% accuracy gains on two datasets.The paper also reports higher GPT-4 accuracy on complex semantic texts and describes its performance as more stable than vanilla ChatGPT.Source: Figure 11 · p. 21, §5.4 · p. 22
• GPT-4’s 25-message-per-three-hours cap constrained experiments, which used three language datasets, 100 randomly selected evaluation samples per experiment, and five rounds.The authors describe these reductions as a trade-off between experimental effectiveness and resource consumption.Source: §5.4 · pp. 21–22

• The study addresses limited systematic evidence on ChatGPT for agricultural classification, a task the paper describes as important for organizing growing agricultural information.Source: §6 · p. 22
• Experiments and ablation studies revealed the superiority of applying ChatGPT in agricultural text classification.Source: §6 · p. 22
• ChatAgri is evaluated against traditional machine learning, PLM fine-tuning, and PLM prompt learning on English, French, and Chinese datasets, with prompt strategies and GPT-4 comparisons also examined.Source: §6 · p. 22
• The authors envision future ChatGPT applications in smart agriculture, including crop and land management, as opportunities for agricultural transformation and sustainable development.Source: §6 · pp. 22–23