Harnessing Large Vision and Language Models in Agriculture: A Review

1.1. The challenges facing the agricultural domain

• Agriculture needs at least 70% more food production for a growing population, but diseases, poor seeds, inefficient tasks, and food-safety concerns threaten yields.Crop diseases, low-quality seeds, and inefficient weeding, planting, watering, and harvesting contribute to economic and production losses.Source: §1.1 · pp. 2–3
• Traditional agricultural solutions are difficult to scale because disease detection requires bulky instruments, seed assessment is time-consuming and destructive, and pesticide use can reduce yields.The section calls for methods that are fast, simple, and easy for agricultural workers to use.Source: §1.1 · pp. 2–3
• Large models show strong performance in agricultural data analysis, pest and disease management, and precision agriculture, despite data, training-efficiency, distribution-shift, and plant-blindness problems.The review aims to analyze large models and their agricultural applications while addressing ethical responsibility, challenges, and future directions.Source: §1.1 · pp. 3–4

1.2. The history of LLM and LVM

• Language-model development progresses from statistical language models to neural language models, pretrained language models, and large language models.Statistical models face parameter, semantic-sharing, and sparsity limitations; neural models improve effectiveness, while pretrained models use feature extraction or fine-tuning.Source: §1.2 · pp. 4–6
• Neural and pretrained language models address earlier limitations through distributed representations, recurrent architectures, bidirectional context, and transferable parameters.LSTM gates address long-term dependence, while Transformer self-attention accelerates training and becomes fundamental to LLMs.Source: §1.2 · pp. 5–6
• LLMs contain billions or more parameters and can exhibit emergent abilities, improved sample efficiency, and behaviors distinct from smaller pretrained models.The cited discussion identifies scale as a distinguishing feature between LLMs and PLMs.Source: §1.2 · p. 6
• MLLMs integrate images, text, language, audio, and other modalities, combining LLM and LVM strengths while addressing their limitations.LLMs perform well on text tasks but struggle with other data types, whereas LVMs perform well in computer vision but provide limited analysis information.Source: §1.2 · pp. 7–8

1.3. The current developed large models

• Industries are investing manpower, material resources, and financial resources to develop large models suited to professional tasks.The section identifies Table 1 as presenting examples of such industry-oriented large models.Source: §1.3 · p. 8

1.4. The current large models in other domains

• Current large models are mainly LLMs and MLLMs, but generic models trained on broad datasets may not provide satisfactory knowledge in specialized fields.Researchers therefore optimize existing models for specific domains, including finance, meteorology, marine science, medicine, and robotics.Source: §1.4 · p. 11

1.5. The importance of large models in the agricultural domain

• Although agricultural technology has improved production efficiency, continued gains remain essential as the global population grows.Global grain production was forecast at 2.84 billion tons in 2023, nearly twice the early-20th-century level.Source: §1.5 · pp. 11–12
• Large models can lower the threshold and improve efficiency for detecting crop pests and diseases, reducing reliance on manual expertise and preventing missed intervention opportunities.Farmers can upload crop photographs to a disease- and pest-focused model for analysis.Source: §1.5 · p. 12
• Large-model-enabled agricultural robots can identify and trim weeds, addressing competition between weeds and crops that can reduce crop yield and quality.Weeds compete with crops for sunlight, water, space, and nutrients, making weeding an important agricultural task.Source: §1.5 · p. 12
• Accurately identifying high-vitality seeds with large models can improve crop yield and quality, while large models can grade mature crops efficiently and reduce time to market.Seed vitality affects stress resistance, germination, emergence, and eventual yield; crop grading includes removing decaying produce and classifying quality.Source: §1.5 · pp. 12–13
• Although large models play a crucial role in agriculture, agricultural papers comprise a comparatively small share of AI research, limiting attention to the field.Fig 1 reports 24 agriculture papers among 3,496 selected post-2019 arXiv papers, compared with larger counts in NLP, medicine, industry, automation, and computer vision.Source: Figure 1 · p. 13, §1.5 · p. 13
• The review aims to identify which large models can be used in agriculture and how, covering the application prospects and practical uses of LLMs and LVMs.Its stated objective is to reveal the importance of large models in the agricultural sector.Source: §1.5 · p. 14

2. Large Language Models in Agricultural Applications

• The described agricultural reasoning process requires a large model to identify abnormal symptoms in soybean leaves, determine bacterial wilt disease and its underlying cause, and develop treatment, such as bactericide spray.Source: §2 · p. 14

2.1. The role of LLM in processing and generating agricultural data, providing insights and decision-making support

• LLMs process and generate agricultural data, provide production insights, and support farmers’ decision-making.Their agricultural roles include extracting structured information, generating data, and supporting decisions across agricultural production work.Source: §2.1 · p. 15

2.1.1. LLMs processing and generating agricultural data

• LLMs can extract structured information from unstructured agricultural text, including relationships between entities, key facts, and information used to populate knowledge graphs.Source: §2.1.1 · p. 15

2.1.2. LLMs provide insights

• LLMs analyze agricultural text to reveal market and pricing trends and can inform design decisions, but their insights require human judgment and do not replace technical support.In robotic tomato-picking gripper design, ChatGPT helped identify challenges and opportunities, assess feasibility, and reduce failure risk, although LLMs cannot yet independently generate comprehensive CAD models, evaluate code, or fabricate robots.Source: §2.1.2 · pp. 16–17

2.1.3. LLMs provide decision-making support for farmers

• LLMs act as conductors: they divide farmers’ total tasks into subtasks, select appropriate AI models, and coordinate those models to complete subtasks and gradually solve complex tasks.Source: §2.1.3 · p. 19
• LLMs play a core role in decision-making support for farmers.Source: §2.1.3 · p. 19

2.2. Few-shot learning of LLM in the agricultural domain

• Few-shot learning is useful in agriculture because data collection and labeling are difficult and expensive, while limited samples can still support generalization.FSL includes few-shot, one-shot, and zero-shot settings, using 2–5 samples per class, one sample per class, or no samples for invisible classes, respectively.Source: §2.2 · pp. 19–20
• FSL has limited generalizability relative to fully supervised learning because related agricultural tasks require dedicated models, while Internet images differ from real agricultural images.Agriculture lacks labeled data, producing poorly generalized supervised models; current LLMs also primarily process text, motivating multimodal models to overcome this limitation.Source: §2.2 · pp. 20–21
• LLMs demonstrate strong few-shot performance and zero-shot generalization, reducing the need for task-specific agricultural data and fine-tuning.GPT-3 performed strongly on diverse NLP tasks including translation, question answering, and cloze.Source: §2.2 · p. 20

3. Large Vision Models in Agricultural Applications

• LVMs use only image data to learn general visual knowledge and adapt across visual tasks, unlike LVLMs, which combine language models with visual capabilities.Agricultural applications of large models currently focus mainly on computer vision for diseases, pests, weeds, seeds, and crops, while also supporting robotic automation.Source: §3 · p. 21

3.1. Image processing and analysis

• Using an LVM to judge crop-related information can greatly improve judgment time, indirectly reduce crop damage, process spectral remote-sensing big data, and help identify and treat diseases.Source: §3.1 · pp. 22–23

3.2. Automation and robotics

• Embedding LVMs such as SAM and TAM into agricultural robots enables crop monitoring, segmentation, yield prediction, harvesting, classification, and close-range pest detection.TAM emphasizes short-term rather than long-term memory, so improving long-term memory could support early disease-change monitoring and farmer warnings; LLMs also assisted tomato-picking gripper design.Source: §3.2 · p. 25
• LVMs can outperform traditional agricultural vision models, but slower inference limits real-time robotics; architecture, efficient algorithms, and hardware acceleration can improve responsiveness.Traditional models are small because agricultural robots require real-time operation, often producing unsatisfactory recognition results, whereas LVMs use pre-trained large-scale datasets.Source: §3.2 · p. 26

3.3. LVLM compared to LVM

• LVLMs combine visual and textual information to comprehend images, generate captions, and answer questions, whereas LVMs rely solely on images.Source: §3.3 · pp. 26–27

4.1. Integration of multimodal models

• Multimodal agricultural systems integrate text, images, audio, video, sensor data, depth information, and point clouds.Agricultural robotics datasets can additionally include stereo colour, thermal, hyperspectral, weather, and soil information.Source: §4.1 · p. 27
• Agricultural multimodal learning is being explored to optimize processes and improve outcomes, including robotics applications involving fruit picking and crop monitoring.The review identifies agricultural robotics as an opportunity to combine images, text, voice, and depth information.Source: §4.1 · pp. 27–29
• 94.84% recognition accuracy was achieved on a moderately sized multimodal cucumber disease dataset using image-text-label contrastive learning.The approach combined multimodal contrastive learning, image self-supervised contrastive learning, and a shared image-text-label space.Source: §4.1 · p. 28
• Existing models primarily rely on text-image data, remain mostly limited to QA tasks, and have few agricultural robotics applications using broader inputs.The review links limited multimodality with fewer tasks and weaker applicability.Source: §4.1 · pp. 28–29

4.2. Results and impact assessment

• The assessment framework distinguishes timely ethical results from outdated or unethical results, with category D representing the worst case.As time passes, category A can become C and category B can become D; evaluations also vary across questioners because ethical judgments differ.Source: §4.2 · p. 29
• POPE-based assessment asks yes-or-no image questions and evaluates answer accuracy across criteria categorized by ethics and timeliness.The framework uses categories A–D and subcategories A1–A5, with priority A1>A2>A3>A4>A5.Source: §4.2 · p. 30

Agriculture

• Agricultural large models raise ethical and responsibility concerns because unequal access creates unfairly distributed risks and benefits, while AI biases exacerbate the divide.The section therefore emphasizes exploring corresponding measures for ethical and responsible use.Source: Agriculture · pp. 30–31

5.1. Ethical issue of large language models to guide farming

• Weidinger et al. identify six risks—malicious uses, interaction harms, automation and environmental harms, information hazards, misinformation, and discrimination, exclusion, and toxicity—in agricultural applications.Poorly performing models may mislead farmers about crop diseases, damage crops, and reduce trust, while data-sharing guidelines should address conduct, privacy, and dataset governance.Source: Figure 6 · p. 31, §5.1 · pp. 32–33

5.2. Responsible use in agriculture

• Large-model development, deployment, and use require agile, effective regulatory oversight and risk evaluation through a risk-based framework focused on transparency and privacy.Source: §5.2 · p. 34
• Developers should obtain explicit consent, explain data use, and offer individuals choices to opt out or request deletion.These practices address potential harm to information owners from unintentionally learned personal information.Source: §5.2 · p. 35
• Agricultural large-model systems should collect and retain only necessary personal or sensitive data, anonymizing or aggregating it whenever possible.This applies the principle of data minimization to privacy protection.Source: §5.2 · p. 35
• Outcome-focused governance combined with ethics can expedite large-model implementation in agriculture and foster innovation at larger scale.The passage emphasizes responsible utilization and governance approaches that focus on outcomes rather than technology itself.Source: §5.2 · pp. 35–36

6.1. Technical and practical challenges

• Agricultural large models require comprehensive, accurate, diverse, and often private time-series data, making collection, labeling, standardization, and sharing difficult.Farmers may hesitate to share data because of privacy or commercial-exploitation concerns, while crop changes require longitudinal collection.Source: §6.1 · p. 36
• Training agricultural large models is costly and slow because they require millions of training examples, thousands of GPU hours, and substantial computational resources.Higher parameter counts and massive-data preprocessing further increase training complexity.Source: §6.1 · pp. 36–37
• Distribution shifts across crops, soils, weather, regions, and practices can reduce applicability and performance on agricultural tasks.The review cites unsatisfactory zero-shot leaf segmentation as an example attributed to possible distribution shifts.Source: §6.1 · p. 37
• Training-data lag can make model outputs deviate from current facts after deployment.The review identifies data timeliness as a distinct challenge and illustrates it with Fig 7.Source: §6.1 · p. 37, Figure 7 · p. 38
• 114?Source: Figure 8 · p. 38, §6.1 · p. 39
• Future development should use model compression, efficient network designs, update and maintenance functions, and usage instructions.These measures are presented as ways to reduce model size, preserve timeliness, and help users adopt models.Source: §6.1 · p. 39

6.2. Future trends in the integration of agricultural and food sectors and large models

• Future agricultural multimodal models may incorporate video and audio alongside text and images, such as analyzing crops in videos or judging watermelon maturity by sound.The review argues that agricultural models should not be limited to text and image inputs.Source: §6.2 · p. 39
• Agricultural large models may promote food-domain models and create mutual positive feedback between the two fields.The review also emphasizes trust-building with governmental agencies, funders, and food-system partners.Source: §6.2 · pp. 39–40

• The study investigates current agricultural large-model applications and outlines their future development directions.Source: §7 · p. 40
• Agricultural large models can greatly improve production efficiency and move agriculture toward “smart agriculture” and “unmanned agriculture”.Source: §7 · p. 40
• Applicability and reliability remain unresolved requirements for deploying large models in agriculture.Source: §7 · p. 40
• Agricultural and food large models are expected to develop together through mutual positive feedback.Source: §7 · p. 40
• Erroneous model information could mislead farmers, damage farmland, and increase distrust of new technologies.Source: §7 · p. 40
• Future agricultural models should serve as a support and cornerstone for continued development.Source: §7 · p. 40