AgriBench: A Hierarchical Agriculture Benchmark for Multimodal Large Language Models

• Conventional ML and DL models require extensive task-specific labeled datasets and do not generalize across tasks, while agriculture poses complex expert multimodal challenges and lacked an MM-LLM benchmark.Source: §1 · p. 2
• MM-LUCAS addresses the lack of agricultural datasets with 1,784 annotated agricultural scenery images from 27 EU countries.Source: §1 · p. 2
• MM-LLMs understand general agricultural content well but struggle with specific problems such as plant-disease diagnosis without fine-tuning.Source: §1 · p. 2
• AgriBench is the first hierarchical benchmark for evaluating MM-LLMs’ agricultural comprehension and reasoning.Source: §1 · p. 2

• Existing MM-LLMs handle diverse multimodal tasks, but still face challenges with specialized research and real-world domains such as agriculture and healthcare.Source: Multimodal Large Language Models… · p. 3
• Current MM-LLM benchmarks emphasize vision-text understanding, and some cover only specific visual-understanding aspects, while specialized-domain evaluation requires expertise and precision.Source: Benchmarks for Multimodal Large… · p. 3
• LUCAS surveys EU land use and land cover through microdata, landscape images, and statistical tables; prior studies built datasets from it, including a segmentation dataset used in this work.The survey collected data from 1,351,293 points at 651,780 unique locations and included 5.4 million scenery photos.Source: Land Use/Cover Area Frame Survey (LUCAS) · p. 3

3 AgriBench

• AgriBench evaluates agricultural MM-LLMs across five task-complexity levels and multiple modalities, treating task and multimodal complexity as distinct but correlated axes.The benchmark uses diverse scenario tasks intended to reflect real-world agricultural challenges.Source: §3 · p. 4

3.1 Hierarchical Standard

• The hierarchy spans L1 Basic Recognition to L5 Human-AI Aligned Suggestion, addressing agriculture’s range from object detection to high-stakes decisions with multiple convincing answers.The ambiguity of high-stakes planning and management motivates a human-centered AI framing.Source: §3.1 · p. 4

3.2 The 5 Levels of Agriculture MM-LLMs Evaluation Strategy

• The evaluation strategy organizes 17 main agriculture tasks across five capability levels, using task examples and descriptions spanning perception to cognition and simple to complex tasks.Tasks are represented with text and image input/output combinations such as T→T, I→T, and T+I→I.Source: §3.2 · pp. 4–5
• The levels distinguish intuitive visual recognition, objectively extractable properties without inference, fine-grained judgments with some subjectivity, knowledge-guided inference, and justified suggestions for future action.Examples include species classification at Level 1, counting at Level 2, image enhancement at Level 3, crop-yield estimation at Level 4, and strategic planning at Level 5.Source: Level 1: Basic Recognition · p. 5, Level 3: Fine-grained Recognition · p. 5, Level 4: Knowledge-guided Inference · p. 6

4 MM-LUCAS Dataset

• MM-LUCAS pairs 1,784 scenery images with corresponding segmentation masks and depth maps, quality and aesthetic assessments, and landscape question answering across four topics.The dataset is designed to follow AgriBench and is based on the segmentation LUCAS dataset.Source: §4 · p. 7

4.1 Data Collection

• Images come from 1,784 sites in 27 EU countries, with geographical location, date, land-cover, and land-use information; masks cover 34 classes in grayscale and color-coded forms.The images are 1600×1200 pixels and were captured horizontally at eye height.Source: Image & Basic Information · p. 8, Segmentation Mask · p. 8

4.2 Data Processing

• Depth maps use Depth Anything V2-Large, while Q-Align supplies image-quality and aesthetic ratings designed to align with human discrete-level ratings.The dataset’s visual assessments are limited to image quality and aesthetics.Source: Depth Estimation · p. 8, Visual Assessment · p. 8
• GPT-4o generates LandQA from images and their land-cover, land-use, quality, and aesthetic annotations, using multiple-choice formats for land labels and direct answers for ratings.The aesthetic scale runs from 0–1 (“Bad”) through 4–5 (“excellent”).Source: Visual Assessment · p. 9, Landscape Question Answering (LandQA) · p. 9

4.3 Data Analysis

• Common wheat is the most frequent land-cover category, while agriculture is the most prevalent land use and forestry is a substantially less frequent second.The land-use categories also include forestry, natural areas, and fallow land.Source: Figure 3 · p. 9, §4.3 · p. 10
• Flower and cropfield have average aesthetic scores of 2.45 and 2.46, respectively, while infrequent orchards average 2.04.The analysis relates segmentation classes to average aesthetic score.Source: §4.3 · p. 10, Figure 4 · p. 11
• Quality scores are higher in France, Germany, and the United Kingdom, while aesthetic scores are higher in France and parts of Romania and Bulgaria and lower in Finland, Estonia, and Lithuania.Romania and Bulgaria show a mix of high- and low-quality images.Source: §4.3 · pp. 10–11, Figure 5 · p. 11
• Aesthetic scores generally peak in summer and fall in winter; frequent classes vary more seasonally than less frequent classes, with flowers and fruit showing the greatest variability.Field margins are more consistent across seasons.Source: §4.3 · p. 12, Figure 6 · p. 12

5.1 Mainstream MM-LLMs for Evaluation and Results

• The evaluation covers two open-source models, InterVL2-26B and mPLUG-Owl2, and three closed-source models, GPT-4o, Gemini-1.0 Pro, and Claude 3.5 Sonnet.The models are evaluated on two tasks using official implementation sources.Source: §5.1 · p. 12

5.2 Results Analysis

• For basic vegetable-crop questions, Claude 3.5 Sonnet and GPT-4o give broader lists than InterVL2-26B and mPLUG-Owl2, while Gemini-1.0 Pro groups crops by production scale.The analysis says Gemini’s categories highlight the economic significance and scale of different crops.Source: §5.2 · p. 13, §5.1 · p. 13
• For contextual scene analysis, all five MM-LLMs rate aesthetic score between 4 and 4.5 out of 5, with Claude 3.5 Sonnet and GPT-4o giving more comprehensive visual analyses.The other models emphasize the tree and contrast; Claude 3.5 Sonnet and GPT-4o additionally highlight seasonal beauty and light.Source: §5.2 · p. 13, §5.1 · p. 13

Correlation of MM-LUCAS and AgriBench

• Segmentation masks and depth maps provide visual and spatial information for assessing agricultural tasks and analyzing relationships between scene objects and their surroundings.The discussion names species classification, object counting, dense object counting, contextual scene analysis, environmental impact prediction, scene projection, scene generation, and sustainability recommendations.Source: §6 · p. 14

Responsible AI

• Responsible AI in agriculture requires interpretable, explainable decisions that support farmers' informed choices and regulatory compliance.The discussion identifies bias, fairness, transparency, and human oversight as relevant considerations.Source: §6 · p. 14

Limitations and Future Works

• The initial evaluation is primarily qualitative; planned additions include perception and cognition metrics.Proposed measures include user satisfaction scores and expert reviews for perception, and accuracy rate and consistency score for cognition.Source: §6 · p. 14

7 Conclusion

• AgriBench evaluates MM-LLMs across multimodality and multitask dimensions to cover a broader range of agricultural scenarios and applications.The conclusion describes it as the first agriculture benchmark of this kind.Source: §7 · p. 14
• MM-LUCAS is introduced as a multimodal agriculture dataset with multiple annotations aligned with AgriBench.The conclusion characterizes the dataset and benchmark as an initial step and work in progress.Source: §7 · p. 14
• The paper compares five MM-LLMs on AgriBench and highlights insights for further exploration of specialized domain models.The conclusion presents this comparison as a source of directions for future expert knowledge-based MM-LLMs.Source: §7 · p. 14