Source-linked AI summary
Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis
Mang Tik Chiu, Xingqian Xu, Yunchao Wei, Zilong Huang, Alexander Schwing, Robert Brunner, Hrant Khachatrian, Hovnatan Karapetyan, Ivan Dozier, Greg Rose, David Wilson, Adrian Tudor, Naira Hovakimyan, Thomas S. Huang, Honghui Shi
TL;DR
Agricultural visual recognition lacks suitable datasets, and aerial farmland segmentation adds challenges from extremely large images and sparse annotations. The paper introduces Agriculture-Vision, evaluates established and specialized segmentation models, and reports benefits from NRGB imagery and effective transfer learning. It also identifies extreme-image-size learning as an area requiring further exploration.
Problem
Agricultural visual pattern recognition has progressed slowly because suitable datasets are scarce, while aerial farmland segmentation involves extremely large images and sparse annotations.
Method
The paper constructs a large-scale aerial farmland dataset with multiple channels and expert anomaly annotations, then evaluates established segmentation models alongside an FPN-based model.
Results
Experiments show that NRGB imagery is more effective than RGB alone, transfer learning remains effective across the RGB-to-multispectral domain gap, and the proposed dataset-trained model performs best across dataset versions.
Takeaways & Limitations
Agriculture-Vision provides a baseline and research resource for large-scale, multi-channel aerial farmland semantic segmentation.
Takeaways & Limitations
The current version uses conventional image sizes, leaving effective use of extremely large aerial images as an open research direction.
Abstract
from arXiv · showhide
The success of deep learning in visual recognition tasks has driven advancements in multiple fields of research. Particularly, increasing attention has been drawn towards its application in agriculture. Nevertheless, while visual pattern recognition on farmlands carries enormous economic values, little progress has been made to merge computer vision and crop sciences due to the lack of suitable agricultural image datasets. Meanwhile, problems in agriculture also pose new challenges in computer vision. For example, semantic segmentation of aerial farmland images requires inference over extremely large-size images with extreme annotation sparsity. These challenges are not present in most of the common object datasets, and we show that they are more challenging than many other aerial image datasets. To encourage research in computer vision for agriculture, we present Agriculture-Vision: a large-scale aerial farmland image dataset for semantic segmentation of agricultural patterns. We collected 94,986 high-quality aerial images from 3,432 farmlands across the US, where each image consists of RGB and Near-infrared (NIR) channels with resolution as high as 10 cm per pixel. We annotate nine types of field anomaly patterns that are most important to farmers. As a pilot study of aerial agricultural semantic segmentation, we perform comprehensive experiments using popular semantic segmentation models; we also propose an effective model designed for aerial agricultural pattern recognition. Our experiments demonstrate several challenges Agriculture-Vision poses to both the computer vision and agriculture communities. Future versions of this dataset will include even more aerial images, anomaly patterns and image channels. More information at https://www.agriculture-vision.com.
1. Introduction
Agricultural visual recognition has progressed slowly because suitable datasets are scarce, while aerial farmland segmentation combines economic importance with unusually large images, sparse annotations, and complex field patterns. Agriculture-Vision addresses this gap with a large, high-quality aerial dataset and baseline semantic-segmentation experiments.
- Agricultural visual recognition has advanced more slowly than other computer-vision domains, partly because relevant datasets are lacking.
- Timely detection of field conditions could help prevent major losses or increase potential yield throughout the growing season.
- Aerial farmland segmentation is challenging because patterns such as weeds may form sparse clusters across vast images.
- Agriculture-Vision combines resolutions up to 10 cm per pixel, aligned non-RGB channels, multiple anomaly patterns, agronomist annotations, and varied annotation sizes.
- The paper introduces a large-scale aerial agricultural image database and conducts pilot experiments that provide a deep-learning segmentation baseline.
2. Related Work
Prior segmentation datasets largely emphasize everyday objects, street scenes, or aerial imagery with different scale and resolution characteristics. The paper positions Agriculture-Vision as addressing the apparent shortage of large-scale aerial agricultural image databases.
- Pascal VOC, MS-COCO, and ADE20K focus on daily object categories, while Cityscapes provides dense street-scene annotations.
- Aerial datasets commonly contain much larger images than object datasets, but often use substantially lower ground resolutions.
- The paper identifies an apparent lack of large-scale aerial agricultural image databases relative to related aerial and agricultural datasets.
- Agricultural imaging studies cover spectral analysis, photogrammetry, color indices, low-level features, integrated systems, land-cover classification, and weed detection.
3. The Agriculture-Vision Dataset
Agriculture-Vision is a high-resolution, multi-band aerial farmland dataset with agronomist-reviewed annotations for nine agricultural patterns. Its construction addresses very large images, sparse annotations, and varied pattern geometry through annotation-centered patching and farmland-level splits.
- Dataset scope: Agriculture-Vision contains 3,432 US farmland images with nine agronomic annotation types, captured across multiple growing seasons.The patterns include double plant, drydown, endrow, nutrient deficiency, planter skip, storm damage, water, waterway, and weed cluster.
- Field image acquisition: Each field image has four aligned channels—Near-infrared, Red, Green, and Blue—with 2018 Blue images scaled to align with NRG imagery.The dataset separates these channels into RGB and single-channel NIR JPG images for visualization and experiments.
- Annotations: Annotation labels were produced by five trained annotators and reviewed by expert agronomists, who improved unsatisfactory annotations.The annotation software provided RGB, NIR, and NDVI visualizations.
- Dataset challenges: Field images can be extremely large and sparsely annotated, making whole-image segmentation inefficient and computationally demanding.One example is 10,875 × 3,303 pixels, while the largest collected image is 33,571 × 24,351 pixels.
- Image sample generation: The dataset generates 512 × 512 patches around annotations, using centered crops for small patterns and non-overlapping sliding windows for larger ones.Patches with over 90% annotation coverage, only background pixels, or redundant overlap above 30% are discarded under the stated rules.
- Dataset splitting: Randomly splitting 3,432 farmland images before assigning crops prevents the same farmland from appearing across train, validation, and test sets.The resulting split contains 56,944/18,334/19,708 train/val/test images.
4. Dataset Statistics
Agriculture-Vision exhibits strong variation in annotation size, frequency, and image coverage. These properties create severe class imbalance, scarce storm-damage examples, and difficult contextual segmentation conditions.
- Annotation area: Large and common field patterns occupy substantially more pixels than small and rare patterns, producing extreme label imbalance.Drydown, nutrient deficiency, and weed cluster account for significantly more annotated pixels than other categories.
- Annotation frequency: The frequency with which models observe each field pattern during training affects their ability to recognize it during inference.Understanding sample distributions is therefore important for interpreting dataset behavior and model performance.
- Annotation counts: Storm damage annotations are extremely scarce, so the authors exclude them from evaluation.The scarcity appears as a sudden drop in the number of images containing storm damage patterns.
- Annotation proportions: Many images contain more than 50% annotated pixels, and some contain more than 80%, making large-pattern segmentation difficult.Recognition relies heavily on contextual information from the surrounding field.
5. Pilot Study on Agriculture-Vision
The pilot study compares established semantic segmentation models with a specialized FPN-based model on Agriculture-Vision, while evaluating multispectral inputs, backbone complexity, and image-sampling scales. The proposed model consistently outperforms DeepLab baselines, and the 512 × 512 dataset version is most stable across cross-evaluation.
- Baseline models: DeepLabV3 and DeepLabV3+ are evaluated alongside a specialized FPN-based model for Agriculture-Vision semantic segmentation.The study compares models with different output strides and uses the FPN-based model for subsequent comparison studies.
- The proposed FPN-based model: The FPN-based model uses a ResNet encoder with a dilated final residual block, feature-pyramid lateral connections, and learned upsampling modules.Its pyramid outputs are upsampled, concatenated, and processed for prediction.
- Baseline comparison: The proposed FPN-based model consistently outperforms DeepLabV3 and DeepLabV3+ on both validation and test results.The model is customized for aerial agricultural images.
- Multi-spectral data and model complexity: Agriculture-Vision experiments compare RGB, NRG, and NRGB inputs to assess the effectiveness of multispectral imagery for semantic segmentation.The dataset contains NRGB images, enabling comparisons beyond conventional RGB inputs.
- Multi-spectral data and model complexity: The experiments also compare ResNet-50 and ResNet-101 backbones across combinations of multispectral inputs and model complexity.Results for these backbone and channel combinations are reported in Table 5.
- Multi-scale data: The 512 × 512 Agriculture-Vision version is the most stable and performs best when models trained on each version are cross-evaluated on all three versions.The comparison includes Agriculture-Vision-1024 and Agriculture-Vision-MS, which retain larger annotated regions through alternative window sizes.
6. Discussion
The discussion highlights Agriculture-Vision as a basis for multispectral recognition and transfer learning, while identifying extreme image sizes as an open challenge. Experiments support NRGB inputs and RGB-to-multispectral transfer, but the current dataset remains a pilot with conventional image sizes.
- Agriculture images beyond RGB: NRGB images produce more effective aerial agricultural semantic segmentation than RGB images alone.The dataset is intended to support further research with additional channels such as thermal, soil, and topographic data.
- Transfer learning: Agriculture-Vision supports transfer learning from RGB common-object pretraining to multispectral agricultural segmentation despite a substantial source-target domain gap.The discussion identifies channel expansion and domain adaptation as directions for further study.
- Dataset versions: The model trained on the proposed 512 dataset version achieves the best results across the 512, 1024, and MS versions.Here, 512 denotes Agriculture-Vision, 1024 denotes Agriculture-Vision-1024, and MS denotes Agriculture-Vision-MS.
- Learning from extreme image sizes: The current dataset provides a pilot study using conventional image sizes, leaving effective learning from extremely large aerial images largely unexplored.The authors position large-scale aerial image analysis as a direction for future research.
7. Conclusion
Agriculture-Vision provides a large aerial agricultural semantic-segmentation dataset and a pilot baseline, while identifying challenges for this setting. Future versions are planned to expand its imagery, patterns, and modalities.
- Agriculture-Vision captures extremely large farmland images, provides multiple field-pattern annotations, and introduces new challenges for aerial agricultural semantic segmentation.
- The pilot study evaluates well-known off-the-shelf semantic-segmentation models and a specialized model on Agriculture-Vision.
- Later versions will add more field images, patterns, and modalities, including thermal images, soil maps, and topographic maps.