PaperlayerView exact revision on arXiv

Source-linked AI summary

Grape detection, segmentation and tracking using deep neural networks and three-dimensional association

Thiago T. Santos, Leonardo L. de Souza, Andreza A. dos Santos, Sandra Avila

arXiv:1907.11819v3cs.CV

TL;DR

Agricultural monitoring needs affordable methods to detect, segment and count individual fruits despite difficult field conditions and grape-cluster variability. This paper combines CNN-based detection and segmentation with annotated data and three-dimensional association, reaching F1 scores above 0.9 for wine-grape instance detection and demonstrating tracking and counting applications.

  • Problem

    Agricultural applications need affordable systems that detect, segment and count highly variable grape clusters under field conditions.

  • Method

    The paper introduces interactive mask annotation, a public grape dataset, CNN detectors, and three-dimensional association for tracking and localized counting.

  • Results

    F1 scores superior to 0.9 were achieved for wine-grape instance detection, while three-dimensional models tracked fruits and avoided double counts.

  • Takeaways & Limitations

    The pipeline can support affordable vineyard fruit counting and inspection using an off-the-shelf 1080p RGB camera and moving-camera tracking.

  • Takeaways & Limitations

    The reported implementation uses computationally intensive structure-from-motion, and the appropriate alternative when fruits are absent in a video segment remains unclear.

Abstract

from arXiv · show

Agricultural applications such as yield prediction, precision agriculture and automated harvesting need systems able to infer the crop state from low-cost sensing devices. Proximal sensing using affordable cameras combined with computer vision has seen a promising alternative, strengthened after the advent of convolutional neural networks (CNNs) as an alternative for challenging pattern recognition problems in natural images. Considering fruit growing monitoring and automation, a fundamental problem is the detection, segmentation and counting of individual fruits in orchards. Here we show that for wine grapes, a crop presenting large variability in shape, color, size and compactness, grape clusters can be successfully detected, segmented and tracked using state-of-the-art CNNs. In a test set containing 408 grape clusters from images taken on a trellis-system based vineyard, we have reached an F 1 -score up to 0.91 for instance segmentation, a fine separation of each cluster from other structures in the image that allows a more accurate assessment of fruit size and shape. We have also shown as clusters can be identified and tracked along video sequences recording orchard rows. We also present a public dataset containing grape clusters properly annotated in 300 images and a novel annotation methodology for segmentation of complex objects in natural images. The presented pipeline for annotation, training, evaluation and tracking of agricultural patterns in images can be replicated for different crops and production systems. It can be employed in the development of sensing components for several agricultural and environmental applications.

1. Introduction

Agricultural vision systems must detect and localize variable plant structures under difficult field conditions, while grape clusters require instance-level separation beyond bounding boxes. This work addresses that need with annotated data, deep detectors, and three-dimensional association for grape monitoring.

  • Field conditions and uncertain plant structure make fine-grained agricultural monitoring harder than industrial automation.
  • Accurate fruit detection and localization support counting, yield estimation, precision agriculture, disease monitoring, automated spraying and harvesting, and phenotyping.
  • Grape clusters require instance segmentation because rectangular bounding boxes do not fit their berries and cannot isolate each cluster precisely.
  • Wine grapes vary substantially in shape, size, color and structure, requiring field datasets that capture this variability.
  • The work introduces interactive mask annotation, a public five-variety grape dataset, evaluations of Mask R-CNN and YOLO, and three-dimensional association for localized counting.

2. Related Work

Earlier fruit-vision systems relied on engineered features and conventional classifiers, while later CNN-based approaches learned representations and enabled end-to-end detection. The related work frames instance segmentation and tracking as responses to grape-cluster variability and counting challenges.

  • Earlier works: feature engineering and machine learning: Earlier fruit-detection methods used hand-designed color, geometric and texture descriptors with Bayesian classifiers, support vector machines and clustering.
  • Earlier works: feature engineering and machine learning: Nuske et al. combined berry detection, classification and clustering, later adding visual-odometry association to avoid double-counting and estimate yield spatially.
  • Deep learning-based works: CNNs learn task-specific representations and can encode variation in pose, color and illumination when training data contains sufficient examples.
  • Deep learning-based works: Faster R-CNN performs detection in two stages, whereas YOLO belongs to the single-shot detector family and predicts classes and bounding boxes in one stage.
  • Deep learning-based works: Grape clusters vary more in size, shape and compactness than several other fruits, motivating cluster instance segmentation rather than only berry or box-based detection.
  • Deep learning-based works: Mask R-CNN jointly optimizes region proposals, bounding-box regression and semantic pixel segmentation, requiring instance-level pixel attribution in training data.

3. Materials and methods

The study combines a public grape dataset, interactive mask annotation, CNN-based detection and instance segmentation, augmentation, and 3-D association for tracking clusters across frames.

  • 3.1. The dataset: The WGISD contains 300 RGB images showing 4,432 grape clusters from five grape varieties captured in a real trellis-system vineyard.
  • 3.1. The dataset: The dataset provides binary masks for 2,020 clusters, supporting supervised training for grape-cluster instance segmentation.
  • 3.1. The dataset: Interactive graph-matching annotation starts from bounding boxes and user scribbles, then propagates grape and non-grape labels to produce cluster masks.
  • 3.2. The perceptual step: CNN architectures: Mask R-CNN combines Faster R-CNN detection with a fully convolutional network for end-to-end instance segmentation, while YOLOv2 and YOLOv3 provide additional detection architectures.
  • 3.3. Training: Training and test images were randomly divided in an approximately 80-20% split, with 88 masked images and 1,307 clusters assigned to training.
  • 3.3. Training: Twenty randomized augmentations per image simulated changes in lighting, focus, noise and dirty lenses to mitigate overfitting.
  • 3.5. Spatial registration: 3-D association: SfM-derived three-dimensional points spatially register CNN detections across frames by linking instances that share projected 3-D points.
  • 3.5. Spatial registration: 3-D association: Depth-first search extracts longest graph paths as grape-cluster tracks, filters paths shorter than five edges, and uses the remaining path count to estimate total clusters.

4. Results

The results evaluate Mask R-CNN for instance and semantic segmentation, compare three networks for object detection, and demonstrate 3-D association for tracking grape clusters. Performance is strong overall, while segmentation disagreement and cluster assignment remain important sources of evaluation error.

  • Tracking: 3-D assignment linked detections across video frames, and longest graph paths produced individual grape-cluster tracks.Paths shorter than five edges were filtered to remove false positives using evidence from multiple frames.
  • Instance segmentation: Mask R-CNN learned grape-cluster variability in shape, compactness, color, size, elongation, and occluding foreground structures.Most instances had confidence near 1.0 at a 0.9 grape-class threshold, while severe occlusion produced lower values.
  • Segmentation: Segmentation divergence produced false positives and false negatives even when grape berries were detected correctly.The disagreement reflected different assignments of berries to clusters and could reduce IoU.
  • Segmentation: 0.89 overall F1 was achieved for semantic segmentation across the masked test set, with no grape variety markedly different.The evaluation covered 27 images grouped by grape variety.
  • Object detection: Mask R-CNN retained more clusters than YOLO networks, especially YOLOv3, and performed better as the required bounding-box IoU increased.The comparison used 837 clusters in 58 images from the entire boxed test set.

5. Discussion

Mask R-CNN outperformed YOLO networks for grape-cluster detection, while qualitative tests indicated generalization across camera poses and developmental stages. The discussion also identifies annotation, computational, and dataset-size trade-offs affecting evaluation and deployment.

  • Model comparison: Mask R-CNN achieved superior results to YOLO networks, with its advantage becoming more salient at IoU values of at least 0.5.The authors caution that comparisons with other crops and datasets are references rather than direct benchmarks.
  • Model comparison: YOLOv3’s multiple scales may offer little advantage over YOLOv2 when camera distance to the vineyard row is nearly constant.The discussion similarly questions whether Mask R-CNN’s feature pyramid is necessary under these agronomical constraints.
  • Data and annotation: Rectangular bounding-box annotation is faster than mask annotation, potentially enabling larger YOLO training datasets than the dataset used for Mask R-CNN.In this work, YOLO training was constrained to the same dataset available to Mask R-CNN.
  • Generalization: Mask R-CNN detected many clusters without tuning in images with different poses, leaf textures, and developmental stages, although textured leaves produced false positives.Earlier-stage clusters were sometimes split in denser regions, suggesting tuning and transfer learning could improve accuracy.
  • Tracking: Structure-from-motion provides spatial integration for tracking but is computationally intensive, motivating consideration of real-time SLAM alternatives.The authors note that fruit-based landmarking may be less robust when fruits are absent from part of a video sequence.

6. Conclusion

The paper combines CNN-based perception with 3-D spatial association to detect, track, and count grape clusters from an affordable camera. It reports strong performance in variable vineyard imagery and identifies broader agricultural applications alongside remaining integration challenges.

  • Conclusion: The pipeline combines structure-from-motion or SLAM with CNNs to support agricultural monitoring and robotics applications.The conclusion presents this combination as a basis for advanced monitoring systems in agriculture and livestock.
  • Conclusion: The methodology detects, tracks, and counts wine grapes using a single off-the-shelf 1080p camera.Three-dimensional models support tracking, helping avoid double counts and increasing tolerance to detection errors.
  • Conclusion: F1 scores superior to 0.9 were reached for instance detection in wine grapes despite substantial variation in shape, size, color, and compactness.The reported crop is explicitly characterized as challenging because of this variability.
  • Conclusion: The methodology could extend to trellis-grown crops such as apples, peaches, and berries, with adaptations for larger-canopy fruits such as citrus and mangoes.For larger canopies, the paper suggests estimating yield from regression using visible fruit counts.
  • Future work: Further research should integrate photogrammetry and perception more closely to handle occlusions and other error sources robustly.The authors frame this as a direction toward more sophisticated scene-understanding systems.

Embrapa Wine Grape Instance Segmentation Dataset – Embrapa WGISD.

This section contains supporting material related to computer vision, photogrammetry, agricultural robotics, and numerical computing. The supplied passages primarily comprise cited references rather than a description of the dataset itself.

  • Related work: The supplied material references prior work on affordable 3-D grape phenotyping using a consumer webcam.The citation concerns automatic grape-bunch detection in vineyards.
  • Photogrammetry: The references include foundational work on visual odometry and structure-from-motion for spatial perception.These include a visual-odometry tutorial and a structure-from-motion study.
  • Computer vision: The cited literature covers semantic segmentation, fully convolutional networks, deep image recognition, and bundle adjustment.These references situate the paper’s computer-vision and photogrammetry methods within established technical areas.
  • Supporting technologies: Additional references concern NumPy, watershed segmentation, edge computing, and agricultural robotics.The supplied passages list these works as related technical or application literature.

Appendix A. Embrapa Wine Grape Instance Segmentation Dataset – Embrapa WGISD

The section introduces a detailed description of the dataset and presents a dataset datasheet following the proposal of Gebru et al. (2018).

  • Dataset documentation: The section presents a detailed description of the dataset together with a datasheet modeled on Gebru et al. (2018).The supplied passage identifies the datasheet as the organizing framework for this section.

Appendix A.1.1. Why was the dataset created?

The Embrapa WGISD dataset was created to support research on grape detection and instance segmentation for image-based viticulture monitoring and field robotics.

  • The dataset provides images and annotations for grape detection and instance segmentation.
  • It contains grape instances from five varieties captured in field conditions.
  • The instances represent variation in pose, illumination, focus, shape, color, and compactness.

Appendix A.1.2. What (other) tasks could the dataset be used for?

Beyond instance segmentation, WGISD supports classification, semantic segmentation, object detection, and grape variety identification.

  • The dataset can support classification of whether a grape appears in an image.
  • It can support semantic segmentation by identifying grape pixels in images.
  • It can support object detection by locating grapes in images.
  • WGISD can also be used for grape variety identification.

Appendix A.1.3. Who funded the creation of the dataset?

The dataset’s creation was supported by Embrapa’s SEG Project and Brazil’s CNPq PIBIC Program.

  • Embrapa SEG Project 01.14.09.001.05.04 supported the dataset’s creation.
  • The supporting Embrapa project focused on image-based metrology for Precision Agriculture and Phenotyping.
  • The CNPq PIBIC Program also supported the work through grants 161165/2017-6 and 125044/2018-6.
Loading 1907.11819v3…