PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Study of Fruit Detection and Counting Methods for Yield Mapping in Apple Orchards

Nicolai Häni, Pravakar Roy, Volkan Isler

arXiv:1810.09499v2cs.CVcs.RO

TL;DR

The paper addresses the challenge of accurately detecting, counting, tracking, and mapping apples in orchard rows, where fruits are viewed from both sides. It compares classical and deep-learning methods across datasets and finds different strengths for detection and counting. Combining the strongest detection and counting components yields high reported yield-estimation accuracy.

  • Problem

    Accurate apple detection and counting for yield mapping remains difficult because orchard environments are complex and fruits must be tracked across both sides of a row.

  • Method

    The study compares three fruit-detection methods and two fruit-counting methods, then combines per-frame processing, tracking, and two-sided 3D reconstruction for row-level yield estimation.

  • Results

    The classical GMM detector achieved the highest F1-score on six of seven datasets, while the CNN counting approach was more accurate, and the combined system achieved 95.56%–97.83% yield accuracy.

  • Takeaways & Limitations

    Classical detection remains strongest on most evaluated datasets, whereas neural-network counting provides more accurate and robust counting results.

  • Takeaways & Limitations

    The limited training data prevents conclusive conclusions about the generalizability of the U-Net and Faster R-CNN approaches.

Abstract

from arXiv · show

We present new methods for apple detection and counting based on recent deep learning approaches and compare them with state-of-the-art results based on classical methods. Our goal is to quantify performance improvements by neural network-based methods compared to methods based on classical approaches. Additionally, we introduce a complete system for counting apples in an entire row. This task is challenging as it requires tracking fruits in images from both sides of the row. We evaluate the performances of three fruit detection methods and two fruit counting methods on six datasets. Results indicate that the classical detection approach still outperforms the deep learning based methods in the majority of the datasets. For fruit counting though, the deep learning based approach performs better for all of the datasets. Combining the classical detection method together with the neural network based counting approach, we achieve remarkable yield accuracies ranging from 95.56% to 97.83%.

1 Introduction

The paper targets automated apple fruit detection, localization, counting, tracking, and yield mapping in complex orchard environments. It compares detection and counting methods and presents an end-to-end yield-estimation system.

  • Motivation: Specialty-crop automation is difficult because orchard geometry is more complex than commodity-crop fields.The paper focuses on apples, where accurate fruit detection and localization precede tasks such as diameter estimation, pruning, and picking.
  • Motivation: Automated yield mapping reduces reliance on sampling a few trees and extrapolating counts across an orchard.Prior work achieved 91.98%–94.81% yield-estimation accuracy while merging counts from both row sides without external navigational sensors.
  • Contributions: The study introduces fruit-detection and image-level fruit-counting methods based on neural networks.Its contributions include a U-net detection approach and an improved counting method over prior work.
  • Contributions: Three fruit-detection and two counting methods are evaluated on common datasets to enable head-to-head comparison.The authors propose that the experiments and datasets could support a future benchmarking platform.
  • Contributions: The yield-estimation contribution provides a complete system for processing fruit counts across an entire orchard row.The paper also reports the system’s strengths and weaknesses through experimental results.

2 Related Work

Related work spans classical, machine-learning, and deep-learning approaches to fruit detection, counting, tracking, and yield estimation. The paper emphasizes that many prior systems lack coherent two-sided row reconstruction, limiting count consistency in complex orchards.

  • Yield estimation: Prior yield-estimation systems combine fruit detection, counting, tracking, and count merging, but often use separate datasets or single-side relationships.The paper positions its comparison as a way to evaluate methods under common conditions.
  • Fruit detection: Classical fruit-detection methods commonly use thresholding with color or shape features, often under controlled illumination.Examples include artificial flashlights, over-the-tree sensors, and static or dynamic thresholding.
  • Fruit detection: Machine-learning approaches include SVM pixel classification, GMM detection and counting, and optical-flow tracking.Some systems manually calibrate counts or use user-supervised models to compensate for errors.
  • Deep learning: Deep-learning systems apply networks such as Faster R-CNN and FCN, sometimes combined with geometric tracking or watershed-based counting.Other work uses KLT tracking, the Hungarian algorithm, and Structure from Motion for fruit localization and size estimation.
  • Two-sided reconstruction: Most prior systems do not build a coherent geometric model from both row sides, which can produce over- or underestimation when trees are poorly pruned.The paper addresses this gap through two-sided reconstruction and count merging.
  • Fruit detection: The paper treats fruit detection as pixel-wise classification with U-Net and compares it against GMM and Faster R-CNN.This creates a direct comparison between a segmentation network, a classical semi-supervised method, and an object detector.
  • Fruit counting and tracking: Circular Hough Transform-based counting is limited by segmentation accuracy, occlusions, and extensive parameter tuning.Tracking remains necessary after detection and counting to avoid counting the same fruit across frames.
  • Fruit counting: The paper formulates clustered-fruit counting as multi-class classification using a CNN trained on apple-cluster image patches.The approach improves earlier work and adds extensive experimental validation.

3 Problem Formulation and Overview of the Entire System

The system estimates total fruit counts from paired front- and back-side image sequences by combining detection, counting, tracking, reconstruction, and duplicate removal. It compares alternative per-frame detection and counting components before merging the resulting views.

  • 3 Problem Formulation and Overview of the Entire System: The task is to estimate fruit counts and locations for the same captured tree-row portion from front- and back-side image sequences.The input can come from a monocular camera such as a cellphone or GoPro.
  • 3 Problem Formulation and Overview of the Entire System: The end-to-end solution requires fruit detection, counting, multi-view tracking, and merging counts from both row sides.These components address the linked subproblems of producing a total count for the captured row portion.
  • 3 Problem Formulation and Overview of the Entire System: Each row side is reconstructed independently and then merged into a coherent 3D model using semantic information.The merged model removes duplicate fruit counts caused by fruits visible from both sides without specialized hardware.
  • 3.1 Per Frame Fruit Detection and Counting: The per-frame component receives an individual image and outputs detected fruit clusters with corresponding counts.The paper evaluates multiple algorithms to identify methods suited to yield estimation.
  • 3.1 Per Frame Fruit Detection and Counting: For detection, the study compares U-Net image segmentation, Faster R-CNN object detection, and GMM color-based clustering.These alternatives represent neural segmentation, neural object detection, and classical clustering.
  • 3.1 Per Frame Fruit Detection and Counting: For clustered-fruit counting, the study compares an improved deep-learning method with a classical GMM and image-segmentation method.The deep-learning approach is evaluated as an alternative to classical counting.
  • 3.2 Tracking Fruits and Merging Fruit Counts Across Multiple Views: Tracking processes a single-side image sequence together with per-frame detections and counts to avoid overcounting.The system uses a previously proposed tracking method for this component.
  • 3.3 Merging Fruit Counts from Both Sides and Yield Estimation: The merging component combines single-side reconstructions and multi-view fruit counts, removes duplicates, and outputs the total captured-row fruit count.This completes the transition from per-frame observations to yield estimation.

4 Technical Approach

The paper develops and compares three fruit-detection approaches, including U-Net semantic segmentation, improved Faster R-CNN, and semi-supervised classical clustering, then addresses clustered-fruit counting and multi-view yield mapping.

  • Fruit detection by semantic segmentation: U-Net assigns apple or background labels to pixels and uses contracting and expansive paths with skip connections to preserve spatial information.The design targets small objects, occlusions, limited training data, and severe fruit-background imbalance.
  • Fruit detection by object detection: Faster R-CNN is improved with a Feature Pyramid Network and focal loss to address small-object detection and class imbalance.The implementation also uses a deeper ResNet50 backbone than the earlier VGG-based approach.
  • Fruit detection by semi-supervised clustering: The classical detector segments LAB-space SLIC super-pixels, clusters them into approximately 25 color classes, and classifies them as apple or background using KL divergence.User supervision supplies labeled color information for the model.
  • Fruit counting: The counting approach formulates clustered-apple counting as finite-class classification of detected regions of interest using a convolutional neural network.The method addresses arbitrary cluster sizes and occlusion-related counting difficulty.
  • Multi-view tracking and yield mapping: The end-to-end system includes fruit detection, counting, tracking across views, and merging counts from both sides of a tree row.Both-side registration is needed to avoid double counting when fruits are visible from both sides.

5 Datasets

The datasets were collected in a Minnesota university orchard across varied tree rows, apple varieties, growth stages, years, and imaging conditions, with separate training, validation, and test sets for detection, yield estimation, and counting.

  • Data collection: All data were collected at the University of Minnesota Horticultural Research Center between June 2015 and September 2016 using smartphone video of single tree-row sides.The orchard contained many apple tree species and datasets differed in year and growth stage.
  • Training sets: Detection training used 10 datasets from six tree rows and 103 annotated 1920 × 1080 images spanning varieties, growth stages, and tree shapes.A separate semi-supervised GMM dataset used 50 frames with user clicks on apples.
  • Validation sets: Validation patches used 80/20 training-validation splits for both U-Net and FRCNN, while GMM required no annotated validation data.These validation procedures were sampled from the training datasets.
  • Test sets: Detection and yield-estimation tests used seven videos from four orchard sections collected in 2016, with fruit boxes, per-tree yield, and post-harvest diameter measurements.Yield experiments used videos recorded from both sides of datasets 1, 2, and 3.
  • Counting datasets: Counting methods were evaluated on small patches annotated with a single ground-truth count from 0 to 6 by at least two human labelers.Disagreements were resolved through a third inspection.

6 Experiments and Results

The experiments quantitatively evaluate fruit detection and counting methods while also examining qualitative behavior and common failure cases.

  • Evaluation scope: The evaluation measures each presented fruit-detection and counting method quantitatively and analyzes qualitative insights and common failure cases.The section evaluates the methods rather than introducing a new experimental component.
  • Detection experiments: The GMM detector is evaluated both without user supervision using a separate semi-supervised dataset and with supervision from clicks on the first five frames of each test video.The semi-supervised dataset used a different year and camera.
  • U-Net experiments: U-Net training used approximately 59,000 annotated 224 × 224 patches extracted with a stride of 50 pixels.These patches were extracted from the original images.
  • Faster R-CNN experiments: Faster R-CNN was initially trained on an open-source dataset and then supplemented with the paper’s annotated data for a fair comparison.The initial dataset contained 1,120 images at 308 × 202-pixel resolution.

6.2 Detection Results

Detection performance varies by dataset and metric: user-supervised GMM generally leads, while U-Net remains competitive where color features are insufficient. The methods are evaluated using precision, recall, and F1 across seven datasets and IoU thresholds.

  • Evaluation metrics: Precision, recall, and F1-measure are computed per frame and averaged per dataset across IoU values from 0.01 to 0.99.Recall, precision, and F1 are defined from true positives, false positives, and false negatives.
  • Recall: The user-supervised GMM outperforms the other approaches on recall in 6 of 7 datasets and remains competitive on the seventh.Its recall drops on datasets whose test color space differs from the training model.
  • Recall: U-Net achieves consistently high recall and surpasses user-supervised GMM on Dataset4 (front), where color features alone are insufficient.Its use of non-color features supports detection under that condition.
  • Precision: User-supervised GMM leads precision by a large margin on 6 of 7 datasets and exceeds 90% on every dataset.Conservative user supervision avoids ambiguous color clusters.
  • Precision: U-Net does not exceed 80% precision on 4 of 7 datasets, partly because it detects yellowing leaves as apples absent from its training data.The approach has higher precision than the other methods on Dataset4 (front).
  • F1-measure: User-supervised GMM is expected to lead F1 on most datasets, but U-Net is competitive on Dataset3 (back) and outperforms GMM on Dataset4 (front).F1 combines precision and recall.
  • Runtime: The GMM runs at 5 frames per second, while U-Net processes a full 1920 × 1080 frame in less than 4.5 seconds.U-Net operates on 224 × 224 patches with zero overlap and takes less than 100 ms per image patch.

6.3 Counting Results

The ResNet50 counting method outperforms GMM across all test sets and generalizes better across varying illumination and fruit colors.

  • Counting performance: ResNet50 outperforms GMM on all test sets for image-patch apple-cluster counting.The experiments use adapted datasets, so their results are not directly comparable with the earlier study.
  • Error handling: The neural network rejects false-positive detections in 87% of cases, compared with 43% for GMM.GMM is precise for predicting a single apple but declines substantially for other count categories.
  • Dataset distribution: The seven-class assumption is broader than the observed distribution because most detected clusters contain between 0 and 4 apples.The test datasets are highly skewed toward single-apple clusters.

6.4 Qualitative Results

Qualitative examples expose dataset-specific failure modes: color-based detection misses apples when color is insufficient, while learned detectors can confuse yellowing leaves with apples.

  • Dataset-specific behavior: Dataset4 (front) shows that insufficient color separation causes problems for user-supervised GMM detection.Dataset1 (front) shows yellowing leaves causing problems for both U-Net and FRCNN.
  • Dataset-specific behavior: On Dataset3 (back), both GMM and U-Net achieve high precision and recall.

6.5 Yield Estimation Results

The yield-estimation system combines fruit detection, counting, tracking across both row sides, and count merging. Using GMM detection with ResNet50 counting produces the most consistent estimates against harvested ground truth.

  • System design: Yield estimation requires accurate detection, counting, multi-view tracking, and merging counts from both sides of an orchard row.The system addresses these subproblems as an end-to-end yield-mapping process.
  • Experimental setup: The experiments use GMM detection because U-Net and FRCNN did not show satisfactory detection rates, then compare GMM and ResNet50 counting.
  • Count merging: Merging counts from both row sides produces more consistent estimates than independently summing side-specific counts.This comparison is shown for both GMM and ResNet50 counting.
  • Yield accuracy: ResNet50 counting achieves 95.56%–97.83% yield accuracy, compared with 91.98%–94.81% for GMM counting.ResNet50 errors range from 2.17% to 4.44% relative to harvested ground truth.
  • Scope: The system counts only visible apples because the camera cannot see apples within the tree foliage.

6.6 Failure Cases

The evaluated detection methods share errors from grouping, false positives, and false negatives, while counting failures often arise from partial visibility and annotation inconsistencies. Additional counting-stage networks can reject many false positives, but false negatives remain challenging across methods.

  • 6.6.1 Detection: Detection errors commonly involve grouped instances, false positives, and false negatives across all three methods.
  • 6.6.1 Detection: Deep learning methods additionally split single objects into multiple detections.
  • 6.6.1 Detection: The U-Net and GMM counting-stage networks reject false positives in approximately 85% of cases.
  • 6.6.1 Detection: False negatives are more challenging because their causes differ across GMM, U-Net, and FRCNN detection methods.
  • 6.6.2 Counting: The counting method reaches 90.5% overall accuracy but fails when fruits are partially visible or annotations are inconsistent, especially under substantial overlap.

7 Conclusion and Future Work

The paper compares fruit detection and counting methods on shared datasets and evaluates them for yield estimation. Classical detection performs best across most detection datasets, while CNN-based counting is more accurate and, combined with classical segmentation, achieves high yield accuracy; limited training data constrains conclusions about deep-learning generalizability.

  • Conclusion: The study compares fruit detection and counting methods on the same datasets to address incompatible dataset-specific evaluations.
  • Conclusion: The GMM detection method achieves the highest F1-score in six of seven datasets, while U-Net performs reasonably and Faster R-CNN has poor precision.
  • Conclusion: CNN-based counting is more accurate for both single-image datasets and yield estimation.
  • Conclusion: 95.56%–97.83% yield accuracies result from combining classical segmentation with CNN-based counting against harvested ground truth.
  • Future Work: Limited training data prevents conclusive insight into the generalizability of the U-Net and Faster R-CNN approaches.
  • Future Work: Synthetic data could reduce fruit-boundary labeling costs, but models trained naively on synthetic data typically do not generalize to real data.
Loading 1810.09499v2…