Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards

Suchet Bargoti; James Underwood

Source-linked AI summary

Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards

Suchet Bargoti, James Underwood

arXiv:1610.08120v1Oct 25, 2016cs.ROcs.CVcs.LG

TL;DR

Manual orchard yield estimation is costly, sparse, and potentially inaccurate, motivating automated analysis of high-resolution UGV imagery. The paper combines learned fruit segmentation with metadata and post-processing for apple detection, counting, and yield estimation, achieving its best reported performance with CNN-based segmentation and Watershed Segmentation.

Problem
Manual orchard yield sampling is labour intensive, expensive, often destructive, and sparse, while natural illumination creates challenging image variation.
Method
The framework uses pixel-level ms-MLP and CNN fruit/non-fruit segmentation, optionally incorporates orchard metadata, and applies WS or CHT to detect and count individual fruits.
Results
The best segmentation F1-score was 0.791 with CNNs, while WS produced the best fruit detection and counting results and CNN-WS count estimates achieved r^2=0.826.
Takeaways & Limitations
CNN segmentation combined with WS provided the best reported performance for fruit detection, counting, and yield estimation on this orchard dataset.
Takeaways & Limitations
Detection algorithms undercount large fruit clusters, and yield estimates can be affected by background-row detections and missing fruit registration across frames.

Abstract

from arXiv · show

Ground vehicles equipped with monocular vision systems are a valuable source of high resolution image data for precision agriculture applications in orchards. This paper presents an image processing framework for fruit detection and counting using orchard image data. A general purpose image segmentation approach is used, including two feature learning algorithms; multi-scale Multi-Layered Perceptrons (MLP) and Convolutional Neural Networks (CNN). These networks were extended by including contextual information about how the image data was captured (metadata), which correlates with some of the appearance variations and/or class distributions observed in the data. The pixel-wise fruit segmentation output is processed using the Watershed Segmentation (WS) and Circular Hough Transform (CHT) algorithms to detect and count individual fruits. Experiments were conducted in a commercial apple orchard near Melbourne, Australia. The results show an improvement in fruit segmentation performance with the inclusion of metadata on the previously benchmarked MLP network. We extend this work with CNNs, bringing agrovision closer to the state-of-the-art in computer vision, where although metadata had negligible influence, the best pixel-wise F1-score of $0.791$ was achieved. The WS algorithm produced the best apple detection and counting results, with a detection F1-score of $0.858$. As a final step, image fruit counts were accumulated over multiple rows at the orchard and compared against the post-harvest fruit counts that were obtained from a grading and counting machine. The count estimates using CNN and WS resulted in the best performance for this dataset, with a squared correlation coefficient of $r^2=0.826$.

1 Introduction

The paper addresses costly, sparse, and potentially inaccurate manual orchard yield estimation by studying image segmentation and contextual metadata for automated fruit analysis.

Manual yield sampling is labour intensive, expensive, and often destructive, so measurements from a few crops are extrapolated across farms.
UGVs with standard colour cameras can capture detailed orchard data at large spatial and temporal scales for extracting crop and yield information.
Orchard image classification must handle variation from illumination, occlusions, clustering, viewpoints, tree types, and seasonal maturity.
Metadata such as camera trajectories, vehicle location, tree or fruit type, distance, sun position, and weather can correlate with appearance variation or class distributions.
The paper studies ms-MLP and CNN fruit segmentation, metadata utility, and the effects of accurate segmentation on fruit detection and yield estimation.
The study evaluates how image segmentation accuracy affects fruit detection and yield estimation.

2 Related Work

Prior agrovision methods are often fruit- and dataset-specific, while natural orchard illumination creates substantial classification challenges. This motivates adaptive feature learning and the integration of contextual metadata with modern segmentation methods.

Agrovision studies address fruit detection and yield estimation across many crops but are often data-specific and heuristic relative to general computer vision.
Orchard fruit analysis commonly transforms image regions into discriminative features, then classifies fruit regions or applies post-processing to densely segmented images.
Key-point extraction narrows the search space by selecting candidate regions before classifying them as fruit or non-fruit.
Image segmentation produces a fruit likelihood map that can be thresholded into a binary mask for identifying fruit-containing regions.
Natural illumination remains an important open problem because orchard images contain variable lighting, shadows, fruit shapes, sizes, and seasonal conditions.
Hand-engineered features often restrict methods to particular fruits and datasets, motivating general-purpose adaptive feature learning.
The paper compares a benchmarked metadata-extended ms-MLP with state-of-the-art CNNs and evaluates improved segmentation for fruit detection and yield estimation.

3 Image Segmentation

The paper performs pixel-level fruit/non-fruit segmentation with multi-scale MLP and CNN architectures, then extends both with orchard metadata. Dense probabilistic outputs support subsequent fruit detection and yield estimation.

Image Segmentation: Pixel-level networks classify orchard image data into fruit and non-fruit classes, producing dense probabilistic outputs for each image.Inference is performed over whole images, enabling binary fruit masks for later detection or yield estimation.
Multi-scale MLP: The ms-MLP uses contextual RGB patches sampled at multiple scales, propagates them through fully connected layers, and outputs pixel class probabilities.Its first-layer activations are concatenated across scales before later dense layers and softmax classification.
Convolutional Neural Networks: The CNN uses larger contextual patches with convolution, ReLU, pooling, and fully connected layers to estimate whether each pixel belongs to fruit.Shared smaller-scale filters allow larger or higher-resolution contextual patches while reducing model parameters.
Adding Metadata: Metadata are appended to network layers to incorporate contextual information such as pixel position, sun position, and tree-related variables.For the ms-MLP, experiments found best performance when raw metadata were merged at the input layer; for the CNN, metadata were added to a fully connected layer.
Inference: The trained models are converted into fully convolutional operations for computationally efficient dense prediction over test images.For the ms-MLP, learned first-layer weights become patch-wise kernels applied across multiple image scales, whose responses are up-sampled and concatenated.

4 Experimental Setup

Experiments used orchard imagery collected by a ground vehicle in a commercial apple block containing multiple varieties and substantial appearance variation. A manually sampled and annotated subset supported architecture training and metadata experiments.

Experimental Setup: The study was conducted in a 0.5 ha Victorian apple orchard with a V-trellis structure and Kanzi and Pink Lady varieties.The scanned block contained fruit ranging from bright red to mixed-color appearances.
Data Collection: The Shrimp ground vehicle carried a six-camera spherical video system with six 2MP cameras providing a 360° panoramic view.Images used for the dataset came from the camera facing the trellis structure.
Data Collection: The vehicle was teleoperated between 15 rows at 1.5 to 2 m s^-1, capturing 1232 × 1616 images at 5 Hz from 2 m away.Rows were planted 4 m apart, and the camera field of view captured trees approximately 4 m tall.
Data Collection: Manual control was chosen because it produced imagery more similar to a manually driven tractor, whereas autonomous row following was expected to yield more consistent imagery.The latter was described as likely to improve image classification performance in subsequent datasets.
Image Dataset: More than 8000 orchard images were collected, while 1100 manually annotated images formed the labelled fruit/non-fruit dataset.Images were divided into 32 sub-images, and random row-based sampling was used to increase data diversity.

5 Segmentation Results

The experiments compare ms-MLP and CNN architectures for pixel-wise fruit segmentation, including metadata, model complexity, training size, and whole-image inference. CNN achieved the strongest segmentation metrics, while metadata substantially benefited ms-MLP but had minimal effect on CNN.

5.1 Multi-scale Multi-Layered Perceptron: The original ms-MLP baseline used 200,000 training instances and a 2-layer network with 200 hidden units.The baseline is denoted ms-MLP-2.
5.1 Multi-scale Multi-Layered Perceptron: All metadata increased the original ms-MLP F1-score by 6.2%, from 0.683 to 0.725.Image height within the original image was the most important individual metadata parameter.
5.1.1 Optimal ms-MLP Architecture: The optimized 3-layer ms-MLP improved F1-score from 0.683 to 0.728, while all metadata added +0.023 to reach 0.751.The optimized network used 200 hidden units per layer, with metadata added alongside the input multi-scale image data.
5.1.2 Varying Training Size: Classification F1-score increased with training size and reached convergence around 500,000 training instances.Dense sampling was unnecessary because neighboring training instances contained substantially overlapping contextual information.
5.2 Convolutional Neural Network: Without metadata, CNN* increased F1-score from 0.728 for ms-MLP-3* to 0.791, whereas metadata added only +0.006.Metadata was placed alongside the first fully connected layer in CNN* experiments.
5.3 Whole Image Segmentation: Whole-image inference took 0.60 seconds/image for ms-MLP and 0.24 seconds/image for CNN on 308 × 202 test images.CNN outputs showed higher precision and recall and smoother segmentation, while metadata sometimes increased ms-MLP false positives.

6 Fruit Detection and Yield Prediction

The framework converts pixel-wise fruit segmentation into individual fruit detections and row-level yield estimates, evaluating Watershed Segmentation (WS) and Circular Hough Transform (CHT) across learned segmentation architectures. WS generally produced stronger counting and yield-estimation results, while calibration and occlusion assumptions were required for row-level estimates.

Fruit Detection: Fruit detection must separate clustered fruits and reconnect fragments caused by occlusion, rather than simply selecting disjoint segmented regions.The paper applies post-processing to segmentation outputs because orchard fruits commonly overlap and appear partially occluded.
Fruit Detection: Watershed Segmentation identifies individual fruits from distance-based local maxima and contours, but cannot merge fragments belonging to one fruit.Morphological erosion and dilation are applied before detection to enforce local consistency.
Fruit Detection: Individual-fruit evaluation used manually annotated circular markers, one-to-one greedy nearest-neighbour matching, and true-positive, false-positive, and false-negative assignments.Boundary-near detections were ignored during evaluation.
Fruit Detection: 0.858 detection F1-score was achieved by the CNN* architectures with both WS and CHT comparable and best performing; WS outperformed CHT for ms-MLP architectures.The authors associate CHT suitability with the CNN's smoother segmentation regions and rounder boundaries.
Yield Estimation: Yield estimation assumes a constant average ratio between visible and occluded fruits, using calibration counts and weights collected separately for 15 orchard rows.The rows contained between 3,000 and 12,000 apples, while routine per-row counting is normally too labour intensive for commercial orchards.
Yield Estimation: WS produced more accurate yield estimates than CHT across segmentation architectures, including r^2=0.753 for ms-MLP-3* without metadata versus the CHT baseline of 0.635.Calibrated linear models related algorithm counts to true row counts; metadata reduced ms-MLP yield-estimation error from 13.3% to 12.2%.
Yield Mapping: Geo-referenced image counts can be interpolated over a fixed grid to produce a yield map showing spatially variable low- and high-yield areas.The map extends row-level estimates into a spatial representation of the orchard block.

7 Discussion

The discussion examines how segmentation architecture, metadata, and downstream detection affect orchard fruit counting and yield estimation, while identifying practical error sources and deployment constraints.

Segmentation performance: 0.791 was the highest fruit segmentation F1-score, achieved with the CNN, while metadata raised ms-MLP performance to 0.751.CNNs produced smoother segmentation but required considerably longer training than ms-MLP networks.
Metadata effects: Metadata gains converged as model complexity or training data increased, and could degrade CNN performance without careful configuration.The paper recommends testing metadata on a held-out validation set before inclusion.
Detection and counting: Watershed Segmentation outperformed Circular Hough Transform for fruit detection because segmented, occluded apples often were not circular.WS also relied on one hyperparameter, compared with six for CHT.
Image Processing Errors: Poor illumination, occlusion, and inconsistent human labels limited segmentation and detection evaluation.Ground-truth discrepancies included missed fruits and inconsistent labelling of background fruits in adjacent rows.
Image Processing Errors: Detection errors arose from poor segmentation, undercounted clusters, double-counted disjoint regions, and CHT failures on occluded non-circular fruit.Large clusters were difficult to split, although they were rare in this orchard because of thinning operations.
Yield estimation: Yield estimates were skewed by erroneous detections, background-row fruit, duplicate or missed detections, and absent fruit registration.Depth-based background removal was identified as necessary to minimise errors from sparse foliage.
Lessons Learned: Natural illumination creates extensive appearance variation, increasing reliance on training examples and complex classifiers.The paper suggests overcast, overhead-sun, or controlled night-time capture to reduce variation, while noting daytime operations are more desirable for farm integration.
Lessons Learned: Pixel-wise labelling took approximately 1.5 hours per 100 images and was difficult to perform consistently under adverse illumination.The authors prioritise methods that reduce annotation cost and improve labelling consistency.

8 Conclusion

The paper presents a feature-learning pipeline for fruit segmentation, detection, counting, and yield estimation from UGV imagery. CNN segmentation combined with WS produced the strongest reported detection and yield-estimation results on the evaluated apple-orchard dataset.

Conclusion: The framework combines ms-MLP and CNN segmentation with metadata, followed by WS or CHT post-processing for individual fruit detection and counting.The pipeline was evaluated on a 0.5 ha apple orchard block using a monocular camera mounted on a UGV.
Conclusion: 0.791 was the best segmentation F1-score, achieved with CNNs, while metadata had negligible impact on CNN performance.Metadata improved performance with the ms-MLP network.
Conclusion: 0.858 was the best fruit detection F1-score, achieved using WS with CNN segmentation results.The improved CNN segmentation performance translated to more accurate fruit detection with both detection algorithms.
Conclusion: r2 = 0.826 was the best yield-estimation fit, obtained by accumulating row-level fruit counts with the CNN and WS configuration.These counts were compared against post-harvest counts.
Conclusion: Future work targets generalisation across fruits and orchards, improved labelling strategies, and transfer learning between datasets.The paper frames these directions as relevant to understanding generalisation and commercial realisation.