PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Fruit Detection in Orchards

Suchet Bargoti, James Underwood

arXiv:1610.03677v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Reliable fruit localisation is needed for orchard automation tasks such as yield mapping and robotic harvesting, but orchard images are large, variable, and densely populated with fruit. The paper adapts Faster R-CNN, evaluates data and transfer-learning choices across apples, mangoes, and almonds, and introduces tiling for high-resolution images. Data augmentation reduced the labelled-image requirement by over 50%, while cross-orchard transfer offered little advantage over ImageNet initialisation; apples and mangoes achieved F1-scores above 0.9.

  • Problem

    Fruit detection must support yield mapping and robotic harvesting despite large, variable orchard images containing many small fruit.

  • Method

    The paper applies Faster R-CNN to apples, mangoes, and almonds, using ablations on training data, augmentation, and transfer learning plus tiling for large images.

  • Results

    Data augmentation reduced the required number of training images by over 50%, while cross-orchard transfer offered little advantage over ImageNet initialisation; apples and mangoes achieved F1-score >0.9.

  • Takeaways & Limitations

    Tiled Faster R-CNN provides a practical approach for fruit counting in large orchard images, with augmentation reducing labelling requirements.

Abstract

from arXiv · show

An accurate and reliable image based fruit detection system is critical for supporting higher level agriculture tasks such as yield mapping and robotic harvesting. This paper presents the use of a state-of-the-art object detection framework, Faster R-CNN, in the context of fruit detection in orchards, including mangoes, almonds and apples. Ablation studies are presented to better understand the practical deployment of the detection network, including how much training data is required to capture variability in the dataset. Data augmentation techniques are shown to yield significant performance gains, resulting in a greater than two-fold reduction in the number of training images required. In contrast, transferring knowledge between orchards contributed to negligible performance gain over initialising the Deep Convolutional Neural Network directly from ImageNet features. Finally, to operate over orchard data containing between 100-1000 fruit per image, a tiling approach is introduced for the Faster R-CNN framework. The study has resulted in the best yet detection performance for these orchards relative to previous works, with an F1-score of >0.9 achieved for apples and mangoes.

I. INTRODUCTION

The paper applies Faster R-CNN to fruit detection in challenging outdoor orchard imagery, targeting practical deployment across apples, mangoes, and almonds. It studies training-data requirements, augmentation, transfer learning, and high-object-count image processing.

  • Motivation: Fruit detection supports yield estimation, yield mapping, and robotic harvesting by localising individual fruit in orchard images.Accurate localisation can facilitate resource utilisation and reduce reliance on labour-intensive orchard tasks.
  • Challenges: Outdoor orchard imagery combines whole-tree coverage, small fruit, illumination and viewpoint variation, clustering, high fruit counts, and hardware constraints.Almond images contain 18 MP and up to 1500 fruit per image, making labelling difficult and computation demanding.
  • Approach: The study evaluates Faster R-CNN for fruit detection across three substantially different orchard types: apples, mangoes, and almonds.The paper provides implementation details, design rationales, and ablation studies for practical deployment.
  • Ablation studies: Ablation studies examine training-data requirements, data augmentation, and transfer learning between orchards to reduce labelling effort.The contributions explicitly target strategies for minimising required labelled data.
  • Deployment and resources: Image modification strategies are proposed for detecting more than 1000 objects in high-resolution orchard images.The paper also releases datasets and an annotation toolbox for rapid fruit labelling.

II. RELATED WORK

Earlier agrovision systems commonly used hand-engineered features, segmentation, or region-based classifiers tailored to particular fruit and capture conditions. This paper focuses on adapting Faster R-CNN to orchard-scale images with thousands of fruit and substantial variation.

  • Earlier agrovision methods: Prior fruit-detection methods transform image regions into features and classify fruit against foliage, branches, or ground, while segmentation provides pixel-wise classification.Post-processing can separate individual fruit from dense segmentation outputs.
  • Fruit-specific methods: Examples include colour and texture features for mangoes, CNN-based apple classification with Watershed Segmentation, and radial-symmetry features with KD-forest classification for berries.These approaches use different representations and post-processing steps for different fruit types.
  • Region-based detection: R-CNN methods combine region proposals with CNN classification and bounding-box regression, while Faster R-CNN unifies region proposal, classification, and localisation.Earlier R-CNN systems used Selective Search to propose candidate regions.
  • Orchard-scale setting: Orchard-scale data contains whole-tree images, thousands of fruit, and large size variation, creating hardware and annotation challenges for direct network input.The paper investigates adaptations specifically for these conditions.
  • Domain differences: Compared with greenhouse and web imagery, orchard images differ substantially because they involve greater distances, lower fruit counts per pixel, and high fruit counts per image.This difference motivates dedicated investigation of orchard data.

III. OBJECT DETECTION

The Faster R-CNN system proposes regions and then classifies and refines them through a shared deep network. Training uses annotated fruit boxes and end-to-end optimisation, while testing thresholds detections and suppresses overlaps.

  • Network components: Faster R-CNN contains a Region Proposal Network for detecting regions of interest and a classification module for classifying regions and regressing fruit bounding boxes.The two modules share convolutional features during end-to-end training.
  • Proposal refinement: The RPN produces class-agnostic object proposals, which the R-CNN component classifies and refines with finer bounding boxes.Shared convolutional layers support both components during end-to-end SGD training.
  • Testing: The network returns 300 bounding-box detections per image before probability thresholding and Non-Maximum Suppression.The threshold is class-specific, and NMS removes overlapping detections.

B. Transfer Learning

The paper asks whether orchard fruit detection should use ImageNet initialisation or knowledge transferred from another orchard dataset. It frames this choice around the potential mismatch between general image features and orchard imagery.

  • Standard transfer learning: CNNs are commonly pretrained on a large base network and fine-tuned for target tasks with fewer labelled examples.ImageNet provides 1000 object categories and 1.2 million images and is widely used for this purpose.
  • Initialisation concern: Faster R-CNN normally initialises its detection network with ImageNet weights, but performance can degrade when target classes differ drastically from base classes.Deeper CNN layers learn features that are more specific to the task.
  • Research question: The study compares ImageNet features with features fine-tuned on another orchard dataset for fruit detection.The comparison addresses whether orchard-specific pretraining is preferable for ground-vehicle imagery.

C. Data Augmentation

The paper uses label-preserving transformations to expand training variability for orchard fruit detection while addressing large, dense, heterogeneous imagery and annotation constraints.

  • C. Data Augmentation: Data augmentation expands training variability through label-preserving transformations such as flipping, rescaling, and colour changes.Random augmentation during each training epoch avoids pre-computing many augmented copies.
  • C. Data Augmentation: The datasets cover apples, almonds, and mangoes captured in Australian orchards under daylight conditions.Apple and mango images came from a ground vehicle, whereas almond imagery used a handheld DSLR camera.
  • C. Data Augmentation: Images contain about 100 fruit for apples and mangoes and over 1000 fruit for almonds, creating hardware and labelling challenges.A 0.25 MP image required approximately 2.5 Gb of GPU memory with VGG16.
  • C. Data Augmentation: Almond and mango annotations used rectangular boxes, while apple circles were converted to equal-width-and-height rectangular boxes for Faster R-CNN.Apples and mangoes were easier to label than almonds because of fruit size, contrast, and surrounding foliage complexity.
  • C. Data Augmentation: Dataset splits used different orchard-block regions to reduce biased results, and fruitless training images were discarded.Each fruit dataset was divided into training, validation, and testing sets.

V. FRUIT DETECTION RESULTS

The evaluation studies Faster R-CNN fruit detection across training configurations, transfer learning, augmentation, and deployment on large orchard images.

  • V. FRUIT DETECTION RESULTS: The ablation studies vary training-image count, transfer learning between orchards, and data augmentation, using ZF for faster training and VGG16 for deeper-network evaluation.A tiled Faster R-CNN method is also proposed for large raw images.
  • V. FRUIT DETECTION RESULTS: The orchard task is treated as a binary detection problem, with a separate model trained for each fruit type.Although Faster R-CNN supports multi-class detection, the experiments restrict each model to one fruit type.
  • V. FRUIT DETECTION RESULTS: Training sub-images were scaled to a 500-pixel shorter side to preserve the networks’ 16-pixel minimum object size.Unless otherwise stated, networks were initialized with ImageNet filters and trained until validation detection performance converged.
  • V. FRUIT DETECTION RESULTS: Performance is reported with average precision and final results with F1-score, using validation-selected class thresholds and NMS thresholds.True positives require sufficient bounding-box overlap, and one-to-one matching penalizes duplicate detections and missed fruit within clusters.

A. Number of Training Images

Detection performance rises rapidly with few training images, but convergence depends on fruit type; apples approach saturation earlier than almonds and mangoes.

  • A. Number of Training Images: 0.6 average precision is reached for apples with only 5 training images.Performance rises quickly when the number of training images is initially increased.
  • A. Number of Training Images: 0.01 average precision is added for apples in the final two-fold increase in training images.Apple performance is close to convergence at the amount of available labelled data.
  • A. Number of Training Images: More than 0.04 average precision is gained for both almonds and mangoes in the final two-fold increase in training data.Neither dataset had reached convergence.
  • A. Number of Training Images: Transfer learning from other orchards provides initial benefits that diminish quickly as apple training-image counts increase.With 5 training images, the reported difference from ImageNet initialization is 0.01.

C. Data Augmentation

The paper compares augmentation and transfer-learning strategies and reports final Faster R-CNN performance across three fruit types, including comparison with an earlier pixel-wise CNN approach.

  • C. Data Augmentation: Flip-scale augmentation provides the largest detection-performance boost, while PCA augmentation gives negligible improvement and can reduce performance at larger magnitudes.Flip and scale augmentations improve detection, indicating greater shape and scale variability than colour variation along principal components.
  • C. Data Augmentation: Transfer learning from almond and mango networks initially benefits apple detection, but the advantage quickly diminishes as training images increase.The comparison uses networks initialized from orchard-specific features versus direct ImageNet initialization.
  • D. VGG16 Network: 0.904, 0.908, and 0.775 are the best VGG16 F1-scores for apples, mangoes, and almonds, respectively.These results use all available training data with flip-scale augmentation.
  • D. VGG16 Network: Faster R-CNN outperforms the previous pixel-wise CNN approach for apples and mangoes.The paper attributes this to the deeper network and end-to-end detection and localisation without heuristic watershed post-processing.
  • D. VGG16 Network: VGG16 training takes 30–120 minutes per fruit class, while prediction averages 0.13 seconds per 500×500 image.The reported prediction time is faster than the pixel-wise CNN’s 2–3 seconds per image.

E. Tiled Faster R-CNN

Tiled Faster R-CNN applies detection across overlapping windows of large orchard images, while ablations examine data requirements, augmentation, transfer learning, and fruit-specific performance constraints.

  • Tiled Faster R-CNN: Tiling applies Faster R-CNN to large raw orchard images by fusing detections from smaller overlapping windows with thresholding and NMS.The overlap is kept greater than the maximum fruit size before proposals are fused on the large image.
  • Tiled Faster R-CNN: 54 true positive mango detections, 2 false negatives, and no false positives are shown for one tiled tree image.
  • Ablation findings: Data augmentation reduced the number of labelled training images required by over 50%, enabling convergence with less data.
  • Ablation findings: 726 apple images were sufficient to span dataset variability, whereas mango and almond datasets would require an order of magnitude more labelled data to reach asymptotic performance.
  • Ablation findings: Transfer learning between farms offered little advantage over ImageNet initialization, despite visual similarities between orchard imagery.
  • Ablation findings: Mango F1-score increased from 0.876 to 0.908 when changing from ZF to VGG16, trading higher accuracy against speed.Precision changed from 0.933 to 0.958 and recall from 0.825 to 0.863.
  • Ablation findings: Almond detection was constrained by low per-fruit resolution and foliage similarity, requiring > 30 MP tree images for mango- and apple-level pixel density.These conditions also made manual labelling and detection difficult.
  • Practical deployment: For smaller fruit, rescaling is recommended so minimum fruit size exceeds 16 pixels, although low resolution can still damage labelling and detection performance.

A. Error Cases

The reported error cases arise from clustered fruit, annotation inconsistencies, and limitations in distinguishing fruit from foliage under difficult imaging conditions.

  • Error Cases: Detection errors partly reflect missed fruit in clusters and ground-truth labelling errors that distort false-positive and false-negative evaluations.
  • Error Cases: Non-maximum suppression can suppress overlapping detections in clusters, producing images with high false-negative counts.
  • Error Cases: Relaxing one-to-one evaluation for clusters increased mango recall from 0.871 to 0.909 with VGG16.
  • Error Cases: Limited resolution, fruit–foliage similarity, and inconsistent object definitions make labelling tedious and error-prone.
  • Error Cases: Missing ground-truth annotations caused many false-positive instances, while consensus voting could reduce annotation error at high expense.

VII. CONCLUSION

The paper adapts Faster R-CNN for orchard fruit detection and evaluates deployment through ablations across apples, mangoes, and almonds. Augmentation reduces data needs, while orchard-to-orchard transfer adds little over ImageNet initialization; tiled inference supports large images, and F1-score exceeds 0.9 for mangoes and apples.

  • VII. CONCLUSION: The system uses Faster R-CNN for fruit detection in orchard images spanning apples, mangoes, and almonds.
  • VII. CONCLUSION: Ablations measure training-data requirements and transfer learning across orchards to assess practical deployment.
  • VII. CONCLUSION: Data augmentation achieved equivalent performance with less than half the number of training images.
  • VII. CONCLUSION: Transfering weights between orchards did not yield significant gains over initialization from ImageNet features.
  • VII. CONCLUSION: F1-score exceeded 0.9 for mangoes and apples, representing the best detection performance in the authors’ prior work.
  • VII. CONCLUSION: Tiled Faster R-CNN implements trained detection over large images required for orchard fruit counting and yield mapping.
Loading 1610.03677v2…