PaperlayerView exact revision on arXiv

Source-linked AI summary

Multispecies fruit flower detection using a refined semantic segmentation network

Philipe A. Dias, Amy Tabb, Henry Medeiros

arXiv:1809.10080v1cs.CVcs.RO

TL;DR

Bloom-intensity estimation still relies on visual inspection. This paper adapts and fine-tunes a segmentation network for flower identification, reporting accurate segmentation across apple, peach, and pear images acquired under varied conditions.

  • Problem

    Bloom-intensity estimation still relies on visual inspection.

    Source: §I · p. 1

  • Method

    The approach adapts a segmentation network for flower segmentation by retaining relevant classification branches and fine-tuning it on apple-tree images.

    Source: A. Network training · pp. 2–3

  • Results

    The apple-trained model achieves high flower-segmentation accuracy across varied datasets and generalizes to peach and pear flowers without dataset-specific fine-tuning or image preprocessing.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The current implementation evaluates high-resolution images of complete trees an order of magnitude faster than human workers, though it does not create block-level flower maps.

    Source: §VI · p. 8

    The current implementation does not create block-level flower maps.

    Source: §VI · p. 8

Abstract

from arXiv · show

In fruit production, critical crop management decisions are guided by bloom intensity, i.e., the number of flowers present in an orchard. Despite its importance, bloom intensity is still typically estimated by means of human visual inspection. Existing automated computer vision systems for flower identification are based on hand-engineered techniques that work only under specific conditions and with limited performance. This work proposes an automated technique for flower identification that is robust to uncontrolled environments and applicable to different flower species. Our method relies on an end-to-end residual convolutional neural network (CNN) that represents the state-of-the-art in semantic segmentation. To enhance its sensitivity to flowers, we fine-tune this network using a single dataset of apple flower images. Since CNNs tend to produce coarse segmentations, we employ a refinement method to better distinguish between individual flower instances. Without any pre-processing or dataset-specific training, experimental results on images of apple, peach and pear flowers, acquired under different conditions demonstrate the robustness and broad applicability of our method.

I. INTRODUCTION

Bloom-intensity estimation guides important orchard decisions, yet visual inspection and existing automated methods have practical limitations. The paper introduces an automated flower-segmentation approach intended to work across species and uncontrolled conditions.

I. INTRODUCTION

  • Bloom-intensity estimates guide pruning and thinning, which affect fruit load, size, coloration, and taste.The estimates can also benefit packing houses by supporting early crop-load estimation for postharvest handling and storage.Source: §I · p. 1
  • Visual inspection is labor-intensive and error-prone, relies on limited tree samples and grower experience, and misses orchard-level spatial variability.Flower appearance is short-lived until petal fall, adding urgency to automated estimation.Source: §I · p. 1
  • Existing hand-engineered vision methods often require controlled conditions and are vulnerable to species, illumination, and occlusion changes.The earlier CNN approach combined superpixel proposals with classification, but was limited by superpixel inaccuracies and network architecture.Source: §I · p. 1
  • The proposed method fine-tunes a fully convolutional network for pixel-wise flower segmentation and is designed for clutter, illumination changes, and multiple species.It evaluates high-resolution images covering a full tree in under 50 seconds; human workers may take up to 50 minutes to count flowers per tree.Source: §I · p. 1
  • The work also contributes an overlapping-window image-partitioning procedure for high-resolution FCN evaluation on commercial GPUs and releases pixel-accurately labeled flower images.The windows reduce artifacts from artificial boundaries when evaluating disjoint image regions.Source: §I · pp. 1–2

II. RELATED WORK

Prior flower-detection methods rely heavily on color or superpixels, while segmentation networks offer a more learned alternative but can produce coarse outputs. The paper situates its approach alongside these methods and refinement strategies.

II. RELATED WORK

  • Color-thresholding methods, and fail in uncontrolled environments and require parameter adjustment for changing illumination, camera position, or flowering density.Aerial multispectral approaches such as also rely solely on color information.Source: §II · p. 2
  • The earlier apple-flower detector used a fine-tuned CNN to classify superpixels, but local-context superpixels can be imprecise when flowers resemble their backgrounds.Other fruit-segmentation work includes Faster R-CNN detection and FCN segmentation followed by CNN-based counting.Source: §II · p. 2
  • Downsampling in classification networks such as AlexNet and VGG enables hierarchical features but reduces pixel-level precision in segmentation.Deconvolution layers, and encoder-decoder skip connections [26], [27] are among strategies proposed to mitigate this effect.Source: §II · p. 2
  • DeepLab combines ResNet-101, atrous convolutions, and spatial pyramid pooling to reduce downsampling, while dense CRF refinement requires supervised grid-search tuning.Region growing refinement (RGR) instead uses CNN score maps to seed high-confidence regions and classify uncertain pixels through appearance-based region growing [32].Source: §II · p. 2

III. OUR APPROACH

The approach adapts a pretrained semantic-segmentation network to flower segmentation, then processes images through patch evaluation and score-map refinement. Network training selects flower and leaf branches and uses multiscale apple-tree training images.

III. OUR APPROACH

  • The method starts from a DeepLab-ResNet model pretrained on COCO-Stuff and uses RGR rather than dataset-tuned dense CRF for fine-grained segmentation.COCO-Stuff includes pixel-level annotations for classes relevant to the application, including leaves, trees, and flowers.Source: A. Network training · p. 2

A. Network training

  • Network surgery retains weights and connections for classes of interest from the original model's 172 segmentation branches.The flower branch is paired with the leaf branch because nearby leaves are a major source of misclassification and the pretrained model strongly activates for leaves.Source: A. Network training · pp. 2–3
  • A two-branch flower-and-background model was chosen because the transferred flower scores lacked softmax normalization and could saturate the final sigmoid.The authors hypothesized that a background branch would help normalize foreground predictions.Source: A. Network training · p. 3
  • Fine-tuning used 100 apple-tree images for 10,000 iterations, with five training scales and mirrored augmentation for flower-containing images.Training labels used the quicker superpixel procedure; validation labels were pixel-accurate, and the images were split into 321 × 321 portraits.Source: A. Network training · p. 3
  • The image pipeline uses sliding windows, evaluates each patch with the fine-tuned CNN, and applies RGR to the resulting score maps.The figure presents the task sequence; its heatmaps encode lower scores in blue and higher scores in red.Source: A. Network training · p. 3, Figure 1 · p. 3

B. Segmentation pipeline

The segmentation pipeline divides a high-resolution image into patches, evaluates them with the fine-tuned CNN, and refines the resulting score maps into a final flower mask.

B. Segmentation pipeline

  • A sliding-window procedure partitions the high-resolution image into smaller patches before CNN evaluation.The patches are processed individually, their score maps are assembled, and refinement produces the final segmentation mask.Source: B. Segmentation pipeline · p. 3
  • The fine-tuned CNN evaluates each image patch to produce score maps for the subsequent refinement step.The pipeline applies refinement to those maps to compute the final segmentation mask.Source: B. Segmentation pipeline · p. 3
  • The pipeline's output is a refined flower-segmentation mask for the input image.Its three operations are patch partitioning, CNN evaluation, and score-map refinement.Source: B. Segmentation pipeline · p. 3

1) Step 1 - Sliding window:

High-resolution images are divided into overlapping 321 × 321-pixel portraits, avoiding the resampling artifacts of resizing. During fusion, overlapping areas are discarded to avoid artifacts from artificial boundaries.

1) Step 1 - Sliding window:

2) Step 2 - CNN prediction:

The CNN evaluates each portrait to produce pixel-dense foreground and background scoremaps. The figure contrasts direct thresholding with the refined segmentation.

2) Step 2 - CNN prediction:

  • For each portrait, the CNN produces pixel-dense foreground and background scoremaps representing flower and background likelihoods.The figure shows example scoremaps for both classes.Source: 2) Step 2 - CNN prediction · p. 4, Figure 3 · p. 4
  • Direct thresholding of the scoremaps yields a coarse segmentation, while RGR produces the refined segmentation.These outputs are shown in panels c and d, respectively.Source: 2) Step 2 - CNN prediction · p. 4

3) Step 3 - Fusion and refinement:

Portrait predictions are fused into image-sized background and foreground scoremaps, with overlapping padded areas discarded so each pixel receives one prediction. Softmax normalization and RGR then produce the refined segmentation.

3) Step 3 - Fusion and refinement:

  • Discarding padded areas during fusion gives each image pixel a prediction from exactly one portrait, avoiding artificial-boundary artifacts.The fused scoremaps have the same resolution as the input image.Source: 3) Step 3 - Fusion and refinement · p. 4
  • Softmax normalization makes the foreground and background scores at each pixel sum to one, yielding class probabilities.The two global scoremaps are normalized after fusion.Source: 3) Step 3 - Fusion and refinement · p. 4
  • Because direct CNN predictions are coarse at flower boundaries, the foreground scoremap and image are passed to RGR for refinement.RGR starts from high-confidence foreground and background regions, grows pixel clusters, then classifies each cluster by majority vote.Source: 3) Step 3 - Fusion and refinement · p. 4
  • RGR uses a flower-score voting threshold of 0.3, with background and foreground thresholds fixed at 0.1 and 1.25 × 0.3.The voting threshold was selected by grid search on the training dataset.Source: 3) Step 3 - Fusion and refinement · p. 4

IV. DATASETS

The four datasets cover apple, peach, and pear flowers collected under varied acquisition conditions. Apple images provide training and testing data, while peach and pear datasets offer different species and imaging conditions for evaluating generalization.

IV. DATASETS

  • The datasets comprise AppleA, AppleB, Peach, and Pear, with multiple flower species and capture conditions represented.The dataset specifications are summarized in Table I.Source: §IV · p. 4, Table I · p. 5
  • AppleA contains 147 hand-held images of apple trees; 100 were randomly selected for CNN training and 30 of the remaining images for testing.The apple datasets were collected in a USDA orchard on a sunny day.Source: §IV · pp. 4–5
  • AppleA flowers vary in size, clutter, and occlusion, and occupy an average of 2.5% of image area amid leaves.Mean flower area is 10,730 pixels, with a standard deviation of 17,150 pixels.Source: §IV · p. 5
  • AppleB images were acquired with a background unit that prevents trees in other rows from appearing.Peach and pear images were captured on an overcast day without a background unit, differing from AppleA in species and acquisition conditions.Source: §IV · p. 5
  • Peach flowers have a pink hue centered at 325°, while pear flowers are greener and differ in morphology.Peach and pear images have lower saturation and value means than AppleA images.Source: §IV · p. 5

A. Labeling

Pixel-accurate segmentation labels are laborious to produce, and polygon approximations may not follow object boundaries accurately. The procedure combines freehand traces with RGR refinement to create ground-truth segmentations.

A. Labeling

  • Pixel-level labels must adhere accurately to object boundaries, but polygon-based tools often produce imprecise ground truth.Inaccurate labels can compromise both supervised training and evaluation.Source: A. Labeling · p. 5
  • Figure 5 distinguishes positive annotations in blue from hard negatives in red, alongside the segmentation after RGR refinement.Source: Figure 5 · p. 5
  • Annotators draw freehand traces over flowers and mark hard negatives when needed; RGR uses those high-confidence points to segment the rest.Figure 5 shows the annotations and refined segmentation.Source: A. Labeling · pp. 5–6

V. EXPERIMENTS AND RESULTS

The experiments evaluate DEEPLAB+RGR across four datasets, training only on AppleA. The method outperforms the baselines across datasets, including unseen species and acquisition conditions.

V. EXPERIMENTS AND RESULTS

  • Experiments use AppleA for training and pixel-level precision, recall, F1, and IoU to compare DEEPLAB+RGR with SPPX+CLARIFAI and HSV-BASED.On the stated Xeon CPU and Quadro P6000 GPU, each high-resolution image takes an average of 50 seconds to evaluate.Source: §V · pp. 6–7
  • DEEPLAB+RGR outperforms baseline methods on every evaluated dataset, with especially strong generalization to unseen datasets.On validation AppleA, prediction and recall rates improve by more than 15%.Source: §V · p. 6
  • On AppleA, DEEPLAB+RGR improves prediction and recall rates by more than 15%.Figure 6 provides a qualitative example of detection accuracy.Source: §V · p. 6, Figure 6 · p. 6
  • On AppleB, DEEPLAB+RGR achieves F1 above 77% despite more flower buds and illumination changes, including sunlight reflected by leaves.Figure 7 shows an example from this dataset.Source: §V · p. 6, Figure 7 · p. 6
  • On Peach, DEEPLAB+RGR reaches F1 near 75% despite pink flowers and cloudy light; HSV-BASED fails on the color shift, while the new model exceeds 80% precision.False negatives often involve flower buds absent from training; poor superpixel segmentation leads SPPX+CLARIFAI to classify sky as flowers. Figure 8 compares detections.Source: §V · pp. 6–7, Figure 8 · p. 7
  • On Pear, DEEPLAB+RGR combines high recall with precision above 90%, despite slight morphology and color variation and clutter from branches.The branch texture compromises SPPX+CLARIFAI's background removal; Figure 9 shows an example.Source: §V · p. 7, Figure 9 · p. 7

VI. CONCLUSION

The method achieved high flower-segmentation accuracy across datasets with varied conditions, including transfer from apple training to peach and pear flowers. The implementation evaluates complete high-resolution tree images faster than human workers, while multi-species training and evaluation remain future work.

VI. CONCLUSION

  • High flower-segmentation accuracy held across datasets varying in illumination, background, resolution, density, and species; an apple-trained model generalized to peach and pear without fine-tuning or preprocessing.The peach and pear flowers differed noticeably in color and morphology.Source: §VI · p. 7
  • Future work aims to improve generalization by training and evaluating the model on multi-species flower datasets.Source: §VI · p. 7
  • The implementation evaluates complete high-resolution tree images an order of magnitude faster than human workers, but this work does not create block-level flower maps.The authors state the method will scale for applications such as predicting thinning spray treatments and timing.Source: §VI · p. 8
Loading 1809.10080v1…