PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion

Xu Liu, Steven W. Chen, Shreyas Aditya, Nivedha Sivakumar, Sandeep Dcunha, Chao Qu, Camillo J. Taylor, Jnaneshwar Das, Vijay Kumar

arXiv:1804.00307v2cs.CV

TL;DR

Counting fruit in video requires handling detection and tracking errors across image sequences. This paper combines FCN segmentation, frame-to-frame tracking, and SfM-based count correction, and reports improved count accuracy and consistency on orange and apple datasets.

  • Problem

    Counting fruit across video frames remains challenging because outliers can arise from tracking and localization errors, and detections must be tracked to avoid counting the same fruit more than once.

    Source: §I · p. 1

  • Method

    The pipeline segments fruit with an FCN, tracks detections using Hungarian assignment with Kalman-corrected KLT, and uses SfM locations and sizes to correct counts.

    Source: §I · p. 1

  • Results

    3D correction reduced orange L1 loss from 593 to 203 and apple L1 error from 673 to 322, with corrected error means of −0.2% and 3.3%, respectively.

    Source: §V · pp. 6–7

  • Takeaways & Limitations

    The evaluated pipeline counts visible fruit across image sequences in datasets featuring orange occlusion and depth variation and apple-foliage color similarity.

    Source: §VI · p. 8

Abstract

from arXiv · show

We present a novel fruit counting pipeline that combines deep segmentation, frame to frame tracking, and 3D localization to accurately count visible fruits across a sequence of images. Our pipeline works on image streams from a monocular camera, both in natural light, as well as with controlled illumination at night. We first train a Fully Convolutional Network (FCN) and segment video frame images into fruit and non-fruit pixels. We then track fruits across frames using the Hungarian Algorithm where the objective cost is determined from a Kalman Filter corrected Kanade-Lucas-Tomasi (KLT) Tracker. In order to correct the estimated count from tracking process, we combine tracking results with a Structure from Motion (SfM) algorithm to calculate relative 3D locations and size estimates to reject outliers and double counted fruit tracks. We evaluate our algorithm by comparing with ground-truth human-annotated visual counts. Our results demonstrate that our pipeline is able to accurately and reliably count fruits across image sequences, and the correction step can significantly improve the counting accuracy and robustness. Although discussed in the context of fruit counting, our work can extend to detection, tracking, and counting of a variety of other stationary features of interest such as leaf-spots, wilt, and blossom.

I. INTRODUCTION

Video fruit counting is difficult because detections must generalize across fruit types and environments, while tracking can still double-count fruit under occlusion or illumination changes. The paper combines deep segmentation, tracking, and 3D localization to correct counts, reporting improved accuracy and error consistency.

I. INTRODUCTION

  • The framework treats fruit counting in video as detection followed by tracking and counting, addressing a challenge that remains despite progress on static-image detection.Video counts can help growers plan labor allocation, storage, packaging, and harvest scheduling.Source: §I · p. 1
  • Fruit detection must generalize across varying illumination, tree shapes, and orchard arrangements, while frame-to-frame tracking must prevent over-counting the same fruit.Occlusion and illumination variation can still lead to double-counting in unstructured environments.Source: §I · p. 1
  • The pipeline trains an FCN on human-generated labels, tracks fruit with a Hungarian assignment using Kalman-corrected KLT costs, and uses SfM to localize fruit and reject count outliers.The SfM-derived size estimates are also useful for estimating yield and analyzing crop growth.Source: §I · p. 1
  • 3D localization and size-distribution estimates correct counts by rejecting false positives, significantly improving count accuracy and error standard deviation.The pipeline applies deep segmentation to monocular image sequences and evaluates counts against human annotations.Source: §I · pp. 1–2

II. RELATED WORK

Prior fruit-counting work spans single-image deep learning and video tracking approaches using combinations of SfM, assignment, optical flow, and Kalman filters. This work distinguishes itself by integrating deep segmentation with SfM in a monocular pipeline.

II. RELATED WORK

  • Single-image deep-learning methods detect, segment, or directly count fruit, whereas extending counting to image sequences adds the challenge of tracking fruit across frames.The cited approaches include Faster R-CNN fruit detection,,, FCN-based candidate regions with CNN counting, and synthetic-data training,.Source: §II · p. 2
  • Earlier video approaches combine SfM, Hungarian assignment, optical flow, or Kalman filters, with examples using 3D apple reconstruction or optical-flow associations,.A night-time stereo-camera approach uses controlled illumination and 3D locations to merge detections of the same apple.Source: §II · p. 2
  • Unlike prior tracking approaches based on traditional fruit segmentation, this work integrates deep segmentation with SfM to remove double-counted fruit using 3D locations and relative size distributions.The authors frame the resulting pipeline as monocular and robust across illumination conditions.Source: §II · p. 2

III. PROBLEM FORMULATION

The problem is to map image sequences of trees bearing one fruit type to estimates of their visible-fruit counts. The discrepancy between estimated and true counts is evaluated using L1 loss.

III. PROBLEM FORMULATION

  • The task maps each image sequence to an integer estimate of the visible-fruit count for that sequence.The sequences concern trees containing fruit of the same type.Source: §III · p. 2
  • L1 loss measures the discrepancy between an estimated count and the true visual count.Source: §III · p. 2

IV. PROPOSED APPROACH

The approach combines fruit-region segmentation, tracking between frames, and 3D reconstruction to estimate sequence counts and detect false positives. It uses RGB images from a monocular camera.

IV. PROPOSED APPROACH

  • The approach combines segmentation, inter-frame tracking, and 3D reconstruction to estimate the image-sequence count and detect false positives.Source: §IV · p. 2
  • The FCN detects most fruits in daytime-orange and night-time-apple images, including highly occluded fruit.Source: Figure 3 · p. 3
  • The pipeline limits its sensor input to monocular-camera RGB images, with off-the-shelf-camera imagery in low-resource settings identified as a potential future scope.Source: §IV · p. 3

A. Segmentation with Deep Learning

The paper uses an FCN to segment fruit pixels from non-fruit pixels, trained with human-generated labels. Its annotations use circles that can also generate label masks and fruit counts.

B. Tracking between frames

The tracking stage associates fruit candidates across frames using a Kalman-corrected KLT flow estimate and Hungarian assignment. It counts sufficiently persistent tracks, while 3D localization can expose outliers and duplicate tracks that are difficult to distinguish in 2D.

  • The tracker converts clustered fruit pixels into candidate objects through contour analysis and associates detections across frames to count a sequence rather than a single image.Source: B. Tracking between frames · pp. 3–4
  • Fruit candidates in adjacent frames are matched as a minimum-cost bipartite assignment solved by the Hungarian Algorithm.Source: B. Tracking between frames · p. 4
  • Noisy KLT flow can send stationary fruits in inconsistent directions as the camera moves; Kalman correction and fruit-velocity initial guesses address this problem.Source: B. Tracking between frames · p. 4
  • The Kalman state includes pixel row and column velocities, and its linear model assumes zero-mean Gaussian process and measurement noise.Source: B. Tracking between frames · p. 4
  • Matching cost uses corrected center distance relative to bounding-box size and favors overlapping boxes.Source: B. Tracking between frames · pp. 4–5
  • After 3D localization and rejection, fruits from a tree behind the target row are removed, while duplicate tracks are very close in 3D despite differing 2D projections.Source: Figure 5 · p. 5
  • Only tracks older than a threshold are counted, when tracking ends; the threshold is one third of the frames with overlap.Source: B. Tracking between frames · p. 5

C. 3D Fruit Localization and Relative Size Estimation

The pipeline uses SfM to localize fruit in 3D and estimate relative size, then uses depth and size outliers to refine the initial count. It focuses feature tracking near fruit regions to reduce computation and improve reconstruction quality, with a fallback when those regions provide too few features.

C. 3D Fruit Localization and Relative Size Estimation

  • SfM matches features across frames to estimate camera poses, using SIFT features near detected fruit regions to improve reconstruction quality and reduce computation.If too few nearby features are available, the algorithm allows SfM to use all visible features.Source: C. 3D Fruit Localization and Relative… · p. 5
  • Restricting tracked features to fruit-adjacent regions can leave too few features in sparse or smooth areas, making reconstruction unstable.The fallback is to let SfM use all visible features.Source: C. 3D Fruit Localization and Relative… · p. 5
  • The method estimates fruit centers by averaging the 3D locations of SIFT features associated with each fruit.Relative size is estimated by multiplying image area in pixels by the square of relative depth.Source: C. 3D Fruit Localization and Relative… · pp. 5–6
  • Fruit tracks can be deduplicated when their estimated 3D locations are close, addressing tracks lost and later reacquired across frames.Occlusion and illumination variation can cause the same fruit to be counted twice.Source: C. 3D Fruit Localization and Relative… · p. 6
  • Relative size outliers, including unusually large detections such as signs classified as fruit, can be rejected using the fruit-size distribution.The relative-size thresholds are shown in Fig. 9.Source: C. 3D Fruit Localization and Relative… · p. 6
  • Fruit in another tree row can be rejected by projecting estimated 3D locations into the camera frame and thresholding depth.The reconstruction view shows tree-canopy distribution consistent with fruit distribution, helping identify fruit from another row.Source: Figure 6 · p. 5, C. 3D Fruit Localization and Relative… · p. 6

V. RESULTS AND ANALYSIS

The pipeline was evaluated against human-annotated counts on orange and apple datasets with different capture conditions. 3D correction improved the reported count metrics, with larger gains on the less controlled orange dataset; it also reduced background-row overcounts in highlighted examples.

V. RESULTS AND ANALYSIS

  • Orange data were collected during the day without artificial lighting, while apple data were collected at night with an external flash in trellis rows.The datasets were selected to represent different illumination, camera stability, and fruit-occlusion conditions.Source: §V · p. 6
  • Orange L1 loss fell from 593 to 203, error mean from 17.1% to −0.2%, and standard deviation from 26.3% to 7.8% after 3D correction.Fig. 7 reports the error metrics; Fig. 8 shows corrected counts by tree.Source: §V · p. 6
  • In two orange-tree examples, correction reduced overcounts from 57 to 2 and from 101 to 9, against ground-truth counts of 206 and 78, respectively.The examples involved visible trees in another row that caused the uncorrected pipeline to overcount.Source: §V · pp. 6–7
  • Relative-size outliers were rejected below 0.5 or above 4, and depth outliers were thresholded at 1.5 times the mean fruit depth.The size thresholds were chosen heuristically.Source: §V · p. 7
  • For apples, L1 error fell from 673 to 322, error mean from 8.5% to 3.3%, and standard deviation from 4.7% to 4.1% after correction.The correction effect was more muted in the structured apple environment.Source: §V · p. 7, Figure 10 · p. 7
  • The correction had a larger effect on oranges, whose non-trellis setting involved more occlusion, depth variation, natural-light variation, and visible background rows than the apple setting.The apples were in a trellis with controlled night illumination and less variation in fruit distance and tree displacement.Source: §V · p. 7

VI. CONCLUSION AND FUTURE WORK

The pipeline counts visible fruit across image sequences, with results reported for apples and oranges and 3D correction improving orange counts more substantially. Future work targets absolute depth and size estimates and evaluation beyond apples and oranges.

VI. CONCLUSION AND FUTURE WORK

  • Evaluation covered orange sequences with high occlusion, depth variation, and uncontrolled illumination, and apple sequences with high color similarity between fruit and foliage.The paper presents segmentation, tracking, and localization as a pipeline for counting visible fruits across image sequences.Source: §VI · p. 8
  • Orange counts had an L1 error of 203, error mean of −0.2%, and standard deviation of 7.8%; apple counts had an L1 error of 322, error mean of 3.3%, and standard deviation of 4.1%.The 3D reconstruction correction improved orange counts more significantly than apple counts; the authors link this to natural illumination, high occlusion, and depth variation in the orange dataset.Source: §VI · p. 8
  • Future work proposes obtaining absolute depth and size, potentially by fusing IMU measurements or using average physical fruit size to convert monocular relative estimates.The average-size approach is proposed for settings where an IMU is unavailable.Source: §VI · p. 8
  • Further evaluation on fruits and farm objects beyond apples and oranges is proposed to explore real-world applications in unstructured, potentially low-resource settings using monocular images.The authors identify performance on other fruits or farm objects as future work.Source: §VI · p. 8

Related papers

Loading 1804.00307v2…