In-field high throughput grapevine phenotyping with a consumer-grade depth camera

1. Introduction

• Manual or hand-held phenotyping is labor-intensive, requires skilled operators, and may be subjective, while grapevines need evaluation under heterogeneous field conditions.Vineyards cover large areas and contain many vines with differing phenotypes.Source: §1 · pp. 2–3
• The proposed vehicle-mounted RGB-D platform acquires visual and geometric 3D information to reconstruct grapevine canopies, estimate volume, and detect and count clusters.RGB-D augments conventional images with per-pixel distance information.Source: §1 · p. 3
• The methods combine 3D and color information for plant modeling and volume estimation with image-only deep learning to segment scenes and detect grape clusters.Source: §1 · p. 3
• The framework was tested in a commercial Swiss vineyard using an Intel RealSense R200 mounted on a Niko caterpillar.Source: §1 · pp. 3–4
• Despite low-quality sensor data, the approaches enable automatic, non-invasive field data collection and processing during vehicle operations using a cost-effective 3D sensor.The authors contrast this with time and cost problems associated with traditional methods.Source: §1 · p. 4

• 2D machine vision has targeted grape-bunch detection and berry size or number, but occlusion and missing 3D geometry limit it.Source: §2 · p. 5
• Field-scale grapevine 3D phenotyping has used a five-camera, RTK-GPS vehicle and multi-view stereo to reconstruct rows and extract bunch, berry, and diameter traits.Source: §2 · p. 6
• Vision systems are sensitive to changing outdoor light, whereas LiDAR is more reliable in such conditions but usually requires greater investment.Source: §2 · p. 6
• Despite improved robustness to illumination, Kinect applications require noise filtering and remain mostly limited to indoor, close-range monitoring (Rosell-Polo et al., 2015; Lachat et al., 2015; Zennaro et al., 2015).Source: §2 · pp. 7–8
• Intel's portable depth cameras use infrared stereoscopy and active texture projection, work from zero lux through broad daylight, and provide RGB, infrared, and depth data for potential affordable agricultural use.Source: §2 · p. 8
• The study presents the first in-field grapevine phenotyping application of this RGB-D sensor, mounted on an agricultural tractor.Source: §2 · p. 8
• The approach uses ground-based 3D information for canopy volume and RGB images to detect and count grape clusters.Source: §2 · p. 8
• Data acquisition and processing can occur automatically during vehicle operations, non-invasively and with low investment.Source: §2 · p. 8
• The study compares four pretrained CNNs for grape-bunch recognition: AlexNet, VGG16 and VGG19 (Simonyan and Zisserman, 2014), and GoogLeNet.Source: §2 · p. 9
• Using transfer learning, the four CNNs are tuned with fewer training-set samples to enable image-based grape-cluster detection.Source: §2 · p. 9

3. Materials and Methods

• The processing pipeline combines 3D row reconstruction, canopy segmentation, and computational-geometry volume estimation with deep-learning-based grape-cluster recognition from color images.Development and testing data came from a commercial vineyard growing the white wine-grape variety Rauschling.Source: §3 · p. 10

3.1. Acquisition platform

• The R200 camera was mounted on a Niko caterpillar vehicle, facing laterally for bunch detection and forward for volume estimation because lateral framing could not capture the whole canopy.The color stream was time-synchronized with the stereo pair, and rolling-shutter artifacts were negligible during the campaign.Source: §3.1 · p. 11

3.2. Canopy volume estimation

• Canopy-volume estimation proceeds through three stages: 3D grapevine-row reconstruction, grapevine segmentation, and plant-by-plant volume estimation.The volume-estimation camera configuration was forward-looking.Source: §3.2 · pp. 12–13

3.2.1. 3D grapevine row reconstruction

• The camera’s onboard depth was not used; an external device ran SGM stereo reconstruction with a disparity range of [8, 40] pixels, corresponding to approximately 1.0–5.5 m.Calibration aligned the IR-derived depth map with the color image and estimated the sensor’s position relative to the vehicle.Source: §3.2.1 · pp. 13–14
• Calibrated stereo disparity was converted into 3D coordinates, retaining only points with color information; statistical filtering and a vehicle-relative distance limit removed noise and non-row points.The selected points formed a color point cloud transformed from camera coordinates to the vehicle frame.Source: §3.2.1 · pp. 14–15
• Successive point clouds were stitched into a 3D map using vehicle-displacement estimates from visual odometry and a box-grid merging filter.The method computes the overlapping region’s bounding box and divides it into cells for merging.Source: §3.2.1 · p. 15
• In row reconstruction, points within grid cells are merged by averaging locations and colours, using a 0.01 m grid step as a compromise between accuracy and computational load.Points outside the overlapping region remain unmerged.Source: §3.2.1 · p. 15

3.2.2. Grapevine segmentation

• Canopy segmentation selects grid cells with more than 70% positive-GRVI points and average height below 75 cm, then uses K-means to identify individual plants.The method first separates canopy from trunks and other scene elements; the stated thresholds gave good segmentation for this dataset.Source: §3.2.2 · p. 17

3.2.3. Plant per plant volume estimation

• The 3D occupancy-grid method estimates each plant’s volume by summing foliage voxel volumes, fitting the reconstructed cloud best but risking underestimation when stereo reconstruction yields few points.Points in each voxel are combined by averaging their coordinates and RGB components.Source: §3.2.3 · p. 18
• Convex-hull, oriented-box and axis-aligned-box alternatives can overestimate volume because they fail to represent plant concavities or voids, or are affected by outliers and long branches.Manual comparisons assumed a constant 0.9 m plant width and measured depth and height with a tape; volume was their product.Source: §3.2.3 · pp. 18–19

3.3. Grape cluster recognition

• Four pretrained CNNs classify input image patches into five classes: bunch, pole, wood, leaves and background.The networks are tuned for this task by retaining earlier layers and rearranging the final fully connected layer; visually similar background and foreground regions can cause false positives or negatives.Source: §3.3 · pp. 19–21
• Each CNN’s final fully connected layer is replaced by five output neurons followed by softmax, with the new layer trained at 20 times the transferred-weight learning rate.This higher rate is used to speed learning in the new final layer.Source: §3.3 · pp. 21–22
• CNN parameters are updated with stochastic gradient descent with momentum, and training stops if validation loss fails to decrease for 10 consecutive epochs.Validation accuracy and loss are computed at the end of each epoch.Source: §3.3 · p. 22

3.3.1. Dataset generation

• The dataset contains 85 manually segmented 640×480 images in five classes, randomly split 70:30 into a training/validation pool and a held-out test set.The test images are not known to the network during training.Source: §3.3.1 · pp. 22–23
• An 80×80-pixel sliding window extracts image patches for bunch localization, and bicubic interpolation resizes them to each CNN’s required input dimensions.The window size reflects the stated working condition that bunches fit within windows no larger than 80×80 pixels.Source: §3.3.1 · p. 23
• Bicubic upscaling may introduce edge-rounding artifacts, but applying the same resizing to training and test patches lets the networks learn to handle those distortions.The paper states that classification will not be altered by the matched upscaling distortions.Source: §3.3.1 · pp. 23–24
• Patch labels use weighted pixel-class sums that emphasize bunch, pole, and wood; a patch is labeled bunch when bunch pixels exceed 20% of its 6,400 pixels.That threshold is more than 1,280 bunch pixels before upscaling.Source: §3.3.1 · pp. 24–25
• The resulting patch set contains 44,400 leaves, 5,295 bunch, 3,260 wood, 7,915 pole, and 81,687 background examples.Source: §3.3.1 · p. 25

4. Results and Discussion

• Field acquisition covered 54 surveyed plants in a Swiss commercial vineyard; the fifth plant was excluded as dead, and stereo frames and color images were captured at 5 Hz while driving at 1.5 m/s.The study considered the row on the vehicle’s right and included Rauschling and Merlot grapes.Source: §4 · pp. 25–26

4.1. Canopy volume estimation

• The reconstruction comprised about 1.3M points; segmentation used GRVI and 3D information, and k-means clustering used 54 clusters, with centroids initialized from plants’ approximately 0.90 m spacing.Source: §4.1 · pp. 26–27
• 3D-OG produced much smaller volumes at δ = 0.05 m because of disconnected structures and voids; increasing the grid size to 0.1 m brought estimates closer to CH and OBB.Source: §4.1 · p. 28
• Per-plant point clouds also enabled canopy-height mapping, with average canopy height of 1.25 m.Height was estimated from the vertical extent of each plant’s minimum bounding box.Source: §4.1 · pp. 28, 31, Figure 15 · p. 31
• AABB most closely approached manual canopy-volume measurements, with about 5% average discrepancy; CH, OBB, and AABB produced higher estimates.The comparison included volume estimates from multiple automatic methods and manual measurements.Source: §4.1 · p. 28, Figure 14 · p. 30, Table 1 · p. 30

4.2.1. Evaluation metrics

• Fine-tuned CNNs classified five manually labeled patch classes; evaluation used patch metrics for training and cluster-detection accuracy, precision, and recall on 25 test images.Test images were assessed using the known number of bunches rather than per-pixel segmentation.Source: §4.2 · p. 31, §4.2.1 · pp. 31–33
• Evaluation counts distinguish true detections, wrong detections, and missed grape clusters relative to the total number of clusters.The supplied passages define GC as total clusters, TGC as true detections, FGC as wrong detections, and NGC as missed detections.Source: §4.2.1 · p. 33, §4.2.2 · p. 33

4.2.2. Training of the CNNs

• AlexNet trained in 4 hours, while the other CNNs required more than about 20 hours to complete training.AlexNet was described as significantly less complex than the other compared CNNs; the reported training times depended on mini-batch sizes set to the GPU memory limit.Source: §4.2.2 · p. 34
• Balanced accuracy exceeded 94.13% for every CNN, while bunch-class precision was lower than recall, indicating that false positives could arise despite high detection rates.The authors note that further processing, such as morphological filtering, is needed to improve grape-cluster detection.Source: §4.2.2 · pp. 34, 36, Table 3 · p. 35

4.2.3. Test of the CNN

• For bunch patches, VGG16 and VGG19 assigned the highest probability (equal to 1); AlexNet and GoogLeNet could fail to label bunches, with background scores higher.Source: §4.2.3 · pp. 36, 38
• Bounding boxes included grape clusters but overestimated their extent; raising the 0.85 threshold could reduce overestimation and false positives while causing more missed detections.With the selected threshold, only a few small clusters were missed, which the authors attributed to low image resolution and quality.Source: §4.2.3 · p. 39
• 91.52% accuracy and 88.52% recall made VGG19 best on those metrics, while its 87.09% precision trailed GoogLeNet (97.91%) and VGG16 (98.0%).Source: Table 5 · p. 41, §4.2.3 · p. 41
• VGG16 and GoogLeNet were the only CNNs with false detections in tests on the whole dataset.Source: §4.2.3 · p. 41
• All tested CNNs could effectively count fruits despite the low quality of the input color images.Source: §4.2.3 · p. 41
• VGG16 achieved 98.0% precision versus GoogLeNet’s 97.91%; in particular, it significantly reduced false predictions to less than 2%, so grape-cluster recognitions effectively referred to actual fruits.Source: §4.2.3 · p. 41
• VGG16 produced fewer missed recognitions than GoogLeNet, suggesting greater sensitivity to grape-cluster presence.Source: §4.2.3 · p. 41
• AlexNet had balanced, nearly comparable precision and recall and may be preferable because it requires less processing time.Source: §4.2.3 · p. 42

• The proposed platform addresses canopy-volume estimation and grape-bunch detection using an Intel RealSense R200 depth camera mounted on an agricultural vehicle.Source: §5 · p. 42
• Canopy-volume modeling approaches used a 3D point cloud of a grapevine row and were compared with manual measurements.Source: §5 · p. 42
• The authors report that results vary with the adopted modeling approach, indicating a need for new standards and procedures.Source: §5 · p. 42
• Four deep-learning frameworks segmented RGB-D images into multiple classes to recognize grape bunches, and all correctly detected fruit despite low-quality input images.Source: §5 · p. 42
• 91.52% accuracy was the maximum reported for fruit detection, achieved by VGG19.Source: §5 · p. 42
• The RealSense-based framework is reported to measure important grapevine characteristics in the field automatically and non-invasively.Source: §5 · p. 42

• Higher-end depth cameras and dedicated hardware are proposed to reduce processing time toward real-time computation; the D435 is noted for improved resolution and outdoor performance.Source: Future work · pp. 43–44
• Volume estimation comprises row reconstruction, canopy segmentation, and per-plant volume estimation, and refining any stage may improve overall performance.Source: Future work · p. 43
• Visual-odometry drift can accumulate on long tasks; GPS/IMU and SLAM are proposed to improve localization and continuous estimates across row-end turns.Source: Future work · p. 43
• Fixed empirical thresholds were used for canopy segmentation; adaptive thresholds and ground classifiers are proposed as possible improvements.Source: Future work · pp. 43–44
• Concavities affect volume estimates: global point-cloud methods tend to estimate higher volumes, while local 3D-OG may fit the canopy better, though reconstruction voids can reflect occlusion or stereo failures.Source: Future work · p. 44
• Future CNN work would extend recognition to other fruit types and maturation stages, requiring expanded datasets and more image classes.Source: Future work · p. 44
• Future work proposes using 3D information to adjust the currently fixed 80×80-pixel patch size to plant volume and sensor distance.Source: Future work · pp. 44–45
• Aligned depth maps are proposed as a fourth input-patch channel, requiring the CNN architectures to be redefined and retrained.Source: Future work · p. 45