Source-linked AI summary
Evaluating the Single-Shot MultiBox Detector and YOLO Deep Learning Models for the Detection of Tomatoes in a Greenhouse
Sandro A. Magalhães, Luís Castro, Germano Moreira, Filipe N. Santos, mário Cunha, Jorge Dias, António P. Moreira
TL;DR
Greenhouse tomato harvesting requires detection across ripeness stages and difficult visual conditions, including occlusion and variable lighting. The paper evaluates SSD and YOLO models, finding SSD MobileNet v2 strongest overall while YOLOv4 Tiny offers faster inference.
Problem
Greenhouse tomato harvesting robots must detect fruit across ripeness stages and heterogeneous conditions including occlusion, variable colour, and lighting.
Method
The paper publicly provides a tomato dataset and evaluates SSD and YOLO models while tuning confidence thresholds to balance precision and recall.
Results
SSD MobileNet v2 was the best-performing model and inferred in 16 ms, while YOLOv4 Tiny was faster because quantisation reduced processing time.
Takeaways & Limitations
The public dataset and benchmark support further deployment of tomato-detection models in robots and perception systems.
Takeaways & Limitations
The models were not generic enough to detect all tomatoes reliably, and performance worsened from validation to test data, indicating a need for more varied data.
Abstract
from arXiv · showhide
The development of robotic solutions for agriculture requires advanced perception capabilities that can work reliably in any crop stage. For example, to automatise the tomato harvesting process in greenhouses, the visual perception system needs to detect the tomato in any life cycle stage (flower to the ripe tomato). The state-of-the-art for visual tomato detection focuses mainly on ripe tomato, which has a distinctive colour from the background. This paper contributes with an annotated visual dataset of green and reddish tomatoes. This kind of dataset is uncommon and not available for research purposes. This will enable further developments in edge artificial intelligence for in situ and in real-time visual tomato detection required for the development of harvesting robots. Considering this dataset, five deep learning models were selected, trained and benchmarked to detect green and reddish tomatoes grown in greenhouses. Considering our robotic platform specifications, only the Single-Shot MultiBox Detector (SSD) and YOLO architectures were considered. The results proved that the system can detect green and reddish tomatoes, even those occluded by leaves. SSD MobileNet v2 had the best performance when compared against SSD Inception v2, SSD ResNet 50, SSD ResNet 101 and YOLOv4 Tiny, reaching an F1-score of 66.15%, an mAP of 51.46% and an inference time of 16.44 ms with the NVIDIA Turing Architecture platform, an NVIDIA Tesla T4, with 12 GB. YOLOv4 Tiny also had impressive results, mainly concerning inferring times of about 5 ms.
1. Introduction
Greenhouse tomato harvesting requires vision systems that handle variable plant conditions and fruit ripeness, including green and reddish tomatoes. The paper benchmarks SSD and YOLO models for fast greenhouse detection on edge-oriented platforms.
- Motivation: Manual harvesting can account for up to 50% of greenhouse production costs, motivating automated harvesting solutions.Labour shortages and seasonal workforce demands further increase the need for automation.
- Motivation: Greenhouse harvesting robots must detect tomatoes amid occlusion, variable plant structures, colour, glare, and lighting conditions.Tomatoes may be harvested green at physiological maturity or later during reddish ripening stages.
- Prior approaches: Existing tomato-detection strategies include classical machine learning and deep-learning approaches based on annotated images.The paper situates SSD and YOLO among the deep-learning architectures considered for greenhouse vision.
- Study objective: The study benchmarks five pre-trained SSD and YOLO models for high-speed tomato detection on edge-oriented hardware.The evaluated models are SSD MobileNet v2, SSD Inception v2, SSD ResNet 50, SSD ResNet 101, and YOLOv4 Tiny.
2. State-of-the-art
Prior tomato-detection research spans classical image-processing, machine-learning, and deep-learning methods, with green and partially occluded fruit remaining challenging. SSD and YOLO provide one-step detection frameworks suited to fast inference, although SSD performance can weaken for small objects.
- Detection challenges: Green tomato detection is difficult because fruit colour resembles the background, and occlusion or overlap from plant structures complicates localization.Machine learning is described as more robust to occlusion and green-tomato detection than conventional methods.
- Classical and machine-learning methods: Prior studies use colour spaces, clustering, morphology, SVMs, and deep-learning detectors across different tomato ripeness levels.Reported methods include ripe, green, intermediate, and clustered-tomato detection in greenhouse settings.
- Deep-learning methods: YOLO models are common in recent deep-learning tomato detection, while CNN-based approaches offer high accuracy but can require longer inference times.The literature review links growing deep-learning interest to newer computers and edge devices such as TPUs.
- SSD architecture: SSD performs localization and classification in one step by mapping image pixels directly to bounding-box coordinates and class probabilities.Its architecture combines a backbone feature extractor with additional feature maps and default anchor boxes.
- SSD limitations: SSD can perform worse on small objects because small objects do not appear in all feature maps.The paper identifies higher-resolution inputs and stronger backbones such as ResNet as possible mitigations.
3. Materials and Methods
The study builds a greenhouse tomato dataset and training pipeline for evaluating deep-learning detectors under robotic harvesting constraints. Images are collected, annotated, split and augmented before TensorFlow-based model training and evaluation.
- Data acquisition: A new greenhouse dataset was collected because existing datasets were not representative of the intended tomato-detection classes.Images were recorded in a Portuguese greenhouse using the AgRob v16 mobile robot.
- Annotation: Images were manually annotated in Pascal VOC format, identifying tomato class, size and position.The dataset focused on green and reddish tomatoes while ignoring ripeness as a class distinction.
- Pre-processing: Input images were split into 300 × 300 px tiles with 20% overlap to meet model and TPU processing constraints.The original video frames were 1280 × 720 px, and the tile scheme was used because the selected models could not process full-sized images.
- Data augmentation: Random transformations such as rotation, translation, scaling, hue modification, saturation, blur and noise expanded the dataset to 23,021 images with 61,204 annotations.The transformations were applied with random factors to increase data variability.
- Dataset split: 18,417 training images and 4,604 validation images were created, while an external annotated set from another greenhouse row was reserved for testing.The training set contained 49,100 annotations and the validation set contained 12,104 annotations.
- Training and evaluation: Models were trained and evaluated in TensorFlow using recall, precision and F1-score, with detections accepted at an IOU threshold of 50%.The confidence threshold was selected by maximizing validation-set F1-score across thresholds from 0% to 100%.
4. Results and Discussion
The evaluation compares four SSD models and YOLOv4 Tiny using threshold tuning, independent test-set analysis, and scenario-specific images. SSD MobileNet v2 performed best overall, while YOLOv4 Tiny offered substantially faster inference and all models behaved similarly for overlapped tomatoes.
- Evaluation methodology: The evaluation used recall × precision curves, mAP, total recall, total precision, F1-score, and inference time.
- Evaluation methodology: The confidence threshold was selected to maximise F1-score by balancing precision and recall while limiting false positives and false negatives.The selected thresholds were then used to characterise each model for prediction.
- Limitations: Performance worsened substantially from validation to test data, indicating that the amount and variability of the dataset should be increased.
- Test-set analysis: Using all predictions produced substantial false positives, whereas threshold filtering increased precision; SSD MobileNet v2 maintained precision above 80% without filtering.SSD MobileNet v2 was balanced between precision and recall and did not approach 0% precision.
- Scenario-specific analysis: SSD MobileNet v2 detected tomatoes with less than 50% of their area occluded, while the other networks did not detect those occluded tomatoes.For darkened tomatoes, all models performed similarly, with SSD MobileNet v2 detecting one additional tomato.
- Scenario-specific analysis: All models performed similarly on overlapped tomato clusters, so they could be used equally in that situation.
- Overall comparison: SSD MobileNet v2 was the best-performing model and the fastest SSD model at 16 ms, whereas quantised YOLOv4 Tiny had faster inference than the other models.The authors note that quantisation reduced YOLOv4 Tiny’s processing time and that its comparison with non-quantised models warrants further analysis.
5. Conclusions
The study benchmarked four SSD models and YOLOv4 Tiny for greenhouse tomato detection, making the dataset publicly available and identifying SSD MobileNet v2 as the strongest overall model. YOLOv4 Tiny offered a faster alternative, while dataset representativeness and model-specific improvements remain future-work priorities.
- Model benchmarking: The benchmark used four pre-trained SSD models and YOLOv4 Tiny to evaluate greenhouse tomato detection.The SSD models came from TensorFlow, while YOLOv4 Tiny came from Darknet.
- Dataset availability: The dataset was made publicly available through the INESC TEC Research Data Repository.The paper identifies this dataset as an openly available research resource.
- Main findings: SSD MobileNet v2 was the best-generalised and best-performing model, with low false positives and fast computation.Its low false-positive rate was highlighted as a practical advantage for detection.
- Main findings: YOLOv4 Tiny processed an image in about 5 ms, making it an alternative when inference speed is especially important.Its speed was attributed to quantisation, which reduced processing time.
- Limitations and future work: SSD ResNet 101 was the worst-performing model, with an F1-score of 38.13 % and an inference time of 60ms.Its complexity was associated with overfitting when the dataset was insufficient or unrepresentative.
- Limitations and future work: Future work includes increasing dataset representativeness and size and adding regularisation to penalise complex models.The paper also identifies further work on contextual subclasses, greenhouse evaluation, and tomato-ripeness assessment.
Abbreviations
The abbreviation list defines the technical terms used throughout the paper, including learning, perception, sensing, and detection concepts.
- Learning terminology: The manuscript provides abbreviations for artificial intelligence, artificial neural networks, deep learning, machine learning, and related technical terms.The list includes AI, ANN, DP, and ML.
- Sensing and robotics: The list includes abbreviations for sensing and robotics technologies such as IMU, GPU, LiDAR, RGB-D, and ROS.These terms describe sensors, computing hardware, image modalities, and robotics software.
- Detection terminology: SSD is defined as Single-Shot MultiBox Detector, while mAP denotes mean Average Precision.The list also defines COCO, OID-related terminology, RGB, and other technical abbreviations.
Appendix A
Appendix A identifies where the evaluated models are stored in the TensorFlow and Darknet model databases.
- Model repositories: All SSD models are located in the TensorFlow Models database.The appendix provides the corresponding TensorFlow download path template.
- Model repositories: YOLOv4 Tiny is located in the Darknet database.The appendix provides the corresponding Darknet release download path.