Source-linked AI summary
Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards
Hanwen Kang, Chao Chen
TL;DR
Dense, overlapping orchard fruits and branches make robotic harvesting difficult and require comprehensive vision of the working environment. DaSNet-V2 addresses this with one-stage multi-task fruit and branch sensing, lightweight computation, and RGB-D 3D visualisation. With LW-net, it reports 0.844 detection F 1 score, 0.858 fruit-segmentation mean intersection of union, and 0.795 branch-segmentation mean intersection of union.
Problem
Dense and overlapping fruits and branches, together with illumination variance and occlusion, challenge vision systems guiding robotic fruit detachment.
Method
DaSNet-V2 combines one-stage fruit detection and instance segmentation, branch semantic segmentation, a lightweight LW-net backbone, and RGB-D 3D visualisation.
Results
With LW-net, DaSNet-V2 achieves 0.844 on the F 1 score of detection, 0.858 on mean intersection of union for fruit instance segmentation, and 0.795 for branch semantic segmentation.
Takeaways & Limitations
The experiments validate DaSNet-V2’s robustness and efficiency for multi-task detection and segmentation in real apple-orchard environments.
Abstract
from arXiv · showhide
Robotic harvesting of fruits in orchards is a challenging task, since high density and overlapping of fruits and branches can heavily impact the success rate of robotic harvesting. Therefore, the vision system is demanded to provide comprehensive information of the working environment to guide the manipulator and gripping system to successful detach the target fruits. In this study, a deep learning based one-stage detector DaSNet-V2 is developed to perform the multi-task vision sensing in the working environment of apple orchards. DaSNet-V2 combines the detection and instance segmentation of fruits and semantic segmentation of branch into a single network architecture. Meanwhile, a light-weight backbone network LW-net is utilised in the DaSNet-V2 model to improve the computational efficiency of the model. In the experiment, DaSNet-V2 is tested and evaluated on the RGB-D images of the orchard. From the experiment results, DaSNet-V2 with lightweight backbone achieves 0.844, 0.858, and 0.795 on the F 1 score of the detection, and mean intersection of union on the instance segmentation of fruits and semantic segmentation of branches, respectively. To provide a direct-viewing of the working environment in orchards, the obtained sensing results are illustrated by 3D visualisation . The robustness and efficiency of the DaSNet-V2 in detection and segmentation are validated by the experiments in the real-environment of apple orchard.
1 Introduction
Apple-orchard harvesting requires vision that can interpret dense, overlapping fruits and branches under variable illumination and occlusion. DaSNet-V2 addresses this with multi-task orchard sensing and 3D visualisation for robotic harvesting.
- Fruit harvesting robots face complex orchard conditions, including densely arranged branches and fruits, illumination variance, and occlusion.
- The vision system senses the working environment and guides the manipulator in detaching target fruits.
- DaSNet-V2 performs fruit detection and instance segmentation together with branch semantic segmentation in a single vision-sensing framework.
- A lightweight residual backbone is used to support computational availability on embedded devices.
- The model is evaluated on apple-orchard data, with processed environments also represented through 3D visualisation.
2 Literature Review
Prior orchard vision research includes traditional machine-learning and newer deep-learning approaches. The supplied literature passage frames deep learning as offering stronger accuracy and generalisation, while Figure 1 presents DaSNet-V2’s multi-branch architecture.
- Earlier agricultural vision systems used extracted 2D or 3D image features with machine-learning classifiers for classification, detection, and segmentation.
- Deep-learning algorithms are described as more recent approaches than traditional machine-learning methods.
- The DaSNet-V2 architecture combines fruit detection and instance segmentation with semantic segmentation of branches.
3 Methodologies and Materials
DaSNet-V2 is a one-stage, multi-task architecture that combines fruit detection and instance segmentation with branch segmentation, using multi-scale feature processing and a lightweight backbone. RGB-D sensing and 3D visualisation extend the output toward orchard-scene understanding.
- Detection and Instance Segmentation: DaSNet-V2 uses a one-stage detector with a mask branch for fruit detection and instance segmentation.
- Detection and Instance Segmentation: A gated-FPN fuses feature maps from C3, C4, and C5 using channel-wise gating and addition.
- Detection and Instance Segmentation: ASPP encodes multi-scale object information, whose output predicts object class, bounding box, and mask.
- Semantic Segmentation: The semantic segmentation branch adds branch-scene information beyond fruit detection and segmentation.
- 3D Visualisation: The model visualises RGB-D orchard sensing as a 3D point cloud, with fruits assigned distinct colours and branches a unified colour.
- Training and Implementation: LW-net serves as a lightweight residual backbone intended for computational availability and real-time performance on embedded devices.
4 Experiment and Discussion
The experiments evaluate DaSNet-V2 across fruit detection, fruit instance segmentation, branch semantic segmentation, orchard robustness, computational efficiency, and 3D visualisation. Results show competitive or improved performance relative to comparison models, with lightweight and embedded-device deployment advantages.
- Evaluation methods: The evaluation covers detection, instance segmentation, and semantic segmentation using F1, IoU, and MIoU metrics.F1 combines detection precision and recall; IoU evaluates predicted box localization, while MIoU evaluates fruit and branch segmentation.
- Semantic segmentation: 0.802 semantic-segmentation score for DaSNet-V2 exceeds 0.772 for DaSNet-V1 and 0.733 for FCN-8s.The paper attributes the improvement over DaSNet-V1 to revised ASPP architecture and augmentation methods.
- Backbone comparison: 0.795 branch-segmentation performance with LW-net is similar to 0.797 with Darknet-53, while processing takes 30ms versus 45ms and 60ms with Darknet-53 and ResNet-101.ResNet-101 achieves the strongest branch-segmentation performance among the compared backbones.
- Real-orchard evaluation: DaSNet-V2 robustly segments overlapping and occluded fruits, presents smooth branch details, and has embedded-device efficiency advantages over two-stage detectors.The model was tested on Jetson-TX2; one-stage YOLO and DaSNet models are reported as more computationally efficient for embedded deployment.
- 3D visualisation: RGB-D sensing results are rendered as 3D orchard scenes showing or estimating fruit shape and orientation, tree structure, and stem-branch-joint location.The visualisation is intended to provide environmental information for fruit detachment guidance and working-environment reconstruction or modelling.
5 Conclusion and Future Work
DaSNet-V2 provides multi-task orchard vision sensing with accurate fruit and branch detection and segmentation, while LW-net improves computational availability for embedded devices. The resulting sensing outputs also support 3D visualisation and future robotic harvesting guidance.
- DaSNet-V2 performs fruit detection, fruit instance segmentation, and branch semantic segmentation in one multi-function detector.Its architecture includes separate fruit detection/instance-segmentation and branch semantic-segmentation branches.
- 0.844, 0.858, and 0.795 were achieved by DaSNet-V2 with LW-net on detection, fruit segmentation, and branch segmentation, respectively.
- 30ms on GTX-1080Ti and 265ms on Jetson-TX2 were recorded for DaSNet-V2 with LW-net.LW-net was developed to improve computational availability on embedded computational devices.
- The obtained orchard sensing results were used for 3D visualisation, with apples shown in colour masks, branches in original colour, and other elements in black.
- Future work will include guiding automatic robotic fruit harvesting.