Source-linked AI summary
Autonomous Sweet Pepper Harvesting for Protected Cropping Systems
Chris Lehnert, Andrew English, Chris McCool, Adam Tow, Tristan Perez
TL;DR
Manual harvesting is costly, while autonomous sweet pepper harvesting remains challenging under crop variability, occlusion, and changing conditions. The paper presents Harvey, combining vision-based detection and grasp selection with a novel end-effector for protected cropping. Field trials achieved 58% harvesting success for modified crops, with 90% detachment for a favourable cultivar.
Problem
Autonomous horticultural harvesting must address crop detection, grasp selection, and manipulation despite variable crop geometry, occlusions, and changing lighting.
Method
Harvey combines vision-based crop detection and 3D grasp selection with a custom end-effector using suction gripping and an oscillating blade.
Results
58% harvesting success, 81% grasping success, and 90% detachment success were achieved for a favourable cultivar after minor crop modifications.
Takeaways & Limitations
The results represent progress toward a commercially viable autonomous sweet pepper harvester, although further advancements are needed for a more general system.
Takeaways & Limitations
Detachment reliability is limited by assuming that sweet-pepper peduncles protrude vertically from the crop centre, an assumption that sometimes fails.
Abstract
from arXiv · showhide
In this letter, we present a new robotic harvester (Harvey) that can autonomously harvest sweet pepper in protected cropping environments. Our approach combines effective vision algorithms with a novel end-effector design to enable successful harvesting of sweet peppers. Initial field trials in protected cropping environments, with two cultivar, demonstrate the efficacy of this approach achieving a 46% success rate for unmodified crop, and 58% for modified crop. Furthermore, for the more favourable cultivar we were also able to detach 90% of sweet peppers, indicating that improvements in the grasping success rate would result in greatly improved harvesting performance.
I. INTRODUCTION
Horticultural harvesting faces substantial labour and environmental pressures, while autonomous harvesting remains difficult because it must integrate perception, planning, and manipulation under variable crop conditions. This paper presents Harvey, a protected-cropping sweet pepper harvester combining vision, grasp selection, and a novel end-effector.
- Motivation: 20% to 30% of total production costs in Australia in 2013-14 came from harvesting labour.Labour costs combine with skilled-labour scarcity and weather-related production volatility to pressure farm profit margins.
- Motivation: Autonomous harvesting could reduce labour costs while enabling more regular and selective harvesting, crop-quality optimisation, and improved scheduling.These potential benefits have motivated agricultural-robot harvesting research for three decades.
- Challenges: Autonomous harvesting requires integrating crop detection, motion planning, and dexterous manipulation despite changing lighting, crop variability, and occlusions.A survey of 50 horticultural robotic-harvesting projects reported that performance had not improved substantially over 30 years.
- Contribution: The proposed system addresses detection, 3D grasp selection, and manipulation using a vision-based algorithm, 3D localisation method, and novel end-effector.The work focuses on protected cropping environments with planar trellis structures to reduce motion-planning complexity and occlusions.
- Results: 58% harvesting success, 81% grasping success, and 90% detachment success were achieved for a favourable cultivar after minor leaf removal.The reported results are presented as significant improvement over the previous state of the art and progress toward commercial viability.
II. LITERATURE
Prior autonomous-harvesting research spans multiple crops, sensing approaches, motion-planning strategies, manipulators, and end-effectors. The literature shows persistent difficulties from occlusion, clutter, crop-specific localisation, and reliable grasping and cutting.
- Sweet Pepper Harvesting: The CROPS platform achieved 6% harvesting success for unmodified sweet pepper crops and 33% after occluding leaves and crop clusters were removed.These results highlight the difficulty and complexity of autonomous sweet pepper harvesting.
- Perception: Crop-perception pipelines generally include detection, segmentation, and 3D localisation before determining grasping or cutting poses.3D localisation may use ToF cameras, stereo vision, or a single-point laser range finder, but methods are generally ad hoc and crop-specific.
- Motion Planning: Open-loop planning is susceptible to unperceived environmental changes, whereas visual servoing requires high update rates but helps in dense vegetation with occlusions.The two approaches represent common motion-planning strategies for autonomous crop harvesting.
- Grasping: Reliable grasping in dense, cluttered environments remains an active research problem and often requires tactile sensing to distinguish rigid from deformable objects.Simplifying the workspace or harvesting operation can improve motion planning in cluttered horticultural environments.
- Manipulators: Autonomous-harvesting systems use configurations ranging from 3DOF Cartesian manipulators to anthropomorphic arms and 6DOF manipulators.Several studies compare joint configurations to optimise target reachability in cluttered environments.
- End-Effectors: Suction cups require access to one exposed crop face, while contact-based grippers are secure but more prone to interference from branches and other fruit.Crops such as sweet pepper and cucumber also require a separate detachment tool, such as a thermal cutter or scissor-like mechanism.
III. SYSTEM DESIGN
Harvey uses a mobile platform, 7DOF manipulator, RGB-D perception, suction attachment, and blade detachment to execute a five-stage autonomous sweet pepper harvesting cycle. Protected cropping rows simplify occlusion, collision avoidance, and motion planning.
- Harvesting Cycle: The harvesting cycle comprises scanning, crop detection, grasp selection, crop attachment, and crop detachment.Scanning builds a 3D model; detection segments and localises peppers; grasp selection computes candidate poses; suction attaches and an oscillating blade detaches the crop.
- System Architecture: Scanning, crop detection, and grasp selection form the perception system, while attachment and detachment use the harvesting tool.The paper separates perception from the procedures that attach and detach the crop.
- Operating Environment: Protected cropping uses two-dimensional planar growing surfaces that significantly reduce occlusion and the need for complex collision avoidance and motion planning.Translucent protective surfaces also diffuse incoming sunlight, creating favourable conditions for perception.
- Operating Environment: Crop rows are up to 2 m tall and spaced approximately 1 m apart, informing the harvesting platform’s workspace requirements.The row layout directly shaped the platform design.
- Platform: Harvey combines a custom differential-drive mobile base with a 6DoF revolute arm and prismatic lift joint.The platform is designed to manoeuvre independently between crop rows for up to 8 hours using an internal 3kWh lead-acid battery.
C. Harvesting Tool Design
The harvesting tool combines suction gripping with oscillating-blade cutting through a passive decoupling mechanism, while software coordinates perception, grasping, planning, and harvesting actions.
- The custom tool grips sweet peppers with a suction cup and cuts them free using an oscillating blade.
- Natural variation in pepper size, shape, and orientation makes simultaneous grasping and cutting challenging, motivating independently positioned operations.
- The passive mechanism magnetically couples the suction cup to the blade during attachment, then lets the blade move independently during cutting.
- The design uses no additional actuators and is intended to improve harvesting success.
- The system coordinates RGB-D sensing, detection, grasp processing, motion planning, robot control, and end-effector control through connected software nodes.
IV. PERCEPTION AND PLANNING
The perception and planning pipeline scans and models the scene, segments sweet peppers, estimates grasp poses, and uses those poses to execute harvesting.
- The perception system detects, segments, and estimates sweet-pepper poses before planning the picking action.
- RGB-D views are fused into a 3D model, peppers are segmented using colour information, and grasp poses are calculated for motion planning.
- The overview directs readers to earlier work for a more detailed description of the perception system.
A. Scanning and Sweet Pepper Detection
The system scans crop rows with an eye-in-hand RGB-D camera, builds and segments a 3D scene, then ranks candidate grasp and cutting poses from pepper geometry.
- Scanning: The RGB-D camera scans the crop row, and registered point clouds are combined into a single 3D scene model.The camera operates at approximately 30 frames per second, with 2 mm depth resolution and a 0.2 m to 1.5 m range.
- Scanning: Multiple views reduce point-cloud noise and leaf-occlusion effects because peppers are generally visible from at least one scanning view.
- Detection and segmentation: Colour-based segmentation separates red sweet peppers from leaves and stems despite variation in crop colour, illumination, and occlusion.
- Detection and segmentation: Euclidean clustering groups segmented points into individual peppers, but peppers touching each other limit this separation method.
- Grasp selection: Candidate grasp poses are generated from segmented 3D point clouds using either superellipsoid model fitting or direct surface-normal estimation.
- Grasp selection: Candidate poses are ranked by a weighted utility combining surface curvature, distance to the point-cloud boundary, and angle relative to the horizontal world axis.
- Grasp selection: The surface-normal method produces multiple grasp poses, unlike the model-fitting method’s single grasp pose.
- Grasp selection: Cutting-pose estimation modifies the utility function to favour vertical normals near the peduncle, while both strategies assume vertically oriented peppers.
C. Motion Planning
Harvesting trajectories are defined relative to grasping and cutting poses, using sequential attachment, separation, and detachment motions suited to the protected-cropping workspace.
- The planner computes harvesting trajectories relative to the selected grasping and cutting poses.
- The attachment motion approaches the grasp pose along the selected axis, followed by vertical movement that separates the suction cup from the cutting tool.
- The cutting trajectory keeps the end effector aligned with the horizontal world frame, which outperformed trajectories aligned with pepper orientation.
- The complete trajectory proceeds through attachment, separation, and detachment stages.
- The relatively planar protected-cropping workspace simplifies planning, allowing mostly in-and-out motions without complex obstacle avoidance around crop structures.
V. EXPERIMENT AND RESULTS
Two field trials evaluated Harvey across two sweet pepper cultivars in a protected cropping system, with platform and sensing changes introduced between trials.
- Platform changes: Trial 2 changed the scanning trajectory from a diamond pattern to a boustrophedon pattern.
- Platform changes: Trial 2 replaced the manual scissor-lift base with a custom mobile platform.
- Platform changes: A prismatic lift joint was integrated into the motion planner to improve workspace and reduce planning failures.
- Sensing changes: A vacuum sensor detected successful attachment, while a micro-switch detected decoupling of the suction cup and cutting blade.
A. Methodology
The field methodology positioned Harvey along crop rows, retried failed attempts, and recorded attachment, detachment, damage, occlusion, obstruction, and shape outcomes.
- Experiment procedure: 24 and 26 sweet peppers were included in the formal experiments for field trials 1 and 2, respectively.Each trial covered a single 10 m stretch of crop row.
- Experiment procedure: The platform was moved between pepper sets, manually in trial 1 and remotely driven in trial 2 to simulate autonomous base movement.
- Retry protocol: Failed attempts were retried, and leaves or pepper positions were modified when obstructions or occlusions caused repeated failures.
- Recorded outcomes: Each attempt recorded attachment and detachment success, plus damage, visual occlusion, physical obstruction, and irregular shape.
- Parameter selection: Scanning parameters were empirically determined for each protected cropping system and cultivar and remained fixed within each trial.
B. Results
Harvey achieved different harvesting outcomes across the two trials, with attachment, detachment, damage, and processing time providing complementary performance measures.
- Harvesting performance: 14/24 (58%) and 11/26 (42%) were successfully harvested in field trials 1 and 2, respectively.Successful harvest required both attachment and detachment.
- Harvesting performance: 22/24 (92%) and 11/26 (42%) sweet peppers were detached in field trials 1 and 2, respectively, irrespective of attachment success.
- Harvesting performance: Attachment rates were 14/24 (58%) in field trial 1 and 21/26 (81%) in field trial 2.
- Failure cases: Irregular pepper shapes produced poor grasp-pose estimates, causing missed peduncle cuts in one case and major pepper damage in another.
- Timing: 35-40 seconds was the average picking time per pepper, including scanning, model fitting, planning, and robot-arm execution.The component times were approximately 15 seconds, 5-10 seconds, and 10-15 seconds, respectively.
VI. DISCUSSION & CONCLUSION
The paper presents an autonomous sweet pepper harvester using a custom suction-and-blade end-effector and reports strong detachment in one trial, while identifying cultivar and peduncle geometry as remaining constraints.
- System contribution: Harvey uses a custom end-effector combining a suction gripper and oscillating blade to remove sweet peppers in protected cropping.
- System contribution: 92% detachment in field trial 1 is identified as promising because detachment is regarded as one of the most challenging harvesting steps.
- Failure analysis: 40% of field-trial-1 attachment failures came from leaves or string obstructions, while irregular shapes caused 30% of total attachment failures.
- Failure analysis: Field trial 2 had higher attachment rates after changing grasp selection and adding pressure sensing for premature separation.
- Remaining challenges: The cutting-point assumption that peduncles protrude vertically from the pepper centre sometimes fails, motivating future peduncle detection.
- Remaining challenges: The Redjet cultivar likely reduced trial-2 detachment because of shorter, thicker peduncles and more peppers inside the canopy.
- Conclusion: Further advancements are necessary for a more general and commercially viable autonomous harvesting system.