Source-linked AI summary
System Design and Control of an Apple Harvesting Robot
Kaixiang Zhang, Kyle Lammers, Pengyu Chu, Zhaojian Li, Renfu Lu
TL;DR
Apple harvesting faces labor and ergonomic pressures, motivating an integrated robotic system. The paper combines RGB-D deep-learning perception, a hybrid-actuated 3 DOF manipulator, vacuum detachment, and nonlinear motion control, with laboratory tests demonstrating the prototype's harvesting functions and sub-2 cm positioning error.
Problem
Apple harvesting requires extensive labor and exposes workers to repetitive motions, awkward postures, and ergonomic injury risks.
Method
The paper integrates Mask R-CNN RGB-D fruit perception, a hybrid pneumatic/motor 3 DOF manipulator, a vacuum end-effector, and nonlinear velocity-based control.
Results
60 picking tests demonstrated desired apple-position reaching with overall error less than 2 cm.
Takeaways & Limitations
The prototype met primary harvesting functionalities and provides a foundation for future advancements in automated apple harvesting.
Takeaways & Limitations
The current open-loop detachment cannot determine whether or when the target fruit has detached, motivating additional sensing and closed-loop detachment.
Abstract
from arXiv · showhide
There is a growing need for robotic apple harvesting due to decreasing availability and rising cost in labor. Towards the goal of developing a viable robotic system for apple harvesting, this paper presents synergistic mechatronic design and motion control of a robotic apple harvesting prototype, which lays a critical foundation for future advancements. Specifically, we develop a deep learning-based fruit detection and localization system using an RGB-D camera. A three degree-of-freedom manipulator is then designed with a hybrid pneumatic/motor actuation mechanism to achieve fast and dexterous movements. A vacuum-based end-effector is used for apple detaching. These three components are integrated into a robotic apple harvesting prototype with simplicity, compactness, and robustness. Moreover, a nonlinear velocity-based control scheme is developed for the manipulator to achieve accurate and agile motion control. Test experiments are conducted to demonstrate the performance of the developed apple harvesting robot.
I. INTRODUCTION
The paper addresses labor and injury pressures in apple harvesting by developing an integrated robotic prototype combining perception, compact manipulation, vacuum detachment, and motion control.
- More than 10 million U.S. seasonal worker hours annually support apple harvesting, representing about 15% of total production cost.
- Existing shake-and-catch systems detach apples efficiently but can bruise fruit through collisions, while fruit-by-fruit systems require coordinated perception, manipulation, and control.
- The paper presents an RGB-D camera, deep learning-based apple detection and localization, a compact 3 DOF manipulator, a vacuum end-effector, and motion control as an integrated prototype.
- The prototype's components are integrated through ROS, with an RGB-D camera providing depth information for computing three-dimensional apple positions.
- Mask R-CNN is trained on 1,243 orchard images covering Gala and Blondee apples, sunny and cloudy weather, and three times of day.
B. Manipulator Design
The manipulator uses two motor-driven revolute joints and one pneumatic prismatic joint to provide a compact workspace extension with fast, efficient actuation.
- Manipulator structure: The proposed manipulator combines two revolute joints forming a pan-and-tilt mechanism with one prismatic joint at its base.
- Manipulator structure: The simple, compact structure provides sufficient degrees of freedom for primary pick-and-place tasks and facilitates efficient motion control.
- Pan-and-tilt actuation: The revolute joints use NEMA 23 servo motors, with maximum velocity of 4,000 RPM and peak torque of 2 N·m.
- Prismatic actuation: The prismatic joint uses a pneumatic rodless air cylinder with a 0.61 m stroke, and the carriage traverses the full stroke in less than one second.
- End-effector integration: A hollow aluminum link measuring 0.71 m long and 0.04 m in diameter also functions as the vacuum tube for fruit grasping.
C. End-effector Design
The system uses a vacuum-based end-effector to grasp and detach apples while addressing common harvesting-end-effector challenges such as fruit damage, weak gripping, and high cycle times.
- End-effector Design: The design addresses common end-effector issues including clustered-fruit isolation, insufficient gripping strength, high cycle times, and damage from bulky components.These issues are identified as significant challenges in fruit-harvesting end-effector design.
- End-effector Design: The vacuum-based end-effector is used to grasp and detach tree fruits while minimizing bruising.It can also attract an apple within a certain distance when sufficient vacuum flow is provided.
- End-effector Design: Vacuum attraction tolerates approaching errors caused by disturbances such as wind, platform motion, and uneven terrain.This tolerance supports operation despite unpredictable in-field conditions affecting localization and robot motion.
A. Kinematic Model
The paper models the 3 DOF manipulator using a kinematical description and Denavit–Hartenberg convention, then exploits its simple structure to obtain analytical inverse kinematics.
- Kinematic Model: The 3 DOF manipulator is represented through a kinematical description of its joint and end-effector configuration.The model parameters are listed in Table I.
- Kinematic Model: The forward kinematics function characterizes end-effector position as a function of joint parameters using the Denavit–Hartenberg convention.The end-effector position is expressed in the manipulator base frame.
- Kinematic Model: Analytical inverse kinematics computes joint parameters from end-effector position for the manipulator’s simple structure.This avoids iterative optimization procedures that can be more time-consuming and induce numerical errors.
B. Controller Development
The controller transforms detected apple positions into the manipulator frame, generates smooth reference trajectories, and uses different control laws for motor-driven and pneumatic joints.
- Controller Development: Camera-frame apple locations are transformed into the manipulator coordinate frame using a calibrated camera-to-base transformation matrix.This provides the apple position used by the control scheme.
- Controller Development: The motor-driven revolute joints use velocity control, while the pneumatic prismatic joint uses position-based PI control because position feedback is unavailable.Inverse kinematics supplies the desired prismatic-joint parameter for PI regulation.
- Controller Development: Quintic functions generate continuously differentiable reference trajectories for the revolute-joint regulation problem.The trajectory coefficients are calculated from the specified constraints over the time interval [0, tf].
- Controller Development: The reference trajectory connects initial and final positions and allows tf to modify the velocity profile and arrival time.Its smoothness supports approaching the desired position along a smooth path.
- Controller Development: A Lyapunov-based velocity controller regulates the revolute joints so end-effector position along the yb-axis and zb-axis tracks the reference trajectory asymptotically.The controller uses angular velocity inputs for the revolute joints.
IV. PERFORMANCE EVALUATION
The evaluation reports experiments that first validate the motion-control scheme and then assess the integrated robotic apple-harvesting system in apple-picking scenarios.
- Performance Evaluation: Experiments validate the motion-control scheme and evaluate the integrated system in apple-picking scenarios.The evaluation is organized into motion-control validation followed by integrated-system testing.
A. Motion Control Validation
The experiments separately validate prismatic-joint position control and compare revolute-joint velocity-control approaches using repeated end-effector positioning tests. The proposed velocity controller achieves higher revolute-joint accuracy, while the prismatic joint reaches commanded positions within one second.
- Revolute-joint control: The revolute-joint comparison evaluates the proposed controller against open-loop velocity control and position control from a common home position.Desired joint values are calculated from desired end-effector positions using inverse kinematics.
- Revolute-joint control: The proposed velocity control scheme achieves higher accuracy than open-loop velocity and position control for all three revolute-joint test cases.Each method was tested five times per case, using average final-position distance errors measured with a QR code.
- Prismatic-joint control: The prismatic joint D is tested at desired positions of 0.1m, 0.2m, and 0.3m with the revolute joints held at zero.This isolates prismatic-joint position-control performance.
- Prismatic-joint control: Measured prismatic-joint positions are 0.111m, 0.208m, and 0.311m for the corresponding desired values.QR-code measurements provide the reported position results.
- Prismatic-joint control: The prismatic-joint position controller reaches the given desired values within one second in different tests.The paper reports that this satisfies the speed requirement for practical applications.
B. Apple Harvesting Validation
Laboratory tests evaluate the integrated harvesting system with apples randomly placed throughout the manipulator workspace. Across 60 trials, the system detects, approaches, detaches, and returns with apples, supporting agile workspace-wide harvesting.
- Integrated harvesting tests: The integrated system completes the full picking sequence: apple localization, manipulator approach, vacuum-based detachment, and return to the home position.Tests use artificial apple trees with apples placed at arbitrary workspace positions.
- Integrated harvesting tests: 60 picking experiments test apples randomly placed within the manipulator’s workspace.The vision system detects and localizes each apple before manipulation begins.
- Approach performance: The manipulator accurately approaches apples in diverse workspace regions by adjusting its inclined approach angle.The paper contrasts this agile adjustment with fixed-path regulation for specific areas.
- Timing and completion: Fruit detection and localization require 0.3 second on average per image, while manipulation approach takes approximately 2.0 seconds per apple.Fruit detaching uses an open-loop command set at 1.0 second.
- Timing and completion: The end-effector firmly holds the detached fruit while the manipulator returns home in all tests.This behavior is reported across the harvesting trials.
C. Discussion on Future Work
The prototype shows promising performance, but future work targets sensing, collision-free planning, and improved fruit detachment. These changes address picking efficiency, robustness, canopy protection, and detaching rate.
- Sensing: Additional sensing is needed to improve picking efficiency and system robustness.The current fruit-detaching command is open-loop and cannot determine whether or when the target fruit has detached.
- Sensing: A pressure sensor could provide feedback on fruit holding and separation after the predetermined detaching movement.This feedback would support closed-loop fruit detaching.
- Perception: The perception system should detect branches and other objects for more comprehensive environment perception.This extension is proposed alongside additional sensing to improve system robustness.
- Path planning: Path planning is needed for deep-canopy apples to avoid collisions with branches and other objects.The planned approach should reduce potential damage to tree branches and other objects in the manipulator’s path.
- End-effector: 80% detaching rate was achieved in 2018 and 2019 field tests, while future end-effector designs target at least 95%.Different vacuum cups and soft materials are proposed to conform to variable apple contours and sizes and build vacuum pressure faster.
V. CONCLUSION
The paper integrates perception, manipulation, and detachment into an apple-harvesting prototype with accurate and agile control. In 60 laboratory picking tests, the manipulator reached desired apple positions with overall error below 2 cm.
- Conclusion: The prototype integrates a vision-based perception system, 3 DOF manipulator, and vacuum-based end-effector for apple picking.Its control scheme targets accurate and agile manipulation motion.
- Conclusion: 60 picking tests showed overall manipulator positioning error below 2 cm, acceptable for vacuum-based fruit detachment.The tests were conducted in the laboratory.
- Conclusion: The prototype met the primary harvesting functionalities and provides a foundation for future advancements.Future work includes sensing fruit holding and detachment, path planning, and improved end-effector design.
APPENDIX A STABILITY ANALYSIS OF THE VELOCITY CONTROLLER
The appendix analyzes stability of the nonlinear velocity controller using a Lyapunov function. It concludes that the position errors along the y_b- and z_b-axes are asymptotically stable and converge to the reference trajectory.
- Theorem: The velocity controller ensures end-effector position along the y_b- and z_b-axes converges to the reference trajectory.The theorem concerns the position vector [y, z]^T.
- Proof: A Lyapunov function is defined to prove the controller’s stability.The proof uses error signals e_y and e_z associated with the controller analysis.
- Proof: Lyapunov’s stability theorem establishes that e_y = 0 and e_z = 0 are asymptotically stable.This stability result supports convergence of the position errors to the reference trajectory.