IVG-UAV: An Intelligent Voice-Guided UAV System for Autonomous Ripe Fruit Harvesting with Vision-Based Classification and Adaptive Path Planning

I. INTRODUCTION

• Existing harvesting robots often rely on predefined algorithms and perception, limiting real-time human interaction and adaptability during fruit search and decisions.The proposed human-in-the-loop approach allows operators to guide trajectories and intervene in edge cases.Source: §I · p. 1
• IVG-UAV combines speech recognition, LLM command interpretation, YOLOv8 fruit detection with RGB-D sensing, and adaptive path planning in one harvesting pipeline.The introduction describes high-level voice commands becoming task constraints for target selection and planning.Source: §I · p. 1
• The UAV uses RRT* [2] to plan a shortest path to the next target tree while avoiding obstacles.At a tree, vision classifies ripe fruit and the system can detach it.Source: §I · p. 2

A. Vision-assisted Avocado Harvesting UAV

• A prior avocado-harvesting UAV combines computer vision and posture estimation with a dual-arm gripper-and-fixer mechanism for picking.The gripper separates the avocado while the fixer holds the stem in place.Source: A. Vision-assisted Avocado Harvesting… · p. 2

B. Deep learning–based computer vision with robotic manipulation

• A prior apple-picking system uses a lightweight detector and predicted gripping locations to guide robotic manipulation.Its image pipeline recognizes apples and estimates suitable grasping areas.Source: B. Deep learning–based computer vision… · p. 2
• The compared harvesting systems differ in architecture but both lack voice-guided interaction for dynamic user supervision.The text identifies this as a usability limitation in practical agricultural settings.Source: B. Deep learning–based computer vision… · p. 2

III. METHODOLOGY

• The proposed framework separates orchard exploration and mission execution, using the map from exploration to plan harvesting and detect ripe fruit.It combines voice-guided commands with existing harvesting technologies.Source: §III · p. 2

A. Exploration Phase

• During exploration, an RGB-D camera detects trees and contributes depth data to an OctoMap probabilistic voxel map of the orchard.The map is updated row by row; occupied voxel clusters provide each tree’s spatial position in the global frame.Source: A. Exploration Phase · pp. 2–3, Figure 3 · p. 2, Figure 2 · p. 2

B. Mission execution phase

• Mission execution has three components: command generation and task planning, navigation, and detection.The execution pipeline receives voice commands and sends the resulting actions through the UAV system.Source: B. Mission execution phase · p. 3, Figure 4 · p. 3

1) Command generation and task planning:

• Whisper transcribes operator speech, then Phi-3.3B [6] verifies and analyzes commands before Gemini 2.5 Flash decomposes the mission into executable actions.The layers handle ambiguous or invalid requests and produce actions such as takeoff, navigation, harvesting, and landing.Source: 1) Command generation and task planning · p. 3
• The dispatcher translates the planned task sequence into UAV control commands and coordinates navigation, perception, and harvesting.An example sequence visits and harvests two trees.Source: 1) Command generation and task planning · p. 3

2) Navigation module:

• RRT* explores the mapped orchard to generate an optimized collision-free trajectory from the UAV’s current position to a target tree.The environmental representation includes vacant space, tree trunks, branches, and canopy structures.Source: 2) Navigation module · p. 3
• Collision checking during trajectory generation ensures that the planned path meets safety constraints and avoids environmental obstacles.Source: 2) Navigation module · p. 3
• After a feasible trajectory is generated, the UAV follows its planned waypoints to reach the target tree.Source: 2) Navigation module · pp. 3–4

3) Detection module:

• Three predefined viewpoints around each target tree provide multi-perspective RGB images, improving visual coverage and reducing fruit occlusion under dense foliage.The UAV sequentially visits the viewpoints to acquire the images.Source: 3) Detection module · p. 4
• Fine-tuned YOLOv8 is trained on manually annotated ripe and unripe fruit images and performs real-time localization and classification from RGB photos.The model learns visual features including fruit color, texture, and shape.Source: 3) Detection module · p. 4
• Pooling detections from the three views increases fruit-counting accuracy and decreases duplicate detections before forwarding results to the harvesting controller.Source: 3) Detection module · p. 4

IV. ENVIRONMENT SETUP

• The simulation integrates Gazebo and PX4 SITL with an x500_depth RGB-D UAV, ROS2 bridging, OctoMap, RViz2, Whisper, Phi-3.3B, Gemini-2.5-Flash, YOLOv8, and MAVSDK.QGroundControl supports manual monitoring and map-building initialization during exploration.Source: §IV · p. 4
• The framework runs on a workstation with an AMD Ryzen 7 Pro 6850U processor and 32 GB RAM.The stated workload includes real-time mapping, LLM inference, navigation planning, and computer vision processing.Source: §IV · p. 4

V. EVALUATION

• Evaluation defines voice accuracy from correctly interpreted commands, fruit detection per target tree against ripe-fruit total, and navigation success as reaching the target without collision within an acceptable distance.The navigation criterion also requires reaching within an acceptable distance threshold from the target coordinate.Source: §V · p. 5
• Figure 7 summarizes ten trials: seven commands were converted into valid text instructions, while three failed—one from incorrect language detection and two from imprecise pronunciation.The overview covers execution flow and performance distribution across the trials.Source: Figure 7 · p. 5, §V · p. 5
• Six of seven successfully interpreted commands generated valid plans, and all six missions reached their target trees without collision before fruit detection from three viewpoints.One interpreted command produced an invalid plan during task verification.Source: §V · p. 5
• Across six successful navigation trials, ripe-fruit detection rates were 1/1, 3/7, 4/5, 2/5, 5/8, and 8/15, varying with canopy density, occlusion, and visibility.The reported conditions include dense foliage, partial fruit occlusion, and varying illumination.Source: §V · p. 5
• 100% successful navigation rate, 85.7% valid plan generation rate, 70% voice command interpretation success rate, and 63.1% average fruit detection rate were reported.Navigation was the highest-performing component; fruit detection was the lowest-performing component.Source: §V · pp. 5–6

• The framework demonstrates the possibility of voice-guided autonomous UAV harvesting.Source: §VI · p. 6
• The proposed operating process emphasizes the critical role of humans in the loop.Source: §VI · p. 6
• The authors state that several limitations of the framework must be addressed.Source: §VI · p. 6

• The detection module cannot fully deduplicate ripe fruits detected from multiple viewpoints, risking repeated detections of the same fruit during harvesting.Source: A. Fruit Detection and Deduplication · p. 6
• Heavily occluded fruit among branches and dense foliage is difficult for the detection framework to recognize.Source: A. Fruit Detection and Deduplication · p. 6
• These detection challenges reduce overall fruit detection accuracy under complex orchard conditions.Source: A. Fruit Detection and Deduplication · p. 6

B. Continuous Mission Execution and Replanning

• Long autonomous operations and dynamic mission replanning are not fully supported.Source: B. Continuous Mission Execution and… · p. 6

C. Dialogue-Based Interaction

• The system does not support dialogue-based interaction for interactive task clarification, contextual feedback, or real-time conversational mission updates.Source: C. Dialogue-Based Interaction · p. 6

VII. FUTURE WORK

• The authors frame future work as enhancements to the framework’s resilience, scalability, and practical usefulness.Source: §VII · p. 6

A. Robust Perception and Fruit Detection

• Future perception work will investigate object tracking for fruit deduplication and advanced methods for detecting partially occluded fruit in dense canopies.Source: A. Robust Perception and Fruit Detection · p. 6

B. Multi-Robot Collaboration

• The framework may be extended to multi-UAV task coordination and a pick-and-catch architecture pairing an aerial robot with a fruit-receiving ground robot.Proposed multi-UAV task allocation would distribute work by fruit location, workload, and mission priority.Source: B. Multi-Robot Collaboration · p. 6

C. Long-Term Autonomous Operation

• Long-term autonomous operation is a future aim, with automatic replanning, adaptive task scheduling, and fault-tolerant mission management proposed for dynamic environments.Source: C. Long-Term Autonomous Operation · p. 6

D. Human-Robot Interaction with LLM-Based Planning

• The authors plan to integrate dialogue-based interaction into LLM planning for task clarification, contextual feedback, and more natural human-robot interaction.Source: D. Human-Robot Interaction with… · p. 6