Source-linked AI summary
Remote Human and Robot Interaction for Greenhouse Gardening Using Virtual Reality
Daniel Udekwe, Hasan Seyyedhasani
TL;DR
This paper evaluates whether VR teleoperation can support remote greenhouse leaf inspection and soil moisture assessment, and whether performance depends more on operator experience or plant structure. Using a camera-equipped mobile manipulator across two experiments with 14 plants, it finds that canopy morphology—and the resulting camera occlusion—more strongly constrains soil-moisture reliability than the limited practice provided by repeated trials. The findings motivate adapting camera viewpoints and sensing strategies to canopy density.
Problem
The study asks how accurately and efficiently VR-teleoperated robots can inspect greenhouse plants, and whether canopy morphology or operator experience determines remote-sensing reliability.
Method
The paper evaluates a VR-operated unmanned ground vehicle and robotic arm with cameras across two experiments involving 14 morphologically diverse plants and expert-established ground truth.
Results
Canopy morphology significantly predicted soil-moisture assessment reliability (p<0.01), while repeated-trial improvements in operator performance were not significant and dense-canopy attempts became faster without becoming more successful.
Takeaways & Limitations
Camera occlusion is identified as a sensing limitation, supporting camera-viewpoint and sensing-strategy adaptation to canopy density.
Takeaways & Limitations
The single-operator design prevents fully disentangling operator learning from system performance, and dense canopies constrain soil-moisture assessment.
Abstract
from arXiv · showhide
This study evaluates the effectiveness of remote human-robot interaction using virtual reality for leaf inspection and soil moisture assessment in a greenhouse environment. The robotic system comprised an unmanned ground vehicle and a robotic manipulator equipped with cameras, governed by kinematic models for navigation and manipulator control. Fourteen distinct plants were inspected across two experiments utilizing VR teleoperation, guided by a set of pre-specified research questions and hypotheses. In the leaf inspection experiments, cycle completion times varied from 3.3 to 8.0 s, and plant-based disease detection was achieved up to 88% accuracy; diseased-spot detection improved numerically in the second experiment, though this change was not statistically significant (p=0.378). For soil moisture assessment, the experiments achieved successful determination of watering needs in up to 64.3% of plants (9 of 14), with consistent success observed for plants 1, 2, 3, 8, 9, 10, and 13; however, this improvement was likewise not statistically significant (p=0.50). A post hoc analysis instead revealed that soil moisture assessment reliability was strongly and significantly predicted by plant canopy morphology (p<0.01): plants with broad, single-leaf canopies reached 100% success by the second experiment, versus only 16.7% for dense, compound canopies. A secondary analysis showed operators became measurably faster at attempting dense-canopy plants without a corresponding gain in success, indicating that camera occlusion, not operator skill or effort, is the dominant limiting factor. These findings show occlusion imposes a sensing limitation rather than a control or training deficiency, and that adapting camera viewpoint and sensing strategy to canopy density is needed to improve the system's accuracy and robustness.
1. INTRODUCTION
This study examines VR-mediated remote human-robot interaction for greenhouse leaf inspection and soil moisture assessment. It investigates whether canopy morphology and operator experience shape the reliability of camera-based teleoperation tasks.
- Motivation: VR-based HRI is positioned as a means to combine human oversight with robotic precision for efficient and scalable greenhouse management.Prior work links agricultural HRI with improved precision, resource management, and reduced labor or environmental costs.
- Study scope: The study combines a camera-equipped robotic arm mounted on an unmanned ground vehicle with remote VR operation for greenhouse inspection.The system targets leaf health and soil moisture monitoring while allowing supervision beyond the greenhouse location.
- Contribution: The paper’s primary contribution is an empirical characterization of how canopy morphology governs camera-based remote-sensing reliability across 14 diverse plants.It specifically tests whether soil-moisture assessment is constrained more by operator skill and familiarity or by unobstructed camera access to the soil.
- Research questions and hypotheses: RQ1–RQ3 assess task accuracy and efficiency, canopy-morphology effects, and performance changes across repeated trials.The hypotheses predict measurable performance, lower success under dense-canopy occlusion, improvement with operator familiarity, and a possible speed-accuracy trade-off.
2.1 System Design
The system design connects the developed VR-based teleoperation framework with its major robotic and remote-environment components. The framework is intended to support remote interaction with the greenhouse platform.
- System overview: Figure 2 presents the complete framework of the developed VR-based teleoperation system.The figure illustrates how the system components are interconnected for teleoperated robotic operation.
- System overview: The system’s component details and capabilities are described as forming the VR subspace and remote environment.These components provide the basis for the detailed operational descriptions that follow.
2.2 VR Subspace
The VR subspace provides an immersive interface for controlling and visualizing the robotic system. Motion smoothing stabilizes controller input, while tracking displays connect operator movements to robot motion.
- VR interface: Unity, a VR headset, hand controllers, robot replicas, and live camera feeds form the VR subspace for greenhouse interaction.The interface visualizes robot pose and collected data while enabling users to manipulate robot movements and perform inspection tasks.
- Motion stabilization: A single exponential filter smooths VR hand-controller data to reduce jitter and noise during teleoperation.The filter produces more stable hand tracking for immersive and responsive interaction.
- Motion stabilization: The smoothing equation computes a weighted average of the previous smoothed observation and the previous raw observation.The smoothing factor α controls responsiveness: values near 1 emphasize recent observations, whereas values near 0 favor slower changes.
- Motion stabilization: With α=0.02, smoothing makes right-controller X-values less sensitive to random hand movements.The system also records end-effector position along the robotic arm’s x, y, and z axes during experiments.
2.3 Remote Environment
The remote environment combines a Husky UGV, Kinova Gen3 arm, cameras, ROS, and Unity-based VR teleoperation through real-time bidirectional communication. Kinematic models convert operator inputs into navigation and manipulation commands, while closed-loop pose feedback corrects robot motion during greenhouse experiments.
- System Architecture: The mobile platform integrates a Husky UGV, Kinova Gen3 arm, Realsense camera, ROS, and Unity through WebSocket communication.The connection exchanges control commands, sensor data, and status updates bidirectionally in real time.
- Mobile Platform Modeling: The Husky uses differential-drive kinematics, relating left- and right-wheel angular velocities to platform linear and angular velocity through wheel radius and wheelbase.The resulting velocity commands determine the UGV pose evolution under a unicycle kinematic model.
- Manipulator Modeling: The Kinova arm uses Denavit–Hartenberg forward kinematics and a manipulator Jacobian to model end-effector motion from joint velocities.The Jacobian supports real-time conversion of VR hand velocities into arm joint commands.
- Closed-Loop Control: The control loop defines error as the difference between the operator-set desired pose and the robot's measured actual pose, then feeds it back to adjust wheel and joint velocities.This feedback drives the tracking error toward zero in the real-time teleoperation loop.
- System Overview: Figures 6–8 present the system architecture, closed-loop control representation, and operational flow from initialization through both experiments.Together, they organize the remote environment's components, feedback structure, and experimental sequence.
- System Architecture: Operator inputs from VR controllers are processed by Unity and transmitted to a robot controller governing both the arm and ground platform.The controller executes commands for the robotic manipulator and mobile platform during teleoperation.
2.4 Experiment Design
The study used two VR-teleoperated greenhouse experiments—leaf disease inspection and soil moisture assessment—with 14 plants examined by a single operator. Performance was evaluated using cycle completion time, task-specific success measures, and human-expert ground truth, within a proof-of-concept design constrained by greenhouse capacity and specimen availability.
- Study setup: Two experiments assessed leaf disease inspection and soil moisture levels using a teleoperated unmanned robotic platform and manipulator.The platform was navigated through the greenhouse while the robotic arm was manipulated to inspect plants or soil.
- Study setup: 14 plants in distinct pots were examined across the greenhouse experiments.The set included plants such as zonal geranium, taro, rose geranium, konjac, flaming flower, Gollum Jade, Giant Taro, and common coleus.
- Study setup: A single operator performed both experiments, so differences between trials combine operator learning with system performance.The design cannot fully disentangle these effects.
- Evaluation: Ground truth came from a human expert’s visual disease assessment and tactile-visual soil assessment, with the latter acknowledged as more subjective than instrument-based measurement.The study used one evaluator and did not assess inter-rater reliability.
- Evaluation: Cycle completion time and robot-plant interaction success rate were the primary performance metrics.Cycle completion time measured the time to inspect a plant or determine whether watering was needed.
- Evaluation: Leaf disease success was scored at spot and plant levels, while soil assessment passed when the watering-needed determination matched human-expert ground truth.Spot-level success required matching an identified diseased spot to a ground-truth spot on the same leaf; plant-level success required identifying at least one confirmed diseased spot.
3. RESULTS AND DISCUSSION
The VR-teleoperated robotic system completed leaf disease inspection and soil moisture assessment across 14 plants, but performance depended more on canopy morphology and camera visibility than on repeated-trial experience. Disease inspection showed modest, non-significant changes, while dense foliage constrained soil assessment through occlusion.
- 3.1 Plant Disease Inspection: 3.7–8.0 s in Experiment 1 and 3.3–7.9 s in Experiment 2 were the observed disease-inspection cycle completion-time ranges.The cross-plant reduction was not statistically significant (t=0.91, p=0.378).
- 3.1 Plant Disease Inspection: 3 of 14 plants had all diseased spots detected in Experiment 2, compared with 2 of 14 in Experiment 1.Plant 6 improved from 0 of 1 to 1 of 1 detected spots, while plant 13 remained undetected in both experiments.
- 3.2 Soil Moisture Level Assessment: 9 of 14 plants were successfully assessed for watering needs in Experiment 2, compared with 7 of 14 in Experiment 1.The increase was not statistically significant (McNemar's test, p=0.50); plants 1, 2, 3, 8, 9, 10, and 13 passed in both experiments.
- 3.3 Canopy Morphology and Occlusion-Driven Failure: 100% of broad/single-leaf plants versus 16.7% of dense/compound plants passed soil assessment in Experiment 2.The morphology association was statistically significant (p=0.0030), consistent with dense foliage blocking the gripper camera's view of the soil surface.
- 3.3.1 Speed-Accuracy Decoupling Under Occlusion: Dense/compound plants became 5.7% faster to attempt, but their pass rate rose only from 0% to 16.7%.This speed-accuracy decoupling indicates that occlusion limits available visual information rather than operator time or attentiveness.
- 3.3.2 Summary of Hypothesis Testing: H3 was not supported because neither cycle completion time nor pass/fail success improved significantly between trials for either task.The study therefore identifies canopy morphology as a stronger determinant of soil-assessment outcome than the limited operator practice provided by two trials.
4. CONCLUSION
Across two VR teleoperation experiments involving 14 plants, the system supported remote leaf inspection and soil moisture assessment, but performance remained constrained by canopy occlusion and study-design limitations.
- 14 plants were inspected across two experiments using VR teleoperation to assess HRI accuracy and efficiency.
- Leaf inspection: 6 s was the mean plant-disease inspection time, with completion times ranging from 3.3 to 8 s and disease detection reaching up to 88%.Diseased-spot detection improved numerically in the second experiment, but the change was not statistically significant (p=0.378).
- Soil moisture assessment: Soil moisture assessment remained less reliable, with initial failures and only a numerical increase in successful watering-need determinations across experiments.
- Limitations and future directions: Future work should adapt gripper-camera viewpoints and auxiliary sensing to canopy density rather than use fixed sensing strategies.The single-operator design also prevents fully separating system performance from operator learning, and post hoc morphology classification may introduce bias.