Source-linked AI summary
Real-Time Service Robot Replanning via Simple Button Interaction for Improved Task Success and User Experience
Ryo Terashima, Yuga Yano, Koshun Arimura, Hakaru Tamukoh
TL;DR
Service robots may encounter failures they cannot detect, while conventional intervention can burden users. The paper proposes a one-button feedback function that enables LLM-based replanning during task execution. Experiments report improved task success and user evaluations supporting the hypothesis that simple interactions do not cause a negative user experience.
Problem
Robots cannot detect all task failures, while intervention designs can impose burden on users.
Method
Users press a single button to provide feedback, which the robot uses with the failure situation and prior skills to replan actions.
Results
Task success improved from 55% to 86% with the feedback system, while questionnaire results indicated reduced anxiety and greater desire for robot intervention.
Takeaways & Limitations
The findings support simple, user-led intervention as a feedback design that can improve task success and user experience.
Takeaways & Limitations
The system relies on users to detect failures, so fewer failures may be caught when users do not constantly monitor the robot.
Abstract
from arXiv · showhide
Service robots must respond to unexpected instructions in real-world environments. However, robots cannot detect all failures and exceptions during a task. To address these issues, we propose a real-time feedback function that enables robots to modify their behavior based on human feedback. In this system, users can intuitively send feedback to the robot by pressing a single button on a tablet when the robot fails to act correctly. Robots use this feedback to consider their failures and replan appropriate actions to complete the task. We conducted experiments with and without the feedback function to verify the following hypothesis: "Simple interactions do not cause a negative user experience." All questionnaire responses are evaluated on a five-point Likert scale. After adding the feedback function, the response score for the question "Did you feel that the robot's behavior was unexpected?" improved by 0.5 points, and that for "Did you feel anxious about the robot's behavior at times?" improved by 0.9 points. These results support the study's hypothesis and indicate that incorporating this real-time feedback function can simultaneously improve task success and the user experience.
I. INTRODUCTION
Service robots need to handle unexpected instructions and failures they cannot detect autonomously, but intervention can burden users. This study proposes a one-operation feedback system and evaluates whether it improves task outcomes without harming user experience.
- 87% success on single-action SayCan tasks falls below 50% for multi-action tasks, motivating failure detection and replanning for complex tasks.
- Robots cannot detect failures such as object or voice misrecognition, creating a need for human intervention to correct behavior.
- The study tests whether simple interactions avoid negative user experience, addressing concerns that interaction frequency and design can increase burden, frustration, and anxiety.
- The proposed system lets users intervene with a single operation while delegating failure reasoning to an LLM that replans from the failure situation and prior skills.
- Experiments compared restaurant-task performance and HRI questionnaires with and without the feedback function in a simulated restaurant environment.
A. LLM-Based Task Planning Methods
LLM-based task-planning methods enable robots to interpret natural-language instructions and select situationally appropriate behaviors, including by combining predefined skills.
- LLM-based planners interpret natural-language instructions and select appropriate robot actions for the situation.
- SayCan combines predefined skills according to natural-language instructions, while related work integrates LLM task understanding with dynamic maps for real-world HRI.
B. Handling Failures in Robot Tasks
Prior failure-handling methods use uncertainty or failure detection to request human assistance, but robots may miss semantic and object-recognition errors. Interaction burden also depends on attention, timing, and intervention design.
- KnowNo requests human assistance when conformal-prediction uncertainty is high, whereas Self-Recovery Prompting replans after detected failures or insufficient information.
- Robots cannot ask for help when they fail to recognize their own semantic or object-recognition failures.
- Interaction burden reflects consumed attention resources and depends on intervention timing, with interruptions during task execution increasing completion time, errors, discomfort, and anxiety.
- Recent robot research has therefore often adopted policies that minimize user interaction.
III. PROPOSAL
The proposal is a seamless real-time feedback function for failures the robot cannot detect itself. Users provide feedback with one button press, allowing the robot to modify its task while minimizing interaction burden.
- The feedback function lets humans intuitively intervene when the robot cannot detect a failure and seamlessly modify its behavior during task execution.
- The proposed system overview presents this feedback-based intervention mechanism for real-time robot task modification.
- Users send feedback with a single button press to minimize the number of interaction operations.
A. Task Planning
The system uses sequential task execution and enables replanning after a user signals failure. Failure analysis is delegated to GPT-4o, allowing feedback through a single button without requiring users to explain the problem.
- The proposed system performs sequential skill execution to respond to sudden task modifications during ongoing tasks.
- GPT-4o analyzes the failure and uses the situation and previously executed skills to replan the robot’s actions.
- Users press one tablet button to report that something failed, regardless of when the failure occurs.
- When the robot grasps the wrong drink, feedback prompts it to identify the mistake and search for the correct drink.
IV. EXPERIMENT
The experiment evaluates waiter-task performance and human-robot interaction under matched conditions with and without the feedback function. Participants completed comparable restaurant scenarios, and outcomes were assessed using task success and five-point Likert questionnaire responses.
- The restaurant experiment focuses on waiter tasks involving orders, food serving, table cleaning, and user interaction.
- The study compares task success rates and questionnaire results under the same experimental conditions with and without feedback.
- Questionnaire responses use a five-point Likert scale to assess human-robot interaction.
A. Experimental Result Without the Feedback Function
Without feedback, the robot completed 21 of 38 requested-item deliveries, while participants reported unexpected and uncomfortable behavior and strongly wanted a way to provide additional instructions or identify failures.
- 21 of 38 requested-item deliveries succeeded without the feedback function, with speech-recognition and object-misrecognition errors among the main failures.
- 3.0 points for unexpected behavior and 3.1 points for anxiety indicate that users did not fully understand or feel comfortable with the robot’s behavior.
- Users rated their desire to give additional instructions or point out failures at 4.5 points overall.
- The results associate unexpected robot behavior with reduced usability when users cannot provide additional instructions or identify failures.
B. Experimental Result With the Feedback Function
With feedback, the robot completed 43 of 50 tasks, including 8 recovered through user feedback. Questionnaire scores also improved for perceived unexpected behavior and anxiety, supporting the hypothesis that simple interaction need not harm user experience.
- 43 of 50 tasks succeeded with feedback, including 8 tasks recovered through the feedback function.
- The robot can reconsider and correct tasks after receiving user feedback.
- Unexpected-behavior responses improved by 0.5 points to 2.5, while anxiety responses improved by 0.9 points to 2.2.
- Despite increasing interaction, feedback reduced anxiety, increased users’ desire for intervention, and remained usable across ages, including children.
V. DISCUSSION
The feedback system improved task success and user experience, while revealing monitoring and interaction-design limitations. Its benefits extended across age groups despite differences in anxiety and intervention timing.
- Users pressed the feedback button only 8 times across 50 tasks, yet anxiety decreased and interest in more robot intervention increased.Over 90% of users did not press the button, suggesting that simply having a means to modify behavior may reduce anxiety.
- Task success improved from 55% to 86% with the proposed feedback system.The result indicates that user-initiated interventions can address failures the robot cannot detect itself.
- The feedback function improved anxiety across both adults and elementary school children, although adults reported lower anxiety overall.This suggests the design was effective across a broad user base despite age-related differences.
- Elementary school children found it harder than adults to predict the robot’s next action and determine when to intervene.Average scores for understanding the next action were 3.5 for children and 4.1 for adults.
- The current design relies on users noticing failures, so missed failures may reduce success in realistic settings where users do not constantly monitor robots.Clearer state information is identified as an important direction for reducing the need for continuous monitoring.
- Button-based intervention is only one design option, and alternatives such as voice, gestures, and gaze detection involve usability, accessibility, and implementation trade-offs.
VI. CONCLUSION
The study introduced a single-operation feedback function for restaurant service robots and evaluated it against a condition without feedback. The function improved serving success and supported positive user impressions despite increased interaction.
- Serving success improved from 55% to 86% when the feedback function was implemented.
- The feedback function made robot behavior easier to understand and reduced anxiety, supporting positive impressions of human-robot interaction.Its simple, accessible design increased users’ choices and supported interaction across ages.
- Future work will improve robot-state information, examine voice and gesture feedback, and test more fluid and complex environments.