Source-linked AI summary
TidyBot: Personalized Robot Assistance with Large Language Models
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas Funkhouser
TL;DR
Personalized household cleanup requires robots to infer where objects belong from a user’s preferences, which vary across people. The paper uses LLM summarization to generalize a handful of examples into rules for robot placement and manipulation. It achieves 91.2% accuracy on unseen benchmark objects and an 85.0% real-world object put-away rate.
Problem
Household-cleanup robots need to choose each object’s proper receptacle despite highly variable personal and cultural preferences.
Method
The system uses an off-the-shelf LLM to summarize a few user examples into generalized rules, then applies them through perception and mobile manipulation.
Results
91.2% accuracy on unseen benchmark objects and 85.0% correct object put-away in real-world test scenarios.
Takeaways & Limitations
LLM summarization provides a way to learn generalized user preferences quickly from only a small set of examples for personalized robotics.
Takeaways & Limitations
The real-world system assumes known receptacle locations and does not work well in excessive clutter because the robot cannot drive over objects.
Abstract
from arXiv · showhide
For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.
1 Introduction
TidyBot addresses personalized household cleanup, where selecting each object’s proper receptacle is difficult because preferences vary across individuals. It uses LLM summarization to generalize a few textual examples into user-specific rules and evaluates the approach on benchmark and real-world systems.
- Motivation: Personalized cleanup requires choosing receptacles according to individual or cultural preferences, so there is no one-size-fits-all placement rule.Examples include storing shirts in drawers, on shelves, or in closets.
- Related challenge: Classical methods require specifying every target location, whereas generic approaches average preferences across users and fail to personalize placement.Prior personalization methods instead extrapolate from a few examples using approaches such as collaborative filtering or spatial relationships.
- Approach: The method asks users for a few textual object-placement examples and uses an LLM to summarize them into generalized preferences for that person.For example, examples involving shirt and sock colors can become rules mapping light-colored and dark-colored clothing to different receptacles.
- Evaluation: 91.2% accuracy on unseen objects across all benchmark scenarios, while TidyBot correctly puts away 85.0% of objects in real-world tests.The approach also extends to inferring manipulation primitive selection in addition to object placements.
- Contributions: The paper contributes LLM-based text summarization for robotics, a benchmark dataset for evaluating generalization, and implementation on a real-world mobile manipulation system.The journal version additionally reports a user study, perception analysis, benchmark statistics, and system limitations.
2 Related Work
Prior household-cleanup and object-sorting research includes rearrangement, preference prediction, and rule-based sorting methods. Existing preference-prediction work can remove the need to specify every object’s location but generally models generic rather than personalized preferences, while LLM robotics work explores language-based planning and commonsense reasoning.
- Household cleanup: Household-cleanup benchmarks and methods ask embodied agents to move objects into specified environmental states.The related work includes simulated indoor household-task benchmarks and object rearrangement tasks.
- Household cleanup: Preference-prediction methods infer where objects typically belong, but these predictions are generic rather than personalized to individual users.Other personalization work uses collaborative filtering or spatial relationships to address variability across users.
- Object sorting: Object-sorting research has used clustering, active learning, metric learning, and heuristic search to execute predefined sorting rules.These systems often sort using physical properties such as color.
- LLMs for robotics: Robotics research increasingly uses LLMs for commonsense reasoning and high-level robotic plans.This work situates personalized cleanup within that broader effort to incorporate language-model capabilities into robotic systems.
3 Method
The method summarizes a few user examples with an LLM into generalized rules for receptacle and manipulation choices. TidyBot grounds summarized categories through open-vocabulary perception and repeatedly executes the resulting decisions during cleanup.
- Personalized receptacle selection: A few user placement examples are summarized by an LLM into personalized generalized rules, which are then applied to unseen objects.The examples and inferred rules can be represented in structured Pythonic prompts and parsed as pick-and-place commands.
- Personalized primitive selection: The LLM can also summarize user-preferred manipulation primitives and use those rules to select actions for unseen objects.Illustrative primitives include pick and place for shirts and pick and toss for socks.
- Robot pipeline: The robot uses predetermined manipulation primitives such as pick and place and pick and toss to move recognized objects into receptacles.Other primitive sets could also be used.
- Robot pipeline: The robot combines summarized rules with perception, using overhead localization, egocentric images, CLIP category prediction, and LLM selection of receptacles and primitives.The pipeline then picks up each object, moves it to the selected receptacle, executes the primitive, and repeats until no objects remain.
- Perception grounding: LLM-generated noun phrases provide candidate generalized categories for CLIP, reducing the label set while supporting arbitrary category sets for different users.This combination is described as robust because fewer categories must be differentiated and flexible because categories vary by user.
4 Experiments
Experiments show that LLM summarization generalizes personalized placement preferences better than several baselines, while human evaluations favor the approach and real-world tests achieve strong cleanup performance. Additional analyses examine summary quality, model choice, perception, and system components.
- Benchmark comparisons: 91.2% unseen-object accuracy exceeds WordNet’s 67.5%, RoBERTa’s 77.8%, CLIP’s 83.7%, and another baseline’s 78.5%.These comparisons evaluate generalization from example placements to unseen objects.
- Ablations: 91.8% seen-object and 91.2% unseen-object accuracy substantially exceed the commonsense-only baseline’s 45.0% seen-object accuracy.The commonsense baseline produces sensible placements but does not reflect particular users’ preferences.
- Ablations: 6% higher accuracy for both seen and unseen objects is achieved by human-written summaries than by LLM-produced summaries.The result suggests that improving LLM summarization could further improve the method.
- User study: 82.2% ± 7.7% of human responses aligned with the benchmark ground truth across 40 participants.Alignment was measured by whether participants preferred the method closer to benchmark unseen-object accuracy.
- Real-world experiments: 85.0% of objects were placed into correct receptacles in real-world tests, while localization reached 92.5%, classification 95.5%, and LLM selection 100%.The evaluation used 240 objects across 8 scenarios, with 3 runs per scenario.
5 Conclusion
The paper shows that LLM summarization can generalize a person’s preferences from a handful of examples for personalized robotic cleanup. The approach outperforms benchmark baselines and achieves 85.0% success in real-world TidyBot scenarios.
- LLM summarization generalizes user preferences for personalized robotics from a handful of examples.
- The approach infers generalized rules that let robots manipulate objects according to a particular user’s preferences.
- The summarization approach outperforms several strong baselines on the benchmark.
- 85.0% success rate was achieved by TidyBot in real-world cleanup test scenarios.
- The approach uses LLMs off-the-shelf instead of costly data collection and model training to achieve generalization in robotics.
Appendix A LLM prompts
Appendix prompts instantiate LLM-based preference summarization for receptacle selection, manipulation primitive selection, and category extraction in the real-world system.
- A.1 Summarization for receptacle selection: A.1 uses object-placement examples to prompt summaries that generalize receptacle preferences across object categories.
- A.2 Receptacle selection: A.2 applies summarized rules to assign individual objects to receptacles through pick-and-place actions.
- A.3 Summarization for primitive selection: A.3 prompts summaries that select pick-and-place or pick-and-toss primitives from object examples.
- A.4 Primitive selection: A.4 converts primitive-selection summaries into category-level rules and applies them to objects.
- A.5 Category extraction for real-world system: A.5 extracts object categories and maps them to receptacles for real-world cleanup scenarios.
- A.6 Receptacle selection for real-world system: A.6 applies category-to-receptacle summaries to real-world object classes using pick-and-place actions.
- A.7 Primitive selection for real-world system: A.7 applies category-level primitive rules to real-world classes, including pick-and-place and pick-and-toss.