Source-linked AI summary
On Bringing Robots Home
Nur Muhammad Mahi Shafiullah, Anant Rai, Haritheja Etukuru, Yiqian Liu, Ishan Misra, Soumith Chintala, Lerrel Pinto
TL;DR
General-purpose home robots must learn diverse household tasks outside curated laboratory settings. Dobb·E combines an ergonomic Stick for collecting demonstrations with household pretraining and adaptation on a commercial mobile robot. It achieved 81% success across 109 tasks in 10 homes, while revealing sensing and hardware limitations.
Problem
Robotics lacks strong evidence for general-purpose task learning in diverse real homes, where standard lab assumptions about objects, scenes, and lighting do not hold.
Method
Dobb·E uses the Stick to collect household demonstrations, pretrains Home Pretrained Representations on them, and fine-tunes the model for new tasks on Hello Robot Stretch.
Results
81% overall success was achieved across 109 tasks in 10 homes over 30 days, with 102 tasks reaching performance ≥50%.
Takeaways & Limitations
Household demonstrations and home-data pretraining support adaptable robot manipulation across varied homes, while the experiments expose challenges absent or overlooked in laboratory robotics.
Takeaways & Limitations
Robot force, reach, battery, and dexterity constraints limit the household tasks the system can physically attempt.
Abstract
from arXiv · showhide
Throughout history, we have successfully integrated various machines into our homes. Dishwashers, laundry machines, stand mixers, and robot vacuums are a few recent examples. However, these machines excel at performing only a single task effectively. The concept of a "generalist machine" in homes - a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective - has long been a goal in robotics that has been steadily pursued for decades. In this work, we initiate a large-scale effort towards this goal by introducing Dobb-E, an affordable yet versatile general-purpose system for learning robotic manipulation within household settings. Dobb-E can learn a new task with only five minutes of a user showing it how to do it, thanks to a demonstration collection tool ("The Stick") we built out of cheap parts and iPhones. We use the Stick to collect 13 hours of data in 22 homes of New York City, and train Home Pretrained Representations (HPR). Then, in a novel home environment, with five minutes of demonstrations and fifteen minutes of adapting the HPR model, we show that Dobb-E can reliably solve the task on the Stretch, a mobile robot readily available on the market. Across roughly 30 days of experimentation in homes of New York City and surrounding areas, we test our system in 10 homes, with a total of 109 tasks in different environments, and finally achieve a success rate of 81%. Beyond success percentages, our experiments reveal a plethora of unique challenges absent or ignored in lab robotics. These range from effects of strong shadows, to variable demonstration quality by non-expert users. With the hope of accelerating research on home robots, and eventually seeing robot butlers in every home, we open-source Dobb-E software stack and models, our data, and our hardware designs at https://dobb-e.com
1 Introduction
Dobb·E addresses the gap between narrow household machines and general-purpose home robots by combining user-friendly demonstrations, household pretraining, and deployment on a commercial mobile manipulator. Across 10 homes, it achieved an 81% overall success rate while exposing hardware, sensing, and task-complexity challenges.
- Motivation: Dobb·E targets general-purpose domestic tasks by shifting robot-learning evaluation from curated labs toward diverse real-world households.The authors identify home settings as substantially different from industrial and academic environments.
- Framework: The framework combines efficiency, safety, and user comfort through large-scale data, human demonstrations instead of trial-and-error, and an ergonomic collection tool.The Stick enables task-specific demonstrations without directly operating the robot.
- Hardware: The Stick combines an affordable reacher-grabber, 3D-printed parts, and an iPhone, while a matching robot mount enables direct demonstration transfer.The setup is centered on the commercially available Hello Robot Stretch.
- Data and models: The HoNY dataset contains 13 hours of household data, 5,620 demonstrations, 216 environments, and 22 New York homes for pretraining.These demonstrations are used to train Home Pretrained Representations (HPR).
- Data and models: Dobb·E uses HPR, a streamlined vision model pretrained with self-supervised learning and fine-tuned with 24 demonstrations for novel tasks.The model incorporates visual and depth information for 3D reasoning.
- Results: HPR improved task success rate by at least 23% over other foundational vision models trained on larger internet datasets.The authors also report that accurate Stick odometry, actions, and iPhone Lidar depth are heavily relied upon.
- Remaining challenges: Robot force, reach, and battery constraints limit physically solvable tasks, while ambiguous sensing and temporally extended tasks challenge the policy framework.The software, models, data, and hardware designs are open-sourced to support future home-robotics research.
2 Technical Components and Method
Dobb·E is a behavior-cloning framework for teaching a mobile robot household tasks through demonstrations collected with an inexpensive Stick. It combines household pretraining data, visual representations, depth, and task-specific policies for deployment in homes.
- Method: Dobb·E combines hardware design, household data collection, self-supervised pretraining, and deployment into a four-stage imitation-learning pipeline.The framework is designed around efficiency, safety, and user comfort.
- Demonstration Collection: The Stick uses a $25 reacher-grabber with an iPhone mount to collect RGB, depth, and 6D motion data while approximating the robot end-effector.The iPhone records these signals at 30 frames per second, and matching mounts support transfer to the robot.
- Pretraining Dataset: The Homes of New York dataset contains 5620 demonstrations collected across 22 homes, totaling 13 hours of recordings and almost 1.5 million frames.Volunteers contributed demonstrations spanning eight broad household task classes.
- Visual Representation Learning: HPR is a ResNet34 visual encoder pretrained with MoCo-v3 self-supervised learning on the household dataset and later fine-tuned for deployment.The model is designed to remain small enough for the robot’s onboard computer.
- Downstream Policy Learning: For each new task, Dobb·E learns a policy from RGB-D inputs that predicts 7-dimensional relative motion and gripper actions at 3.75 Hz.The output contains three translations, three axis-angle rotations, and one gripper value.
3 Experiments
Dobb·E was evaluated in household environments across 10 homes and 109 tasks, with task outcomes and demonstration characteristics reported through aggregate figures and task-level tables.
- Overall Evaluation: 81% success was achieved across 109 tasks evaluated in 10 households in the New York and New Jersey area.The evaluation also found that the robot completed 102 tasks with at least even odds.
- Task-Level Results: The task-level evaluation reports success rates out of 10 trials for household tasks grouped by category.Table 2 lists the attempted tasks, their categories, and associated success rates.
- Task-Level Results: Success rates are summarized across 20 task groups with variance shown in the error bars.The figure provides a group-level view of performance variability.
- Action-Type Analysis: Action-type analysis plots the number of successes out of 10 rollouts against the number of tasks attaining each success count.This breakdown organizes outcomes by the actions required to solve tasks.
- Demonstration Analysis: Demonstration analysis reports a mean duration of 3.82 seconds and a median duration of 3.49 seconds, alongside correlation with policy success rate.The figure compares demonstration time with the success rate of the associated robot policy.
3.2 Understanding the Performance of Dobb·E
Performance varied substantially by task and required motion. Tasks without wrist rotation performed better overall, while full 6D motion was associated with more difficult and less concentrated outcomes.
- Task Groups: 100% was the highest average success rate for Air Fryer Closing and Cushion Flipping, while 6D pick & place had the lowest at 56%.The authors attribute many 6D pick-and-place failures to trajectories requiring translations and rotations across multiple axes.
- Action Types: Tasks without wrist rotation had more positively skewed success-rate distributions than tasks requiring yaw, roll, or combined yaw, pitch, and roll.The comparison is based on action-type buckets shown in Figure 12.
- Action Types: Tasks requiring full 6D motion had the flattest success distribution, indicating greater difficulty than tasks without full 6D motion.The reported pattern concerns the distribution of successes across task outcomes.
Correlation between demo time and difficulty
Robot difficulty is only weakly related to the time humans need to demonstrate a task. Lighting changes further show that execution conditions can destabilize otherwise successful policies.
- Correlation between demo time and difficulty: r = −0.24 (p = 0.012 < 0.05) indicates a weak negative correlation between human demonstration time and robot task success.Longer human demonstrations may correspond to harder robot tasks, but other factors also contribute to difficulty.
- Lighting and shadows: Strong shadows absent during data collection may cause the policy to behave erratically during execution.The effect is condition-dependent: ambient day–night changes were often tolerated, whereas strong task-space shadows were problematic.
- Lighting and shadows: The Home 1 Air Fryer Opening policy failed under a strong arm shadow but had no further failures after an overhead light produced even lighting.A separate table-pulling task succeeded despite strong shadows, so the shadow effect was inconsistent across tasks.
- Lighting and shadows: Low-light iPhone photography helped preserve similar robot views when demonstrations occurred during the day and execution occurred at night.This was demonstrated for Home 8 cushion straightening, where the perceived light-level difference was negligible.
Sensor limitations
Depth and visual sensing fail in specific household conditions, especially reflective surfaces and views through windows. These failures can make depth-aware policies leave their training distribution and produce task failures.
- Reflective surfaces: Lidar-based depth sensing is brittle around shiny or reflective objects, producing out-of-distribution depth values that can hinder policy execution.Mirrors can also change the recorded reflection from demonstrator during collection to robot during rollout, creating a visual mismatch.
- Reflective surfaces: 0/10 successes occurred for Home 3 Air Fryer Opening because its shiny handle produced bad depth and mismatched demonstrator and robot reflections.The task combined reflective-surface depth failure with a change in the reflected actor between demonstrations and execution.
- Noisy depth: 10/10 successes with depth-free models versus 2/10 with depth-using models occurred for Home 1 outward-facing window blinds.The outward camera view in darkness generated many out-of-distribution depth values.
- Physical sensing and execution: A heavy door handle can tip the bottom-heavy robot when pulled high, whereas pulling closer to the ground succeeds.The failure reflects the interaction between manipulation direction and the robot’s physical balance.
- Visual task cues: A behavior-cloning policy failed to place an object when the pickup and placement shelf looked identical, continuing left without a clear placement signal.A red book on a different shelf provided a visual cue that enabled successful completion in the contrasting demonstrations.
Robot hardware limitations
Hardware constraints limit Dobb·E’s reachable workspace, force application, and fine manipulation. These constraints interact with task geometry and the absence of temporal state in the policy.
- Workspace and stability: The Stretch’s manipulation space caps at 1 m, making high-position tasks such as light-switch flicking and elevated pick-and-place difficult.Its tall, bottom-heavy body can tilt when the arm applies strong pulling or pushing force near the robot’s top.
- Fine manipulation: 65% success on knob turning reflects the fine-motor requirement for centering the grasp precisely on the knob.An off-center grasp can move the wrist without turning the knob properly.
- Temporal dependencies: 0/10 successes occurred when pick and place used visually identical pickup and placement shelf locations, while a red-book cue produced 7/10 successes.The last-observation-only policy cannot distinguish moving forward from moving backward when the views look the same.
- Representation comparison: VC-1 was the closest alternative representation to Dobb·E’s model but showed bimodal behavior, either performing comparably or failing entirely in household environments.The comparison used task-completion-rate distributions with 95% confidence intervals.
3.4 Ablations
Ablations show that representation choice and demonstration quantity materially affect household manipulation performance. Home-pretrained representations outperform most alternatives, while additional demonstrations improve performance until saturation.
- 3.4 Ablations: The ablation benchmark compares visual representations, demonstration counts, depth perception, and demonstrator expertise across home-like laboratory tasks.The experiments evaluate alternate system components and their relative contributions to performance.
- Alternate visual representation models: VC-1 was the only alternative representation close to the trained representation in benchmark environments but underperformed it in household environments.VC-1’s household behavior was bimodal, with either comparatively strong performance or complete task failure.
- Number of demonstrations required for tasks: Adding more demonstrations always improves performance, with gains scaling until the model saturates.When the model can partially solve a task, additional demonstrations can improve average performance.
Depth Perception
Depth information improves Dobb·E’s performance, especially when the camera is close to featureless objects where RGB-only imitation struggles. Success rates generally converge with more demonstrations, and depth outperforms RGB-only input in most settings.
- Depth Perception: Adding depth perception helps Dobb·E perform much better than using RGB-only input.Depth provides approximate 3D structure that supports manipulation near task surfaces.
- Depth Perception: Depth is especially useful near featureless doors and drawers, where RGB-only models cannot reliably determine when to close the gripper.The depth model uses camera-to-surface distance to judge when to open or close the gripper.
- Depth Perception: Success rates converge as the number of demonstrations increases across five different tasks.
- Depth Perception: Using depth outperforms not using depth in most experimental settings, although exceptional cases remain.The exceptions are discussed in the paper’s Section 3.3.2.
- Depth Perception: Novice demonstrators produced sufficient demonstrations after practice, despite initially variable task success rates.Across two tasks, collected data achieved 100% and 70% success, while individual novice results ranged from 0% to 90%.
Odometry
Dobb·E compares iPhone hardware odometry with OpenSfM visual odometry for extracting robot actions from demonstrations. iPhone odometry is more robust when the camera approaches surfaces, where OpenSfM loses visual features.
- Odometry: The experiment compares iPhone hardware-based odometry with OpenSfM visual odometry and evaluates the quality of their extracted actions.
- Odometry: Figure 24 presents open-loop rollouts using actions extracted from iPhone odometry and OpenSfM.
- Odometry: iPhone hardware odometry produces reliable extracted actions even when the camera is close to surfaces.
- Odometry: OpenSfM-extracted actions are generally adequate at a distance but fail near surfaces when visual features disappear.
4 Open Problems and Request for Research
The paper identifies unresolved challenges for deploying learned robots at home, including scene memory, limited validation on long-horizon tasks, missing force sensing, and constrained robot hardware. Longer tasks can compound subtask errors, while current platforms limit the task range.
- 4 Open Problems and Request for Research: Dobb·E primarily focuses on short-horizon tasks, leaving longer-horizon household behavior as an open problem.The authors test multi-step tasks in the lab as an initial extension.
- 4 Open Problems and Request for Research: Compound failures reduce long-horizon success even though Dobb·E can chain subtasks.The muffin-in-toaster task achieved 1 success out of 10 trials, while the cup-in-drawer task achieved 6 success out of 10 trials.
- 4 Open Problems and Request for Research: Figure 26 depicts three temporally extended tasks, each composed of five to seven subtasks.
- 4 Open Problems and Request for Research: Single-view operation creates a need for robotic scene memory, but existing memory algorithms are rigid and difficult to edit during deployment.
- 4 Open Problems and Request for Research: The absence of tactile and force sensors can cause the robot to apply too much or too little force during deployment.The paper points to cheap sensors and learned visual contact-force estimation as possible directions.
- 4 Open Problems and Request for Research: Hello Robot Stretch’s height, force output, and dexterity prevent attempting many household tasks.The paper argues that more cheap, versatile, home-ready platforms are needed.
5 Reproducibility and Call for Collaboration
Dobb·E is designed for reproducibility through broad open-sourcing of its hardware, code, data, and models. The authors also provide deployment documentation and invite researchers to contribute and collaborate.
- 5 Reproducibility and Call for Collaboration: The project open-sources hardware designs, code, datasets, and models to support progress in home robotics.
- 5 Reproducibility and Call for Collaboration: Dobb·E uses the commercially available Hello Robot Stretch, with shared 3D-printable hardware files for the gripper and robot attachment.The Stretch was listed at US$24,000 as of November 2023.
- 5 Reproducibility and Call for Collaboration: The project shares model fine-tuning code, controller code, a household deployment guide, and a best-practices handbook.
- 5 Reproducibility and Call for Collaboration: The authors offer 30-minute online meetings to help researchers set up the framework in labs or homes.
- 5 Reproducibility and Call for Collaboration: The authors describe Dobb·E as an early step toward learned household robots and welcome credited contributions to their repositories and datasets.