Source-linked AI summary
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Abhinav Gupta, Adithyavairavan Murali, Dhiraj Gandhi, Lerrel Pinto
TL;DR
Robotic learning policies often rely on curated lab data, while real homes introduce distribution shifts and noisy labels from inexpensive hardware. This paper collects grasping data across diverse homes and models execution noise as a latent variable, finding improved generalization to unseen homes and better performance from noise modeling.
Problem
Existing robot-learning datasets and policies are often developed in labs, where models can overfit and may not generalize to diverse household environments.
Method
The paper collects household grasping data with a mobile manipulator costing under 3K USD and uses Robust-Grasp to disentangle low-cost robots’ commanded and actual executions.
Results
Models trained on diverse household data generalize better to unseen home environments, while explicitly modeling low-cost robot error significantly improves grasping performance.
Takeaways & Limitations
Household data collection and explicit treatment of low-cost robot noise support robot grasping beyond curated laboratory settings.
Takeaways & Limitations
Low-cost hardware introduces inaccurate control, calibration errors, and noisy labels; the noise model also assumes conditional independence given global information.
Abstract
from arXiv · showhide
Data-driven approaches to solving robotic tasks have gained a lot of traction in recent years. However, most existing policies are trained on large-scale datasets collected in curated lab settings. If we aim to deploy these models in unstructured visual environments like people's homes, they will be unable to cope with the mismatch in data distribution. In such light, we present the first systematic effort in collecting a large dataset for robotic grasping in homes. First, to scale and parallelize data collection, we built a low cost mobile manipulator assembled for under 3K USD. Second, data collected using low cost robots suffer from noisy labels due to imperfect execution and calibration errors. To handle this, we develop a framework which factors out the noise as a latent variable. Our model is trained on 28K grasps collected in several houses under an array of different environmental conditions. We evaluate our models by physically executing grasps on a collection of novel objects in multiple unseen homes. The models trained with our home dataset showed a marked improvement of 43.7% over a baseline model trained with data collected in lab. Our architecture which explicitly models the latent noise in the dataset also performed 10% better than one that did not factor out the noise. We hope this effort inspires the robotics community to look outside the lab and embrace learning based approaches to handle inaccurate cheap robots.
1 Introduction
Data-driven robot learning often relies on curated or simulated environments, while home deployment requires broader environmental diversity. The paper addresses this gap by collecting grasp data in homes with inexpensive robots and modeling the resulting label noise.
- Motivation: Simulators and lab datasets may not capture the reality gap or the environmental diversity needed for robust robot learning.Existing approaches often overfit despite larger datasets, while real-world deployment involves unstructured visual environments.
- Motivation: Real-world robot datasets have expanded across tasks, but learned models often overfit and performance plateaus as data volume increases.This motivates asking why robotic action data does not yield gains comparable to computer vision and natural language processing.
- Challenges: Home data collection is constrained by expensive robots, unreliable low-cost control, limited supervision, and external sources of noise.These constraints create a chicken-and-egg problem: improving robots requires home data, but current robots are not yet good enough to collect it reliably.
- Contribution: The paper presents a systematic home-data collection effort using robots costing under 3K USD, data from six training homes and three testing homes, and a model that factors label noise.The authors report superior performance from diverse home environments with little-to-no domain adaptation.
2 Overview
The paper targets biased grasp-learning data by moving collection into diverse homes with a scalable, inexpensive mobile manipulator. Because low-cost hardware introduces execution and calibration errors, the approach explicitly models noisy labels.
- Overview: The paper focuses on grasping because existing datasets contain biases such as simple backgrounds and repeated environment dynamics.The stated goal is to diversify data and environments so learned models can better generalize.
- Hardware: The authors propose a customized mobile manipulator to reduce the cost of collecting large-scale data across many homes.The scaling objective is motivated by a need to collect data from hundreds or thousands of homes.
- Noise: Low-cost motors cause inaccurate control, calibration errors, and hardware execution errors, making the dataset diverse but noisy.A commanded grasp at (x, y) may execute at (x + δx, y + δy), so the success label can correspond to another location.
- Noise: The proposed solution models execution noise as a latent variable using separate networks for likely noise and the action to execute.This design addresses unreliable labels produced by inexpensive robots.
3 Learning on Low Cost Robot Data
The method learns planar grasping from low-cost robot data while separating the true grasp distribution from structured execution noise. It combines a grasp predictor, a noise model, and marginalization over possible actual executions.
- Method: The method addresses noisy labels caused by hardware execution, inaccurate kinematics, camera calibration, proprioception, and wear.Its architecture aims to disentangle the low-cost robot’s actual and commanded executions.
- Grasp prediction: Planar grasping predicts a configuration (x, y, θ) from an object observation, with fixed height and end-effector pitch.Training uses image patches, executed grasps, success labels, discretized angles, and binary cross entropy.
- Noise modeling: Robust-Grasp models grasp success by marginalizing over nine neighboring patches that represent possible actual executions.The latent variable z identifies the true patch executed when the observed patch may be inaccurate.
- Architecture: The Grasp Prediction Network estimates success probabilities, while the Noise Modelling Network estimates environment-dependent noise distributions.A marginalization operator combines the two distributions into the final grasp model.
- Noise model training: The noise model uses global scene and robot information, assuming latent noise is conditionally independent of local patch variables given that information.The inputs include the whole scene image, robot identity, and grasp pixel location; NMN is trained jointly with GPN despite lacking explicit noise labels.
- Training: The model is initialized with pretrained visual features and trained in two stages: GPN initialization followed by joint end-to-end optimization with NMN and marginalization.The implementation uses a pretrained ResNet-18 and separate training stages for grasp prediction and noise modeling.
4 Results
The experiments test generalization across held-out data, unseen homes, and a curated lab setting. Models trained with diverse home data generalized better, while explicitly modeling low-cost robot noise improved grasping performance.
- Experimental Design: The Robust-Grasp architecture models low-cost robot error and uses Home-LCA data collected across diverse households and environmental conditions.
- Evaluation Settings: The evaluation uses held-out binary classification, physical grasping on 20 novel objects in three unseen homes, and physical grasping on Sawyer in the lab.
- Held-out Data: Home-LCA training generalized across both home and curated lab environments, whereas lab-only models overfit their respective environments.
- Real-LCA: 43.7% improvement was achieved when training on Home-LCA rather than Lab-Baxter in physical grasping on novel objects in unseen homes.
- Real-Sawyer: 77.5% grasping accuracy was achieved by Robust-Grasp on Real-Sawyer, compared with 56.25% for Patch-Grasp trained on the same dataset.
5 Related Work
Prior work established large-scale robot-learning datasets and data-driven grasping, but largely relied on restrictive laboratory environments and expensive collection setups. Related research also examined learning with low-cost robots and noisy data.
- Robot Datasets: Large-scale grasping datasets progressed from hand-annotated data to self-supervised collection and multi-arm reinforcement-learning systems.
- Robot Datasets: Existing data-collection efforts commonly used restrictive lab environments and high-cost labeling mechanisms.
- Data-Driven Grasping: Data-driven grasping methods learn mappings from visual observations to grasp control, using both physical datasets and simulation.
- Low-Cost Robots: Research on learning with low-cost robots and manipulators remains limited compared with work using collaborative or industrial robots.
- Noisy Data: Prior noisy-data research distinguishes noise independent of the environment from noise statistically dependent on the environment and motivates modeling and correction.
6 Conclusion
The paper introduces large-scale grasping data collected in diverse homes with a low-cost mobile manipulator and a framework that factors out label noise. Home-trained models improved performance in unseen homes and curated lab settings.
- Conclusion: The dataset contains about 28K grasps collected in six homes under varying environmental conditions using a mobile manipulator costing under 3K USD.
- Conclusion: The proposed architecture factors out noise introduced by cheap, inaccurate robots during data collection.
- Conclusion: 43.7% improvement was obtained over a model trained with laboratory data when evaluating on novel objects in three unseen home environments.
- Conclusion: The framework performed 33% better than DexNet, which struggled with poor depth sensing in naturally lit household environments.