PaperlayerView exact revision on arXiv

Source-linked AI summary

BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities

Yunfan Jiang, Ruohan Zhang, Josiah Wong, Chen Wang, Yanjie Ze, Hang Yin, Cem Gokmen, Shuran Song, Jiajun Wu, Li Fei-Fei

arXiv:2503.05652v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Household whole-body manipulation requires coordinated control across arms, navigation, and reachability, while the resulting embodiment complexity challenges data collection and policy learning. BRS combines JoyLo teleoperation with the WB-VIMA policy and achieves strong performance across real-world household tasks, while remaining limited by perception and long-horizon error accumulation.

  • Problem

    Household tasks require bimanual coordination, stable navigation, and extensive reachability, but hardware complexity makes scalable data collection and coordinated whole-body policy learning difficult.

  • Method

    BRS combines the low-cost JoyLo whole-body teleoperation interface with WB-VIMA, which models coordinated actions using embodiment hierarchy and multimodal observations.

  • Results

    BRS achieves 88% average sub-task success and 93% peak full-task success across five real-world household tasks.

  • Takeaways & Limitations

    BRS integrates robotic embodiment, data collection, and whole-body learning into a framework demonstrating strong performance on real-world household tasks.

  • Takeaways & Limitations

    Long-horizon multi-stage execution suffers from compounding errors that can reduce overall task success despite high sub-task success.

Abstract

from arXiv · show

Real-world household tasks present significant challenges for mobile manipulation robots. An analysis of existing robotics benchmarks reveals that successful task performance hinges on three key whole-body control capabilities: bimanual coordination, stable and precise navigation, and extensive end-effector reachability. Achieving these capabilities requires careful hardware design, but the resulting system complexity further complicates visuomotor policy learning. To address these challenges, we introduce the BEHAVIOR Robot Suite (BRS), a comprehensive framework for whole-body manipulation in diverse household tasks. Built on a bimanual, wheeled robot with a 4-DoF torso, BRS integrates a cost-effective whole-body teleoperation interface for data collection and a novel algorithm for learning whole-body visuomotor policies. We evaluate BRS on five challenging household tasks that not only emphasize the three core capabilities but also introduce additional complexities, such as long-range navigation, interaction with articulated and deformable objects, and manipulation in confined spaces. We believe that BRS's integrated robotic embodiment, data collection interface, and learning framework mark a significant step toward enabling real-world whole-body manipulation for everyday household tasks. BRS is open-sourced at https://behavior-robot-suite.github.io/

1 Introduction

BRS targets whole-body manipulation for household tasks by combining hardware, teleoperation, and policy-learning innovations. Its evaluation spans five challenging real-world tasks and reports strong learned-policy performance.

  • BEHAVIOR-1K analysis identifies bimanual coordination, stable and accurate navigation, and extensive end-effector reachability as essential capabilities.
  • Household tasks combine capability demands: lifting heavy objects requires bimanual manipulation, retrieving objects requires navigation, and opening doors while carrying groceries requires both.
  • Hardware with dual arms, a mobile base, and a flexible torso enables whole-body manipulation but complicates scalable data collection and coordinated-action modeling.
  • BRS introduces JoyLo, a low-cost whole-body teleoperation interface, and WB-VIMA, a learning algorithm for coordinated whole-body actions.
  • 88% average success on short-horizon sub-tasks and 93% peak success on long-horizon full tasks were achieved across five challenging real-world household tasks.

2 JoyLo: Joy-Con on Low-Cost Kinematic-Twin Arms

JoyLo is a low-cost whole-body teleoperation interface built around kinematic-twin arms and thumbstick control. It combines constrained, intuitive operation with bilateral haptic feedback for data collection.

  • JoyLo is implemented on a wheeled dual-arm manipulator with a 4-DoF torso, while its design principles are intended to adapt to similar mobile manipulators.
  • Kinematic-twin arms with thumbsticks provide a puppeteering-based approach intended to balance intuitiveness, ease of use, and manipulation precision.
  • Kinematic constraints prevent operators from generating infeasible or undeployable actions, supporting smooth and reliable demonstrations.
  • Bilateral teleoperation provides haptic feedback without extra force sensors, with proportional resistance when the robot experiences contact.
  • JoyLo uses 3D-printed links, low-cost Dynamixel motors, and Joy-Con controllers, totaling under $500.

3 WB-VIMA: Whole-Body VIsuoMotor Attention Policy

WB-VIMA learns coordinated whole-body actions by hierarchically decoding embodiment components and fusing visual and proprioceptive observations. Its training and deployment setup supports high-frequency control.

  • WB-VIMA is a transformer-based model trained on JoyLo data that autoregressively decodes whole-body actions and aggregates multimodal observations with self-attention.
  • A 0.17 rad knee movement can shift the end-effector by up to 0.14 m, motivating hierarchical coordination of mobile base, torso, arms, and grippers.
  • The model predicts the mobile base trajectory first, then conditions torso prediction on it, and finally predicts arm and gripper trajectories from the preceding outputs.
  • Using action-readout tokens with lightweight UNet action heads balances transformer expressivity with inference latency for high-frequency control.
  • Point clouds and proprioception are encoded into tokens and fused across current and past observations through causal self-attention.
  • Training minimizes mean-squared error between ground-truth and predicted noise across three action decoders, with deployment latency of 0.02 s.

4 Experiments

Experiments evaluate BRS on five real-world household tasks, compare WB-VIMA with baselines, ablate its components, and assess JoyLo for teleoperation and policy-data collection. WB-VIMA achieves strong task performance, while its coordinated action decoding, multimodal attention, and JoyLo interface each support reliable whole-body manipulation.

  • Experiment setup: The experiments evaluate five long-horizon household tasks using JoyLo-collected trajectories, sub-task and entire-task success rates, and baselines including DP3, RGB-DP, and ACT.Each task lasts 60–210 seconds and is segmented into sub-tasks for evaluation.
  • Task performance: 88% average sub-task success and 93% peak entire-task success demonstrate strong WB-VIMA performance across the evaluated tasks.Average entire-task success is 58%.
  • Baseline comparison: 13× and 21× higher end-to-end success than DP3 and RGB-DP, respectively, show WB-VIMA’s advantage over baseline policies.WB-VIMA also exceeds DP3 and RGB-DP in average sub-task performance by 1.6× and 3.4×, respectively.
  • Ablations: Removing autoregressive whole-body action decoding causes up to a 53% performance drop, while removing multimodal attention degrades performance across all evaluated tasks.The multimodal-attention ablation also produces four collisions because the model ignores visual inputs and overfits to proprioception.
  • User study: JoyLo achieves a 5× higher task success rate and 23% shorter median completion time than VR controllers while producing higher-quality policy-learning data.It also has the lowest singularity ratio and consistently replays successful trajectories.

5 Related Work

Related work covers household robotics, low-cost robot-learning hardware and interfaces, and whole-body manipulation methods. WB-VIMA is positioned as explicitly modeling action hierarchy, embodiment interdependencies, and task-relevant multimodal observations for high-DoF mobile manipulation.

  • Household robotics: Household robots must generalize across diverse, complex home environments, motivating benchmarks and learning-based robotic systems for everyday activities.Prior work commonly addresses either data collection or policy learning rather than both comprehensively.
  • Low-cost hardware: Low-cost robot-learning hardware includes robots, teleoperation interfaces, and wearable or portable data-collection devices, with JoyLo categorized as a cost-effective whole-body interface.The related work distinguishes these three hardware categories.
  • Whole-body manipulation: Whole-body manipulation methods use arms, torso, and base, spanning motion planning, reinforcement learning, behavior cloning, and large pretrained models.The surveyed methods cover both traditional planning and several learning paradigms.
  • WB-VIMA: WB-VIMA explicitly models action hierarchy and embodiment interdependencies through autoregressive whole-body decoding, while dynamically fusing multimodal observations through visuomotor attention.This targets coordinated policies on a high-DoF wheeled dual-arm robot with a torso.

6 Conclusion

BRS combines a cost-effective whole-body interface with a policy that models interdependent actions and embodiment hierarchy. The system demonstrates strong performance on real-world household tasks in natural, unstructured environments.

  • Conclusion: BRS targets diverse real-world household tasks through JoyLo for data collection and WB-VIMA for learning coordinated whole-body actions.The framework addresses both data and modeling challenges.
  • Conclusion: BRS demonstrates strong performance with unmodified objects in natural, unstructured environments, marking a step toward greater household-robot autonomy and reliability.The conclusion identifies bimanual coordination, stable navigation, and extensive end-effector reachability as essential capabilities.

7 Limitations

BRS has several limitations involving camera-view mismatch, accumulated long-horizon errors, imperfect point clouds, and robot-specific training data. The authors identify these as scope boundaries and propose directions including active perception, correction data, task planning, improved stereo, and multi-embodiment training.

  • Camera field of view: Camera-view mismatch requires operators to maintain workspace visibility while ensuring target objects remain visible to the robot’s cameras.Otherwise, collected data can become partially observable and complicate policy training.
  • Long-horizon execution: Long-horizon multi-stage tasks can accumulate errors across sub-tasks, reducing overall task success despite high sub-task success rates.The authors suggest human correction data or model-based task planning as possible mitigations.
  • Perception: Point-cloud quality can degrade under difficult lighting or reflective surfaces, and daytime training may not generalize well to nighttime environments.FoundationStereo is suggested as a future way to improve point-cloud quality.
  • Training-data scope: WB-VIMA is trained exclusively on R1 data, limiting current evidence for multi-embodiment transfer and scene-level generalization.The authors propose multi-embodiment, synthetic, human, and large-model data as future directions.

A.2 Hardware Specifications

The R1 hardware combines dual arms, a four-joint torso, an omnidirectional mobile base, and multi-camera sensing for whole-body household manipulation. Its sensing pipeline fuses RGB-D point clouds into a common robot-centered representation for policy input.

  • Torso: The torso uses four revolute joints for waist rotation, hip bending, and two knee-like motions.It measures 1223 mm in height when fully extended, excluding the head.
  • Mobile base: The omnidirectional base supports planar translation and yaw rotation through three steering motors and three wheel motors.It uses a 3-DoF velocity interface for forward, lateral, and rotational velocities.
  • Sensors: The sensor suite includes one head RGB-D camera, two wrist RGB-D cameras, and a tracking camera for visual odometry.RGB-D streams operate at 60 Hz, while odometry is reported at 200 Hz.
  • Perception pipeline: RGB-D point clouds are transformed using forward kinematics, fused in the robot base frame, then spatially cropped and downsampled.The resulting representation is a fused egocentric colored point cloud.
  • JoyLo hardware: JoyLo uses 3D-printed arm links, low-cost Dynamixel motors, and off-the-shelf Joy-Con controllers.The arms require 13 hours of printing and 317 g of PLA filament.

B.2 Controller Implementation

The controller maps JoyLo and robot state into coordinated commands for the torso, arms, grippers, and mobile base. WB-VIMA encodes point clouds and proprioception with attention, then predicts base, torso, and arm actions through hierarchical diffusion decoders.

  • Controller implementation: The R1 controller sends torso and arm joint targets plus gripper commands and mobile-base velocities through ROS at 100 Hz.Joint impedance controls the torso and arms, while velocity control operates the mobile base.
  • JoyLo control: JoyLo arm teleoperation uses current-based joint impedance with coupled motors for load-bearing joints.The controller sets proportional and derivative gains for the six arm joints.
  • Data collection: Data collection records RGB, depth, point clouds, joint states, odometry, and action commands at 10 Hz while the robot runs at 100 Hz.Joy-Con buttons control starting, pausing, saving, and discarding recordings.
  • Observation encoding: WB-VIMA represents colored point clouds and proprioception as separate observation tokens before multimodal attention.Proprioception includes base velocity, torso and arm positions, and gripper widths.
  • Multimodal attention: Causal self-attention fuses observation tokens across time, while passive action readout tokens attend only to earlier observations.The final action readout token is used for autoregressive whole-body decoding.
  • Action decoding: Three independent UNets sequentially predict mobile-base, torso, and arm actions, conditioning each upper-body stage on preceding predictions.The action horizon is Ta = 8, and encoded readout tokens avoid rerunning observation encoding during denoising.

C.4 Policy Training Details

The appendix specifies optimization, data splitting, diffusion schedules, and asynchronous deployment procedures for policy training. Evaluation uses separate training and validation data and fewer denoising steps during inference than training.

  • Optimization: Policies use AdamW with 90% of collected data for training and 10% for validation.Models are trained for equal steps and evaluated using the last checkpoint.
  • Diffusion schedules: Training uses a DDPM scheduler with 100 denoising steps, whereas evaluation and inference use DDIM with 16 steps.This reduces the denoising burden during deployment.
  • Hyperparameters: Training uses a 7 × 10^-4 learning rate, 0.1 weight decay, 1000 warm-up steps, and 300,000 cosine-decay steps.The minimum learning rate is 5 × 10^-6.
  • Deployment: Asynchronous inference runs continuously in the background and discards initial actions from newly predicted trajectories to compensate for latency.This design aims to prevent delays during execution.

D.1 Task Definition

BRS evaluates whole-body manipulation across five randomized household activities, each emphasizing a different capability and combining navigation, manipulation, and multi-stage execution. The task suite includes both long-horizon evaluations and simulated ablations of whole-body action decoding.

  • Task suite: Five household activities target stable navigation, end-effector reachability, or bimanual coordination as their most critical capability.The activities cover cleaning, trash disposal, shelving, and clothing manipulation in realistic rooms.
  • Task suite: Clean House After a Wild Party requires navigation between rooms, dishwasher interaction, bowl collection, and final placement.The study collects 138 demonstrations and randomizes robot starts, bowl instances and placements, and table distractors.
  • Task suite: Clean the Toilet combines sponge pickup, articulated toilet-cover manipulation, wiping, and flush-button pressing.This task emphasizes extensive end-effector reachability and uses 103 demonstrations with randomized starts and sponge placements.
  • Task suite: Take Trash Outside requires navigation, carrying a trash bag, opening a door, moving outdoors, and depositing the bag in a bin.Stable and accurate navigation is identified as most critical; 122 demonstrations use randomized starts and trash-bag placement.
  • Task suite: Put Items onto Shelves requires lifting a box and placing it on an appropriate level of a four-level shelf, while Lay Clothes Out requires coordinated wardrobe and jacket manipulation.The tasks collect 100 and 98 demonstrations, respectively, with randomized object and scene configurations.
  • Ablation setting: A simulated table-wiping task supplies 100,000 cuRobo-generated whole-body trajectories for ablations of autoregressive action decoding and related design choices.The mobile base and first two torso joints are locked to constrain the motion space.

D.4 User Study Details

The user study compares JoyLo with VR controllers and Apple Vision Pro in simulated household-task operation, using randomized interface exposure and post-session rollout annotation. The surrounding evaluation materials report task and sub-task success rates, safety violations, and ablation results across several activities.

  • Study design: Ten participants compare JoyLo, VR controllers, and Apple Vision Pro on the simulated Clean House After a Wild Party task.Participants use a Meta Quest 3 headset for equalized depth perception, and interface order is randomized to reduce bias.
  • Annotation and safety: Post-session rollouts are manually segmented and annotated for task and sub-task completion using a GUI.The study also identifies singular or near-singular configurations through Jacobian condition numbers for IK interfaces and excessive joint velocities for JoyLo.
  • Questionnaire: The questionnaire asks about prior data-collection experience, expected and experienced user-friendliness, and whether JoyLo arms and thumbsticks improved control.These questions separately probe usability expectations, physical arm holding, and torso or mobile-base control.
  • Policy evaluation: Task evaluation reports success rates as percentages, with parenthesized counts giving successful trials out of total trials.The materials include numerical tables for Clean House After a Wild Party, Clean the Toilet, Take Trash Outside, Put Items onto Shelf, and Lay Clothes Out.
  • Policy evaluation: Human teleoperation baselines are reported as 61% (100/164), 76% (96/127), 89% (93/104), and 50% (54/108) in the supplied evaluation rows.These values correspond to separate task tables and should not be interpreted as a single aggregate rate.
  • Ablations: Removing whole-body action denoising yields 40% (6/15), while removing multi-modal observation attention yields 13% (2/15) in one supplied result row.A separate supplied row reports 13% (2/15) without whole-body action denoising and 0% (0/15) without multi-modal observation attention.
Loading 2503.05652v2…