Source-linked AI summary
BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, Li Fei-Fei
TL;DR
BEHAVIOR addresses the need for embodied-AI benchmarks covering realistic, diverse, and complex everyday household activities, alongside challenges in definition, simulation, and evaluation. It combines predicate-logic activity definitions, simulator requirements and an iGibson implementation, and human-relative metrics with VR demonstrations. The resulting activities challenge existing embodied-AI solutions, including state-of-the-art reinforcement-learning baselines.
Problem
Existing embodied-AI benchmarks cover only a fraction of everyday challenges, while realistic activities require solutions to definition, simulation realization, and evaluation.
Method
BEHAVIOR uses BDDL to define symbolic initial and goal conditions, specifies simulator requirements, implements the benchmark in iGibson 2.0, and evaluates agents with human-relative metrics.
Results
BEHAVIOR’s 100 realistic, diverse, and complex activities challenge existing solutions, as shown by experiments with two state-of-the-art reinforcement-learning algorithms.
Takeaways & Limitations
BEHAVIOR provides a benchmark intended to guide embodied-AI development through well-defined household challenges and comparable human-grounded evaluation.
Takeaways & Limitations
BEHAVIOR requires simulators to support object-centric state, physical interaction, non-kinematic properties, instance generation, and goal evaluation.
Abstract
from arXiv · showhide
We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation. These activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world. Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation. BEHAVIOR addresses these with three innovations. First, we propose an object-centric, predicate logic-based description language for expressing an activity's initial and goal conditions, enabling generation of diverse instances for any activity. Second, we identify the simulator-agnostic features required by an underlying environment to support BEHAVIOR, and demonstrate its realization in one such simulator. Third, we introduce a set of metrics to measure task progress and efficiency, absolute and relative to human demonstrators. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth. Our experiments demonstrate that even state of the art embodied AI solutions struggle with the level of realism, diversity, and complexity imposed by the activities in our benchmark. We make BEHAVIOR publicly available at behavior.stanford.edu to facilitate and calibrate the development of new embodied AI solutions.
1 Introduction
Existing embodied-AI benchmarks cover only a fraction of everyday human challenges, motivating BEHAVIOR’s realistic, diverse, and complex household activities. BEHAVIOR addresses benchmark definition, simulation realization, and evaluation with a symbolic language, simulator implementation, and human-relative metrics.
- Existing embodied-AI benchmarks represent only a fraction of the realistic, diverse, and complex activities humans perform daily.
- The benchmark must overcome three challenges: defining meaningful activities, realistically supporting them in simulation, and evaluating success objectively.
- BEHAVIOR defines 100 activities with BDDL, enables potentially infinite initial states and solutions, and provides a functional iGibson 2.0 implementation with 1217 object models across 391 categories.
- BEHAVIOR evaluates success and efficiency using metrics relative to human performance and includes 500 successful VR demonstrations as evaluation ground truth.
- Experiments with two state-of-the-art reinforcement-learning algorithms show that BEHAVIOR’s realistic, diverse, and complex activities remain extremely challenging for existing solutions.
2 Related Work
Embodied-AI benchmarks benefit from simulation’s speed, safety, reproducibility, accessibility, and fairness, but existing activities remain limited in realism, diversity, or complexity. BEHAVIOR is positioned as a broader benchmark for realistic household work and human-relative evaluation.
- Passive image and video datasets are poorly suited to embodied AI because agents must control actions and understand their consequences.
- Simulation provides faster, safer learning and more reproducible, accessible, and fair evaluation than real-world testing.
- BEHAVIOR combines realistic simulation, broad activity and scene diversity, complex household tasks, infinite scene-agnostic instances, and human VR demonstrations.
- Existing benchmarks commonly trade off visual realism, physical realism, activity diversity, or task complexity, often focusing on narrow rearrangement or manipulation settings.
3 BEHAVIOR: Benchmarking Realistic, Diverse, Complex Activities
BEHAVIOR grounds household activities in population time-use data and realistic home simulation, while varying tasks, objects, scenes, configurations, and state changes. Its activities also match real chores in length, object count, and capability demands.
- Realism in BEHAVIOR Activities: BEHAVIOR derives activities from the American Time Use Survey, whose distribution approximates the full space of simulatable household activities.
- Realism in BEHAVIOR Activities: Realistic home reconstructions, Bullet physics, high-quality sensors, 15 interactive homes, and 1217 object models support ecological simulation.
- Diversity in BEHAVIOR Activities: BEHAVIOR includes 100 activities spanning state changes such as moving objects, soaking materials, cleaning surfaces, and heating or freezing food.
- Diversity in BEHAVIOR Activities: Each activity has two example definitions, while BDDL samples potentially infinite instances varying scenes, object models, and configurations.
- Complexity in BEHAVIOR Activities: BEHAVIOR activities involve an average of 10.5 objects, 2–8 capabilities, and sometimes several thousand steps, creating long-horizon household challenges.
4 Defining Realistic, Diverse, and Complex Household Activities with BDDL
BEHAVIOR represents activities as acceptable initial and goal-state sets, then uses BDDL to express them symbolically rather than as scene- or pose-specific physical configurations. This enables diverse instances and semantically valid solutions.
- BEHAVIOR models an activity as sets of possible initial states and acceptable goal states within a partially observable physical-interaction environment.
- BDDL predicates represent object kinematics, temperature, wetness, and other physical or functional properties and relationships.
- BDDL is a predicate-logic language whose meaningful predicates map simulated physical states into symbolic representations over object categories.
- An initial BDDL condition maps to infinitely many physical states, while a goal condition accepts all semantically satisfactory solutions.
- BDDL’s declarative state-based definitions support systematic generation of unlimited activity definitions and practically infinite unique instantiations.
5 Instantiating BEHAVIOR in a Realistic Physics Simulator
BEHAVIOR requires a simulator that represents objects and their changing physical and non-kinematic properties, supports realistic sensing and interaction, and exposes agent action and observation spaces. The benchmark is implemented in iGibson 2.0 with humanoid and Fetch embodiments.
- Simulator requirements: BEHAVIOR simulators must maintain object-centric identities and states, simulate physical motion and forces, generate sensor signals, and model properties such as temperature, wetness, and cleanliness.They must also provide functionality for generating activity instances and solutions, as specified by the benchmark’s functional requirements.
- Embodiments and interfaces: The simulator must expose the underlying POMDP’s action space A and observation space O to embodied AI agents.BEHAVIOR supports a 24-DoF bimanual humanoid avatar and a 12/13-DoF Fetch robot, both capable of navigation, grasping, and hand interaction.
- Simulator-supported evaluation: The evaluation visualization separates success-score completion of activity-relevant steps from efficiency signals capturing exploration, manipulation, and scene disruption.The figure uses Q for success, Dk for kinematic disarrangement, and Lright and Lleft for hand interaction displacement.
- iGibson 2.0 realization: iGibson 2.0 provides BEHAVIOR’s required object-centric states, heat and water sources, dust and stain particles, and appearance changes tied to extended object states.Its implemented embodiments receive proprioception and realistic RGB, depth, LiDAR, normals, flow, and semantic and instance-segmentation signals.
6 Evaluation Metrics: Success, Efficiency and Human-Centric Metric
BEHAVIOR evaluates whether activities are completed and how efficiently agents perform them, combining a logical success score with complementary efficiency metrics. Efficiency can be reported in absolute units or normalized against successful human VR demonstrations.
- Metric framework: BEHAVIOR uses success as its primary metric and secondary metrics to characterize efficiency.The framework extends metrics proposed for Rearrangement while distinguishing task accomplishment from execution quality.
- Success score: The success score Q is the maximum fraction of satisfied goal literals across conjunctions in an activity’s flattened goal condition.An activity is complete when all literals in at least one conjunction are satisfied, yielding Q = 1 (100%).
- Efficiency metrics: BEHAVIOR’s efficiency metrics include simulated time, kinematic and logical disarrangement, body distance navigated, and left- and right-hand displacement.These metrics respectively characterize time, scene displacement, logical-state changes, navigation, and object-contact interaction.
- Human-centric efficiency: Absolute efficiency metrics support scene- and activity-specific comparisons, while human-centric efficiency expresses each secondary metric as a fraction of maximum human performance.The normalization uses a human demonstration for the same activity instance in VR.
- Human ground truth: 500 successful VR demonstrations totaling 758.5 minutes provide human performance data for efficiency analysis and comparison.The dataset records manipulation, navigation, and gaze tracking while humans control the same embodiment used by AI agents.
7 Evaluating Reinforcement Learning in BEHAVIOR
BEHAVIOR tests embodied AI under realistic sensing and actuation, diverse instances, and long-horizon household activities. State-of-the-art reinforcement-learning agents perform poorly on the fully realistic benchmark, with complexity, realism, and diversity each contributing to the challenge.
- Experimental setup: The evaluation trains SAC with low-level continuous actions and PPO with action primitives on the 12 simplest activities, using RGB, depth, and proprioception.Reward is provided by the staggered success score Q, and the experiments aim to expose challenges from realism, diversity, and complexity.
- Overall results: Under the fully realistic setup, SAC achieves Q = 0 on all 12 evaluated activities and fulfills no predicate in any goal condition.This result uses realistic robot actions and onboard sensing even for the benchmark’s simpler activities.
- Complexity: With temporally extended action primitives, PPO still fails on all but one activity, achieving Q = 0.13 on bringingInWood.Oracle-driven SAC experiments show that agents commonly fail even when initialized near a goal, and success becomes zero for all activities at 10 s from the goal.
- Realism: Full observability does not overcome the complexity of original continuous-control policies, while primitive-based policies achieve partial success in only five activities.Removing physics difficulties produces significant success only for a handful of activities, indicating that realistic sensing and actuation remain important challenges.
- Diversity: Performance degrades quickly as diversity increases even in single-predicate activities, indicating that current RL algorithms struggle to generalize across BEHAVIOR’s scenes, objects, and initial states.Table 3 isolates diversity while simplifying activity complexity.
8 Conclusion and Future Work
BEHAVIOR is a household-activity benchmark combining realistic, diverse, complex tasks with symbolic representations, simulation, and human-centric evaluation. Its activities introduce state changes beyond object repositioning, and experiments with two reinforcement-learning baselines expose substantial challenges.
- BEHAVIOR presents 100 realistic, diverse, and complex household activities for embodied AI.
- The benchmark combines a logic-symbolic representation, functional simulation, and metrics based on human VR performance.
- Its tasks require agents to cause state changes such as cleaning surfaces and changing object temperatures, beyond repositioning objects.
- Experiments with two state-of-the-art reinforcement-learning baselines reveal challenges from BEHAVIOR’s realism, diversity, and complexity.
- BEHAVIOR is intended to support open, fair access to embodied-AI research tools and future benchmark development.
A.1 Visualizing 100 BEHAVIOR Activities
Figure A.1 visualizes BEHAVIOR’s 100 household activities by pairing virtual-reality execution frames with corresponding real-life video frames. The activities span everyday chores across several household domains.
- Figure A.1 shows paired views of BEHAVIOR activities from virtual reality and corresponding real-life YouTube videos.
- The visualized activities are selected from the American Time Use Survey and correspond to simulatable household chores.
- The activity set covers cleaning, maintenance, preparation for social activities, and household management.
- Examples include putting away cleaned dishes and re-shelving library books.
- The figure continues with additional BEHAVIOR activities in a second panel.
A.2 Additional Comparison between BEHAVIOR and other Embodied AI Benchmarks
The appendix provides an expanded comparison of BEHAVIOR with other embodied-AI benchmarks. The listed comparison includes rearrangement and interactive-environment benchmarks.
- The comparison includes AI2THOR Visual Room Rearrangement.
- The comparison includes the Interactive Gibson Benchmark.
- The comparison includes Rearrangement T5 in Habitat.
- Table A.1 is an expanded comparison between BEHAVIOR and existing embodied-AI benchmarks.
A.3 Defining BEHAVIOR Activities
This section explains how BEHAVIOR’s 100 activities are defined, covering activity selection, BDDL, crowdsourced conditions, and concrete examples.
- The section explains how BEHAVIOR selects its 100 activities from the American Time Use Survey.
- It describes BDDL, the predicate-logic language used to define activities.
- It documents crowdsourcing initial and goal conditions for the activities.
- It provides real BDDL examples of the generated activity definitions.
A.3.1 Selection of 100 Activities for BEHAVIOR
BEHAVIOR selects 100 household activities to span semantic domains and diverse environmental state changes, while capturing activities with varying complexity and horizons.
- The 100 activities span semantic areas including cleaning, food preparation, maintenance, and repairing.
- Activity selection targets diversity in the environmental properties that agents must manipulate.
- BEHAVIOR covers a realistic distribution of activities relative to categories in the American Time Use Survey.
- Activities require varied state changes, including transport or rearrangement, visual changes, and other environmental transformations.
- Goal-condition volume ranges from one to 25 literals, and even one-literal activities can require long horizons such as cleaning large surfaces.
A.3.2 BDDL– BEHAVIOR Domain Definition Language
BDDL provides a predicate-logic representation connecting simulated objects and states to general activity conditions, while BEHAVIOR combines this representation with embodied simulation, demonstrations, and evaluation infrastructure.
- BDDL maps simulated states to semantic symbols and defines activities through initial and goal conditions.
- Crowdsourcing with simulator feedback produced two alternative definitions per activity, each feasible in at least three simulated scenes.
- The simulator provides fully simulated and partially simulated variants of six temporally extended action primitives for agent development and complexity studies.
- 36-59 steps per second are achieved under the highest-fidelity physics setup, while 47-71 steps per second are achieved with a larger simulated timestep.
- The benchmark curates 1217 object models across 391 categories and includes rich VR demonstrations of long-horizon, multi-step human activities.
- Activity volume strongly correlates with total activity time, while room occupancy and hand use vary with activity type.
- BEHAVIOR is simulation-based, enabling reproducible evaluation and accessible robot learning while leaving a gap from equivalent real-world activities.