Source-linked AI summary
Rearrangement: A Challenge for Embodied AI
Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, Hao Su
TL;DR
The paper proposes rearrangement as a canonical Embodied AI task: bringing a physical environment into a specified goal state. It defines goal specifications, evaluation metrics, scenario dimensions, and simulation testbeds to support research and deployment-oriented development.
Problem
Embodied AI lacks a task definition that can align and accelerate research across practical interaction scenarios.
Method
The paper defines rearrangement, supports multiple goal-state specifications, proposes predicate-based evaluation metrics, and presents testbeds across four simulation environments.
Results
The framework characterizes rearrangement scenarios, provides metrics for benchmarking performance, and establishes experimental simulation testbeds for research and exploration.
Takeaways & Limitations
Rearrangement offers a unified framework spanning tasks such as setting tables, cleaning bedrooms, loading dishwashers, and manipulating objects in complex scenes.
Takeaways & Limitations
Simulation-based development may produce techniques whose performance does not transfer to real-world domains.
Abstract
from arXiv · showhide
We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. A standard task can focus the development of new techniques and serve as a source of trained models that can be transferred to other settings. In the rearrangement task, the goal is to bring a given physical environment into a specified state. The goal state can be specified by object poses, by images, by a description in language, or by letting the agent experience the environment in the goal state. We characterize rearrangement scenarios along different axes and describe metrics for benchmarking rearrangement performance. To facilitate research and exploration, we present experimental testbeds of rearrangement scenarios in four different simulation environments. We anticipate that other datasets will be released and new simulation platforms will be built to support training of rearrangement agents and their deployment on physical systems.
1. Introduction
The paper proposes rearrangement as a canonical Embodied AI task: agents bring physical environments into specified goal states while integrating perception, manipulation, planning, and navigation. It defines a flexible evaluation protocol and releases scenarios across four simulation environments.
- Task definition: Rearrangement asks an agent to bring a physical environment into a specified goal state across practical settings such as homes and fulfillment centers.The task covers moving objects and changing object states, and can involve stationary manipulators or mobile systems.
- Embodied abilities: Rearrangement integrates recognizing object states, inferring differences between current and goal states, manipulating in clutter, estimating forces, planning actions, and navigating complex environments.The framework therefore combines component abilities into a broader Embodied AI research and evaluation roadmap.
- Task definition: Goal states can be specified geometrically, visually, linguistically, formally, or through an embodied demonstration.The evaluation protocol scores object and object-set compliance with predicates and assigns each episode a score between 0 and 1.
- Research guidelines: The proposed framework emphasizes realistic onboard perception and strong generalization to unseen objects and environments without privileged information.Agents should access previously unseen scene content and layout solely through realistic sensing.
- Research infrastructure: The authors release rearrangement scenario suites in THOR, RLBench, SAPIEN, and Habitat to support near-term research and development.They anticipate additional datasets and simulation platforms for training and deployment on physical systems.
2. Background
The paper situates rearrangement within task and motion planning but shifts emphasis toward standardized end-to-end evaluation from raw sensory input to actuation. Its framework is intended to compare diverse approaches across broad Embodied AI capabilities on the same footing.
- Planning approaches: Modular rearrangement systems can decompose perception and planning, with perception producing a world representation from which planning generates actions.This connects rearrangement to established task-planning and motion-planning traditions.
- Planning approaches: Task and motion planning research searches for action sequences transforming initial object configurations into goals, often under different transformation assumptions.Existing formulations include operations such as pushing or grasping.
- Planning approaches: Classic rearrangement planners typically assume object shapes and poses are already available rather than processing raw sensory input.This assumption may be partial and uncertain, but it remains distinct from end-to-end perception and action.
- Evaluation framework: The paper's point of departure is standardized end-to-end evaluation, which measures a complete system from raw sensory input through actuation while remaining agnostic to internal decomposition.This places real perception and action, including physical sensors and actuators, within the evaluation scope.
- Evaluation framework: The proposed framework aims to cover broad capabilities and compare varied algorithmic approaches on the same footing rather than constrain evaluation to narrow tasks.The authors position this breadth as compatible with both modular and differing design philosophies.
3. Rearrangement
Rearrangement formalizes Embodied AI as transforming an initial environment state into an acceptable goal state using sensory observations. The framework supports multiple goal specifications, embodiment fidelities, and evaluation-oriented abstractions.
- Task definition: The formalization uses a POMDP-style state, observation, action, transition, and observation framework without prescribing a solution approach.World states are factorized into Cartesian products of rigid-body pose spaces, while achievable configurations remain constrained by joints and physical conditions.
- Task definition: Rearrangement models an agent transforming an initial state s0 into an acceptable goal state s∗∈S∗ using observations o∈O.The goal may be non-unique, with S∗ representing the set of acceptable goal states.
- Goal specification: Goal specifications can encode geometric transformations, images, language, prior experience, or predicate sets, each defining acceptable goal states differently.Image and language goals may be underspecified, whereas predicate goals can be precisely specified and evaluated; experience goals let agents construct their own representation.
- Goal specification: PredicateGoal provides a common substrate for interpreting other goal types, with predicates evaluated from state through programs, classifiers, or logical checks.Predicates operate on symbols grounded to objects or object sets and can express relations such as being on or inside another object.
- Embodiment: A kinematic articulated arm with abstracted grasping balances robotic-body control and geometric constraints while omitting inertia and contact-force complexities.Objects sufficiently close to the end effector are virtually grasped after a pick command, making the setting suitable for simplified manipulation and perception-focused methods.
- Embodiment: Full physical simulation offers the highest interaction fidelity but requires closed-loop perception and control, physical grasping, and simulated contact forces.SAPIEN and RLBench use this embodiment, which also introduces actuation noise and additional end-effector manipulation challenges.
4. Evaluation
The paper proposes task completion as a unified primary metric for rearrangement, supplemented by efficiency measures and predicate-based evaluation that can accommodate varied goals and object relationships.
- Primary Metric: Task completion is the percentage of binary target-state tests passed for relevant objects without harming other scene elements.
- Primary Metric: Completion percentage supports progression because increasingly complex tasks involve more objects and produce a more continuous score.
- Primary Metric: Individual object errors can use translation, rotation, or 3D IoU, with task-specific thresholds determining whether each object passes.
- Primary Metric: General scene predicates extend evaluation beyond independent object poses to relative configurations and other binary task conditions.
- Evaluation Programs: Automatic completion evaluation compares final and target configurations, but accurate real-world evaluation remains technically difficult under occlusion and contact.
- Secondary Metrics: Secondary metrics should report agent and computational efficiency because completion alone may not establish real-world utility.
5. Experimental Testbeds
The paper contributes rearrangement testbeds spanning environment complexity and navigation or manipulation demands, from fixed-base sweeping and tabletop tasks to room-scale cleanup.
- Suite Overview: The testbed suite is ordered roughly by environment scale and navigational requirements while varying manipulation capabilities.
- T1: Bimanual Sweeping: T1 uses two fixed-base arms with spatulas to sweep simulated trash into a bin in SAPIEN.
- T2: Table Organization: T2 uses a fixed-base UR5e arm and parallel-jaw gripper to rearrange randomly configured tabletop objects in SAPIEN.
- T3: Storing Groceries: T3 requires a Franka Panda arm to place scattered groceries into a constrained shelf volume, creating difficulty as available space decreases.
- T4: Room Rearrangement: T4 requires room-scale object rearrangement and state changes such as opening or closing cabinets, with difficulty depending on changed objects and action complexity.
- T5: House Cleanup: T5 uses a mobile agent with camera and depth sensors to relocate household objects across rooms in Habitat, requiring longer-range navigation.
6. Why Rearrangement?
Rearrangement provides a unified testbed for evaluating perception, navigation, manipulation, memory, planning, communication, and coordination. Its complexity can be adjusted to emphasize individual capabilities or combinations of capabilities.
- Agent capabilities: Rearrangement scenarios exercise a broad spectrum of agent abilities, allowing researchers to evaluate different subsets of those capabilities.The task can be configured to stress specific abilities or their combinations.
- Perception: Perception supports object detection, rigid-object state estimation, tracking, and localization, with difficulty shaped by sensors and environment complexity.These abilities help agents identify movable objects and judge states relevant to action.
- Navigation and manipulation: Navigation and manipulation extend rearrangement beyond static recognition by requiring locomotion, grasping, and object control in the environment.Navigation matters in larger environments, while manipulation complexity can range from abstract interfaces to low-level control.
- Memory and planning: Memory and planning are required because rearrangement involves sequences of manipulations or navigations and actions with pre-conditions, post-conditions, and dependencies.For example, setting a table requires ordered subtasks and hierarchical decision making.
- Communication: Language grounding connects perceptions and actions to object relations, instructions, final-state descriptions, and higher-level goals.The task can involve spatial relations, action distinctions, and concepts such as setting or clearing a table.
- Multi-agent coordination: Rearrangement can also study communication and emergent behavior in coordination between multiple agents.This extends the platform from individual capabilities to collaborative interaction.
7. Discussion
The discussion positions rearrangement as a more physically interactive challenge than navigation benchmarks while identifying simulation, task-complexity, and scope boundaries. It also outlines extensions toward richer processes, physical robots, deformable objects, object-state transformations, and multiple agents.
- Benchmark scope: Rearrangement requires substantially more physical interaction than prior navigation benchmarks and can encompass PointGoal navigation in its mobile variant.The resulting agents are expected to need more sophisticated world representations, sequential decision-making, perception, grasping, and physics simulation.
- Complex tasks and processes: Current examples largely specify final states directly, leaving complex multi-subtask processes and subgoal ordering as future work.Cleaning rooms and putting away groceries are examples requiring sequential subgoals and dependency resolution.
- Physical robotic systems: Simulation offers more data, precise evaluation, and efficient experiments, but performance may not transfer to real-world robotic systems.The authors propose developing physical robot variants in parallel, while noting that robust real-world evaluation remains challenging.
- Simulation fidelity: Simulation fidelity in physics and sensing influences task complexity and the effort required to transfer learned models to reality.Actuation and sensing noise can be simulated, while efficient accurate simulation of deformable phenomena remains challenging.
- Future extensions: Deformable-object rearrangement is excluded from the initial formulation because deformation simulation, goal specification, and evaluation remain challenging.Clothing, towels, curtains, and bed sheets are identified as practical future extensions.
- Future extensions: Object-state transformations are outside the presented formulation because they require deeper world knowledge, causal reasoning, complex actions, and separate evaluation procedures.The predicate-based evaluation is described as a possible basis for future extensions.
- Future extensions: The report focuses on single-agent rearrangement, while cooperative and adversarial multi-agent scenarios are proposed as extensions.Cooperation could improve efficiency or enable subtasks such as moving a heavy sofa.
- Future extensions: Human interaction is identified as a future direction, including real-time interaction, dialog-based interfaces, and active learning from human input.These directions target systems intended to assist users with everyday tasks.
A. Experimental Testbed Details
The authors release rearrangement scenarios in existing simulators to support near-term research and provide a common basis for task specifications.
- Released testbeds: Rearrangement scenarios are released within AI2-THOR, Habitat, RL-Bench, and SAPIEN, while future work can use any simulator supporting the task.A summary of the task specifications is provided in Table 1.
A.1. Bimanual Sweeping in SAPIEN
The SAPIEN testbed models two robot arms collaboratively sweeping or moving objects into a target bin under realistic sensing and control interfaces. Performance is measured by placement success and efficiency within a fixed time budget.
- Task setup: The bimanual sweeping scenario uses two simulated robot arms to move boxes or cubes into a target bin efficiently.The setup includes two arms, ten boxes, and a randomly configured bin on a table.
- Episodes: Episodes terminate at a maximum time limit or when the agent returns False, including when it believes the task is accomplished or stuck.The environment does not automatically terminate when all boxes are correctly placed.
- Embodiment: The SAPIEN embodiment uses fully simulated Panda 7-axis arms with spade end effectors and physics-based dynamics.The robot arms replace their usual end effectors with spades.
- Sensors: Observations include RGBD views from four fixed cameras, optional ground-truth segmentation, and robot joint positions and velocities.The camera viewpoints are front, left, right, and top.
- Action space: Agents can use joint position and velocity control through PD controllers or direct joint torque control.The environment also exposes physical quantities such as inertia matrices, kinematic Jacobians, and dynamics.
- Metrics: Success rate measures the fraction of correctly placed boxes, while efficiency measures correctly placed boxes per minute.Under a fixed time budget, efficiency equals the total number of correctly placed boxes.
A.2. Cloud Robot Table Organization Challenge (SAPIEN & Real robots)
The Cloud Robot Table Organization Challenge evaluates rearrangement across simulation and real-robot stages, with increasing scene, object, and relational complexity. Performance is measured primarily by how many target objects are correctly organized.
- Stages: The challenge contains separate simulation and real-robot stages, each with a trial period followed by a contest.Participants can first familiarize themselves with the working environment before competing.
- Scenario design: Five scene levels increase difficulty through more objects, complex geometry, disturbing objects, and hierarchical relative-position specifications.The levels range from five simple objects without heaps to scenes with ten complex objects and ten disturbing objects.
- Scenario design: Object difficulty ranges from known objects with precise models to undisclosed novel objects and novel instances represented only by imprecise category-level models.Novel and imprecisely modeled objects are reserved for selected mid- to high-difficulty contest tasks.
- Robot interface: The setup provides RGBD cameras, robot-arm state observations, and ROS Control interfaces for commanding joint trajectories and other controllers.The real-robot hardware includes stationary and wrist-mounted cameras, a UR5e manipulator, and a parallel-jaw gripper.
- Evaluation: Task completion is the primary metric, computed from the number of target objects whose pose errors fall within object-specific thresholds.The episode aggregates correctly rearranged objects, while secondary metrics provide additional information.
A.3. Storing Groceries in RLBench
The RLBench grocery-storage testbed asks a simulated robot arm to place varied grocery objects into a constrained shelf space. It combines physical simulation, realistic sensing, automated task variation, and pose-based evaluation.
- Testbed: RLBench offers over 100 realistic single-arm robot tasks and automatically generates task variations with corresponding state-informed demonstrations.These demonstrations can seed reinforcement-learning algorithms.
- Task: The task combines varied object shapes in constrained shelf space, requiring high-level planning and precise perception.It is presented as a representative rearrangement task among several RLBench tasks involving object placement and organization.
- Task: The selected scenario requires placing seven randomly positioned grocery objects onto a box shelf in front of the robot arm.The grocery objects are accurately modeled from the YCB dataset.
- Embodiment: The simulated embodiment is a physically simulated Franka Panda arm with a two-finger gripper, supporting joint-level and end-effector control modes.The simulation uses Bullet physics within CoppeliaSim.
- Evaluation: Evaluation tests whether each object’s final translational pose lies within the shelf volume using simulated proximity sensors.The state tests are implemented in RLBench’s specification language.
A.4. Room rearrangement in AI2-THOR
The AI2-THOR room-rearrangement testbed asks an agent to restore selected object locations and open/close states from an initial household scene. It evaluates object-level predicate satisfaction across varied rooms and scenarios.
- Task: The agent must recover an initial room configuration after selected objects are moved or their open/close states are changed.The agent can navigate the scene in its initial configuration and collect data before acting.
- Dataset: The dataset uses 120 rooms across bathrooms, bedrooms, kitchens, and living rooms, split into 80 training, 20 validation, and 20 test scenes.The test scenes remain interactable in their initial configuration without object-position metadata.
- Embodiment and sensing: The agent uses RGB, depth, and haptic sensors, while its virtual arm can manipulate objects within the camera frustum and a default 1.5m interaction distance.The haptic signal indicates contact with an object and returns virtual-arm length.
- Evaluation: Success requires object bounding-box IoU above 50% and open/close state error within 20%, while changing unaffected objects makes the episode unsuccessful.The task metric averages the percentage of satisfied predicates for each scenario.
A.5. House Cleanup in Habitat
The Habitat house-cleanup testbed requires a virtual Locobot to find and reposition scattered objects in photorealistic indoor scenes. It emphasizes object-level completion and evaluates efficiency against shortest-path behavior.
- Scenes: The testbed uses 55 photorealistic Gibson scans, split into 35 training, 10 validation, and 10 testing scenes.Scanned object meshes are inserted into empty apartments and houses to support controlled scenario generation.
- Task: Each episode requires the agent to rearrange 25 objects according to specified initial and desired center-of-mass positions.The episode also specifies the scene, agent spawn pose, and each object’s initial location, rotation, and type.
- Embodiment and actions: The Locobot uses an RGB-D camera, GPS+Compass sensing, and navigation actions combined with interactive grab/release actions.The magic pointer selects a visible object under a fixed viewport crosshair within a distance threshold, and the agent carries one object at a time.
- Evaluation: Task completion is the percentage of goal objects placed within 1m of their desired goal locations.Distance is measured between the desired and final object center of mass.
- Evaluation: Episode-level success requires every specified object to be correctly placed, while SPL measures successful path efficiency relative to the shortest path.The shortest path is computed by formulating rearrangement as an extension of the traveling-salesman problem.