PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied Scene Rearrangement Planning

Canzhi Chen, Zan Wang, Siqi Zhu, Qi Wu, Yixuan Li, Wei Liang

arXiv:2608.27371v1cs.RO

TL;DR

Existing rearrangement research does not fully capture furniture-scale planning in complex 3D scenes under partial observability. The paper introduces ESRP and ESRP-Bench with four baseline paradigms, and finds that current methods remain weak, especially in complex scenarios.

  • Problem

    Prior rearrangement settings do not fully address furniture-scale 3D planning with egocentric observations, global-state restrictions, and long-horizon inter-object dependencies.

  • Method

    The paper introduces ESRP, ESRP-Bench, three evaluation metrics, and four baselines spanning planning, vision-language modeling, imitation learning, and reinforcement learning.

  • Results

    Current methods struggle on ESRP: the strongest baseline reaches a 30.20% success rate and falls to nearly 0% in complex scenarios.

  • Takeaways & Limitations

    ESRP exposes persistent challenges in scene understanding and long-horizon task planning for embodied agents.

  • Takeaways & Limitations

    The study focuses on room-level furniture rearrangement with egocentric RGB observations and lacks real-robot results because furniture-scale manipulation remains a hardware bottleneck.

Abstract

from arXiv · show

This paper introduces Embodied Scene Rearrangement Planning (ESRP), a novel task requiring embodied agents to rearrange furniture in 3D scenes to match a target configuration using only egocentric observations and a top-down target layout. Unlike prior rearrangement tasks, ESRP precludes global state access and introduces mutual object occlusions, reflecting the practical constraints of real-world robotic deployment. These factors make aligning partial egocentric observations with the global target layout particularly challenging for long-horizon planning. To facilitate research, we present ESRP-Bench, a comprehensive benchmark built on OmniGibson featuring over 5,400 scene pairs and 8,200 objects. We define three multi-level metrics to evaluate rearrangement quality and provide four baselines: a hierarchical task-and-motion planning method, a vision-language-model-based method, and two learning-based approaches (IL and RL). Experimental results demonstrate that current methods struggle to complete the task efficiently, highlighting ESRP as a challenging frontier for embodied agents in scene understanding and long-horizon task planning. This work serves as a stepping stone toward deploying intelligent agents in real-world scenarios. Project page: https://pie-lab.cn/ESRP/.

I. INTRODUCTION

ESRP frames furniture rearrangement as an embodied 3D planning task under partial observability, long-horizon dependencies, and dynamic scene changes. ESRP-Bench and four baseline paradigms support systematic evaluation, while initial results show substantial difficulty.

  • ESRP requires agents to rearrange furniture from an initial 3D layout to a target configuration using egocentric observations and a top-down target image.
  • Partial observability forces policies to align first-person views with the target layout and infer spatial information from local perception.
  • Furniture manipulation creates long-horizon planning demands involving action sequencing, inter-object dependencies, physical deadlocks, and mutual occlusions.
  • ESRP-Bench contains over 5,400 furnished indoor scenes, more than 8,200 movable items, three multi-level metrics, and expert demonstrations.
  • Four baseline policies span imitation learning, reinforcement learning, vision-language modeling, and hierarchical task-and-motion planning.
  • 30.20% is the strongest baseline success rate, falling to nearly 0% in complex scenarios.

II. RELATED WORK

Prior rearrangement research commonly targets small tabletop objects or larger objects in simplified 2D settings. These abstractions leave room-level 3D spatial relationships and long-horizon planning insufficiently addressed.

  • Tabletop rearrangement methods reposition small objects in confined workspaces but do not address room-scale navigation and spatial reasoning.
  • Scene-level rearrangement methods for larger objects often simplify environments to 2D planes, overlooking intricate 3D spatial relationships.
  • Standard task-and-motion planning combines high-level task planning with low-level motion planning for executable long-horizon actions.

III. ESRP-BENCH

ESRP-Bench defines unseen 3D rearrangement from egocentric observations and a top-down target image, with explicit action and success criteria. Its dataset spans diverse rooms, objects, and difficulty levels.

  • Task Definition: ESRP requires agents to perceive an unseen scene, plan rearrangement, and execute actions that reposition every furniture item to its target.
  • Dataset: Scene clutter is measured by summed L2 displacement between initial and target object positions, with larger values indicating more challenging scenes.
  • Task Definition: The action space contains four navigation primitives—forward, backward, left, and right—and two manipulation primitives: fetch and release.
  • Task Definition: Success requires all rearrangeable objects to reach their target configurations, with correctness defined by a horizontal-plane bounding-box IoU above 0.3.
  • Dataset: The benchmark includes 5,495 scenes across 24 room types, with 1–6 rearrangeable objects per scene and Easy, Medium, and Hard difficulty levels.
  • Dataset: The dataset contains 8,213 movable objects, led by Cabinet/Shelf/Desk items (3,718) and Chairs (1,721).

C. Metrics

The benchmark evaluates rearrangement across episodes using three complementary metrics: complete-episode success, object-level placement, and remaining distance to target positions.

  • Success Rate: Success Rate (SR) measures the proportion of episodes in which every object reaches its goal.Results are aggregated as mean performance across N independent episodes.
  • Object Success Rate: Object Success Rate (OSR) measures the proportion of rearrangeable objects correctly placed in their target positions.It captures partial rearrangement effectiveness even when an entire episode is unsuccessful.
  • Remaining Distance Ratio: Remaining Distance Ratio (RDR) evaluates how close the final arrangement is to the target configuration using summed Euclidean distances.The ratio compares distances at the episode’s start and end.
  • Remaining Distance Ratio: Lower RDR values indicate better alignment, with values approaching zero representing perfect placement of all furniture items.This metric distinguishes arrangements that make progress toward goals without fully succeeding.

D. Simulation

ESRP is implemented in OmniGibson with a Fetch-derived embodiment and collision-aware physics, while low-level grasping is abstracted to emphasize high-level planning.

  • Simulation Setup: The simulation uses OmniGibson, a physics-based framework built on NVIDIA Isaac Sim, with collision detection enforcing physical fidelity.The embodiment is derived from Fetch, with the robotic arm removed and an overhead camera mounted.
  • Action Execution: Low-level grasping is simplified by instantaneously attaching fetched objects to the agent upon interaction.The agent can fetch objects within a 1.0 m radius, prioritizing high-level planning over control fidelity.

E. Baselines

The paper evaluates four ESRP baselines spanning imitation learning, reinforcement learning, foundation-model reasoning, and hierarchical task-and-motion planning. The learning-based agents use egocentric observations and target-layout information, while the planning baseline assumes privileged global state.

  • Baselines: Four baselines span imitation learning, reinforcement learning, foundation-model-based VLM reasoning, and hierarchical task-and-motion planning.The agents are ESRP-BC, ESRP-PPO, ESRP-VLM, and ESRP-PLAN.
  • Imitation Learning: ESRP-BC uses a conditional diffusion policy to predict future action sequences from visual observations and contextual features.The policy conditions on egocentric observation history and target-layout information, then predicts a sequence over a finite horizon.
  • Reinforcement Learning: ESRP-PPO combines visual encoders, gripper-state embeddings, and an LSTM memory module to produce the next action.Its dense reward includes task completion, arrival or leaving signals, and distance-based potential terms.
  • Foundation-Model Agent: ESRP-VLM prompts a pretrained vision-language model in a ReAct loop without task-specific fine-tuning.The agent alternates between short reasoning traces and discrete actions.
  • Planning-Based Agent: ESRP-PLAN uses hierarchical task-and-motion planning with privileged global state and separate navigation, fetching, goal-reaching, and release phases.The task planner selects rearrangement order, while motion planning uses BFS and A* over discretized positions and orientations.

A. Experimental Setup

The experiments train and evaluate learning-based agents on fixed dataset splits, specify implementation settings for all baselines, and illustrate both successful and approximate rearrangements. The failure cases emphasize increasing difficulty as more objects must be rearranged.

  • Implementation Details: The RL policy trains for 6 million environment steps with PPO, a 2048 batch size, 64-sample minibatches, and a 64-step history context.The reported PPO learning rate is 1.5 × 10^-4, and the value-function loss coefficient is 5 × 10^-3.
  • Implementation Details: The IL policy trains for 500 epochs with 50 diffusion timesteps and predicts action sequences of length 8 while executing only the first 6 actions.Inference uses DDIM sampling with 16 denoising steps and includes a 0.1 probability of random action selection.
  • Implementation Details: The planning baseline uses BFS and A* with maximum 1000 iterations and discretizes positions at 0.05 m and orientations at π/32 radians.Task planning exhaustively searches object orderings when feasible and selects the ordering that places the most objects successfully.
  • Dataset: The dataset uses a 90% training split and a 10% test split, with the training scenes supporting both RL and IL training.The split is defined at the scene level.
  • Qualitative Results: Successful examples show agents moving objects to match the target layout, with red curves for agent trajectories and blue curves for object trajectories.The figure visualizes both navigation behavior and object motion.
  • Qualitative Results: As the number of rearrangeable objects increases, long-horizon complexity makes the task more challenging and agents may only approximate the target layout.The reported difficulty trend reflects increasing demands from longer rearrangement sequences.

B. Results

ESRP-PLAN leads across all three metrics, while learning-based and vision-language baselines remain weaker, especially as scene complexity and long-horizon execution demands increase.

  • ESRP-PLAN outperforms all other baselines across all three metrics by using global scene state for explicit navigation, fetching, and obstacle avoidance.Learning-based methods rely on partial observations and perform more weakly.
  • ESRP-VLM performs comparably to ESRP-BC but does not surpass ESRP-PPO, indicating off-the-shelf foundation-model priors are insufficient for ESRP’s long-horizon spatial reasoning.
  • Performance degrades progressively as the number of objects requiring rearrangement increases, with particularly pronounced degradation in complex scenes.The trend reflects compounded long-horizon planning, object-ordering, partial-observability, and dynamic-scene challenges.
  • Even unsuccessful scenes can achieve lower RDR when objects are placed closer to their designated goals, but efficient planning under time constraints remains difficult.

C. Discussion

The discussion identifies 3D scene understanding and long-horizon task planning as distinct sources of difficulty, while defending the benchmark’s abstractions and outlining deployment boundaries.

  • The gap between ESRP-PLAN and learning-based methods reflects difficulty inferring global scene states from partial egocentric views.
  • ESRP-PLAN achieves 0% SR on Hard-level scenes despite perfect perception, showing that multi-object rearrangement complexity challenges planning independently of perception.
  • The benchmark abstracts low-level control with “magic snapping” and strict collision checks to isolate embodied-AI challenges, consistent with mainstream rearrangement benchmarks.
  • Simulation-only evaluation supports controlled, reproducible comparisons before real-world deployment, while prior work reports routine sim-to-real adaptation for these abstractions.
  • Top-down layout images compactly specify which objects to move and where to place them, and can be produced through commodity RGB-D or LiDAR room scans.

V. CONCLUSION

The paper presents ESRP and ESRP-Bench for room-level furniture rearrangement from egocentric RGB observations, while identifying broader sensing and house-level planning as future extensions.

  • ESRP focuses on repositioning furniture within complex 3D rooms using egocentric images as observations.
  • ESRP-Bench provides a large-scale, diverse indoor-scene dataset and evaluation metrics in a realistic simulation environment.
  • The work is limited to room-level furniture rearrangement and exclusively egocentric RGB observations.
  • Future extensions include house-level planning and additional modalities such as depth, bird’s-eye-view cameras, and LiDAR.
Loading 2608.27371v1…