PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Room Rearrangement

Luca Weihs, Matt Deitke, Aniruddha Kembhavi, Roozbeh Mottaghi

arXiv:2103.16544v1cs.CVcs.RO

TL;DR

Embodied-AI benchmarks often omit interaction with objects and changing their states, motivating a more comprehensive room-rearrangement task. The paper introduces RoomR and baseline models for restoring rooms after object perturbations, and reports that modern methods perform only marginally above chance. The task is therefore still far from solved, especially under the paper’s object-state and visual-reasoning setting.

  • Problem

    Existing embodied-AI tasks commonly assume static environments, limiting evaluation of agents that must interact with objects and change their states.

  • Method

    The paper introduces the RoomR dataset and two room-rearrangement variants in which agents observe a goal configuration, then unshuffle altered object poses or states.

  • Results

    Modern deep-RL methods obtain test-set performance only marginally above chance on the RoomR rearrangement task.

  • Takeaways & Limitations

    Room rearrangement combines navigation, planning, object-state reasoning, memory, and manipulation in a challenging interactive benchmark.

  • Takeaways & Limitations

    The task restricts attention to piece-wise rigid objects rather than modeling soft-body physics.

Abstract

from arXiv · show

There has been a significant recent progress in the field of Embodied AI with researchers developing models and algorithms enabling embodied agents to navigate and interact within completely unseen environments. In this paper, we propose a new dataset and baseline models for the task of Rearrangement. We particularly focus on the task of Room Rearrangement: an agent begins by exploring a room and recording objects' initial configurations. We then remove the agent and change the poses and states (e.g., open/closed) of some objects in the room. The agent must restore the initial configurations of all objects in the room. Our dataset, named RoomR, includes 6,000 distinct rearrangement settings involving 72 different object types in 120 scenes. Our experiments show that solving this challenging interactive task that involves navigation and object interaction is beyond the capabilities of the current state-of-the-art techniques for embodied tasks and we are still very far from achieving perfect performance on these types of tasks. The code and the dataset are available at: https://ai2thor.allenai.org/rearrangement

1. Introduction

The paper frames room rearrangement as a comprehensive interactive embodied task that requires agents to observe, remember, navigate, and manipulate objects. It introduces RoomR and evaluates two task variants, finding that strong baselines struggle even in the easier setting.

  • Static navigation and instruction-following tasks do not test agents’ ability to interact with or change object states.The paper positions rearrangement as a more comprehensive interactive task encompassing manipulation, planning, and action-condition reasoning.
  • Room rearrangement requires restoring a room after objects are moved or their states are changed.The agent first records the goal configuration during walkthrough, then performs an unshuffle stage after objects have been altered.
  • RoomR contains 6,000 rearrangement tasks across 120 rooms and more than 70 unique object categories.Task difficulty varies with the differences between initial and goal object configurations.
  • The 1-Phase task aligns walkthrough and unshuffle observations, whereas the 2-Phase task separates them and requires longer-term reasoning.The two variants differ in whether the agent receives aligned images during the two stages.
  • Strong baselines struggle even in the easier 1-Phase task, showing that rearrangement remains challenging for current embodied-AI methods.The paper identifies navigation, visual comparison, object-state inference, memory, and action-condition learning as central challenges.

2. Related Work

Related rearrangement methods often assume idealized settings, while RoomR targets visually rich embodied environments with imperfect observability, mobile agents, and broader object variation.

  • Rearrangement is a broader embodied task that encompasses navigation, exploration, instruction following, manipulation, and other learned skills.
  • Prior robotics rearrangement methods commonly assume complete state observability, perfect perception, planar environments, static robots, or limited object variation.Some prior work addresses generalization to new objects or imperfect perception, but the related-work discussion identifies these assumptions as common limitations.

3. The Room Rearrangement Task

Room rearrangement asks an agent to transform a disturbed room from its observed initial state back to a goal configuration. The task models object pose and state, uses walkthrough and unshuffle stages, and evaluates restoration with approximate pose-based metrics.

  • 3.1. Definition: The task restricts objects to piece-wise rigid bodies to avoid requiring soft-body physics.
  • 3.1. Definition: Each object state records its 3D pose, openness when applicable, bounding-box corners, and broken status.The state representation includes pose, openness, geometry, and whether the object is broken.
  • 3.1. Definition: During walkthrough, the agent observes the goal room; after selected object states are changed, unshuffle requires converting the disturbed state back to the goal state.State changes may affect an object’s pose or openness, and the walkthrough is limited to a maximum number of actions.
  • 3.2. Metrics: Object poses are judged approximately equal using thresholds rather than exact coordinate matching.The metric definitions use pose comparisons based on object type and geometric overlap, including 3D bounding-box IOU for pickupable objects.
  • 3.2. Metrics: SUCCESS equals one only when all object poses match the goal approximately, while %FIXEDSTRICT measures the proportion of initially misplaced objects correctly restored.%FIXEDSTRICT becomes zero if the agent moves an object that should not have been moved.
  • 3.2. Metrics: %E gives partial credit by measuring remaining pose-distance energy, whereas #CHANGED counts objects whose poses the agent altered.The energy function decreases as poses approach one another, and #CHANGED is not intrinsically better when larger or smaller.

4. The RoomR Dataset

RoomR is an automatically generated benchmark of room-rearrangement tasks spanning diverse rooms, objects, and configuration changes. Its design requires agents to recover altered object poses or states, often involving substantial travel and small movable objects.

  • RoomR contains 6,000 unique rearrangements across 120 AI2-THOR rooms, with each datapoint specifying initial state, starting position, and goal state.The rooms span kitchen, living room, bathroom, and bedroom categories.
  • The generation procedure randomizes agent poses, shuffles background objects, and changes sampled pickupable or openable objects to create varied goal configurations.Changed objects may be moved within the room or opened to a randomly selected degree.
  • Dataset properties: Objects changed in position are significantly smaller than objects changed in openness across room categories.Figure 3 represents object size using the cube root of bounding-box volume.
  • The dataset includes 118 object categories, including pickupable and openable non-pickupable objects, with category overlap across training, validation, and testing rooms.Object instances can still have unique physical appearances across splits.
  • Dataset properties: Changed objects can be far from their goal positions, requiring agents to travel relatively far to recover the target configuration.Figure 2 reports horizontal Manhattan and vertical distance distributions between initial and goal positions.

5. Model

The model uses a recurrent actor-critic architecture for both walkthrough and unshuffle stages, combining visual processing, discrete-input embeddings, and temporal memory. The 1-Phase variant simplifies the model because walkthrough and unshuffle observations are aligned, whereas 2-Phase requires longer-term memory.

  • The model supports both walkthrough and unshuffle stages, with stage-specific connections and recurrent links from previous time steps.Figure 4 distinguishes walkthrough and unshuffle connections and identifies inputs, outputs, intermediate features, and trainable parameters.
  • The 2-Phase model requires longer-term memory because walkthrough and unshuffle occur sequentially, while 1-Phase aligns the two stages in lock step.The 1-Phase model is a simplification of the model used for 2-Phase rearrangement.
  • The architecture combines a CNN for egocentric images, embedding layers for discrete inputs, and an RNN for reasoning through time.This structure is used as the basic model architecture for embodied tasks.
  • The actor-critic agent maps observations and a summary of history to an action policy and an estimate of future rewards.At each timestep, the model produces both a policy distribution and a value estimate.
  • The implementation uses a pretrained ImageNet ResNet18 encoder, a one-layer LSTM with 512 hidden units, and separate policy and value heads for walkthrough and unshuffle.The ResNet18 produces 7×7×512 feature tensors before recurrent processing.

6. Experiments

The experiments evaluate simplified RoomR action and sensor settings with baseline models, training procedures, and qualitative trajectories. Results indicate persistent difficulty, including saturation, overfitting, and challenges even under abstract actions.

  • Action space: The baselines use a simplified action space with 84 possible actions, including navigation, rotation, looking, pickup, opening, placement, and termination.High-level PLACEOBJECT abstracts object-manipulation complexities, but planning and visual reasoning remain challenging.
  • RoomR variants: The 1-Phase task provides aligned walkthrough and unshuffle images, while the 2-Phase task separates them and requires reasoning over longer time spans.The supplied experiments include 1-Phase and 2-Phase baseline variants with different training procedures.
  • Training dynamics: Over approximately 75 million training steps, PPO models quickly saturate, whereas imitation-learning models continue improving but begin to overfit training scenes.Figure 5 reports #CHANGED and %FIXEDSTRICT with smoothed curves and 95% error bars.
  • Training: Reward-based reinforcement learning is extremely challenging for RoomR, even with dense shaped rewards, motivating a hybrid DD-PPO and imitation-learning strategy.The hybrid strategy assigns DD-PPO to walkthrough training and imitation learning to the unshuffle stage.
  • Baseline results: Pretrained CNN backbones improve SUCCESS and %FIXEDSTRICT across splits in the imitation-learning comparison, while results are more mixed for PPO baselines.The authors note that these results are preliminary because parameter counts were not carefully balanced.

7. Discussion

RoomR frames rearrangement as a demanding combination of navigation, planning, and object-pose reasoning, and reports that current deep reinforcement-learning methods remain only marginally above chance.

  • Task challenges: RoomR combines navigation, planning, and reasoning about object poses and states in visually rich interactive environments.The dataset is presented as a challenging testbed for rearrangement research.
  • Findings: Modern deep RL methods achieve test-set performance only marginally above chance on the proposed task.The discussion suggests that stronger models may need comparative mapping, visual reasoning, physics-aware manipulation, and new joint-training methods.

A. Implementation details

Training uses substantial compute and distinct reward structures for walkthrough and unshuffle stages. The reported setup supports large-scale experimentation but does not eliminate the task’s learning difficulty.

  • Compute: Agents are trained for approximately 75 million steps using AllenAct on instances with 4 NVIDIA T4 GPUs and 48 CPU cores.Training takes approximately 4.3 days with expert supervision and 1.8 days with pure PPO.
  • Compute: Training throughput is approximately 125 FPS with expert supervision and 300 FPS with pure PPO.The reported runtimes correspond to the same approximately 75-million-step training schedule.
  • Rewards: The walkthrough and unshuffle stages use different reward structures, with the unshuffle reward based on changes in scene-state energy.The supplied description begins defining the unshuffle transition reward from states s1 and s2.

B. Heuristic Expert

The heuristic expert supplies imitation targets through greedy object-by-object rearrangement with shortest-path navigation and backtracking, rather than solving the task optimally.

  • Expert design: The expert uses ground-truth scene layouts and object poses, so its performance cannot be fairly compared with visually guided agents.Its purpose is to produce expert actions for learning agents to imitate.
  • Expert procedure: The heuristic repeatedly selects the closest misplaced object, navigates to it, picks it up, navigates to a placement position, and places it.The procedure uses shortest paths computed from the scene layout.
  • Expert procedure: Backtracking lets the expert postpone an obstructed placement and try other objects when physics-based interactions cause failure.The expert is not optimal but can restore all but a small fraction of objects according to the supplied discussion.

C. Lower-level actions

The paper supplements high-level rearrangement actions with lower-level controls for motion, object placement, opening, and pickup. These controls expose more physical details of interaction.

  • Lower-level actions support continuous rotation and movement by arbitrary degrees and distances.
  • Object manipulation: Low-level placement lets agents move held objects through space, potentially collide with other objects, and explicitly drop them at goal locations.
  • Opening and picking up objects: High-level opening abstracts target openness and object location, whereas low-level opening requires image coordinates and an opening amount.

D. Semantic Mapping

The semantic-mapping baselines adapt Active Neural SLAM to predict object-aware maps from egocentric images. The module is pretrained, frozen, and incorporated into the baseline model.

  • Two baselines incorporate an adapted Active Neural SLAM module for semantic mapping.
  • The pretrained module maps a 224×224×3 image to a 40×40×75 tensor covering a 2m×2m region directly ahead.
  • Three output channels predict free space, while the remaining 72 channels predict probabilities for the rearrangement object types.
  • After pretraining, the semantic-mapping module is frozen and incorporated into the baseline model.

E. Computing the energy between two poses

The energy function D measures discrepancy between two object poses using special handling for broken, openable, and pickupable objects. For pickupable objects, it uses overlap or spatial distance, and it decreases as poses become closer.

  • The energy function D maps pairs of object poses to values in [0, 1].
  • If either pose represents a broken object, D equals 1.
  • For openable but non-pickupable objects, D equals 0 when openness differs by at most 0.2 and otherwise equals 1.
  • For pickupable objects with positive IOU, D uses 0.5 · max(0, 0.5 − IOU); otherwise it uses a distance-based expression involving the minimum corner distance.
  • D decreases monotonically as the two object poses move closer together.
Loading 2103.16544v1…