Source-linked AI summary
ManipulaTHOR: A Framework for Visual Object Manipulation
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, Roozbeh Mottaghi
TL;DR
Embodied AI has advanced mainly through navigation, but object manipulation requires mobile agents to handle complex scenes, occlusion, collisions, and long-horizon multi-object tasks. The paper introduces ManipulaTHOR and ArmPointNav by extending AI2-THOR with articulated arms for visual manipulation. Evaluations show that the end-to-end model generalizes to novel objects and unseen environments, while substantial room for improvement remains.
Problem
Object manipulation remains challenging because agents must handle manipulator motion, collisions, occlusion, long-horizon planning, mobile settings, and generalization to unseen environments and objects.
Method
The paper extends AI2-THOR with arms and introduces ArmPointNav, which requires moving to an object, picking it up, and moving it to a target location.
Results
The end-to-end ArmPointNav model generalizes to novel objects in unseen environments and outperforms a disjoint model with separate policies for each skill.
Takeaways & Limitations
ManipulaTHOR provides a framework, dataset, and task for studying generalizable visual object manipulation beyond navigation.
Abstract
from arXiv · showhide
The domain of Embodied AI has recently witnessed substantial progress, particularly in navigating agents within their environments. These early successes have laid the building blocks for the community to tackle tasks that require agents to actively interact with objects in their environment. Object manipulation is an established research domain within the robotics community and poses several challenges including manipulator motion, grasping and long-horizon planning, particularly when dealing with oft-overlooked practical setups involving visually rich and complex scenes, manipulation using mobile agents (as opposed to tabletop manipulation), and generalization to unseen environments and objects. We propose a framework for object manipulation built upon the physics-enabled, visually rich AI2-THOR framework and present a new challenge to the Embodied AI community known as ArmPointNav. This task extends the popular point navigation task to object manipulation and offers new challenges including 3D obstacle avoidance, manipulating objects in the presence of occlusion, and multi-object manipulation that necessitates long term planning. Popular learning paradigms that are successful on PointNav challenges show promise, but leave a large room for improvement.
1 Allen Institute for AI 2 University of Washington https://ai2thor.allenai.org/manipulathor
Figure 1 illustrates visual object manipulation as moving an object between two scene locations in a visually rich environment. The sequence shows a virtual robot picking up a vase and stacking it on a plate.
- Visual object manipulation moves an object between two locations in a scene.
- The task involves operating in visually rich and complex environments while avoiding collisions and planning a route to the destination.
- The illustrated AI2-THOR sequence picks up a vase from a shelf and stacks it on a plate on a countertop.
1. Introduction
The paper motivates object manipulation as a progression beyond navigation, introducing ManipulaTHOR and ArmPointNav to study mobile manipulation in complex scenes. The proposed end-to-end model provides baseline evidence for generalization to unseen environments and novel objects.
- Motivation: Embodied AI has primarily progressed on navigation, while the broader goal includes agents that actively manipulate objects.
- Challenges: Object manipulation requires obstacle avoidance, generalization to novel environments and objects, and visual reasoning for multi-object tasks in cluttered scenes.
- ManipulaTHOR: ManipulaTHOR extends AI2-THOR with arms that support both forward and inverse kinematics for navigation and active object manipulation.
- ArmPointNav: ArmPointNav asks an agent to move toward an object, pick it up, and move it to a desired location.
- ArmPointNav: ArmPointNav adds joint agent-and-arm motion, occlusion-aware collision avoidance, and possible multi-object manipulation requiring long-term planning.
- Results: The end-to-end model generalizes to new environments and novel objects and outperforms a disjoint model with separate skill policies.
2. Related Works
Related work spans robotic object manipulation, simulated environments, and visual navigation. The paper distinguishes its framework through joint navigation and manipulation in visually complex scenes with novel objects and environments.
- Object Manipulation: Prior manipulation work studies cluttered multi-step interaction, grasp planning, object dynamics, skill composition, model-based planning, and mobile manipulation.
- Environments for Object Manipulation: Existing environments include tabletop arms, crowdsourced trajectories, grasping datasets, and interaction frameworks with varying support for interactive or low-level manipulation.
- Visual Navigation: The paper distinguishes its framework by combining visually complex scenes, novel-object and novel-scene generalization, joint navigation and manipulation, and navigation while carrying an object.
3. ManipulaTHOR
ManipulaTHOR extends AI2-THOR with physically simulated arms for low-level object interaction in visually rich environments. It supports flexible arm control, abstracted grasping, collision-aware physics, and training at 300 fps.
- Framework: ManipulaTHOR extends AI2-THOR with arms for studying object manipulation in visually rich environments with varied objects and scenes.The framework uses AI2-THOR’s physics engine, realistic indoor scenes, and manipulable object library.
- Arm design: The arm uses a simple three-jointed design with swivel joints, rotating shoulder and wrist components, and a 6DOF hand grasper.The design supports either forward or inverse kinematics, allowing joint-level or wrist-level control.
- Grasping: Objects intersecting the spherical grasper can be picked up, abstracting grasping complexity rather than modeling jaw or humanoid-hand mechanics.The authors identify detailed grasping as a future extension.
- Arm interface: Agents can control wrist location and orientation, arm height, grasping, dropping, and grasper radius through the arm interface.The arm supports inverse-kinematic wrist control and exposes joint-state metadata.
- Performance: 300 fps enables training for approximately 20M steps per day, compared with roughly 800 fps for PointNav with simpler collision checks and no manipulator.The manipulation simulator trades speed for accurate collision detection and object displacement estimation.
4. ARMPOINTNAV
ARMPOINTNAV extends point navigation into mobile object manipulation: agents navigate to an object, pick it up, carry it, and place it at a target. The benchmark adds arm-aware 3D collision avoidance, occlusion, and carefully timed pickup challenges.
- Task definition: ARMPOINTNAV requires navigating to an object, positioning the gripper, picking it up, carrying it to a target, and releasing it carefully.Source and target object locations are specified using relative coordinates, following PointNav’s task structure.
- Dataset: Evaluation uses 60 randomly subsampled tasks per object per scene, alongside a larger pool of training tasks.The subsampled tasks provide benchmark evaluation splits.
- Metrics: Reported metrics include success rate, pickup success rate, episode length, successful episode length, and pickup-successful episode length.The metrics distinguish overall task completion, pickup performance, and episode duration.
- Challenges: The agent must coordinate body navigation with 3D arm and object motion while avoiding collisions during carrying and reaching.Reaching is especially difficult when source or target locations are occluded and pickup timing must be selected precisely.
- Dataset statistics: The dataset includes target distances from initial object states and marks agent and arm movement step sizes for comparison.The initial distance between the agent and object is excluded from this distribution.
5. Model
The model investigates end-to-end learning for ARMPOINTNAV, combining visual observations and relative arm-object and arm-goal locations to produce actions. Its architecture uses a recurrent controller trained with DD-PPO and shaped rewards for pickup and placement.
- Task formulation: ARMPOINTNAV requires agents to navigate on a 2D floor while navigating the arm and manipulated objects through surrounding 3D space.This combines mobile navigation with manipulation in a single task.
- Task challenges: The dataset includes occluded, collision-sensitive, and enclosed-object configurations that require different manipulation strategies.Opening a microwave before reaching an enclosed object is explicitly outside the paper’s scope.
- Learning approach: The work contrasts modular manipulation pipelines with an end-to-end model that directly learns the task policy.The modular alternatives use components such as detection, segmentation, point-cloud estimation, and planning.
- Architecture: The policy uses visual embeddings, relative arm-to-object and arm-to-goal coordinates, and a recurrent controller to output distributions over discrete actions.The architecture combines convolutional image features and coordinate embeddings before processing them with a GRU.
- Training objective: DD-PPO training combines terminal rewards with intermediate shaping based on pickup success and changes in arm-object and object-goal distances.The reward design balances the pickup and placement phases of the task.
6. Experiments
Experiments evaluate ARMPOINTNAV on seen and novel scenes and objects, compare sensor and architectural choices, and analyze distance-to-goal and qualitative behavior. The end-to-end model generalizes beyond training conditions, but object transport remains harder than pickup and unseen-object generalization is challenging.
- Experimental setup: The experimental setup uses RGB, depth, GPS, and compass-related inputs with 13 discrete actions, and depth-only input outperforms RGB input.Models are trained for 20M frames with maximum episode length 200.
- Quantitative results: The agent achieves 39.4% SRwD and 89.9% PuSR on interacted-with objects, showing pickup is easier than transporting objects through scenes.Allowing disturbance raises SRwD to 68.7%.
- Generalization: The model generalizes moderately to novel objects, with performance similar in unseen scenes and seen scenes containing novel objects.The authors attribute the challenge to needing object geometry understanding for collision avoidance.
- Distance to goal: SRwD decreases as the target becomes farther away, but the model remains relatively robust to longer distances and performs better on seen than novel objects.Figure 8 compares SRwD by initial distance to goal.
- Qualitative results: Qualitative examples show pickup and transport of differently sized objects, including careful arm movement when placing an object on a cluttered table.The examples include an apple, lettuce, and a third-person view of navigation around obstacles.
- Architectural comparison: The end-to-end model shares one policy across pickup and transport, while the disjoint comparison separates these skills and improves pickup metrics but fails to learn the full task.The end-to-end design is motivated by shared physical understanding between grasping and moving objects.
7. Discussion / Conclusion
ManipulaTHOR extends AI2-THOR for visual object manipulation and supports ARMPOINTNAV, in which agents pick up objects and move them to target locations. Results show current embodied-AI models still leave substantial room for improvement, while relaxing GPS and visual-sensor requirements remains future work.
- Framework: ManipulaTHOR provides diverse, visually complex scenes and object categories for studying joint navigation and manipulation.The framework builds upon AI2-THOR.
- Task: ARMPOINTNAV evaluates picking up an object and moving it to a target location within the environment.
- Conclusion: State-of-the-art models effective on embodied navigation are less effective for object manipulation, leaving substantial room for improvement.
- Future work: The experiments use GPS and visual sensors, and relaxing these sensor requirements is identified as future work.
A. Data generation
Data generation enumerates possible object locations and retains initial placements reachable by a valid agent pose. This ensures every generated task has a solution for picking up and dropping off the object, whether or not the target is visible.
- Location generation: The process finds all possible locations for each object in every room.
- Reachability: Initial object locations are filtered to those with a valid agent pose that can reach them.
- Task validity: The filtering ensures each task contains reachable pickup and drop-off locations, including targets that may be visible or hidden.
B. More Dataset Stats
Additional dataset statistics characterize the task pool, per-object location counts, and object–receptacle co-occurrences. These statistics emphasize broad variation in possible object placements and task configurations.
- Training task pool: The training split is a large streaming pool of tasks sampled from pairs of possible object locations.
- Task counts: Figure 9 reports the number of possible tasks per training object, with smaller objects yielding more possible locations under physical space constraints.
- Location statistics: Table 3 reports possible object locations per object for each data split to characterize dataset diversity.
- Object–receptacle co-occurrences: Objects can occur on diverse source and target receptacles, requiring agents to reach and place them in varied situations.