Source-linked AI summary
From Passive Execution to Active Exploration: Agentic Embodied Manipulation in Realistic Environments
Shilin Ma, Chubin Zhang, Xulong Bai, Zifeng Gao, Shiyi Zhang, Yansong Tang
TL;DR
Existing embodied manipulation frameworks often assume passive execution with visible targets, leaving realistic semantic cues, distractors, and initially invisible objects as a challenge. This paper combines planning, perception, and execution in an active exploration loop with fine-grained perception–action interleaving. On a real-world Find-and-Place task, success rates remain high across target variants and decrease only slightly with visually similar distractors.
Problem
Many embodied manipulation frameworks assume passive execution with visible task-relevant objects, unlike real-world scenes with occlusion, distractors, and indirect semantic cues.
Source: §1 · p. 1Method
The framework coordinates planning, perception, and execution in a feedback-driven loop, interleaving perception with fine-grained manipulation operations.
Source: §1 · p. 1, §2.2 · p. 3Results
Success rate decreases only slightly from 95.7% to 94.3% with visually similar distractors, while remaining consistently high across target-object variants.
Source: §3.4 · pp. 4–5Takeaways & Limitations
Real-world Find-and-Place experiments support the effectiveness of active exploration for hidden targets, distractors, and category-guided search.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Recent advances in agentic systems have substantially enhanced the long-horizon capability of embodied manipulation. However, many existing frameworks still follow a passive execution paradigm, which limits their applicability to real-world scenarios involving textual semantic cues, distractors, and initially invisible targets. To bridge this gap, we propose an agent-based active exploration framework that enables robots to dynamically interact with the environment rather than merely execute predefined instructions. Specifically, our framework consists of three collaborative modules: a planning module for high-level task reasoning, a perception module for visual scene understanding, and an execution module for low-level manipulation. This design allows the robot to actively acquire task-relevant information, adapt its behavior based on environmental feedback, and complete manipulation tasks under partial observability. Furthermore, we introduce a fine-grained perception-execution interleaving strategy, which tightly couples visual feedback with skill execution to improve exploration robustness. We evaluate our method on a realistic Find-and-Place task, demonstrating its effectiveness in challenging environments where target objects must be actively discovered before manipulation.
1. Introduction
Many embodied manipulation systems assume visible targets and passive execution, leaving a gap to real-world scenes with occlusion, distractors, and semantic cues. The paper proposes a closed-loop active exploration framework and evaluates it on a realistic Find-and-Place task.
1. Introduction
- Passive execution under visible-object assumptions leaves a gap to real-world manipulation, where targets may be occluded, initially invisible, surrounded by distractors, or indicated by semantic cues.Some planning-based methods address such scenarios but rely on carefully engineered modules and less generalizable paradigms.Source: §1 · p. 1
- The framework coordinates planning, perception, and execution in a closed loop, adapting subsequent actions to visual observations and environmental feedback.Its fine-grained perception–execution interleaving couples visual feedback with skill execution.Source: §1 · pp. 1–2
- On a realistic Find-and-Place task, the framework improves robustness and adaptability when targets are initially invisible or hidden among distractors.The task requires finding a target object and placing it at a specified location.Source: §1 · pp. 1–2
2. Method
The method treats manipulation as closed-loop decision-making: planning selects exploratory or manipulative steps, perception grounds decisions in visual observations, and execution carries out skills. Fine-grained perception–action interleaving lets the system update its scene understanding and replan as interactions reveal information.
2. Method
- Manipulation is framed as a closed-loop process that actively acquires missing information in partially observable, semantically complex environments.Unlike conventional instruction-following manipulation, the task does not assume relevant objects are visible and directly reachable.Source: §2 · p. 2
2.1. Overall Framework
- The planner selects subtasks from the instruction, current observation, and interaction history, including searching, inspecting cues, opening containers, retrieving, or placing.Perception extracts object, location, OCR, and scene-semantic information; execution converts planned subtasks into robot actions.Source: §2.1 · p. 2
- After each manipulation skill, perception updates the environment state so planning can adapt to occlusion, distractors, and initially invisible targets.The closed loop selects an action, executes it, and updates the scene understanding from the resulting observation.Source: §2.1 · p. 3, Figure 2 · p. 3
2.2. Fine-grained Interleaving
- Fine-grained interleaving breaks long-horizon tasks into atomic operations and re-perceives after each step, letting the planner verify progress and decide whether to explore or continue.For example, opening a drawer is followed by inspecting the revealed space before choosing the next operation.Source: §2.2 · p. 3
- Interleaved verification can correct earlier assumptions in cluttered scenes, limiting the accumulation of errors from distractors and semantic ambiguity.The approach addresses the limitation of perceiving only before or after the entire task, since manipulation can reveal new information.Source: §2.2 · p. 3
3. Experiment
The evaluation uses a real-world Find-and-Place task in which the robot searches labeled cabinets and drawers for a hidden target amid distractors, then places it in a box. Across target variants, success rates remain high, with only a slight decrease when visually similar distractors are present.
3.1. Real-World Task Setup
- The real-world task uses two category-labeled, multi-drawer cabinets, with the target hidden in an unknown drawer and distractors potentially present.The robot infers the relevant cabinet from semantic labels, searches candidate drawers, identifies the visible target, and places it in a box.Source: §3.1 · p. 4
3.2. Implementation Details
- The system runs on a Franka robotic arm, with Open-Claw for high-level planning and local perception and execution modules for scene understanding and robot actions.The authors report that it runs on a consumer-grade NVIDIA RTX 3090 GPU.Source: §3.2 · p. 4
3.3. Evaluation Protocol
- Evaluation varies pear-toy appearance across color, size, material, and texture and adds lemon-shaped distractors; success requires finding the target and completing placement.Failures include choosing the wrong cabinet, stopping before finding the target, grasping a distractor, or failing to place it.Source: §3.3 · p. 4
3.4. Results and Analysis
- Success rate decreases only slightly from 95.7% to 94.3% with visually similar distractors, while results remain consistently high across target-object variants.Table 1 reports results across three target variants and conditions with and without visually similar distractors.Source: §3.4 · pp. 4–5, Table 1 · p. 5
4. Conclusion
The conclusion presents agent-based active exploration as a way to handle realistic manipulation conditions through feedback-driven planning, perception, and execution. It reports effectiveness on real-world Find-and-Place tasks with hidden targets, distractors, and category-guided search.
4. Conclusion
- The framework integrates planning, perception, and execution in a closed loop to acquire task-relevant information and adapt behavior to feedback.The conclusion frames this as active exploration rather than passive execution.Source: §4 · p. 5
- Perception–action interleaving updates scene understanding after each fine-grained operation to support more reliable long-horizon execution.Source: §4 · p. 5
- Real-world Find-and-Place experiments demonstrate effectiveness with initially invisible targets, distractors, and category-guided search.Source: §4 · p. 5