PaperlayerView exact revision on arXiv

Source-linked AI summary

I Won't Know Until I Try: Interactive Information Gathering for Long-Horizon Manipulation

Linfei Yuan, Rachel Holladay

arXiv:2610.33959v1cs.RO

TL;DR

Long-horizon manipulation can require properties that appearance alone cannot reveal, and physical tests can also change the scene. The framework combines optimistic interactive actions with execution monitoring and recovery replanning; completion time is lower than Global in seven of eight settings, and it recovers from two tested disturbances in all ten trials per condition.

  • Problem

    Task-relevant properties may be unobservable from appearance, while interactions used to reveal them can alter the physical state and invalidate later actions.

    Source: §I · p. 1

  • Method

    The framework plans optimistically with interactive information-gathering actions, verifies outcomes during execution, and reconnects to reusable plan portions through recovery replanning.

    Source: §I · p. 1

  • Results

    Completion time Ttotal is lower than Global in 7 of 8 settings, while the framework completes 10/10 trials in each of two unmodeled disturbance conditions.

    Source: B. Effectiveness and Efficiency of… · p. 6

  • Takeaways & Limitations

    Execution monitoring and recovery replanning allow the robot to adapt to interaction outcomes and execution deviations while continuing toward the task goal.

    Source: §VII · p. 7

Abstract

from arXiv · show

Long-horizon manipulation often depends on object properties that cannot be reliably inferred from appearance alone. Physical interaction can reveal these properties, but it can also change the world in ways that invalidate subsequent actions. We present a task and motion planning (TAMP) framework that integrates task-directed interactive information-gathering with interleaved planning and execution. Building on PDDLStream, our framework plans optimistically with these interactive actions, verifies their outcomes during execution, and adapts the remaining plan by retaining still-applicable actions and replanning only the connections needed to continue execution. We evaluate the framework in simulation and on a real robot across tasks that require determining whether an object is movable, a pen has ink, a drawer can be opened, or a container is empty. These experiments demonstrate how interaction-driven state updates and recovery replanning support continued execution under uncertain outcomes and external disturbances.

I. INTRODUCTION

Long-horizon manipulation requires information about task-relevant properties that appearance alone may not reveal, while gathering that information can also alter the world. The paper integrates interactive information gathering with planning and execution, then adapts plans when observations invalidate actions.

I. INTRODUCTION

  • Physical tests can reveal hidden properties while changing the scene: pushing may expose movability but displace an object by an uncertain offset.These effects make downstream actions potentially invalid.Source: §I · p. 1
  • Information gathering must be planned around both which properties the task requires and the physical effects that demand downstream adaptation.Interactive perception addresses properties that cannot be reliably inferred from appearance alone.Source: §I · p. 1
  • The framework combines optimistic interactive actions, execution monitoring, and recovery replanning to adapt plans while retaining reusable portions.It reconnects the current state to a valid plan suffix rather than discarding the entire remaining plan.Source: §I · p. 1
  • The framework is evaluated in simulation and on a real robot across movability, pen-ink, drawer-openability, and container-emptiness tasks.Its stated contributions include task-directed information-gathering actions and interleaved planning and execution.Source: §I · pp. 1–2

II. RELATED WORK

Prior work addresses partial observability, task-directed information gathering, and adaptive execution. This paper distinguishes its approach through explicit property-knowledge predicates and by accounting for interactions that both reveal information and change the physical scene.

II. RELATED WORK

  • The paper situates its work among TAMP under partial observability, interaction-directed information gathering, and adaptive execution methods.TAMP jointly searches action sequences and constrained discrete and continuous parameters; real-world manipulation can depend on outcomes uncovered during execution.Source: §II · p. 2, A. Task and Motion Planning under… · p. 2

A. Task and Motion Planning under Partial Observability

  • The paper uses explicit predicates for property knowledge and adapts plans as observations update both those predicates and geometric relationships, rather than tracking continuous belief distributions.Its representation builds on existing symbolic encodings rather than introducing a new epistemic formalism.Source: A. Task and Motion Planning under… · p. 2

B. Task-Directed Information Gathering through Interaction

  • Unlike approaches that assume information-gathering actions leave the world unchanged, this framework models coupled informational and physical effects, such as a push revealing movability while changing object pose.Execution must account for both discovered information and physical-state changes.Source: B. Task-Directed Information Gathering… · p. 2

C. Adaptive Execution and Online Replanning

Prior adaptive-execution work reuses planning effort through action or motion repair, constrained replanning, reactive execution, and learned policies. This framework retains validated, potentially reusable action suffixes and generates new actions only to bridge execution gaps.

C. Adaptive Execution and Online Replanning

  • Planning repair reuses components of an existing solution by removing obstructing actions or constraints and adding actions needed to restore a valid plan.Source: C. Adaptive Execution and Online… · p. 2
  • SS-Replan initially reuses the unexecuted plan tail while recomputing observation-dependent poses, configurations, and trajectories, relaxing that constraint if no plan is found before timeout.Source: C. Adaptive Execution and Online… · p. 2
  • TAMPER uses closed-loop actions that cannot be fully grounded before execution, then connects the resulting configuration to the next planned action through online motion repair.Source: C. Adaptive Execution and Online… · p. 2
  • Other strategies limit the planning horizon, express motions relative to objects, replan from scene hypotheses or monitored beliefs, or use logical conditions for reactive execution.Source: C. Adaptive Execution and Online… · p. 2
  • Learning-based approaches update failure-probability models across tasks or train bridge policies that handle state discrepancies and decide when to return control to the planner.Source: C. Adaptive Execution and Online… · p. 2
  • The framework retains validated, potentially reusable instantiated action suffixes and generates new actions only when execution encounters a gap.Source: C. Adaptive Execution and Online… · p. 2

III. PROBLEM FORMULATION

The problem is to plan manipulation when task-critical properties are initially unknown and must be discovered through physical interaction. The formulation represents knowledge status explicitly and uses optimistic tests whose execution reveals outcomes while potentially changing the geometric state.

III. PROBLEM FORMULATION

  • PDDL represents world states as sets of facts, while actions are controllers with preconditions and effects, parameterized by discrete and continuous variables.The domains combine basic motion and manipulation actions with domain-specific information-gathering actions.Source: §III · pp. 2–3
  • The TAMP objective is to find an action plan that achieves a goal state, jointly solving for its action sequence and discrete and continuous parameters.The framework uses PDDLStream, which reduces hybrid planning to discrete problems through focused parameter sampling.Source: §III · p. 3
  • Initial states include object types, poses, and robot configuration, while task-critical binary properties can remain unknown and necessary for achieving the goal.Examples include object movability and whether a pen contains ink.Source: §III · p. 3
  • Information-gathering actions such as pushing or writing both reveal property facts and alter geometric relations, so plans must adapt to both effects.The interaction can change object poses while uncovering the property being tested.Source: §III · p. 3
  • The representation distinguishes an unevaluated property from a tested negative result, because those states require different responses from the robot.The robot must track whether evaluation occurred as well as its outcome.Source: §IV · p. 3
  • For each property, CheckedX records evaluation and IsX records a positive outcome, encoding unknown, tested-positive, and tested-negative statuses.A negative result is represented by CheckedX without IsX; evaluations are treated as deterministic, without confidence scores or probabilities.Source: §IV · p. 3
  • Optimistic test actions predict that a property is positive during planning, but the robot verifies that outcome during execution.A dependent manipulation action requires the positive-property predicate; the test’s optimistic effect is a planning prediction, not a claim of prior knowledge.Source: §IV · p. 3
  • When a property is unknown, the planner sequences its test before dependent actions, and CheckedX prevents repeating the test after a negative outcome.For movability, pushing tests an object before downstream pick, move-holding, and place actions that require IsMovable.Source: §IV · p. 3

V. INTERLEAVED PLANNING AND EXECUTION

The framework interleaves observation, plan validation, execution, and recovery so it can continue when outcomes or disturbances change the state. It retains usable actions and repairs connections on demand rather than rebuilding the whole plan after each mismatch.

V. INTERLEAVED PLANNING AND EXECUTION

  • The algorithm repeatedly observes the state, validates the remaining plan, executes ready actions, and replans when needed.It terminates when the goal is satisfied; if the retained plan is exhausted first, it generates a new plan.Source: §V · p. 4
  • The VALIDATOR removes actions whose preconditions no longer hold, including motions made collision-infeasible by changed object poses.For example, a mug revealed to be immovable invalidates its planned pick, move-holding, and place actions.Source: §V · p. 4
  • Plan gaps are repaired lazily, avoiding premature planning under uncertain conditions; reusable actions may still need a PRECHECK before execution.Small perceptual discrepancies are handled with state-matching tolerances.Source: §V · p. 4
  • When PRECHECK finds the next retained action is not executable, RECOVERYREPLAN searches for a bridge to that action or a later one.A successful bridge is prepended to the retained suffix; if no bridge succeeds, the framework plans from scratch.Source: §V · p. 4
  • After an interactive action, AFTERCHECK records the observed property and revises the active goals to reflect the result.In the example, the immovable mug is removed from the target set while the movable can remains.Source: §V · p. 4

VI. EXPERIMENTS

The experiments cover four domains requiring different kinds of interactive information gathering. They compare efficiency with global replanning and demonstrate handling of execution disturbances and interdependent information-gathering actions.

VI. EXPERIMENTS

  • The experiments span four domains requiring different types of interactive information gathering.Source: §VI · p. 4
  • The framework’s efficiency is evaluated against a global replanning baseline.Source: §VI · p. 4
  • The experiments demonstrate handling of unexpected execution disturbances and multiple interdependent information-gathering actions.Source: §VI · p. 4

A. Domains

The domains test interactive discovery of object and task-relevant properties through physical actions or measurements. Figure 2 illustrates how execution updates a plan using retained actions and newly introduced bridges.

A. Domains

  • The four domains test object movability by pushing, pen ink with a test stroke, drawer openability by pulling, and container emptiness by measuring mass.Source: A. Domains · pp. 4–5
  • Figure 2 traces online recovery, with updated plans combining retained actions and newly introduced bridge actions.The trace begins with actions a1 through a12.Source: Figure 2 · p. 5, A. Domains · p. 5
  • The tasks route objects according to test outcomes: movable objects, functional pens, openable drawers, and empty containers are handled according to their respective goals.Objects assigned to locked drawers remain on the table; nonfunctional pens return to the workspace, and non-empty containers are poured out first.Source: A. Domains · p. 5
  • Each domain has easy settings with 2–3 objects and hard settings with 4–6 objects; positive and negative targets vary across trials.Source: A. Domains · p. 5

B. Effectiveness and Efficiency of Recovery Replanning

The section compares recovery replanning with a global-from-scratch baseline and evaluates completion, efficiency, recovery, and robustness. Recovery replanning reduces completion time in most settings and handles execution disturbances without modification.

B. Effectiveness and Efficiency of Recovery Replanning

  • Global replanning serves as a controlled ablation: both methods share action sets, state monitoring and updates, and goal updates.The comparison isolates reconnecting to retained actions through bridge construction.Source: B. Effectiveness and Efficiency of… · p. 5
  • In Easy cases both methods achieve high task completion, while recovery replanning outperforms Global in Hard cases.The evaluation spans four domains and two difficulty levels, with 20 trials per condition.Source: B. Effectiveness and Efficiency of… · p. 6, Table I · p. 6
  • Completion time Ttotal is lower in 7 of 8 settings; versus Global, it falls by approximately 44% for Object-Movability (Hard), 35% for Pen-Functionality (Hard), and 65% for Container-Emptiness (Hard).Drawer-Openability (Easy) is the only setting in which Global is faster.Source: B. Effectiveness and Efficiency of… · p. 6, Table II · p. 7
  • Online planning time Ton drops by up to 91% because short bridge actions cost less to generate than replanning from scratch.The computational savings outweigh execution costs, and overall runtime remains lower even when the framework requires additional actions.Source: B. Effectiveness and Efficiency of… · p. 6
  • Across 152 successful trials, recovery required 14 fallbacks to global replanning, with none in Pen-Functionality or Container-Emptiness.Fallbacks primarily occurred when a short remaining plan suffix offered few viable reconnection points.Source: B. Effectiveness and Efficiency of… · p. 6, Table III · p. 7
  • Without modifications, the monitoring and replanning loop achieved 100% completion (10/10) under both unmodeled scene changes and action failures.The trials used the Container-Emptiness task in the easy setting.Source: B. Effectiveness and Efficiency of… · p. 6

C. Transfer across Interaction Types and Task Composition

A combined task tests whether the framework can coordinate several interdependent information-gathering actions within one planning horizon.

C. Transfer across Interaction Types and Task Composition

  • In a combined domain requiring movability, emptiness, pen-functionality, and drawer-openability reasoning, the framework completed 5 of 5 trials.The goal was to store functional pens and empty, movable mugs inside a drawer.Source: C. Transfer across Interaction Types… · p. 7

D. Real-Robot Demonstrations

The real-robot demonstrations use a Franka FR3 to test object movability and pen functionality through physical interaction and visual inspection.

D. Real-Robot Demonstrations

  • On the Franka FR3, the robot demonstrated movability tests with a force-limited push and pen tests using before-and-after images of a writing surface.The push controller retracts and records an object as immovable when contact force exceeds 10 N without displacement.Source: D. Real-Robot Demonstrations · p. 7
  • PRECHECK accommodates hardware and pose-estimation variability with translation, orientation, and joint-configuration tolerances.It ignores rotationally symmetric axes, such as those of mugs and markers, during orientation checks.Source: D. Real-Robot Demonstrations · p. 7
  • Across two-object scenes, the robot transferred movable objects and functional pens while leaving immovable objects in place and returning nonfunctional pens to the table.The demonstrations used pose estimates from two RealSense D435i cameras and supplied object meshes.Source: D. Real-Robot Demonstrations · p. 7

VII. CONCLUSION AND DISCUSSIONS

The framework combines physical interaction, execution monitoring, and recovery replanning to continue task execution when observations or physical effects invalidate planned actions.

VII. CONCLUSION AND DISCUSSIONS

  • The framework plans optimistically that objects are movable, pens functional, drawers openable, and containers empty, then verifies these assumptions during execution.Observed outcomes trigger replanning strategies.Source: §VII · p. 7
  • Negative results in Object-Movability and Drawer-Openability prompt goal pruning and management of postinteraction pose uncertainty.For nonfunctional pens, goal pruning is paired with actions that return the pen to the table.Source: §VII · pp. 7–8
  • A non-empty container does not invalidate the goal; it requires pouring before the container can be placed.The unified framework balances goal revision, action insertion, and execution recovery under uncertainty.Source: §VII · pp. 7–8
Loading 2610.33959v1…