PaperlayerView exact revision on arXiv

Source-linked AI summary

ADM-Planner: LLM-Guided Long-Horizon Planning for Mobile Manipulators with Attention-Enhanced Dynamic Memory

Jiaping Xiao, Pingyuan Ji, Mir Feroskhan

arXiv:2609.29212v1cs.RO

TL;DR

Long-horizon mobile manipulation needs task state that remains current without accumulating stale history. ADM-Planner maintains object-centric state and selectively retrieves records, replanning when changes affect the remaining plan. It achieves 100% success in DN-14 and reduces context by 95.8% versus unfiltered memory, with further validation in live, rigid-body, and physical studies.

Abstract

from arXiv · show

Large language models can decompose mobile-manipulation goals into long action sequences, but the resulting plans remain reliable only while their world context is current. A fixed scene description becomes stale when objects are discovered, moved, or completed while retaining every observation instead produces a growing history with redundant and conflicting state. To resolve this tension, we present an LLM-guided planning framework ADM-Planner with attention-enhanced dynamic memory (ADM). Persistent workspace knowledge is separated from object-centric state, asynchronous observations and action outcomes update that state, and a bounded retriever exposes only the entries that can affect the next decision. The LLM replans when an update invalidates the remaining plan. Across 1,500 task-simulator episodes, the proposed ADM achieved 100% full-task success in the 14-container noisy dynamic setting, compared with 62% for static memory and 97% for unfiltered dynamic memory, while reducing the context-size proxy by 95.8% relative to the latter. In a six-episode live GPT-5 Mini planner, both dynamic memory variants completed every mission, while ADM reduced provider-reported input tokens by 14.4% and mean planner calls from 7.0 to 6.0. A separate 60-trial PyBullet study retained 100% success for ADM, compared with 50% for static memory. Finally, the mobile manipulator with ADM-Planner completed various missions in indoor and outdoor physical experiments while incorporating targets revealed after execution began. The results show that selective state maintenance with ADM, rather than prompt history alone, is a practical basis for long-horizon planning in changing environments. Project page: https://xjp99v5.github.io/ADM-Planner

I. INTRODUCTION

Long-horizon mobile manipulation requires task state that stays current as objects change, while remaining compact enough to guide the next decision. ADM-Planner addresses this through dynamic state updates and bounded retrieval, building on prior work in LLM planning, persistent maps, and aerial-ground systems.

I. INTRODUCTION

  • A planner must preserve progress as deliveries change active objects and off-view objects are discovered or moved, without assuming the initial scene remains correct.Fixed maps become stale, while appending every observation grows context and retains conflicting records.Source: §I · p. 1
  • ADM-Planner separates stable workspace knowledge from object-centric state, updates records from asynchronous evidence, and retrieves only top-K records relevant to the next decision.It replans from the achieved state when an update changes an unexecuted action.Source: §I · p. 1
  • The framework formulates planning as joint state maintenance and bounded retrieval, and demonstrates aerial detections extending the mobile manipulator’s task state in real-world recycling scenarios.The update-retrieve-replan algorithm responds to asynchronous evidence without replaying the complete interaction history.Source: §I · p. 1

II. RELATED WORK

  • LLMs translate goals into robot actions –; feedback methods and scene-graph or integrated systems – leave open what state to retain in changing missions.Semantic maps and scene graphs also provide persistent memory for multi-object navigation and manipulation –.Source: §II · p. 1
  • Long-term embodied agents retrieve prior observations, while this work focuses on maintaining one current task record per entity and bounding retrieval by the decision.Aerial perception supplies asynchronous task evidence rather than acting as a separate planner; onboard observations and skill outcomes use the same update process.Source: §II · p. 2

III. PROBLEM FORMULATION

The formulation represents long-horizon planning using a goal, robot execution state, persistent workspace knowledge, and current task-entity state. It constrains retrieval to a bounded subset and keeps low-level control outside the LLM.

III. PROBLEM FORMULATION

  • The planner conditions on the manipulation goal, execution state, static workspace and skill knowledge, and dynamic memory containing task-relevant entity state.Static memory includes the map, station poses, task rules, and available skills.Source: §III · p. 2
  • Entity state tracks identifiers, classes, estimated poses, task statuses, confidence, update times, and observation sources, with status progressing through observed, assigned, grasped, delivered, or failed.Source: §III · p. 2
  • At each planning event, the system selects a bounded subset of dynamic memory that can affect the next decision while excluding stale, completed, or irrelevant records.The objective includes mission completion while limiting failed actions, unnecessary travel, replanning calls, and context cost.Source: §III · p. 2
  • Collision avoidance, trajectory generation, and grasp control remain with the reliable low-level controller; the planner invokes only skills exposed by the robot interface.Source: §III · p. 2

IV. METHODOLOGY

ADM-Planner closes the loop between state estimation and high-level planning through event-driven memory updates, bounded retrieval, and validated skill execution. Planning is triggered when needed rather than for every observation.

A. Overview

  • An event interface converts asynchronous perceptual reports and synchronous skill outcomes into updates to a single object-centric world state.Source: A. Overview · p. 2
  • An attention-based retriever filters terminal or unsupported records and serializes at most K scored entities before the LLM proposes a typed skill sequence.Source: A. Overview · p. 2
  • The proposed skill sequence is validated before dispatch, and outcomes return through the event interface so later plans retain achieved progress.Planning occurs when no plan exists, execution fails, or an update affects the remaining plan; the LLM does not directly modify memory or command actuators.Source: A. Overview · p. 2

B. Object-Centric State Updates

Object-centric updates preserve stable workspace knowledge while maintaining one current record per task entity. The system associates incoming observations, arbitrates updates by field, and replans only when changed fields affect the remaining plan.

B. Object-Centric State Updates

  • Static memory holds maps, station poses, semantic regions, task rules, and skill definitions, while dynamic memory keeps one current record per task entity.Grasps and placements advance task status; observations may revise pose and confidence, while earlier values may remain in an audit log.Source: B. Object-Centric State Updates · p. 2
  • Observations are normalized in the shared map frame, then matched to the nearest class-compatible record within an association radius or assigned a new identifier.The gate preserves class identity while allowing unseen targets to enter memory during execution.Source: B. Object-Centric State Updates · p. 3
  • Accepted perceptual evidence revises pose, confidence, time, and source, while verified skill outcomes advance task status; observations cannot demote verified grasp or delivery.Failed skills mark an entity for reconsideration without discarding completed work.Source: B. Object-Centric State Updates · p. 3
  • Replanning is required exactly when changed fields overlap fields used by arguments or preconditions in the remaining plan.A revised target pose invalidates navigation to it, whereas an unrelated detection does not interrupt the active skill.Source: B. Object-Centric State Updates · p. 3

C. Attention-Enhanced Dynamic Memory Retrieval

ADM scores candidate object locations using decayed observation support and distance to the robot, then exposes a bounded set of current records for planning.

C. Attention-Enhanced Dynamic Memory Retrieval

  • The method maintains candidate-location support separately for each active entity and updates it when positive observations are accepted.The observations are matched to an existing record or assigned a new one before the update.Source: C. Attention-Enhanced Dynamic Memory… · p. 3
  • Past pose hypotheses are discounted over time, while accepted new evidence adds support at its reported location.Remote observations below a confidence threshold are rejected; verified onboard observations can reset pose belief, and verified absence deletes a candidate.Source: C. Attention-Enhanced Dynamic Memory… · p. 3
  • At planning time, delivered and carried entities are excluded, and the remaining entities are reduced to one record each.For each remaining entity, the selected pose is the one with maximum decayed support.Source: C. Attention-Enhanced Dynamic Memory… · p. 3
  • The retrieval score favors reliable nearby targets, with recency already encoded in the decayed confidence.The selected records are ordered by score, and the dynamic portion of the planner context is bounded by K.Source: C. Attention-Enhanced Dynamic Memory… · p. 3

D. LLM-Guided Skill Planning and Execution

The LLM converts a structured query of current task state into a typed skill sequence, while validation and execution feed outcomes back into memory. Only the next skill is committed, and the remaining plan is retained while valid.

D. LLM-Guided Skill Planning and Execution

  • The query combines the goal, robot and execution state, static rules and skills, retrieved active records, and a compact summary of terminal entities.Entity identifiers and poses come from memory rather than free-form scene text.Source: D. LLM-Guided Skill Planning and… · p. 3
  • The LLM produces a finite sequence of skills with typed arguments, choosing a target, compatible station, and skill order from the current query.If an object is already held, the plan omits the pickup prefix; the LLM does not generate actuator commands or directly edit memory.Source: D. LLM-Guided Skill Planning and… · pp. 3–4
  • A deterministic validator checks skill names, argument types, entity references, carrying capacity, target status, and object–station compatibility before dispatch.The low-level controller handles trajectory generation, navigation, collision avoidance, and grasp control.Source: D. LLM-Guided Skill Planning and… · p. 4
  • Only the next validated skill is committed; its observed outcome updates dynamic memory before the following skill.The remaining plan suffix is kept while preconditions hold, but cleared after failure or when new evidence affects the plan, prompting replanning from achieved state.Source: D. LLM-Guided Skill Planning and… · p. 4

V. EXPERIMENTS

The experiments move from controlled simulation to physical full-system validation, progressively testing memory updates, retrieval, and execution.

V. EXPERIMENTS

  • The evaluation proceeds from controlled simulation through deployed-planner testing and higher-fidelity execution to physical full-system validation.The sequence is designed to isolate the core mechanism before testing the complete system.Source: §V · p. 4
  • The experiments ask whether dynamic updates recover from stale state and bounded retrieval preserves that benefit without unbounded context.Source: §V · p. 4
  • The evaluation also tests whether plans produced by the approach remain executable.Source: §V · p. 4

A. Task-Level Memory Evaluation

In the 14-container noisy dynamic scenario, attention memory reaches 100% success while substantially reducing context relative to unfiltered dynamic memory. The evaluation uses matched event tapes and a deterministic policy to isolate memory effects.

A. Task-Level Memory Evaluation

  • The simulator tests 6, 10, and 14 containers on progressively larger grids, with relocation, false reports, latency, and detection varied across scenarios.Step budgets are 80, 150, and 240, respectively.Source: A. Task-Level Memory Evaluation · p. 4
  • 100% success for attention memory in DN-14 compares with 97% for unfiltered dynamic memory and 62% for static memory; context falls 95.8% versus unfiltered memory.Attention gains 38 points over static memory (95% CI: 28–48); its three-point gain over unfiltered memory is inconclusive (95% CI: 0–7), and it matches oracle success.Source: Table I · p. 4, A. Task-Level Memory Evaluation · p. 4
  • Five memory conditions are compared on identical event tapes across 100 seeds per scenario, using a deterministic nearest confidence-weighted policy.Context cost is a size proxy, not tokenization; success intervals use Wilson estimates and paired differences use bootstrap intervals.Source: A. Task-Level Memory Evaluation · p. 4
  • The confidence–distance heuristic may suppress a distant entity needed for a future subgoal, and parameter calibration and individual effects remain open.Fixed K, source weights, and confidence and decay thresholds require calibration; individual effects remain to be ablated.Source: A. Task-Level Memory Evaluation · pp. 4–5

B. Live LLM Planning and Trajectory Analysis

The live-planner evaluation compares two dynamic-memory variants using identical event tapes and a frozen GPT-5 Mini snapshot. Both complete every mission, while ADM uses fewer input tokens and planner calls.

B. Live LLM Planning and Trajectory Analysis

  • The comparison used a frozen gpt-5-mini-2025-08-07 snapshot, strict structured output, and three paired seeds per dynamic variant with identical event tapes.The 6×6, six-container scenario used a 110-step budget, two-step latency, and specified relocation, detection, and false-report rates.Source: B. Live LLM Planning and Trajectory… · p. 5
  • Across six live GPT-5 Mini episodes, both dynamic-memory variants completed every mission, while ADM reduced provider-reported input tokens from 2,983 to 2,553 (14.4%) and planner calls from 7.0 to 6.0.Mean request latency changed only slightly, from 2.27 to 2.24 s, and was treated as comparable rather than a runtime benefit.Source: B. Live LLM Planning and Trajectory… · p. 5
  • Because both variants used the same structured schema, the input-token reduction shows that bounded retrieval remained effective under provider tokenization.This supports the context-size proxy comparison beyond its own size-based accounting.Source: B. Live LLM Planning and Trajectory… · p. 5

C. Rigid-Body Execution Validation

Rigid-body trials show that dynamic memory preserves task completion under contact and collision constraints, while attention reduces stale-target visits and context cost. Physical demonstrations also complete missions with online-revealed targets across varied layouts, with limits around uncertainty and communication still to be tested.

C. Rigid-Body Execution Validation

  • The rigid-body test used PyBullet in a 3.5-m arena with a mobile base, articulated arm, contact-gated grasps, two stations, six containers, and 20 paired seeds per method.The simulation supports feasibility under physical constraints, not calibrated sim-to-real prediction.Source: C. Rigid-Body Execution Validation · pp. 5–6
  • Across 20 rigid-body trials, both dynamic variants completed all 20 (100%; 95% CI: 83.9–100%), compared with static memory’s 10 of 20 (50%; 29.9–70.1%).Source: C. Rigid-Body Execution Validation · p. 5, Figure 4 · p. 6
  • Attention reduced the context proxy from 3,976 to 1,035 (74.0%), targeting informational efficiency and stale-state failures rather than motion optimality, without improving every execution cost.Successful-trial time/path were 232.6 s/17.1 m for attention versus 208.3 s/16.2 m for unfiltered memory.Source: C. Rigid-Body Execution Validation · p. 6

D. Physical System Validation

  • The physical setup paired a JetRover mobile manipulator with a Tello drone, whose aerial detections entered the shared map and could trigger replanning with the frozen GPT-5 Mini.The ground robot used LiDAR mapping and localization, an RGB-D camera for grasping, and onboard ROS perception and control.Source: D. Physical System Validation · pp. 6–8
  • Each physical mission involved five containers, one initially known and four revealed online, with varied poses, obstacles, connectivity, lighting, and dynamically assigned stations.Completion required every container at its assigned station and no object held by the robot.Source: D. Physical System Validation · p. 8
  • All physical configurations completed collection while incorporating online reveals, demonstrating the update-replan loop across the tested layouts.Localization error may create duplicate or misplaced records; uncertainty-aware association, active observation requests, communication loss, and longer missions remain future-study topics.Source: D. Physical System Validation · p. 8

VI. CONCLUSION

The conclusion frames long-horizon LLM planning as a memory problem: state must stay current while remaining selective. ADM-Planner maintains separate persistent and dynamic memory, retrieves plan-relevant records, and replans when new evidence affects the remaining skills; experiments across increasing realism support its use between asynchronous perception and mobile-manipulator planning.

VI. CONCLUSION

  • Long-horizon LLM planning requires state that remains current enough to react to change and selective enough to support the next decision.The paper identifies this as the core memory problem for robot missions.Source: §VI · p. 8
  • ADM-Planner separates persistent workspace knowledge from object-centric dynamic state, retrieves bounded plan-relevant records, and replans when new evidence affects the remaining skill sequence.The framework is presented as an interface between asynchronous perception and long-horizon planning.Source: §VI · p. 8
  • Controlled simulation, live-LLM, rigid-body, and physical experiments support attention-enhanced dynamic memory as a practical interface for mobile-manipulator planning.The experiments increase in realism while testing the same memory mechanism.Source: §VI · p. 8

Related papers

Loading 2609.29212v1…