Source-linked AI summary
2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation
Yutong Hu, Fengjiao Chen, Xuezhi Cao, Renaud Detry
TL;DR
Long-horizon manipulation needs task memory, but it is unclear whether that memory must reside inside the action policy and how precisely an Agent should communicate physical intent. 2AM keeps memory in a multimodal Agent and uses a single RGB, episodically stateless Action Model steered by subtask language and optional 2D hints. On LIBERO-Mem, it reaches 76.3% completion, while the paper concludes that grounded steering helps translate remembered intent into action but exact stopping remains unresolved.
Problem
Existing agentic manipulation systems blur whether performance comes from policy capability, richer sensing, alternative motor tools, or an under-specified language interface.
Method
2AM gives task memory to a multimodal Agent and task-relevant motion to one RGB-based, episodically stateless Action Model connected by subtask language and optional 2D grasp, place, and move hints.
Results
76.3% completion is achieved on LIBERO-Mem without depth, online geometry, or planner-based object motion, with 63.0% relaxed and 11.8% strict success.
Takeaways & Limitations
Grounded steering helps an Agent turn remembered intent into action, while Action Model capability depends on how precisely the Agent asks it to act.
Takeaways & Limitations
Evidence is limited to a single simulated benchmark and task memory whose current consequence can be grounded in RGB; strict termination remains a separate bottleneck.
Abstract
from arXiv · showhide
Long-horizon robot manipulation requires memory, but not necessarily inside the action policy. To address such tasks, current agentic systems often combine VLAs with planners and geometric tools, sometimes using additional depth or calibrated geometry. These systems confound attribution: gains may come from richer observations or alternative motor tools, while failures may stem from either the policy or an under-specified language interface. We isolate this question through a deliberately constrained design: less tool breadth, but greater interface bandwidth. 2AM makes a multimodal Agent the sole holder of task memory and a single RGB-based, episodically stateless Action Model the sole executor of task-relevant motion. The Agent compiles interaction history into subtask language and optional 2D grasp, place, and move hints that bind its physical intention at different time scales. To teach this steerability to the VLA, we augment demonstrations with structured hint labels and train under condition dropout, spatial noise, and temporal jitter to tolerate imperfect Agent outputs. On LIBERO-Mem, without depth, online geometry, or planner-based object motion, 2AM reaches 76.3% average completion, a 61.5-point improvement over the strongest reported baseline of 14.8%, together with 63.0% relaxed and 11.8% strict success. These results show that task memory can remain Agent-side. They further show that Action Model capability depends not only on what the policy has learned, but on how precisely the Agent can steer it.
1 Introduction
2AM isolates task memory in a multimodal Agent and task-relevant motion in a single RGB, episodically stateless Action Model, connected by language and optional 2D steering hints. On LIBERO-Mem, this constrained design improves completion while showing that interface bandwidth is central to translating remembered intent into action.
- Motivation: The design addresses attribution problems in agentic manipulation by reducing tool breadth and sensing while increasing interface bandwidth.Broader systems may confound gains from richer observations or alternate motor tools with policy capability, while language-only commands can omit resolved physical arguments.
- Design: 2AM assigns task memory exclusively to the Agent and all task-relevant motion to one RGB-based, episodically stateless Action Model.The Action Model receives current RGB and robot state without task history, while the Agent consolidates history and re-steers through fresh observations.
- Interface: Subtask language plus optional 2D grasp, place, and move hints provide a compositional interface for grounding remembered physical intent.The hints bind physical arguments at different time scales while guiding rather than prescribing approach, contact, transport, and release.
- Results: 76.3% completion is achieved on LIBERO-Mem without depth, online geometry, or planner-based object motion, improving over an aligned π0 reproduction.The introduction reports a 5.5-point completion improvement and a 25.6-point relaxed-success improvement, with comparable strict success.
- Training: Condition dropout, spatial noise, and temporal jitter train the Action Model to tolerate imperfect Agent steering.The supervision is recovered from demonstrations and structured around robust Agent-to-Action Model communication.
2 Contract between Agent and Action Model
2AM separates long-horizon task memory from local control: the Agent consolidates history into a steering command, while an episodically stateless Action Model executes motion from current visual and robot-state inputs. Language and optional 2D hints expose object, destination, and near-term movement arguments without transferring history or prescribing trajectories.
- 2 Contract between Agent and Action Model: 2AM assigns memory and decision-making to the Agent while one Action Model executes all task-relevant motion.The Action Model is RGB-based and episodically stateless, with no persistent episode state across Agent calls.
- 2 Contract between Agent and Action Model: The deployment contract excludes depth, online geometry, object poses, backprojection, and analytic object-specific motion from the task-solving path.Every task-relevant movement must pass through the same Action Model.
- 2 Contract between Agent and Action Model: The Agent consolidates task history into a steering command, while the Action Model receives current visual inputs, robot state, and that command to predict an action chunk.The Agent updates once per short action chunk; the Action Model performs local control within the chunk.
- 2 Contract between Agent and Action Model: The steering command contains local language plus optional normalized 2D grasp, place, and move hints, rather than end-effector poses, gripper commands, durations, or trajectories.These inputs guide the Action Model, which retains responsibility for approach geometry, grasping, transport, collision avoidance, and release.
- 2 Contract between Agent and Action Model: Grasp, place, and move hints operate at complementary temporal scales: grasp selects the current object, place preserves the destination commitment, and move supplies a near-term direction cue.Fields can appear or disappear by stage, and the move hint is not a required waypoint or trajectory.
3 Training-time supervision for inference-time contract usage
2AM derives structured steering supervision from offline demonstrations and trains one Action Model to tolerate incomplete, noisy, and temporally misaligned Agent commands. The Agent refreshes those commands after each action chunk while the controller uses surviving hints, RGB, and robot state.
- 3.1 Recovering supervision without a model teacher: Structured annotations recover subtask, grasp, place, and move supervision from synchronized demonstrations without a model teacher.The trajectories provide RGB, robot state, actions, stages, object annotations, gripper contact, and object motion.
- 3.1 Recovering supervision without a model teacher: 2AM derives grasp and place targets from visible object annotations while treating them as semantic targets rather than optimal geometric contact points.The learned controller must infer feasible contact from images, state, and demonstrations.
- 3.1 Recovering supervision without a model teacher: Future-window sampling makes move hints robust to variable Agent latency instead of tying them to one exact temporal offset.The sampled index is clipped at episode boundaries.
- 3.1 Recovering supervision without a model teacher: Action labels preserve continuous motion across annotated subtask boundaries by using next-frame end-effector poses and clipping windows only at episode boundaries.This avoids teaching the motor policy to pause for the Agent near grasp, lift, and release.
- 3.2 Training for imperfect steering: Condition dropout, spatial noise, and temporal jitter expose the Action Model to omission, localization, and timing errors in Agent steering.Fields are independently masked while at least one intent source remains, and surviving points receive coordinate noise.
- 3.2 Training for imperfect steering: After each action chunk, the Agent re-observes global RGB, updates task history, and may revise the subtask or any hint.The benchmark does not expose state or a learned completion oracle during this loop.
4 Experiments
2AM evaluates a constrained RGB-only Agent–Action Model system on LIBERO-Mem, comparing aggregate performance, interface bandwidth, and task-level progression. The results show stronger completion and relaxed success than an aligned π0 reproduction, but not uniformly higher strict success.
- Benchmark setup: At deployment, the system uses task language, agent-view and wrist RGB, and robot state while excluding depth, poses, oracle subgoals, rewards, and stage flags.Strict success requires ordered completion and the requested repetition count; relaxed success permits later overshoot.
- Aggregate results: 76.29% completion and 63.00% relaxed success exceed the aligned π0 reproduction by 5.50 and 25.58 percentage points, while strict success is slightly lower at 11.83% versus 12.25%.The comparison uses an in-house π0 reproduction rather than weaker published baselines.
- Interface ablation: Grounded hints raise strict success by 4.58, relaxed success by 43.58, and completion by 22.57 percentage points over language-only steering.Agent-side memory and the motor stack are held fixed in this ablation.
- Task-level results: 2AM achieves higher relaxed success on all ten tasks and higher completion on seven, especially on later tasks where relation and occlusion increase object and destination-binding demands.The reproduced π0 has higher strict success on T1–T6, whereas 2AM is stronger on T7–T10.
- Steering errors: Figure 3 shows that localization, stage-selection, and object-binding errors enter through the same Agent-to-Action Model steering interface.The examples mark grasp and place hints alongside incorrect subtasks.
5 Related Work
Related work spans agentic systems that select skills, ground plans spatially, and connect memory to execution policies. 2AM instead studies whether Agent-side memory can become actionable through one compositional command with language and spatial arguments.
- Agentic VLA systems: Prior systems combine language-model planning, executable programs, 3D value maps, keypoints, learned primitives, structured memory, and low-level VLA commands.These approaches distribute planning, grounding, memory, and execution across richer toolchains.
- Grounded interfaces: 2AM’s controlled question is whether Agent-side memory can steer one Action Model through language plus independently available grasp, place, and move arguments.The study also tests tolerance to missing and noisy online combinations.
- Memory and execution boundaries: MemoryVLA, Embodied-SlotSSM, RoboMME, and related methods place visual, object-centric, symbolic, or recurrent memory partly inside execution policies.MemER is structurally closest to 2AM but communicates retrieved task information to π0.5 through text instructions.
6 Discussion and Limitations
The evidence is a controlled LIBERO-Mem case study, not a claim of universal long-horizon competence. Its RGB-grounded 2D interface leaves hidden geometry, force, continuous motion history, hint-specific effects, stronger Agents, and latency trade-offs unresolved.
- Scope: The evidence is limited to one simulated benchmark and task memory whose current consequence can be grounded in RGB.The authors explicitly frame the result as a controlled study of the Agent–Action Model interface.
- Interface limits: A 2D interface cannot expose hidden geometry, force, or continuous motion history, and strict termination remains a separate bottleneck.High completion alone is therefore not evidence that memory has been solved.
- Open limitations: The study has not isolated each hint’s contribution, compared stronger Agent backbones, or measured capability–latency trade-offs across steering frequencies.The current Agent is Qwen3.8-27B.
- Future validation: Future work will test the same RGB-only interface across multiple memory benchmarks and on real robots without privileged geometry or scripted object motion.The proposed benchmarks include LIBERO-Mem, RMBench, RoboMemArena, and RoboMME.
7 Conclusion
2AM keeps persistent task memory in the Agent and uses a compositional steering contract to guide one RGB Action Model. Grounded steering improves remembered intent-to-action progression, while exact stopping remains unresolved.
- 2AM places persistent task memory in an Agent, continuous motion in one RGB Action Model, and a compositional steering contract between them.
- Grounded steering helps the Agent turn remembered intent into action, improving task progression over language-only control on LIBERO-Mem.
- Exact stopping remains unresolved, and Action Model capability depends on how precisely the Agent asks it to act.