PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Agentic Memory for Generalist Robot Policies

Yuyou Zhang, Yunbei Zhang, Miao Li, Janet Wang, Zijian Jin, Shilong Liu, Ding Zhao

arXiv:2609.36595v1cs.ROcs.AI

TL;DR

Robot control needs interaction-derived state that may not appear in the current view. SimpleARM maintains and retrieves task-relevant state for frozen robot policies; across 16 RoboMME tasks, it achieves 67.17% mean success versus 44.51% for the strongest previously reported non-oracle method.

  • Problem

    Robot control over longer, compositional tasks requires remembering observations, action effects, and unfinished work, including information not represented in the current view.

    Source: §1 · p. 1

  • Method

    SimpleARM specifies task-relevant state, maintains it online, and uses structured retrieval and current-view grounding to expose it to a frozen policy.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across all 16 RoboMME tasks, SimpleARM achieves 67.17% mean success over three seeds, versus 44.51% for the strongest previously reported non-oracle method.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support maintaining compact task-relevant state derived from interaction history rather than relying on retained visual history alone.

    Source: §5 · p. 10

    RoboMME evaluates memory jointly with other capabilities, including temporal perception, fine-grained imitation, and execution.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Visual-memory systems commonly retain or compress past observations. Robot control additionally requires interaction-derived state that no individual frame may explicitly represent, such as persistent identity relations, accumulated progress, or ordered procedures. We introduce Simple Agentic Robot Memory (SimpleARM), a training-free memory layer for frozen generalist robot policies. From the task instruction, SimpleARM specifies what to monitor; frozen perceptual tools maintain compact typed state online; structured access retrieves that state only when a proposed subgoal depends on history; and current-view grounding resolves recalled entities before execution. We evaluate SimpleARM on RoboMME, a benchmark of memory-dependent robot manipulation tasks that require history information no longer available in the current observation. Across all 16 tasks and three policy seeds, SimpleARM achieves 67.17% mean success, compared with 44.51% for the strongest non-oracle baseline. Matched ablations show mechanism specificity: removing relation, reference, progress, or route state produces large losses where the affected state is retrieved for control, while largely sparing other tasks. These results support a state-based view of robot memory: effective memory for control is not simply retained visual history, but compact task-relevant state derived from the interaction history.

1 INTRODUCTION

Robot memory must preserve task-relevant state arising from action–observation history, not just selected visual evidence. SimpleARM maintains and retrieves that state for frozen robot policies, achieving higher mean success than the supplied baselines on RoboMME.

1 INTRODUCTION

  • Under partial observability, control-relevant state must summarize past actions as well as observations because needed information may not appear in any single frame.The interaction history can determine identity, progress, or route information that selected frames alone do not recover.Source: §1 · pp. 1–2
  • SimpleARM uses instruction-conditioned specifications and frozen perceptual tools to maintain typed state, retrieving history-dependent fields and re-grounding recalled entities before execution.The state includes entity–relation bindings, event progress, and ordered trajectories or procedures.Source: §1 · p. 2
  • The paper frames robot memory as task-conditioned maintenance of policy-sufficient state over action–observation history, rather than visual-history compression alone.SimpleARM combines compact online state with structured retrieval and current-view grounding for a frozen policy.Source: §1 · p. 2
  • 67.17% mean success across 16 RoboMME tasks and three seeds exceeds the 44.51% strongest previously reported non-oracle method and 32.70% no-memory baseline.Matched ablations show large losses when relation, reference, progress, or route state is removed where retrieved for control, while largely sparing other tasks.Source: §1 · p. 2

2 RELATED WORK

Related work spans visual-history retrieval, online video memory, and memory-augmented robot policies. SimpleARM instead maintains task-conditioned state for control and re-grounds recalled entities in the current view.

2.1 AGENTIC AND ONLINE VIDEO UNDERSTANDING

  • Robot control couples perception to action, so memory must provide control-relevant variables and re-ground recalled identities before action, not only return historical evidence.This contrasts with video-memory approaches organized around selecting evidence from visual history.Source: §2.1 · pp. 2–3

2.2 MEMORY FOR VLAS

  • Memory-augmented VLA methods use varied representations, while SimpleARM maintains instruction-conditioned entity–relation, progress, and trajectory state with both the subgoal predictor and VLA frozen.Its structured retrieval is limited to compatible history-dependent subgoals, with recalled entities re-grounded in the current view.Source: §2.2 · p. 3

2.3 SELECTIVE MEMORY FOR GROUNDED ACTION

  • SimpleARM maintains episode-specific task state with fixed model parameters and prompt templates; closed-loop control also requires tracking action-induced scene changes.The paper relates selective structuring of identity, progress, and trajectory variables to prior work on visual working memory and event models.Source: §2.3 · p. 3

3 METHOD

SimpleARM maintains compact interaction state selected by the task, then retrieves it selectively to condition a frozen robot policy when a subgoal depends on history. It separates historical state from current-view grounding and execution.

3.1 POLICY-SUFFICIENT STATE

  • SimpleARM maintains policy-sufficient interaction state—not merely past frames—and adds it to frozen hierarchical control through specification, online maintenance, selective retrieval, and grounding.The state can encode persistent entity relations, accumulated progress, and ordered procedures.Source: §3.1 · p. 4

3.2 TASK-CONDITIONED STATE ABSTRACTION

  • The instruction selects candidate relation, progress, and procedure variables, while frozen tools establish and update their values from interaction evidence.The factorization retains only variables selected as potentially relevant to future control, rather than serving as a complete world model.Source: §3.2 · pp. 4–5

3.3 SELECTIVE MEMORY RETRIEVAL AND CURRENT-VIEW GROUNDING

  • Retrieval depends jointly on the task specification and proposed subgoal; compatible history resolves identity, progress, or procedure, with current-view grounding when localization is required.Memory modifies the proposal only when compatible historical state is available and any required grounding succeeds.Source: §3.3 · p. 6

3.4 TRAINING-FREE INTEGRATION

  • The training-free integration keeps the subgoal predictor and VLA frozen, uses episode-scoped non-parametric memory, and resets records between episodes.Frozen perception and event tools maintain state, and current-view perception grounds recalled entities before execution.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare SimpleARM with retained-frame and no-memory baselines across RoboMME, then use matched ablations and failure analyses to examine when maintained state helps. The results distinguish task-specific state use from challenges involving perception, identity, imitation, and control.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers all sixteen RoboMME tasks across three seeds, with frozen subgoal and VLA policies and no memory training or test-time optimization.Baselines include published RoboMME results and Recent32+ModuL, using the latest 32 frames.Source: §4.1 · pp. 6–7

4.2 RQ1: WHEN DOES STATE MAINTENANCE IMPROVE OVER VISUAL-HISTORY RETENTION?

  • On RoboMME, SimpleARM achieves 67.17% average success, exceeding the strongest FrameSamp baseline by 22.66 points and π0.5 without memory by 34.5 points.The evaluation uses three policy seeds and 50 episodes per task per seed.Source: §4.2 · p. 7, Table 1 · p. 7
  • SimpleARM beats FrameSamp by 53.56 points on Permanence and 42.17 on Reference, but trails by 8.22 on Counting.It improves Imitation by 3.11 points; excluding StopCube, Counting averages 76.0% for SimpleARM and 73.0% for FrameSamp. Figure 2 compares systems on matched episodes and reports gains concentrated in Permanence and Reference.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 RQ2: WHICH COMPONENTS OF STATE MAINTENANCE MATTER?

  • Removing route state reduces PatternLock and RouteStick success by 68.0 points, while removing progress counting reduces SwingXtimes by 62.0 points.Demonstration-reference state removal lowers its three corresponding tasks by 44.7 points on average; unrelated tasks are largely spared.Source: §4.3 · p. 8, Figure 3 · p. 9
  • Removing identity tools cuts success by 49.0 points when bindings are retrieved, versus a +1.5-point change without retrieval; replacing structured retrieval lowers overall success by 11.9 points.The retrieval replacement changes success by −17.2 points on tasks with retrievable history and −3.0 on remaining tasks; free-form retrieval can expose irrelevant history and overwrite subgoals resolved from the current view.Source: §4.3 · p. 8

4.4 RQ3: WHAT CHALLENGES REMAIN BEYOND MEMORY?

  • 82/88 SwingXtimes episodes with exact counts succeed, while PickHighlight falls from 74/75 successes with one initialized binding to 2/27 with four or more.ButtonUnmask and ButtonUnmaskSwap failures often end with every carrier internally accepted, which does not establish ground-truth identity; StopCube, imitation, and execution also expose challenges beyond memory.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

5 CONCLUSION AND LIMITATIONS

SimpleARM maintains task-conditioned state online and selectively retrieves it for frozen policies. RoboMME results and matched ablations support state-based memory for episode-level hierarchical manipulation, while real-robot evaluation and broader state schemas remain future work.

5 CONCLUSION AND LIMITATIONS

  • SimpleARM achieves 67.17% success on RoboMME, and matched ablations find relation, reference, progress, and route state matter primarily when retrieved for control.The method leaves the subgoal predictor and VLA frozen and re-grounds recalled entities before execution.Source: §5 · p. 10
  • The paper frames robot memory as task-conditioned maintenance of policy-sufficient state rather than reliance on retained visual history alone.SimpleARM specifies candidate state variables, maintains them online, and retrieves them through structured access.Source: §5 · p. 10
  • The method targets episode-level state maintenance for hierarchical manipulation, not high-frequency motion estimation or low-level control.The authors identify additional policy-relevant variables and real-robot evaluation as next steps.Source: §5 · p. 10

A ADDITIONAL METHOD AND IMPLEMENTATION DETAILS

The appendix details how SimpleARM specifies what to monitor, maintains typed state, and uses fixed tools in RoboMME. It separates language-based specification from online evidence and policy-facing retrieval.

A.1 INSTRUCTION-CONDITIONED SPECIFICATION AND ACCESS

  • Four instruction-only queries specify candidate state and retrieval operations before control; perceptual and event tools establish stored values, while structural tests decide whether history is read.The plan’s coarse NEED field is affirmative in all 2,400 episodes and does not gate writes; WATCH phrases and observable predicates discriminate what to monitor.Source: §A.1 · p. 15

A.2 TYPED SCHEMAS, VERIFICATION, AND STATE LIFECYCLE

  • Episode-scoped typed records are admitted only after type-specific evidence, and entity records move through candidate, verified, uncertain, and reacquired states.Entity reads require current-view localization before modifying a subgoal; memory abstains when compatible fields, verified state, or safe grounding are absent.Source: §A.2 · p. 15, Table 2 · p. 16
  • Progress counts summarize verified event transitions, while trajectory and procedure state preserve ordered elements and a current progress index.The same observation or waypoint can imply different behavior at different positions in an ordered procedure.Source: §A.2 · p. 16

A.3 ROBOMME TOOL INSTANTIATION

  • RoboMME uses frozen detection, feature, optical-flow, mask, and event tools; recovery widens the search at most three times, and fixed settings persist through evaluation.The method uses ten text-only language-model calls before the first action and averages 2.02 seconds per episode for perception tools.Source: §A.3 · p. 16

B EVALUATION PROTOCOL

The protocol combines a three-seed, sixteen-task evaluation with matched episode comparisons and single-seed ablations. It also records limits on what the diagnostics and pooled effects support.

B EVALUATION PROTOCOL

  • The final benchmark uses three policy seeds and 50 episodes per task and seed; the sixteen-task mean is 67.17% with a 1.23-point seed-to-seed standard deviation.A separate comparison evaluates the released FrameSamp-Modul checkpoint on the same 800 episode identities, controlling episode difficulty but not same-session execution.Source: B EVALUATION PROTOCOL · p. 17
  • Recent32 succeeds on 254, 254, and 248 of 800 episodes across three evaluation rounds, averaging 31.50 ± 0.43%.The rounds repeat one trained model rather than three training runs, and host effects are not separated from seed effects.Source: B EVALUATION PROTOCOL · p. 17
  • Each of ten final-method ablations uses seed 11 and matches 50 episodes per task to full-method controls with identical task and episode identifiers.Effects are interpreted through their size and concentration on predicted-active tasks, not as multi-seed replication.Source: B EVALUATION PROTOCOL · p. 17
  • Diagnostics cannot visually certify every accepted identity or uniquely attribute residual failures to grounding versus downstream control.Pooled effects near run-to-run variability are interpreted only when localized mechanism and matched task signature support them.Source: B EVALUATION PROTOCOL · p. 17

C.1 TASK-STATE ANNOTATION

The task-state annotation maps task semantics to candidate history variables, while the episode-matched comparison evaluates SimpleARM against FrameSamp on shared episodes. The annotation is interpretive rather than causal evidence.

C.1 TASK-STATE ANNOTATION

  • Task-state labels come from task semantics and may overlap; ButtonUnmaskSwap combines persistent relation with progress, while VideoPlaceButton combines historical identity with event-relative order.MoveCube and InsertPeg require manipulation choices not encoded by route state.Source: §C.1 · pp. 17–18, Table 5 · p. 19
  • Recent-frame baselines use the latest 4, 16, or 32 native frames, with task cells reporting three-seed mean success.Each variant was trained once and evaluated with policy seeds 7, 11, and 23; Recent32 is the variant reported in Table 1.Source: Table 3 · p. 18
  • Table 4 reports per-seed success percentages for the final method, with each seed covering 50 episodes for each of sixteen tasks and the last column giving sample standard deviation across seeds.Source: Table 4 · p. 18
  • The task-state annotation predicts where explicit state may help but is not causal evidence; the all-task ablations provide causal attribution.VideoUnmask and BinFill show gains despite weak sensitivity to the tested ablations, and PickHighlight’s mark pathway has no removal ablation.Source: §C.1 · p. 19

C.2 EPISODE-MATCHED FRAMESAMP COMPARISON

  • +21.6 points separated SimpleARM from FrameSamp in the episode-matched comparison, with 254 ours-only versus 81 FrameSamp-only successes.The runs shared episode identities but were executed on different hosts; the suite comparison showed +54.5 points on Permanence and +40.0 on Reference.Source: §C.2 · p. 19, Table 6 · p. 19

D FULL MECHANISM ABLATIONS

Matched ablations test the contribution and task specificity of SimpleARM’s mechanisms across sixteen tasks. Large losses cluster on tasks where the removed state is used, while isolated cells require care against run-to-run variability.

D FULL MECHANISM ABLATIONS

  • All ten ablations cover sixteen tasks and 800 matched episodes; active task sets were specified from the state or access mechanism exercised by the final method.Figure 5 plots ablated-minus-full success with paired-bootstrap 95% intervals; its lower three rows are robustness controls, not typed-memory mechanisms.Source: D FULL MECHANISM ABLATIONS · p. 20, Figure 5 · p. 21
  • Route removal cuts PatternLock and RouteStick success by 90 and 46 points, while removing the flash count cuts SwingXtimes by 62 points.The matrix also reports 34–52-point losses on demonstration-reference tasks, 64- and 42-point losses on the ButtonUnmask pair without live binding, and a 50-point loss on VideoUnmaskSwap without demonstration cover-following; off-target changes are much smaller.Source: D FULL MECHANISM ABLATIONS · p. 20, Table 9 · p. 21
  • A −16-point off-target cell occurs where route state is never formed, so it cannot reflect removing that state.Single-cell effects of this size should be read against the run-to-run variability band rather than alone.Source: D FULL MECHANISM ABLATIONS · pp. 20–21

D.1 ACTIVITY, ACCESS, AND COMPILED READERS

Mechanism activity alone does not establish a contribution to control. The results distinguish mechanisms that are exercised from state that is actually retrieved and allowed to affect a decision.

D.1 ACTIVITY, ACCESS, AND COMPILED READERS

  • Demonstration notes are written in 92–96% of VideoPlaceButton and VideoPlaceOrder episodes but are never read, and removing them changes neither task.Carrier bindings and flash counters can also be active without corresponding effects; activity confirms exercise, not contribution to control.Source: §D.1 · p. 21, Table 10 · p. 22
  • Structured access holds entity reads to 0.0 on BinFill, PickXtimes, and SwingXtimes, versus 37.2, 35.5, and 29.0 under agent-decided access.On worsened SwingXtimes pairs, a free-form reader exposes a plausible carrier track to pick-and-move subgoals, replacing current-view coordinates and disrupting the subsequent count.Source: §D.1 · p. 21
  • Compiled retrieval programs affect control only when the structural gate finds a compatible history-dependent field; the evidence does not support perfect compiler routing.Programs work on PickHighlight, VideoRepick, and VideoPlaceButton, while inert or fallback outputs occur on other tasks.Source: §D.1 · p. 22
  • Removing the frozen feature anchor and optical-flow continuity check causes large losses when maintained bindings are retrieved, but almost no change when they are not.The comparison retains the color verifier, and SAM is absent from both arms.Source: §D.1 · p. 22, Table 11 · p. 22

D.2 ROBUSTNESS CONTROLS

Supplementary robustness comparisons indicate sensitivity to instruction rewording and other configuration changes, but are separate from the final matched ablations. A no-flow control without SAM does not establish flow’s necessity in the final tool stack.

D.2 ROBUSTNESS CONTROLS

  • Forcing a planning vote changes the pooled score by −3.0 points and refining every verb by −2.0, differences not treated as core mechanisms.Refining button-press coordinates nevertheless produces localized 20- and 22-point losses on ButtonUnmask and ButtonUnmaskSwap through extra composer calls and delay before the historical-container read.Source: §D.2 · p. 22
  • Instruction rewording produces the largest sensitivity; supplying the evidence source from the environment reduces, but does not eliminate, that loss.Detector substitution, write-policy changes, removing the write gate, and loosening structural access have smaller effects; these use a distinct evaluation configuration.Source: §D.2 · p. 23, Table 12 · p. 23

D.3 OPTICAL-FLOW CONTROL UNDER A DIFFERENT TOOL STACK

  • The no-flow control ran without SAM, unlike the final method, so its localized loss does not establish flow’s necessity when SAM is available.A matched no-flow run with SAM available would be needed for that claim.Source: §D.3 · p. 23, Table 13 · p. 23

E.1 REPRESENTATIONAL COVERAGE

  • Table 14 summarizes which representative tasks have required control variables in the current schema, interpreting task requirements alongside matched ablations and oracle headroom.It is not an additional performance comparison.Source: §E.1 · p. 23

E.2 MEMORY-STATE ATTAINMENT ACROSS TASKS

The analyses describe where memory state is attained, how state correctness and task difficulty relate to outcomes, and what the episode traces reveal about memory-to-action operation. They also identify limits on causal attribution, including pathways not isolated by current ablations.

E.2 MEMORY-STATE ATTAINMENT ACROSS TASKS

  • In 12 of 16 tasks, successful and failed episodes receive essentially the same memory-attainment label; 559/788 failures occur on tasks with a constant label.These episode-level totals show the furthest attained state or access event, not when or why failure occurs; VideoPlaceButton and VideoPlaceOrder use a separately reported specialized reference path.Source: §E.2 · pp. 23–24, Table 15 · p. 24

E.3 STATE CORRECTNESS AND BINDING COMPLEXITY

  • For SwingXtimes, all 17 episodes with an under-counted state fail, while 82/88 episodes with the exact count succeed; matched no-count ablation evidence links missed event detections to failure without showing exact counts are sufficient.For PickHighlight, success counts decline as initialized bindings increase: 74/75, 18/24, 5/24, and 2/27 for one, two, three, and at least four bindings; this trend is diagnostic, without a matched mark-memory ablation.Source: §E.3 · p. 24
  • On ButtonUnmask and ButtonUnmaskSwap, failures average more rejected tracks and exhausted recoveries than successes, while most failed-episode reads do not abstain.Internal verification reflects identity and geometry checks, not ground-truth identity; the diagnostics cannot distinguish undetected identity errors from later control failure.Source: §E.3 · p. 24

E.4 ROUTE LENGTH AND DOWNSTREAM EXECUTION

  • PatternLock is reliable through route length three and less stable thereafter, while RouteStick declines broadly across lengths two through six; these are observational difficulty slices, not causal length effects.None of the PatternLock or RouteStick failures reaches the 80-subgoal horizon, whereas MoveCube and InsertPeg reach it in 37/49 and 61/144 failures, respectively.Source: §E.4 · p. 25, Table 16 · p. 25

E.5 SEED-INVARIANT DEMONSTRATION FAILURES

  • Successful episode identities are nearly seed-invariant on VideoPlaceButton and VideoPlaceOrder, suggesting residual demonstration-reference failures primarily reflect demonstration interpretation rather than policy-seed variation.VideoRepick’s internally correct cube record is marked verified in all 150 episodes, including all 33 failures; internal verification does not certify that the selected demonstration entity is correct.Source: §E.5 · p. 25

E.6 EPISODE-LEVEL STATE-TO-ACTION LIFECYCLE

  • A ButtonUnmaskSwap episode trace shows memory written, updated, accessed, and re-grounded before passing a coordinate to the VLA, illustrating mechanism operation rather than accuracy or necessity.The trace includes cube identities while visible, cube–cover bindings after occlusion, and binding updates as covers move; memory retrieval corrects the frozen composer’s proposed container at history-dependent reads.Source: §E.6 · p. 25, Figure 6 · p. 26

E.7 TOOL-USE EXAMPLES

  • Figures 7–13 provide successful examples spanning binding, progress, reference, route, and mark mechanisms, with rollout frames paired to traces transcribed from episode logs.Figures 7–11 are seed-7 replications on a host without the mask tool; Figures 12–13 use earlier generic-gate runs, before the unified entity tool stack.Source: §E.7 · p. 25

E.8 REMAINING CAUSAL QUESTIONS

  • Current ablations do not isolate the PickHighlight mark reader, SAM, or current-view grounding, and they do not uniquely decompose failures across state, access, grounding, and control.A mark-reader removal and a matched oracle ladder are proposed as ways to provide finer attribution.Source: §E.8 · p. 25

Related papers

Loading 2609.36595v1…