PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Memory Belongs: Ledger, an Object Ledger for Memory-Augmented VLAs

Tanguy Dieudonné, Jack B. Jedlicki, Heng Yang

arXiv:2609.34554v1cs.RO

TL;DR

Long-horizon robotic manipulation requires memory for information absent from the current frame, while in-policy approaches do not cover all spatio-temporal needs and trail oracle methods. LEDGER pairs in-policy perceptual memory with an external object record, then lets a planner choose between them at runtime; it achieves the highest evaluated average on RoboMME.

  • Problem

    On RoboMME, the best in-policy memory trails an oracle-conditioned policy, leaving open how to resolve what the robot should act on from memory.

    Source: §1 · p. 1

  • Method

    LEDGER pairs one fine-tuned π0.5 policy’s frame-sampling memory with an external object-centric record and uses a planner to choose between them at runtime.

    Source: §1 · p. 2

  • Results

    64.3% average success on RoboMME gives LEDGER the highest evaluated-method average, versus 45.9% for the strongest prior baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported memory split keeps short-term perceptual state inside the policy and persistent spatial and event state in an explicit external record.

    Source: §6 · p. 8

    The implementation is limited to simulation with 2D image-coordinate grounding, and transcript errors leave ordinal placement below oracle performance.

    Source: §5 · p. 8

Abstract

from arXiv · show

Memory is essential for long-horizon, partially observed robotic manipulation: a robot must remember which object was placed in a drawer, whose cup it moved, or how many action cycles have elapsed. Recent vision-language-action (VLA) models embed memory directly inside the policy, but benchmarks show no single in-policy mechanism covers all spatio-temporal dimensions, trailing oracle methods by a wide margin. We argue that memory type dictates where memory should reside: short-term perceptual memory (repetition, timing, retracing) belongs inside the policy, while long-term object memory (persistent spatial state, containment, event history) belongs outside as an explicit, readable record. We present Ledger, a harness that realizes this split over a single fine-tuned $\pi_{0.5}$ policy by pairing an in-policy frame-sampling memory with an external spatio-temporal object memory, the ledger, built from a SAM3 tracker and a VLM captioner of the demonstration and read by an LLM planner that decides at step boundaries. On RoboMME, Ledger reaches the highest four-suite average among the evaluated methods, 64.3% (vs. 45.9% for the strongest prior method under identical evaluation), leading object reference (60.7% vs. 40.3%) and object permanence (86.7% vs. 56.2%) using a single set of weights. Choosing the memory source at runtime, from the instruction and the record, removes the need for a task-level router.

1 Introduction

Long-horizon manipulation requires remembering both recent perceptual history and persistent object state, but in-policy memories trail oracle performance. LEDGER separates these memory types and lets a planner choose between them at runtime using one policy.

1 Introduction

  • In-policy memory averages 44.5% on RoboMME’s four suites, compared with 84.1% when an oracle supplies a correctly resolved target.The cited results identify target resolution as a major remaining bottleneck rather than low-level execution alone.Source: §1 · p. 1
  • Short-term perceptual memory stays inside the VLA, while persistent spatial and event state is kept outside as a readable record.The distinction is between what the robot observed and executed and what happened to objects in the world.Source: §1 · p. 1
  • At runtime, a planner uses the instruction and record to ground a target or defer to the VLA’s short-term memory, without a task-level router.A single set of π0.5 weights is fine-tuned for both inputs.Source: §1 · p. 2
  • LEDGER combines tracker-derived object intervals and occlusion events, demonstration transcripts, and a geometric co-location rule in an explicit object-centric memory.Source: §1 · p. 2
  • 64.3% average success on RoboMME gives LEDGER the highest evaluated-method average, versus 45.9% for the strongest prior baseline.Source: §1 · p. 2

2 Related Work

Prior benchmarks characterize memory demands and gaps, while existing methods use policy-internal histories or explicit records with limits in what they retain. LEDGER addresses the open question of where memory should live and which representation to use at runtime.

2 Related Work

  • RoboMME, LIBERO-Mem, RMBench, and RoboMemArena characterize distinct memory demands, but leave representation, memory location, and memory selection unresolved.Source: §2 · pp. 2–3
  • In-policy methods compress histories or retain selected perceptual information, but none keeps per-object records of resting locations and covering events.The passage contrasts these methods with an object-centric record that persists across occlusion.Source: §2 · p. 3
  • Scene graphs and harnesses expose state, while DIRECT chooses a memory before execution using per-task success and cost; LEDGER records event history and routes at runtime.Source: §2 · p. 3

3 LEDGER

LEDGER splits long-horizon manipulation memory between a low-level policy’s short-term perceptual trace and a high-level planner’s long-term object record. At step boundaries, the planner can ground a target in that record or defer to the policy’s own memory.

3.1 Preliminaries

  • The hierarchy separates low-level action chunks conditioned on recent observations from high-level language subgoals and long-term memory, since current observations omit prior events.The current observation comprises multi-view RGB input and proprioception; it alone cannot recover the demonstration, object covering, or repetition count.Source: §3.1 · pp. 3–4

3.2 Long-Term Memory: Object-Centric Spatio-Temporal

  • The ledger combines tracked object descriptions, stationary intervals, cover timestamps, and demonstration transcripts to represent spatial relations and event order.A geometric co-location rule supports queries about placements and current coverage; the transcript supplies action semantics that geometric tracking alone cannot determine.Source: §3.2 · p. 4

3.3 Short-Term Memory: Frame Sampling as Feature Modulation

  • The low-level policy pools 32 uniformly sampled historical observations into tokens and uses task-conditional subgoal dropout to keep both prompt modes in-distribution.Without dropout, procedural subgoals can become a shortcut, leaving the policy without perceptual memory when subgoals are withheld; in-policy memory also cannot preserve object identity through severe occlusions or explicitly record moved-object state.Source: §3.3 · p. 5

3.4 Deciding at Step Boundaries

  • At episode start and step completion, the planner reads the ledger and one annotated frame, then issues the next subgoal or an empty handoff.Step-boundary replanning occurs when the gripper reaches a stable state, mitigating mid-motion intervention.Source: §3.4 · p. 5

4 Experimental Evaluation

Across four RoboMME suites, LEDGER uses one unified policy and planner, with its largest gains on Permanence and Reference. The suite results show that the best memory source depends on the task, while runtime planning selects between sources.

4 Experimental Evaluation

  • Evaluation uses the official RoboMME test split across 16 tasks, 50 episodes per task, and three seeds; a tracker-only variant isolates persistent geometric tracking before event history is added.Results use each method’s final checkpoint; the tracker-only variant uses SAM3 tracks for spatial/object tasks and the policy’s internal memory for temporal/procedural tasks.Source: §4 · p. 6

4.1 One Policy, Best Across Memory Types

  • FrameSamp+Modul and MemER show suite-specific strengths but marked drops elsewhere: their Counting/Permanence/Reference/Imitation rates are 69.2%/25.7%/37.3%/51.3% and 49.2%/56.2%/40.3%/27.3%, respectively.These comparisons are reported under the paper’s evaluation protocol.Source: §4.1 · p. 6
  • LEDGER reaches 86.7% on Permanence (vs. 56.2% for MemER) and 60.7% on Reference (vs. 40.3%).On Counting and Imitation, it trails FrameSamp+Modul by two points (66.8% vs. 69.2%) and eight points (43.0% vs. 51.3%), respectively.Source: §4.1 · p. 6

4.2 The Optimal Memory Source is Task-Dependent

  • On Permanence, Tracker and LEDGER score 87.2% and 86.7%, versus 25.7% without a subgoal and a 93.3% oracle; both reach at least 88% on the first three tasks.The results support tracked targets as sufficient for this spatial-memory suite.Source: §4.2 · p. 7
  • FrameSamp leads Counting and Imitation at 69.2% and 51.3%; LEDGER’s grounded target costs 15 points on MoveCube (60.7% vs. 76.0%).On Counting, LEDGER’s 66.8% is close to the same policy without subgoals at 66.0%.Source: §4.2 · p. 7
  • For Reference, LEDGER reaches 60.7%, versus 32.7% for tracks without past records and 37.3% for in-policy memory; the oracle reaches 95.2%.The results distinguish identifying a target from executing the skill once the target is known.Source: §4.2 · p. 7
  • Carrying the remaining plan after each decision restores Repick from 28% to 54% and retains PlaceButton at 66%; UnmaskSwap reaches 92%.The ablation uses one evaluation seed and 50 episodes per cell, with binomial standard error up to 7 points.Source: §4.2 · p. 7, Table 2 · p. 7
  • The planner chooses at runtime from the instruction and ledger, deferring on PatternLock and RouteStick while grounding targets on BinFill and PickXtimes.The choice uses neither task nor suite metadata.Source: §4.2 · p. 7

5 Discussion and Future Work

The implementation is limited to simulation and 2D image-coordinate grounding, while transcript errors can constrain ordinal placement matching. The authors highlight interpretable object records and direct human intervention as advantages, and identify 3D mobile manipulation and dynamic object discovery as future work.

5 Discussion and Future Work

  • The implementation operates in simulation with 2D image-coordinate grounding; mobile manipulation will require 3D spatial grounding and camera-pose tracking.For ordinal placement matching, the ledger reaches 64.0% on VideoPlaceOrder versus 100% for oracle grounding because geometry alone cannot verify temporal event semantics.Source: §5 · p. 8
  • Transparent object records make high-level reasoning interpretable and allow people to update object properties without changing policy weights.Source: §5 · p. 8
  • Future work includes 3D mobile manipulation, real-world deployment, dynamic mid-episode object discovery, and reducing supervisory fine-tuning as foundational VLMs evolve.Source: §5 · p. 8

6 Conclusion

LEDGER separates short-term perceptual memory inside the policy from long-term spatial and event memory in an external record, with a planner choosing between them at runtime. On RoboMME, it is the only evaluated method competitive across all four suites and achieves the highest average success, 64.3%.

6 Conclusion

  • 64.3% average success makes LEDGER the highest-performing evaluated method on RoboMME and the only one competitive across all four suites.The system pairs π0.5 with a tracker-maintained ledger, a VLM-generated action transcript, and an LLM planner acting at proprioception-confirmed step boundaries.Source: §6 · p. 8
  • Short-term perceptual memory stays inside the policy, while long-term spatial and event memory resides in an explicit external record.A high-level planner chooses between the two memory sources at runtime.Source: §6 · p. 8

A The sixteen RoboMME tasks

The appendix describes RoboMME’s sixteen tasks through task instructions and first-episode frames, organized across four suites. Instructions and starting inputs vary, including whether a task begins with a demonstration video or a single observed frame.

A The sixteen RoboMME tasks

  • Figure 2 presents first test-episode frames from the four suites: Counting, Permanence, Reference, and Imitation.The front-camera images are 256 × 256, the resolution seen by both policy and tracker.Source: Figure 2 · p. 11
  • Table 3 lists one instruction instance per task, with object colours, counts, and targets varying across episodes.Instructions are given to both the planner and the low-level policy.Source: Table 3 · p. 12
  • Tasks beginning “watch the video carefully” start with a demonstration video; the remaining tasks start from a single observed frame.Source: Table 3 · p. 12

B Training and inference details

The appendix specifies the fine-tuned policy and perception components, along with planner-query frequency during evaluation.

B Training and inference details

  • The π0.5 VLA uses a PaliGemma-2B backbone and flow-matching action expert, fine-tuned from pi05_base for 80k steps at batch size 64.Training uses the standard flow-matching objective and the Sec. 3.3 subgoal dropout schedule.Source: B Training and inference details · p. 12
  • SAM3 serves as tracker, while Qwen3-VL-30B captions demonstrations in about one minute per demonstration on one A100.Source: B Training and inference details · p. 12
  • Planner queries average 2.7 per episode, ranging from once on StopCube, PatternLock, and RouteStick to 4.8 times on PickXtimes.Seeding, planning, and occlusion queries use Claude Sonnet 5.Source: B Training and inference details · p. 12

C Seed variance

Table 4 reports variation across three evaluation seeds for suite-level and overall success averages.

C Seed variance

D The ledger as the planner reads it

The example shows the planner grounding a hidden-object instruction in the ledger’s current object state despite an inaccurate demonstration transcript. After the grasp, it defers and the episode succeeds.

D The ledger as the planner reads it

Related papers

Loading 2609.34554v1…