PaperlayerView exact revision on arXiv

Source-linked AI summary

REMM: Revisable Embodied Memory for Manipulation

Revanth Krishna Senthilkumaran, Ruwan Wickramarachchi, Alessandro Oltramari, Jonathan Francis

arXiv:2610.12710v1cs.RO

TL;DR

Long-horizon manipulation requires retaining useful history while revising conclusions no longer supported by physical evidence. REMM links an append-only evidence ledger to revisable task state and schedules recovery after supported facts change. Across 51 physical trials, it improves recorded intervention-recovery progress over the strongest baseline from 57% to 91%.

  • Problem

    Long-horizon manipulation requires retaining task-relevant history while revising conclusions that later physical evidence no longer supports.

    Source: §I · p. 1

  • Method

    REMM maintains an append-only evidence ledger and a revisable task-state projection linked by task-semantic dependencies, withdrawing affected conclusions and scheduling bounded recovery when accepted evidence changes a supporting fact.

    Source: §I · p. 2

  • Results

    57% to 91%: REMM improves mean recorded task progress over the strongest baseline on intervention-recovery tasks across 51 physical trials.

    Source: §VII · p. 8

  • Takeaways & Limitations

    In matched video diagnostics, REMM achieves 8/12 fully grounded T2 decisions versus 5/12 or fewer for every baseline.

    Source: §VII · p. 8

    The physical study uses one robot, a small tabletop object set, controlled interventions, and limited per-condition samples.

    Source: B. Scope and Limitations · p. 8

Abstract

from arXiv · show

Long-horizon manipulation requires robots to retain task-relevant history while revising conclusions no longer supported by physical evidence. Retaining observations alone does not specify which completed conditions should be withdrawn after a disturbance or which obligations require repair. We introduce REMM, Revisable Embodied Memory for Manipulation, a supervisory memory layer for a frozen G0.5 vision-language-action executor. REMM separates an append-only evidence history from revisable task state and explicitly links derived relations and completed conditions to their supporting facts. When accepted evidence changes a supporting fact, it withdraws dependent conclusions and schedules targeted recovery at verified skill boundaries without updating the executor. Across 51 physical trials, including operator-assisted runs, REMM improves mean recorded task progress over the strongest baseline from 57% to 91% in intervention recovery and from 43% to 70% in sequence recall. Complementary video diagnostics show improved grounding of retained history into the next instruction, including comparisons with $\pi_{0.5}$-compatible high-level planner adaptations. These results support revisable memory as a practical interface for continued manipulation without policy retraining. Project: https://revanthsenthil.github.io/remm/.

I. INTRODUCTION

Long-horizon manipulation needs memory that preserves what happened while revising conclusions when physical evidence changes. REMM links physical evidence to task state and recovery, and is evaluated on real-robot tasks involving disruptions, recall, and hidden objects.

I. INTRODUCTION

  • A capable executor and retained history alone do not determine what should happen next when evidence is hidden, old, or contradicted by physical intervention.Long-horizon decisions depend on interaction history under partial observability.Source: §I · p. 1
  • REMM preserves an append-only evidence history while linking physical relations to derived and completed task conditions, so verified changes can withdraw affected conclusions and schedule bounded recovery.Recovery instructions are issued at the next verified stationary skill boundary without REMM-specific parameter training.Source: §I · p. 2
  • 57% to 91% in intervention recovery and 43% to 70% in sequence recall: REMM improves mean recorded task progress over the strongest baseline across 51 physical trials.The evaluation also includes hidden-object rearrangement tasks.Source: §I · p. 2
  • The evaluation traces observed changes through memory revision to subsequent control decisions across transport, sequence-recall, and hidden-object tasks.It distinguishes retaining history through delay or occlusion from withdrawing and repairing task state after transport interventions.Source: §I · p. 2

A. Memory for Vision–Language–Action Control

REMM places explicit, revisable task-state memory above a frozen executor rather than learning memory-conditioned action generation. It connects retained physical relations to task conditions and uses those dependencies to revise downstream obligations after evidence changes.

A. Memory for Vision–Language–Action Control

  • REMM differs from policy-integrated memory approaches by leaving G0.5 frozen and revising task state between bounded skills rather than learning a memory-conditioned action model.Prior approaches described here learn retention and retrieval representations inside the policy.Source: A. Memory for Vision–Language–Action… · p. 2

B. Closed-Loop State Revision and Recovery

A. Problem Setting and System Overview

  • REMM converts the long-horizon instruction and multi-view observations into an atomic instruction for the frozen G0.5 executor, while accepted typed events update its ledger, active-state projection, and support graph.New observations and robot state close the loop after action-chunk execution.Source: Figure 2 · p. 3

B. Revisable Memory State

REMM separates recorded evidence from the currently supported task state. Its assertion records preserve support dependencies, while source-specific gates determine which candidate changes become accepted evidence.

B. Revisable Memory State

  • The ledger preserves active and inactive assertions, while the state projection contains only propositions REMM currently treats as supported.A later intervention can change the projection without erasing the record that an object was placed.Source: B. Revisable Memory State · p. 3
  • Each derived record requires every parent in its conjunctive support set; losing any required parent withdraws the record until accepted evidence supports it again.Invalidation means the stored evidence no longer justifies the proposition, not that the proposition has been observed false.Source: B. Revisable Memory State · p. 3
  • Only candidate events that pass their source-specific acceptance gate enter the ledger; visual relations require grounded identities, confidence of at least 0.8, and two agreeing observations.A missing detection alone never retracts a fact, and confidence is an acceptance signal rather than a calibrated probability.Source: B. Revisable Memory State · p. 3

C. Dependency Revision and Recovery

REMM derives task state and completion from supported physical relations, then invalidates downstream records when a required relation becomes inactive. Lost completed obligations become repairs inserted or deferred according to their overlap with the active skill.

C. Dependency Revision and Recovery

  • A placement goal is complete only when an accepted physical postcondition supports its completion record.This keeps completion linked to the observed condition that justified it.Source: C. Dependency Revision and Recovery · p. 4
  • When a record becomes inactive, REMM invalidates every active descendant, and invalidated completed goals can generate or update a repair skill.The support links are declared from task semantics rather than learned from trajectories.Source: C. Dependency Revision and Recovery · p. 4
  • REMM inserts a repair after the active skill when their persistent entities overlap; otherwise it appends the repair, while leaving the active instruction unchanged.Equivalent repairs are deduplicated and pending repairs may be retargeted using newer evidence.Source: C. Dependency Revision and Recovery · p. 4
  • If a verified placement supports an object's containment and transport completion, later evidence placing it outside the container withdraws the inferred destination and reopens the transport obligation.The independently verified container location remains active, and repair is compiled from current supported state rather than a stale instruction.Source: C. Dependency Revision and Recovery · p. 4

D. From Retained History to an Atomic Instruction

The section describes how REMM turns retained history into bounded instructions: it grounds remembered cues or object identity in the current scene, then passes an instruction—not the full event ledger—to the executor. The experiments test recall, identity retention, and revision under distinct task conditions.

D. From Retained History to an Atomic Instruction

  • In T2, the controller binds remembered cues to visible objects and destinations, then compiles the demonstrated order into the next bounded placement instruction.The executor receives that instruction and its native observations, not the full event ledger.Source: D. From Retained History to an Atomic… · p. 4
  • Sequence recall tests whether ordered demonstration history survives delay and occlusion and can guide the next placement instruction.Remembering the color sequence alone does not provide current object binding or placement destination.Source: D. From Retained History to an Atomic… · p. 4
  • In T3, the selected cup must retain its target-object identity through rearrangement, and its final position is converted into a lift instruction.The offline path uses pixel-initialized MIL tracking and is distinct from T1 dependency revision and open-ended VLM planning.Source: D. From Retained History to an Atomic… · p. 4
  • Hidden-object search tests whether an object–cover binding stays attached to the same cover through a shuffle, rather than to its initial position.Source: D. From Retained History to an Atomic… · p. 4

IV. EXPERIMENTS

  • The study covers 51 physical robot trials and 42 matched generated-video clips, testing revision and recovery, history retention and grounding, and control effect and cost.Table I marks mechanisms tested by at least one predeclared condition; only T2 disturbed repeats test revision and recovery.Source: §IV · p. 4, Table I · p. 5
  • The evaluation also asks how memory decisions affect task progress, end-to-end success, failure modes, and system cost against no-memory and hierarchical-memory baselines.Source: §IV · pp. 4–5

A. Task Families

The task families test recovery, ordered recall, and hidden-object identity under different memory demands. A separate mechanism control compares stored invalidation, on-demand recomputation, and direct physical-relation updates under the same accepted evidence.

A. Task Families

  • The study organizes tasks by memory demand rather than atomic motor skill, covering transport, sequence recall, and hidden-object search.Source: A. Task Families · p. 5
  • Transport tests whether recovery is deferred while unrelated work remains queued and retargeted after a destination change.Source: A. Task Families · p. 5
  • Sequence recall tests ordered cue-to-object grounding and queued placements, with disturbed repeats testing revision and repair without corrupting the remaining order.Source: A. Task Families · p. 5
  • Hidden-object search tests whether an object–cover binding remains attached to the same cover through a shuffle.Source: A. Task Families · p. 5

B. Dependency Mechanism Controls

  • The explicit strategy reads stored invalidations, recomputation checks saved postconditions on demand, and direct update changes only the physical relation.All three receive the same accepted evidence; this software-level control does not isolate stored propagation's advantage over recomputation in the full-system physical comparisons.Source: B. Dependency Mechanism Controls · p. 5

C. Evaluation Metrics

The evaluation reports task progress and strict end-to-end success alongside memory-decision, cost, and offline readout views. Pooled outcomes show full completions for REMM in every task family, but the counts are descriptive because task mixes and denominators differ.

C. Evaluation Metrics

  • The evaluation defines partial task progress and strict end-to-end success using evaluator-defined checkpoints fixed before execution and shared across methods.The exported progress measure may use recorded checkpoint fractions or completion-label or T3 decision-label fallbacks when progress is absent.Source: C. Evaluation Metrics · p. 5
  • For disturbed trials, accepted change, invalidated task state, recovery placement in the queue, and physical repair are separate events; a memory update alone is not physical success.Source: C. Evaluation Metrics · p. 5
  • Memory-traffic ratios normalize method-native logical write and retrieval payloads by fresh G0.5 action-chunk queries, while current observations, audit recordings, and model weights are reported separately.Source: C. Evaluation Metrics · p. 5

V. RESULTS

  • The study combines physical outcome decomposition, task-level decision and cost profiles, and matched offline diagnostics of high-level memory readout.Figure 3 groups physical tasks into recovery scheduling and retargeting, ordered recall and disturbed repeats, and hidden-object identity; blue and orange borders mark human motion and robot execution.Source: §V · p. 5, Figure 3 · p. 6

A. End-to-End Physical Outcomes

  • 10 of 17 REMM trials were full completions overall, compared with 2 of 16 for HiMe and 0 of 18 for vanilla G0.5; none of 17 REMM trials was a complete failure.These pooled counts are descriptive rather than a single comparable success rate because task mix and denominators differ.Source: A. End-to-End Physical Outcomes · p. 6
  • Two of REMM’s full completions had recorded task assistance, so the pooled counts are not unassisted-autonomy success rates.Source: A. End-to-End Physical Outcomes · p. 6

B. Task-Level System Profiles

Task-level profiles compare REMM’s recorded progress and system costs across T1–T3; offline video diagnostics test whether retained history becomes an actionable instruction. The diagnostics distinguish sequence recall and target tracking from grounding decisions, with results bounded by their passive-video setting.

B. Task-Level System Profiles

  • REMM raises mean recorded task progress from 57% to 91% on T1 and from 43% to 70% on T2, and completes all four T3 trials while neither baseline records a full completion.Fig. 5 reports task-level progress profiles; T1 pools T1a and T1b.Source: B. Task-Level System Profiles · p. 6, Figure 5 · p. 6
  • Fig. 4 presents full, partial, and failed physical outcomes as raw counts, including assisted runs; the relative-efficiency profiles have important measurement limits.G0.5-query efficiency excludes planner, sentry, and verifier VLM calls; missing records, assistance, and early stopping mean the relative efficiencies do not establish lower cost at equal task performance.Source: Figure 4 · p. 6, B. Task-Level System Profiles · p. 7
  • REMM is not the most time- or query-efficient method on every task, so the paper reports efficiency beside success rather than combining them.Additional perception and verification, and continuing runs that baselines terminate early, contribute to the visible systems cost.Source: B. Task-Level System Profiles · pp. 6–7

C. Offline Generated-Video Diagnostics

  • T2 and T3 diagnostics use generated videos to score high-level decisions without executing a motor policy; T2 uses LTX-Video clips and T3 deterministic shell-game renders.All methods use the same 12 T2 clips and 30 T3 clips; the adaptations emit language subtasks compatible with a downstream π0.5 executor, which is not run.Source: C. Offline Generated-Video Diagnostics · p. 7, Figure 6 · p. 7
  • On T2, REMM and HiMe both achieve cue LCS 35/36 and exact sequence 11/12, but REMM grounds the first object on 11/12 clips and commands on 8/12, versus 1/12 for each with HiMe.REMM’s five-metric partial score is 86.1%, versus 73.9% for RoboMME-style, 55.0% for MemER-style, 54.4% for HiMe, and 38.9% for no-memory; the score is not a successful-clips percentage.Source: C. Offline Generated-Video Diagnostics · p. 7
  • On T3, REMM’s persistent target–cover track is correct on all 30 deterministic clips, while the no-memory VLM matches the fixed-set majority at 11/30.The result is specific to the generated set and does not establish perfect tracking under real occlusion or appearance shift.Source: C. Offline Generated-Video Diagnostics · p. 7

VI. DISCUSSION

REMM separates retained history from revisable task state, linking accepted evidence to supported conditions so disturbances can trigger bounded repair. The reported gains concern recorded task progress and grounded decisions, within evaluation and system boundaries that limit causal attribution and generalization.

A. Interpreting the Results

  • REMM connects evidence to completed conditions so accepted changes can withdraw unsupported conditions and create bounded repairs rather than merely add observations to history.Nominal T2 and T3 test retained information after delay, covering, or rearrangement; T1 and disturbed T2 repeats additionally test whether a supported condition becomes inactive.Source: A. Interpreting the Results · p. 7
  • REMM’s offline T2 advantage is in persistent entity binding and compiling retained state into an atomic instruction, not dominance on every diagnostic column.MemER-style and RoboMME-style slightly exceed REMM on sequence storage and visible destination recognition, respectively; the comparison is system-level and does not isolate one component.Source: A. Interpreting the Results · pp. 7–8
  • Because methods vary in observation queries, stopping, and instructions, end-to-end failures are not necessarily memory failures, and physical gains cannot be attributed to graph propagation alone.The dependency controls isolate software state-update logic, whereas the physical comparison measures complete systems.Source: A. Interpreting the Results · p. 8

B. Scope and Limitations

  • REMM’s dependency graph comes from a task compiler and fixed predicate ontology; it does not discover causal structure from trajectories or predict arbitrary physical effects.It also does not automatically compile unconstrained language into new dependency rules.Source: B. Scope and Limitations · p. 8
  • Updates rely on conservative 2D detections, persistent identifiers, frozen VLM judgments, and source-specific evidence gates; missing detections can delay necessary updates.Corrections occur only at verified stationary skill boundaries, with no guarantee of recovery when identity is lost under complete occlusion.Source: B. Scope and Limitations · p. 8
  • The physical study uses one robot, a small tabletop object set, controlled interventions, and limited per-condition samples.The video diagnostics use generated T2 clips and deterministic T3 renders; the compared high-level adaptations are not official reproductions or measurements of π0.5 manipulation success.Source: B. Scope and Limitations · p. 8

VII. CONCLUSION

  • 91% versus 57% mean recorded task progress on intervention-recovery tasks, and 8/12 fully grounded T2 decisions versus 5/12 or fewer for every baseline.The physical-trial result includes operator-assisted runs; the T2 comparison comes from matched video diagnostics, not physical rollouts.Source: §VII · p. 8

APPENDIX A MEMORY RECORDS AND UPDATE SEMANTICS

REMM stores accepted evidence separately from revisable facts and task conditions, then updates dependent obligations at verified stationary boundaries. Its worked example distinguishes an unsupported conclusion from evidence that a proposition is false.

A. Stored Records and References

  • Accepted observations become immutable events, while revisable facts and subgoal records retain completion and recovery evidence linked back to the supporting evidence.The supersedes_fact_ids and depends_on_fact_ids fields trace updates from visual evidence to affected goals and repairs.Source: A. Stored Records and References · p. 9

B. Boundary Updates and Recovery

  • At stationary boundaries, accepted events supersede conflicting records, invalidate unsupported descendants, and expose affected obligations for a deduplicated repair bounded by the affected skill contract.Visual relations require grounded identities, confidence of at least 0.8, and agreement across two distinct observations; missing detections alone do not create negative facts.Source: B. Boundary Updates and Recovery · p. 9
  • Figure 7 shows inspectable evidence, supersession, and recovery references in an assisted partial rollout using recompute-on-demand, not a successful explicit-invalidation ablation.The example illustrates the references for mechanism inspection; raw history and its interpretation remain distinct.Source: B. Boundary Updates and Recovery · p. 9, Figure 7 · p. 10

C. A Worked Support-Withdrawal Example

  • When a superseded containment fact supported an inferred location and completed placement, REMM withdraws dependent conclusions while leaving an unrelated bowl-location fact active.The location and completion must be established again before the transport goal is satisfied; an invalidated destination is not automatically evidence of the opposite proposition.Source: C. A Worked Support-Withdrawal Example · p. 9

A. Shared Executor and Baseline Boundary

  • The physical comparison uses a Galaxea R1 Lite and frozen G0.5, with all methods sending language instructions through the same motor-policy interface.Editable memory is not unique to REMM; its distinction is retaining support for completed conditions and connecting withdrawal to a repair obligation.Source: A. Shared Executor and Baseline Boundary · pp. 9–10

B. Assistance and Outcome Labels

The section distinguishes physical outcome and decision measures from offline video diagnostics, while making assistance and comparison boundaries explicit. The diagnostics separately score remembered sequence, object grounding, destinations, commands, and hidden-target choice.

B. Assistance and Outcome Labels

  • Two task-assisted REMM full completions were a T1b run with an orange-block nudge and a T2 run with help completing strawberry pickup.Assistance remains visible in the recorded outcomes; controller stops and completion claims are not independent evidence of physical success.Source: B. Assistance and Outcome Labels · p. 10
  • The outcome export separates assistance from outcome, but its figures do not establish a complete independent re-adjudication of the saved ledger and annotations.Assistance counts include unrecorded vanilla trials, which should not be treated as unassisted.Source: B. Assistance and Outcome Labels · p. 10

C. Reading the Radial Profile

  • The radial profile combines recorded task progress, decision accuracy, non-failure completion, and relative median-cost scores, which use available labels and do not establish equal-completion cost.A run can have a correct decision or be non-failed without completing the task; efficiency scores are relative medians, not elapsed time or query counts.Source: C. Reading the Radial Profile · p. 10

A. What Each Baseline Predicts

  • Offline methods output high-level readouts rather than action chunks, and π0.5-compatible subtasks are not tested with motor inference or physical rollouts.The paths differ in observation selection and compute, so matched clips do not control every aspect of inference budget.Source: A. What Each Baseline Predicts · pp. 10–11, Figure 8 · p. 11

B. Prediction Versus Expected Answer

  • The T2 scorer separates exact three-cue order from first-object, destination, and command correctness, so sequence recall alone does not earn grounding credit.For example, the MemER-style row has 36 LCS and 12 exact sequences, but zero first-object and command counts.Source: B. Prediction Versus Expected Answer · p. 11
  • T3 scores the final position of the cover bound to the hidden target after shuffling, rather than the cover’s original position.The snapshot reports 12 T2 clips and 30 T3 clips.Source: B. Prediction Versus Expected Answer · p. 11

C. Partial Accuracy and Full T2

Partial T2 accuracy averages normalized metrics equally, while full T2 requires all key fields to be correct together. The mechanism controls distinguish update strategies, but the physical comparison does not isolate support-graph propagation.

C. Partial Accuracy and Full T2

  • Fig. 8’s T2 partial composite gives each of five normalized metric accuracies equal weight, rather than summing raw counts.It is a descriptive summary, not five independent trials or a probability of motor success.Source: C. Partial Accuracy and Full T2 · p. 11
  • 8/12 REMM clips versus 5/12 RoboMME-style clips had exact sequence, first object, destination, and command all correct together.These full-decision counts are stricter than the respective partial scores of 86.1% and 73.9%.Source: C. Partial Accuracy and Full T2 · p. 11

APPENDIX D SCOPE OF THE MECHANISM EVIDENCE

  • Direct update, recompute-on-demand, and explicit invalidation differ in how they handle an accepted change, but the physical outcomes compare complete systems rather than isolating graph propagation.Separating support-graph benefits would require matched comparisons holding perception, accepted evidence, goal checks, and execution fixed.Source: Appendix D · p. 12
Loading 2610.12710v1…