PaperlayerView exact revision on arXiv

Source-linked AI summary

Externalise the Memory: A Pre-Registered Study of Where an Embodied Agent's Experience Should Live

Cheng-Yin Tu

arXiv:2610.30478v1cs.ROcs.AI

TL;DR

Whether embodied-agent experience should live in policy weights or external retrieval remains unresolved, and its performance under controlled conditions had not been tested. In a pre-registered comparison, retrieval added +34 pp on fixed student weights and the registered protocol contrast agreed at +23 pp, while distillation beat an action-only control by +1 pp (n.s.).

  • Problem

    Whether embodied-agent experience should be folded into policy parameters or kept external for retrieval remains unresolved, with controlled evidence for embodied skill absent to the authors’ knowledge.

    Source: Opening problem · pp. 1–2

  • Method

    The study pre-registered its hypotheses, arms, primary tests, and analysis rules, then compared memory supplied through student weights or inference-time retrieval.

    Source: Pre-registration as a stance · p. 2, §2 · p. 3, §3.4 · p. 6

  • Results

    Retrieval added +34 pp on fixed student weights (identical-weights contrast, C_retrieval vs A_ctrl_rat) and +23 pp in the pre-registered protocol contrast (T4, C_retrieval vs B_main), while distillation beat an action-only control by +1 pp (n.s.).

    Source: Contributions · pp. 2–3

  • Takeaways & Limitations

    The results are consistent with marginal knowledge transferring into weights while situation-specific conditional knowledge belongs in context, supporting externalising memory.

    Source: Contributions · p. 2, Summary · p. 3

    T1’s [−0.2, +22.0] confidence interval leaves the study underpowered to resolve a real effect, so no directional claim is made.

    Source: §4.5 · p. 18

Abstract

from arXiv · show

Should an embodied agent's experience be baked into its policy weights, or kept external and retrieved at inference? A memory-augmented teacher (a VLM that retrieves lessons, with their episode images, from its own past attempts) produces trajectories on a vision-conditioned reach task; from the same trajectories we distil LoRA student policies and evaluate them under matched protocols, including one that re-attaches the teacher's frozen retrieval buffer to a fixed set of student weights. We pre-registered the hypotheses, arms, tests, and analysis rules before training and before any p-value. Distilling the memory-augmented behaviour into weights beat an action-only control by +1 pp (n.s.); the registered test proves underpowered for its own minimum effect, so we withdraw - rather than refute - the memory-in-weights claim. Attaching retrieval to a fixed set of student weights added +34 pp (z=5.15), reaching the pooled memory-augmented teacher reference (49.3%); the pre-registered protocol contrast agrees (+23 pp, z=3.36); under a pre-registered auto-reset sensitivity count the two are +36 and +29 pp. An exploratory behavioural probe with pre-registered per-arm predictions is consistent with behavioural distillation transferring the marginal correction but not the conditional, situation-specific correction that retrieval supplies. The recipe that follows is to externalise the memory: in follow-up controls registered before they ran, the same frozen buffer lifts the base model with no adapter to 48% (sensitivity 48%) and the other three adapters from 19-26% to 53-55% (sensitivity 17-21% to 50-55%), and the adapter adds nothing detectable once the buffer is attached. On a second, harder task, the retrieval advantage points the same direction but falls short of significance (+6 pp, 95% CI [-4.5, +16.4]); exploratory diagnostics tie the attenuation to memory coverage.

1. Introduction

The paper asks whether embodied experience should be consolidated into policy weights or retrieved from context, and tests that choice with matched student protocols and pre-registration. Distillation shows no resolved memory-in-weights benefit, while retrieval improves performance and the exploratory probe is consistent with memory supplying situation-specific corrections.

Opening problem

  • The open question is whether embodied agents should bake experience into weights or retrieve it, a comparison not previously tested under controlled conditions for embodied skill.Retrieval context can grow with accumulated experience while weights do not, making the choice both architectural and empirical.Source: Opening problem · pp. 1–2

Why this test bed answers it

  • A memory-augmented teacher supplies trajectories for student distillation, while matched protocols compare memory in targets with retrieval reattached at inference to fixed student weights.The identical-weights contrast isolates memory supplied through weights versus retrieval context.Source: Why this test bed answers it · p. 2

Pre-registration as a stance

  • Hypotheses, arms, and primary tests were registered before training, and analysis rules were locked after collection but before any p-value.The locked rules kept visible success counts out of test selection.Source: Pre-registration as a stance · p. 2

Contributions

  • +1 pp (n.s.): distilling memory-augmented behaviour beat the action-only control, so the authors withdrew the memory-in-weights claim under the registered falsification clause.The withdrawal is scoped to this recipe, task, data scale, and LoRA rank-16 capacity.Source: Contributions · p. 2
  • +34 pp: attaching inference-time retrieval to fixed student weights improved the identical-weights contrast; the pre-registered protocol contrast was +23 pp, and retrieval reached the 49.3% pooled memory-teacher reference.The +34 pp contrast was exploratory; the +23 pp contrast was confirmatory.Source: Contributions · p. 3
  • The exploratory probe is consistent with distillation transferring a marginal distribution shift but not the conditional, situation-specific correction supplied by retrieval.The paper identifies the correction’s carrier as the action distribution rather than memory content.Source: Contributions · p. 3

Summary

  • The authors’ takeaway is to externalise memory: with the buffer attached, the adapter adds nothing detectable.Source: Summary · p. 3

2. Related Work

The related work contrasts parametric and external memory, situates the study among agent-memory and embodied-policy research, and distinguishes its controlled substrate comparison from prior work.

Parametric vs. retrieved knowledge in language models

  • Prior language-model studies often find retrieval competitive with or superior to weight-based knowledge, but results vary across settings and no memory substrate is uniformly dominant.The paper positions its contribution as an embodied controlled analogue using the same weights and memory supplied in two ways.Source: §2 · p. 3

Agent memory and experiential learning

  • Agent-memory work commonly stores experience as retrievable artifacts, while other approaches distil experience into weights or route different experience types to different substrates.A GUI-agent study links destination to item type, recurrence, and state-conditionality; its lessons-in-weights result differs from this paper’s lessons helping only in context.Source: §2 · p. 3

Memory for VLAs and embodied policies

  • Embodied-policy research includes both external retrieval systems and architectures that incorporate memory, with no retrieval method reported to dominate across settings.The authors distinguish their contribution as testing which substrate carries fixed retrieved memory, rather than proposing a memory system.Source: §2 · pp. 3–4

VLA post-training and RL

  • The study’s distilled, KTO-refined students place its null in the context of VLA post-training research on what refinement moves into parameters.Source: §2 · p. 4

Lifelong / continual LoRA

  • The authors frame marginal versus conditional knowledge as a lens on what continual LoRA adapters can absorb, scoped to a static prior versus a conditional map at this recipe and capacity.Source: §2 · p. 4

Reasoning distillation; Pre-registration in ML

  • The paper relates rationale-in-target distillation to reasoning distillation and presents pre-registration as a stance that makes its null interpretable, without claiming primacy.It says explicit pre-registration with an amendment-tracked chain is uncommon in embodied learning.Source: §2 · p. 4

3. Methods

The methods use a controlled, vision-conditioned reach task, a memory-augmented teacher, and matched student protocols. Hypotheses and analysis procedures were registered or locked in advance of the relevant training or testing decisions.

3.1 Task and environment

  • The evaluation uses a vision-conditioned single-arm reach task with the right arm frozen; success requires the active end-effector to come within 0.05 m of the target.Runs use a fixed task distribution and are subject to an inherited 24-second environment auto-reset.Source: §3.1 · p. 4

3.2 Teacher and memory pipeline

  • The teacher retrieves visually and state-similar past recaps with episode images and injects them as a PAST_LESSONS preamble; its trajectories are the distillation source and its frozen buffer is reused for student retrieval.Retrieval uses DINOv2 image embeddings and state-aware ranking.Source: §3.2 · p. 5

3.3 Students, arms, and the 2×2 factorial

  • Four student arms vary whether retrieved-lesson gist and the teacher’s native thought appear in targets, while all use the same canonical action lines and pinned trajectory pool.Students are LoRA adapters on the shared base; all arms follow the same two-stage training recipe.Source: §3.3 · p. 5

3.4 Evaluation protocols

  • C_retrieval reuses A_ctrl_rat’s adapter and adds the frozen teacher buffer at inference; its identical-weights contrast is exploratory, while C_retrieval versus B_main is the registered protocol contrast.The buffer is read-only during evaluation.Source: §3.4 · p. 6

3.5 Pre-registered hypotheses and tests

  • The primary registered test compares B_main with A_action_only, requiring a significant improvement of at least +10 pp to reject the memory-in-weights null.The registered R1 ∆X probe compares round-1 target X with initial EE X against two reference fingerprints and has pre-registered per-arm predictions.Source: §3.5 · p. 6

3.6 Analysis rules (locked after collection, before any p-value)

  • After collection, the pairing-integrity gate and mechanical test-selection rule were locked before any p-value; because the literal gate failed, the two-proportion z test governed and McNemar was a sensitivity analysis.A frozen-arm diagnostic confirmed physically real pairing despite the failed literal check.Source: §3.6 · p. 7

3.7 Data curation history

  • The rationale filter used in training dropped no rows, leaving the four arms’ 992/2791 counts filter-invariant.The first two rules were demoted to advisory flags, and the vacuity rule had zero drop authority under the policy used.Source: §3.7 · p. 7

Appendix A1. Pre-registration amendment chain

The amendment chain records when the study’s plan changes were filed relative to training and analysis.

Amendment timing

Pre-flight amendments

  • Amendments 5–13 were filed during the pre-flight period, with frozen commits serving as anchors and full amendment text in the pre-registration document.Source: Appendix A1. Pre-registration… · p. 7

Training-count update

Appendix A2. Filter audit trail

The audit found that Rules 1–2 had low precision on correctness because they measure coherence, so they were demoted to advisory flags. The final flags-only policy guarantees zero row drops, leaving the data pool and cross-arm SHA pins unchanged.

Appendix A2. Filter audit trail

4. Results (Experiment 1)

Experiment 1 reports the outcomes of memory distillation, retrieval attachment, and follow-up buffer controls, alongside exploratory probes of what distillation transfers. It also explains how registered tests, confidence intervals, the registered floor, and alpha allocation qualify the findings, leaving T1a unresolved while T4’s interval excludes zero.

4.1 A pre-registered null: memory does not bake into weights (T1)

  • +1 pp for B_main over A_action_only (26/100 vs 25/100; z=0.16, p=0.87), with no detectable effect.The registered +10 pp minimum effect lies inside the 95% CI [−11.0, +13.0], and estimated power for that effect was about one in four; the authors withdraw, rather than refute, the memory-in-weights claim. This null is scoped to single-round SFT + composable-KTO on L0a-Left at this data scale and LoRA rank-16; stronger consolidation is untested.Source: §4.1 · pp. 8–9

4.2 The reversal: retrieval on fixed weights nearly triples success

  • +34 pp on fixed student weights with retrieval (z=5.15); the registered protocol contrast was +23 pp (z=3.36).The first contrast compares C_retrieval with A_ctrl_rat, the same adapter without retrieval (15/100 vs 49/100); it was exploratory. The registered C_retrieval vs B_main contrast was 49/100 vs 26/100. Both reach the same direction as the memory-augmented teacher reference (49.3%).Source: §4.2 · p. 9, Figure 1 · p. 10

4.2b Adapter controls: the buffer without the adapter, and on the other adapters (Amendment 15)

  • With the frozen buffer, the base model scored 48/100 versus C_retrieval’s 49/100 (−1 pp, p=0.887), an undetected—not demonstrated zero—adapter contribution.The three other adapters rose from 25, 26, and 19 to 53, 53, and 55; none differed detectably from C_retrieval. Auto-reset sensitivity counts were 48 for the base and 53, 50, and 55 for the other adapters.Source: 4.2b Adapter controls: the buffer… · p. 12, Table 1 · p. 12

4.3 Mechanism: distillation bakes the margin, retrieval supplies the condition (R1 ∆X probe)

  • The exploratory R1 probe places distilled arms near the memory-teacher fingerprint (about −16.7), while C_retrieval shifts to −7.10 with σ=14.4.The authors interpret this as distillation transferring a marginal, static correction, while retrieved context supplies a situation-specific correction; the study does not separate lesson text from episode images.Source: §4.3 · p. 13, Figure 2 · p. 14

4.4 Which episodes need memory, and what “conditional” is not

  • 29/100 episodes were solved only by C_retrieval; exploratory tests found no support for starting difficulty or mid-trajectory recovery as the explanation.Rescue-only episodes had identical initial cube distance (19.9 cm), and C_retrieval showed diverge-then-converge in only 10% of rescue episodes. The proposed account is sustained per-round supply of the correct direction, rather than a hard start or mid-course save.Source: §4.4 · p. 15

4.4b Is the null just too little data? A no-training diagnostic

  • At round 1, distilled-arm residual correlations with C_retrieval fell inside the chance band (about ±0.20), with no detectable conditional trace.The analysis is limited by noisy proxy corrections and n=100; later-round positive correlations were confounded by shared target geometry and survivorship, and B_main did not significantly exceed A_action_only.Source: 4.4b Is the null just too little data?… · pp. 15–16, Table 2 · p. 16

4.5 Limitations

  • The study does not establish whether more data or adapter capacity would enable conditional transfer, and its T1 null is limited to one task and recipe.The round-1 diagnostic holds capacity fixed and does not test data scaling; a rank sweep was also not run. Raw rationale text was not logged, the pairing gate failed its literal check and the registered z-test fallback governed, and no inference-cost advantage was claimed. Retrieved lesson text and episode images were not separated; disclosed environment and prompt-template defects also remain.Source: §4.5 · pp. 16–17
  • T3 tests whether success rate exceeds a 36.2% floor, set at 0.7 × the 51.7% pooled memory-teacher success rate; B_main is below the floor.The registered T3 test is one-sided.Source: §4.5 · p. 18
  • Both registered readings of T1—T1-strong (∆≥+10 pp and significant) and T1-weak (∆>0 and significant at α=0.010)—fail.They are graded readings within T1’s single α=0.020 budget line, not separate family members.Source: §4.5 · p. 18
  • The registered family-wise alpha budget is 0.060: T1 and T1a receive 0.020 each, while T3 and T4 receive 0.010 each.Source: §4.5 · p. 18
  • T1a’s 95% CI [−0.2, +22.0] includes zero, so the study is underpowered to resolve a possible effect and makes no directional claim.The result is reported as n.s. at the pre-registered alpha.Source: §4.5 · p. 18
  • T4’s 95% CI [+9.6, +35.3] excludes zero comfortably.Source: §4.5 · p. 18

5. Experiment 2 (L2 position-goal reach)

On the harder L2 reach task, the identical-weights retrieval contrast favored retrieval by 6 pp but was not significant; exploratory diagnostics point to limited buffer coverage and start-distance gating as qualifications. The experiment used per-arm scoring and an arm-aligned retrieval buffer to test the question it was powered to answer.

5.1 Task: dual-arm position-goal reaching in a 6-DoF control space (Amendment; 5.2 Per-arm scoring (Amendment 1 §1, disclosed as post-hoc)

  • L2 is a dual-arm position-goal reach task, but its success gate scores position only; joint success was 1/100 while per-arm success was 28/100 for each arm.Per-arm scoring was added post hoc because joint success at episode end differs from each arm reaching its goal at some point.Source: §5.1 · p. 18, §5.2 · p. 19

5.3 Buffer re-tag: arm-aligned success-floor (Amendment 1a)

  • 45 of the 46 episodes with at least one goal-reaching arm were labeled unsuccessful by the joint-outcome recap tag, so the buffer added separate left_reached and right_reached labels.The success-floor filter uses the label for the arm currently scored; the lesson prose remains episode-level and may still refer to either arm.Source: §5.3 · p. 19

5.4 Scope: L2 tests the answerable question (Amendment 2)

  • L2 tested only the identical-weights retrieval contrast because its power analysis indicated that the bake-in effect was not answerable at this sample size.The study trained one adapter on 56 desirable episode-instances and evaluated bare weights against the same weights with a frozen, re-tagged buffer.Source: §5.4 · p. 20

5.5 Per-arm evaluation: single-arm inference matches single-arm training (Amendment 3)

  • Evaluation matched the single-arm training target: only the scored arm was prompted and gated, while the other arm was held in place.The left-scored contrast used shared seeds; freezing the other arm was justified for separate position goals with minimal inter-arm coupling, but not for contact-coupled tasks.Source: §5.5 · pp. 20–21

5.6 Analysis rules (carried from D11 §3.6, per-arm generalisation); 5.7 R1-bias probe, L2 redefinition (locked before data); 5.8 Pipeline integrity (verify-layer interceptions, carried from D11)

  • The per-arm round-1 probe was redefined and locked before L2 data, using each arm’s signed displacement toward its own target and the task-matched teacher distribution.The analysis rules used a pre-specified pairing-integrity gate and mechanical fallback; a format-contract check was added after an initial train/evaluation mismatch was caught.Source: §5.7 · pp. 21–22, §5.6 · p. 21, §5.8 · p. 22

5.9.1 Confirmatory contrast: the identical-weights retrieval effect

  • +6.0 pp for C_retrieval − A_ctrl_rat (20% vs 14%) was not significant (95% CI [−4.5, +16.4] pp; p=0.259).The direction agreed with L0a, but the estimate was below the pre-registered +20 pp MDE; the protocol therefore routed to diagnosis rather than adding arms.Source: §5.9.1 · pp. 22–23

5.9.3 Diagnostic appendix: why the effect is small, not absent

  • Exploratory diagnostics found retrieval restored the teacher’s round-1 mean displacement (C_retrieval −17.2; teacher −17.1) from the distilled arm’s −4.8, while the registered σ test was null.Across two tasks, thinner retrieval coverage co-occurred with attenuation; within L2, rescues began closer to the goal (13.9 cm vs 21.6 cm for both-fail episodes), not with better-matched memories.Source: §5.9.3 · pp. 23–24

5.9.4 Headroom-normalized cross-task framing (post-hoc, exploratory)

  • The post-hoc headroom framing estimates that L2 retrieval closed 42.9% of the per-arm gap to the teacher, versus about 99% on L0a.The L2 estimate remains +6 pp and non-significant; the framing is exploratory.Source: §5.9.1 · p. 22, §5.9.4 · p. 24

Appendix: L2 amendment chain (extends D11 Appendix A1)

The L2 amendment chain records decisions made before evaluation results, including per-arm scoring, arm-aligned buffer labels, and a single-arm evaluation contract.

L2-1

L2-1a; L2-2

  • The buffer was re-tagged with dual arm labels and an arm-aligned success-floor before the retrieval evaluation.The amendment fixed the order: re-tag, update retriever, pin the buffer, then run C_retrieval-L2.Source: Appendix: L2 amendment chain (extends… · p. 24

L2-2 addenda; L2-3

  • The amendments recorded label-based pool selection, a GGUF conversion fix, and single-arm evaluation with a format-contract gate before evaluation.The per-arm evaluation specification was fixed before the crashed evaluation was restarted.Source: Appendix: L2 amendment chain (extends… · pp. 24–25

6. Discussion

The discussion frames experience as having marginal and situation-specific components: the former transferred into weights in this recipe, while retrieval supplied the conditional correction. The L2 result points in the same direction mechanistically but is not a significant replication, and contact-rich tasks remain outside the demonstrated scope.

6.1 What the results say to a system builder: externalise the memory

  • The main results support externalising retrieval context: baking behavior into weights yielded +1 pp over the action-only control, while attaching retrieval to fixed student weights added +34 pp and reached the pooled teacher reference.The +34 pp identical-weights contrast was exploratory; the registered protocol contrast agreed at +23 pp.Source: §6.1 · p. 25

6.2 Two poles, a bridge, and a loop

  • The proposed substrate distinction is that average corrections transfer into weights, whereas situation-specific corrections are supplied by context; the study does not show that retrieval generally beats parametric memory.The student retrieved the teacher’s frozen buffer, so whether a student’s own self-generated buffer performs similarly remains untested.Source: §6.1 · p. 25, §6.2 · p. 26

6.3 Why this does not extend to contact-rich manipulation (a bounded feasibility probe)

  • The feasibility probe found zero success across LIBERO grasping, pushing, and knob-twist tasks despite intact teacher spatial reasoning; the authors locate the boundaries in contact precision and primitive expressivity.They identify closed-loop contact control and an expanded primitive vocabulary as needed extensions.Source: §6.3 · p. 27

6.4 The open problem: consolidating a conditional function, not a marginal prior

  • The discussion reframes the open problem as consolidating a conditional function rather than only a marginal prior.The authors identify surprise-gated writing, per-round distillation targets, and workspace-readout probes as possible routes, not established solutions.Source: §6.4 · pp. 27–28

6.5 The cross-task result (Experiment 2, L2)

  • On L2, retrieval’s direction and mechanism agreed with the substrate account, but its +6 pp end-task effect was non-significant and did not constitute replication.The discussion also treats marginal transfer as potentially dose-dependent and presents the competence-gating account as an exploratory hypothesis.Source: §6.5 · p. 28

Appendix A. Pre-registration amendment chain

The amendment chain is documented with the Methods and anchored to frozen commits. Amendments 15 and 16 post-date the D11 results, while derived statistics are recomputed from persisted counts.

Appendix A. Pre-registration amendment chain

  • Amendments 1–14 for D11 are in Appendix A1, and L2 amendments 1/1a/2/3 are tabulated with the Experiment-2 Methods.Every amendment is signed, dated, and anchored to a frozen commit in the pre-registration document.Source: Appendix A · p. 28
  • Amendment 15 registers inference-only reinforcement arms after the D11 results; Amendment 16 is a documentation-only erratum relabeling R1 ∆X from cm to scalar-guard grid units.The erratum makes no analysis change.Source: Appendix A · p. 28
  • Derived statistics not printed by the registered analysis scripts are recomputed from persisted counts by derived_stats.py.These include Newcombe intervals, the identical-weights z, the T1 critical difference, and the pooled teacher rate.Source: Appendix A · pp. 28–29

Shared configuration

The teacher launch used the same model-serving configuration across the D10 and D10-ext campaign and the first L2 collect.

Shared configuration

  • The teacher server used gemma-4-31B-it Q4_K_M, the same mmproj, ctx-size 16384, and parallelism 1 across both tasks.The launch script kept reasoning-budget 512 unchanged from commit a2aebb9 through the campaign and first L2 collect.Source: Shared configuration · p. 29

Recap outcomes under the shared configuration

Under the shared teacher configuration, inline recap generation succeeded on D10 but failed in every L2 first-collect episode. The difference was traced to reasoning consuming the output budget when the longer L2 prompt triggered unbounded reasoning.

Recap outcomes under the shared configuration

  • D10 produced 547 recap records across seven runs, with no empty-content failures; isolated empty-recap skips coincided with teacher-server outages.D10 was the L0a-Left single-arm task.Source: Recap outcomes under the shared… · p. 29
  • L2's first collect had empty-content recap failures in all 8 of 8 episodes.Each VLM response reported “Empty content in VLM response”.Source: Recap outcomes under the shared… · p. 29
  • Reasoning mode could consume the entire 400-token output budget, and the longer L2 recap prompt triggered this while the shorter L0a prompt did not.The L0a recap stayed within budget because its prompt was brief, not by design.Source: Recap outcomes under the shared… · p. 29

Change for the L2 collects

Turning reasoning off restored L2 recap generation. The resulting L0a–L2 recap-buffer provenance differs, although each student is trained from its own buffer.

Change for the L2 collects

  • With reasoning off and reasoning-budget 0, the previously empty L2 recap call returned a 469-character lesson in 7 seconds.Reasoning-off was also safe for L0a, whose shorter prompt had not needed reasoning tokens.Source: Change for the L2 collects · p. 29

Implications for comparing L0a and L2

  • D10 recaps used reasoning on with budget 512, whereas L2 recaps used reasoning off with budget 0.This configuration difference is a provenance distinction if the L0a and L2 memory buffers are compared directly, but does not affect students trained from their respective buffer text.Source: Implications for comparing L0a and L2 · p. 29
  • Stage-1 reasoning quality may differ subtly between reasoning-on D10 and reasoning-off L2; a clean comparison requires rerunning L0a recaps with reasoning off or reporting the difference explicitly..Source: Implications for comparing L0a and L2 · p. 29

Related papers

Loading 2610.30478v1…