PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

Yuchen Guo; Junli Gong; Weicheng Wang; Hongmin Cai; Yiu-ming Cheung; Weifeng Su

Source-linked AI summary

PEAM: Parametric Embodied Agent Memory through Contrastive Internalization of Experience in Minecraft

Yuchen Guo, Junli Gong, Weicheng Wang, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

arXiv:2605.27762v2May 26, 2026cs.AI

TL;DR

Embodied agents commonly keep experience in externally retrieved memory, leaving a gap in how selected skills become parametric competence without repeated prompt re-injection. PEAM internalizes verified experience and failure-correction trajectories into isolated adapters, and experiments in Minecraft show improved long-horizon performance, preserved consolidated skills, and lower retrieval inference cost.

Problem
Current embodied agents store experience externally and repeatedly retrieve it, leaving unspecified how selected experience becomes part of parametric competence.
Method
PEAM uses a deliberative LLM, episodic staging, failure-correction training, worthiness scoring, and self-triggered consolidation into isolated category adapters.
Results
PEAM improves long-horizon task performance, preserves consolidated skills under continual learning, and reduces the inference cost of retrieval-based memory.
Takeaways & Limitations
The results support internalizing selected experience as isolated parametric skills for repeated execution in the evaluated Minecraft setting.
Takeaways & Limitations
All experiments use Minecraft 1.19 with VOYAGER’s Mineflayer-based framework, so transfer to other embodied domains is outside the present scope.

Abstract

from arXiv · show

We present PEAM, a Parametric Embodied Agent Memory framework in Minecraft that transforms agent memory from inference-time retrieval into parameter-resident skills internalized through experience. PEAM pairs a slow deliberative LLM for open-ended reasoning with a fast parametric module for reflexive execution of consolidated skills. The fast module is a multimodal Mixture-of-Experts LoRA architecture with per-category physically isolated adapters, enabling parameter-level continual learning without catastrophic forgetting. We treat failure as a first-class training signal: failure--correction trajectory pairs are internalized through a joint behavioral-cloning and contrastive objective, so the agent learns not only what succeeds but also how corrected actions differ from failed ones. To govern consolidation, PEAM introduces a parameterization-worthiness score for deciding which experience should be internalized, and a scale-free self-triggered consolidation mechanism for deciding when to internalize without task-specific hand-tuned thresholds, making the agent self-evolving as the trigger transfers across task distributions without re-tuning. Experiments in Minecraft show that PEAM improves long-horizon task performance, mitigates forgetting on previously consolidated skills, and improves parametric-versus-retrieval efficiency over retrieval-based embodied agents and parametric memory variants.

1 Introduction

PEAM addresses the limits of external, non-parametric memory in embodied agents by consolidating selected experience into isolated parametric skill adapters. Its Minecraft evaluation studies skill reuse, forgetting, efficiency, and cross-distribution behavior against retrieval-based and parametric-memory agents.

Motivation: Current embodied agents store trajectories, reflections, and skills externally, leaving their parametric policies unchanged across tasks.This makes memory dependent on inference-time reinjection rather than parameter updates.
Motivation: External recall consumes context budget, adds retrieval and prompt-construction latency, and requires repeated reintroduction of experience during deployment.The introduction frames this as a missing consolidation pathway, not a failure of retrieval-augmented memory itself.
Approach: PEAM consolidates experience into per-category adapters, using parameter isolation to reduce cross-category forgetting.This differs from replaying traces into a shared substrate or distilling specialists into one model.
Approach: PEAM uses a two-tier agent in which a slow deliberative LLM reasons and verifies outcomes while a fast multimodal Mixture-of-Experts LoRA module executes consolidated skills.An external episodic store stages successful and corrected trajectories, and the tiers communicate through a consolidation pipeline.
Evaluation: In Minecraft, PEAM is evaluated against retrieval-based embodied agents and parametric memory variants on task success, forgetting, inference efficiency, and cross-distribution behavior.The introduction also notes that forward-pass preference margins may fail to predict generate-path deployability and that quantized on-device serving can introduce deployment-specific failures.

2 Related Work

Related work frames embodied-agent memory as retrieval-based external storage and situates continual learning within established regularization, replay, and architectural-isolation approaches. Recent parameter-efficient methods extend this landscape with LoRA routing, dynamic adapter expansion, and mixture-of-LoRA designs.

Retrieval-based memory in embodied agents: Embodied-agent memory commonly stores past trajectories, reflections, and skills externally, retrieving them into the context during inference.This retrieval-based design is presented as a dominant approach in LLM agents.
Continual learning: Continual-learning methods are organized around regularization, replay, and architecture- or isolation-based strategies.Recent LLM continual-learning surveys preserve this taxonomy while adapting it to continual pretraining, fine-tuning, and alignment.
Parameter-efficient adaptation: Parameter-efficient continual-learning variants use LoRA routing, dynamic adapter expansion, and mixture-of-LoRA architectures.

3 PEAM

PEAM combines slow deliberative reasoning with a fast parametric module that executes consolidated skills through category-isolated adapters. It stages successful and corrected experience, selects parameterization-worthy skills, and self-triggers consolidation using joint behavioral cloning and trajectory-level preference learning.

Architecture: PEAM pairs a slow deliberative LLM for reasoning, code synthesis, and verification with a fast multimodal Mixture-of-Experts LoRA module for reflexive skill execution.The fast module uses Qwen3-VL-8B-Instruct with per-category isolated adapters.
Architecture: At inference, PEAM routes tasks to applicable category adapters, verifies fast-path trajectories, and falls back to the slow tier when no adapter applies or verification fails.Successful or corrected fallback trajectories are written to episodic memory as future consolidation candidates.
Consolidation: Episodic memory stores verified success demonstrations and failure-correction pairs, while consolidation updates only the adapter corresponding to each skill category.Per-category routing prevents updates to one category from modifying another category’s adapter.
Consolidation: Joint behavioral cloning and adapter-conditioned DPO train on successful trajectories and matched failure-correction pairs, treating corrected trajectories as chosen and failed trajectories as rejected.The frozen fast-policy checkpoint before each consolidation cycle serves as the DPO reference policy.
Experience selection: Parameterization-worthiness scores candidates using retrieval-cost savings, cross-context stability, redundancy, and category-based forgetting risk before internalization.The forgetting proxy reflects that adapters do not share trainable parameters across categories, and prior heuristic selection is recovered as a degenerate special case.
Consolidation triggering: Self-triggered consolidation compares recent failure rates with each skill’s historical baseline and requires high PV ranking, avoiding task-specific absolute failure thresholds.Experiments use W = B = 10, α = 0.05, and q = 0.5 without requiring retuning across task distributions.

4 Experiments

Experiments in Minecraft show that PEAM improves held-out long-horizon task success and inference efficiency, while category-isolated adapters preserve prior skills during continual consolidation. Ablations and distribution-slice evaluations support the roles of failure-aware training, selective consolidation, and self-triggered timing, while also exposing deployment and failure-mode limitations.

Held-out task performance: 69.7% task success (23/33) outperforms VOYAGER’s 54.5% (18/33) by +15.2 percentage points on 11 held-out long-horizon tasks.A McNemar paired test gives p = 0.018.
Inference efficiency: PEAM reduces median per-call latency from 5.5s (B1) to 3.2s (PEAM, −42%) and tokens per task from ∼31,200 to ∼4,600 (−85%).These gains result from eliminating per-call skill-library re-injection after consolidation.
Continual-learning retention: PEAM retains full early-craft performance after sequential craft→gather→combat consolidation, versus forgetting of 32.4% with B4 Single shared LoRA, 43.3% with B5 EWC, and 78.5% with B3 Naive full-FT.Replacing per-category adapters with a shared LoRA increases forgetting from 0% to 32.4% over two sequential consolidations.
Ablations: Joint BC+DPO training produces parser-compatible code in 12/12 held-out cases, whereas pure-DPO produces it in 0/12, while the reward margin rises from +6.51 to +37.92.The result identifies the BC term as necessary for generate-path deployability, not merely an auxiliary preference-learning component.
Selective self-triggered consolidation: STC fires 4 times on 80-skill instrumented data with identical hyperparameters across craft-heavy and combat-heavy slices, yielding top-10 PV overlap of Jaccard = 0.538 and 0.61.A fixed failure-rate threshold instead requires distribution-specific tuning.
Limitations: Quantized on-device serving exhibits high per-step latency, silently zeroed low-magnitude BC updates, and parser-required outputs exceeding feasible max_new_tokens; navigation yields no usable failure-correction pairs.Navigation failures are predominantly environmental rather than code-level, unlike craft and gather failures handled by the retry loop.

5 Conclusion

PEAM presents a parametric embodied memory framework that internalizes experience in the agent’s parameters rather than relying on inference-time retrieval. It combines deliberative exploration and correction with fast parametric storage, using scoring and self-triggered consolidation to govern internalization.

PEAM turns agent memory from inference-time retrieval into experience internalized by the agent’s own parameters.
The framework couples a slow deliberative LLM, an episodic staging store, and a fast MoE-LoRA parametric module.
Failure-correction trajectories, parameterization-worthiness scoring, and self-triggered consolidation govern how, what, and when experience is internalized.
The slow tier explores, verifies, and corrects behavior, while the fast tier stores selected experience.

Limitations

The paper’s claims are scoped to Minecraft 1.19 and a limited set of consolidated skill categories. Transfer to other embodied domains is outside the present scope.

Single environment: Experiments use only Minecraft 1.19 with VOYAGER’s Mineflayer-based execution framework, so transfer to robotic manipulation or web agents is untested.The stated claims cover long-horizon task success, forgetting resistance, parametric-versus-retrieval efficiency, and trigger stability only in this setting.
Consolidated category set: The parametric tier covers three consolidated skill categories, limiting the demonstrated scope of category-level continual learning.

Ethical Considerations

PEAM is evaluated only in Minecraft, without human subjects, personal data, or real-world deployment. Its main ethical risks concern reduced inspectability and reinforcement of incorrect behaviors, mitigated by environment-side verification before consolidation.

Scope and risks: PEAM involves no human subjects, personal data, or real-world deployment because its evaluation is conducted in Minecraft.The paper frames its ethical risks as indirect rather than arising from direct human participation or deployment.
Scope and risks: Internalizing experience into parameters may make embodied agents harder to inspect than agents with external, retrievable memories.This reduced inspectability is identified as a primary indirect ethical risk of parametric memory.
Risk mitigation: Environment-side verification is required before trajectories enter consolidation to reduce the risk of reinforcing incorrect behaviors.The mitigation targets failures in verification that could otherwise cause incorrect behaviors to be reinforced.

A Held-Out Task Suite

The held-out evaluation suite comprises 11 long-horizon Minecraft tasks across recipe planning, resource extraction, and constrained combat, with success verified after up to 200 interaction steps. Each method is evaluated over 33 trials using three random seeds.

Task composition: The suite contains 11 held-out long-horizon tasks spanning recipe planning and inventory management, resource location and extraction, and constrained combat.Tasks T1–T5 cover recipes and inventory, T6–T9 resource extraction, and T10–T11 combat under lighting and projectile-evasion constraints.
Evaluation protocol: Success requires VOYAGER’s environment-side verifier to confirm task completion without manual intervention within N = 200 agent-environment interaction steps.The evaluation reports success only when the verifier confirms completion.
Evaluation protocol: Each method runs every task with three random seeds—42, 43, and 44—for 33 trials per method.The three seeds produce 33 trials across the 11-task suite.

B Training Hyperparameters and Implementation Details

PEAM’s implementation uses a shared Qwen3-VL-8B-Instruct backbone with simultaneously loaded, category-specific adapters and a GPT-4o slow tier. Hyperparameters were selected by coarse grid search, while experiments used three seeds and reported averaged results with Wilson confidence intervals.

Training Hyperparameters: Hyperparameters were selected through a coarse grid search on a held-out validation subset separate from the held-out task suite.This selection covered the PEAM training and consolidation pipeline.
Backbone and serving: The fast module uses Qwen3-VL-8B-Instruct, loads all category adapters simultaneously, and switches them through a routing classifier without adapter merging.Inference runs on a single A100 80GB GPU in bf16 precision with 3.2 seconds median per-call latency.
Slow tier: The slow deliberative tier uses Azure GPT-4o deployment version 2024-11-20 for curriculum proposal, code synthesis, verification, and outcome judgment.LLM-reasoning baselines B1, B2, B6, and B7 use the same GPT-4o version.
Reproducibility: Experiments use random seeds {42, 43, 44}, with reported numbers averaged across seeds and Table 1 intervals computed as Wilson 95% confidence intervals over 33 trials per method.The 33 trials comprise 11 tasks × 3 seeds.

C Per-Task Results Breakdown · D Contrastive-Pair Construction Details

PEAM matches or exceeds VOYAGER on 10 of 11 Minecraft tasks, while both methods fail on precise ranged combat. Its contrastive-pair pipeline identifies, matches, context-filters, and quality-checks failure–correction trajectories before consolidation.

C Per-Task Results Breakdown: PEAM matches or exceeds B1 VOYAGER on 10 of 11 tasks, with strict gains concentrated in multi-step crafting and resource extraction.The gather adapter consolidates location and mining patterns for extraction tasks.
C Per-Task Results Breakdown: Both methods fail on T11, where precise ranged-combat timing exceeds the JavaScript bot interface’s action-space granularity.Combat is also the lowest-yield cpair category in the collected data.
D Contrastive-Pair Construction Details: Failure–correction pairs Dcpair are extracted from VOYAGER trajectory logs through a four-stage pipeline.The stages cover failure identification, correction matching, context matching, and quality gating.
D Contrastive-Pair Construction Details: A trajectory is marked failed when verification returns false after the bot exhausts its retry budget, defaulting to 4 attempts; final action, terminal state, and verifier reason are recorded.This defines the failure signal and preserves the information needed for later correction matching.
D Contrastive-Pair Construction Details: For each failure τf, the pipeline searches the same task for a verified success τc produced within ∆= 5 episodes, limiting pairs to local refinement.The temporal bound reduces contamination from long-horizon curriculum drift.
D Contrastive-Pair Construction Details: Pairs must match inventory, biome, time-of-day, and 32-block-grid location, preventing corrections that succeed because environmental conditions changed.The discrete pre-action state includes inventory composition and counts, with day/dusk/night time buckets.
D Contrastive-Pair Construction Details: The quality gate requires at least 3 non-whitespace token differences, valid VOYAGER wrappers, and category agreement; failed pairs are discarded and logged.Category labeling uses craft, gather, or combat, routes mixed signatures to the dominant category, and preserves a mixed flag.
D Contrastive-Pair Construction Details: Roughly 40% of ∼80 failure trajectories yield usable pairs within ∆= 5; craft and gather yield around 50%, combat ∼25%, and navigation none.These pairs form the |Dcpair| used for consolidation.

E Deployment-Realism Details

Deploying PEAM on consumer hardware exposes three bottlenecks: extreme latency, quantized adapter merging that erases updates, and parser–token-budget incompatibility. A hardware-parity protocol resolves these issues using bf16 serving on 8 A100 GPUs with un-merged, hot-swapped adapters.

Hardware-parity protocol: The hardware-parity protocol serves the fast tier on 8 A100 80GB GPUs at bf16, keeping adapters un-merged and hot-swapped through PEFT’s set_adapter API.Under this protocol, median latency drops to 3.2 seconds per call, BC updates are preserved, and max_new_tokens = 2048 remains tractable.
Failure modes: Consumer deployment encountered prohibitive per-step latency from VRAM-bound activation recomputation when long prefill contexts exceeded KV-cache headroom.On an RTX 4070 with Qwen3-VL-8B quantized to 4-bit, median latency reached ∼2,000 seconds at max_new_tokens = 512 across three approximately 8K-token prompts.
Failure modes: Quantized merge_and_unload silently zeroed low-magnitude LoRA updates, disproportionately erasing the broad BC contribution.∼37% of LoRA delta-W entries below 5 × 10−3 were zeroed on the 4-bit path, versus 0% on bf16.
Failure modes: The action parser requires complete async function wrappers, while craft-task skill bodies often exceed 1,500 generated tokens.Raising max_new_tokens to ≥2,048 worsens latency, whereas lowering it truncates trajectories and produces parser rejection rates of ∼84%.