Source-linked AI summary
Experience Transfer for Multimodal LLM Agents in Minecraft Game
Chenghao Li, Jun Liu, Songbo Zhang, Huadong Jian, Hao Ni, Lik-Hang Lee, Sung-Ho Bae, Guoqing Wang, Yang Yang, Chaoning Zhang
TL;DR
Multimodal LLM agents often treat memory as a passive repository, leaving the structures that support transferable experience underexplored. Echo organizes multimodal experience along explicit transfer dimensions and uses analogical learning to improve learning efficiency, task success, and interpretability, particularly in long-term and cross-task settings.
Problem
Existing methods often treat memory as a passive warehouse or library of past behaviors, while the deeper structures that enable experience transfer remain underexplored.
Method
Echo decomposes environmental knowledge and experience into five transfer dimensions and uses structured in-context analogical learning to retrieve, adapt, and build upon past experiences.
Results
Echo significantly improves learning efficiency, task success rates, and interpretability, with an explosive item-unlocking phenomenon emerging after the cold-start phase.
Takeaways & Limitations
The results support explicit experience transfer as a promising approach for improving multimodal agents’ long-term learning and cross-task transfer in complex interactive tasks.
Takeaways & Limitations
Echo is less effective at actively exploring unfamiliar environments and is evaluated mainly in Minecraft, whose simple, consistent rules limit applicability to the real physical world.
Abstract
from arXiv · showhide
Multimodal LLM agents operating in complex game environments must continually reuse past experience to solve new tasks efficiently. In this work, we propose Echo, a transfer-oriented memory framework that enables agents to derive actionable knowledge from prior interactions rather than treating memory as a passive repository of static records. To make transfer explicit, Echo decomposes reusable knowledge into five dimensions: structure, attribute, process, function, and interaction. This formulation allows the agent to identify recurring patterns shared across different tasks and infer what prior experience remains applicable in new situations. Building on this formulation, Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples. Experiments in Minecraft show that, under a from-scratch learning setting, Echo achieves a 1.3x to 1.7x speed-up on object-unlocking tasks. Moreover, Echo exhibits a burst-like chain-unlocking phenomenon, rapidly unlocking multiple similar items within a short time interval after acquiring transferable experience. These results suggest that experience transfer is a promising direction for improving the efficiency and adaptability of multimodal LLM agents in complex interactive environments.
1. Introduction
Existing multimodal agents use memory and planning, but often treat memory as passive records rather than transferable experience. Echo structures experience across five dimensions and uses analogical retrieval to improve adaptation and learning efficiency.
- Background: Existing agents combine perception, reasoning, action, and memory to explore environments, decompose goals, plan subtasks, and invoke tools.These systems can self-improve through environmental feedback without large-scale task-specific supervision.
- Background: Explicit memory structures support skill reuse, scene understanding, hierarchical planning, and self-verification in complex scenarios.Examples include spatiotemporal event indices, multimodal knowledge graphs, and structured memory.
- Motivation: Most existing methods treat memory as a passive warehouse, behavior index, or skill library, leaving deeper structures for experience transfer underexplored.The paper motivates transfer through recurring crafting patterns, material hierarchies, and causal chains in Minecraft-like worlds.
- Results: 1.3×–1.7× faster equivalent milestones were achieved by Echo than by MP5, Voyager, JARVIS-1, and MrSteve in item unlocking.The method also shows rapid mid-stage unlocking after accumulating knowledge during cold start.
- Contribution: Echo decomposes environmental and experiential knowledge into Structural, Attribute, Procedural, Functional, and Interaction dimensions.A Contextual State Descriptor aligns visual, textual, and interactive signals into comparable semantic snapshots.
- Results: Echo combines explicit transfer axes, ICAL, self-consistency checking, and memory replay to improve learning efficiency, task success rates, and interpretability.The reported gains are particularly emphasized for long-term learning and cross-task transfer scenarios.
2. Related Work
Related work establishes multimodal Minecraft agents for exploration, perception, and skill reuse, while the proposed framework is motivated by shortcomings in transfer and control generalization.
- Embodied Agents in Minecraft: Minecraft agents such as MineDojo and Voyager use internet-scale knowledge and large models to support open-ended exploration.Other systems enhance active perception or use modular and hierarchical architectures for skill reuse.
- Motivation and Challenges: Traditional MLLM agents struggle to transfer across tasks with different state transitions and causal relationships and may exhibit unstable control from hallucinations.The proposed framework responds by decomposing knowledge into five explicit transfer dimensions.
3. Proposed Method
Echo uses Structured In-Context Learning to represent and align multimodal task knowledge across five transfer axes, retrieve analogous experiences, and iteratively plan, verify, execute, and update memory.
- Structured ICL: Structured ICL models task-transfer relationships along Structural, Attribute, Procedural, Functional, and Interaction axes.Explicit dimensions support interpretable cross-task alignment and analogy-based reasoning.
- Explicit Transfer Dimensions: The five dimensions encode what the world is like, how it changes, what things do, and how the agent interacts with it.Structural and attribute dimensions capture static regularities, while procedural and functional dimensions capture dynamics and transformations.
- Contextual State Descriptor: A unified CSD compresses visual, textual, and interactive inputs into a comparable and verifiable semantic snapshot.The representation organizes multimodal content along the five explicit transfer dimensions.
- Contextual State Descriptor: CSD-ICAL uses structured examples and a memory bank to support analogical learning and knowledge transfer.Instruction tuning trains reliable CSD generation from task instructions, execution traces, and verifier feedback; successful tasks enter long-term memory.
- Overall Iterative Process: The iterative system loops through perception, memory retrieval, planning, verification, and execution using short- and long-term memory.Its architecture has perception, decision, and execution layers supporting structured ICAL and case-based transfer.
- Transfer System Formalization: The transfer system stores multimodal trajectories, CSDs, plans, validation results, and execution traces in symbolic and vector memory channels.The agent produces hierarchical plans and self-verification assertions, then updates memory after execution.
4. Experiment
The experiments evaluate Echo’s transfer axes, structured ICAL, continuous learning, and interpretability across cold-start, long-horizon, and task-specific settings. Echo shows stronger late-stage learning and axis-dependent performance effects, while the case study illustrates functional transfer between pickaxe tasks.
- Experimental objectives: Echo combines multimodal long-term memory with transfer learning to evaluate cross-world and cross-task generalization, consistency checking, and continual learning.The experimental objectives explicitly target open-world distribution shift, long-horizon stability, and progressive performance improvement through memory replay.
- Cross-world learning from scratch: Echo remains competitive in the 2-shot setting, while baseline strengths vary across functional-equivalence, structural, visual-variation, and multi-step reasoning tasks.JARVIS-1 is described as the most stable overall baseline, and removing its SelfCheck causes a 10–20 point drop for most tasks.
- Continuous learning test: 45% is Echo’s episode-30 success rate, ahead of MP5 (43%), JARVIS-1 (35%), MrSteve (33%), and Voyager (18%).Echo starts more slowly but continues gaining from episodes 10–30, whereas JARVIS-1 saturates after 20 episodes.
- Ablation of explicit transfer axes: Removing the Procedural axis causes a -12% decline on long-horizon tasks, while other axes produce task-family-specific degradations.Reported effects include Attribute -11% on Recipe, Structural -7% on Functional Eq. and -9% on Crafting Chain, Functional -9% on Functional Eq., and Interaction -7% on Utility Blocks.
- Case study: The wooden-to-stone pickaxe case study shows ICAL transferring functionally similar crafting knowledge to infer a stone-pickaxe procedure.The retrieved example uses functional similarity between planks and stone, while the inferred procedure includes mining stone, crafting sticks, placing a crafting table, and arranging materials.
5. Conclusion and Discussion
Echo emphasizes skill acquisition and learning rather than exploration or perception, and its transfer evaluation is mainly conducted in Minecraft’s predictable, idealized environment. These choices limit performance in information-sparse settings and constrain direct applicability to more diverse real-world tasks.
- Research focus comparison: Echo is less effective at actively exploring unfamiliar environments because it relies more on prior knowledge and retrieval than active perception.The comparison specifically contrasts Echo with MP5, which continuously gathers new information through active perception.
- Research focus comparison: Echo has a slower initial learning rate than representative exploration- or perception-focused agents.The discussion frames this as a difference in research focus rather than a universal performance failure.
- Applicability to the real physical world: Minecraft’s simple and consistent rules facilitate skill learning and transfer but limit the method’s real-world applicability.The paper states that real-world tasks are more diverse, ambiguous, and causally complex, so transfer there is unlikely to be as straightforward.