PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Filesystem Memory For Long-Horizon Embodied Agents

Chloe Su, Jiahan Li, Zhenting Qi, Chaoran Cheng, Haoyu Chen, Sejoon Jun, Weirui Ye, Yilun Du

arXiv:2610.22686v2cs.CL

TL;DR

Long-horizon agents need memory that remains searchable without losing visual and spatial details that may matter later. MFM links complementary memory types to multimodal episodes through searchable cues and consolidation. Across Minecraft and computer-use tasks, it outperforms memory-free and memory-augmented baselines.

  • Problem

    For long-horizon agents, retaining all interaction history makes relevant experience difficult to find, while summaries may discard visual or spatial details needed later.

    Source: §1 · p. 1

  • Method

    MFM organizes episodic, spatial, semantic, procedural, and working memory in a filesystem, linking searchable cues and agent-written knowledge to supporting episodes.

    Source: §1 · pp. 1–2

  • Results

    Across Minecraft and KiCad with three base models, MFM outperforms memory-free and memory-augmented baselines; GPT-6-Astra reaches 16 of 24 Ender Dragon milestones versus 12.7 for the best baseline.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    MFM’s memory transfers across tasks, with agent-written knowledge, procedures, and plans remaining linked to the episodes that support them.

    Source: §6 · p. 11

    The evaluation covers single agents in Minecraft and KiCad, and even GPT-6-Astra at maximum reasoning effort did not complete Ender Dragon.

    Source: §6 · p. 11

Abstract

from arXiv · show

Agents operating over long horizons need an effective memory that allows them to reuse their experience throughout long interactions. If the entire history of an interaction is retained, it becomes difficult to find the relevant experience, while compressing it into summaries may result in the loss of visual and spatial details that are required later on. We present \textbf{MFM}, a multimodal filesystem memory architecture that keeps complementary types of agent memory. \textbf{Episodic memory} stores the observations, actions, and outcomes, and \textbf{spatial memory} associates experiences with locations. \textbf{Semantic and procedural memory} build up general knowledge and reusable methods of acting, and \textbf{working memory} holds the current plans. MFM makes use of searchable textual records to connect these different representations to the underlying multimodal episodes. The agents are able to find previous experiences by means of contextual cues and to examine the original observations in order to recover details that are missing from their textual descriptions. Through consolidation, the agents convert experience into higher-level knowledge and procedures while still keeping references to the original episodes. In the case of long-horizon Minecraft and computer-use tasks, MFM leads to improved performance with three base models. With GPT-6-Astra, MFM completes \emph{Mine Diamond from Scratch} and reaches 16 out of 24 milestones on the \emph{Ender Dragon} task, as against 12.7 for the best baseline. On CUABench-KiCad it attains a mean reward of 0.440, as compared to 0.400 for the best baseline and 0.267 in the case of no persistent memory.

1 INTRODUCTION

Long-horizon agents need memory that preserves visual and spatial detail without making growing histories difficult to search. MFM organizes complementary memory types in a filesystem, links searchable cues to underlying episodes, and improves performance on Minecraft and computer-use tasks.

1 INTRODUCTION

  • Keeping an entire interaction history makes relevant experience harder to find, while summaries may discard visual or spatial details needed later.Long-horizon agents may need those details to recover routes to previously seen objects.Source: §1 · p. 1
  • MFM organizes episodic, spatial, semantic, procedural, and working memory in one filesystem, linking agent-written knowledge, procedures, and plans to supporting episodes.Episodic memory preserves observations and interactions; spatial memory connects experiences to locations; the other forms hold facts, reusable methods, and current plans.Source: §1 · pp. 1–2
  • MFM uses searchable textual records to locate episodes and inspect original observations, addressing the difficulty of finding relevant experience as memory grows.The textual cues need not describe every episode detail, because the agent can inspect the referenced observation.Source: §1 · p. 2
  • Consolidation turns observations into facts, reusable procedures, and revised plans while retaining references to supporting episodes.Agents can revisit the evidence if an interpretation becomes uncertain or is contradicted.Source: §1 · p. 2
  • MFM completes Mine Diamond from Scratch in all instances and reaches 16 of 24 Ender Dragon milestones, versus 12.7 for the best baseline; CUABench-KiCad mean reward is 0.440 versus 0.400 and 0.267 without persistent memory.These results use GPT-6-Astra.Source: §1 · p. 3

2 RELATED WORK

Prior work includes filesystem-based textual logs, visual-memory systems, and methods for organizing or reasoning over accumulated observations. MFM is positioned as a multimodal episodic filesystem combining searchable access with retained visual evidence.

Filesystem and Multimodal Memory

  • Filesystem memory has used searchable interaction records, while multimodal systems organize visual sequences for question answering and reasoning.Related work also examines task-conditioned retention, retrieval reliability, relational organization, and analysis of accumulated observations.Source: §2 · p. 3

Memory for interactive agents with a long horizon

  • Long-horizon agent memory spans visual records, spatial navigation, executable skills, retrieved plans, and accumulated knowledge for self-improvement.PRO-LONG indexes text and VISTA revisits frames without a searchable index; MFM is described as a general-purpose multimodal episodic filesystem.Source: §2 · p. 3

3 MULTIMODAL FILESYSTEM MEMORY

MFM stores multimodal episodes, indexes them with contextual cues, and connects them to agent-written memory. Retrieval and consolidation let agents inspect past evidence and update knowledge, procedures, and plans while preserving the recorded history.

3 MULTIMODAL FILESYSTEM MEMORY

  • MFM combines episodic records, a searchable index, and agent-written memory, linking reusable knowledge and plans to multimodal evidence.The architecture’s loop covers recording and indexing, retrieval and inspection, and consolidation without losing original evidence.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 PROBLEM SETUP

  • As an agent’s interaction history outgrows its context budget, MFM represents external memory as retained episodes, an index for locating them, and agent-written knowledge.Episodes contain completed interaction history; the index and agent-written knowledge support later retrieval and use.Source: §3.1 · p. 4

3.2 MEMORY REPRESENTATION

  • MFM’s episodic store retains observations, states, actions, outcomes, and metadata; its index uses contextual and spatial cues to address records, while agent-written memory holds facts, procedures, and plans.Spatial memory also uses traversed-cell grids, coordinates, egocentric views, and landmark notes to connect experiences across time.Source: §3.2 · p. 5

3.3 MEMORY RETRIEVAL

  • Retrieval first searches textual or structured cues for relevant episodes, then selectively loads referenced frames, states, or records for inspection.The agent can compare frames, inspect a region, or place a prior observation beside the current view, without loading the full interaction history.Source: §3.3 · pp. 5–6

3.4 SELF-EVOLVING CONSOLIDATION

  • Consolidation updates environmental knowledge, reusable procedures, and working plans while preserving episodic records and their index.Agents can revise earlier guidance when new evidence qualifies it, or revisit the referenced observation to correct an interpretation and change a plan.Source: §3.4 · pp. 6–7

4 EXPERIMENT RESULTS

The evaluation tests MFM on long-horizon Minecraft and computer-use tasks, compares it with memory baselines, and examines component ablations and retrieval behavior. MFM improves milestone and KiCad results, while the analyses show how its components and retrieved experience relate to performance and interaction.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers Minecraft and CUABench-KiCad, compares three base models against direct evaluation, PRO-LONG, and VISTA, and averages results over three seeds.The authors limit configurations to three seeds because long-horizon runs with frontier models are expensive.Source: §4 · p. 7, §4.1 · p. 7

4.2 PERFORMANCE ON LONG-HORIZON TASKS

  • MFM completes all 10 Diamond milestones, while reaching 16/24 Dragon milestones versus 12.7 for VISTA.The Diamond comparison also reports 6.3 for direct evaluation, 9.3 for PRO-LONG, and 9.7 for VISTA; Dragon reports 6.3 for direct evaluation and 11.0 for PRO-LONG.Source: §4.2 · pp. 7–8, Table 2 · p. 8
  • On CUABench-KiCad, MFM’s GPT-6 mean reward is 0.440 and it outperforms memory baselines across models.The reported rewards are 0.413 for GPT-5.6-Sol and 0.389 for Opus 5; direct-evaluation improvements range from 0.149 to 0.173.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 ABLATION STUDIES

  • Removing stored frames, associative indexing, agent-written procedures, or consolidation lowers performance across the reported Diamond and Dragon ablations.For GPT-6-Astra, removing procedures yields 8.7 ± 0.6 Diamond milestones and 10.7 ± 1.2 Dragon milestones.Source: Table 4 · p. 8, §4.3 · p. 8

4.2 PERFORMANCE ON LONG-HORIZON TASKS

  • MFM retrieves much earlier history than VISTA, and MFM and PRO-LONG execute more actions per turn than VISTA with fewer total model queries.The comparison associates MFM’s temporal reach with its addressing layer and reports reduced execution overhead.Source: §4.2 · p. 8, Figure 6 · p. 9

4.4 HOW MEMORY RETRIEVAL HELPS IN LONG-HORIZON TASKS

  • Retrieved visual evidence helps agents revise uncertain interpretations, recover spatial context, and connect past observations to the current environment.Examples include distinguishing a wolf from a sheep, locating hole rims and mine entrances, and correcting unfavorable viewing angles.Source: §4.4 · p. 9
  • Only MFM builds the Nether portal within the 6,000-step budget, using 30 model calls, 77 tool calls, and 77 environment actions.VISTA uses 141/140/129 calls/actions and PRO-LONG uses 43/131/151, respectively.Source: Figure 4 · p. 9, §4.4 · p. 9, Figure 5 · p. 9

5 SELF-IMPROVEMENT WITH MFM

MFM agents turn interaction experience into reusable procedures and can inherit accumulated world knowledge for new tasks. Reusing procedures reduces steps on several tasks, while the benefit of inherited memory depends on whether it records the target ore’s location.

5.1 SKILL LEARNING THROUGH MEMORY

  • The agent creates and reuses procedures such as cut_step and quarry_north_pair, and bridge_north carries over between lives to support later obsidian mining.The agent also disables an inherited procedure after tracing a previous lava death to its release of the sneak key.Source: §5.1 · p. 10
  • Reusing skills acquired during earlier tasks reduces environment steps by 30% for Iron shield, 7% for Climb out, and 26% for Cook meat.The reported procedure ablation also lowers GPT-5.6-Sol mean milestone completion from 9.7 to 8.7 on Diamond and from 10.7 to 9.0 on Dragon.Source: §5.1 · pp. 10–11
  • Without agent-written visual-input skills, the evaluated methods and models cannot progress beyond milestone 16 to milestones such as killing a blaze or an enderman.The authors caution that not every stored procedure is beneficial and that a larger skill library does not necessarily improve performance.Source: §5.1 · p. 11

5.2 TRANSFERRING WORLD KNOWLEDGE TO NEW TASKS

  • Across evaluated trials, success in finding and mining iron, diamond, or gold rises from 1/12 without inherited memory to 11/12 with full memory.Agents whose memory records the target ore’s location generally succeed within budget; memory from before its discovery often does not prevent budget exhaustion.Source: §5.2 · p. 11

6 CONCLUSION

MFM organizes multimodal episodes, searchable cues, and revisable agent-written knowledge in one filesystem, linking knowledge and plans to supporting evidence. The conclusion reports gains across Minecraft and KiCad, while noting that exploration remains difficult and the evaluation is limited to single agents in two environments.

6 CONCLUSION

  • Across Minecraft and KiCad with three base models, MFM outperforms memory-free and memory-augmented baselines, while each component contributes and memory transfers across tasks.The conclusion also presents MFM as a possible foundation for memory in long-horizon multimodal and embodied agents.Source: §6 · p. 11
  • The evaluation is limited to single agents in Minecraft and KiCad, and even GPT-6-Astra at maximum reasoning effort did not complete Ender Dragon.The paper identifies coupling memory with exploration and route planning, shared memory across agents, and broader environments as future directions.Source: §6 · p. 11

A ADDITIONAL EXAMPLES

  • The appendix adds results and qualitative examples on Minecraft milestone progression and interaction, visual memory inspection, and CUABench-KiCad.It also includes qualitative examples on KiCad.Source: A ADDITIONAL EXAMPLES · p. 15

A.1 MILESTONE PROGRESSION AND INTERACTION IN MINECRAFT

  • VISTA reaches diamond equipment earlier, but MFM continues through obsidian, flint and steel, and netherrack within the 36,000-step budget.The figure shows one run per method; seed-averaged results are reported in Table 2.Source: §A.1 · p. 15
  • MFM generally executes more environment steps per turn than VISTA and PRO-LONG, but its comparison with PRO-LONG varies by task.On Ender Dragon, MFM exceeds both baselines for 41–56% of each run; on Mine Diamond, it does so for 5–14%, while PRO-LONG usually executes the most steps per turn.Source: §A.1 · p. 15

A.2 VISUAL MEMORY INSPECTION IN MINECRAFT

Minecraft examples show agents retrieving indexed stored frames and inspecting them to answer questions that textual records alone cannot resolve. The examples include identifying a place from an earlier life and diagnosing a failed mining cut.

A.2 VISUAL MEMORY INSPECTION IN MINECRAFT

  • Stored frames let the agent answer questions that the textual record alone cannot, including what animal it saw and where an earlier hole is located.The examples also ask whether a place from an earlier life matches the current view and why a previous mining cut failed.Source: §A.2 · pp. 15–16
  • Within 36,000 steps on Ender Dragon, GPT-6-Astra MFM reaches 16 of 24 milestones, versus 13 for VISTA, 12 for PRO-LONG, and 6 for Direct Evaluation.The figure shows one run per method; Table 2 reports seed-averaged results.Source: Figure 8 · p. 16
  • After death, the agent retrieves an earlier-life frame, recognizes the grass bank as the iron-mine entrance, and mines iron ore at step 6,565.It compares the retrieved frame with its current view.Source: §A.2 · p. 16

A.3 QUALITATIVE EXAMPLES ON CUABENCH-KICAD

KiCad examples illustrate MFM checking application state after edits and reusing retrieved episodes as precedents. Other examples show visual memory helping agents resolve uncertain observations and recover spatial context.

A.3 QUALITATIVE EXAMPLES ON CUABENCH-KICAD

  • MFM checks the resulting schematic state rather than trusting an edit operation’s “OK” response, catching an incomplete supply-label change that VISTA records as successful.On Task b7148878, MFM leaves one pin unconnected, compared with three for VISTA.Source: §A.3 · p. 16
  • MFM verifies edits by inspecting component values and net labels, running the Electrical Rules Checker, and confirming that only expected undriven-power errors remain.The agent reuses this three-step routine from an earlier task; the passage says it checks that no pins are left unconnected.Source: §A.3 · p. 16
  • The interaction figure reports rolling median environment steps per turn across three base models and two Minecraft tasks; higher values mean more steps between model calls.Shaded intervals mark where MFM exceeds both VISTA and PRO-LONG.Source: Figure 9 · p. 17
  • Magnifying stored frames resolves ambiguous animal observations: the agent distinguishes a wolf from a sheep and confirms a pink sheep rather than a pig.The first correction leads the agent not to attack; the second helps it hunt for food.Source: Figure 10 · p. 17
  • For netlist export, the agent retrieves an earlier episode with a nearly identical action sequence and follows it as a precedent.The sequence includes saving the blank schematic, opening Export →Netlist, selecting native KiCad format, and setting the output path.Source: §A.3 · p. 17
  • Retrieved and current views line up closely enough for the agent to identify a hole’s rim and recover its exact location.The example illustrates spatial relocation through visual memory.Source: Figure 11 · p. 18

A.4 INSPECTION STATISTICS

The inspection analysis measures whether opening a stored frame is followed by a new destination near the retrieved episode’s location. The pseudocode describes how MFM records and indexes episodes, retrieves evidence, and updates knowledge while preserving the episode archive.

A.4 INSPECTION STATISTICS

  • The analysis measures retargeting and recall targets across main Minecraft runs spanning three models, two tasks, and three seeds.A retarget names a go_to/look_at coordinate absent from the running plan; a recall target falls within 8 blocks of the inspected episode’s recorded position.Source: §A.4 · p. 18
  • On Mine Diamond, turns that inspect a stored frame retarget 17% of the time versus 7% without image inspection, and 42 of 48 new destinations lie near the retrieved location versus a 37% null.Ender Dragon shows 33% versus 14% retargeting and 15/17 nearby destinations against a 24% null; pooled results are 19% versus 11% and 57/65 against 33%.Source: §A.4 · p. 18

B PSEUDOCODE

  • MFM’s loop optionally retrieves past experience and consolidates knowledge before selecting an action, then records each interaction and finalizes episodes at boundaries or termination.Episode boundaries and consolidation triggers are supplied by the interaction protocol; retrieval may repeat as needed.Source: B PSEUDOCODE · p. 22
  • Episode recording stores each interaction interval with visual observations, structured states, actions, and metadata, then preserves the evidence and adds searchable cues and references to the index.The cues include available temporal, spatial, action, and outcome descriptions.Source: B PSEUDOCODE · p. 22
  • Retrieval filters the index by query predicates, selects references, and loads the referenced records for inspection.Textual cues locate candidate episodes, while the underlying evidence can contain details absent from those cues.Source: B PSEUDOCODE · p. 23
  • Consolidation can retrieve additional evidence before revising agent-written knowledge, while leaving the episode store and index unchanged.The update may revise plans, findings, relational memory, general guidance, or reusable procedures, and does not execute environment actions.Source: B PSEUDOCODE · p. 23
  • Stored evidence remains linked to its records rather than being replaced or compressed by organization and indexing.The agent selects which referenced observations or records to inspect.Source: B PSEUDOCODE · p. 23

C SKILLS

This section describes executable Minecraft skills, including a repeatedly verified spruce-approach refinement, an image-based blaze detector, and a combat skill. It also documents harness records for indexing episodes and mapping interacted world cells, alongside shared method configurations and implementation details.

C SKILLS

  • Parameterized routines cover descending cuts, paired-wall mining, one-block bridging, bounded tunnel extension, and three-block side probes.The routines specify preconditions and action sequences for these Minecraft tasks.Source: C SKILLS · pp. 24–27
  • The blaze detector segments warm-colored image blobs but accepts groups only when they include enough yellow rod-colored pixels, distinguishing blazes from orange fire.It requires at least one yellow-marked pixel and a yellow fraction of 10% or more of the group.Source: C SKILLS · pp. 28–29
  • The combat skill ranks detected targets by area, adjusts camera aim toward a target, and swings while turning when within specified angle limits.The code fragments also show movement and sneaking actions in the attack logic.Source: C SKILLS · p. 29

D.1 EPISODE INDEX

  • The harness writes one episode-index row per completed episode, recording its identifier, step range, action, summary, tags, and agent position.Summaries combine procedure calls with outcomes such as duration, movement, inventory changes, and milestones; the harness derives fields from game state rather than asking the model to write them.Source: §D.1 · p. 30

D.2 SPATIAL GRID

  • The append-only spatial grid records interacted cells as walked, placed, opened, blocked, or hazard, with item, pose, or damage details for relevant kinds.Placed and opened entries include the item and agent pose; hazard entries include damage amount and type.Source: §D.2 · p. 30

D HARNESS AND IMPLEMENTATION DETAILS

  • Methods share an action and procedure reference, knowledge sections, ground-truth rules, and a per-turn state block, while Direct Evaluation omits [RECENT] and [MAP].All methods with memory can write files and execute stored procedures through the same interface.Source: Table 6 · p. 31

D.3 METHOD CONFIGURATIONS

Minecraft methods share a 12 KB standing brief and closed-loop atoms, while PRO-LONG and VISTA retain their core memory algorithms. MFM and VISTA have matched visual-inspection tools, differing in evidence addressing and memory use.

D.3 METHOD CONFIGURATIONS

  • All Minecraft methods receive the same 12 KB standing brief and use closed-loop atoms that read structured game state at every step.No method has task-specific macros such as tree harvesting, strip mining, or portal building.Source: §D.3 · p. 31
  • PRO-LONG retains programmatic text logs, while VISTA retains visual observations and an inspection interface; both were reimplemented without changing their core algorithms.Source: §D.3 · p. 31
  • MFM and VISTA both store lossless observations and support reopening, cropping, enlarging, and comparing frames; they differ in how agents locate evidence and use it.VISTA uses turn and frame identifiers, while MFM adds searchable cues and agent-written memory linked to supporting episodes.Source: §D.3 · p. 31
  • MFM's index and linked frames let agents check prior actions and outcomes to verify candidate procedures, unlike the baselines' less structured durable memory.PRO-LONG's log lacks structured action and outcome fields and visual links; VISTA addresses frames by turn, and its writable memory is limited to two notes files.Source: §D.3 · p. 31

D.4 STORAGE AND COST

The reported evaluation spans repeated Minecraft and KiCad runs, while cost and storage tables are explicitly presented as single-run, order-of-magnitude examples. Ender Dragon runs exhaust their step budget and can incur substantial model-call and monetary costs.

D.4 STORAGE AND COST

  • All main and ablation results average 3 seeds; the study ran 96 Minecraft runs and 1,125 KiCad episodes.Minecraft runs were capped at 12k steps for Diamond and 36k for Dragon; seeds fix the world or task order, with stochastic sampling adding variation.Source: §D.4 · p. 31
  • All 48 Dragon runs exhaust their budget, consuming about 1.7M steps and up to 14 hours each.For Claude Opus 5 on Dragon, a run makes 0.9–3.9k model calls and costs $108–$969.Source: §D.4 · p. 32
  • Tables 7 and 8 report single-run storage, retrieval, memory-use, and Ender Dragon cost statistics as orders-of-magnitude illustrations.Table 7 covers a GPT-6-Astra Mine Diamond run reaching all 10 milestones; Table 8 covers Claude Opus 5 Dragon runs at 36,000 steps using Claude Code.Source: §D.4 · p. 32, Table 7 · p. 32, Table 8 · p. 32

E BENCHMARK SETUP AND PROMPTS

The appendix describes the Minecraft benchmark tasks, skill-reuse evaluation, and example prompts and action formats. It specifies task objectives and initialization as well as the information and instructions supplied to agents.

E.1 MCU TASKS DESCRIPTIONS

  • Mine Diamond has 10 milestones, while Kill Ender Dragon shares its early progression and adds 14; milestones need not be completed in list order.The Dragon list inserts Diamond as milestone 10, making Diamond milestone 11 and continuing added objectives from 12.Source: Figure 18 · p. 33, §E.1 · p. 33

E.2 NEW TASKS FOR EVALUATING SKILL REUSE

  • Three added tasks test skill reuse through new objectives: crafting an iron shield, climbing from an underground pocket, and cooking meat.They mix familiar intermediate objectives with new goals; Climb Out requires upward digging, unlike the supplied downward-or-level procedures.Source: §E.2 · pp. 33–34
  • The skill-reuse evaluation compares steps needed to complete each task with and without previously written skills; task columns align to Diamond milestones and distinguish shared, task-specific, and unscored initialized milestones.Source: §E.2 · p. 34, Figure 19 · p. 34

E.3 PROMPTS AND OUTPUTS EXAMPLES

  • The Nether Portal prompt specifies a 4-by-5 frame, six empty interior cells, ignition, and entering the portal, with ample materials supplied.The four frame corners may be absent; the task asks only for the shape to be built correctly and the portal entered.Source: §E.3 · p. 34
  • The Minecraft memory prompt treats logs and episode files as the life record, carries the agent's mine directory forward, and directs it to inspect frames around remembered places or surprising outcomes.Source: §E.3 · pp. 34–35
  • The working-memory prompt puts the current plan, its next prediction, and unresolved contradictions before the agent each turn, and says it survives a context reset.It distinguishes the current plan in working.md from world knowledge in GUIDE.md.Source: §E.3 · p. 35
  • The Minecraft per-turn example combines milestone progress, state, recent actions, inventory, and map information in the observation block.Source: §E.3 · p. 35
  • KiCad actions use screen coordinates normalized from 0 to 1000; task rules require screenshot checks before typing, saving, and terminating.The action reference includes click, drag, typing, key, wait, and screenshot operations.Source: §E.3 · pp. 35–36
Loading 2610.22686v2…