Source-linked AI summary
Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Exploration
Sen Wang, Bangwei Liu, Zhenkun Gao, Lizhuang Ma, Xuhong Wang, Yuan Xie, Xin Tan
TL;DR
Long-horizon embodied agents need to use episodic memory, yet mainstream one-shot tasks underemphasize exploration and memory utilization. The paper introduces LMEE-Bench and MemoryExplorer, which combine multi-goal navigation, memory-based question answering, and reinforcement-trained active retrieval. The approach reports strong performance on LMEE-Bench and GOAT-Bench, while practical deployment remains limited by inference speed and long-term-memory accuracy and efficiency.
Problem
Existing one-shot embodied tasks emphasize task outcomes while neglecting exploration processes and memory utilization needed for long-horizon lifelong learning.
Method
The paper introduces LMEE-Bench and trains the MLLM-based MemoryExplorer with reinforcement learning to actively retrieve multimodal memories while jointly predicting actions, frontiers, and answers.
Results
MemoryExplorer achieves strong performance on both LMEE-Bench and GOAT-Bench, with experiments reporting enhanced autonomous exploration and active memory retrieval.
Takeaways & Limitations
Unifying episodic-memory reasoning with multi-goal exploration provides a benchmark and training framework for evaluating cognitive understanding, memory utilization, and decision-making together.
Takeaways & Limitations
MLLM-based embodied exploration has slow inference speed, and current methods still struggle with challenging tasks requiring long-term memory.
Abstract
from arXiv · showhide
An ideal embodied agent should possess lifelong learning capabilities to handle long-horizon and complex tasks, enabling continuous operation in general environments. This not only requires the agent to accurately accomplish given tasks but also to leverage long-term episodic memory to optimize decision-making. However, existing mainstream one-shot embodied tasks primarily focus on task completion results, neglecting the crucial process of exploration and memory utilization. To address this, we propose Long-term Memory Embodied Exploration (LMEE), which aims to unify the agent's exploratory cognition and decision-making behaviors to promote lifelong learning. We further construct a corresponding dataset and benchmark, LMEE-Bench, incorporating multi-goal navigation and memory-based question answering to comprehensively evaluate both the process and outcome of embodied exploration. To enhance the agent's memory recall and proactive exploration capabilities, we propose MemoryExplorer, a novel method that fine-tunes a multimodal large language model through reinforcement learning to encourage active memory querying. By incorporating a multi-task reward function that includes action prediction, frontier selection, and question answering, our model achieves proactive exploration. Extensive experiments against state-of-the-art embodied exploration models demonstrate that our approach achieves significant advantages in long-horizon embodied tasks. Our dataset and code will be released at https://wangsen99.github.io/papers/lmee/
1. Introduction
Existing embodied tasks emphasize outcomes while underrepresenting exploration and memory use. The paper introduces LMEE and MemoryExplorer to unify exploration, episodic-memory reasoning, and decision-making for lifelong embodied intelligence.
- Motivation: Lifelong embodied agents must operate continuously in dynamic, unfamiliar environments while retaining memories that support later recall and reasoning.The motivating example links multi-step exploration with answering a later question about a previously observed object.
- Research gap: Current paradigms emphasize goal navigation and embodied question answering, but one-shot tasks largely neglect the exploration process itself.Multi-goal navigation extends task horizons yet still overlooks how exploration contributes to scene understanding and decision-making.
- Research gap: Existing MLLM approaches often use memory passively through imitation, limiting autonomy and generalization to unseen scenarios.The paper contrasts passive trajectory replication with proactive memory use for exploration.
- Contributions: LMEE unifies multi-goal navigation with memory-based question answering to evaluate both exploration outcomes and episodic-memory use.Its benchmark measures navigation success and efficiency together with memory-based question-answering accuracy.
- Contributions: MemoryExplorer uses reinforcement fine-tuning and a multi-task reward combining action, frontier, and question-answering objectives to promote active memory retrieval.The framework integrates scene understanding, memory utilization, and planning-based decision-making.
3. Data Construction of LMEE
LMEE constructs exploration data from multi-goal navigation, stores multimodal observations as episodic memory, and generates goal-oriented questions for memory evaluation. The resulting benchmark varies task difficulty and supports training and testing of long-term exploration.
- Construction pipeline: The dataset construction pipeline contains task-instruction generation, exploration-trajectory generation, and memory-based question-answering generation.It builds a memory bank from observations collected during multi-goal navigation.
- Task instruction generation: Task instructions are generated from HM3DSem scenes, semantic regions, and object categories using an LLM.The source includes 145 training scenes and 36 test scenes with semantic labels.
- Memory construction: Each memory entry stores position, text features, and image features, with CLIP used to encode the latter modalities for similarity computation.The method weights text, visual, and distance similarities when comparing current states with memory entries.
- Memory construction: Adaptive filtering aggregates recent similarity scores and selects contextual memories using a threshold that preserves temporal consistency.Task difficulty is categorized as easy, medium, or difficult according to regions, goals, and target distance.
- Question-answering generation: Goal-oriented questions are generated from navigation-target observations to assess memory reliably and help the model track long-term task progress.Focusing questions on navigation targets avoids objects that may not have been observed because trajectories are non-unique.
4. Method
MemoryExplorer trains an end-to-end MLLM to retrieve relevant long-term memories and jointly predict actions, frontiers, and answers. Reinforcement fine-tuning uses GRPO and a multi-task reward to encourage autonomous tool use and structured decisions.
- Task formulation: MemoryExplorer receives task instructions, multi-view observations, goal-oriented questions, and externally stored multimodal memories, then outputs an action, frontier, and answer.The formulation targets integrated scene planning and decision-making under long-term-memory constraints.
- Memory retrieval: The model first generates a tool call and query, retrieves relevant memories, and produces a final response conditioned on the retrieved information.If retrieval fails, it falls back to the first-round response based on the original inputs.
- Reinforcement fine-tuning: RFT evaluates final responses rather than intermediate retrieval-tool responses, using GRPO to encourage autonomous decisions through final reward feedback.This training design aims to learn whether tool calls improve the resulting action, frontier, and answer.
- Multi-task reward: The multi-task reward combines action accuracy, frontier correctness, answer precision, and output-format completeness.A consistency coefficient penalizes logically inconsistent action-frontier pairs, while component weights control each reward contribution.
- Multi-task reward: Tool-use scaling lowers sub-rewards without external tools and amplifies them under tool-based reasoning, while clipping the total reward to [0, 1] stabilizes comparisons.The scaling factor is designed to encourage efficient memory-tool utilization.
5. Experiments
Experiments evaluate LMEE-Bench and GOAT-Bench using navigation, memory-based question answering, and ablations of retrieval and question-type supervision. Results indicate that active memory retrieval and varied question types improve exploration and answer quality.
- Benchmark and metrics: LMEE-Bench evaluates multi-goal navigation with Success Rate and SPL, alongside open-ended MLLM-Score and multiple-choice answer accuracy.A navigation episode succeeds when the agent finishes within 1 meter of the target.
- LMEE-Bench comparison: Table 2 compares embodied exploration methods on LMEE-Bench using open-ended scores and multiple-choice accuracy.
- Main results: MemoryExplorer achieves higher robustness and efficiency than existing embodied exploration methods on LMEE-Bench.
- GOAT-Bench comparison: On GOAT-Bench, RA-Mem improves success rate by actively generating memory queries, while MemoryExplorer further improves success rate and navigation efficiency.
- Ablation studies: Training with memory-retrieval tools significantly improves performance, and combining multiple question types outperforms using any single type.The model gradually learns more accurate tool invocation, which accompanies improved answer accuracy.
6. Conclusion
The paper introduces LMEE and MemoryExplorer to connect exploration, episodic memory, and decision-making in embodied agents. Experiments on LMEE-Bench and GOAT-Bench report strong performance, while ablations identify active memory retrieval as important for long-horizon navigation and question answering.
- Conclusion: LMEE unifies multi-goal exploration with memory-based reasoning through an episodic memory bank.
- Conclusion: MemoryExplorer uses reinforcement learning and a multi-task reward combining action prediction, frontier selection, and question answering to train proactive memory retrieval.
- Conclusion: MemoryExplorer achieves strong performance on both LMEE-Bench and GOAT-Bench.
- Evaluation scope: The subset evaluation shows no significant difference from the full test set, supporting its use for model evaluation.The main-text comparison used approximately 35% of the test set, or 58 of 166 tasks.
- Evaluation scope: Open-ended counting and relational questions remain more difficult than their multiple-choice counterparts for large models.
- Ablation evidence: Adding memory retrieval produces a significant gain, whereas autonomous-navigation supervision alone does not clearly improve performance.
9. Real World Testing
MemoryExplorer was tested on a ROSMASTER X3 in unstructured office environments to assess sim-to-real transfer. The agent completed navigation and memory-based question answering in two office scenarios, while a longer task exposed retrieval-related failures.
- Experimental setup: MemoryExplorer was deployed on a ROSMASTER X3 to test sim-to-real generalization in unstructured office environments.The evaluation used a meeting room and a reception room.
- Meeting-room test: In the meeting-room case, the agent found a bottle of water and rubbish bin, then answered that the bottle was on the chair.The memory bank was constructed from observation images before question answering.
- Reception-room test: In the reception-room case, the agent answered an unrelated tripod question by locating it in the room corner near a wall with a blue triangle.The question target was not among the navigation targets.
- Summary: The real-world experiments are reported as demonstrating robustness and generalization while coupling cognition with decision-making.This conclusion is based on the qualitative office tests described in the section.
- Complete task: A complete task combined five navigation goals with two memory-based questions, but incorrect retrieval caused failures on two navigation targets and one question.The agent successfully located the refrigerator, coffee machine, and nightstand; the second question was answered correctly after selecting the correct memory entry.
11. Failure Case Analysis
The failure analysis attributes errors to both navigation problems and memory-based question-answering limitations. These include ambiguous data, incorrect retrieval, and incorrect spatial descriptions by the multimodal model.
- Memory-based question answering: Memory-based question answering can fail because data-generation ambiguities produce incorrect answers even when the correct memory is retrieved.This is identified as one source of question-answering error.
- Memory-based question answering: MLLM spatial-understanding limitations can cause either retrieval of the wrong memory or an incorrect description after retrieving the correct memory.The analysis distinguishes retrieval errors from description-generation errors.
- Navigation: Navigation failures include selecting the wrong object in memory and exceeding the maximum exploration steps.These are listed alongside the question-answering failure modes.
12. Data Construction Details
The dataset construction maps labeled objects to rooms, generates multi-object navigation tasks, filters invalid instructions, and derives memory-based question-answer pairs from successful trajectories. Examples include spatial-relation and counting questions with corresponding ground-truth answers and predictions.
- Task generation: HM3DSem object and region labels are mapped to room names before language-model task generation.The construction uses an LLM to generate multi-object navigation tasks from object and room information.
- Question types: The examples include questions about object location, spatial relations, and visible-object counts, each paired with ground truth and model prediction.Examples cover a bottle of water, a wooden chair relative to a stone wall, and sinks visible in a kitchen.
- Task examples: A complete navigation example orders goals from the kitchen to the bedroom and then the living room and hall area.The illustrated task includes the refrigerator, coffee machine, dresser, nightstand, and wall picture.
- Task generation: Erroneous instructions containing non-existent or incorrect objects are automatically filtered during Habitat-sim trajectory generation.Only successfully navigated targets are passed onward for question-answer generation.
- Question-answer generation: Observation images from successfully navigated targets are used with VLM prompts to generate question-answer pairs.The resulting trajectory data supports multimodal training-sample construction.
- Quality examples: The displayed examples show incorrect predictions for both a chair’s relation to a stone wall and the number of sinks in a kitchen.The ground truths state that the chair is directly against the wall and that one sink is visible.
13. Training Details
MemoryExplorer training uses EasyR1, a simplified Verl framework, with hyperparameters specified in the paper and a training prompt illustrated separately. The reported run used eight NVIDIA H200 GPUs for approximately 60 hours.
- Training setup: MemoryExplorer was trained with EasyR1, a simplified version of the Verl framework.The paper provides the specific training hyperparameters in Table 10 and the training-format prompt in Figure 15.
- Training setup: The reported training run used 8 NVIDIA H200 GPUs and took approximately 60 hours.This is the paper’s stated hardware and duration for training.
14. Experimental Details
The paper builds on memory-based embodied exploration methods and implements frontier-based navigation with multimodal observations. Its approach uses memory snapshots or retrieved memories to support object finding, navigation, and embodied question answering.
- Related methods: 3D-Mem constructs a 3D memory bank from multi-view observations and uses object-category relevance to filter memories before MLLM input.The method stores observed images with object categories and masks, addressing context-window limits.
- Related methods: RA-Mem actively retrieves relevant memories through an MLLM-generated query and feature-similarity matching for navigation and embodied question answering.The retrieval mechanism improved model performance and reduced task completion time according to the cited passage.
- Proposed method: MemoryExplorer extends RA-Mem with reinforcement-learning fine-tuning to improve active memory retrieval and exploration capabilities.Unlike RA-Mem, which uses MLLM inference, MemoryExplorer trains the MLLM end to end.
- Navigation implementation: The navigation implementation selects a frontier as the target location and collects three egocentric views at 60° angular intervals for each time step.The original 1280 × 1280 views are resized to 360 × 360 for input.
15. Limitations
The paper identifies slow MLLM inference as the primary limitation of embodied exploration, preventing real-time execution. It also reports that current methods remain inadequate for challenging tasks requiring long-term memory, leaving storage and retrieval efficiency as open issues.
- Limitations: Slow MLLM inference prevents real-time execution of embodied tasks.The paper identifies this as the primary limitation of MLLM-based embodied exploration.
- Limitations: Current methods do not yet effectively handle challenging embodied tasks that require long-term memory.This boundary is reported from results on LMEE-Bench.
- Future directions: Improving the accuracy and efficiency of long-term memory storage and retrieval is identified as crucial for practical deployment.The paper presents this as a future research direction rather than a demonstrated capability.