Source-linked AI summary
RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, Yang You
TL;DR
LLM agents face a challenge in using past experiences to inform current decisions. RAP stores and retrieves experiences matched to the current context, including multimodal information, and reports improvements across textual and multimodal benchmarks, with gains over ReAct on four named benchmarks.
Problem
Comprehensively leveraging past experiences in LLM-agent planning remains challenging, despite approaches that analyze failures or extract insights.
Source: §1 · p. 1Method
RAP stores context and action-observation trajectories, retrieves relevant experiences based on current context and multimodal information, and uses them through in-context learning to guide actions.
Source: §1 · p. 1Results
RAP reports improvements over prior SOTA methods across textual and multimodal benchmarks, with gains over ReAct of 33.6% on ALFWorld, 13.0% on Webshop, 18.2% on Franka Kitchen, and 12.7% on Meta World.
Source: §1 · pp. 1–2Takeaways & Limitations
Across four agent and robotics benchmarks, RAP demonstrates flexible use of past experiences according to current situations.
Source: §6 · p. 9The formulation considers an agent completing a task through interaction over a finite horizon H.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Owing to recent advancements, Large Language Models (LLMs) can now be deployed as agents for increasingly complex decision-making applications in areas including robotics, gaming, and API integration. However, reflecting past experiences in current decision-making processes, an innate human behavior, continues to pose significant challenges. Addressing this, we propose Retrieval-Augmented Planning (RAP) framework, designed to dynamically leverage past experiences corresponding to the current situation and context, thereby enhancing agents' planning capabilities. RAP distinguishes itself by being versatile: it excels in both text-only and multimodal environments, making it suitable for a wide range of tasks. Empirical evaluations demonstrate RAP's effectiveness, where it achieves SOTA performance in textual scenarios and notably enhances multimodal LLM agents' performance for embodied tasks. These results highlight RAP's potential in advancing the functionality and applicability of LLM agents in complex, real-world applications.
1. Introduction
RAP addresses the challenge of using comprehensive past experiences in LLM-agent planning. It stores and retrieves experiences according to current context, including multimodal information, and reports gains across textual and multimodal benchmarks.
Motivation
- The framework targets a gap in comprehensive use of past experiences for LLM-agent planning, beyond approaches focused on failure cases or within-task reflection.The paper identifies this gap as limiting agents’ adaptability and applicability in complex, real-world scenarios.Source: §1 · p. 1
Framework
- RAP stores past experiences and retrieves them by similarity to the current situation, using in-context learning to generate subsequent actions.Its memory includes context and action-observation trajectories; retrieval considers multimodal information.Source: §1 · p. 1
Results
- 33.6%, 13.0%, 18.2%, and 12.7% gains over ReAct on ALFWorld, Webshop, Franka Kitchen, and Meta World, respectively.The paper also reports significant improvements over prior SOTA methods in textual and multimodal environments.Source: §1 · pp. 1–2
Contribution
- RAP applies memory retrieval to both textual and multimodal embodied tasks, including decision-making and robotics settings.The paper describes this as a pioneering effort in memory retrieval for multimodal agents, to its knowledge.Source: §1 · pp. 1–2
2. Related Work
The related work covers language and vision-language model foundations, LLM agents, and retrieval-based memory. RAP is presented as using implicit retrieval across diverse experiences rather than relying on within-task reflection or explicit insight extraction.
2.1. Language Models and Vision-Language Models as Foundations
- LLMs provide linguistic and reasoning abilities, while VLMs integrate textual and visual inputs; the paper uses them for text-based and embodied agents, respectively.The work focuses on memory retrieval to support sequential decision-making in these settings.Source: §2.1 · p. 2
2.2. Language Models as AI Agents
- ReAct interleaves generated actions and environmental states; the paper extends this approach with object identification and context-based retrieval of relevant experiences.The agent can receive different experiences at different points as the situation changes.Source: §2.2 · pp. 2–3
2.3. Retrieval-Augmented Generation with Memory
- Unlike Reflexion and ADaPT’s within-task trajectory focus or ExpeL’s text-based reflection, RAP implicitly draws relevant components from diverse experiences without an extra insight-extraction step.The paper describes this approach as supporting generalization from successful tasks and extending memory use from textual to multimodal contexts.Source: §2.3 · p. 3
3. RAP: Retrieval-Augmented Planning
RAP organizes experience-based planning into memory, reasoning, retrieval, and execution: it stores successful trajectories, retrieves relevant past logs, and uses them to guide the Executor’s next actions.
3. RAP: Retrieval-Augmented Planning
- RAP comprises four components: Memory, Reasoner, Retriever, and Executor.The framework is presented for agents completing tasks in an environment over a finite horizon, with actions and observations forming the interaction trajectory.Source: §3 · p. 3, §3.1 · p. 3
3.1. Preliminaries
- At each timestep, the agent selects an action and receives an observation, with plans, actions, and observations comprising its trajectory.The agent first forms an overall plan for its assigned task and then interacts with the environment over a finite horizon.Source: §3.1 · p. 3
3.2. Memory
- Memory logs store successful task executions, including task information, an overall plan, and a trajectory of plans, actions, and observations.Text environments store textual observations, while multimodal environments store images from a fixed-viewpoint camera after each action.Source: §3.2 · p. 3
3.3. Reasoner
- The Reasoner generates overall plans, action plans, and retrieval keys from the task and the agent’s current situation and trajectory.When an action plan is generated, its retrieval key is based on that plan; for example, “I need to find the watch” yields “search watch.”Source: §3.3 · p. 4
3.4. Retriever
- The Retriever ranks logs using weighted task, overall-plan, and retrieval-key similarity, adapting key comparisons and component weights to the environment.Multimodal retrieval compares current and logged visual observations; textual search keys compare with logged textual observations, while object-interaction planning uses action similarity. For constrained task spaces such as Franka Kitchen, it retrieves logs of the same task type.Source: §3.4 · p. 4
3.5. Executor
- The Executor uses retrieved experiences aligned with the current context through in-context learning to generate the next action.It uses the current task trajectory in the same way as past experiences, while limiting new trajectories to a constant number.Source: §3.5 · p. 5
4. Experiments
RAP was evaluated on the text-based ALFWorld and WebShop benchmarks and the multimodal Franka Kitchen and Meta-World benchmarks. Reported results show higher textual-task scores than prior methods and enhanced embodied-agent planning on both multimodal benchmarks.
4.1.1. ALFWORLD
- ALFWorld evaluation covered 134 unseen games across six task types, with RAP run recursively to depth three.RAPtrain used 1,000 training tasks and memory from successful training tasks and prior trials.Source: §4.1.1 · p. 5
- 85.8% success rate for RAP and 91.0% for RAPtrain exceed ReAct (52.2%), Reflexion (74.6%), and ADaPT (71.6%) on ALFWorld with GPT-3.5.RAP was also effective with GPT-4 and Llama2-13b; across three trials, its success rates were higher than ReAct’s.Source: §4.1.1 · p. 5, Table 1 · p. 6
4.1.2. WEBSHOP
- WebShop memory expands with reward-1 successes, while retrieval relates remembered actions to both current reasoning and their task scenarios.WebShop actions are highly scenario-dependent, motivating the use of intra-task relationships when generating actions.Source: §4.1.2 · p. 6
- 48.0% success rate and 76.1% overall reward score: RAP exceeds the listed WebShop baselines on both measures.For GPT-3.5, the comparisons are against ReAct, Reflexion, and ADaPT.Source: Table 2 · p. 6, §4.1.2 · p. 6
4.2. Multimodal Environments
- The embodied evaluation used five subtasks and two camera views per benchmark, with 50 seeded trials for each task-view combination.The benchmarks require planning from visual observations in interactive 3D environments.Source: §4.2 · p. 7
- RAP significantly enhances embodied multimodal-agent planning on both Franka Kitchen and Meta-World.The evaluation used VLM agents and reports task success rates with and without RAP augmentation.Source: §4.2 · p. 7
5. Ablation Study
The ablations examine retrieval choices across ALFWorld and Webshop, report embodied-task results, and test whether memory transfers across models.
5. Ablation Study
- The ablation section introduces additional experiments and examines which components of experiences should be stored in memory.Source: §5 · p. 7
- Table 3 reports average success rates on Franka Kitchen and Meta-World, while Figure 6 evaluates LLaVA and CogVLM with and without RAP and reports that RAP notably enhances embodied-task performance.Source: Table 3 · p. 7, Figure 6 · p. 8
5.1.1. ALFWORLD EVALUATION
- RAPact and RAPobs show the effectiveness of changing retrieval information according to the situation, while RAPclip using image observations performs slightly better than text-observation retrieval.The ALFWorld retriever comparison includes action-only, observation-only, and CLIP-based image retrieval.Source: §5.1.1 · p. 7, Table 4 · p. 7
5.1.2. WEBSHOP EVALUATION
- 14.3% and 13.0% gains on GPT-3.5 Webshop overall reward and success rate, respectively, accompany gains of 6.5% and 5.0% on Llama2-13b.The combined RAP configuration uses observation, intra-task, and product-category retrieval.Source: §5.1.2 · pp. 7–8
5.2. Transfer Learning via Memory
- RAP permits memory constructed with GPT-3.5 to remain effective when evaluated with Llama2-13b, supporting experience sharing across models.The transfer test uses memory with dmax = 1.Source: §5.2 · p. 8, Table 6 · p. 8
6. Conclusion
RAP stores past experiences, retrieves relevant information from text or images, and guides subsequent actions. Across four agent and robotics benchmarks, it outperforms baseline methods and supports flexible use of experience for current situations.
6. Conclusion
- RAP stores past experiences and retrieves pertinent information from multimodal sources such as text and images to guide subsequent actions.Source: §6 · p. 9
- RAP demonstrates superior performance to baseline methods across various LLMs and four diverse agent and robotics benchmarks.Source: §6 · p. 9
- The reported results support flexible use of past experiences according to the current situation in language-agent decision-making.Source: §6 · p. 9
A. Parameters
The appendix specifies a Webshop temperature setting and provides prompts for plan generation, retrieval-key generation, and action selection.
A. Parameters
- Webshop experiments use temperature 0.0 because interactions with the web application require precise commands.Source: A. Parameters · p. 11
B.1.1. PROMPT FOR OVERALL PLAN IN REASONER
- The overall-plan prompt asks the reasoner to plan a new task from examples, such as planning to put a watch on a safe.The examples include a task to put a vase on a safe.Source: §B.1.1 · p. 11
B.1.2. PROMPT FOR RETRIEVAL KEY IN REASONER
- The retrieval-key prompt pairs prior reasoning examples with a new task and asks the reasoner to continue planning from them.One example asks for a plan to put a watch on a safe.Source: §B.1.2 · p. 12
B.1.3. PROMPT FOR ACTION PLAN IN REASONER AND FOR ACTION IN EXECUTOR
- The action prompt provides task information and examples, then asks for an action consistent with the plan.The example and target task concern locating an object and putting it on a safe.Source: §B.1.3 · p. 13
C. ALFWorld trajectories
The ALFWorld trajectories illustrate retrieval keys changing as tasks move from finding an object to carrying out later actions.
C. ALFWorld trajectories
- For the mug task, retrieval searches for a mug, finds it on shelf 1, then uses an action-put key to place it on desk 1.Source: C. ALFWorld trajectories · p. 14
- For the cloth task, the plan and retrieval key search for a cloth, and the example trajectory locates one in cabinet 2.Source: C. ALFWorld trajectories · p. 15
- After finding the cloth, the trajectory uses action-specific keys for cleaning and putting it away, and ends with the cloth placed in cabinet 3.Source: C. ALFWorld trajectories · p. 15
D. Webshop trajectories
The Webshop trajectories show agents selecting product options and completing purchases, with scores varying according to how closely the selected options match the request.
D. Webshop trajectories
- Selecting lw2 instead of the requested mn4 for the foundation led to a purchase score of 0.6666666666666666.The trajectory records a click on lw2 before Buy Now.Source: D. Webshop trajectories · p. 16
- Selecting mn4 for the requested foundation led to a purchase score of 1.0.The agent inspected the available color options, clicked mn4, and then clicked Buy Now.Source: D. Webshop trajectories · p. 16
- Buying the Capri Sun product after selecting assorted flavor received a score of 0.6666666666666666.The trajectory identifies the product as a pack of 10 priced at $2.55, then selects assorted flavor.Source: D. Webshop trajectories · p. 17
- Selecting Pacific Cooler flavor and the 6 fl oz (pack of 10) option before purchase received a score of 1.0.The trajectory also records selecting assorted flavor before clicking Buy Now.Source: D. Webshop trajectories · p. 17
E.1. Task Selection
RAP’s multimodal evaluation uses Franka Kitchen and Meta-World, with five selected tasks from each benchmark to assess manipulation and object-control abilities.
E.1. Task Selection
- Franka Kitchen and Meta-World are the two benchmarks selected for evaluating multimodal agents in controlled, real-world-simulating scenarios.The selected tasks are listed in Tables 8 and 9, respectively.Source: §E.1 · p. 18, Table 8 · p. 18, Table 9 · p. 18
- Five Franka Kitchen tasks test the agent’s ability to manipulate objects and controls.Table 8 lists the selected tasks.Source: §E.1 · p. 18, Table 8 · p. 18
- Five Meta-World tasks represent a broad range of object-manipulation actions.Table 9 lists the selected tasks.Source: §E.1 · p. 18, Table 9 · p. 18
E.2. Model Specification
The multimodal agent uses LLaVA or CogVLM with deterministic language generation, and an MLP policy network maps high-level plans to environment-specific control actions.
E.2. Model Specification
- The agent uses LLaVA-v1.5 13B or Cog-VLM 17B as its vision-language model.LLaVA-v1.5 uses Vicuna-v1.5 13B; Cog-VLM allocates 10 billion parameters to visual understanding and 7 billion to language processing.Source: §E.2 · p. 18
- Language generation uses temperature 0.0 to prioritize precision and determinism in planning responses.Source: §E.2 · p. 18
E.3. Policy Network details
- An MLP policy network translates high-level vision-language-model plans into precise, environment-specific low-level control actions.Source: §E.3 · p. 18
- The policy network is trained few-shot using 25 expert demonstrations per environment from D4RL.Each Franka Kitchen demonstration has 50 state-action pairs, while each Meta-World demonstration has 500.Source: §E.3 · pp. 18–19