Source-linked AI summary
CogPlanner: Unveiling the Potential of Agentic Multimodal Retrieval Augmented Generation with Planning
Xiaohan Yu, Zhihan Yang, Chong Chen
TL;DR
Existing MRAG systems often use rigid, single-step retrieval despite the need to acquire relevant information and reformulate multimodal queries. The paper defines MRAG Planning and proposes CogPlanner, an iterative framework, reporting improved MRAG performance with limited added costs.
Problem
Existing MRAG frameworks rely on rigid, single-step retrieval and face challenges in acquiring relevant information and reformulating ambiguous, incomplete, or multi-hop queries.
Source: §1 · pp. 1–2Method
The paper defines MRAG Planning and proposes CogPlanner, which iteratively reformulates queries and selects retrieval actions, alongside CogBench for evaluation and fine-tuning.
Source: §1 · p. 2Results
CogPlanner reports more than 15% improvement over various MRAG approaches with less than 10% additional costs using Qwen2-VL-7B.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that MRAG Planning and CogPlanner improve response quality while minimizing redundant retrieval and can integrate with existing MRAG systems.
Source: §4.3 · p. 5, §7 · p. 9
Abstract
from arXiv · showhide
Multimodal Retrieval Augmented Generation (MRAG) systems have shown promise in enhancing the generation capabilities of multimodal large language models (MLLMs). However, existing MRAG frameworks primarily adhere to rigid, single-step retrieval strategies that fail to address real-world challenges of information acquisition and query reformulation. In this work, we introduce the task of Multimodal Retrieval Augmented Generation Planning (MRAG Planning) that aims at effective information seeking and integration while minimizing computational overhead. Specifically, we propose CogPlanner, an agentic plug-and-play framework inspired by human cognitive processes, which iteratively determines query reformulation and retrieval strategies to generate accurate and contextually relevant responses. CogPlanner supports parallel and sequential modeling paradigms. Furthermore, we introduce CogBench, a new benchmark designed to rigorously evaluate the MRAG Planning task and facilitate lightweight CogPlanner integration with resource-efficient MLLMs, such as Qwen2-VL-7B-Cog. Experimental results demonstrate that CogPlanner significantly outperforms existing MRAG baselines, offering improvements in both accuracy and efficiency with minimal additional computational costs.
1 Introduction
CogPlanner addresses rigid MRAG retrieval by planning information acquisition and query reformulation dynamically. The paper introduces CogBench and reports substantial gains over existing MRAG approaches with limited added cost.
1 Introduction
- Rigid, single-step MRAG can retrieve irrelevant information, miss opportunities to use MLLM reasoning, and fail on ambiguous or incomplete queries and multi-hop tasks.The cited limitations include redundant or counterproductive retrieval and the need for multi-step reasoning.Source: §1 · pp. 1–2
- MRAG Planning optimizes information seeking and integration through information acquisition and query reformulation tailored to each multimodal query.Its sub-tasks include choosing retrieval strategies and decomposing complex queries into clearer atomic sub-queries.Source: §1 · p. 2
- CogPlanner iteratively refines queries and selects image search, text search, or no retrieval through a centralized planning expert.It supports parallel and sequential modeling, which differ in the ordering of reformulation and retrieval selection.Source: §1 · p. 2
- CogBench contains over 5,000 samples, including a 401-sample high-quality test set, and supports evaluation and fine-tuning for resource-efficient MLLMs.The introduction describes lightweight integration of Qwen2-VL-7B-Instruct as the planning expert.Source: §1 · p. 2
- More than 15% improvement over various MRAG approaches with less than 10% additional costs is reported for Qwen2-VL-7B.The reported result accompanies the CogPlanner and CogBench contributions.Source: §1 · p. 2
2 Related Work
Related work traces query processing from heuristic multi-stage pipelines toward LLM prompting, while MRAG research expands retrieval to multimodal sources. Existing MRAG methods and benchmarks motivate query-specific planning beyond rigid single-modality search.
2.1 Query Processing in IR
- Earlier IR systems used human-defined, multi-stage query-processing techniques, while LLMs can perform several traditional tasks within a prompt.In RAG, the challenge shifts toward choosing an effective strategy for processing user queries.Source: §2.1 · p. 2
2.2 Multimodal Retrieval Augmented Generation
- MRAG extends RAG to external visual knowledge, and prior systems include visual retrieval, text-based descriptions, and combined text-and-image retrieval.Examples discussed include MuRAG [5], Plug-and-Play [26], RAMM, Wiki-llava, mR2AG [33], and M2RAG [21].Source: §2.2 · pp. 2–3
- MRAG benchmarks such as MRAG-Bench [8] and MMSearch [12] evaluate retrieval, but prevailing methods largely use rigid, single-modality search.The paper introduces MRAG Planning to determine query-processing strategies according to each multimodal query and underlying MLLM.Source: §2.2 · p. 3
3 Task Formulation
MRAG Planning frames multimodal retrieval and generation as a sequence of decisions that moves from an initial query toward a sufficiently informed state. Its decisions jointly cover acquisition strategy and query reformulation, with retrieval, refinement, and response generation defined as connected stages.
3 Task Formulation
- A multimodal query combines textual and visual components, and planning frames progress from the initial state toward goal conditions for a comprehensive, accurate response.The formulation describes available information as state and planning as a state-transition function.Source: §3 · p. 3
3.1 Multimodal Retrieval Augmented Generation Planning
- The planning decision space has two components: an information-acquisition strategy and a query-reformulation result.The paper conceptualizes these as interrelated optimization sub-tasks.Source: §3.1.1 · p. 3, §3.1 · p. 3
- Planning proceeds iteratively over T rounds, with each decision based on the current information state under a Markovian transition assumption.The state can include historical queries or retrieved document elements.Source: §3.1.1 · p. 3
3.1.2 Information Aquisition.
- The acquisition action is chosen among text search, image search, and non-search based on available information quality and the estimated utility of more external context.The acquisition mechanism supplements knowledge gaps in MLLMs.Source: §3.1.2 · p. 3
- After an action is selected, in-document retrieval finds and extracts relevant elements under that strategy to produce retrieved document elements..Source: §3.1.2 · p. 3
3.1.3 Query Reformalization.
- Query reformulation uses textual and visual cues to refine ambiguous or incomplete queries and decomposes complex multi-hop queries into manageable sub-queries.The decomposition is intended to preserve semantic relationships.Source: §3.1.3 · p. 3
- Once the final state is sufficiently informed, response generation combines the initial query, final refined query, and relevant document elements.The formulation makes sufficient information the condition for concluding planning and generating a response.Source: §3.1.4 · p. 3
4 Methodology
CogPlanner replaces fixed, modality-specific retrieval with adaptive planning that iteratively reformulates queries and selects retrieval actions. It supports parallel and sequential planning, then generates a response once the acquired information and query are sufficiently clear.
4.1 Baselines
- Fixed text and visual retrieval workflows are modality-specific and can add noise in non-search scenarios, limiting flexibility across contexts.The text workflow retrieves documents from textual queries; the visual workflow retrieves images and captions from visual queries.Source: §4.1 · pp. 3–4
4.2 CogPlanner
- CogPlanner adapts to complex multimodal queries through iterative decisions about query reformulation and retrieval actions.The framework is inspired by human information gathering and query refinement.Source: §4.2 · p. 4
4.2.1 Planning Expert.
- The planning expert is an MLLM that evaluates gathered information and determines follow-up retrieval and query reformulation at each iteration.Parallel modeling makes those decisions concurrently, while sequential modeling reformulates first and then evaluates the need for further retrieval.Source: §4.2.1 · pp. 4–5
4.2.2 Retrieval and Generation.
- Retrieval continues until the expert judges the information sufficiently comprehensive and the query adequately clear, after which CogPlanner generates the response.The retrieval pipeline can invoke text or image search and preprocesses retrieved material for MLLM consumption.Source: §4.2.2 · p. 5
4.3 Compatibility
- CogPlanner is model-agnostic: its planning expert can be an MLLM or a traditional classifier, supporting integration with different MRAG systems.The evaluated planning experts include GPT-4o [9], Qwen-VL models [3], and Pixtral models [7].Source: §4.3 · p. 5
5 CogBench Construction
CogBench is a benchmark for MRAG planning, built from reviewed multimodal queries and records of planning and generation. Its analysis describes the benchmark’s query complexity and its use in fine-tuning resource-efficient planning experts.
5 CogBench Construction
- CogBench comprises over 5,000 samples and a test set of more than 400 samples for evaluating MRAG planning frameworks.The benchmark can also support fine-tuning to strengthen MLLM decision-making.Source: §5 · p. 5
5.1 Query Collection
- CogBench queries are developed from web-sourced screenshots and manually reviewed image-query pairs designed to be distinct, unambiguous, natural, and specific.Claude-3.5-sonnet generates candidate queries, and reviewers retain one to five compelling queries per image.Source: §5.1 · p. 5
5.2 MRAG Planning and Generation
- Each sample records the multimodal query, retrieval actions, iterative reformulations, retrieved documents, and a golden answer; annotations target answer correctness and completeness rather than a unique planning path.The split is reported as 5,307 training samples and 401 test samples.Source: §5.2 · pp. 5–6
5.3 CogBench Analysis
- 24.19% of CogBench queries are open-ended, their answers average 40.13 tokens, and 79.55% of cases explicitly require MRAG Planning.The benchmark contains 5,718 queries across nine cognitive domains.Source: §5.3 · p. 6, Table 1 · p. 6
5.4 Lightweight Integration of CogPlanner
- The lightweight integration fine-tunes Qwen2-VL-7B-Instruct on CogBench training data, mixed 1:1 with general instructions, to produce Qwen2-7B-VL-Cog.The stated aim is to reduce resource requirements while retaining the base model’s general capabilities.Source: §5.4 · p. 6
6.1 Experimental Settings
The experiments compare direct MLLM answering, fixed retrieval, self-reflective RAG, and CogPlanner’s parallel and sequential variants across several backbones. Evaluation covers end-to-end answer quality and query reformulation, with token- and claim-level metrics.
6.1 Experimental Settings
- Six configurations compare direct MLLM answering, fixed image and text retrieval, self-reflective RAG, and parallel and sequential CogPlanner.The selected backbones include GPT-4o, Qwen2-VL-72B-Instruct, Pixtral-Large-Instruct, and Qwen2-7B-VL-Cog.Source: §6.1.1 · p. 6, §6.1 · p. 6
6.1.1 End-to-End MRAG Performance.
- CogPlanner shows substantial improvements over fixed-pipeline baselines on CogBench, where diverse planning procedures degrade those baselines.Table 2 reports claim-level precision and recall and token-level F1.Source: Table 2 · p. 7
6.1.2 Planning Procedure Performance.
- Query reformulation is evaluated against human-expert annotations using BLEU, ROUGE, and F1 across parallel, sequential, and direct GPT-4o approaches.The end-to-end evaluation also uses token-level F1 and claim-level precision and recall.Source: §6.1.1 · pp. 6–7, §6.1.2 · pp. 7–8
6.1 Experimental Settings
- CogPlanner is capped at three planning iterations, and fine-tuning uses two epochs with batch size 32 and learning rate 2e-6.The experiments use the Llama-Factory framework and eight NVIDIA A800 GPUs.Source: §6.1.3 · p. 8
6.2 CogPlanner Performance
CogPlanner improves MRAG performance, with stronger gains on multi-hop queries; its reformulation performance also depends on the planning-expert model. Fixed search strategies can hurt direct MLLM responses, while sequential modeling offers no substantial advantage over parallel modeling.
6.2 CogPlanner Performance
- 12.4% to 52.5% end-to-end gains over baseline MRAG systems, with at least a 41.45% improvement over self-reflective variants, are reported for CogPlanner with GPT-4o.CogPlanner with GPT-4o yields the best performance among the compared configurations.Source: §6.2.1 · p. 8
- The two fixed search strategies generally underperform direct MLLM responses, while sequential modeling shows no substantial advantage over parallel modeling.Fixed retrieval can add noise for concise queries and screenshot images; the authors suggest current MLLMs struggle to evaluate complementary information sequentially.Source: §6.2.1 · p. 8
- CogPlanner’s improvements are more pronounced on multi-hop queries, where retrieval from multiple sources and multi-step reasoning are needed.Performance varies across one- to three-hop categories.Source: §6.2.1 · p. 8
6.2.1 End-to-end Performance.
- GPT-4o with parallel modeling is the highest-performing tested query-reformulation setting, and reformulation performance depends strongly on the planning-expert MLLM.CogPlanner improves the accuracy and informativeness of the original multimodal queries.Source: §6.2.2 · p. 8
6.3 Lightweight Model Performance
The fine-tuned Qwen2-7B-VL-Cog planning expert approximates the larger model on most evaluation metrics. It uses fewer resources for planning, while Table 5 characterizes retrieval-action proportions across methodologies.
6.3 Lightweight Model Performance
- Qwen2-7B-VL-Cog closely approximates Qwen2-VL-72B-Instruct across most evaluation metrics as a CogPlanner planning expert.Both use Qwen2-VL-72B-Instruct for final-answer generation.Source: §6.3.1 · p. 8, Table 4 · p. 9
6.3.2 Efficiency Evaluation.
- 10% more token consumption and latency reduced to 30% of Qwen2-VL-72B-Instruct are reported for Qwen2-7B-VL-Cog as the planning expert.The efficiency comparison excludes final-answer generation; parallel execution outperforms sequential modeling on both efficiency metrics.Source: §6.3.2 · p. 8, Table 4 · p. 9
- Table 5 reports retrieval-action proportions across methodologies, with categories for no search, text search, and image search.The table caption labels these categories # No, # Text, and # Image.Source: Table 5 · p. 9
6.4 Analysis
The analysis finds that MLLMs often take more retrieval actions than human annotators expect, while Qwen2-7B-VL-Cog produces a more reasonable number. Most tasks resolve around two retrieval hops, leaving optimal strategy selection challenging.
6.4 Analysis
- Figure 4 shows that all examined MLLMs tend to perform more retrieval actions than human annotators expect, with Qwen2-VL-72B-Instruct exceeding two rounds even on one-hop queries.Qwen2-7B-VL-Cog produces the most reasonable number of retrieval actions, and parallel modeling reduces redundant actions relative to sequential modeling.Source: Figure 4 · p. 9, §6.4.1 · p. 9
6.4.1 Analysis on planning procedure of Cogplanner.
- Qwen2-7B-VL-Cog predominantly uses text search, and most tasks appear to resolve around two retrieval hops even when queries require more.The authors note that additional web searches for reassurance do not necessarily harm overall performance.Source: §6.4.1 · p. 9
- Selecting an optimal retrieval strategy matched to an MLLM’s knowledge base and reasoning capabilities remains challenging.The analysis reports that MLLMs generally acquire more information and make conservative decisions to ensure accuracy.Source: §6.4.1 · p. 9
7 Conclusion
The conclusion presents MRAG Planning as a way to dynamically optimize information acquisition and query reformulation. CogPlanner and CogBench support this approach through planning and evaluation, respectively.
7 Conclusion
- CogPlanner refines queries and selects retrieval strategies to minimize redundant retrieval and enhance response quality.The framework offers flexible modeling approaches and integrates with existing MRAG systems.Source: §7 · p. 9
- CogBench is proposed to assess MRAG Planning decision-making, addressing a gap in current evaluation methods.Source: §7 · p. 9
- Experiments validate MRAG Planning and CogPlanner, reporting substantial performance improvements with minimal additional computational costs.The conclusion frames this work as groundwork for more adaptive MRAG frameworks.Source: §7 · p. 9