Source-linked AI summary
MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences
Shijian Wang, Jiarui Jin, Runhao Fu, Zexuan Yan, Xingjian Wang, Mengkang Hu, Eric Wang, Xiaoxi Li, Kangning Zhang, Li Yao, Wenxiang Jiao, Xuelian Cheng, Yuan Lu, Zongyuan Ge
TL;DR
Multimodal agents need better ways to exploit past experience for complex, multi-step reasoning than coarse trajectory retrieval. MuSEAgent abstracts interactions into stateful atomic experiences, retrieves guidance through complementary wide- and deep-search strategies, and consistently outperforms trajectory-level baselines across fine-grained perception and multimodal reasoning tasks.
Problem
Existing experience-augmented agents commonly rely on coarse trajectory-level retrieval, which can introduce irrelevant noise when matching long, rigid interactions to new problems.
Method
MuSEAgent uses hindsight reasoning to abstract trajectories into atomic state-action experiences, organizes them in a quality-filtered bank, and retrieves them through compositional semantic viewpoints with Deep-and-Wide search.
Results
MuSEAgent consistently outperforms strong trajectory-level experience retrieval baselines by nearly 8% in average accuracy across fine-grained multimodal reasoning tasks.
Takeaways & Limitations
Stateful experiences capture generalizable multimodal reasoning and tool-use skills, supporting robust cross-domain adaptation without prior in-domain exploration.
Takeaways & Limitations
Performance on MME-RealWorld-Lite remains lower than Expel and zero-experience ReAct, indicating that specialized real-world visual tasks may benefit from target-domain experience.
Abstract
from arXiv · showhide
Research agents have recently achieved significant progress in information seeking and synthesis across heterogeneous textual and visual sources. In this paper, we introduce MuSEAgent, a multimodal reasoning agent that enhances decision-making by extending the capabilities of research agents to discover and leverage stateful experiences. Rather than relying on trajectory-level retrieval, we propose a stateful experience learning paradigm that abstracts interaction data into atomic decision experiences through hindsight reasoning. These experiences are organized into a quality-filtered experience bank that supports policy-driven experience retrieval at inference time. Specifically, MuSEAgent enables adaptive experience exploitation through complementary wide- and deep-search strategies, allowing the agent to dynamically retrieve multimodal guidance across diverse compositional semantic viewpoints. Extensive experiments demonstrate that MuSEAgent consistently outperforms strong trajectory-level experience retrieval baselines on both fine-grained visual perception and complex multimodal reasoning tasks. These results validate the effectiveness of stateful experience modeling in improving multimodal agent reasoning.
1 Introduction
MuSEAgent addresses the noise and inflexibility of trajectory-level retrieval by modeling multimodal reasoning as state-aware experience learning. It combines atomic state-action experiences with Deep-and-Wide retrieval and improves performance on fine-grained perception and complex multimodal reasoning tasks.
- Contribution: MuSEAgent abstracts historical trajectories into atomic state-action experiences through hindsight reasoning and stores them in a quality-filtered experience bank.Retrieval is conditioned on the agent’s current decision state rather than static initial task contexts.
- Method: Its compositional state representation indexes experiences across perceptual intent, tool execution history, and interaction context.These semantic viewpoints support retrieval over heterogeneous multimodal states.
- Method: Deep-and-Wide search enables adaptive retrieval by combining cross-task strategic knowledge with iterative refinement across complementary semantic viewpoints.The agent repeatedly queries its experience bank during inference.
- Results: Nearly 8% average-accuracy improvement over strong trajectory-level experience retrieval baselines is reported, particularly on fine-grained multimodal reasoning tasks.The authors attribute the benefit to state-level guidance mitigating contextual noise.
- Results: MuSEAgent improves fine-grained perception and complex multimodal reasoning compared with trajectory-based agents.This result is presented as a core contribution of the paper.
2 Related Work
Prior multimodal agents support interactive visual reasoning through tool use and planning, while experience-driven agents commonly reuse past interactions at the coarse trajectory level. MuSEAgent is positioned against the resulting rigidity and irrelevant retrieval noise.
- Visual Reasoning: Multimodal large language models have advanced vision-language understanding but remain fragile in multi-step visual reasoning.The related benchmarks include V* Bench, HR-Bench, MME-RealWorld-Lite, and ZoomBench.
- Multimodal Agent: Multimodal agents shift visual reasoning from single-pass inference toward interactive decision-making with tool use and planning.Prior systems coordinate vision modules or structured programs for complex visual reasoning.
- Experience-driven Agent Learning: Experience-driven agents reuse past trajectories for long-horizon reasoning, but most methods retrieve experiences at the coarse trajectory level.Long and rigid trajectories can introduce irrelevant noise when matched directly to new problems.
3 MuSEAgent
MuSEAgent abstracts high-quality state-level experiences from multimodal trajectories, indexes them through complementary semantic viewpoints, and exploits them with adaptive deep-and-wide retrieval to guide actions.
- Framework overview: Figure 1 summarizes the framework as Experience Abstraction followed by Experience Exploitation for determining the next action.The exploitation phase performs deep-and-wide search over the experience bank.
- Problem formulation: MuSEAgent models agentic reasoning as a stateful experience-driven process over multimodal observations and execution history.Its decision state includes the user instruction, visual observation, task descriptor, and execution history.
- Stateful Experience Abstraction: Experience Abstraction decomposes trajectories into atomic transitions and retains high-quality decisions as reusable experience units.Each retained unit contains the state, executed action, and abstracted decision guidance after hindsight evaluation.
- Compositional State Representation: Each experience is indexed with multiple embeddings derived from predefined compositions of heterogeneous state components.The viewpoints support retrieval through different contextual cues, and the policy selects a viewpoint at inference time.
- Deep-and-Wide Search: Wide Search retrieves Top-K experiences under a selected viewpoint to expose diverse contextual patterns for cross-task reuse.Retrieval breadth K controls how many experiences are returned.
- Deep-and-Wide Search: Deep Search iteratively queries complementary semantic viewpoints to refine guidance across task intent, perception, and tool-use history.The unified procedure selects viewpoints across refinement rounds and combines retrieved guidance before generating the execution action.
4 Experiments
MuSEAgent is evaluated across multimodal reasoning benchmarks, search-scaling settings, out-of-distribution transfer, and ablations of experience construction. It consistently improves over trajectory-level baselines, with performance depending on model size, search scope, experience sources, hindsight models, and filtering thresholds.
- Overall Performance: MuSEAgent’s relative gains are larger for smaller models, while absolute performance increases with model size to 69.76% on Qwen3.5-397B-A17B.Relative improvements are 7.97%, 3.25%, and 3.81% for Qwen3-VL-32B-Instruct, Qwen3-VL-235B-A22B-Instruct, and Qwen3.5-397B-A17B, respectively.
- Deep-and-Wide Search: Deep Search improves V* Bench accuracy from about 71% to 85% and HR-Bench accuracy from 66% to 78% as iterations increase from 0 to 3.Wide Search shows a similar improvement when retrieved experiences per search increase from 1 to 3.
- Deep-and-Wide Search: Performance peaks at 3 Deep Search iterations and 3 retrieved experiences, then slightly declines at scales of 4 or 5.The reported explanation is that excessive historical experiences add redundant context and dilute guidance for the current state.
- Generalization: OOD stateful experiences reach 59.64% average accuracy, exceeding Reflexion at 57.33% and Expel at 56.96%.On Zoom-Bench and HR-Bench, MuSEAgent exceeds the strongest in-domain baselines by 2.58 and 3.00 percentage points, respectively.
- Generalization: MME-RealWorld-Lite remains a boundary case, where MuSEAgent scores 40.00%, below Expel at 41.56% and zero-experience ReAct at 41.46%.The passage attributes this to distinctive real-world visual characteristics benefiting from target-domain experience.
- Ablation Studies: Combining successful and failed trajectories performs best, reaching 85.42% on V* Bench and 78.00% on HR-Bench.Failed-only experiences outperform successful-only experiences, suggesting correction of erroneous reasoning provides stronger guidance than reinforcing successful paths.
- Ablation Studies: GPT-4o produces the best hindsight-model results, reaching 85.42% on V* Bench and 78.00% on HR-Bench.Qwen3.5-397B-A17B reaches 83.33% on V* Bench, while Qwen3-VL-32B-Instruct reaches 78.12% and 75.00% on the two benchmarks.
5 Conclusion
MuSEAgent uses atomic state-action experiences, hindsight reasoning, and Deep-and-Wide retrieval to reduce trajectory-level noise and improve multimodal reasoning. Experiments report stronger performance and out-of-domain transfer, while future work targets online memory updates.
- MuSEAgent decouples historical episodes into atomic state-action pairs to mitigate noise and inflexible context modeling in trajectory-level retrieval.
- Hindsight reasoning constructs a granular stateful experience bank that agents dynamically query through a Deep-and-Wide search paradigm.
- Evaluations across diverse benchmarks show state-aware guidance consistently outperforms existing trajectory-based methods across multiple base model architectures.
- Out-of-domain experiments indicate that stateful experiences capture generalizable reasoning and tool-use skills without prior in-domain exploration.
- Future research will examine online experience construction so stateful memory can update during active interaction with open-ended environments.
A Datasets and Metrics
The evaluation uses multiple-choice benchmarks covering fine-grained visual grounding, real-world high-resolution scenarios, minute visual details, and intricate high-resolution perception.
- V* Bench evaluates detailed visual grounding through attribute recognition and spatial relationship reasoning on high-resolution images.It contains 191 curated multiple-choice samples.
- MME-RealWorld-Lite tests high-resolution perception across practical real-world scenarios using 1,919 multiple-choice items.The final score is average task accuracy.
- Zoom-Bench assesses fine-grained multimodal perception when decisive visual evidence is small and easily overshadowed by global context.The evaluation uses overall VQA accuracy on 621 multiple-choice questions.
- HR-Bench measures perception and reasoning on true high-resolution images, including fine-grained single-instance and cross-instance tasks.The evaluation selects 200 multiple-choice questions based on 8K-resolution images.
B Baseline Methods
The study compares MuSEAgent with standalone reasoning, tool-interacting, reflective-memory, and trajectory-distillation baselines.
- Vanilla CoT generates intermediate reasoning steps using only pretrained parametric knowledge, without tools or environmental feedback.
- ReAct interleaves reasoning and task-specific actions while using external environments to obtain observations and adjust plans.
- Reflexion stores evaluator-generated verbal reflections from task attempts in episodic memory to revise strategies and avoid repeated mistakes.The implementation extracts reflective experiences only from failed trajectories.
- Expel distills reusable generalized rules from both successful and unsuccessful interaction trajectories into an experience pool.
C Tool Descriptions
The model is equipped with diverse tools for visual interaction, calculation, and external information access, organized into four functional modules.
- MuSEAgent’s toolset spans visual inputs, calculations, and external information access through four functional modules.
C.1 Image Manipulation Tools
Image manipulation tools operate on specified image regions to crop, enlarge, or annotate visual areas for inspection and grounding.
- Zoom In enlarges a specified image region using a 2D bounding box and zoom factor.The tool returns the magnified image segment for inspecting minute details.
- Crop extracts a sub-region from the original image and returns it as a new image.It removes peripheral visual context to isolate the target area.
- Zoom In and Crop require valid normalized bounding boxes, while Zoom In additionally requires a zoom factor greater than 1.The tools return errors for invalid box ordering, out-of-range coordinates, or invalid zoom factors.
- Visualize Regions overlays labeled bounding boxes on an image to highlight regions of interest.Regions use normalized coordinates, with optional color and line-width controls.
C.2 Visual Perception Tools
Visual perception tools extract text, localize objects, estimate depth, and connect semantic objects with spatial measurements.
- OCR detects and extracts visible text from an input image.It returns recognized textual content to support reading and parsing tasks.
- Localize Objects identifies user-specified object categories and returns their 2D bounding boxes.The tool is intended to provide bounding boxes for other region-based tools.
- Estimate Region Depth computes statistical depth values within a designated bounding box.Supported modes include mean, minimum, and maximum depth; smaller values indicate closer regions.
- Estimate Object Depth combines semantic object recognition with monocular depth estimation to determine an object's distance.The tool estimates depth from an object specified by text description.
C.3 Semantic Similarity Tools
Semantic similarity tools compare images and text across modalities, producing alignment scores and identifying the best match.
- Image-to-Image Similarity compares a source image with reference images using visual embedding distances.It returns similarity scores and the best-matching reference image.
- The similarity tools return ranked-style scores and a selected best match for the compared images or texts.Each tool supports one reference modality against multiple candidates in the other modality.
- Image-to-Text Similarity evaluates how well multiple textual descriptions match a given image.The output includes scores for each text and the best match.
- Text-to-Image Similarity measures how well multiple images correspond to one text query.It functions as a targeted text-based image retrieval metric.
C.4 Reasoning and External Tools
Reasoning and external tools provide arithmetic, equation solving, web search, and safe external-query handling for agent workflows.
- Calculator evaluates standard arithmetic expressions and returns exact numerical results.The implementation restricts expressions to approved arithmetic syntax and handles division-by-zero errors.
- Solve Math Equation handles algebraic, calculus, and symbolic or numeric queries through WolframAlpha.It accepts natural-language mathematical questions or equations.
- WolframAlpha querying supports API-key rotation and retries for retryable failures such as rate limits and selected HTTP errors.The tool returns parsed results or an error after available keys and retries are exhausted.
- Web Search queries external search engines with text keywords and returns up to five relevant results.Results contain titles and snippets, with a default top_k of 3.
D Prompts
The prompts guide multimodal agents in planning tool use, retrieving and applying stateful experiences, and abstracting useful decision guidance from reasoning traces. They distinguish atomic state-action evaluation from trajectory-level summaries and calibrate advice by outcome and action quality.
- D.1 Evaluation Prompt: Agents should analyze task requirements, identify needed information, select suitable tools, plan tool sequences, and combine outputs with direct observation.
- D.1 Evaluation Prompt: Tools are intended for difficult visual details, precise localization or counting, and complex images requiring step-by-step regional analysis.
- D.1 Evaluation Prompt: Agents should avoid tools when cropping removes context or direct observation already determines the answer, while adapting when tool calls fail.
- D.1 Evaluation Prompt: When enabled, stateful experience retrieval occurs before each action, using a matching retrieval view and retrieved advice to guide the next decision.
- D.2 Prompt for Stateful Experience Abstraction: Stateful abstraction evaluates each state-action pair with a q_value and produces concise, generalizable advice for future agents without revealing task answers.
- D.2 Prompt for Stateful Experience Abstraction: The q_value scale ranges from essential or helpful actions to reasonable, wasteful, harmful, and critical-error steps based on their contribution to the outcome.
- D.2 Prompt for Stateful Experience Abstraction: Correct traces are judged by strategy and efficiency, whereas incorrect traces require identifying responsible steps and converting high-q_value errors into cautionary guidance.
- D.3 Prompt for Trajectory-level Experience Abstraction: Trajectory-level abstraction instead summarizes what worked or failed across the complete trace and gives one or two actionable, generalizable sentences for similar future tasks.