PaperlayerView exact revision on arXiv

Source-linked AI summary

XSkill: Continual Learning from Experience and Skills in Multimodal Agents

Guanyu Jiang, Zhaochen Su, Xiaoye Qu, Yi R. Fung

arXiv:2603.12056v3cs.AIcs.CL

TL;DR

Multimodal agents need better ways to overcome inefficient tool use, inflexible orchestration, and limited learning from past trajectories without parameter updates. XSKILL uses visually grounded task-level skills and action-level experiences in a continual dual-stream loop, and consistently outperforms tool-only and learning-based baselines across diverse benchmarks and models. Its knowledge streams also show complementary effects on robustness and zero-shot generalization.

  • Problem

    Multimodal agents struggle with inefficient tool use and inflexible orchestration, while existing continual-learning approaches rely mainly on text logs that miss visual decision signals.

  • Method

    XSKILL extracts, consolidates, retrieves, and adapts visually grounded task-level skills and action-level experiences from multi-path rollouts without parameter updates.

  • Results

    XSKILL consistently outperforms strong tool-only and learning-based baselines across diverse multimodal benchmarks and four backbone models, improving Average@4 by 2.58 to 6.71 points over tool-only baselines.

  • Takeaways & Limitations

    The complementary knowledge streams improve multimodal agent performance, tool-use robustness, and zero-shot cross-task transfer.

  • Takeaways & Limitations

    Accumulated knowledge may inherit and amplify biases from previous trajectories, and more capable agents may be misused for malicious automation involving sensitive visual data.

Abstract

from arXiv · show

Multimodal agents can now tackle complex reasoning tasks with diverse tools, yet they still suffer from inefficient tool use and inflexible orchestration in open-ended settings. A central challenge is enabling such agents to continually improve without parameter updates by learning from past trajectories. We identify two complementary forms of reusable knowledge essential for this goal: experiences, providing concise action-level guidance for tool selection and decision making, and skills, providing structured task-level guidance for planning and tool use. To this end, we propose XSkill, a dual-stream framework for continual learning from experience and skills in multimodal agents. XSkill grounds both knowledge extraction and retrieval in visual observations. During accumulation, XSkill distills and consolidates experiences and skills from multi-path rollouts via visually grounded summarization and cross-rollout critique. During inference, it retrieves and adapts this knowledge to the current visual context and feeds usage history back into accumulation to form a continual learning loop. Evaluated on five benchmarks across diverse domains with four backbone models, XSkill consistently and substantially outperforms both tool-only and learning-based baselines. Further analysis reveals that the two knowledge streams play complementary roles in influencing the reasoning behaviors of agents and show superior zero-shot generalization.

1. Introduction

Multimodal agents struggle with inefficient tool use, inflexible orchestration, and inadequate text-centered knowledge reuse. XSKILL addresses these limitations with visually grounded skills and experiences, achieving consistent gains across benchmarks and models.

  • Motivation: Multimodal agents often waste steps on simple problems, underexplore complex queries, and struggle to compose tools across tasks.Structured skills support reusable workflows, while context-sensitive experiences address execution decisions.
  • Motivation: Existing continual-learning approaches largely rely on textual trajectory logs, which inadequately capture decision signals grounded in visual observations.This limits reliable retrieval and adaptation of prior knowledge in multimodal settings.
  • XSKILL: XSKILL retrieves and adapts recalled knowledge to current visual contexts, producing grounded execution plans such as rotation and cropping for difficult visual tasks.Figure 1 contrasts this behavior with a baseline that fails to correct inverted images or isolate small objects.
  • Results: 2.58 to 6.71 points: XSKILL improves Average@4 over the tool-only baseline across four backbone models and gains up to 11.13 points over the strongest baseline.Evaluation spans visual agentic tool use, multimodal search, and comprehensive multimodal reasoning.
  • XSKILL: XSKILL unifies visually grounded task-level skills and action-level experiences in a training-free dual-stream framework.It extracts and consolidates knowledge from visual-tool interactions through multi-path rollouts, summarization, critique, and hierarchical consolidation.
  • Results: The two knowledge streams play complementary roles in improving tool-use robustness and enabling stronger zero-shot cross-task transfer.These analyses extend the reported performance gains beyond the main benchmark comparisons.

2. Methodology

XSKILL organizes multimodal-agent knowledge into complementary task-level skills and action-level experiences, then continually extracts, consolidates, retrieves, and adapts them using visual context. Its two-phase pipeline combines multi-path rollout analysis with task-decomposed, context-aware knowledge injection.

  • Knowledge Representation: XSKILL uses skills for structured task-level workflows and experiences for concise action-level guidance tied to execution context and failure patterns.These knowledge types are stored in a Markdown-based Skill Library and a JSON-based Experience Bank.
  • Phase I: Accumulation: During accumulation, multiple rollouts are summarized with visual observations, then cross-rollout critique distills generalizable knowledge from successful and failed trajectories.Summaries record decision points, tool-use patterns, failure reasons, and the visual evidence influencing subsequent decisions.
  • Phase II: Inference: During inference, XSKILL decomposes a test task into subtasks, retrieves relevant knowledge for each need, and adapts it to the current visual context before prompt injection.Subtask-specific queries improve coverage over single-query retrieval by targeting distinct technical demands.
  • Problem Formulation: The framework formulates multimodal tool use as a POMDP and optimizes an external knowledge base combining skills and experiences with the execution model.The formulation reflects that visual observations reveal only partial information about the underlying task state.
  • Phase I: Accumulation: Cross-rollout updates add or modify experiences, while hierarchical consolidation merges similar entries, removes low-quality items, and refines skill content for generalizability and conciseness.Skill refinement can update, merge, or remove sections and replace overly specific details with reusable placeholders.

3. Experiments

XSKILL is evaluated across diverse multimodal benchmarks, backbone models, and tool-use settings, showing consistent gains over tool-only and learning-based baselines. Ablations and behavioral analyses indicate complementary contributions from skills and experiences: skills improve execution reliability, while experiences support context-aware tool orchestration.

  • Experimental Setup: XSKILL is evaluated on five benchmarks spanning visual tool use, multimodal search, and comprehensive multimodal reasoning, using four backbone models.The evaluation includes VisualToolBench, TIR-Bench, MMSearch-Plus, and additional multimodal reasoning benchmarks.
  • Main Results: 2.58 to 6.71 points: XSKILL improves Average@4 over the tool-only baseline across models, with a maximum 11.13-point gain over Agent-KB on TIR-Bench with Gemini-3-Flash.Transferred knowledge also improves GPT-5-mini and o4-mini by 2.58 to 4.16 points over the tool-only baseline.
  • Ablation Study: Removing experiences or skills reduces performance by 3.04 or 3.85 points, while removing Phase 1 managers causes larger drops than removing Phase 2 components.The reported Phase 1 drops are 4.09 and 3.62 points, compared with 1.28 and 1.52 points for Phase 2.
  • Analysis: 29.9% (168 errors) to 15.3% (95 errors): skills reduce overall execution errors in the VisualToolBench error analysis.Syntax errors fall from 114 (20.3%) to 71 (11.4%), and tool name errors decrease from 16 to 2.
  • Analysis: Experiences shift tool selection toward task-specific strategies, increasing code-interpreter use on VisualToolBench and both code-interpreter and image-search use on MMSearch-Plus.On VisualToolBench, code-interpreter usage rises from 66.63% to 74.49% and reaches 76.97% in the full pipeline; on MMSearch-Plus, code-interpreter use rises from 6.18% to 13.21% and image-search calls from 15.43% to 24.63%.
  • Analysis: Performance improves consistently as rollout count increases, with Pass@4 showing steeper gains because richer trajectory diversity improves extracted knowledge.The analysis attributes this to cross-rollout critique and identification of common patterns across execution paths.
  • Analysis: Zero-shot transfer experiments show average improvements of 2 to 3 points over Agent-KB across benchmarks and backbone models.The reported transferability is associated with hierarchical consolidation and task adaptation.

4. Related Work

Multimodal agents have evolved into active tool-using problem solvers, but remain largely stateless and struggle to accumulate reusable reasoning capabilities without costly parameter updates.

  • Multimodal Agentic Reasoning: Modern multimodal agents actively manipulate visual data, synthesize executable code, and orchestrate web searches using visual cues.These capabilities extend beyond passive visual perception toward active “thinking with images.”
  • Multimodal Agentic Reasoning: Most existing agentic frameworks operate in isolated episodes, preventing agents from internalizing successful tool-use patterns or corrective feedback across tasks.The resulting redundancy motivates agents that accumulate procedural expertise from lifelong interactions.
  • Multimodal Agentic Reasoning: Parametric approaches such as reinforcement learning face high domain-specific training costs and difficulty adapting to evolving toolsets.These scalability bottlenecks motivate non-parametric mechanisms for continuous and flexible capability accumulation.
  • Multimodal Agentic Reasoning: Non-parametric agents can learn from retrieved execution trajectories or abstract them into reusable experiences and skills.Experiences and skills provide complementary forms of knowledge for improving decision-making and generalization.

5. Conclusion

XSKILL unifies visually grounded task-level skills and action-level experiences to help multimodal agents leverage past interactions. Its knowledge streams provide complementary benefits, while experiments support consistent improvements and zero-shot transfer.

  • 5. Conclusion: XSKILL unifies task-level skills with action-level experiences through visually grounded extraction and hierarchical consolidation.Visually grounded retrieval and adaptation connect accumulated knowledge to task-specific inference requirements.
  • 5. Conclusion: Skills improve tool-execution robustness, whereas experiences guide strategic tool selection according to task-specific contexts.Ablation studies identify these as distinct yet complementary advantages.
  • 5. Conclusion: Experiments across diverse benchmarks show consistent performance improvements over strong baselines and strong zero-shot transferability.The conclusion attributes these findings to the effectiveness and generalization of the unified framework.

Impact Statement

XSKILL externalizes skills and experiences into structured, human-readable knowledge for continual improvement without parameter updates. The framework also raises risks involving malicious automation and bias propagation, motivating oversight and auditing.

  • Impact Statement: XSKILL enables continual improvement without parameter updates by externalizing task-level skills and action-level experiences into structured representations.The representations make agent decision-making more transparent and interpretable.
  • Impact Statement: Human operators can audit, edit, or remove specific accumulated knowledge because skills and experiences are stored in explicit representations.This supports direct intervention in the knowledge base rather than requiring model retraining.
  • Impact Statement: More capable agents may be misused for malicious automation involving sensitive visual data.The impact statement identifies this as a potential misuse risk of the framework.
  • Impact Statement: Accumulated trajectories may transmit and amplify bias across models through the continual learning loop.Cross-model transfer means biased knowledge from one model could propagate to others without safeguards.
  • Impact Statement: The authors recommend human oversight, periodic bias audits, and access controls for cross-model knowledge transfer.These safeguards target accumulated knowledge bases, skill documents, experience banks, and transfer pathways.

A.1. Open-Source Model Evaluation

The open-source evaluation transfers knowledge accumulated by Gemini-3-Flash to Qwen3-VL models on VisualToolBench and MMSearch-Plus. Transfer helps on MMSearch-Plus but can interfere with weaker models’ native tool-use behavior on VisualToolBench.

  • A.1. Open-Source Model Evaluation: Qwen3-VL-235B-Instruct and Qwen3-VL-32B-Instruct are evaluated on VisualToolBench and MMSearch-Plus using knowledge accumulated by Gemini-3-Flash.The setup tests cross-model transferability without model-specific accumulation.
  • A.1. Open-Source Model Evaluation: Knowledge transfer improves performance on MMSearch-Plus for both evaluated Qwen models.The cited analysis reports gains for both open-source models on this benchmark.
  • A.1. Open-Source Model Evaluation: Knowledge transfer has negative Average@4 effects on VisualToolBench relative to the tool-only baseline.The table caption identifies Average@4 as a reported performance measure and Avg Turns as average tool invocations per task.
  • A.1. Open-Source Model Evaluation: Transferred knowledge can interfere with weaker models’ native tool-use behaviors while increasing exploratory behavior through more tool invocations.The analysis links the transfer setting to mixed effectiveness across benchmarks.

B.1. Dataset Details

XSKILL is evaluated on five multimodal benchmarks using disjoint training and test partitions, with Phase I accumulating knowledge and Phase II evaluating inference. The experiments also standardize key baseline settings and use two-phase hyperparameter configurations.

  • Benchmark suite: Five benchmarks cover visual agentic tool use, multimodal search, and comprehensive multimodal reasoning.
  • Data partitioning: Training and test sets are completely disjoint, with training data used for knowledge accumulation and test data used for inference evaluation.
  • Benchmark suite: MMBrowseComp is reserved exclusively for testing as an out-of-distribution transfer target from MMSearch-Plus.Its limited size motivates using all 130 samples for testing.
  • Experimental configuration: Phase I uses MLLMkb for trajectory processing, while Phase II uses lower-temperature task decomposition and retrieves the top-k = 3 most similar experiences.
  • Baseline comparison: Baselines share text-embedding-3-small retrieval, MLLMkb-based knowledge extraction, identical data splits, and aligned inference settings for fair comparison.

C. Method Details

The method details specify the multimodal tool environment and prompt-driven workflow for creating, consolidating, refining, and adapting reusable skills. Skill construction emphasizes generalizability, deduplication, actionable structure, and task-specific adaptation.

  • Tool environment: Agents receive four primary multimodal tools, with image references distinguishing original inputs from images generated by prior code executions.The Code Interpreter preserves execution state across calls.
  • Skill extraction: Trajectory-to-skill extraction asks an architect to identify reusable workflows, effective tool sequences, failure lessons, placeholders, and executable knowledge.
  • Skill consolidation: Skill consolidation integrates new material by rewriting better content, deleting redundancy, merging complementary elements, and adding genuinely distinct variants.
  • Skill refinement: Refinement removes task-specific details, converts hardcoded values into placeholders, consolidates overlapping workflows, and improves logical structure and scanability.
  • Skill adaptation: At inference, the agent selects relevant skill components, integrates experiences, preserves useful templates, and produces a focused methodology rather than a pre-filled answer.The adapted skill is limited to approximately 400 words and omits frontmatter metadata.

D.2.1. TASK DESCRIPTION

The task asks the agent to locate the second occurrence of “GOOD” on a taxi and identify the color area containing it. Solving it requires both text-instance counting and objective color assessment under visual ambiguity.

  • Task description: The question asks which color area contains the second “GOOD” text on the car body.
  • Task description: The input is a street-scene image containing a taxi with “GOOD GIFTING GOOD VALUES” text on its side.
  • Task requirements: The task requires locating and counting text instances on a small object, then identifying its background color objectively despite lighting variation and ambiguity.

D.2.2. CASE 1: DIRECT ANSWER VS. SYSTEMATIC ANALYSIS (ROLLOUT 2)

The case contrasts direct visual answering with systematic analysis guided by injected knowledge. XSKILL’s workflow combines text localization, ROI isolation, and forensic color analysis to correct the tool-only error.

  • Phase 1: Injected Knowledge: Injected knowledge recommends sequential text-instance scanning and objective color identification through ROI isolation and pixel sampling.
  • Phase 2: Agent Reasoning & Execution: The tool-only agent answers “red” incorrectly, whereas XSKILL’s guided ROI extraction and forensic color analysis produces the correct answer, “purple.”
  • Phase 2: Agent Reasoning & Execution: The case demonstrates that XSKILL can guide iterative refinement and more sophisticated color-space analysis.
  • Phase 2: Agent Reasoning & Execution: Across different rollouts, XSKILL improves consistency while enabling advanced strategies when initial RGB analysis is ambiguous.
Loading 2603.12056v3…