PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topology

Zheng Nie, Ruolin Shen, Xinlei Yu, Bo Yin, Jiangning Zhang, Xiaobin Hu

arXiv:2604.17503v1cs.AIcs.MA

TL;DR

Visual multi-agent systems often use fixed topologies and static skills, despite the need to align collaboration with multimodal query content. SkillGraph jointly evolves skills and topology through MMGT and a self-evolving Skill Bank. It reports consistent superiority over fixed-topology and static-skill baselines across diverse VLM backbones and benchmarks.

  • Problem

    VMAS commonly use fixed topologies blind to query content and static agent skills, limiting adaptation to multimodal tasks.

    Source: §1 · p. 2

  • Method

    SkillGraph uses MMGT to predict query-conditioned collaboration graphs and a Skill Designer to refine skills from failure cases, feeding updated skill representations back into MMGT.

    Source: §1 · p. 3

  • Results

    SkillGraph demonstrates consistent superiority over fixed-topology and static-skill baselines across diverse VLM backbones and benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SkillGraph couples evolving agent expertise with routing, allowing updated skill representations to reshape topology prediction without additional parameter updates.

    Source: §5 · p. 17

Abstract

from arXiv · show

Scaling vision-language models into Visual Multiagent Systems (VMAS) is hindered by two coupled issues. First, communication topologies are fixed before inference, leaving them blind to visual content and query context; second, agent reasoning abilities remain static during deployment. These issues reinforce each other: a rigid topology fails to leverage richer agent expertise, while static agents lack incentives to specialize for a given query. We address this with SkillGraph, a joint framework that evolves both agent expertise and communication topology. Within this framework, a Multimodal Graph Transformer (MMGT) encodes visual tokens, instruction semantics and active skill embeddings to predict a query-conditioned collaboration graph, replacing hand-crafted routing with dynamic, content-aware information flow. Complementing this, a Skill Designer distills and refines reasoning heuristics from failure cases, constructing a self-evolving multimodal Skill Bank. Crucially, updated skill embeddings are fed back into the MMGT, enabling the topology to adapt alongside capability growth. Experiments show that SkillGraph achieves consistent improvements across four benchmarks, five common MAS structures and four base models. Code is available at https://github.com/niez233/skillgraph.

1 Introduction

Current VMAS frameworks are limited by fixed communication topologies and static agent skills, despite visual tasks requiring query-specific collaboration. SkillGraph jointly evolves agent expertise and routing through a multimodal topology predictor and a self-evolving Skill Bank.

1 Introduction

  • Fixed topologies and frozen skills leave current VMAS unable to adapt routing or agent reasoning to a query’s multimodal demands.Existing pathways are set before query content is analyzed, and agents lack a mechanism to refine heuristics from persistent test-time blind spots.Source: §1 · p. 2
  • MMGT predicts query-conditioned communication topologies by jointly modeling visual tokens, instruction semantics, and agents’ active skill representations.This replaces hand-crafted wiring with content-aware routing aligned to task requirements.Source: §1 · p. 3
  • Skill-Conditioned Agents use dynamically retrieved Skill Bank entries encoded as node features to represent each agent’s current reasoning state.The contribution also introduces a Skill Designer that diagnoses recurring failures and refines reasoning heuristics.Source: §1 · p. 3
  • SkillGraph closes the loop by feeding updated skill representations back into MMGT, linking capability growth with changes to communication structure.The framework aims to condition orchestration jointly on the query and active skills while allowing skills to improve through interaction experience.Source: §1 · p. 3

2 Related Work

Prior graph-based multi-agent work developed structured, learnable, and query-adaptive collaboration, but topology prediction remained text-only. Skill-library research developed reusable and evolving skills, while generally leaving skills and collaboration structure decoupled.

2.1 Multi-Agent Systems as Graphs

  • Graph-based multi-agent systems progressed from workflow and debate structures to learnable or query-adaptive topologies, but existing methods left visual query content outside topology prediction.SkillGraph conditions topology on question semantics and per-agent image attention.Source: §2.1 · pp. 3–4

2.2 Self-Improving Agents with Skill Libraries

  • Skill-library methods support experience reuse and skill evolution, but prior frameworks largely decoupled skill updates from collaboration structure and visual features.SkillGraph links the Skill Bank to MMGT node representations and feeds topology predictions into skill-selection signals during training.Source: §2.2 · p. 4

3 Method

SkillGraph couples dynamic agent skills with query-conditioned communication topology in a loop: agents retrieve skills, MMGT designs the graph, and a Skill Designer refines skills from failures. The method encodes multimodal task context and active skills for routing, then feeds evolved skills back into topology prediction.

3 Method

  • The framework links skill-based agent construction, query-conditioned topology design, and failure-driven skill evolution in a closed loop.Updated skills feed back into MMGT, coupling capability changes with communication structure.Source: §3 · pp. 4–5

3.1 VMAS Construction

  • Agents retrieve structured reasoning skills from a hierarchical bank, and their trigger conditions and strategies are encoded as node features that update when skills change.Each skill also tracks a running accuracy estimate, a bounded failure buffer, and a version counter.Source: §3.1 · pp. 5–6

3.2 Multimodal Graph Topology Design

3.3 Adaptive Skill Evolution

  • The Skill Designer uses accumulated failures to identify repeatedly hard skills, then revises their trigger or strategy or creates a skill for an uncovered visual sub-task.Modified skills clear their failure buffers; new skills enter retrieval with accuracy initialized to zero.Source: Failure Accumulation · p. 9
  • Skill changes update MMGT’s skill-derived representations, while the predicted topology determines which agents and skills are exercised and which failure records accumulate.This couples topology prediction and skill refinement in both directions.Source: Co-Evolution Loop · pp. 9–10

3.4 Training Objective

  • Because graph sampling is discrete, MMGT is trained as a stochastic policy that increases the probability of sampled edges in topologies yielding correct answers.The update uses a binary correctness utility and a policy-gradient objective.Source: §3.4 · p. 10

4 Experiments

SkillGraph is evaluated on four multimodal benchmarks, using a primary Qwen3-VL-8B backbone and additional vision-language backbones. Its results cover overall benchmark performance, component ablations, model scale, iteration trends, and skill-bank evolution.

4.1 Settings

  • Evaluations cover MMBench, MathVista, RealWorldQA, and InfoVQA, using accuracy for the first three and ANLS for InfoVQA.The benchmarks assess perception, reasoning, and knowledge-intensive understanding; ANLS measures soft string matching for text-centric VQA outputs.Source: §15 · p. 11

4.2 Main Results

  • SkillGraph improves accuracy across all four primary benchmarks, with stronger gains on MathVista and RealWorldQA and smaller gains on InfoVQA.Table 1 compares SkillGraph enhancements with multi-agent topology baselines on Qwen3-VL-8B-Instruct/Thinking.Source: §4.2 · p. 11, Table 1 · p. 12
  • SkillGraph consistently improves performance across diverse vision-language backbones, including LLaVA-OneVision and InternVL3.Table 2 compares models and methods across multiple benchmarks.Source: §4.2 · p. 12, Table 2 · p. 13
  • Skill Evolution and MMGT each contribute positively across most benchmark-setting combinations, and their combination achieves the best overall trade-off across benchmarks.Skill Evolution refines and specializes skills; MMGT learns query-specific, vision-aware communication, with the full model especially effective under Thinking.Source: §4.2 · pp. 12–14, Figure 3 · p. 12
  • Across model scales from 2B to 38B parameters, SkillGraph yields positive gains on every benchmark, though absolute improvements generally decrease as model scale increases.Spatial and mathematical tasks retain more headroom at 32B–38B than document-heavy InfoVQA.Source: §4.2 · p. 14, Table 3 · p. 14
  • Performance improves across benchmarks with more iterations, gains most noticeably from iteration 5 to 10, and largely plateaus around iterations 15–20.Thinking outperforms Instruct on all benchmarks; MathVista has the largest gains, while InfoVQA improves less consistently in magnitude.Source: §4.2 · pp. 14–15, Figure 4 · p. 15
  • Skill-bank evolution actions decrease monotonically across benchmarks, while MathVista ends with the largest inventory and InfoVQA saturates earliest.The figure describes the frequency of evolution actions and stabilization of the skill inventory.Source: §4.2 · p. 15, Figure 5 · p. 15
  • A qualitative case study traces a skill from commonsense-based assumptions toward evidence-based hypothesis testing through failure-driven refinements.The refinements adjust both the skill’s trigger condition and strategy description across versions v5–v7.Source: §4.2 · p. 16, Table 4 · p. 16

5 Conclusion

SkillGraph jointly evolves agent expertise and communication topology. It combines query-conditioned visual routing with failure-driven skill refinement, and experiments report consistent gains over static-topology and fixed-skill baselines, especially on spatial and mathematical reasoning tasks.

5 Conclusion

  • SkillGraph jointly optimizes agent expertise and communication topology to address structural and cognitive bottlenecks in visual multi-agent systems.The framework couples topology design and skill evolution in a closed loop.Source: §5 · pp. 16–17
  • MMGT predicts a query-conditioned communication graph from fine-grained visual features and instruction semantics, replacing hand-crafted routing.Updated skill representations can reshape routing without additional parameter updates.Source: §5 · p. 17
  • A Self-Evolving Skill Bank refines multimodal reasoning heuristics through continuous diagnosis of failure experiences.Experiments report consistent outperformance of static-topology and fixed-skill baselines, with particularly strong gains on complex spatial and mathematical reasoning tasks.Source: §5 · p. 17

Related papers

Loading 2604.17503v1…