Source-linked AI summary
SkillMOO: Multi-Objective Optimization of Agent Skills for Software Engineering
Jingzhi Gong, Ruizhen Gu, Zhiwei Fei, Yazhuo Cao, Lukas Twist, Alina Geiger, Shuo Han, Dominik Sobania, Federica Sarro, Jie M. Zhang
TL;DR
Existing skill practice can improve software-engineering agents while increasing inference cost or adding misleading guidance, leaving a need for cost-aware validation. SkillMOO evolves skill bundles with LLM-proposed edits and NSGA-II Pareto selection, and achieves the top pass-rate rank on 11 of 12 non-zero-pass tasks while finding cost-efficient configurations. The paper also reports that pruning and substitution dominate successful edit patterns, while limiting conclusions to one benchmark, expanded tests, and one LLM.
Problem
Skills can improve software-engineering task success while adding substantial token overhead or misleading guidance, but existing evaluation and evolution often focus on pass rate alone.
Method
SkillMOO evolves task-specific skill bundles through LLM-proposed edits and NSGA-II Pareto selection on verifier pass rate and inference cost.
Results
SkillMOO achieves the top pass rate rank on 11 of 12 non-zero-pass tasks, with gains up to +42 percentage points and cost reductions up to 32.1% versus static deployment.
Takeaways & Limitations
Effective SE skills should be evaluated and evolved for both task success and inference cost rather than treated as static assets or optimized for pass rate alone.
Takeaways & Limitations
The evaluation uses one benchmark, LLM-expanded tests that may share reasoning biases with the solver, and one LLM, leaving transferability open.
Abstract
from arXiv · showhide
Agent skills are increasingly used to configure coding agents for software engineering (SE) tasks, yet current practice treats them as static, hand-crafted assets, or evolved on pass rate alone. This is insufficient: a skill can improve task success while substantially raising token cost, or introducing misleading guidance. We argue that SE agent skill bundles can be treated as multi-objective search objects and present SkillMOO, a framework that evolves skill bundles through LLM-proposed edits and NSGA-II Pareto selection on pass rate and inference cost. Evaluated across all 16 SkillsBench SE tasks, SkillMOO achieves the top pass rate rank on 11 of 12 non-zero-pass tasks while achieving cost reductions of up to 31.7% over static bundles, with pass rate gains up to 21 percentage points. Analysis of 38 skill edits shows that pruning and substitution dominate successful operations, offering actionable principles for skill bundle design. Thereby, the current practice of deploying skills without cost-aware validation leaves better skill configurations unexplored, motivating a new class of cost-aware, search-based skill engineering.
1 Introduction
Existing evidence shows that skills can improve software-engineering agents but also add token overhead or misleading guidance, motivating cost-aware search over skill bundles. SkillMOO frames bundles as multi-objective candidates and combines LLM edits with NSGA-II selection.
- Motivation: Skills create a tradeoff between task success, inference cost, and contextual compatibility rather than being simply helpful or unhelpful.SkillsBench reports modest SE gains, while SWE-Skills-Bench finds many public SWE skills add substantial token overhead without improving pass rate.
- Motivation: SkillMOO treats SE agent skill bundles as multi-objective optimization objects with pass rate and deployment cost as competing objectives.This challenges evaluating skills on pass rate alone.
- Approach: SkillMOO combines LLM-proposed edits with NSGA-II survivor selection in a solver-optimizer loop.The optimizer can propose pruning, substitution, reordering, and rewriting edits to skill bundles.
- Evaluation: The framework is evaluated on 16 SkillsBench SE tasks and analyzes 38 filtered skill-edit patterns to identify design principles.The paper also releases data, scripts, and skill artifacts for replication.
2 Related Work
Prior work established systematic skill evaluation, routing, and iterative skill refinement, but skill-evolution methods focused on pass rate alone. SkillMOO extends this line toward multi-objective search using LLM-proposed edits and NSGA-II.
- Prior work: Earlier studies examined skill benefits, routing over skill pools, and iterative skill refinement, but reported mixed SE performance and pass-rate-focused optimization.SkillsBench found modest SE gains, SWE-Skills-Bench found limited benefit from many public SWE skills, and SkillRouter framed usage as retrieval.
- Positioning: SkillMOO targets SE tasks with multi-objective search using LLM-proposed skill edits and NSGA-II survivor selection.This builds on prior skill-evolution approaches while adding explicit pass/cost optimization.
3 SkillMOO
SkillMOO iteratively evolves task-specific skill bundles through solver evaluation, optimizer-proposed edits, and NSGA-II selection. Its biobjective search balances verifier pass rate against LLM inference cost.
- Workflow: SkillMOO represents each candidate as a task-specific subset or edited version of the available skill pool and evaluates it through a two-agent workflow.The workflow alternates task solving with skill optimization across generations.
- Optimization: The optimizer proposes changes such as removing distracting guidance, replacing misaligned skills, adding missing instructions, reordering skills, or rewriting stale content.Each child bundle is evaluated by the same task solver.
- Selection: NSGA-II ranks evaluated candidates by non-dominated sorting and crowding distance, and top-ranked survivors seed the next generation.Optimization and selection repeat for subsequent generations.
- Objectives: The biobjective formulation minimizes negative verifier pass rate and inference cost, distinguishing SkillMOO from skill discovery or routing.The resulting question is which bundle offers the best pass/cost tradeoff.
4 Experimental Setup
The evaluation covers all 16 SkillsBench SE tasks and compares SkillMOO with original-skill and no-skill baselines using repeated full-test-suite runs. Three research questions address effectiveness, economic recovery of search overhead, and edit patterns.
- Evaluation design: Each method runs 10 times per task, with a pass requiring the full automated test suite to pass.The suites were expanded to 40–43 tests covering behavior, safety, and retained compile/build gates.
- Evaluation design: The optimization budget uses population size 4 and 3 generations, with per-candidate costs of $0.1–$14 and a practical per-task overhead of $1–$15.Runs also use a 900 s timeout and GLM-5 for both agents.
- Analysis: RQ2 computes hypervolume after per-task min-max normalization of pass rate and cost, while RQ3 filters edits from fully evaluated Pareto-archive candidates.RQ3 requires meaningful bundle changes and non-zero pass-rate, cost, and runtime deltas against ori_skill.
- Evaluation design: SkillMOO, ori_skill, and no_skill are compared across all 16 SkillsBench SE tasks through three research questions on effectiveness, economic worth, and edit patterns.RQ1 concerns pass rate and cost, RQ2 accounts for optimization overhead, and RQ3 examines edits associated with improvements.
5 Results and Analysis
SkillMOO improves pass-rate and cost tradeoffs over static skill deployment across non-zero-pass SE tasks, while exploratory evidence indicates that pruning and substitution are the most successful edit patterns.
- 5.1 RQ1: Effectiveness of SkillMOO: SkillMOO achieves the top pass-rate rank on 11 of 12 non-zero-pass tasks, with gains up to 42 percentage points and cost reductions up to 32.1%.Four tasks produce zero pass for all methods; no_skill has the lowest mean cost but trails SkillMOO on pass rate for 11 of 12 non-zero-pass tasks.
- 5.2 RQ2: Optimization Efficiency: SkillMOO improves hypervolume on all 12 non-zero-pass tasks, with optimization costs ranging from $1.76 to $13.73.Estimated cost-only break-even ranges from 5 to 682 reuses, with a median of 45; the Pareto archive retains alternatives for different deployment budgets or reliability targets.
- 5.1 RQ1: Effectiveness of SkillMOO: Four tasks with zero pass for all methods are excluded from comparative analysis because specialized or heavy runtime dependencies stress container memory and build-time budgets.These tasks are among the 16 evaluated SkillsBench SE tasks.
- 5.3 RQ3: Skill-Edit Pattern Evidence: Pruning and substitution are most often associated with successful edits, with 16 and 7 edits respectively, but these filtered traces do not establish causality.Removing peripheral or redundant skills and replacing misaligned content are associated with pass-rate improvements, whereas adding skills shows 0/9 pass improvements in the logged cases.
- 5.4 Threats to Validity: The evaluation is limited to one benchmark, skill bundles containing 1–8 initial skills, and the GLM-5 model family.Expanded tests were generated by the same model family used for solving, and transfer to other coding agents remains an open empirical question.
6 Conclusion and Research Agenda
SkillMOO motivates evaluating and evolving SE agent skills for both task success and inference cost, while identifying open directions for validation, reuse, transfer, and maintenance.
- 6 Conclusion and Research Agenda: SkillMOO motivates evaluating and evolving skills for both task success and inference cost, rather than treating them as static assets or optimizing pass rate alone.This framing motivates skill engineering as a search-based software engineering research direction.
- 6 Conclusion and Research Agenda: Future work should causally test edit operations because the current edit-pattern evidence is descriptive and drawn from filtered search traces.Suggested approaches include controlled ablations, randomized edit proposals, and repeated validation.
- 6 Conclusion and Research Agenda: Learned Pareto fronts or edit histories could route future tasks to existing skill configurations under user-specified budgets, avoiding full re-optimization at inference time.
- 6 Conclusion and Research Agenda: Future studies should test whether optimized bundles, edit heuristics, or routing policies transfer across unseen SE tasks and different coding agents.Skill interactions may differ with task type, verifier structure, and the underlying LLM.
- 6 Conclusion and Research Agenda: Lightweight SkillMOO-style revalidation could support continuous maintenance as APIs, dependencies, models, and project conventions evolve.Revalidation could detect when a previously useful skill becomes costly, misleading, or obsolete.