Source-linked AI summary
SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe
Yifei Shen, Bo Li, Xinjie Zhang
TL;DR
Existing skill-optimization methods leave open which pipeline components are genuinely necessary. The paper formalizes skill optimization through zeroth-order optimization and PAC-learning principles, then proposes a file-centric minimal pipeline with consensus mining and independent validation. SkillOpt-Lite improves performance and convergence across benchmarks, extends to harness optimization, and supports one-line IDE-triggered evolution.
Problem
Existing skill-optimization frameworks rely on increasingly complex pipelines without resolving which components are theoretically or empirically necessary.
Method
SkillOpt-Lite treats rollout trajectories as editable files and uses trajectory exploration, consensus attribute mining, and independent validation gating within a zeroth-order and PAC-learning framework.
Results
SkillOpt-Lite consistently matches or outperforms engineered baselines across benchmarks, while HarnessOpt reaches 0.7758 accuracy on SpreadsheetBench versus 0.7620 for GPT-5.5 under standard pipelines.
Takeaways & Limitations
The file-centric pipeline generalizes from skill editing to full harness optimization and is encapsulated for one-line agent evolution in an IDE.
Takeaways & Limitations
Extending containment mechanisms to internet-augmented or multimodally grounded execution domains remains unaddressed.
Abstract
from arXiv · showhide
While skill optimization for autonomous agents has gained traction, existing methods rely on complex pipelines. This leaves a fundamental question unaddressed: What constitutes a minimal viable pipeline for skill optimization, where every component is justified by theory or empirical necessity? We formalize skill optimization via Zeroth-Order (ZO) optimization, mapping classical counterparts (central difference, trust regions) to recent literature. Noting that unlike blind numerical perturbations in classical ZO, skill trajectories serve as interpretable debugging feedback. Grounded in Claude Code philosophy and PAC learning, we establish three principles for convergence and generalization: file-system-based trajectory exploration, consensus attribute mining, and independent validation gating. Eliminating redundancies, we propose SkillOpt-Lite. It accelerates convergence and outperforms full SkillOpt: improving LiveMath by +8.8 points on GPT-5.5 and +25.4 points on GPT-5.4-nano, allowing the nano model to surpass standard GPT-5.4 optimized by SkillOpt. Finally, we integrate our framework into production coding agents like VSCode Copilot, enabling developers to evolve agent skills via one line of vibe. Because our framework treats all agent components simply as standard editable code, this minimal pipeline naturally generalizes to full harness optimization (HarnessOpt). On SpreadsheetBench, HarnessOpt enables GPT-5.4-nano to achieve 0.7758 accuracy, outperforming the larger GPT-5.5 running standard pipelines (0.7620). Code is available at https://github.com/EvolvingLMMs-Lab/SkillOpt-Lite.
1 Introduction
The paper asks which components are theoretically or empirically necessary for skill optimization and proposes SkillOpt-Lite as a simpler alternative. Its file-centric pipeline removes complex mechanisms while improving convergence and benchmark performance, and extends to harness optimization and IDE-based use.
- Motivation: Existing skill-optimization frameworks have become architecturally complex despite leaving the minimal viable pipeline question unresolved.They incorporate mechanisms such as mini-batch tree merging, textual learning-rate schedules, and rejected-edit buffers.
- File-centric optimization: Primitive file-system exploration can outperform heavily engineered optimization pipelines, including a four-epoch full SkillOpt run.The pilot isolates trajectories into standalone files and lets an autonomous coding agent refine them with primitive file-system tools.
- Empirical results: +12.6 points on Spreadsheet, from 57.1 to 69.7, is the reported average improvement over SkillOpt across the six benchmarks.SkillOpt-Lite also lifts GPT-5.4-nano to 81.3% on ALFWorld, +9.5 over SkillOpt.
- Harness extension: HarnessOpt achieves 0.7758 accuracy on SpreadsheetBench for GPT-5.4-nano versus 0.7620 for GPT-5.5 under a standard harness and full SkillOpt pipeline.Because agent components are treated as editable code, the workflow extends from skill optimization to harness optimization.
- IDE integration: The workflow is encapsulated in a VS Code Copilot extension that lets developers trigger skill optimization through a single slash command.The extension operationalizes the paper’s one-line agent-evolution interface.
- Theoretical framing: SkillOpt-Lite maps agentic skill training to zeroth-order optimization and derives design principles for stability, convergence, and validation.The framework connects reflection paradigms to classical zeroth-order operators and uses PAC-learning theory to motivate its principles.
2 Analysis of Skill Optimization
The analysis formalizes skill optimization as zeroth-order search over discrete text artifacts while emphasizing that readable trajectories provide richer feedback than classical black-box numerical queries. PAC-learning arguments motivate consensus mining and independent validation as safeguards against overfitting and evaluation bias.
- ZO formulation: Skill optimization maximizes expected reward over text-based skills executed by a frozen language model and harness, with interaction treated as a zeroth-order oracle.The objective is non-differentiable because the skill space is discrete and the model-harness composition is non-differentiable.
- Algorithmic connections: Text edits correspond to zeroth-order operators such as stochastic gradient estimation, coordinate descent, trust regions, and control variates.Execution traces, error logs, and validation scores provide the oracle feedback for these discrete-space analogues.
- Conceptual divergence: Unlike classical zeroth-order optimization, agentic skill optimization exposes semantically rich rollout trajectories that support targeted debugging and code patches.The paper characterizes the process as language-mediated program compilation rather than blind numerical perturbation.
- Stability and overfitting: Consensus mining reduces instability by discarding single-sample eccentricities and extracting attributes shared across heterogeneous rollouts.Overfitting to episodic anomalies increases βexp, whereas stabilizing compression targets cross-task invariants.
- Validation: Independent validation removes βexp from the model-selection upper bound, provided the validation data are disjoint and sufficiently large.Using training failures or subsampled training data compromises the statistical validity of the gate.
3 SkillOpt-Lite
SkillOpt-Lite treats rollout trajectories as independent files and uses autonomous exploration, consensus mining, minimal edits, and independent validation instead of complex batch-level optimization machinery.
- Motivating Example: A pilot found that single-batch file-system exploration could outperform SkillOpt after four full epochs, although Spreadsheet performance degraded without closed-loop validation.This motivates retaining validation gating while removing unnecessary optimization structure.
- Pipeline Design: SkillOpt-Lite removes batch merging, learning-rate scheduling, slow updates, and rejected-edit buffers from the original pipeline.The streamlined design operates directly on local files through an autonomous debugging loop.
- Pipeline Design: Each rollout is staged as an independent text file, allowing constrained file-system exploration to identify clustered failures and high-leverage files.The optimizer uses native tools to list directories, inspect selected logs, and avoid loading the entire corpus into context.
- Pipeline Design: Consensus mining converts shared failure patterns into compact code patches that implement bounded, targeted skill updates.The optimizer searches for cross-task invariants before applying minimal edits.
- Pipeline Design: Independent validation accepts a candidate only when it improves the current baseline, while a new historical best overwrites the production skill file.This gate preserves the best validated skill rather than relying solely on exploratory edits.
- Developer Integration: The pipeline is exposed through a production-ready VS Code extension, enabling developers to trigger optimization with a single-line slash command.The integration embeds the lightweight debugging loop within the native IDE environment.
4 Experiments
Across six benchmarks, SkillOpt-Lite generally matches or exceeds SkillOpt, with its clearest gains on reasoning-heavy tasks and faster early convergence over ten optimization steps.
- Benchmark Results: +17.9 points on Spreadsheet-GPT-5.4: SkillOpt-Lite reaches 79.4 versus 61.5 for the full SkillOpt framework.The comparison illustrates the larger advantage on a reasoning-intensive benchmark.
- Mechanism: Removing reflection pooling avoids averaging distinct textual updates, enabling localized file edits that target logical and algorithmic deadlocks.The proposed mechanism attributes the reasoning-task advantage to preserving discrete, localized update signals.
- Benchmark Results: On SearchQA, ALFWorld, and OfficeQA, SkillOpt-Lite matches or slightly exceeds SkillOpt within a typical +0.1 to +1.5 point range.These semantics-heavy tasks show marginal performance differences between the frameworks.
- Efficiency: On shallow semantic-heavy domains, both methods approach similar local optima, while SkillOpt-Lite reaches that bound with lower computational overhead.The passage links this pattern to rapid saturation of the available optimization margin.
- Convergence Speed: SkillOpt-Lite reaches substantial gains within the first 2 to 3 steps on LiveMath-GPT-5.5 and LiveMath-GPT-5.4-nano.SkillOpt shows slower early trajectories, which the passage associates with mini-batch partitioning and update damping.
- Convergence Speed: By the final step, SkillOpt-Lite establishes an equal or higher performance ceiling without premature convergence or destabilization.The convergence analysis tracks the best validation score achieved so far across ten optimization intervals.
5 Towards Harness Optimization
HarnessOpt extends the file-based optimization loop from skills to executable harness code, using safety gates and joint skill–harness optimization to address model-specific execution bottlenecks.
- Motivation and Scope: HarnessOpt extends optimization beyond skills when declarative prompts and heuristics approach a performance ceiling, targeting imperative control flows and tool scaffolding.It applies the same three-pillar architecture to runtime code within the workspace.
- Safety Guardrails: Harness modifications are restricted to framework scaffolding, while task-specific skills and internal configurations remain read-only.This allowlist prevents optimization from drifting into task-specific artifacts.
- Safety Guardrails: Candidate harnesses must pass compilation and an isolated N = 5 smoke test before full validation.The gate reduces the risk of executing invalid or unsafe control-flow changes.
- Safety Guardrails: HarnessOpt uses reversible git rollbacks, feature toggles, and a statistical dead band to filter stochastic or non-functional improvements.Patches below the dead band require progress on continuous secondary metrics or are rolled back.
- Model-Specific Changes: HarnessOpt targets different bottlenecks by expanding spreadsheet visibility and final-answer verification for GPT5.4-mini and GPT5.4, while breaking repetitive tool-failure loops for GPT5.5 and GPT5.4-nano.These changes address parsing, formatting, observation, verification, and recovery behavior across model tiers.
- Results: GPT5.4-nano reaches 0.7758 with HarnessOpt w. skill, exceeding GPT5.5 at 0.7620 under SkillOpt.The best joint results are 0.8505 for GPT5.4 and 0.8577 for GPT5.5.
6 Conclusion and Future Work
The framework presents SkillOpt-Lite as a minimal, file-centric approach that matches or outperforms engineered baselines and extends naturally to harness optimization. Future work targets distillation, harness databases, continual harness learning, and foundation-model training.
- Conclusion: SkillOpt-Lite treats rollout trajectories as independent flat files and uses autonomous agents for targeted, semantic-driven debugging.The workflow is framed through zeroth-order optimization and statistical learning theory.
- Conclusion: The framework consistently matches or outperforms heavily engineered baselines across multiple benchmarks while accelerating optimization convergence.
- Future Work: Future research includes skill optimization for frontier-model distillation, where compute-efficient evaluation protocols for data selection remain a challenge.
- Future Work: Scaling harness optimization requires curated minimal harness templates and adaptive sandboxing for heterogeneous execution environments.Extending current containment mechanisms to internet-augmented or multimodal domains remains unaddressed.
- Future Work: Further directions include continual learning through evolving harnesses and skills, plus extending the pipeline to foundation-model training.These directions address independently evolved harness lineages and optimization of the model-training phase.