Source-linked AI summary
EffiSkill: Agent Skill Based Automated Code Efficiency Optimization
Zimu Wang, Yuling Shi, Mengfan Li, Zijun Liu, Jie M. Zhang, Chengcheng Wan, Xiaodong Gu
TL;DR
Existing LLM-based code-optimization methods do not explicitly distill reusable knowledge from recurring transformations, limiting reuse beyond individual instances. EffiSkill mines Operator and Meta Skills offline from slow/fast program pairs and applies them through execution-free diagnosis, retrieval, planning, and candidate generation. On EffiBench-X, it consistently improves optimization success across evaluated models and languages, while its skill library provides a reusable resource for broader agent workflows.
Problem
Existing LLM-based optimization methods remain tied to individual examples rather than distilling recurring optimization knowledge into reusable principles.
Method
EffiSkill mines Operator and Meta Skills from slow/fast program pairs, then applies them to unseen programs through execution-free diagnosis, retrieval, plan composition, and candidate generation.
Results
EffiSkill consistently improves optimization success across GPT-5-mini and Qwen3-Coder-30B-A3B-Instruct in Python and C++, achieving the best OPT@1 and OPT@8 in all settings.
Takeaways & Limitations
Explicit skill reuse provides a practical basis for execution-free code optimization and a portable toolbox for LLM-based agent workflows.
Takeaways & Limitations
Evaluation uses competitive-programming benchmarks, two backbones, and two languages, so broader validation across software domains remains needed.
Abstract
from arXiv · showhide
Code efficiency is a fundamental aspect of software quality, yet how to harness large language models (LLMs) to optimize programs remains challenging. Prior approaches have sought for one-shot rewriting, retrieved exemplars, or prompt-based search, but they do not explicitly distill reusable optimization knowledge, which limits generalization beyond individual instances. In this paper, we present EffiSkill, a framework for code-efficiency optimization that builds a portable optimization toolbox for LLM-based agents. The key idea is to model recurring slow-to-fast transformations as reusable agent skills that capture both concrete transformation mechanisms and higher-level optimization strategies. EffiSkill adopts a two-stage design: Stage I mines Operator and Meta Skills from large-scale slow/fast program pairs to build a skill library; Stage II applies this library to unseen programs through execution-free diagnosis, skill retrieval, plan composition, and candidate generation, without runtime feedback. Results on EffiBench-X show that EffiSkill achieves higher optimization success rates, improving over the strongest baseline by 3.69 to 12.52 percentage points across model and language settings. These findings suggest that mechanism-level skill reuse provides a useful foundation for execution-free code optimization, and that the resulting skill library can serve as a reusable resource for broader agent workflows.
1 Introduction
EffiSkill addresses the instance-centric limits of existing code-optimization methods by distilling recurring slow-to-fast transformations into reusable agent skills. Its two-stage framework mines and applies these skills execution-free, while improving optimization success across evaluated models and languages.
- Motivation: Existing LLM-based optimizers mainly rewrite individual examples through prompting, retrieval, or search, leaving optimization knowledge tied to specific instances.This makes structured, multi-step optimization difficult and limits reusable principles.
- Motivation: Recurring mechanisms such as aggregation, algebraic reformulation, dynamic-programming compression, and better data structures motivate transferable optimization knowledge.Treating tasks independently misses recurring transformation structure.
- Approach: EffiSkill models optimization knowledge as reusable agent skills that capture transformation mechanisms and shift optimization from instance matching toward mechanism-level reuse.Skills are distilled from recurring slow-to-fast program transformations and retrieved for new tasks.
- Approach: The framework separates offline skill mining from execution-free optimization: Stage I builds a library, while Stage II diagnoses programs, retrieves skills, composes plans, and generates candidates.The library contains Operator Skills for concrete mechanisms and Meta Skills for selecting and composing them.
- Results: 12.03 percentage points on C++ and 4.98 points on Python with GPT-5-mini, and 12.36 and 8.67 points respectively with Qwen3-Coder-30B-A3B-Instruct, EffiSkill improves optimization success over the strongest baseline.The evaluation uses EffiBench-X and compares against prompting-, retrieval-, and search-based methods.
- Implications: The mined skills form a portable, plug-and-play optimization toolbox that can be reused in broader LLM-agent workflows.The explicit intermediate abstraction also makes the optimization process more analyzable than purely black-box rewriting.
2 Background
EffiSkill formulates efficiency optimization as generating a functionally equivalent but faster program without execution feedback during inference. It represents recurring optimization knowledge as reusable Operator and Meta Skills rather than instance-specific examples or prompts.
- Problem Definition: The objective is to generate a program that preserves the input program’s behavior while reducing its benchmark-specific efficiency cost.The formulation requires both semantic correctness on the test suite and lower cost than the input program.
- Execution-Free Setting: EffiSkill performs optimization without executing candidates or using runtime traces during inference.It instead infers bottlenecks and opportunities from the input program, task context, and mined skill library.
- Execution-Free Setting: This execution-free setting reflects deployments lacking suitable environments, representative workloads, dependencies, or budget for repeated online candidate evaluation.Examples include code-review assistants, repository-scale optimization tools, and enterprise development environments.
- Skill-Based Optimization: An agent skill is a reusable structured unit of optimization knowledge encoding either a concrete transformation mechanism or higher-level control knowledge.This definition distinguishes skills from raw prompts, retrieved exemplars, and fully optimized programs.
- Skill-Based Optimization: EffiSkill abstracts recurring mechanisms into skills with applicability conditions, transformation procedures, expected effects, and possible failure modes.Operator Skills encode transformations, while Meta Skills govern diagnosis, retrieval, composition, and execution-free candidate generation.
3 Proposed Approach: EffiSkill
EffiSkill separates offline mining of reusable optimization knowledge from execution-free application to unseen programs. Its pipeline diagnoses bottlenecks, retrieves multiple skill routes, composes plans, and generates candidates while preserving intended behavior.
- Framework Overview: EffiSkill treats optimization as composing reusable transformation mechanisms learned from prior slow-to-fast examples rather than one-shot rewriting.The two-stage paradigm decouples learning optimization knowledge from applying it to new programs.
- Skill Library: Operator Skills encode transformation mechanisms, while Meta Skills control how those mechanisms are selected, combined, and executed.Examples include algorithm replacement, state compression, and constant-factor reduction.
- Stage II: Skill-Guided Optimization: The workflow generates candidates without iterative execution feedback while preserving the original program interface and intended behavior.Meta Skills guide plan construction, and candidate generation follows those plans directly.
- Stage I: Skill Mining: Stage I extracts structured traces, abstracts compact signatures, clusters similar mechanisms, distills Operator Skills, and constructs Meta Skills.The pipeline uses paired slow and fast solutions as its mining source.
- Stage II: Skill-Guided Optimization: Stage II diagnoses each unseen task, retrieves three candidate Operator-Skill sets, composes two to three plans per set, and generates optimized candidates.Multiple routes preserve diversity because inefficient programs may admit several valid optimization directions.
4 Experimental Setup
The evaluation tests EffiSkill on paired-solution corpora and EffiBench-X across Python and C++ using controlled candidate generation, ranking, and private-test evaluation. It compares prompting, retrieval, search, and fine-tuning baselines under a fixed eight-candidate budget.
- Research Questions: The study asks whether EffiSkill improves efficiency, which components contribute, whether performance transfers across languages, and what skills are learned.These questions cover overall effectiveness, ablations, cross-language consistency, and learned-skill analysis.
- Datasets: The skill-mining corpus uses paired slow and fast competitive-programming solutions from PIE and additional Python datasets, with runtime or instruction-count measurements.C++ uses gem5-simulated runtime, while Python efficiency is measured using CPU instruction counts.
- Benchmark: EffiBench-X contains 623 optimization tasks across six languages, with Python as the primary setting and C++ used to assess cross-language transfer.The evaluation focuses on the Python and C++ subsets and uses expert-written canonical solutions.
- Evaluation Protocol: Candidates are selected using 20% public tests and evaluated finally on the remaining 80% private tests.Only candidates passing all public tests are retained and ranked by public-test runtime before private evaluation.
- Controlled Comparisons: Each method generates k = 8 candidates, and all compared methods use the same LLM backbones to isolate optimization-strategy effects.The backbones are GPT-5-mini and Qwen3-Coder-30B-A3B-Instruct.
- Baselines: Baselines span direct prompting, chain-of-thought, retrieval augmentation, evolutionary search, and fine-tuning.SBLLM represents evolutionary search, while EffiCoder is reported only for the open-weight Qwen3-Coder backbone.
- Metrics: OPT@k is the percentage of tasks where at least one top-k candidate passes all tests and reduces runtime by at least 10% on private tests.The metric combines functional correctness with a minimum efficiency improvement relative to the input program.
5 Experimental Results
EffiSkill consistently achieves the strongest optimization success across model–language settings, candidate budgets, and programming languages. Its gains reflect relevant skill retrieval, multi-plan composition, and reusable mechanism-level optimization knowledge.
- RQ1: Overall Effectiveness: EffiSkill achieves the best OPT@1 and OPT@8 across all four model–language settings, converting a small execution-free candidate budget into successful optimizations.The reported advantage concerns candidates that are both functionally correct and measurably faster than the input program.
- RQ1: Overall Effectiveness: EffiSkill benefits more consistently from additional candidates, wins more tasks than it loses against the strongest baseline, and shifts more tasks into moderate and strong improvement ranges.These analyses indicate broader candidate-set quality and more reliable optimization discovery under a fixed execution-free budget.
- RQ1: Overall Effectiveness: Mechanism-level skill reuse, explicit diagnosis, retrieval, and multi-plan composition outperform direct prompting, retrieval-based, and optimization-oriented baselines in execution-free optimization.The paper presents explicit, reusable skills as inspectable optimization ideas, including constant-factor cleanup, closed-form arithmetic, and simplified combinatorial computation.
- RQ2: Ablation Study: Ablations show that relevant retrieval and multi-plan composition materially contribute to performance, while random skills and single-plan generation substantially reduce OPT@8.On GPT-5-mini, replacing retrieved skills with random ones reduces OPT@8 from 37.40% to 27.82%; on Qwen3-Coder-30B-A3B-Instruct, it falls from 36.60% to 13.80%, while single-plan generation reduces it to 14.29%.
- RQ2: Ablation Study: Removing retrieval degrades OPT@1 and OPT@8, indicating that diagnosis alone is insufficient and that retrieval helps promote strong candidates among top-ranked outputs.The robustness effect is especially pronounced for OPT@1 on GPT-5-mini.
- RQ3: Cross-Language Generalization: EffiSkill improves optimization success over the strongest baselines across Python and C++, with particularly large gains on C++ under both backbones.The framework reaches 26.48% / 37.40% on Python with GPT-5-mini and 21.35% / 36.60% with Qwen3-Coder-30B-A3B-Instruct; reported C++ gains are larger.
- Skill Library Analysis: The learned library contains 29 operator skills grouped into five semantic families, with implementation and constant-factor optimization comprising 54.2% of candidate-skill pairs across 623 problems.Algebraic reformulation and dynamic-programming compression also appear frequently in the library.
6 Discussion
EffiSkill reuses explicit transformation mechanisms and composes them through structured diagnosis, retrieval, and planning, enabling algorithmic rather than merely local optimization. Its execution-free design improves efficiency without runtime feedback, while evaluation validity remains bounded by benchmark and construction choices.
- Why Does Skill-Based Optimization Work?: EffiSkill shifts reuse from surface-level program patterns to explicit transformation mechanisms, supporting generalization across syntactically different programs with similar bottlenecks.The library contains 29 Operator Skills, while the top five account for 68.0–69.7% of usage.
- Why Does Skill-Based Optimization Work?: The diagnosis–retrieval–planning pipeline composes multiple skills to address coordinated changes in state representation, transitions, and constant-factor optimization.This composition constrains candidate search while preserving multiple optimization plans.
- Case Study: The prompting baseline preserves pairwise reasoning and remains O(Q^2), whereas EffiSkill produces a non-trivial algorithmic rewrite.The baseline mainly refines implementation without changing the underlying computational structure.
- Case Study: EffiSkill reformulates repeated pairwise checks into aggregated mask operations, reducing the case-study complexity from O(Q^2) to O(Q+N).The improvement comes from changing the algorithmic formulation rather than cleaning up local implementation details.
- Case Study: For the shown examples, EffiSkill reduces runtime from 3838.11 ms to 1434.23 ms, a 2.68× efficiency improvement.The example illustrates a qualitative difference in optimization behavior rather than the largest possible absolute gain.
- Threats to Validity: The evaluation uses competitive-programming benchmarks whose controlled slow/fast pairs do not fully represent real-world software involving libraries, compilers, hardware, or cross-file context.Reported results should therefore be interpreted as evidence in a controlled code-efficiency setting.
- Threats to Validity: Effectiveness is measured by execution-free optimization success, which does not capture readability, maintainability, broader resource trade-offs, or transfer across more models, languages, and domains.The skill corpus and benchmark also share the competitive-programming domain, despite no direct task overlap.
7 Related Work
Prior code-efficiency methods use prompting, retrieval, or search, but their optimization knowledge remains implicit, instance-specific, or runtime-dependent. EffiSkill instead distills language-agnostic optimization strategies into reusable skills for execution-free inference.
- Background: Code-efficiency optimization transforms functionally correct but slow code into faster equivalents, yet LLM-generated programs remain behind human-expert solutions.The paper presents this as a distinct software-engineering challenge among broader LLM coding tasks.
- Existing Approaches: Prompting-based approaches ask models to rewrite code without supplying additional optimization knowledge, while related methods differ in how they represent and deliver reusable transformations.The cited discussion contrasts prompting with classical rule-based and search-based techniques.
- EffiSkill: Across existing paradigms, optimization knowledge is either implicit in model weights or narrow retrieval instances, or dependent on runtime feedback.EffiSkill addresses this by distilling language-agnostic strategies into a reusable skill library validated offline.
8 Conclusion
EffiSkill separates offline skill mining from execution-free, skill-guided optimization so agents can reuse mechanism-level knowledge rather than one-shot rewrites or instance matching. Across models and languages, it improves optimization success, with gains attributed to the interaction of diagnosis, retrieval, and plan diversity.
- Framework: EffiSkill is a two-stage framework that models reusable optimization knowledge as agent skills and separates offline mining from execution-free optimization.This design replaces one-shot rewriting and instance-level exemplar matching with mechanism-level reuse.
- Results: EffiSkill achieves the best OPT@1 and OPT@8 across GPT-5-mini and Qwen3-Coder-30B-A3B-Instruct in both Python and C++ settings.The results indicate greater likelihood of producing at least one correct, efficiency-improving candidate within a fixed execution-free budget.
- Implications: The reported gains arise from the interaction of structured diagnosis, relevant skill retrieval, and plan diversity rather than any single component.The skill library contains both broadly applicable constant-factor improvements and specialized transformations, and extends beyond Python to C++.
- Implications: The mined skills can serve as a reusable optimization toolbox for coding agents and may support integration of optimization knowledge into development workflows.Future work includes broader languages and software-engineering scenarios and improving the library’s quality, scope, and novelty.