PaperlayerView exact revision on arXiv

Source-linked AI summary

PEACE: Towards Efficient Project-Level Efficiency Optimization via Hybrid Code Editing

Xiaoxue Ren, Jun Wan, Yun Peng, Zhongxin Liu, Ming Liang, Dajun Chen, Wei Jiang, Yong Li

arXiv:2510.17142v2cs.SE

TL;DR

Project-level efficiency optimization remains underexplored, although real-world work can involve interactions among multiple functions. PEACE combines dependency-aware function sequencing, valid historical-edit identification, and iterative optimization with internal and external code knowledge. On PEACEXEC, it outperforms the selected baselines in correctness and efficiency.

  • Problem

    Prior LLM-based efficiency optimization focuses on individual functions and overlooks function interactions, while project-level editing must address misleading historical edits and suboptimal internal implementations.

    Source: §I · pp. 1–2

  • Method

    PEACE sequences functions by dependency, filters historical edits for valid associations, and iteratively edits the sequence using a fine-tuned optimizer with internal and external high-performance implementations.

    Source: §I · p. 2, 2) Code Efficiency Optimizing · p. 5

  • Results

    On PEACEXEC, PEACE achieves 69.2% pass@1 and +46.9% opt rate versus GPT-4o instruction prompting, outperforming the selected baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    PEACE’s project-level approach outperforms state-of-the-art methods in correctness and efficiency on 146 tasks from 47 Python GitHub projects.

    Source: §IX · p. 11

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated significant capability in code generation, but their potential in code efficiency optimization remains underexplored. Previous LLM-based code efficiency optimization approaches exclusively focus on function-level optimization and overlook interaction between functions, failing to generalize to real-world development scenarios. Code editing techniques show great potential for conducting project-level optimization, yet they face challenges associated with invalid edits and suboptimal internal functions. To address these gaps, we propose Peace, a novel hybrid framework for Project-level code Efficiency optimization through Automatic Code Editing, which also ensures the overall correctness and integrity of the project. Peace integrates three key phases: dependency-aware optimizing function sequence construction, valid associated edits identification, and efficiency optimization editing iteration. To rigorously evaluate the effectiveness of Peace, we construct PeacExec, the first benchmark comprising 146 real-world optimization tasks from 47 high-impact GitHub Python projects, along with highly qualified test cases and executable environments. Extensive experiments demonstrate Peace's superiority over the state-of-the-art baselines, achieving a 69.2% correctness rate (pass@1), +46.9% opt rate, and 0.840 speedup in execution efficiency. Notably, our Peace outperforms all baselines by significant margins, particularly in complex optimization tasks with multiple functions. Moreover, extensive experiments are also conducted to validate the contributions of each component in Peace, as well as the rationale and effectiveness of our hybrid framework design.

I. INTRODUCTION

PEACE addresses the gap between function-level optimization and real-world projects, where efficiency improvements can involve interacting functions. It combines dependency-aware editing, filtered historical edits, and optimization knowledge, and is evaluated on a new project-level benchmark.

I. INTRODUCTION

  • Prior LLM optimization work focuses on individual functions, while real-world efficiency optimization often requires project-wide context and coordinated edits to multiple functions.The introduction frames code efficiency as important to system scalability, efficiency, and maintainability.Source: §I · p. 1
  • PEACE constructs a dependency-aware function sequence, identifies valid associated historical edits, and iteratively edits functions using internal and external high-performance implementations.The framework aims to improve project efficiency while preserving correctness and stability.Source: §I · p. 2
  • The evaluation measures correctness with pass@1 and efficiency with opt rate and CPU-instruction-count-based speedup, comparing opt rate against GPT-4o instruction prompting.The authors note that project-level tasks lack pre-optimization test cases, motivating their baseline for opt rate.Source: §I · p. 2
  • 69.2% pass@1 and +46.9% opt rate versus GPT-4o instruction prompting; PEACE also achieves 0.840 speedup and outperforms the selected baselines.The authors report that its speedup is closest to human-written patches and that it performs well on both single-function and multi-function tasks.Source: §I · p. 2
  • PEACEXEC contains 146 tasks from 47 Python GitHub projects, including 80 single-function and 66 multi-function tasks with executable projects, prompts, historical edits, and test cases.The benchmark was built because existing benchmarks rarely target project-level efficiency optimization; prompts and edits are drawn from project commits.Source: §I · p. 2
  • The authors open-source a replication package containing PEACEXEC, PEACE’s source code, and results.Source: §I · p. 3

II. MOTIVATION

The motivation identifies two project-level optimization challenges: misleading historical edits and limited optimization knowledge within a project. PEACE addresses them by filtering associated edits and using internal and external optimization knowledge in a three-phase workflow.

II. MOTIVATION

  • Dependency similarity can wrongly associate _is_pareto_front_nd with _compute_exclusive_hv, leading to an edit that fails tests and degrades functionality instead of selecting _is_pareto_front.The example motivates filtering historical edits for semantic validity rather than relying on dependency similarity alone.Source: §II · p. 3
  • Internal project implementations can be suboptimal; the example’s external count_zeros_ones is more efficient than the internal compute_class_py.Source: §II · p. 3

III. APPROACH

  • PEACE’s workflow constructs a function sequence, identifies valid associated edits, then iteratively optimizes those functions using associated edits and internal and external high-performance functions.The workflow targets a target function’s efficiency while maintaining overall project correctness and integrity.Source: §III · p. 3, §II · p. 3

A. Dependency-Aware Optimizing Function Sequence Construction

PEACE builds an edit sequence from a target function’s call context, combining structural dependency and semantic relevance to select and order related functions. For each function, it ranks historical edits and uses an LLM agent to identify valid associated edits.

A. Dependency-Aware Optimizing Function Sequence Construction

1) Context Extraction:

  • Context extraction parses the project into an AST and traces direct and indirect callers and callees of the target function to build a call graph.The implementation uses tree-sitter to parse code.Source: 1) Context Extraction · p. 4

2) Relevance Analysis:

  • Relevance analysis combines structural dependency scores with semantic similarity, using RobertaModel and CodeBERT embeddings to assess related functions.Structural similarity captures dependencies; semantic similarity filters syntactically related but functionally irrelevant functions.Source: 2) Relevance Analysis · p. 4

3) Relevance Filtering & Ranking:

  • Functions below the 0.5 relevance threshold are removed; the remaining functions are ranked by score, with callees before the target and callers after it.The example’s resulting sequence is (fd, ft, fa, fb).Source: 3) Relevance Filtering & Ranking · p. 4

B. Valid Associated Edits Identification

  • For each function, historical edits are relevance-ranked, then an LLM agent retrieves edits sequentially and summarizes those deemed valid.The agent is prompted with a definition of valid associated edits and can continue retrieval until it has enough edits or reaches 10 iterations.Source: B. Valid Associated Edits Identification · pp. 4–5

C. Code Efficiency Optimization Editing Iteration

The section combines a fine-tuned efficiency optimizer with an iterative editing process to improve project code while preserving contextual correctness. The optimizer is trained on function- and project-level examples and supplies efficiency-focused edits to the LLM.

1) Efficiency Optimizer Fine-tuning:

  • The optimizer’s training set combines 3,450 function-level cases with performance improvements greater than 10% and 4,043 additional project-level GitHub cases.The project-level cases were excluded from PEACEXEC because of execution-feasibility limitations.Source: 1) Efficiency Optimizer Fine-tuning · p. 5
  • LoRA adapts a small language model using original functions, semantically similar alternatives, and performance benchmarks to identify the best-performing implementation.At inference, the optimizer is prompted to generate the most efficient version of a function.Source: 1) Efficiency Optimizer Fine-tuning · p. 5

2) Code Efficiency Optimizing:

  • The LLM generates an initial edit, retrieves functionally similar internal and external snippets, and uses them with a fine-tuned optimizer to produce an efficiency edit.The LLM then integrates that edit while preserving correctness in the broader project context, and the edit joins the history used for subsequent functions.Source: 2) Code Efficiency Optimizing · p. 5
  • The editing iteration proceeds function by function through the optimizing sequence and ends after all its functions have been optimized.The fine-tuned optimizer assists the LLM in selecting the most efficient code from candidate implementations.Source: 2) Code Efficiency Optimizing · p. 5, 1) Efficiency Optimizer Fine-tuning · p. 5

D. Implement

PEACEXEC is a project-level code-efficiency benchmark built from GitHub optimization tasks, with project context, historical edits, executable environments, and tests. Its tasks cover both single-function and interdependent multi-function optimization.

IV. PEACEXEC CONSTRUCTION

  • PEACEXEC was constructed because existing project-level datasets do not specifically address performance optimization or are unavailable.The benchmark targets project-level code optimization rather than limiting tasks to function-level edits.Source: §IV · p. 6

A. PEACEXEC Overview

  • Each task includes a target function, its executable project, an optional commit-message prompt, historical edits, and evaluation test cases.When a commit does not describe the optimization intent, a generic optimization instruction is used instead.Source: A. PEACEXEC Overview · p. 6
  • PEACEXEC contains 146 optimization tasks from 47 projects: 80 single-function tasks and 66 multi-function tasks.Multi-function tasks require jointly editing interdependent functions while maintaining project consistency and functionality.Source: A. PEACEXEC Overview · p. 6

B. PEACEXEC Construction

PEACEXEC construction filters GitHub commits for execution-speed optimization, then validates the associated tests and executable project environments. Manual review and test-coverage screening yield 146 tasks across 47 projects.

1) Project Selection:

  • The benchmark starts from more than 2,000 top-starred GitHub Python projects to support evaluation across varied coding styles and application domains.The authors cite complex dependencies and unclear installation guidance as challenges in configuring executable environments.Source: 1) Project Selection · p. 6

2) Candidate Optimization Task Filtering:

  • Filtering begins with about 440,000 recent candidate commits, then uses TF-IDF keywords to reduce the set to around 16,000.The candidates come from up to the most recent 2,000 commits per selected project.Source: 2) Candidate Optimization Task Filtering · p. 6
  • A second filter excludes commits changing fewer than five lines, more than 150 lines, or five or more files, leaving 10,998 candidates.Source: 2) Candidate Optimization Task Filtering · p. 6
  • LLMs semantically rank remaining commits using commit messages and code diffs; the retained 509 commits have strong relevance to code-efficiency optimization.The dataset focuses on a single optimization goal: execution speed.Source: 2) Candidate Optimization Task Filtering · p. 6

3) Project Environment Setup & Test Case Validation:

  • Docker-based project environments and commit-derived test cases are validated to support reliable evaluation of correctness and performance.Cases lacking sufficient test coverage are filtered out before benchmark construction.Source: 3) Project Environment Setup & Test… · p. 7
  • Manual review of commits and removal of cases without sufficient test coverage yielded 146 tasks across 47 diverse projects.The benchmark collects each task’s target function, optional prompt, historical commits, executable project, and test cases; target functions are selected manually.Source: 3) Project Environment Setup & Test… · p. 7

V. EXPERIMENT SETUP

The experiment setup evaluates PEACE’s overall performance, task-specific comparisons, component contributions, and hybrid-framework design. It compares against code-editing and efficiency-optimization baselines and defines component ablations.

A. Research questions

  • The setup poses four research questions on overall performance, single- versus multi-function tasks, component contributions, and the fine-tuned optimizer’s role in the hybrid design.The optimizer question also compares its capability with popular proprietary LLMs and examines the framework design by replacing the optimizer.Source: A. Research questions · p. 7

B. Baselines

  • The baseline set includes instruction prompting, fine-tuning, SBLLM, DeepDev-PERF, RAPGen, and CoEdPilot.The selected techniques span project-level and function-level optimization methods.Source: B. Baselines · p. 7
  • The OFS ablation tests using only the target function or multiple relevant functions in random order instead of the optimized function sequence.Source: B. Baselines · p. 8
  • The VAE ablation compares giving no associated edits with providing edits extracted by a dependency analyzer.Source: B. Baselines · p. 8
  • The OKA ablation tests no function augmentation, external-function augmentation, or internal-function augmentation of initial edits.Source: B. Baselines · p. 8
  • The optimizer comparison replaces PEACE’s fine-tuned optimizer with Llama-3.1-405B, DeepSeek-V3-671B, GPT-4o, Claude3.5, or fine-tuned CodeLlama-13B.Source: B. Baselines · p. 8

C. Metrics

The section defines metrics for code-editing correctness and efficiency, including instruction-count comparisons against a GPT-4o instruction-prompting baseline and human-written patches.

C. Metrics

  • Pass@1 measures the proportion of generated solutions that pass the test cases on the first try.It is used as an indicator of code-editing correctness.Source: C. Metrics · p. 8
  • Opt rate quantifies relative changes in CPU instruction consumption against the instruction-prompting GPT-4o baseline.The paper uses instruction counts because prior research identifies them as more stable than execution time.Source: C. Metrics · p. 8
  • Speedup is the ratio of the human-written solution’s CPU instruction count to the generated solution’s count.Values approaching 1.0 from above indicate convergence toward human-level quality.Source: C. Metrics · p. 8

VI. EVALUATION

PEACE is evaluated against six baselines on project-level optimization tasks using correctness and efficiency metrics. It outperforms all selected baselines overall, while the comparative analysis also examines task types and correctness-qualified efficiency cases.

1) Correctness:

  • 69.2% pass@1 exceeds the baselines by 11.0% to 23.3%; SBLLM is second at 58.2%.The authors attribute SBLLM’s lower correctness to limited fine-grained function-dependency modeling.Source: 1) Correctness · pp. 8–9

2) Efficiency:

  • +46.9% opt rate surpasses baselines by 18.4% to 39.1%, and PEACE’s 0.840 speedup is closest to human-written patches.The reported efficiency gains draw on internal and external optimization knowledge.Source: 2) Efficiency · p. 9

A. Overall Performance of PEACE (RQ1)

  • 69.2% pass@1 and +46.9% opt rate, with 0.840 speedup, are PEACE’s reported overall results against the baselines.The authors report that these results outperform all selected baselines; the speedup is closest to human-written patches.Source: 1) Correctness · p. 8, 2) Efficiency · p. 9

B. Comparative Analysis (RQ2)

1) Correctness:

PEACE performs well on both single- and multi-function tasks, especially complex tasks, while ablations show that coordinated edit order and historical-edit knowledge matter for its results.

1) Correctness:

  • PEACE’s correctness comparison reports overlap between methods, with PEACE ahead on single-function tasks and on complex multi-function tasks.CoEdPilot overlaps substantially on multi-function tasks but does not outperform PEACE there.Source: 1) Correctness · p. 9

2) Efficiency:

  • PEACE outperforms baselines in correctness and efficiency across single- and multi-function tasks, with stronger results on complex tasks.On Figure 6, PEACE optimizes seven unique single-function tasks and five unique multi-function tasks; Figure 7 compares efficiency on 23 tasks all methods edit successfully.Source: 2) Efficiency · p. 9, 1) Correctness · p. 9

1) Ablation of OFS:

  • Random-order multi-function edits raise opt rate to +42.7% but lower correctness to 56.2%, while single-target edits reduce opt rate to +29.5%.Single-target editing slightly improves correctness to 70.5% from 69.2%, showing the tradeoff between scope and coordinated ordering.Source: 1) Ablation of OFS · p. 10

2) Ablation of VAE:

  • Removing historical edits drops correctness to 47.2% pass@1, while dependency-based retrieval reaches 59.6% correctness and +41.9% opt rate.The authors conclude that structural dependencies alone are insufficient and semantic context also matters for identifying valid associated edits.Source: 2) Ablation of VAE · p. 10

3) Ablation of OKA:

Ablations associate knowledge augmentation with efficiency gains, while optimizer comparisons and the architecture discussion describe a hybrid design balancing correctness, execution efficiency, and cost.

D. Performance of Optimizer (RQ4)

  • 69.2% pass@1, +46.9% opt rate, and 0.840 speedup are reported for PEACE’s fine-tuned optimizer, outperforming larger fine-tuned models and closed-source LLMs in execution efficiency.GPT-4o and DeepSeek-V3 have higher pass@1 at 71.2% and 70.8%, respectively, while Llama-3.1-405B reaches +32.9% opt rate and 0.665 speedup.Source: D. Performance of Optimizer (RQ4) · p. 10, Table III · p. 10
  • PEACE combines Llama-3.1-405B for project-wide reasoning and dependency-aware editing with a fine-tuned Llama-3-8B for performance optimization.The authors describe this hybrid design as balancing correctness and efficiency.Source: D. Performance of Optimizer (RQ4) · p. 10

A. Computational Costs and Hybrid Architecture Design

  • Most tasks converge within five interaction rounds, with a hard cap of ten iterations; lightweight models run within seconds on an Nvidia A800.LLM interactions account for the primary overhead, which relevance filtering in Phase I controls by reducing token use and model calls.Source: A. Computational Costs and Hybrid… · p. 11

B. Threads to Validity

PEACE is presented as a project-level optimization framework that captures interdependencies among functions and outperforms the evaluated methods. The section also notes limits in its evaluation measures, static call-graph construction, benchmark scope, and the generalizability of results across languages and model generations.

1) Internal Validity:

  • The evaluation uses pass@1 for correctness and an opt rate measured against GPT-4o because the GitHub-sourced tasks lack pre-optimization test cases.Static call-graph analysis may miss dynamic calls, including those involving inheritance or polymorphism.Source: 1) Internal Validity · p. 11

2) External Validity:

  • Because PEACEXEC draws on popular, highly starred Python projects, findings may not generalize to other languages or less-maintained codebases.Rapid changes in LLM architectures and finetuning techniques also bound the findings to the current generation of models.Source: 2) External Validity · p. 11

A. Code Optimization

  • Project-level optimization methods include fine-tuning, retrieval, iterative self-validation, and compiler feedback, but general-purpose scalable solutions remain limited.Examples include DeepDev-PERF, RAPGen [7], CodeDPO [41], and COCOGEN [42].Source: A. Code Optimization · p. 11

B. Code Editing

  • Prior code-editing methods target general edits, whereas PEACE is tailored to targeted performance optimization.Earlier approaches include AST-based edit modeling, pre-trained repair models, structured edit plans, and prompt-based LLM methods.Source: B. Code Editing · p. 11

IX. CONCLUSION

  • On 146 tasks from 47 Python GitHub projects, PEACE significantly outperforms CoEdPilot, SBLLM, and DeepDev-PERF in correctness and efficiency.The framework captures interdependencies among functions while aiming to preserve correctness.Source: §IX · p. 11
Loading 2510.17142v2…