PaperlayerView exact revision on arXiv

Source-linked AI summary

ECO: An LLM-Driven Efficient Code Optimizer for Warehouse Scale Computers

Hannah Lin, Martin Maas, Maximilian Roquemore, Arman Hasanzadeh, Fred Lewis, Yusuf Simonson, Ameya Shringi, Hongwen Dai, Patrick Musau, Tzu-Wei Yang, Amir Yazdanbakhsh, Deniz Altinbüken, Florin Papa, Maggie Nolan Edmonds, Aditya Patil, Don Schwarz, Satish Chandra, Chris Kennelly, Milad Hashemi, Parthasarathy Ranganathan

arXiv:2503.15669v2cs.SE

TL;DR

ECO targets the gap between promising LLM optimization on benchmarks and the challenges of production deployment. It combines profiling, embedding-based search, and mined anti-patterns to localize opportunities; deployed at Google, it has saved hundreds of thousands of normalized CPU cores.

  • Problem

    LLM-based optimization shows promise on benchmarks, but practical adoption in large-scale production systems presents significant challenges.

    Source: §1 · p. 1

  • Method

    ECO uses continuous profiling and embedding-based similarity search guided by mined anti-patterns to locate optimization opportunities, then applies LLMs with robust verification.

    Source: §1 · pp. 1–2, §10 · p. 14

  • Results

    Several hundred thousand normalized CPU cores saved; ECO landed over 6,400 commits changing over 25k lines, and >99.5% of the commits did not have to be reverted.

    Source: §1 · p. 2, §10 · p. 14

  • Takeaways & Limitations

    ECO’s core techniques can be replicated elsewhere with openly available tools and LLMs, including on smaller and decentralized codebases.

    Source: §9 · p. 14

    Wider adoption often faces organizational challenges, including integration with existing developer workflows and review processes.

    Source: §9 · p. 14

Abstract

from arXiv · show

This paper introduces ECO, a system that automatically modifies source code to improve performance at scale. ECO overcomes the localization problem by combining fleet-wide continuous profiling to identify performance-critical code with an embedding-based search to pinpoint specific optimization candidates, guided by a mined dictionary of performance anti-patterns. It overcomes the reliability problem through a multi-stage verification approach that uses automated testing, LLM-based self-review, and post-deployment monitoring to ensure changes are both correct and effective. Fully productionized and deployed within Google's hyperscale production fleet, ECO has successfully landed over 6,400 commits, changing more than 25,000 lines of production code. Incorrect changes are caught before they are submitted to production, and 99.5% of the submitted commits did not cause any rollbacks. These optimizations have resulted in savings equivalent to several hundred thousand normalized CPU cores, showing that ECO makes LLM-based optimization both practical at scale and highly impactful in real-world settings.

1 Introduction

Production-scale LLM optimization faces localization and reliability challenges that benchmark demonstrations do not resolve. ECO combines targeted opportunity discovery with a multi-stage verification workflow and has been deployed at Google with substantial code and CPU savings.

1 Introduction

  • Production-scale optimization requires localizing high-impact opportunities and ensuring edits are reliable, since repository-wide LLM use is computationally prohibitive and can generate false positives.Incorrect changes can cause outages, and benchmarks do not capture system integration or attribution challenges.Source: §1 · p. 1
  • ECO combines fleet-wide continuous profiling with embedding-based similarity search guided by a mined database of performance anti-patterns to identify optimization candidates.Once a candidate is localized, a fine-tuned LLM generates a transformation.Source: §1 · pp. 1–2
  • ECO checks generated edits with automated testing, LLM self-review, and post-deployment monitoring, then integrates them into a human-in-the-loop engineering workflow.The paper also studies human adoption through an evaluation of 960 edits and a production study at Google.Source: §1 · p. 2
  • Over 6,400 commits changed over 25k lines of code and saved the equivalent of several hundred thousand normalized CPU cores; >99.5% did not have to be reverted.ECO filters out and discards incorrect changes before production submission.Source: §1 · p. 2

2 Background & Overview

ECO uses LLMs to handle code variations and contextual optimizations that can challenge rule-based tools, while its system combines mined anti-patterns, similarity search, and fleet profiling. Engineers direct automated candidate discovery and generation, with human review and production monitoring in the workflow.

2.1 Why LLMs for Code Optimization?

  • LLMs can recognize and fix anti-patterns across syntactic variations and infer intent from surrounding code, enabling semantic optimizations that static checkers may miss.The paper notes that LLM inference cost is negligible compared with fleet-wide savings (Section 7.4).Source: §2.1 · p. 2

2.2 High-Level Approach

  • ECO mines historical code changes for canonical optimizations, groups similar anti-patterns into categories, and uses vector similarity search to find candidate locations.Fleet-wide continuous profiling ranks candidates by resource consumption.Source: §2.2 · pp. 2–3
  • A fine-tuned LLM generates the optimization, with self-review, automated tests, project-specific checks, human code review, and post-deployment monitoring supporting reliability.Monitoring detects regressions and measures performance impact.Source: §2.2 · p. 3
  • Engineers select and rank anti-patterns, launch candidate discovery, choose prompt recipes, and submit generated commits for review; ECO automates actions while engineers intervene when needed.The paper describes review and submission as the main points of human involvement.Source: §2.2 · p. 3

3 Performance Optimization Datasets

ECO builds an efficiency anti-pattern dataset from historical performance-improving changes and uses related data for model fine-tuning. The approach is applied across a large, diverse repository, while the paper’s evaluations use Gemini Pro 1.0.

3 Performance Optimization Datasets

  • Google’s repository spans over 100,000 packages and varied projects, and the paper states ECO’s approach would also work across smaller, disparate GitHub repositories.The work targets C++, though ECO is not fundamentally limited to it.Source: §3.1 · pp. 3–4

3.1 A Dataset of Efficiency Anti-Patterns

  • Human experts guide searches of commit messages and curated sources to find performance improvements, and a distributed computation framework builds a database of optimizations and solved anti-patterns.The repository history includes code, commit messages, and review comments.Source: §3.1 · p. 4

3.2 LLM Fine-Tuning Datasets

  • ECO’s fine-tuning dataset combines source code, selected commits, metadata, and performance-specific data derived from an anti-pattern database of approximately 55k commits.The examples and evaluation use Gemini Pro 1.0, although ECO’s architecture is described as model-agnostic.Source: §3.2 · p. 4
  • All evaluations in the paper use Gemini Pro 1.0 and are not intended to evaluate LLM capabilities at producing code-efficiency optimizations.The authors note that ECO benefits as models improve.Source: §3.2 · p. 4

4 Optimization Opportunity Localization

ECO narrows optimization search to costly functions, then uses anti-pattern-guided code retrieval and similarity ranking to identify candidates. Its localization pipeline combines fleet profiling, performance-aware attribution, and function embeddings.

4 Optimization Opportunity Localization

  • ECO combines vector similarity search with fleet-wide profiling and a mined anti-pattern database to find costly code likely to match known optimizations.The approach targets localized candidates rather than applying an LLM across the entire codebase.Source: §4 · p. 4

4.1 Performance IR

  • ECO represents functions in an IR, annotates them with AST-derived types, and filters out functions that are not performance-critical.Function-level granularity balances optimization context with tractability across millions of daily repository changes.Source: §4.1 · p. 4

4.1.1 Continuous Profiling

  • Fleet-wide GWP profiles sample program counters, stack traces, and performance counters, identifying costly functions and enabling commit-level resource-impact measurement.Profiles are aggregated globally and correlated with code versions.Source: §4.1.1 · p. 5

4.1.2 Code Annotations with Performance Metrics

  • Because cycles often accrue in shared library functions, ECO re-attributes resource use to parent functions more meaningful for optimization.The shared libraries are generally already heavily optimized and are not usually profitable targets at ECO’s level.Source: §4.1.2 · p. 5

4.1.3 Identifying Costly Functions

  • Over 10M costly functions form ECO’s search space, after call-tree pruning attributes shared and application-specific costs to actionable callers.The empirically tuned bounds are Cmin=0.1% and Cmax=25% of a binary’s cycles.Source: §4.1.3 · pp. 5–6

4.2 Code Retrieval

  • For each anti-pattern query, ECO uses ScaNN approximate nearest-neighbor search to retrieve the top 500 functions from the costly-function set.Candidates are normalized by removing custom names, static strings, and comments before syntactic ranking.Source: §4.2 · p. 6

4.2.1 Embedding Design for Semantic Similarity

  • ECO explores bag-of-words embeddings and a fine-tuned 1B-parameter dual-encoder trained on 25M samples of similar functions and code diffs with before-states.Diff training focuses retrieval on relevant optimization lines rather than entire functions.Source: §4.2.1 · p. 6

4.2.2 Ranking Performance Opportunities

  • ECO’s higher-is-better syntactic score combines BLEU, ROUGE-L, type overlap, and control-flow keyword similarity to refine approximate matches.The components capture textual, type, and control-flow similarity.Source: §4.2.2 · p. 6

5 Code Edit Generation Strategies

ECO explores zero-shot, few-shot, Chain-of-Thought, and ReAct prompting for code edits. All four approaches work for some tasks, but no recipe is universally superior, so engineers choose among them case by case.

5 Code Edit Generation Strategies

  • Zero-shot prompting directly instructs the model to perform an edit, but the paper notes it struggles with complex tasks.The approach provides no examples and relies on the model interpreting the instructions.Source: §5 · p. 7

5.1 Prompting Recipes

  • Few-shot prompts use examples of a target optimization, CoT elicits stepwise ideas before implementation, and ReAct iterates through thought, action, and observation.ReAct lets the model examine code, propose a patch, and observe the result.Source: §5.1 · p. 7
  • All four prompting approaches are effective for certain tasks, but no single recipe is universally superior, so engineers select one case by case.ECO maintains a library of strategies for engineers to choose from.Source: §5.1 · p. 7

6 Verification, Submission, & Monitoring

ECO checks generated edits with automated tests and LLM self-review before human review, then monitors deployed changes for performance regressions and impact.

Automated testing

  • Affected monorepo tests run through continuous integration; failed edits receive automated or LLM fixes, and are abandoned if they still fail.The process checks build and test outcomes.Source: §6 · p. 7

Self-review and human review

  • Because tests may miss errors and usually do not assess performance, ECO adds checklist-based LLM self-review before code-owner review.Code owners make the final decision on changes.Source: §6 · p. 7

Submission

  • Rosie automates large-scale commit management by identifying code owners and requesting reviews, but code owners must approve submissions.The paper says Google engineers perform these reviews carefully.Source: §6 · p. 8

Post-deployment monitoring

  • After deployment, ECO correlates commit uptake in binaries with performance metrics and can revert changes associated with regressions.Such reverts are rare, occurring for <0.5% of ECO’s commits.Source: §6 · p. 8

7 Evaluation

ECO’s evaluation combines controlled tests of prompting and candidate retrieval with studies of generated edits and a year of production deployment. Results cover retrieval quality, edit quality and adoption, production savings, rollbacks, and operating costs.

7 Evaluation

  • Evaluation combined production-fleet measurements with controlled experiments on examples from a subset of anti-patterns; production edits were assessed with CodeBLEU and human scoring because isolated performance tests were unavailable.The authors focus on production impact rather than comparing generation quality across models.Source: §7 · p. 8, §7.3 · p. 10

7.1 Microbenchmark Evaluation

  • Prompt choice was task-dependent: CoT generated more line changes but more invalid edits, while ReAct usually achieved higher maximum speedups; conservative CBS met or exceeded median speedup in nearly all cases.CBS achieved the maximum speedup in a third of cases.Source: §7.1 · p. 9

7.2 Candidate Retrieval

  • MAP@5 rose from 0.0728 with BOW to 0.1633 with DTE and 0.2036 with DCE; syntactic reranking helped BOW and text embeddings but added little for DCE.Deep text embeddings also outperformed BOW, while deep code embeddings improved further.Source: §7.2 · p. 10

7.3 Edit Generation on Production Code

  • Zero-shot and ReAct achieved the highest CodeBLEU and human quality scores, while CoT scored lowest; CodeBLEU could misrate non-performance edits or performance-improving edits with extensive documentation.Figure 9 reports CodeBLEU comparisons and human quality scores from 960 rated examples.Source: §7.3 · pp. 10–11, Figure 9 · p. 11

7.4 Production Impact Evaluation

  • Human reviewers directly approved 40%, 5%, and 41% of Copy, Map, and Vector changes, respectively; other changes could require comments or be rejected.Reviewers sometimes rejected changes because they judged them non-performance-enhancing or preferred not to alter the code.Source: §7.4 · p. 11
  • Map changes had comparatively low success: broader restructuring and subtle C++ semantics expose control-flow, pointer-stability, corner-case, and object-identification errors.Examples include confusing map operations, assuming pointers remain valid after modification, and mistaking a proto for a map.Source: §7.4 · p. 12
  • Over one year, ECO submitted 6.4k commits spanning 25k lines, with a rollback rate below 0.5% and savings worth several hundred thousand normalized CPU cores.Figure 10 summarizes submitted edits, performance savings, and landed production edits over time.Source: Figure 10 · p. 12, §7.4 · p. 12
  • Under conservative Gemini 3.1 Pro Preview assumptions, 10k commits cost $3,000; additional human and infrastructure load was less than 0.1% of overall resources, while savings accrue ongoing.The paper characterizes the cost as one-off and the resource savings as ongoing.Source: §7.4 · p. 12

8 Related Work

Related work spans low-level and learned optimization, code transformation, and repository-level refactoring. ECO differs by applying anti-pattern examples to optimization opportunities at warehouse scale, rather than narrowly specified edits or code-editing tasks.

8 Related Work

  • Prior optimization work spans instruction-level superoptimization, reinforcement-learning algorithm discovery, and model approaches such as PIE [2], Chen et al., and Supersonic evaluated on competition datasets.Scalene [3] interfaces with ChatGPT to optimize Python code.Source: §8 · p. 12
  • Among efforts beyond competition datasets, Aroma [36] does not target performance, while Garg et al. and EditLord [60] focus less on large-scale deployment or on localization and reliability.ECO emphasizes automation and warehouse-scale optimization, reporting thousands of real-world improvements.Source: §8 · p. 13
  • Transformation systems generalize or propagate example edits (Meng et al.; Miltner et al.), while CodePlan [64] applies specified multi-file edits; ECO uses LLMs to generalize anti-pattern examples across much larger codebases.CodePlan’s largest repository had approximately 20k lines of code.Source: §8 · p. 13

9 Discussion & Future Work

ECO outperformed deployed static analyzers across the examined vector-reserve patterns, while the discussion highlights static-analysis boundaries, retrieval and model challenges, and deployment lessons. Future extensions include bottom-up optimization and proactive developer suggestions.

Static Analysis

  • ECO optimized all 6 vector-reserve patterns, while Clang-Tidy optimized 1 and CppCheck none; CARAMEL and CLARITY target different loop optimizations.Clang-Tidy’s reserve check works only for the standard encoded pattern and trivial control flow.Source: §9 · pp. 13–14
  • Static analysis covers only designed-in optimizations; some patterns need external or domain knowledge, while custom rules for corner cases can cost more than their savings.Static analysis can more easily perform inter-procedural analysis across disparate code locations, a task that was challenging for earlier LLMs.Source: §9 · p. 14

Deployment Insights

  • ECO’s more diverse commits expanded automation beyond rigid repetitive patterns, making careful monitoring and iteration on review feedback critical.The production success rate exceeded initial expectations.Source: §9 · p. 14

Challenges

  • ECO’s capabilities expanded incrementally, while prompt-strategy selection remained empirical and depended heavily on the underlying model.Some multi-file optimizations, such as proto arenas, became feasible only in later versions.Source: §9 · p. 14
  • Retrieval requires precision-recall calibration, and increasingly complex refactorings raise false positives; early LLMs also struggled with context-dependent sorting changes.Agent-based LLM validators are being deployed during localization to catch false positives in more complex targets.Source: §9 · p. 14

Future Work

  • A complementary bottom-up approach could begin with profiling data, and a similar framework could proactively suggest edits as developers write code.The current approach is top-down and reactive, improving performance after deployment.Source: §9 · p. 14
  • The core techniques are considered replicable with openly available tools and applicable to smaller codebases, though wider adoption may depend on organizational workflow integration.The authors identify organizational rather than technical challenges as primary to broader adoption.Source: §9 · p. 14

10 Conclusion

ECO combines code similarity search and LLMs to optimize code at data-center scale, with production deployment changing more than 25,000 lines and saving hundreds of thousands of normalized CPU cores. The accompanying appendices describe anti-pattern mining, prompting trajectories, and multi-file edits.

10 Conclusion

  • Deployed at Google, ECO changed more than 25k lines of code and saved hundreds of thousands of normalized CPU cores.The system uses code similarity search over a dataset of more than 55k performance-improving repository edits, with LLMs applying changes automatically.Source: §10 · p. 14

A Anti-Patterns

B LLM Trajectory Examples

  • The trajectory examples show prompts and model responses, with target-code placeholders standing in for representative inputs.The examples illustrate the prompting mechanisms described in the paper.Source: B LLM Trajectory Examples · p. 22

B.1 Zero-Shot

  • Zero-shot prompting supplies the requested optimization and input code, and the model responds with a snippet based on its training.The appendix illustrates a request to reuse object allocations where possible.Source: §B.1 · p. 22

B.2 Few-Shot

  • Few-shot prompting adds positive and negative optimization examples before the target code, after which the model produces a diff.The examples include both overallocation and reserve-use cases.Source: §B.2 · pp. 22–23

B.3 Chain-of-Thought

  • The chain-of-thought prompt asks the model to reason step by step; the illustrated arena edit describes faster destruction and reduced allocation overhead as potential benefits.The example notes arena growth may matter when the object is frequently set and cleared.Source: §B.3 · p. 24

B.4 ReAct

  • ReAct repeatedly gives the model its execution trajectory, runs its proposed actions, and returns outputs to the prompt until the final action generates a diff.Source: §B.4 · p. 26

C Multi-File Edits

  • ECO supports multi-file edits using code embeddings, targeted prompting, and testing, self-review, and code review; descriptive prompts help capture dependencies across files.Protocol-buffer edits can involve more than five files at a time.Source: C Multi-File Edits · p. 27

D Self-Validation

ECO’s self-validation prompts check whether a vector’s final size is knowable before reserving, including whether conditional or surrounding additions make the count uncertain. Examples show the LLM rejecting a maximum-size reserve when some inputs may be absent, but approving it when each input adds exactly one element.

Self-validation criteria

  • The self-validation prompt asks whether the reserved amount is appropriate, with exact size known in advance as the key criterion.It also asks whether push_back occurs conditionally, outside a loop, or under nested loops.Source: D Self-Validation · p. 30

Example judged No

  • When some input names may be missing, reserving for all names can over-allocate because the exact output size is unknown.The response says determining an exact count first would require another map-checking pass, duplicating the main loop.Source: D Self-Validation · pp. 30–31
  • The LLM judged the maximum-size reserve reasonable when the expected number of missing names is small, while noting possible over-allocation.The example says the approach would be efficient if 99% of names were present.Source: D Self-Validation · p. 31

Example judged Yes

  • The LLM judged reserve(criterions.size()) appropriate when each input adds one output element, making the final vector size known in advance.The response distinguishes reserve from resize: resize would create elements before the loop, whereas reserve lets push_back grow the vector.Source: D Self-Validation · pp. 30, 32

E Interactions of ECO and Code Review

Human reviewers approve some ECO changes directly, but may also request unrelated edits or reject changes for performance or other reasons. Reviewer feedback can itself introduce regressions, while ECO has also produced a change committed to TensorFlow’s upstream repository.

E Interactions of ECO and Code Review

  • 40% of copy changes, 5% of map changes, and 41% of vector changes were directly approved without reviewer feedback needing resolution.The paper says every ECO change is reviewed by a human code owner before production submission.Source: E Interactions of ECO and Code Review · p. 33
  • Reviewers sometimes left comments unrelated to the optimization, which ECO’s authors addressed before submitting the change.Reviewers also rejected changes they judged non-enhancing or wanted left untouched for reasons such as style preferences.Source: E Interactions of ECO and Code Review · p. 33
  • Reviewer feedback sometimes introduced a regression: replacing a redundant map lookup with try_emplace caused repeated make_unique calls and increased CPU consumption.The change was reverted after the regression.Source: E Interactions of ECO and Code Review · p. 33

F Reproducibility

  • An ECO-generated change removing a redundant map lookup was committed to TensorFlow’s upstream repository.The paper presents it as an example of ECO-generated code on open-source code.Source: F Reproducibility · p. 35

G Comparison With Static Analysis

Across six vector-reserve patterns, ECO optimized all six, while Clang-Tidy could not resolve five and CppCheck resolved none. The examples illustrate gaps involving algorithm wrappers, custom containers, templates, and information spanning compilation boundaries.

G Comparison With Static Analysis

  • 5 of 6 vector-reserve patterns were not resolvable by Clang-Tidy, none were optimized by CppCheck, and ECO optimized all six.The reported microbenchmarks span container sizes from 8 to 4096 elements.Source: G Comparison With Static Analysis · p. 36
  • For standard vector insertions such as push_back in loops, both Clang-Tidy and ECO can optimize the pattern.The paper also describes ECO reserving from a known size before the loop.Source: G Comparison With Static Analysis · p. 36
  • Clang-Tidy misses bulk moves wrapped in algorithms and iterator adapters, while ECO recognizes element transfer and recommends reserving.The local syntax contains no explicit loop for Clang-Tidy to analyze.Source: G Comparison With Static Analysis · p. 36
  • Clang-Tidy’s standard-library-focused rules miss custom containers such as protocol-buffer RepeatedPtrField, while ECO can use a custom Reserve() method.The example concerns iterator-based assignment to a protocol buffer.Source: G Comparison With Static Analysis · p. 36
  • For generic templates, Clang-Tidy ignores uninstantiated Range parameters, whereas LLMs can infer range properties and apply block insertions.The example copies from a templated source range into a destination vector.Source: G Comparison With Static Analysis · p. 36
  • Clang-Tidy cannot trace sizing information across compilation units, but ECO can use a pre-calculated size member for conditional-loop insertions.The example reserves using stats.total_size rather than mapping the conditional loop’s source size directly.Source: G Comparison With Static Analysis · pp. 36–37
  • For custom pointer buffers using Add(), Clang-Tidy misses implicit array expansion, while ECO can recommend Reserve().The described buffer stores pointers rather than copied structures.Source: G Comparison With Static Analysis · p. 37
Loading 2503.15669v2…