PaperlayerView exact revision on arXiv

Source-linked AI summary

Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve

Yuanzhe Liu, Ryan Deng, Tim Kaler, Xuhao Chen, Charles E. Leiserson, Yao Ma, Jie Chen

arXiv:2505.23946v2cs.AIcs.LGcs.MAcs.SE

TL;DR

Code LLMs can have complementary strengths across optimization categories, raising the question of how agents can collaborate when those strengths are not known in advance. LessonL lets agents learn from one another’s successes and failures through lessons and reports that small teams can outperform a much larger LLM and other collaboration methods. The paper also identifies added time to first token and function-level-only evaluation as limitations.

  • Problem

    LLMs vary in their code-optimization strengths across categories, so it is unclear how to leverage multiple agents when their complementary strengths are unknown beforehand.

    Source: §1 · p. 1

  • Method

    LessonL enables multiple agents to collaboratively solve coding problems by soliciting, banking, and selecting lessons learned from one another’s solutions.

    Source: §1 · p. 2

  • Results

    A team of small LLMs using lessons can significantly outperform a much larger LLM under similar resource consumption, and LessonL reports state-of-the-art performance on code benchmarks.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Lesson-based collaboration can improve agents’ solutions by allowing them to learn from other agents’ successes and failures.

    Source: §5 · p. 10

    Lesson solicitation and subsequent trials defer time to first token, and LessonL has so far been applied only to function-level code snippets.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent studies show that LLMs possess different skills and specialize in different tasks. In fact, we observe that their varied performance occur in several levels of granularity. For example, in the code optimization task, code LLMs excel at different optimization categories and no one dominates others. This observation prompts the question of how one leverages multiple LLM agents to solve a coding problem without knowing their complementary strengths a priori. We argue that a team of agents can learn from each other's successes and failures so as to improve their own performance. Thus, a lesson is the knowledge produced by an agent and passed on to other agents in the collective solution process. We propose a lesson-based collaboration framework, design the lesson solicitation--banking--selection mechanism, and demonstrate that a team of small LLMs with lessons learned can outperform a much larger LLM and other multi-LLM collaboration methods.

1 Introduction

Code LLMs have complementary strengths across optimization categories, motivating collaboration that does not assume those strengths are known beforehand. LessonL has agents exchange lessons from successes and failures through an iterative solicitation, banking, and selection process.

Motivation

  • Across fine-grained optimization categories, different small models outperform GPT-4o, including Qwen7B on geometry by 2.5× and Deepseek7B and Qwen14B on histogram by 1.6×.Other category strengths also vary across Deepseek7B, Qwen7B, and Qwen14B.Source: §1 · pp. 1–2

Lesson-based collaboration

  • LessonL lets agents learn from one another’s successful and failed solutions by exchanging lessons that can improve their own problem-solving.Lessons may be optimization strategies, pitfalls, or profiler feedback, and can be summarized and reused iteratively.Source: §1 · p. 2

LessonL

  • LessonL uses solicitation, banking, and selection to let multiple LLM agents collaboratively solve coding problems without requiring preassigned agent roles.Its lessons are more concise than code and are interpretable and reusable.Source: §1 · pp. 2–3

Contributions

  • The authors report state-of-the-art performance on code optimization and code generation benchmarks.They also report that small LLM teams can significantly outperform a much larger LLM under similar resource consumption.Source: §1 · p. 3
  • The paper provides representative code examples and lessons in Appendices K–M.Code optimization is the framework’s focal application, with code generation as another demonstrated task.Source: §1 · pp. 2–3

2 Related Work

The related-work section surveys multi-agent collaboration approaches and prior LLM work on code optimization. It positions LessonL as combining multiple agents with a lesson mechanism, without relying on an external code dataset.

Multi-agent collaboration

  • Prior multi-agent frameworks use role-based agents or aggregate independent solution proposals; LessonL belongs to the latter category and adds lessons for collaborative learning.The section cites role-based approaches including AgentVerse [12], MetaGPT [23], and MapCoder [27], and proposal-based approaches including MoA and LLM-Blender [28].Source: §2 · p. 3

Code optimization

  • Prior code-optimization work includes specialized fine-tuned models and adaptation methods, while agentic approaches use retrieved examples or self-generated feedback.LessonL instead uses multiple agents and does not rely on an external code dataset.Source: §2 · p. 3

3 The LessonL Framework

LessonL iterates between generating solutions and lessons, selecting lessons from a shared bank, and updating solutions. Its components adapt lesson solicitation and selection to the task and account for how lessons perform when applied.

3 The LessonL Framework

  • LessonL repeatedly updates agents’ solutions using banked lessons, then solicits new lessons from the updated solutions and returns the best solution.Each agent initially generates a solution and lesson; subsequent rounds select lessons, update solutions, deposit new lessons, and adjust selected lessons’ effectiveness.Source: §3 · p. 4

3.1 Lesson Solicitation

  • Lessons explain why a solution is correct or wrong, with compiler, timing, test, and profiler feedback distinguishing speedup, slowdown, incorrectness, and syntax-error scenarios.For speed changes, solicitation includes measured speedup; for syntax errors, it includes the compiler message, while incorrectness prompts omit test cases to encourage general lessons.Source: §3.1 · p. 4

3.2 Lesson Banking and Selection

  • To limit prompt capacity and token cost, LessonL selects at most k lessons, prioritizing high-speedup lessons and then semantic relevance to the original code.Selection considers negative lessons for avoiding mistakes as well as relevance to guard against irrelevant lessons.Source: §3.2 · pp. 4–5

3.3 Effectiveness Adjustment

  • LessonL dynamically ranks a lesson by its original speedup multiplied by an adjustment factor reflecting its performance when later applied.The factor increases when an application’s speedup falls below the lesson’s original speedup; a value above 1 means more output codes achieve speedup greater than that original value.Source: §3.3 · p. 5

3.4 Extension to Other Coding Tasks

  • For Python code generation, LessonL solicits lessons for functional incorrectness and selects them by test cases passed and semantic relevance rather than speedup.If generated code passes all test cases, the iteration terminates immediately.Source: §3.4 · p. 5

4 Experiments

Across code optimization and generation, LessonL performs competitively against single-agent and multi-agent alternatives. Ablations, cost analysis, and case studies examine its lesson-selection strategy, scaling, and scope.

4.1 Setup

  • The evaluation covers six coding benchmarks, comparing LessonL with single-agent models, reasoning and reflection methods, and multi-agent frameworks.It tests code optimization on ParEval and PolyBench and code generation on HumanEval, HumanEval+, MBPP, and MBPP+.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 Benchmark Results

  • LessonL achieves best-or-second-best results across code-optimization metrics and benchmarks, and ranks first on three of four code-generation benchmarks.In code generation, MoA comes next, with the two frameworks swapping ranking positions on the remaining benchmark.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7

4.3 Ablation Study

  • Ablations show serial mode benefits most from high-speedup lesson prioritization, while OpenMP favors high-relevance lessons and speedup adjustments.Without lessons, the framework underperforms across metrics; in OpenMP, LessonL has lower speedup variability than random selection (0.03 vs 0.28 standard deviation).Source: §4.3 · p. 8, Table 3 · p. 8
  • LessonL and Reflexion benefit from more collaboration rounds, unlike MoA, whose performance declines; MapCoder shows no clear trend beyond two rounds.Even with a smaller model, Reflexion surpasses GPT-4o and continues improving with more rounds.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4 Cost Analysis

  • LessonL is Pareto optimal on ParEval, combining superior speedup with reasonable costs, while MapCoder and MoA appear least cost-effective.The analysis estimates monetary costs and uses FLOPS as a proxy for latency; its cost estimates may vary over time and include educated guesses.Source: Figure 4 · p. 9, §4.4 · p. 9

4.3 Ablation Study

  • Adding agents generally improves LessonL across metrics, with the largest gain from three agents and diminishing returns as more are added..Source: Table 4 · p. 8, §4.3 · p. 9

4.5 Case Study

  • In case studies, lesson-guided optimization reaches 74.31x speedup for a geometry task and 10.83x for DFT by precomputing repeated exponentials.The geometry example changes from a straightforward O(n^2) algorithm to divide-and-conquer at O(n log n).Source: §4.5 · p. 10
  • Even collaborating small LLMs sometimes fail on problems that are too challenging for them; external knowledge is one proposed mitigation.The reported example includes syntax errors, semantic errors, and lack of speedup.Source: §4.5 · p. 10

5 Conclusions

LessonL lets multiple agents learn from one another’s successes and failures, and evaluations report strong performance across coding tasks. Its current scope and latency trade-offs constrain deployment, while its lessons are presented as interpretable educational information.

5 Conclusions

  • LessonL lets agents learn from one another’s successes and failures, and evaluations report state-of-the-art results and small-model teams outperforming a much larger LLM under a similar resource budget.Case studies include divide-and-conquer and precomputation optimizations.Source: §5 · p. 10
  • Lesson solicitation and subsequent trials defer first-token time, and LessonL has so far been applied to function-level snippets rather than repository-level tasks.The authors say lesson information may offset the user-experience impact; broader coding scenarios remain future work.Source: §5 · p. 10
  • Lessons are interpretable and educational, but practitioners should verify them before applying their knowledge elsewhere because LLM outputs can be flawed.Source: §5 · p. 10

A Complementary Strengths of LLM Agents

The section motivates multi-agent coding from models’ varied category-level strengths and outlines LessonL’s lesson mechanism. It also describes related collaboration work, prompt designs, and the adaptation of ParEval for code optimization.

A Complementary Strengths of LLM Agents

B More Related Work

  • Prior agent approaches include layered aggregation, debate, response ranking and fusion, dynamic team adjustment, and graph-based collaboration.The section also describes single-LLM tool use, planning, iterative refinement, and search approaches.Source: B More Related Work · p. 18

C Algorithm of LessonL

  • LessonL’s algorithm banks lessons, selects them using speedup-weighted factors and code relevance, then adjusts lesson factors based on subsequent performance.The framework takes code, agent count, rounds, and selected-lesson count as inputs; selection divides lessons between high speedup and high relevance.Source: C Algorithm of LessonL · pp. 18–19

D Prompt Templates for Code Optimization

E Prompt Templates for Code Generation

F ParEval Modification for Code Optimization

  • The adapted ParEval optimization benchmark asks models to optimize source code and evaluates both correctness and speedup over that source.The adaptation uses the baseline as the code to optimize and may add definitions, such as custom types, so source code runs independently.Source: F ParEval Modification for Code… · p. 22, §F.1 · p. 22

F.2 Benchmarking

The benchmark adaptation improves runtime measurement and test-input coverage, and fixes baseline-code and overflow bugs. These changes address noisy timings and cases where insufficient input variation could let incorrect code pass.

F.2 Benchmarking

  • Nanobench varies run counts according to code runtime and variance to produce stable measurements, including for short-running code.The authors use it because fixed-run averages can be noisy when runtimes are very short.Source: §F.2 · p. 23

F.3 Input Generation

  • The adaptation seeds matching inputs for baseline and generated code and varies graph and search inputs to avoid narrow test coverage that can let incorrect code pass.For graph problems it varies edge counts; for search, the original narrow value range could make targets almost certain to appear in large lists.Source: §F.3 · p. 23

F.4 Other Fixes

  • The authors fix overflow issues from large generated values and incorrect baseline implementations in several problems, including matrix LU decomposition and convex hull calculations.Source: §F.4 · p. 23

G Additional Information of Evaluation Setup

The evaluation uses configured iterative baselines for optimization and generation, with shared hardware and hyperparameters described for the experiments. Supplementary examples show that additional iterations do not necessarily improve code-optimization performance.

G.1 Baseline Implementation for Code Optimization

  • Optimization baselines adapt Reflexion and MapCoder to select code by speedup, while MoA uses three layers with GPT-4o as aggregator.Reflexion runs ten iterations; MapCoder uses three plans with five iterations each and selects the best-speedup code.Source: §G.1 · p. 24

G.2 Baseline Implementation for Code Generation

  • For code generation, baseline iterations use six synthetic test cases generated by Qwen14B, and LessonL stops when synthetic tests indicate correct code while reporting pass@1 on given tests.The authors prefer synthetic tests over the small supplied sets because they consider overfitting more harmful than potentially incorrect generated tests.Source: §G.2 · p. 24

G.3 Hyperparameters, Hardware, and Timing

  • Code-optimization models run on an EC2 instance with four L40S GPUs, while generation uses a server with eight A6000 GPUs; a full LessonL ParEval run takes a few hours.All methods use one c5.4xlarge exclusively for performance evaluation; both task types share hyperparameters except generation does not use SELECTHIGHSPEEDUP.Source: §G.3 · pp. 24–25, Table 6 · p. 24

H Supplementary Information for Figure 3

  • Across the illustrated iterative methods, the authors find that more rounds do not necessarily improve code-optimization performance.The supplementary section discusses speedup variation across rounds for MoA, MapCoder, and Reflexion.Source: H Supplementary Information for Figure 3 · p. 25

H.1 LessonL

This section presents examples of code-agent outcomes, including optimization speedups across rounds and a `cellsXOR` implementation. It covers how reported results vary across LessonL, MoA, MapCoder, and Reflexion.

H.1 LessonL

  • 9.50x was LessonL’s round 4 speedup after its convolution optimization sequence.Round 0 removed redundant zero-additions; round 2 separated interior and boundary processing; round 3 removed temporary storage and extra data copies, and round 4 kept the round 3 code.Source: §H.1 · pp. 25, 28

H.2 MoA

  • 12.36x was MoA’s round 0 speedup, its highest; later rounds slowed after a type cast and a switch to std::nth_element, with rounds 3–5 unchanged.The task was finding an array’s kth smallest element.Source: §H.2 · pp. 29–30
  • MoA’s reported rounds 2–5 achieved speedups of 6.71x, 6.72x, 6.74x, and 6.72x, respectively.Source: §H.2 · pp. 31–32

H.3 MapCoder

  • MapCoder never beat the original code: speedups were 1.00x, 0.96x, 0.65x, 0.74x, and 1.00x across rounds 0–4.The outputs used loop unrolling, which the passage says should be left to the compiler.Source: §H.3 · pp. 32–35

H.4 Reflexion

  • Reflexion reached 10.71x speedup in round 4, after a 10.81x speedup in round 3.Round 0 failed to compile, round 1 matched the original code at 1.00x, and round 2 was 0.95x; rounds 3 and 4 used identical code, with the round-3 improvement attributed to eliminating half the if-tests.Source: §H.4 · pp. 35, 36, 38
  • `cellsXOR` counts input[idx - N] when i > 0, input[idx - 1] when i < N - 1, and input[idx + 1] when j < N - 1; it sets output[idx] to 1 only when count equals 1.The function takes input and output vectors and N; the excerpt shows the index calculation and these conditional neighbor checks.Source: §H.4 · p. 38

I Supplementary Information for Figure 4

This section estimates monetary cost, FLOPS per token, and token use for ParEval experiments, documenting the assumptions behind those estimates. It also compares methods’ LLM calls and token consumption per problem.

I Supplementary Information for Figure 4

J.1 Complementary Strengths of Agents

LessonL improves speedup over the best single agent on many ParEval categories, while longer interaction improves speedup until returns nearly plateau around round 8.

J.1 Complementary Strengths of Agents

  • In the “sort” category, Qwen7B fails to offer any speedup without LessonL, but with LessonL achieves 2.35× speedup on average; Deepseek7B and Qwen14B perform quite well.Source: §J.1 · p. 40
  • LessonL improves speedup on many categories, with the strongest gains on “fft” and “stencil”; degradations are minor and reflect LLM-generation randomness.The comparison is against the best single agent across categories.Source: §J.1 · p. 40, Table 10 · p. 41
  • For complex-number sorting, lessons identify avoiding square roots with std::norm and precomputing magnitudes; Qwen7B follows the first lesson to gain speedup.Without lessons, Qwen7B fails to optimize the problem and may produce uncompilable code or slower hypot calls.Source: §J.1 · pp. 40–41

J.2 Running LessonL for More Rounds

  • More interaction rounds improve speedup as computational resources increase, but gains nearly plateau around round 8.Table 11 reports LessonL speedups over rounds on ParEval serial mode.Source: §J.2 · p. 41, Table 11 · p. 41

K Case Study: Geometry

The Geometry case replaces an O(n^2) closest-pair search with divide-and-conquer, achieving 74.31x speedup. Its lessons identify both the optimization and the implementation errors agents encountered.

K Case Study: Geometry

  • 74.31x speedup accompanies a change from O(n^2) pairwise search to an O(n log n) divide-and-conquer closest-pair algorithm.The algorithm sorts by x-coordinate, solves each half recursively, then checks a y-sorted strip for closer cross-half pairs.Source: K Case Study: Geometry · pp. 41–43
  • Lessons sometimes credit x-sorting alone, but the authors identify it as a precursor to divide-and-conquer and its reduced O(n log n) cost.Sorting appears in lessons 1, 2, and 4; divide-and-conquer appears in lessons 1, 4, and 5.Source: K Case Study: Geometry · p. 42
  • The nine lessons also expose implementation pitfalls, including misplaced directives, missing comparisons, type mismatches, and undefined or prematurely used functions.The authors note that the complex divide-and-conquer implementation can cause errors even for human programmers.Source: K Case Study: Geometry · pp. 42, 44, 45
  • The authors conclude that the lessons help guide an LLM toward code that is both correct and high-performing.The final code's strip check filters points by distance from the dividing line and compares only candidates within the current minimum distance.Source: K Case Study: Geometry · p. 42

L Case Study: DFT

The DFT case shows a 10.83x speedup from precomputing and reusing complex exponential factors. Attempts to use FFT were unsuccessful, while compiler constraints also ruled out Intel intrinsics in the evaluation pipeline.

L Case Study: DFT

  • Some agents attempted FFT to reduce DFT's O(N^2) cost to O(N log N), but sign errors caused incorrect results and the lessons discouraged further attempts.The lessons characterized the failures as nonequivalence or logical errors, although the authors trace them to a sign error.Source: L Case Study: DFT · p. 46
  • Intel-intrinsic attempts failed because the evaluation pipeline did not support the required CPU support and compiler flags.The authors say this lesson discourages using Intel intrinsics in benchmarking.Source: L Case Study: DFT · p. 46
  • 10.83x speedup accompanies precomputing complex exponential factors and reusing them in the inner loop instead of recalculating them.The authors attribute the optimization to periodicity that makes many exponential factors repeat.Source: L Case Study: DFT · pp. 45–47

M Case Study: Sum of Prefix Sum (A Failure Example)

The prefix-sum example illustrates failures across compilation, problem understanding, and performance: initial agents produced no compilable code, and some later outputs computed the wrong quantity.

M Case Study: Sum of Prefix Sum (A Failure Example)

Related papers

Loading 2505.23946v2…