PaperlayerView exact revision on arXiv

Source-linked AI summary

MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computing

Asif Rahman, Veljko Cvetkovic, Kathleen Reece, Aidan Walters, Yasir Hassan, Aneesh Tummeti, Bryan Torres, Denise Cooney, Margaret Ellis, Dimitrios S. Nikolopoulos

arXiv:2505.03906v3cs.DCcs.LGcs.SE

TL;DR

General-purpose LLMs struggle to produce optimized HPC code because parallelism, memory efficiency, and specialized domain knowledge remain difficult requirements. MARCO combines specialized agents, iterative performance feedback, and real-time web search to refine generated code. It reports a 14.6% average runtime reduction versus Claude 3.5 Sonnet and a 30.9% improvement from web search over base MARCO.

Abstract

from arXiv · show

Large language models (LLMs) have transformed software development through code generation capabilities, yet their effectiveness for high-performance computing (HPC) remains limited. HPC code requires specialized optimizations for parallelism, memory efficiency, and architecture-specific considerations that general-purpose LLMs often overlook. We present MARCO (Multi-Agent Reactive Code Optimizer), a novel framework that enhances LLM-generated code for HPC through a specialized multi-agent architecture. MARCO employs separate agents for code generation and performance evaluation, connected by a feedback loop that progressively refines optimizations. A key innovation is MARCO's web-search component that retrieves real-time optimization techniques from recent conference proceedings and research publications, bridging the knowledge gap in pre-trained LLMs. Our extensive evaluation on the LeetCode 75 problem set demonstrates that MARCO achieves a 14.6\% average runtime reduction compared to Claude 3.5 Sonnet alone, while the integration of the web-search component yields a 30.9\% performance improvement over the base MARCO system. These results highlight the potential of multi-agent systems to address the specialized requirements of high-performance code generation, offering a cost-effective alternative to domain-specific model fine-tuning.

I. INTRODUCTION

HPC demands specialized parallel, memory, and execution optimizations that general-purpose LLMs often miss. MARCO addresses this gap with specialized agents, iterative feedback, real-time web search, and a reported runtime improvement over Claude 3.5 Sonnet.

I. INTRODUCTION

  • General-purpose LLMs struggle with HPC because specialized parallelism, memory efficiency, and execution-speed requirements exceed their domain knowledge.The scarcity of HPC code examples and lower parallel-code accuracy contribute to this gap.Source: §I · p. 1

A. Research Contributions

  • MARCO separates code generation, performance evaluation, and testing so agents can specialize in optimization, measurement, and correctness.The architecture targets cache locality, OpenMP or CUDA parallelization, vectorization, execution time, memory usage, algorithmic complexity, and output correctness.Source: A. Research Contributions · p. 2
  • MARCO iteratively feeds performance measurements back into subsequent optimization attempts instead of relying on single-pass generation.This feedback loop creates progressive refinement of generated code.Source: A. Research Contributions · p. 2
  • MARCO retrieves recent optimization techniques from research publications and conference proceedings through web search, addressing the knowledge gap in pretrained LLMs.The component is presented as an alternative to relying only on static model knowledge or fine-tuning.Source: A. Research Contributions · p. 2
  • 14.6% lower average runtime was reported for MARCO than Claude 3.5 Sonnet alone on LeetCode 75, while web search improved the base MARCO system by 30.9%.The evaluation compared MARCO with leading language models and included component ablation studies.Source: A. Research Contributions · p. 2

II. BACKGROUND & RELATED WORK

LLMs are effective for general code generation but degrade on larger and more complex programs. HPC exposes additional weaknesses in parallel code, optimization, domain knowledge, and iterative error correction.

A. LLMs for Code Generation

  • LLM code-generation effectiveness is primarily limited to small-scale tasks such as individual functions and single-file programs.Performance degrades as problem complexity and scale increase, partly because context processing becomes more demanding.Source: A. LLMs for Code Generation · p. 2
  • Larger context windows help models process more documentation and maintain awareness of broader code structures during complex generation tasks.The passage contrasts GPT-3.5’s 16,385-token window with the larger capacity of GPT-4 Turbo.Source: A. LLMs for Code Generation · p. 2

B. Challenges in LLM-Generated HPC Code

  • LLM-generated HPC code often has inefficient cache utilization and suboptimal parallelization, reflecting insufficient specialized knowledge for HPC applications.General coding data provide breadth but do not supply the domain knowledge required for high-performance workloads.Source: B. Challenges in LLM-Generated HPC Code · pp. 2–3
  • Parallel-code generation performs worse than serial generation because modern parallel paradigms and HPC optimization techniques remain difficult for current LLMs to understand and debug.Threading, synchronization, advanced algorithms, and architecture-specific techniques increase the complexity of diagnosis and correction.Source: B. Challenges in LLM-Generated HPC Code · p. 3
  • HPC-Coder shows that fine-tuning on HPC-specific data can improve parallel-code generation and OpenMP pragma generation over general-purpose LLMs.This result motivates specialized approaches to HPC optimization.Source: B. Challenges in LLM-Generated HPC Code · p. 3
  • Agent-based workflows provide multiple attempts and iterative improvement, addressing the limitations of single-pass generation on complex programming tasks.The approach mirrors repeated error identification and refinement in human problem solving.Source: B. Challenges in LLM-Generated HPC Code · p. 3

III. METHODOLOGY

MARCO combines automated code generation with systematic performance testing, dynamic feedback, and live research retrieval. Its methodology evaluates multiple proprietary and open-source models across parallelization, memory optimization, and algorithmic efficiency tasks.

A. Problem Statement and Approach

  • MARCO addresses HPC code-generation limitations by coupling automated optimization with systematic performance testing through API integrations.The modular separation is intended to improve on conventional monolithic workflows.Source: A. Problem Statement and Approach · p. 3
  • The optimizer applies cache-locality, OpenMP or CUDA parallelization, and vectorization strategies, while the testing agent measures time, memory, complexity, correctness, and incremental improvement.The division assigns optimization and evaluation responsibilities to separate components.Source: A. Problem Statement and Approach · p. 3
  • MARCO feeds testing metrics back to the optimizer in an iterative loop that progressively refines implementations instead of seeking an immediate single-pass solution.The loop is designed around multi-step agentic workflows.Source: A. Problem Statement and Approach · p. 3
  • Separating optimization and evaluation reduces unnecessary API calls and token consumption at each iteration, supporting a resource-aware alternative to computationally expensive fine-tuning.The design targets controlled system costs and scalability.Source: A. Problem Statement and Approach · p. 3
  • MARCO dynamically combines live performance data with web-sourced research rather than relying on narrow task-specific fine-tuning and model retraining.The stated design goal is applicability across diverse applications.Source: A. Problem Statement and Approach · p. 3

B. Selected LLMs

  • The evaluation considered GPT-4o, Claude 3.5 Sonnet, DeepSeek Coder-V2, and Llama 3.1 8b, using proprietary and locally deployable open-source models.ChatGPT and Claude APIs were selected for MARCO because of their developer ecosystems, telemetry, and workflow integration.Source: B. Selected LLMs · p. 4

C. Benchmark Design

  • The benchmark covered parallelization, memory optimization, and algorithmic efficiency across OpenMP, CUDA, MPI, asyncio, vectorization, cache utilization, sorting, and matrix multiplication.The testing agent supplied runtime and space-complexity comparisons against baseline implementations.Source: C. Benchmark Design · p. 4

D. Evaluation Metrics

MARCO was evaluated across execution performance, code quality, and cost-efficiency. The framework measured runtime, throughput, memory use, correctness, maintainability, API cost, and token consumption.

D. Evaluation Metrics

  • The evaluation framework covered execution performance, code quality, and cost-efficiency as three dimensions of MARCO’s assessment.This structure combines technical outcomes with correctness, maintainability, and deployment practicality.Source: D. Evaluation Metrics · p. 4
  • Execution performance used runtime, FLOPS-based throughput, and memory usage measured with system timers, tracemalloc, and psutil.These measurements supported comparisons among optimization strategies and resource utilization.Source: D. Evaluation Metrics · p. 4
  • Code quality assessment validated correctness through LeetCode submissions and preserved readability and maintainability through localized modifications, comments, and explanations.The correctness procedure had library-import restrictions but followed established code-generation evaluation practices.Source: D. Evaluation Metrics · p. 4
  • Cost analysis recorded API spending and token consumption to assess the practicality of multi-agent optimization relative to fine-tuning or dedicated model retraining.Reported totals included $4.22 for Anthropic usage and $10.50 for OpenAI usage over the stated periods.Source: D. Evaluation Metrics · p. 4

IV. EXPERIMENTAL RESULTS

MARCO was evaluated under controlled, repeated experiments on LeetCode 75 and additional hard problems, with runtime comparisons against leading LLM baselines. Its runtime gains were strongest on harder problems, including a 51.9% average improvement over Claude-3.5 Sonnet.

A. Experimental Setup

  • Experiments used an AMD EPYC 7502 processor, NVIDIA A100 GPUs, 256 GB memory, Ubuntu 22.04, Python 3.10, and CUDA 12.2.Each experiment was repeated five times, and reported execution-time and memory metrics were averaged across runs.Source: A. Experimental Setup · pp. 4–5

B. Performance Analysis

  • MARCO was evaluated against Claude-3.5 Sonnet, GPT-4o, DeepSeek Coder-V2, and Llama 3.1 8b using LeetCode 75 and 10 additional randomly selected hard problems.The analysis focused on runtime performance, consistency, and relative improvements, with Claude-3.5 Sonnet providing the detailed comparison baseline.Source: B. Performance Analysis · p. 5
  • 14.6% average runtime reduction was achieved on LeetCode 75 easy and medium problems, with MARCO matching or outperforming Claude-3.5 Sonnet in 44 of 75 cases.Mean runtime was 49.1 ms for MARCO versus 57.5 ms for Claude-3.5 Sonnet.Source: B. Performance Analysis · p. 5
  • 51.9% average runtime improvement was achieved on 10 hard problems, with MARCO outperforming Claude-3.5 Sonnet in all cases.MARCO solutions ranked in the top 20% of all LeetCode submissions for 7 of the 10 hard problems.Source: B. Performance Analysis · p. 5
  • MARCO’s effectiveness increased with problem complexity, with the largest improvements observed on hard problems where sophisticated optimization strategies provide greater benefit.Table I summarizes average runtime performance across problem difficulties for the two best-performing approaches.Source: B. Performance Analysis · p. 5, Table I · p. 5

C. Memory Usage Analysis

MARCO’s primary strength was runtime optimization rather than memory reduction. Across 85 problems, memory performance was usually comparable, with modest average savings and occasional speed–memory trade-offs.

C. Memory Usage Analysis

  • MARCO improved memory usage in 15 of 85 problems, while baseline LLMs performed better in 10 and 60 cases were comparable within measurement variance.The evaluation covered 75 easy/medium problems and 10 hard problems using peak memory consumption metrics.Source: C. Memory Usage Analysis · p. 5
  • MARCO reduced average peak memory consumption by 1.2%, with maximum observed savings of 8 MB on hard problems.Table II presents memory usage comparisons across problem difficulties.Source: C. Memory Usage Analysis · p. 5, Table II · p. 5
  • MARCO’s optimizations sometimes increased memory usage by up to 1.9% to achieve runtime speedups, exposing a deliberate time–space trade-off.This trade-off characterizes the framework’s balance between execution speed and memory efficiency.Source: C. Memory Usage Analysis · p. 5

D. Summary

  • MARCO delivered 21.2% improvement on easy problems, 15.6% on medium problems, and 51.9% on hard problems while maintaining comparable or better memory efficiency across categories.The summary identifies runtime optimization as the framework’s primary strength and calls for further cross-model and domain-specific evaluation.Source: D. Summary · pp. 5–6

A. Motivation

MARCO addresses limitations of general-purpose LLMs in HPC by combining specialized agents, real-time scholarly retrieval, and iterative performance feedback. The framework is designed to reduce dependence on static model knowledge, costly fine-tuning, and manual optimization expertise.

A. Motivation

  • MARCO uses web search to retrieve current HPC optimization techniques from IEEE, ACM, arXiv, and ResearchGate beyond the base model’s training cutoff.This targets limitations in access to current architectures, optimization strategies, software libraries, and toolchains.Source: A. Motivation · p. 6, Figure 3 · p. 6
  • MARCO avoids fine-tuning by enriching pretrained LLMs with system-level context and externally sourced insights, reducing training overhead and GPU-resource requirements.The approach is presented as a more scalable and accessible alternative to fine-tuning monolithic HPC models.Source: A. Motivation · p. 6

B. MARCO Framework

  • The code optimizer agent queries the web-search engine, applies retrieved strategies, and performs multiple refinement iterations using evaluator feedback.It targets memory efficiency, execution performance, and scalability.Source: B. MARCO Framework · p. 6
  • The web-search engine supplies current scholarly techniques, while the performance evaluator measures execution time, memory usage, algorithmic complexity, and computational cost.Evaluator feedback guides subsequent optimization iterations until convergence or user-defined stopping criteria.Source: B. MARCO Framework · pp. 6–7

C. Evaluation of MARCO

MARCO with Tavily web search was evaluated against baseline models and against MARCO without web search. The results show consistent benefits from dynamic knowledge retrieval and iterative refinement, especially on complex problems.

C. Evaluation of MARCO

  • MARCO with Tavily web search consistently outperformed both Claude-3.5 Sonnet and MARCO without web-search enhancement across the evaluated problems.The evaluation compared MARCO variants against Claude-3.5 Sonnet and GPT-4o to isolate dynamic knowledge retrieval.Source: C. Evaluation of MARCO · p. 7
  • 30.9% average improvement was achieved in the final iteration over base MARCO without web search on harder LeetCode problems.Subsequent iterations generally outperformed initial attempts, supporting the value of performance-based refinement for complex optimization tasks.Source: C. Evaluation of MARCO · p. 7

A. Key Findings and Analysis

  • MARCO’s runtime gains scaled with problem complexity, reaching 51.9% on hard problems compared with 21.2% on easy problems.This pattern supports the reported value of multi-agent optimization for computationally demanding scenarios.Source: A. Key Findings and Analysis · p. 7
  • Web search produced superior optimizations across all problem categories by retrieving techniques that address knowledge gaps in static pretrained models.The reported improvement with web search was 30.9%, linking dynamic retrieval to the observed performance difference.Source: A. Key Findings and Analysis · p. 7
  • Iterative refinement is particularly effective for complex algorithmic problems because multiple strategies can be layered and adjusted using performance feedback.The process is described as resembling progressive human expert analysis and technique application.Source: A. Key Findings and Analysis · p. 7

B. System Limitations and Trade-offs

MARCO’s practical trade-offs include iterative overhead, dependence on the underlying LLM’s expertise, restricted library and language support, and higher API costs. The findings suggest complexity-aware orchestration and real-time retrieval are most valuable for demanding, specialized tasks.

B. System Limitations and Trade-offs

C. Unexpected Findings and Implications

D. Broader Implications and Future Directions

  • Dynamic web retrieval suggests hybrid pretrained-model and real-time-information approaches may suit rapidly evolving technical domains better than purely static or fine-tuned approaches.Source: D. Broader Implications and Future… · p. 8
  • Future work should expand languages and GPU support, improve complexity assessment and cost-performance optimization, and evaluate real-world HPC applications with domain-specific metrics.Source: D. Broader Implications and Future… · p. 8
  • MARCO’s approach may extend to systems programming, embedded software, and domain-specific scientific computing where current LLMs remain limited.Source: D. Broader Implications and Future… · p. 8

VII. CONCLUSIONS AND FUTURE WORK

MARCO combines specialized agents, iterative refinement, and real-time knowledge retrieval to optimize HPC code. The paper reports improvements over standalone LLMs while identifying richer communication and broader infrastructure as future priorities.

A. Summary of Contributions

  • 14.6% average runtime reduction on LeetCode 75 and 51.9% improvement on complex problems were achieved versus Claude 3.5 Sonnet.Source: A. Summary of Contributions · p. 8
  • 30.9% performance improvement over base MARCO resulted from integrating real-time web search for current optimization techniques.Source: A. Summary of Contributions · p. 8
  • MARCO separates code generation, evaluation, and testing, adds adaptive feedback, retrieves scholarly knowledge, and avoids expensive fine-tuning.Source: A. Summary of Contributions · p. 8

B. Future Research Directions

1) System Enhancements:

  • Richer bidirectional communication with diagnostics, bottlenecks, and targeted suggestions could improve refinement efficiency and effectiveness.Source: 1) System Enhancements · p. 9
  • Docker-based execution, broader library support, and C++, Fortran, and domain-specific-language support would expand cross-platform HPC applicability.Source: 1) System Enhancements · p. 9
  • Persistent code artifacts and reinforcement learning could enable finer-grained edits, reduce redundant computation, and improve optimization strategies over time.Source: 1) System Enhancements · p. 9

2) Evaluation and Validation:

Future evaluation should test MARCO across real-world applications, hardware, workloads, and competing optimization approaches. The paper also identifies broader applications and communication-history mechanisms for practical deployment.

2) Evaluation and Validation:

  • Real-world HPC evaluations should use domain-specific metrics for parallelization efficiency, hardware utilization, and scalability.Source: 2) Evaluation and Validation · p. 9
  • Studies across hardware architectures and workload types, plus comparisons with compiler optimizations and fine-tuned models, would test generalizability and positioning.Source: 2) Evaluation and Validation · p. 9

3) Broader Applications and Impact:

C. Concluding Remarks

  • Multi-agent orchestration with real-time retrieval improves on traditional single-model approaches and supports scalable, cost-efficient HPC optimization pipelines.Source: C. Concluding Remarks · p. 9
  • The greatest demonstrated benefits occur on complex problems, where sophisticated optimization provides the most value.Source: C. Concluding Remarks · p. 9
  • Further progress depends on technical development, broader real-world evaluation, and testing applicability across specialized programming domains.Source: C. Concluding Remarks · p. 9
Loading 2505.03906v3…