PaperlayerView exact revision on arXiv

Source-linked AI summary

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

Zehua Zhang, Ati Priya Bajaj, Divij Handa, Siyu Liu, Arvind S Raj, Hongkai Chen, Hulin Wang, Yibo Liu, Zion Leonahenahe Basque, Souradip Nath, Vishal Juneja, Nikhil Chapre, Tiffany Bao, Yan Shoshitaishvili, Adam Doupé, Chitta Baral, Ruoyu Wang

arXiv:2509.25248v2cs.SEcs.AIcs.PL

TL;DR

OSS compilation is challenging because projects vary in dependencies, documentation, and environment requirements, and prior benchmarks focused on popular repositories. The paper introduces a broader benchmark and an agent combining instruction retrieval with iterative compilation; OSS-Build-Agent achieves a 66.4% success rate with a strong base LLM. The authors also analyze design choices and bound their evaluation to C/C++ repositories on Linux.

  • Problem

    OSS compilation requires resolving missing dependencies and undocumented environment requirements, while prior evaluation on popular repositories may underrepresent the challenges of less-profiled projects.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper constructs Build-Bench and evaluates OSS-Build-Agent, which retrieves compilation instructions from documentation and iteratively revises build commands using execution feedback.

    Source: §1 · p. 2, §3.1 · p. 5, §6.2 · p. 9

  • Results

    66.4% success rate: OSS-Build-Agent with a strong base LLM surpassed the strongest rule-based baseline by roughly 50% on Build-Bench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Build-Bench and the analysis of agentic module designs provide a benchmark and findings intended to inform future compilation methods.

    Source: §9 · p. 11

    The evaluation covers only C/C++ repositories on Linux; other languages and operating systems remain future work, and the authors note a relatively small compilable test set.

    Source: §8 · p. 11

Abstract

from arXiv · show

Automatically compiling open-source software (OSS) projects is a vital, labor-intensive, and complex task, which makes it a good challenge for LLM Agents. Existing methods rely on manually curated rules and workflows, which cannot adapt to OSS that requires customized configuration or environment setup. Recent attempts using Large Language Models (LLMs) used selective evaluation on a subset of highly rated OSS, a practice that underestimates the realistic challenges of OSS compilation. In practice, compilation instructions are often absent, dependencies are undocumented, and successful builds may even require patching source files or modifying build scripts. We propose a more challenging and realistic benchmark, BUILD-BENCH, comprising OSS that are more diverse in quality, scale, and characteristics. Furthermore, we propose a strong baseline LLM-based agent, OSS-BUILD-AGENT, an effective system with enhanced build instruction retrieval module that achieves state-of-the-art performance on BUILD-BENCH and is adaptable to heterogeneous OSS characteristics. We also provide detailed analysis regarding different compilation method design choices and their influence to the whole task, offering insights to guide future advances. We believe performance on BUILD-BENCH can faithfully reflect an agent's ability to tackle compilation as a complex software engineering tasks, and, as such, our benchmark will spur innovation with a significant impact on downstream applications in the fields of software development and software security.

1 Introduction

OSS compilation is difficult because projects vary in dependencies, toolchains, and platform requirements, while existing rule-based workflows handle such mismatches poorly. The paper introduces Build-Bench and OSS-Build-Agent, evaluates five methods, and inspects design choices in retrieval and error resolution.

1 Introduction

  • OSS compilation requires manual intervention for missing dependencies, version mismatches, and undocumented environment requirements, which rule-based methods handle unreliably.The paper connects reliable, scalable compilation to software integration, binary-data use, program analysis, vulnerability discovery, and maintenance workflows.Source: §1 · pp. 1–2
  • Build-Bench contains 148 humanly verified repositories selected from 385 randomly sampled C/C++-heavy GitHub projects, with labels for compilation success and instruction retrieval.A separate set of 70 carefully chosen projects supported development of OSS-Build-Agent.Source: §1 · p. 2
  • OSS-Build-Agent reached a 66.4% success rate with a strong base LLM and surpassed the strongest rule-based baseline by roughly 50%.The authors evaluated five rule-based and AI build methods on Build-Bench, including two agentic methods; the benchmark remains challenging for future research.Source: §1 · p. 2
  • The paper inspects compilation-instruction retrieval and error-resolution designs and their effects on task performance.It compares OSS-Build-Agent’s LLM-assisted retrieval and multi-agent compilation modules with another agentic solution.Source: §1 · p. 2

2 Constructing Build-Bench

Build-Bench is designed to represent heterogeneous, long-tail C/C++ repositories rather than only popular projects. Its construction combines random sampling, human verification, and labeling, and its repositories vary in popularity, build systems, and instruction sources.

Data Filtering and Selection

  • From 6.57M filtered C/C++ repositories, the authors randomly selected 385 projects and manually verified them, yielding 148 compilable repositories for the test set.The filtering excluded forks and projects judged extremely low quality; verification excluded projects that could not be compiled in a best-effort setting.Source: §2 · p. 3

Popularity

  • Build-Bench mostly contains repositories with 50–500 stars, unlike CompileAgentBench projects aggregated between 2k and 10k stars.The paper argues that popular repositories may be unusually well documented and maintained, while low-profile repositories can lack documentation or need customization.Source: §2 · pp. 2, 4, Figure 2 · p. 4

Build Systems

  • Build-Bench includes 62 Make, 60 CMake, 29 Autotools, and 14 Visual Studio projects, alongside smaller subsets using alternative systems.Ten repositories provide no explicit build system, and some projects offer multiple build systems as alternative compilation approaches.Source: §2 · p. 4

Build Instruction Sources

  • 136 of 148 Build-Bench test repositories have build instructions, and 39 of those host instructions outside the repository.Source: §2 · p. 4

Representativeness

  • The authors describe Build-Bench as representing a wide variety of real-world C/C++ projects and as suitable for evaluating automated build techniques.Source: §2 · p. 4

3 Agentic Building Methods

OSS-Build-Agent combines optional instruction retrieval with an iterative, multi-agent compilation loop. The retrieval module explores documentation when README instructions are insufficient, while the agents revise and rerun commands using execution feedback.

3 Agentic Building Methods

  • OSS-Build-Agent starts with the README, optionally extends it with retrieved compilation instructions, then iteratively generates and executes build steps.The retrieval module precedes the multi-agent build system and is optional.Source: §3 · p. 4, Figure 3 · p. 5

3.1 Compilation Instruction Retrieval

  • When README information is insufficient, LLM-Assisted Retrieval follows promising internal or external links, summarizes up to three new sources per iteration, and stops at confidence or three iterations.The process distills instructions and checks their sufficiency before seeking more documentation; final instructions pass to the compilation framework.Source: §3.1 · pp. 4–5

3.2 Multi-Agent Compilation System

  • The Bash Command Generator proposes commands and the Execution Agent runs them in a container; the generator revises commands using execution results until success or the turn limit.The iterative loop is intended to recover from missing dependencies, incorrect flags, and environmental mismatches.Source: §3.2 · pp. 5–6

4 Baseline Methods

The paper compares rule-based repository builders, a single-turn LLM baseline, and a multi-agent compilation system. These baselines differ in how they discover build files, use execution feedback, and delegate instruction retrieval and error resolution.

GHCC

  • GHCC discovers build-system files such as Makefiles and CMakeLists.txt, then applies a rule-based routine customized to those systems.Prior research has used datasets GHCC created.Source: §4 · p. 6

Assemblage

  • Assemblage automates construction of primarily Windows executable datasets and follows a rule-based compilation workflow similar to GHCC.Source: §4 · p. 6

Single-turn LLM baseline

  • The single-turn LLM baseline generates Bash build commands from the README and root directory listing, then executes them without feedback-driven revision.Source: §4 · p. 6

CompileAgent

  • CompileAgent uses a master agent with CompileNavigator and ErrorSolver modules, supported by five specialized tools and four auxiliary LLM agents.The evaluation includes its official open-source implementation as a representative agentic baseline.Source: §4 · p. 6

5 Experiment Setup and Evaluation Methods

The evaluation tests build methods in fresh Ubuntu 22.04 containers and uses expert-generated binary names to validate success. This distinguishes validated builds from completion inferred merely from the presence of a binary.

5 Experiment Setup and Evaluation Methods

  • Build methods are evaluated on freshly cloned repositories in fresh Ubuntu 22.04 Docker containers, with minimal preinstalled packages and no injected or mutated source-code failures.The methods use different base models; OSS-Build-Agent is tested with five models, while CompileAgent uses GPT-4o.Source: §5 · p. 6
  • A binary’s presence alone is unreliable evidence of success because failed builds can leave intermediates or successfully build only a submodule.The evaluation compares produced binary filenames against an expert-generated per-repository list, with strict success requiring all listed names and flexible success requiring at least one.Source: §5 · p. 6
  • Table 1 reports completion and validated-success performance for all evaluated build techniques on the Build-Bench test set.The supplied caption identifies the table’s scope but does not provide its cell values.Source: Table 1 · p. 7

6 Evaluating Build Methods

OSS-Build-Agent achieves strong validated build results, while repeated runs expose variability and retrieval quality affects performance. The analysis also identifies dependency, troubleshooting, and command-generation failures that can impede agentic builds.

6 Evaluating Build Methods

  • CompileAgent falls from 89% strict validated success on CompileAgentBench to 49.7% strict and 55.7% flexible on Build-Bench, indicating a distribution shift and greater benchmark difficulty.GHCC achieves 30.2% completions and 13.4% flexible validated successes; Claude 3.7-Sonnet’s single-turn baseline reaches 37.2% strict and 39.9% flexible.Source: §6 · p. 7
  • 67.6% strict and 73.0% flexible validated success: OSS-Build-Agent with LLM-assisted retrieval and Claude 3.7-Sonnet surpasses the single-turn baseline using the same model.The paper reports this as the best configuration and says agentic methods substantially outperform rule-based baselines.Source: §6 · p. 7
  • Agentic performance is model-agnostic but scales with model capability; Claude 3.7-Sonnet performs best, while Qwen3 Coder reaches 38.9% flexible success.The authors attribute stronger performance to adjusting outputs based on errors and applying targeted fixes; they suggest coding specialization may be a drawback when retrieval depends on documentation comprehension.Source: §6 · p. 7

6.1 Instability and Repeated Experiments

  • Across three GPT-4o runs, OSS-Build-Agent achieves 53.0% ± 6.8 strict and 57.6% ± 6.5 flexible validated success; pass@3 rises to 65.5% strict and 70.3% flexible.The reported pass rates rise from 54.7% strict and 59.5% flexible at pass@1, and from 59.5% strict and 64.2% flexible at pass@2.Source: §6.1 · p. 8, Figure 4 · p. 8

6.2 Retrieval and Error Resolution

  • OSS-Build-Agent retrieval accuracy is 73.8% versus CompileAgent’s 46.2%; LLM-assisted retrieval also exceeds RAG by 4.3% in strict validated success.Its README-led documentation traversal avoids distraction by build scripts and better handles instructions scattered across sources; all retrieval variants improve over No Retrieval.Source: §6.2 · pp. 8–9
  • Validated success scales with error-resolution attempts; OSS-Build-Agent averages 6.6 attempts versus CompileAgent’s 7.5, whose one-command-at-a-time approach counts trivial steps.OSS-Build-Agent emits a fuller command set and retains command history for troubleshooting; the system achieves effective retrieval and resolution with two agents rather than CompileAgent’s seven.Source: §6.2 · pp. 9–10

6.3 Failure Modes of Agentic Build Methods

  • Common agentic failures include missing packages or version mismatches, premature troubleshooting stops, under-exploration of build alternatives, and incorrectly ordered or fabricated command flags.Complex repositories may require incremental repairs, and link order and library grouping can make command-order mistakes fail during compilation or linking.Source: §6.3 · p. 10

7 Related Work

Prior work applies LLM agents to several software-engineering tasks, but automatic C/C++ OSS compilation remains underexplored. Earlier compilation evaluations may favor familiar repositories, motivating a more challenging benchmark.

7 Related Work

8 Limitations

The evaluation is limited to compiling C/C++ repositories on Linux, and the test set contains relatively few compilable repositories. The authors also note that agentic-framework instability may cause reproduction results to vary.

8 Limitations

  • Evaluation covers C/C++ repositories on Linux; results for other languages and operating systems remain outside the tested scope.The authors leave those settings as plausible future work.Source: §8 · p. 11
  • The test set has relatively few compilable repositories, though intensive manual verification supplies ground-truth binary-file and retrieval labels.The authors describe manual verification as compensation for the limited quantity.Source: §8 · p. 11
  • Agentic-framework instability may introduce performance variation when experiments are reproduced.The authors also suggest retrieval could be further improved.Source: §8 · p. 11

9 Conclusion

The paper presents Build-Bench as a challenging benchmark for C and C++ repositories and reports that analysis of agent module designs highlights the difficulty of compilation. It also discusses potential benefits and security risks of compilation agents, alongside a filtering-keyword example.

9 Conclusion

  • Build-Bench supports rigorous evaluation of C and C++ repository compilation, while module-design analysis highlights the task’s challenges.The authors hope the benchmark will inspire better OSS-compilation agents.Source: §9 · p. 11

Broader Impact Statement

  • More reliable automated compilation could reduce environment-setup time and software bit rot, and support workflows using compiled artifacts.The cited downstream examples include large-scale program analysis and security research.Source: Broader Impact Statement · p. 11
  • Running untrusted build scripts or fetching external dependencies without strong isolation can risk code execution, data exposure, or supply-chain compromise.The paper recommends ephemeral sandboxed environments, minimal privileges, and review of agent-proposed changes before reuse.Source: Broader Impact Statement · p. 11

A Filtering Keywords

  • The filtering-keyword examples include homework, assignment, tutorial, exercise, and demo.The appendix introduces these as part of the keywords used to filter low-quality OSS.Source: A Filtering Keywords · p. 14

B Sample Size Estimation

The appendix estimates a repository sample size using proportion-estimation parameters and a finite-population correction. It also provides prompts for retrieving build instructions and for LLM-based compilation and execution agents.

B Sample Size Estimation

  • With 95% confidence, a 5% margin of error, and p = 0.5, the finite-population-corrected sample size is rounded up to 385 repositories.The chosen p maximizes variance to yield a conservative sample size.Source: B Sample Size Estimation · pp. 14–15

C.1 LLM-Assisted Retrieval Prompt

  • The retrieval prompt extracts Linux source-build instructions and dependency requirements, then seeks relevant external links and internal documentation paths.If no build instructions are found, it asks the model to report that explicitly while still seeking useful links.Source: §C.1 · p. 15

C.2 LLM baseline prompts

  • The LLM baseline prompt requests sequential Ubuntu commands to install dependencies, detect and configure the build system in Debug mode, and install the main binary.It supplies a Docker environment, README, and top-level file list as repository context.Source: §C.2 · p. 16

C.3 System Prompt for Bash Command Generator

  • The Bash Command Generator prompt requires executable commands that use repository-specific paths, retain debug information, follow repository instructions, and attempt to fix errors.It also directs the agent to run make install and avoid make check or make test.Source: §C.3 · p. 17

C.4 System Prompt for Executor Agent

  • The Executor Agent is prompted as an assistant that can run bash commands or function calls during repository compilation.Source: §C.4 · p. 17

D Retrieval Benchmark

The retrieval benchmark compares instruction-source retrieval strategies, while the cost analysis gives reference per-repository costs and notes limits on timing comparisons. A case study shows an agent patching outdated OpenCV calls to compile an old repository.

D Retrieval Benchmark

  • The LLM-assisted retrieval module captured around 80% of intermediate instruction URLs or internal files, along with potentially useful build instructions.The study measured retrieval trajectory coverage against ground-truth labels.Source: D Retrieval Benchmark · p. 18
  • Hit@5 was 63.2% for TF-IDF and 77.2% for RAG on in-repository documentation, while LLM-assisted retrieval reached 27.8% versus 0% for RAG on 18 repositories with external-URL ground truths.The LLM-assisted retriever also used fewer HTTP requests than RAG to build its knowledge base, and the results suggest it improves coverage on long-tail external instruction pages.Source: D Retrieval Benchmark · p. 18

E Time and Monetary Cost

  • Using GPT-4o, compilation consumed roughly 87K input and 1.6K output tokens per repository, costing $0.23; retrieval added an average $0.11, for about $0.34 total.The cost comparison is scoped to per-repository time and API costs across compilation agents.Source: E Time and Monetary Cost · p. 19, Table 6 · p. 19
  • Timing comparisons are only reference estimates because OSS-Build-Agent ran on Kubernetes with 10 CPU cores per repository, while baseline experiments ran on a local server with default settings.Assemblage did not report a reference time-cost analysis in its manuscript.Source: E Time and Monetary Cost · p. 19
  • For CompileAgent, the study reports costs from its manuscript after observing minor differences between that manuscript and the public implementation.GHCC's minimal time consumption reflects its one-shot strategy: it terminates after an unsuccessful initial attempt.Source: E Time and Monetary Cost · p. 19

F Case Study 1: Agentic Compilation patching source files

  • An agent patched a ten-year-old repository's outdated OpenCV v3 calls to current API names, then successfully compiled it on a recent Ubuntu environment.For example, it changed CV_LOAD_IMAGE_COLOR to IMREAD_COLOR after receiving compilation-error feedback.Source: F Case Study 1: Agentic Compilation… · p. 20

G Case Study 2: CMake Error due to Directory Changes

The case studies show that compilation failures can arise from incorrect build-directory choices or insufficient troubleshooting of an earlier error. The appendix tables list BuildBench composition, but the supplied captions do not provide their values.

G Case Study 2: CMake Error due to Directory Changes

  • The agent failed to compile optimal-transport after running CMake from the wrong directory; its CMakeLists.txt was in the src subdirectory, and further troubleshooting would have revealed the fix.The passage characterizes changing directories as a simple and straightforward solution.Source: G Case Study 2: CMake Error due to… · p. 21

H Case Study 3: Insufficient Troubleshooting

  • In the package case, a dubious-ownership error during submodule initialization preceded a make install error, so resolving the root cause required addressing submodule initialization first.Focusing only on the follow-up make error would not resolve the problem.Source: H Case Study 3: Insufficient… · p. 21

I Information about Repositories

Loading 2509.25248v2…