PaperlayerView exact revision on arXiv

Source-linked AI summary

XREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language Models

Dung Le Quang, Dong Cao Van, Nam Le Hai, Linh Ngo Van, Anh M. T. Bui, Phuong T. Nguyen

arXiv:2608.25939v1cs.SE

TL;DR

Existing unit-test-generation evaluations often omit repository constraints and cover few programming languages, limiting practical evidence about LLM performance. XREPOTEST introduces a multilingual repository-level benchmark and containerized, context-aware evaluation framework, then evaluates 14 state-of-the-art LLMs and adds Invocation Rate to assess direct functionality exercise. The study reports substantial repository-level challenges, language- and context-dependent effects, and a trade-off between improved test effectiveness and invocation reliability.

  • Problem

    Existing evaluations emphasize standalone settings and a narrow language set, limiting evidence about realistic multilingual repository-level unit-test generation.

  • Method

    XREPOTEST combines repository-level benchmark tasks, containerized execution, multiple context-augmentation strategies, and execution-based metrics including Invocation Rate.

  • Results

    Experiments with 14 state-of-the-art LLMs reveal substantial challenges from language-specific constraints, domain complexity, and semantic issues in generated tests.

  • Takeaways & Limitations

    XREPOTEST provides a challenging benchmark for analyzing scalable and robust multilingual unit-test generation in realistic repository contexts.

  • Takeaways & Limitations

    Results may not generalize directly beyond the five languages and curated repositories, and retrieval findings depend on retriever design choices.

Abstract

from arXiv · show

Large language models (LLMs) have shown promise for automated unit test generation, but existing evaluations largely rely on standalone settings and a narrow set of programming languages, overestimating real-world readiness. We introduce XREPOTEST, a multilingual repository-level benchmark for unit test generation spanning five underexplored languages: Rust, Go, Julia, PHP, and Ruby. XREPOTEST evaluates tests under realistic repository constraints using a containerized execution framework and multiple context augmentation strategies, including file-level, LSP-based, and retrieval-based context. Beyond standard metrics such as test pass rate and coverage, we propose Invocation Rate (IR) to assess whether generated tests meaningfully exercise the intended functionality. Experiments with 14 state-of-the-art LLMs, including Claude 4.5, GPT-5.2, DeepSeek V4-Pro, and Qwen families, reveal a substantial gap between standalone and repository-level performance, as well as trade-offs between richer context and test reliability. Overall, XREPOTEST provides a challenging and informative benchmark to advance scalable and robust unit test generation in realistic software environments. The dataset and code are publicly available at: https://github.com/solis-team/XRepoTest

1 Introduction

XREPOTEST addresses limitations in unit-test-generation evaluation by testing LLMs under repository-level constraints across five underexplored programming languages. It combines executable evaluation, context augmentation, and Invocation Rate to expose gaps between test execution and meaningful functionality coverage.

  • Existing benchmarks often isolate functions from repository build systems, dependencies, and framework conventions, overstating practical deployment readiness.
  • Repository-level evaluations remain concentrated in Java and Python, limiting evidence about LLM test generation across diverse programming ecosystems.
  • The experiments show that richer context can improve test effectiveness while introducing noise that reduces invocation reliability.
  • XREPOTEST covers over 3,642 focal functions from real-world Rust, Go, Julia, PHP, and Ruby repositories.
  • Its containerized framework evaluates generated tests with test pass rate, compilation success rate, line coverage, mutation score, and Invocation Rate.
  • The benchmark supports file-level, LSP-based, and retrieval-based context augmentation to study how context design shapes generated-test quality.

2 Related Work

Prior unit-test benchmarks provide limited evidence about multilingual repository-level generation because they emphasize standalone settings and high-resource languages. XREPOTEST responds with executable evaluation across diverse programming ecosystems.

  • Traditional and LLM-based benchmarks commonly center on Java and Python, limiting applicability to other programming ecosystems.
  • UniTSyn is multilingual but evaluates tests using standalone benchmarks that abstract away repository-level constraints.
  • Broader coding datasets evaluate test generation within larger tasks, but do not necessarily provide standalone repository-level test-generation evaluation.
  • Repository-level multilingual benchmarks remain scarce because reliable execution environments require language-specific build, testing, and dependency knowledge.
  • XREPOTEST addresses this gap with a multilingual repository-level benchmark and executable data pipeline across diverse programming languages.

3 Methodology

XREPOTEST constructs repository-level tasks from real projects, supplies configurable context, and executes generated tests in reproducible language-native environments. Its design spans five languages, diverse domains, and extensible evaluation infrastructure.

  • Dataset Construction: Each task targets a focal function, inserts generated tests into the original project, and validates them within the repository environment.
  • Dataset Construction: The workflow selects repositories, extracts non-trivial focal functions, pairs them with static or retrieval-based context, and executes tests in containers.
  • Language Selection: The benchmark focuses on PHP, Go, Ruby, Rust, and Julia, excluding heavily studied Java and Python settings.
  • Repository Selection: Repositories must meet quality criteria including at least 500 GitHub stars, build configuration files, and coverage of at least two application domains per language.
  • Focal Method Extraction: Tree-sitter parses source files into ASTs, from which top-level functions and methods are extracted and filtered for quality and validity.
  • Context Augmentation: Context augmentation combines file-level content, LSP-resolved symbols and signatures, and optional sparse BM25 or dense UniXCoder retrieval.
  • Containerized Execution: Docker containers standardize toolchains and dependencies while running each repository's native testing framework.
  • The framework is designed to extend to additional languages when supported testing frameworks and configuration files are available.

4 Evaluation Metrics

XREPOTEST evaluates generated tests using execution and coverage metrics while adding Invocation Rate to distinguish genuine focal-function testing from accidental indirect execution.

  • Standard Metrics: Test Pass Rate measures the fraction of compiled tests that run without exceptions or assertion failures, while line coverage measures exercised focal-method lines.
  • Metric Set: The evaluation reports test pass rate, compilation success rate, line coverage, mutation score, and Invocation Rate across generated tests.
  • Metric Motivation: Standard metrics can count passing tests and covered lines even when a wrapper indirectly invokes the focal method without directly validating it.
  • Invocation Rate: Invocation Rate verifies whether generated tests directly call the focal function, providing an indicator of behavioral relevance.

5 Findings and Discussion

Across five languages, XREPOTEST shows that unit test generation remains difficult for frontier LLMs, with correctness, setup, and repository context limiting success. Context augmentation and agentic repair can improve execution outcomes, but richer context and pass-rate gains may reduce invocation reliability.

  • Overall Trends: 57.4% CSR and 3.2% mean MS show that generated tests often compile without strongly detecting behavioral faults.The mean mutation score covers Go, Rust, and Ruby, with Ruby values reported as lower-bound estimates.
  • Context Augmentation: File-level context is the most consistently effective augmentation, while retrieval gains vary by language and model and LSP benefits are selective.File-level context achieves the best results for Go, Julia, and Rust across the reported settings.
  • Context Augmentation: Ruby file-level context raises TPR and coverage but reduces IR from 85.63% to 53.19%, exposing a test-effectiveness versus invocation-reliability trade-off.The IR decline is reported across all three evaluated models, while Go, PHP, and Julia show negligible IR changes in comparable settings.
  • Overall Trends: At most ∼27% TPR under the standard setting confirms that repository-level test generation remains challenging across languages.Frontier models dominate many metric highs, while Qwen variants and GPT-OSS remain competitive open-weight baselines.
  • Failure Modes: API Hallucination dominates most language-specific failures, while richer Rust context reduces hallucinations but increases Logic & Assertion failures.In Julia, hallucination slightly increases under context from 47.90% to 50.10%, whereas Ruby’s rate is largely unchanged.
  • Agentic and Repair Workflows: Agentic execution improves TPR by 1.2–2.6× across all five languages, but PHP shows a 33.3% coverage drop and 71.3% IR drop despite an 11.5% TPR increase.A one-round GPT-OSS-120B repair loop improves TPR by 9–26% with IR nearly unchanged.
  • Failure Modes: 9.7% of TPR-passing suites fail IR, including 28.3% in Rust and 19.2% in Ruby, because they invoke mocked, stubbed, or unrelated APIs.This disagreement demonstrates that pass rate alone misses whether tests exercise the assigned focal method.

6 Conclusion

XREPOTEST is a multilingual repository-level benchmark and execution-based framework for assessing how LLMs generate, invoke, and validate tests under realistic conditions. Across 14 LLMs, it reveals language- and context-dependent challenges, while showing that IR complements pass rate and coverage for assessing behavioral validity.

  • Contributions: XREPOTEST combines realistic repository settings, controlled context augmentation, and execution-based metrics across multiple programming languages and domains.The benchmark uses containerized environments and supports systematic analysis of generated test behavior.
  • Findings: Evaluation of 14 state-of-the-art LLMs reveals challenges from language-specific constraints, domain complexity, and semantic issues in generated tests.These challenges are observed across the benchmark’s multilingual evaluation.
  • Metrics: Invocation Rate complements test pass rate and coverage by assessing whether generated tests meaningfully exercise the intended functionality.The metric supports behavioral-validity analysis beyond whether tests pass or execute covered lines.
  • Context Augmentation: File-level context provides the most reliable gains, whereas retrieval- and LSP-based augmentation produce selective, language-dependent benefits.The conclusion also identifies adaptive context selection, language-aware augmentation, and stronger oracle reasoning as directions for future work.

7 Limitations

XREPOTEST’s evaluation design is constrained by fixed prompting and decoding choices, a limited language and repository scope, and several implementation dependencies. Its context construction and execution pipeline also relies on language-specific tooling and retrieval configurations that may not be universally optimal.

  • Scope and evaluation choices: Fixed prompting and decoding configurations limit conclusions about alternative strategies or model-specific performance.Alternative prompting, decoding, or tool-augmented settings may produce different absolute results.
  • Scope and evaluation choices: The benchmark covers five languages and curated repositories, so results may not generalize to other languages, domains, or testing frameworks.The released containerized pipeline is intended to support future expansion.
  • Context construction: Retrieval-based results depend on retriever design choices such as chunking, indexing granularity, and relevance scoring.The reported retrieval pipeline is not necessarily optimally tuned.
  • Context construction: LSP augmentation requires repository-specific type and symbol analysis, including argument definitions, callee signatures, and control-flow references.These retrieved elements are attached as context for the focal method.
  • Execution pipeline: Evaluation depends on Dockerized language toolchains, native test runners, coverage instrumentation, and language-specific placement of generated tests.The pipeline supports Go, Rust, Julia, PHP, and Ruby through their respective runners and conventions.

C Data Contamination Risk

The paper examines contamination risk and formalizes evaluation metrics for XREPOTEST. Perplexity results suggest low leakage risk, while mutation and invocation analyses show why passing or covering code does not fully capture test quality.

  • Data contamination risk: 3.52–32.74 perplexity across languages indicates models have not memorized the dataset, suggesting low data-leakage risk.Ruby has higher perplexity than Go, Rust, PHP, and Julia.
  • Evaluation metrics: Test Pass Rate uses an all-or-nothing criterion: every test case in a suite must pass for the suite to count.This prevents partially correct suites from inflating pass rates.
  • Evaluation metrics: Invocation Rate counts generated suites that syntactically parse and directly call the focal function, penalizing both syntax errors and missed targeting.Tree-sitter parsing and static analysis are used before computing IR over all generated suites.
  • Mutation evaluation: Mutation testing differentiates fault-detection capability beyond pass rate, with GPT-5.2 achieving the highest MS across Go, Rust, and Ruby.Reported scores are Go: 19.62%, Rust: 4.14%, and Ruby: 6.29%.
  • Mutation evaluation: Mutation coverage is incomplete across languages because Ruby tooling yields lower-bound estimates, while Julia and PHP tooling constraints prevent inclusion.Julia libraries are incompatible with the required environment, and PHP tools operate only at file-level granularity.

E.4 Correlation Analysis

The correlation analyses compare XREPOTEST metrics with one another and with SWE-bench performance. Invocation Rate is related to compilability and coverage but is less aligned with pass rate and shows a negative observational association with SWE-bench.

  • Within-benchmark correlations: CSR and Cov correlate at r = 0.85, while TPR and Cov correlate at r = 0.88 across 14 evaluated models.CSR and TPR show a moderate correlation of r = 0.57.
  • Within-benchmark correlations: IR correlates strongly with CSR (r = 0.87), moderately with Cov (r = 0.65), and weakly with TPR (r = 0.39).This pattern indicates that invocation is not redundant with pass rate.
  • SWE-bench correlations: TPR has the strongest relationship with SWE-bench, with Pearson r = 0.44, Spearman ρ = 0.59, and Kendall τ = 0.46.Cov has a weaker positive Spearman correlation of ρ = 0.38.
  • SWE-bench correlations: CSR shows essentially no linear relationship with SWE-bench, with Pearson −0.12 and Spearman 0.13.The rank-order tendency is slight and positive.
  • SWE-bench correlations: IR shows negative correlations with SWE-bench: Pearson r = −0.57, Spearman ρ = −0.39, and Kendall τ = −0.33.Because the analysis covers only 12 models, the paper presents this as an observational pattern rather than a conclusive finding.
  • Hardness and overlap: The Venn analysis identifies shared solutions, model-specific strengths, and pairwise complementarity among three flagship models.Samples count as solved when tests pass and achieve non-zero coverage, motivating possible multi-model strategies.

F Agentic and Iterative-Repair Evaluation

Agentic execution improves test pass rates across all five languages, but multi-metric evaluation reveals that passing tests may reduce focal coverage or invocation reliability. A single feedback-driven repair iteration further improves pass rates while largely preserving invocation rate.

  • Agentic setup: Agentic execution improves Test Pass Rate (TPR) by 1.2×–2.6× over the passive baseline across all five languages.The evaluation uses a headless Claude Code agent with full repository access and iterative repair capabilities.
  • Agentic versus passive: PHP gains +11.5% TPR but lose 33.3% line coverage and 71.3% IR under agentic execution.The agent often exercises easier helper or public API functions instead of the assigned focal function.
  • Invocation and coverage: Julia reaches 100% IR while coverage decreases because multiple dispatch selects a non-focal method overload.The test invokes the intended API name but resolves to a forward-iteration overload rather than the focal reverse-iteration method.
  • Iterative repair: A single execution-feedback repair iteration improves TPR by 9–26% across all five languages while keeping IR nearly unchanged.The repair setting uses compiler and test-execution feedback with GPT-OSS-120B.
  • Invocation and coverage: 9.7% of TPR-passing suites fail IR overall, including 28.3% in Rust, 19.2% in Ruby, and 6.1% in Julia.Manual inspection attributes these failures to mocked, stubbed, or unrelated public-API calls rather than focal-function invocation.

G Retrieval Parameter Ablation

Retrieval performance is stable under moderate window-size and top-k changes, supporting a representative configuration without tuning to a particular model. The ablation applies only to retrieval-based augmentation because file-level and LSP contexts are deterministic.

  • Sensitivity results: Yi-Coder’s BM25 coverage ranges from 5.16% to 8.41% across nine configurations, a ≤3.3% absolute range.The ablation varies window size ws ∈ {30, 50, 70} and top-k k ∈ {5, 10, 15} on Go.
  • Sensitivity results: Codestral’s BM25 coverage varies only from 17.64% to 18.59%, a ≤1.0% absolute range.Similar stability patterns hold under dense retrieval.
  • Configuration choice: The chosen ws=50, k=10 configuration falls in the median-to-upper tier without peaking for any single model.This choice avoids overfitting to a particular architecture while providing a representative evaluation setting.
  • Configuration choice: The ablation supports low sensitivity of retrieval performance to moderate parameter changes.The reported results cover BM25 and Dense (UniXCoder) retrieval on Go using Codestral and Yi-Coder.
  • Scope: File-level and LSP-based contexts are deterministic and invariant across runs, so the ablation concerns retrieval augmentation only.The parameter study does not evaluate sensitivity of static augmentation contexts.

H.2 McNemar’s Test Results

McNemar’s tests and bootstrap confidence intervals assess whether model rankings differ reliably across 25 language–mode settings. The analyses identify a consistent leading model and confirm significant benefits from repository-aware context, while low pass-rate settings limit confidence in small differences.

  • Pairwise comparisons: 56 of 90 top-model pairwise comparisons (62%) are statistically significant at α = 0.05.Pairwise comparisons use McNemar’s test with Holm-Bonferroni correction for multiple comparisons.
  • Model rankings: In Rust under file-level context, Claude Sonnet 4.5 reaches 21.16% TPR versus 8.06% for GPT-OSS-120B, with non-overlapping confidence intervals.McNemar’s test confirms the separation: χ²=89.63, p=2.87e-21 with Bonferroni correction.
  • Context effects: Claude Sonnet 4.5’s Rust TPR rises from 12.78% to 21.16% with file-level context, confirmed by McNemar’s test.PHP increases from 26.95% to 31.43%, and Go from 23.66% to 34.14%, showing the same context effect.
  • Model rankings: Claude Sonnet 4.5 ranks first in 21/25 settings and records 32 significant pairwise wins, making it the most consistent top performer.GPT-5.2 records 19 significant wins, primarily in Go and Ruby.

J.3 API Hallucination Sub-category Analysis

The analysis decomposes API hallucinations into four sub-categories and shows that context changes their composition differently across languages. Failures primarily reflect ecosystem grounding, while Julia has an unusual concentration of signature mismatches.

  • Failure taxonomy: API Hallucination failures comprise Phantom Library, Non-existent API, Signature Mismatch, and Other categories.These represent nonexistent packages, fictitious functions, incorrect arguments or types, and primarily deprecated APIs, respectively.
  • Cross-language patterns: Context redistributes hallucination types rather than uniformly reducing them across languages.Table A.13 compares standard and context-augmented shares within the hallucination category, whose columns sum to 100%.
  • Cross-language patterns: 77.9% →72.9%: Rust Phantom Library errors remain dominant under context, while PHP increases from 88.8% →92.4%.The passage attributes these patterns to broader repository exposure amplifying incorrect assumptions about dependency structure.
  • Cross-language patterns: Go Phantom Library errors fall from 23.2% →3.9%, while Non-existent API errors rise from 71.2% →85.9% under context.The shift is described as context anchoring correct package structure without preventing API overgeneralization.
  • Interpretation: Signature Mismatch stays below 10% in most languages, except Julia at 21–33%, indicating an ecosystem-grounding problem more than type-level reasoning.Repository-aware metadata such as dependency graphs, import trees, and module registries is identified as a potentially more impactful constraint.
  • Illustrative examples: Concrete Go examples show hallucination through static type errors, references to unexported fields, overspecified traversal behavior, and tests that bypass the focal function.One passing test achieved zero focal coverage because it exercised a local wrapper rather than the real function.
Loading 2608.25939v1…