PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu, Chen Xing

arXiv:2502.06556v5cs.SEcs.CL

TL;DR

LLM unit-test benchmarks largely focus on single-file code, despite the cross-file dependencies in realistic projects. MultiFileTest benchmarks three languages and evaluates LLMs with manual and self-error-fixing. Most models perform moderately, and self-fixing remains behind human fixes.

  • Problem

    Existing LLM unit-test benchmarks mainly assess function-, class-, or single-file code, whereas multi-file projects require handling cross-file dependencies and test setup.

    Source: §1 · p. 1

  • Method

    MultiFileTest provides multi-file projects in Python, Java, and JavaScript and evaluates eleven LLMs with and without manual fixing and LLM self-fixing.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    Most evaluated frontier LLMs perform moderately; Gemini-3.0-Pro performs best overall, and self-fixing remains behind human fixes in quality and reliability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Executability and cascade errors can prevent assessment of more advanced aspects of generated tests, including correctness and coverage.

    Source: §1 · p. 2

    The benchmark covers only Python, Java, and JavaScript, with projects limited to approximately 1,600 lines rather than production-scale codebases.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Unit test generation has become a promising and important Large Language Model (LLM) use case. However, existing evaluation benchmarks for LLM unit test generation focus on function- or class-level code (single-file) rather than more practical and challenging multi-file-level codebases. To address such a limitation, we propose MultiFileTest, a multi-file-level benchmark for unit test generation covering Python, Java, and JavaScript. MultiFileTest features 20 moderate-sized and high-quality projects per language. We evaluate eleven frontier LLMs on MultiFileTest, and the results show that most frontier LLMs tested exhibit moderate performance on MultiFileTest, highlighting the difficulty of MultiFileTest. We also conduct a thorough error analysis, which shows that even advanced LLMs, such as Gemini-3.0-Pro, exhibit basic yet critical errors, including executability and cascade errors. Motivated by this observation, we further evaluate all frontier LLMs under manual error-fixing and self-error-fixing scenarios to assess their potential when equipped with error-fixing mechanisms. Our code and dataset is available at \href{https://github.com/YiboWANG214/ProjectTest}{MultiFileTest}.

1 Introduction

Existing LLM unit-test benchmarks mainly target function-, class-, or single-file code, while realistic projects require cross-file dependency handling. MultiFileTest evaluates unit-test generation across three languages and examines model errors and error-fixing.

1 Introduction

  • Existing benchmarks mostly cover function-, class-, or single-file code, leaving systematic evaluation of cross-file dependencies and testing environments limited.DevBench briefly includes multi-file unit testing but focuses broadly on software-development tasks rather than in-depth multi-file testing.Source: §1 · p. 1
  • MultiFileTest includes 20 projects each in Python, Java, and JavaScript, with cross-file dependencies enforced in every repository.Its 60 targets are moderate-sized, with each project under 1,600 lines of code.Source: §1 · pp. 1–2
  • Most evaluated frontier LLMs perform moderately; Gemini-3.0-Pro performs best overall, while language-specific leaders differ.Gemini-3.0-Pro ranks first in Python and Java, and GPT-o1 ranks first in JavaScript.Source: §1 · p. 2
  • Manual fixes change model rankings, while LLM self-fixing corrects some errors but remains behind human fixes in quality and reliability.Executability and cascade errors can prevent assessment of correctness and coverage.Source: §1 · p. 2

2 Related Work

Prior unit-test-generation work includes automated coverage-oriented methods and a range of LLM approaches. Benchmark coverage remains limited for systematic multi-file unit-test generation.

2.1 Unit Test Generation

  • Search-, constraint-, and random-based test generators target coverage, but their tests are often less readable and meaningful than developer-written tests.Source: §2.1 · p. 2
  • LLM unit-test-generation approaches include zero-shot prompting, iterative querying, fine-tuning, context selection, dynamic scaling, and code-difference-focused methods.Source: §2.1 · p. 2

2.2 LLM Unit Test Generation Benchmark

  • Existing benchmarks often target single-file code, while multi-file benchmarks focus on other tasks; DevBench's broad scope limits systematic multi-file unit-test analysis.The passage also notes quantity and quality concerns in DevBench's dataset.Source: §2.2 · pp. 2–3

3 Methodology

The methodology builds a three-language, multi-file benchmark and evaluates generated tests through execution, correctness, and coverage measures. It also compares vanilla generation with manually fixed and LLM self-fixed tests.

3.1 Benchmark Dataset

  • MultiFileTest contains 20 projects per language, selected for size, inter-file dependencies, and reliable sources, with preprocessing to make projects suitable for testing.Projects contain 2–15 files and fewer than 1,600 lines; preprocessing checks syntax, makes extracted projects self-contained, and consolidates multiline statements for coverage measurement.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Evaluation Metrics

  • The evaluation measures executability, correctness, and line and branch coverage to assess whether generated suites run, contain correct tests, and explore project code.Executability is measured across projects, correctness across generated tests per project, and coverage as the covered share of project lines or branches.Source: §3.2 · pp. 3–4
  • Executability and cascade errors can reduce correctness and coverage, motivating additional evaluations after manual fixes and LLM self-fixing.A non-executable suite yields zero correctness and coverage for its project; other test failures, such as missing dependencies, can also affect coverage.Source: §3.2 · p. 4

3.3 Unit Test Generation

  • The pipeline evaluates vanilla tests, tests after manual fixes, and tests refined by LLMs using error messages and conversation history.Manual fixes target mechanical executability and cascade errors while preserving test intent; self-fixing is not restricted to those error types.Source: §3.3 · pp. 4–5
  • Language-specific prompts give LLMs the full project and request tests across project files, with evaluation focused on executability, correctness, and coverage.Generated tests are extracted from the model response; manual correction addresses issues such as import failures that stop test collection or invalidate multiple tests.Source: §3.3 · pp. 4–5, Figure 3 · p. 4, Figure 4 · p. 4

4 Experimental Settings

The experiments evaluate seven closed-source and four open-source models. They use zero-shot prompts, language-specific test frameworks, and model-specific maximum input lengths.

4.1 Models

  • The evaluation includes seven closed-source models and four open-source models.The models include GPT, Gemini, Claude, CodeQwen, DeepSeek-Coder, CodeLlama, and CodeGemma variants.Source: §4.1 · p. 5

4.2 Implementation Details

  • Experiments use zero-shot prompting at temperature 0, maximize input length to each model’s token limit, and run on eight NVIDIA A100 GPUs.The testing frameworks are Pytest for Python, JUnit for Java, and Jest for JavaScript; JaCoCo measures Java coverage.Source: §4.2 · p. 5

5 Experiments

The experiments compare vanilla generation, manual fixing, and LLM self-fixing using correctness, executability, and coverage metrics. Results show moderate performance, gains from fixing, differences between models and languages, and persistent errors and redundancy.

5 Experiments

  • The evaluation compares vanilla, manually fixed, and self-fixed tests using correctness, executability, line-coverage, and branch-coverage metrics.It also includes unique-contribution and error analyses.Source: §5 · p. 5

5.1 Main Results

  • Gemini-3.0-Pro performs best overall, language leaders vary, and Java is hardest because stricter syntax lowers executability.Gemini-3.0-Pro leads in Java but trails GPT-o1 in JavaScript.Source: Table 2 · p. 6, §5.1 · p. 6

5.2 Manual Fixing Results

  • Manual fixing substantially improves results across languages and models; Java benefits most, and fixing can reorder model rankings.For example, CodeQwen1.5 outperforms DeepSeek-Coder in Java after fixing.Source: §5.2 · p. 6, Table 3 · p. 7

5.3 LLMs Self-fixing Results

  • Most closed-source models improve over vanilla through self-fixing, whereas evaluated open-source models lack reliability; GPT-o1’s JavaScript coverage remains below manual fixing.CodeQwen1.5’s self-fixing can exceed manual fixing in JavaScript when error messages help it interpret a prompt it had previously misunderstood.Source: §5.3 · p. 7, Table 4 · p. 7

5.4 Unique Contribution of Unit Tests

  • All tested LLMs have low unique-contribution rates; GPT-o1 produces more tests but lower unique contribution than GPT-5-mini despite higher coverage.Unique contribution measures the coverage each test adds without overlap with other tests.Source: §5.4 · p. 8, Table 5 · p. 8

5.5 Error Analyses

  • Executability and cascade errors include bad imports, syntax and mocking problems, and framework misuse, with patterns varying across Python, Java, and JavaScript.Java errors include missing or improper mocks; JavaScript errors include confusion between named and default imports and Jest compliance issues.Source: §5.5 · p. 8
  • After fixes, expected-versus-actual mismatches are most common; hallucinated or missing code and dependency misunderstandings also persist across languages.The paper links mismatches to incorrect expected values and weak reasoning, and links missing code to failures to understand codebase structure and dependencies.Source: §5.5 · p. 8

6 Conclusion

The paper presents MultiFileTest as a high-quality multi-file unit-test benchmark spanning three programming languages. It evaluates eleven LLMs with and without fixing mechanisms and analyzes errors by language.

6 Conclusion

  • MultiFileTest is presented as a reliable, high-quality multi-file unit-test benchmark spanning three programming languages.Source: §6 · p. 9
  • The study evaluates eleven LLMs with manual fixing and LLM self-fixing, alongside comprehensive error analyses for each language.Source: §6 · p. 9
  • The benchmark’s evaluation covers unit-test generation both with and without error-fixing mechanisms.Source: §6 · p. 9

Limitations

The study limits its benchmark to three languages and projects of approximately 1,600 lines or fewer. The accompanying appendices describe dataset and prompt materials, models, validation, and manual-edit statistics.

Limitations

  • The benchmark covers Python, Java, and JavaScript, excluding other relevant languages such as C and C#.Source: Limitations · p. 9
  • Projects are limited to approximately 1,600 lines, below many production-scale codebases, partly because of current LLM input-length and context-window constraints.The projects retain multiple files and meaningful inter-file dependencies, but the study does not represent the full complexity of production systems.Source: Limitations · p. 9

A Dataset

  • Dataset tables list each project’s language, name, license, link, stars, forks, file count, and lines of code.The appendix notes that “Author Permission” indicates permission obtained from the repository author.Source: A Dataset · p. 12

B.1 Prompts

B.2 Models

  • Model license and link information is provided in a table.Source: §B.2 · p. 12

C.1 Assert Statistics

  • A table reports, by language and model, the percentages of vanilla-generated tests containing expected-versus-actual comparisons.Source: §C.1 · p. 12

C.2 Robustness Analysis

  • Three independent GPT-3.5-Turbo runs show minimal variance; paired correctness-rate comparisons across nine models and three scenarios are significant, with p-values ≤0.009 and Cohen’s d > 1.1.Source: §C.2 · p. 12

C.3 Changed LOC Statistics of Manual Fixing

  • Manual edits to Python tests are modest and consistent across models, suggesting many errors can be fixed with minimal human effort.Source: §C.3 · p. 12

C.4 Comparison with Other Methods

The appendix compares LLM-generated tests with other methods and examines prompt components. On Java projects, vanilla LLMs trail EvoSuite, while self-fixing is comparable; prompt-component effects vary by metric.

C.4 Comparison with Other Methods

  • On Java projects, vanilla LLMs fall behind EvoSuite, while GPT-o1 self-fixing has comparable performance; HITS with iterative debugging and method slicing still achieves low coverage.Source: §C.4 · p. 12

D.1 Ablation Study on Prompts

  • Correctness- and executability-related prompt requirements can improve vanilla performance, while coverage-related requirements are not always beneficial.The authors suggest that coverage may be too abstract for LLMs to interpret effectively.Source: §D.1 · p. 13
  • Language-specific requirements improve correctness but reduce executability and coverage; comment-marked prompts marginally improve ER, LC, and BC, while the original prompt has an advantage in CR.Source: §D.1 · p. 13

D.2 Effect of Executability Errors and Cascade Errors

Manual fixes reveal how strongly evaluation depends on executable tests, while multi-turn self-fixing offers modest gains but risks context-related confounds. A single-turn design is used to isolate error-correction reasoning from context management.

D.2 Effect of Executability Errors and Cascade Errors

  • Fixing executability errors significantly improves results across all languages and LLMs; CodeQwen1.5, CodeGemma, and CodeLlama are more sensitive to cascade errors.The comparison with Java results is primarily affected by missing or invalid mocks of user interactions.Source: §D.2 · pp. 13–14, Table 9 · p. 14

D.3 Ablation Study on Multi-turn Self-fixing

  • On 10 Python projects, GPT-3.5-Turbo shows modest gains over three self-fixing turns, with coverage rates showing signs of plateauing by Turn 3.The multi-turn ablation was conducted to justify the single-turn design.Source: §D.3 · p. 14
  • Multi-turn conversation history rapidly consumes context and can omit critical multi-file dependencies, confounding model failures with context truncation.This is the stated attribution problem in the multi-turn setting.Source: §D.3 · p. 14
  • The single-turn design aims to isolate core error-correction reasoning from context management and external tooling.The authors frame long-context handling and external tooling as separate topics for future investigation.Source: §D.3 · p. 14

E Detailed Error Analyses

The error analysis identifies language-specific execution and cascade failures, followed by incorrect tests after manual fixes. Import and package handling are recurring sources of problems, while Java also faces mocking-related errors.

Executability Errors

  • Python package-detection failures can misdirect imports; Java errors include hallucinated symbols, package declarations, access violations, syntax and mocking problems, while JavaScript often has incorrect import paths.Java also encounters argument or type mismatches and improper Mockito use; CodeQwen1.5 may produce empty tests or invalid output.Source: E Detailed Error Analyses · pp. 14–15, Figure 10 · p. 16

Cascade Errors

  • Cascade errors include missing imports across languages; Java interaction-mocking failures can stop execution and prevent coverage data, while JavaScript errors also involve import styles and Jest.Python examples include missing project imports and FileNotFoundError; JavaScript examples include missing chai or three imports.Source: E Detailed Error Analyses · p. 15, Figure 11 · p. 16

Post-Fix Errors

  • After manual fixes, AssertionError from mismatched expected and actual values is the most common remaining error across all three languages.Python also frequently has AttributeError; Java and JavaScript have additional language- and framework-specific failures.Source: E Detailed Error Analyses · pp. 15–16
  • Other post-fix failures include Python hallucinated attributes, Java null-pointer and mock-use errors, and JavaScript TypeErrors from hallucinated functions or invalid mocking.Spring-specific errors are also common in Java projects tested with Spring.Source: E Detailed Error Analyses · pp. 15–16

F Comparison with Other Benchmarks

MultiFileTest is presented as a multi-language, multi-file unit-test benchmark with fine-grained error analysis and manual and LLM self-fixing mechanisms. Its comparison with DevBench emphasizes its dedicated unit-testing focus and larger project set.

Comparison with Other Benchmarks

  • MultiFileTest provides 60 projects, 20 per language, and uniquely combines fine-grained error analysis with manual-fixing and LLM self-fixing mechanisms.The comparison describes DevBench as covering broader software-engineering tasks and having a smaller unit-testing subset.Source: F Comparison with Other Benchmarks · p. 16

G Comparison between Unique Contribution and Other Metrics

The discussion motivates a project-agnostic metric for comparing heterogeneous projects and explains the benchmark’s scope and error-fixing protocol. The appendix specifies language-specific fix patterns and a validation process intended to support objective, reproducible annotations.

G Comparison between Unique Contribution and Other Metrics

  • The unique contribution metric is presented as a project-agnostic proxy for test utility, avoiding normalization challenges of execution time and code size across heterogeneous projects.The passage says execution times vary with external dependencies and environment, while code size varies by language and coding style.Source: G Comparison between Unique… · p. 16

H Discussion on Context Window Limitations

  • The benchmark limits projects to 1,600 lines to avoid truncation and isolate test-generation ability from long-context management or external tooling.The discussion identifies context extension, retrieval-augmented generation, and hierarchical decomposition as future directions for larger codebases.Source: H Discussion on Context Window… · p. 17

I.1 Overview

  • The fixing protocol targets only executability errors and cascade errors, using minimal changes determined objectively from error messages and codebase structure.Correctness errors, coverage issues, and test-logic flaws are explicitly excluded.Source: §I.1 · p. 17

I.2 General Decision Framework

  • The general decision framework identifies an error type, checks the matching protocol rule and its conditions, then reruns tests and documents the fix.Source: §I.2 · p. 17

I.3 Python Fixing Rules

  • Python cascade errors are treated as mechanically fixable when a single root cause causes multiple failures, such as an undefined name requiring an import.The Python rules also enumerate conditions and actions for fixable executability errors.Source: §I.3 · p. 17, §I.3.2 · p. 18, §I.3.1 · p. 18

I.4 Java Fixing Rules

  • Java fixes address missing package declarations and import paths; cascade errors also include test-framework imports and user-interaction mocks, often involving Scanner.A package declaration can be derived deterministically from the file’s directory path.Source: §I.4.1 · p. 18, §I.4 · p. 18, §I.4.2 · p. 18

I.5 JavaScript Fixing Rules

I.6 Validation Protocol

  • Each fix undergoes a five-step validation process applied identically by annotators to support objectivity and reproducibility.Source: §I.6 · p. 18, Table 25 · p. 21

Related papers

Loading 2502.06556v5…