PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion

Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, Bing Xiang

arXiv:2310.11248v2cs.LGcs.CLcs.SE

TL;DR

Existing code-completion benchmarks underrepresent repositories where correct completions depend on cross-file context. CrossCodeEval constructs multilingual examples requiring that context and evaluates models and retrieval methods, finding substantial gains when context is supplied but persistent room for improvement.

  • Problem

    Existing benchmarks mainly evaluate single-file completion, although real software repositories contain cross-file dependencies needed for accurate completion.

  • Method

    CrossCodeEval builds a multilingual benchmark requiring cross-file context and identifies such examples with static analysis of repository imports and unresolved names.

  • Results

    Adding cross-file context significantly improves code-completion accuracy, yet even the top-performing model with the best retrieval method remains imperfect.

  • Takeaways & Limitations

    CrossCodeEval evaluates models’ ability to complete code using cross-file context and can also measure the capability of code retrievers.

  • Takeaways & Limitations

    CrossCodeEval currently supports only four programming languages, although its construction method is potentially extensible.

Abstract

from arXiv · show

Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion tasks within a single file. This over-simplified setting falls short of representing the real-world software development scenario where repositories span multiple files with numerous cross-file dependencies, and accessing and understanding cross-file context is often required to complete the code correctly. To fill in this gap, we propose CrossCodeEval, a diverse and multilingual code completion benchmark that necessitates an in-depth cross-file contextual understanding to complete the code accurately. CrossCodeEval is built on a diverse set of real-world, open-sourced, permissively-licensed repositories in four popular programming languages: Python, Java, TypeScript, and C#. To create examples that strictly require cross-file context for accurate completion, we propose a straightforward yet efficient static-analysis-based approach to pinpoint the use of cross-file context within the current file. Extensive experiments on state-of-the-art code language models like CodeGen and StarCoder demonstrate that CrossCodeEval is extremely challenging when the relevant cross-file context is absent, and we see clear improvements when adding these context into the prompt. However, despite such improvements, the pinnacle of performance remains notably unattained even with the highest-performing model, indicating that CrossCodeEval is also capable of assessing model's capability in leveraging extensive context to make better code completion. Finally, we benchmarked various methods in retrieving cross-file context, and show that CrossCodeEval can also be used to measure the capability of code retrievers.

1 Introduction

Existing code-completion benchmarks largely provide only current-file context, unlike real repositories whose cross-file dependencies often determine correct completions. CrossCodeEval addresses this gap with a multilingual benchmark and shows that models improve with cross-file context but remain imperfect.

  • Motivation: Most existing code-completion benchmarks evaluate models using only context from the current file, oversimplifying modern repository-based development.Real repositories contain extensive dependencies across multiple source files.
  • Motivation: CodeGen-2B-mono fails to complete a CaseConverter API call when only the current file is provided.The example illustrates that the missing information resides in another repository file.
  • Contribution: CrossCodeEval contains 10k examples from 1k repositories in Python, Java, TypeScript, and C#, each requiring cross-file context for correct completion.The dataset uses quality filters and minimizes overlap with code-LM training data to reduce leakage and memorization concerns.
  • Results: Models perform suboptimally with current-file context, while adding cross-file context significantly improves code-completion performance, including in zero-shot settings.The evaluation covers CodeGen, StarCoder, and GPT-3.5-Turbo across multiple model sizes.
  • Results: Even the strongest models remain notably imperfect after receiving cross-file context, leaving CrossCodeEval challenging for extensive-context utilization.The benchmark therefore evaluates both cross-file completion and the ability to leverage retrieved context effectively.

2 CROSSCODEEVAL: A Benchmark for Cross-File Code Completion

CrossCodeEval is a multilingual scope-completion dataset whose examples are constructed to require local repository APIs and cross-file resolution. Static analysis, repository filtering, and quality controls produce examples suitable for completion and fill-in-the-middle evaluation, while the benchmark remains limited to four languages.

  • Dataset: CrossCodeEval covers Python, Java, TypeScript, and C# with prompts ending at an imagined cursor and references completing the statement.Each example includes at least one local API defined in the repository.
  • Static-analysis generation: The construction method replaces intra-project imports with empty classes and uses static analysis to identify names that become unresolved.Undefined-name locations are mapped back to the original file to determine prompt and reference boundaries.
  • Static-analysis generation: Language-specific tools detect unresolved names: Pylint for Python, javac for Java, and tsc for TypeScript.The C# pipeline uses the csc compiler from the mono image, while statement boundaries are identified for reference construction.
  • Quality control: Quality filters remove examples with insufficient prompts, references outside the 3–30-token range, cross-file reference duplication, or duplicate references.These filtering steps collectively remove 15%-20% of examples.
  • Quality control: A StarCoderBase-1B screening step removes examples whose references can be inferred exactly from current-file prompts.This removes <10% of generated examples and further reduces memorization risk.
  • Scope: Including suffix lines after references enables CrossCodeEval to evaluate fill-in-the-middle capabilities in addition to ordinary completion.The prompt or prefix and suffix provide code surrounding the reference span.
  • Future extensions: CrossCodeEval currently supports four popular languages, although the construction method can potentially extend to additional languages.The paper also advises excluding the benchmark from future pre-training datasets to minimize memorization effects.

3 Experiments

The experiments benchmark public and proprietary code language models under zero-shot prompting and examine how cross-file retrieval affects completion, scaling, and retrieval quality. Results show that in-file-only prompting performs poorly, while retrieved context substantially improves performance, though effective use and retrieval remain imperfect.

  • 3.1 Models: The study benchmarks CodeGen, StarCoder, GPT-3.5-Turbo, and model variants across sizes using zero-shot generation without training.Experiments use standardized generation hyperparameters and report code-match and identifier-match metrics.
  • 3.3 Context Retrieval: Baseline prompts contain only current-file context, while retrieval prompts prepend retrieved cross-file code fragments to that context.The retrieve-and-generate setup extracts ten-line fragments and queries them using the last ten in-file context lines.
  • 3.4 Results: 8.82% code exact match is achieved by the best 15.5B StarCoder model in Python with in-file context only.All models perform poorly without cross-file context, supporting the benchmark’s requirement for repository-level information.
  • 3.4 Results: Up to 3.0× and 4.5× better exact code match is reported for StarCoder with retrieved and retrieved-with-reference context, respectively.Performance improves across models and sizes when cross-file context is added to prompts.
  • 3.5 Analysis and Discussions: Correct completion quality depends strongly on retrieving the right cross-file context, while fixed-line fragments can be difficult for zero-shot models to digest.For StarCoder on Python, 327 correct generations became incorrect and 468 incorrect generations became correct when switching retrieval settings.
  • 3.5 Analysis and Discussions: Model performance follows a power-law increase with size but remains far from perfect, and the best retriever achieves below 20 EM in every language.BM25 often outperforms UniXCoder, whereas OpenAI’s ada embedding is generally stronger, especially for Java and C#.

4 Related Works

Earlier repository-level code generation work highlights cross-file context, but often targets a specific approach, language, or completion scope. CROSSCODEEVAL instead provides a broader multilingual benchmark designed to evaluate code LMs generally.

  • Recent repository-level benchmarks emphasize the importance of cross-file context but were often collected to evaluate their associated approaches.
  • Shrivastava et al. and Ding et al. collect data in only one programming language, while Pei et al. restrict completion to function arguments.
  • CROSSCODEEVAL covers Python, Java, TypeScript, and C# to evaluate general code completion capacity rather than one application type.
  • CROSSCODEEVAL draws from diverse permissively licensed GitHub repositories and avoids overlap with The Stack to reduce potential memorization issues.

5 Conclusion

The paper introduces CROSSCODEEVAL as a multilingual benchmark for code completion requiring cross-file context. Its experiments show that added cross-file context improves model accuracy, while substantial room for improvement remains.

  • CROSSCODEEVAL is a diverse and multilingual benchmark for cross-file code completion.
  • The benchmark uses static analysis to identify cross-file-dependent code fragments and evaluates whether models can use that context accurately.
  • Including cross-file context significantly improves code language models’ completion accuracy, demonstrating the benchmark’s intended evaluation capability.
  • Even the top-performing model with the best retrieval method leaves substantial room for progress in leveraging extensive context and building better retrievers.

Appendices

The appendices detail language-specific procedures for detecting cross-file references through static analysis or compiler errors. They describe how these references become benchmark prompts and completions.

  • Python: Python static analysis runs Pylint on standalone files after imported classes are replaced by empty classes, detecting undefined members without project-level information.
  • Java: Java extraction compares javac errors before and after replacing an imported class with a dummy class to locate original cross-file method calls.
  • TypeScript: TypeScript processing identifies imports and dependencies, removes imports or inserts dummy classes, then uses tsc errors and line numbers to form references.
  • C#: C# generation replaces the original class with a dummy class, compiles all files with the Mono compiler, and locates references through incremental errors.
  • Example construction: The Java workflow compares compiler errors from original and modified code to identify cross-file entities for prompt and ground-truth construction.

B Human Annotation for Quality Control

Human annotation assessed whether references required cross-file information, were predictable from the current file, and deserved dataset inclusion. Annotators largely agreed that the examples captured cross-file dependencies, while a small minority flagged removal candidates.

  • Six experienced authors annotated sampled Python and Java examples, with each example reviewed by two annotators.
  • The annotation asked whether references required associated cross-file information, could be predicted from current-file context, and should remain in the dataset.
  • Annotators reported near-universal cross-file dependence in references, while only 2% were judged predictable using current-file context alone.
  • Removal was suggested for 2.5% of examples in one set and 7% in the other, often because long reference strings were difficult to predict and caused disagreement.
  • High agreement scores across most questions indicate substantial annotator consensus for both languages.

C Retrieve-and-Generate Modeling Details

The Retrieve-and-Generate framework retrieves relevant cross-file code by comparing in-file context with chunked snippets from other files. An oracle variant uses target-aware context to identify a directly relevant completion hint, while the accompanying annotation setup remains sparse.

  • Retrieve-and-Generate Framework: Figure 8 illustrates additional details of the Retrieve-and-Generate framework and its context-retrieval process.The figure is presented as an illustration of context retrieval within the RG framework.
  • Cross-file Context Retrieval: The Retrieve-and-Generate framework uses the last 10 code lines before the cursor as a query for cross-file context retrieval.The query is constructed from in-file context, and the goal is to find relevant code from other files.
  • Cross-file Context Retrieval: Other-file code is chunked into 10-line candidates, and the candidate most similar to the in-file query is selected.Similarity calculation compares the in-file context with code snippets from other files.
  • Cross-file Context Retrieved w/ Reference: The oracle-context variant constructs its query from the last 10 lines of concatenated in-file context and target completion, selecting lines 8-17 as the most similar candidate.Unlike ordinary retrieval, the selected cross-file context is directly used as oracle context because the query includes the target completion.
  • Annotation Limitation: The study omits Krippendorff’s α and Cohen’s κ because examples and annotations are limited and annotations are relatively sparse.Only limited examples and annotations per example were available.

D.1 Evaluation with Additional Models

The appendix extends CROSSCODEEVAL evaluation beyond the main reported models by testing additional CodeGen and StarCoder variants.

  • Additional Models: Additional CodeGen and StarCoder variants, including StarcoderBase, are evaluated on CROSSCODEEVAL.The results are reported in Table 7 alongside the main evaluation.

D.2 Nucleus Sampling w/ Re-ranking

The appendix tests nucleus sampling with mean-log-likelihood reranking and finds results comparable to greedy decoding. Cross-file and oracle context retain equivalent improvements across search settings.

  • Sampling and Re-ranking: Nucleus sampling uses temperature 0.2 and top-p 0.95, followed by mean-log-likelihood reranking.
  • Comparison with Greedy Search: The sampling-and-reranking results are quite comparable to the main greedy-search results, with only marginal differences.The experiments are reported in Table 8.
  • Context Effects: Cross-file and oracle context bring equivalent improvement under sampling and reranking as under greedy search.
  • Context Effects: Cross-file context remains helpful regardless of the sampling or search algorithm.

D.4 Qualitative Analysis

The qualitative analysis shows that retrieved cross-file context helps models reuse repository patterns and inspect imported dependency implementations. These two retrieval roles correct different kinds of completion errors.

  • Pattern Reuse: Retrieved snippets can expose repetitive repository coding patterns that help models adapt existing implementations to complete code.One example retrieves a similarly purposed step() function that generates a token.
  • Dependency Implementations: Cross-file retrieval can also provide implementation details for imported dependencies rather than similar API usage.Figure 10 retrieves store_by_text from EntitySessionStorage, instantiated as self.entity_repository.
  • Dependency Implementations: Without the dependency context, the model hallucinates an undefined function usage; prepending it enables the ground-truth prediction.

E Limitations

The evaluation uses zero-shot prompting and a retrieval framework whose fixed context window and token-based similarity can return unhelpful cross-file information. Results should also be interpreted cautiously because prior model exposure to evaluation code cannot be ruled out.

  • Zero-shot Evaluation: Zero-shot evaluation limits performance because models never see the cross-file-context format during training or prompting.The authors omit few-shot evaluation because most benchmarked models have limited maximum sequence lengths for additional prompt examples.
  • Cross-file Context Retrieval Quality: RG retrieval can select useless information or fail to improve generation because of its fixed context window and token-based similarity calculation.The authors expect a more advanced retrieval approach to provide more accurate cross-file contexts.
  • Memorization: Prior memorization of evaluation data cannot be ruled out because code language models were trained on vast amounts of unlabeled code.The authors excluded popular packages from annotation but still advise caution when interpreting results.
Loading 2310.11248v2…