PaperlayerView exact revision on arXiv

Source-linked AI summary

RepoQA: Evaluating Long Context Code Understanding

Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, Lingming Zhang

arXiv:2406.06025v1cs.SEcs.CLcs.LG

TL;DR

Long-context evaluations have largely left repository code understanding untested, although developers need to find functions in large codebases. RepoQA introduces SNF, which tests function retrieval from code context using natural-language descriptions, across 500 tasks from 50 repositories and five languages. The evaluation finds that proprietary models still outperform the best open-source models overall, while language performance varies and removing comments may help.

  • Problem

    Long-context benchmarks have not evaluated how LLMs understand code repositories, even though developers may need to locate functions across large codebases.

    Source: §1 · p. 1

  • Method

    RepoQA evaluates function retrieval from long code contexts using natural-language descriptions, with 500 tasks from 50 repositories across five languages.

    Source: §1 · p. 1

  • Results

    Proprietary models still outperform the best open-source models overall, language performance differs by model, and removing comments may help model understanding.

    Source: §5 · p. 4

  • Takeaways & Limitations

    RepoQA provides a benchmark for evaluating long-context code understanding through the Searching Needle Function task.

    Source: §5 · p. 4

Abstract

from arXiv · show

Recent advances have been improving the context windows of Large Language Models (LLMs). To quantify the real long-context capabilities of LLMs, evaluators such as the popular Needle in a Haystack have been developed to test LLMs over a large chunk of raw texts. While effective, current evaluations overlook the insight of how LLMs work with long-context code, i.e., repositories. To this end, we initiate the RepoQA benchmark to evaluate LLMs on long-context code understanding. Traditional needle testers ask LLMs to directly retrieve the answer from the context without necessary deep understanding. In RepoQA, we built our initial task, namely Searching Needle Function (SNF), which exercises LLMs to search functions given their natural-language description, i.e., LLMs cannot find the desired function if they cannot understand the description and code. RepoQA is multilingual and comprehensive: it includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages. By evaluating 26 general and code-specific LLMs on RepoQA, we show (i) there is still a small gap between the best open and proprietary models; (ii) different models are good at different languages; and (iii) models may understand code better without comments.

1. Introduction

Long-context benchmarks have not addressed how LLMs understand code repositories, despite developers needing to locate functions across large codebases. RepoQA introduces Searching Needle Function (SNF), a repository-based function search task, and evaluates models on a multilingual benchmark.

1. Introduction

  • RepoQA targets the gap in long-context code understanding by testing repository tasks that reflect real-life uses, such as locating a function.Developers may need to ask how to locate a specific function across massive lines of code.Source: §1 · p. 1
  • SNF asks models to retrieve a function from long code context using its natural-language description, requiring them to understand both the description and code.Each test includes an instruction, code context, function description, and repeated instruction.Source: §1 · p. 1
  • RepoQA comprises 500 code-search tasks from 50 repositories across five programming languages, and the paper reports evaluating 33 models.The contribution list describes an automatic pipeline for building SNF evaluation sets.Source: §1 · p. 1

2. Related Work

Existing long-context benchmarks cover realistic general tasks and synthetic retrieval, while code benchmarks cover repository completion or complex software engineering. RepoQA is positioned as a benchmark for core code understanding over very long contexts.

2. Related Work

  • Long-context benchmarks include realistic task suites such as ZeroSCROLLS, L-Eval, LongBench, and ∞Bench, as well as synthetic retrieval benchmarks such as NIAH and RULER.The cited benchmarks cover multiple tasks and context lengths, with ∞Bench extending beyond 100k tokens.Source: §2 · p. 2
  • RepoBench and CrossCodeEval assess repo-level code completion using cross-file context, but their input token counts remain limited.Source: §2 · p. 2
  • RepoQA addresses the missing evaluation of core code understanding over very long contexts, which the authors distinguish from repository completion and complex agent-oriented software engineering tasks.SWE-BENCH contains 2,294 real-world problems and is typically used for code-agent evaluation rather than model-only evaluation.Source: §2 · p. 2

3. RepoQA

RepoQA builds Searching Needle Function tests from selected repositories and evaluates whether models can retrieve described functions from long code contexts. Its curation and scoring procedures define the test inputs and the conditions for successful retrieval.

3. RepoQA

  • RepoQA’s two phases are data curation of SNF tests from repositories and model evaluation on those tests.The overview describes evaluation as comparing a model’s retrieved function with candidate functions in the context against a similarity threshold.Source: §3 · p. 2

3.1. Dataset Curation

  • The dataset uses 50 repositories across Python, C++, Java, TypeScript, and Rust, selecting 10 repositories per language and focusing on source files under each main-package entry directory.Repositories are chosen for topic diversity, quality, and popularity; dependency analysis annotates files for later steps.Source: §3.1 · p. 2
  • For each repository, SNF selects 10 needle functions by sampling functions from evenly sized chunks of a topologically ordered concatenation of source files.The chunks are 64 in number; candidate functions must have unique names and bodies under 2,000 bytes.Source: §3.1 · pp. 2–3
  • Descriptions specify a function’s purpose, inputs, outputs, and general procedures, with GPT-4-Turbo instructed not to reveal function or variable names.The annotation prompt also asks for descriptions specific enough to distinguish the target from other functions.Source: §3.1 · p. 3

3.2. Model Evaluation

  • Each evaluation prompt places a needle function at one of 10 evenly paced depths in a long, topologically ordered code context and includes its description and repeated task instruction.Context length is measured in CodeLlama-tokenizer tokens.Source: §3.2 · p. 3
  • Accuracy aggregates retrieval success across tests, requiring a syntactically correct extracted code block whose BLEU similarity is highest for the target and meets the default 0.8 threshold.The threshold is user-given and defaults to 0.8 in the paper.Source: §3.2 · p. 3

4. Evaluation

RepoQA evaluation compares 33 models on 500 tasks using 16k code contexts and a 0.8 matching threshold. Results vary across model families, comment conditions, and programming languages.

Experimental setup

  • Evaluation uses 16k code contexts and a minimum similarity threshold of 0.8, reporting the best results when training-free context-extension methods are used.For models limited to 8k or 16k contexts, the tested methods are dynamic RoPE scaling or overwriting the maximum length.Source: §4 · p. 3

Overall results

  • The best-tier models, mostly proprietary, exceed 90% accuracy; the best open-source models exceed 80% and slightly outperform Claude-3-haiku and GPT-4-Turbo.Within most model families, larger models perform better, with CodeLlama-13B-Instruct surpassing the 33B version; Llama-3-Instruct performs strongly with dynamic RoPE scaling.Source: §4 · p. 3

Impact of natural comments

  • Removing comments improves retrieval for models except Gemini, with code contexts padded by synthetic comments to align positions.The authors report that Gemini-1.5-Flash sometimes forgets the task and continues counting line numbers in comment-free mode.Source: §4 · p. 4

Impact of programming languages

  • Most models perform best on Java and TypeScript, followed by Python, C++, and Rust; some smaller models perform best on C++.The authors suggest language difficulty might relate to training-corpus size; JavaScript/TypeScript and Java have the largest amounts in the deduplicated Stack v2 dataset.Source: §4 · p. 4

5. Conclusion and Future Work

RepoQA evaluates long-context LLM understanding through Searching Needle Function, with 500 problems from 50 repositories across five programming languages. The authors report model- and language-dependent results, potential benefits from removing comments, and plans to broaden the benchmark and its tasks.

Conclusion

  • RepoQA's Searching Needle Function task asks models to retrieve a function from its natural-language description, spanning 500 problems, 50 repositories, and five programming languages.Evaluation of 33 models found proprietary models still outperform the best open-source models, while language performance varies and removing comments may help understanding.Source: §5 · p. 4

Future Work

  • Future work will expand Searching Needle Function to more programming languages and models.The authors also plan more complex tasks inspired by real-life development that can benefit from long-context LLMs.Source: §5 · p. 4

Appendix

The appendix includes additional tables and figures, including retrieval accuracy for all evaluated models at a matching threshold of 0.8, and a GPT-4-Turbo needle-function annotation prompt.

Needle function annotation prompt

  • The annotation prompt asks GPT-4-Turbo to briefly describe a function's purpose, input, output, and procedure, while avoiding function and variable names.The prompt requests a customized description that differentiates the function from others.Source: Appendix · p. 7

Additional results

  • Table 2 reports retrieval accuracy for all evaluated models using a matching threshold of 0.8.The appendix also notes that it includes additional tables and figures.Source: Appendix · p. 7, Table 2 · p. 8

Related papers

Loading 2406.06025v1…