LongCodeBench: Evaluating Coding LLMs at 1M Context Windows

• Existing long-context evaluations often use synthetic tests or reading-comprehension tasks that do not fully capture realistic codebase comprehension and repair.Many coding benchmarks evaluate shorter contexts, up to 64K–200K tokens, or isolated tasks such as commit-message generation.Source: §1 · p. 2
• LongCodeBench evaluates code comprehension and repair across context sizes from tens of thousands of tokens to one million, using issue-derived QA and real bug-fixing tasks.The benchmark contains 1,043 instances from 108 repositories, curated with human supervision.Source: §1 · p. 2
• LCB supports evaluation from frontier to smaller models by pairing repair with QA, where weaker models can achieve nonzero performance and receive graded feedback.The authors describe QA as a performance “gradient” for model improvement.Source: §1 · p. 2
• All LCB data come from public GitHub repositories and real issues, grounding both tasks in practical software-engineering scenarios rather than synthetic or simplified tasks.Source: §1 · p. 2
• LCB evaluates multiple context scales—32K, 64K, 128K, 256K, 512K, and 1M—to examine context behavior while retaining smaller-scale options for resource-constrained evaluation.Source: §1 · p. 3
• Claude 3.5 Sonnet’s LongSWE-Bench accuracy falls from 29% at 32K to 3% at 256K, illustrating that long context remains challenging even for strong models.The authors also report degradation with increasing context, difficulty predicting longer-context performance from shorter contexts, and a relationship between length sensitivity and task complexity.Source: §1 · p. 3

2.1 Benchmarks for LCLMs

• Long-context benchmarks extend evaluation beyond 128K tokens and diversify capabilities, yet synthetic tasks and some artificial recall or reranking settings limit real-world coverage.Source: §2.1 · p. 3
• LCB complements these benchmarks with software-engineering tasks, emphasizing long contexts, practical scenarios, and coding capabilities underrepresented in prior evaluations.Source: §2.1 · p. 3

2.2 Coding benchmarks for LCLMs

• Prior coding benchmarks include repository-based tasks and real-issue debugging, but LongCodeBench adds a code-comprehension challenge that the cited benchmarks lack.SWE-Bench uses real-world GitHub issues for debugging; LCB builds on that realism while adding comprehension.Source: §2.2 · p. 4
• LongCodeBench extends coding evaluation beyond SWE-Bench’s 50K-token limit, with samples reaching one million tokens.Source: §2.2 · p. 4

3.1 Code comprehension and repair

• LCB pairs repository-level comprehension with code repair, testing targeted information retrieval and reasoning alongside generation of patches consistent with project structure.The tasks also stress different bottlenecks: QA requires finding definitions, tracing functions, and locating usage; repair requires synthesizing precise, executable edits grounded in local project logic.Source: §3.1 · pp. 4–5
• The benchmark balances sample size against million-token inference costs, aiming for robust estimates while keeping evaluation costs practical.Source: §3.1 · p. 5

3.2 LongCodeQA

• LongCodeQA asks multiple-choice questions derived from public Python GitHub repositories, using the repository context to test code comprehension.Questions originate from closed issue discussions; GPT-4 filters issue types and helps convert selected issues into questions, with context-free answering used to remove questions solvable from general knowledge alone.Source: §3.2 · pp. 5–6
• Manual checks found that 52 of 54 sampled LongCodeQA questions required in-depth repository research, and questions were generated from issue discussions without access to the full codebase.Source: §3.2 · p. 6
• LongCodeQA contains 443 question-answer instances from 98 public GitHub repositories, sampled across context-length brackets for balanced evaluation.Source: §3.2 · p. 6

3.3 LongSWE-Bench

• LongSWE-Bench asks models to fix bugs from real GitHub issues using multiple repository files, and evaluates patches by whether all associated unit tests pass.Any failed test or compilation error counts as failure; performance is the percentage of solved issues.Source: §3.3 · p. 7
• The LongSWE-Bench pipeline manually excludes ill-posed issues and uses dedicated Docker environments to reproduce the original code version and verify bugs and tests.Issues are manually checked for clarity and unambiguous solutions; instances involving feature additions or design choices are excluded.Source: §3.3 · p. 7
• LongSWE-Bench contains 600 bug-fixing instances, with 100 in each of six context brackets from 32K to 1M, drawn from 61 public Python repositories.Source: §3.3 · p. 8

4.1 Selection of LCLMs

• The evaluation spans open- and closed-source models with context windows up to 1M; the authors assess long-context ability rather than optimizing RAG.They report Claude 3.5 Sonnet with RAG for reference and characterize those scores as a narrow view of RAG capabilities.Source: §4.1 · p. 8

4.2 Performance on LongCodeQA

• LongCodeQA performance was neither monotonic nor uniform: Gemini models stayed between 63–72% up to 1M, while Qwen2.5 fell from 70.2% at 512K to 40.0% at 1M.GPT-4.1 had the best task results, while Jamba fell from 72.8% at 128K to 54.2% at 256K.Source: §4.2 · p. 9

4.3 Performance on LongSWE-Bench

• LongSWE-Bench repair was markedly harder: Claude 3.5 Sonnet solved 29% of issues at 32K but 3% at 256K, while several models solved none.Open-source Qwen2.5, Llama 3.1, Llama 4, and GPT-4.1 solved no issues across the tested brackets.Source: §4.3 · p. 9

4.4 Discussion

• Newer models showed task-specific shifts: GPT-4.1 improved on LongCodeQA but underperformed on LongSWE-Bench, while Gemini 2.5 Pro gained on repair without comparable QA improvement.The authors suggest newer models may be increasingly optimized for particular domains or problem types.Source: §4.4 · p. 10

• LongCodeBench evaluates long-context language models on real-world coding tasks with inputs up to one million tokens.Source: §5 · p. 10
• The benchmark combines GitHub-sourced code comprehension and repair tasks in a scalable, realistic evaluation framework.Source: §5 · p. 10
• Results indicate that current LCLMs often degrade at scale, revealing a gap between claimed and effective context capabilities.Source: §5 · p. 10

• The study uses only publicly available data from open-source GitHub repositories and includes no personal or sensitive information.Source: Ethics statement · p. 11
• The authors state that experiments comply with the terms of service of the platforms and APIs used.Source: Ethics statement · p. 11
• The authors say they will release the dataset and code to support transparency and reproducibility.Source: Ethics statement · p. 11

• The appendix reports further experiments intended to guide the design of LCB’s data collection.Source: A Further analysis on data collection · p. 13

A.1 Performance on the small-scale dataset before and after the first filter

• Accuracy dropped substantially after filtering out questions that did not require repository-specific knowledge.The comparison tested two models on an initial dataset of around 60 samples.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 Performance on the complete dataset before final GPT-4o-based filtering

• At 1M tokens, Qwen2.5 accuracy drops to 54.5%, while the reported results show reduced accuracy for most models at extreme lengths.The dataset in this evaluation precedes a final filter that retains challenging questions requiring detailed repository analysis.Source: §A.2 · p. 13, Table 5 · p. 13

B Prompts

• The prompts appendix collects the prompts used to generate LongCodeQA and prompt LLMs.Source: B Prompts · p. 14

B.1 Prompt to identify issues convertible into questions

• The first prompt selects issues that can become repository-grounded multiple-choice questions, excluding issues requiring code changes or new features.It asks for a correct answer and three distractors and instructs the model to be conservative.Source: §B.1 · p. 14

B.2 Prompt to filter questions requiring repository-specific knowledge

• The second filter asks whether answering accurately requires detailed, up-to-date repository knowledge rather than prior or general programming knowledge.It requests reasoning and a final Yes or No decision.Source: §B.2 · pp. 14–15

B.3 Prompt for LongCodeQA task

• The LongCodeQA prompt provides repository contents and a question, and requests only the answer's letter.Source: §B.3 · p. 15

B.4 Prompt for LongSWE-Bench task

• The LongSWE-Bench prompt provides a partial codebase and issue, then requests a single applicable patch file as the response.The prompt includes an example patch format.Source: §B.4 · p. 15

C Inference time and cost

• Claude 3.5 Sonnet costs 100 USD for LongCodeQA and 140 USD for LongSWE-Bench, the highest reported API-model costs.Source: C Inference time and cost · p. 16, Table 6 · p. 16
• Qwen2.5 inference used 8×A100 80GB GPUs for LongSWE-Bench over 12 hours and 4×A100 64GB GPUs for LongCodeQA over 35 hours.Source: C Inference time and cost · p. 16
• GPT-4o costs 50 USD per task, and the reported figures underscore the financial and computational burden of evaluation near one million tokens.Source: C Inference time and cost · p. 16, Table 6 · p. 16

D Comparison with SWE-Bench

• LongSWE-Bench extends the maximum context length to one million tokens and evaluates across multiple length brackets.Source: D Comparison with SWE-Bench · p. 16
• LCB always supplies ground-truth files and adds random distractor files, aiming to measure model performance without conflating it with retrieval accuracy.SWE-Bench uses BM25 retrieval; ground-truth file inclusion or omission can create a sizable performance gap.Source: D Comparison with SWE-Bench · p. 16
• Random additional-file selection reduces relevance-based sampling bias at the cost of higher variance, which can be controlled with enough instances.Different random file selections also allow more samples per issue.Source: D Comparison with SWE-Bench · p. 16
• p=0.005 for random retrieval is statistically significant at the 0.01 threshold, whereas BM25 retrieval has p=0.025, above that threshold.The logistic-regression analysis concerns the relation between context length and resolution rate.Source: D Comparison with SWE-Bench · p. 16, Figure 5 · p. 17, Figure 6 · p. 17