PaperlayerView exact revision on arXiv

Source-linked AI summary

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

Adarsh Kumarappan, Pareesa Ameneh Golnari, Wen Wen, Xiaoyu Liu, Gabriel Ryan, Yuting Sun, Shengyu Fu, Elsie Nallipogu

arXiv:2601.11895v3cs.LGcs.AIcs.SE

TL;DR

Existing code-completion benchmarks provide limited realism, diagnostics, and contamination resistance because they rely on public code and static completion rules. DevBench addresses these gaps by deriving task categories from over one billion developer interactions and evaluating synthetic instances across languages with multiple complementary metrics. Across its benchmark design and evaluations, it supports realistic, fine-grained model comparison while revealing distinct capability patterns and practical scope boundaries.

  • Problem

    Existing benchmarks rely on public code and static completion rules, limiting realism, diagnostic specificity, and resistance to training-data contamination.

  • Method

    DevBench derives six categories from over one billion developer interactions and evaluates 1,800 synthetic instances across six languages using telemetry-grounded scenarios and multiple evaluation methods.

  • Results

    DevBench reveals distinct strengths in low-context pattern recognition and persistent challenges in bidirectional natural-language–code translation and syntactic alignment across nine evaluated models.

  • Takeaways & Limitations

    DevBench provides ecological validity, contamination resistance, and scenario-specific diagnostics for more accountable and targeted evaluation of code-generation models.

  • Takeaways & Limitations

    Raw user code is not released or reused; all benchmark instances are synthetic to satisfy privacy and compliance requirements.

Abstract

from arXiv · show

DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six programming languages and six task categories derived from real developer telemetry and synthesized using generator models from multiple provider families to mitigate single-source bias. Unlike prior benchmarks, it emphasizes ecological validity, avoids training data contamination, and enables detailed diagnostics. The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments focused on usefulness and contextual relevance. 9 state-of-the-art models were assessed, with the strongest achieving only 43.5% Pass@1, confirming the benchmark remains challenging and revealing differences in syntactic precision, semantic reasoning, and practical utility. Our benchmark provides actionable insights to guide model selection and improvement, detail that is often missing from other benchmarks but is essential for both practical deployment and targeted model development.

1 Introduction

Existing code-generation benchmarks often use publicly sourced, rule-defined completions that poorly represent real developer usage, provide limited diagnostics, and risk contamination. DevBench addresses these gaps with telemetry-grounded, synthetic, cross-language evaluation designed for realism and fine-grained analysis.

  • Motivation: Existing benchmarks use open-source or coding-challenge data and static completion rules, limiting realism, diagnostic specificity, and contamination resistance.Their aggregate metrics cannot attribute performance differences to specific usage areas.
  • Contribution: DevBench derives six task categories from over one billion developer interactions and synthesizes 1,800 instances across six languages.The instances use generator models from multiple provider families and are reviewed for quality and realism.
  • Contribution: DevBench combines observed developer behavior, synthetic controlled generation, fine-grained utility evaluation, and six-language coverage.The benchmark assesses semantic alignment and developer utility rather than only aggregate correctness.
  • Contribution: DevBench supports overall rankings and scenario-specific diagnostics while providing a contamination-resilient foundation for model research.The benchmark and evaluation code are released as an open-source 1,800-instance resource.

2 Benchmark Design

DevBench constructs realistic code-completion tasks from telemetry-derived capability categories, then instantiates and validates synthetic examples across languages, generators, and practical context lengths. Its categories capture distinct but related skills, while balanced, diverse instances support realistic evaluation.

  • Benchmark Design: DevBench treats code generation as a composite task requiring API usage, intent understanding, and syntax control from telemetry-derived scenarios.Synthesis instantiates empirically derived patterns without releasing raw user code.
  • From User Telemetry to Categories: Over one billion anonymized completion interactions provide the telemetry basis, while raw user code remains unreleased and benchmark instances are synthetic.The corpus includes context, completions, and user actions across diverse developer settings.
  • Generation and validation: Three generator families produced candidates, and validation reduced 2,052 candidates to 1,800 final instances through syntax checks, deduplication, and blinded human review.The final set has near-balanced provenance across GPT-4o, Claude 3.5 Sonnet, and DeepSeek-V3.
  • Benchmark categories: Six categories target distinct developer intents across languages, including API usage, code-purpose understanding, code–natural-language translation, low-context completion, pattern matching, and syntax completion.Language-specific adaptations preserve idioms and ecosystem differences.
  • Category relationships: Category Pass@1 correlations average ρ̄ = 0.44 across 54 model–language combinations, while PCA indicates two to three capability dimensions beyond one general factor.Pattern Matching and Code Purpose Understanding correlate at ρ = 0.74, but individual models still diverge diagnostically.
  • Benchmark construction: Each instance contains prefix context, a golden completion, optional suffix context, and hidden assertions, with cursor positions reflecting natural telemetry-observed boundaries.This supports both prefix-only and fill-in-the-middle evaluation.
  • Complexity and Diversity: 57.6 LOC average instance length is paired with 117.9 prefix tokens and 108.2 completion tokens, while mean within-prompt TF-IDF cosine similarity is 0.094.The balanced structure and low similarity indicate practical completion proportions and high instance diversity.

3 Evaluation methods

DevBench combines functional correctness, similarity metrics, and LLM-judge scoring to evaluate generated completions from complementary perspectives. The design measures execution success, semantic and strict textual alignment, and contextual usefulness.

  • Evaluation methods: The evaluation combines functional correctness, similarity-based metrics, and LLM-judge assessments to address the difficulty of evaluating LLM-generated code.These methods provide complementary measurements across languages and tasks.
  • Functional correctness: Pass@1 measures the probability that at least one generated sample passes all test cases, using k = 1.The evaluation reports Pass@1 with n = 5 samples.
  • Similarity metrics: Average Cosine Similarity measures semantic overlap across completions, while Line 0 Exact Match Rate measures exact agreement on the first completion line.Both metrics are averaged over five generated completions per test case.
  • LLM-judge assessment: The LLM judge scores relevance and helpfulness from 0–5 each, producing a 0–10 combined score averaged over five completions.Gemini 2.5 Flash judgments correlate with averaged human scores at Spearman ρ = 0.81.

4 Experiments

Across nine models, DevBench exposes substantial differences in functional correctness, similarity to reference completions, and developer-perceived usefulness. The results show both broad capability gaps and targeted weaknesses by category, language, and metric.

  • Functional correctness: 43.5% Pass@1 is GPT-5.5’s leading result, followed by DeepSeek V4 Pro at 43.3% and Claude Opus 4.7 at 40.5%.No model exceeds 44% Pass@1, indicating that the benchmark remains challenging for frontier models.
  • Functional correctness: A roughly 25-point Pass@1 gap separates the strongest and weakest models, producing a gradient from frontier systems to compact and smallest open models.The reported range runs from frontier models at 40–44% to Qwen3.6-27B at 18.0%.
  • Functional correctness: Low Context is strongest at 46–54% for top performers, whereas Code2NL/NL2Code is hardest at 29–32% for leading models and below 26% for most others.Pattern Matching also differentiates models sharply, ranging from DeepSeek V4 Pro at 60.4% to Qwen3.6-27B at 22.7%.
  • Similarity-based evaluation: DeepSeek V4 Pro, Llama 4 Maverick, and Mistral Medium 3.5 achieve the strongest average cosine similarity across most languages.DeepSeek V4 Pro also has the highest Line 0 Exact Match Rates in Python and JavaScript, at 60.67% and 62.67%.
  • Similarity-based evaluation: DeepSeek V4 Pro leads similarity despite GPT-5.5’s slightly higher Pass@1, showing that syntactic closeness and functional correctness can diverge.The reported Pass@1 values are 43.3% for DeepSeek V4 Pro and 43.5% for GPT-5.5.
  • LLM-judge evaluation: The LLM-judge ranking differs from Pass@1 because it evaluates relevance and helpfulness rather than executable correctness.Llama 4 Maverick and Mistral Medium 3.5 receive the highest judge scores, 9.13 and 9.22, despite lower Pass@1.
  • LLM-judge evaluation: GPT-5.5 leads Pass@1 but ranks below Llama 4 Maverick and Mistral Medium 3.5 on the judge, demonstrating divergence between correctness and contextual usefulness.The judge’s frontier ranking is DeepSeek V4 Pro at 7.71, GPT-5.5 at 6.11, and Claude Opus 4.7 at 6.07.
  • Diagnostic case study: DeepSeek V4 Pro shows category and language-specific weaknesses despite strong overall performance, including 29.3% Pass@1 in Code2NL/NL2Code and a 5.42 JavaScript judge score.The model’s judge scores range from 9.70 in C++ to 5.42 in JavaScript, while its TypeScript Pass@1 is 36.1%.

5 Related Work

Existing code-generation benchmarks span problem solving, natural-language-to-code translation, and repository-based evaluation, but use varied task formats and sources.

  • Problem-solving benchmarks evaluate coding problems across difficulty levels, including HumanEval, MBPP, and APPS.
  • Concode and CoNaLa assess natural-language-to-code translation.
  • Repository-based benchmarks evaluate code generation within existing codebases, including masking tasks.

6 Conclusion

DevBench provides a telemetry-grounded benchmark for realistic, fine-grained code-completion evaluation across languages and task categories. Its evaluation reveals recurring capability patterns and supports more targeted assessment of model behavior.

  • DevBench contains 1,800 synthetic evaluation instances grounded in developer telemetry across six languages and six task categories.
  • The benchmark emphasizes ecological validity, contamination resistance, and interpretability through realistic, fine-grained code-completion evaluation.
  • Across nine state-of-the-art models, low-context pattern recognition was a consistent strength, while bidirectional natural-language–code translation and syntactic alignment remained challenging.
  • Functional correctness, similarity metrics, and LLM-judge assessments exposed differences in cross-language consistency and robustness across task types.

A.1 Telemetry derivation details

The telemetry derivation pipeline samples difficult real-world completions, annotates their scenarios and failure modes, and converts validated clusters into balanced benchmark instances. It combines category analysis, human review, and controlled generation to improve realism and comparability.

  • Category derivation: Category derivation began with a difficulty-enriched sample of 50,000 completions balanced across six languages and weighted toward low-acceptance or high post-acceptance-edit cases.
  • Category derivation: Scenario and failure-mode annotations were clustered and consolidated into six cross-language categories after substantial inter-annotator agreement.Agreement before adjudication was Fleiss’ κ = 0.74.
  • Sampling design: Python and JavaScript account for approximately 55% of telemetry completions, but the benchmark assigns 300 instances to each language for balanced comparison.
  • Benchmark structure: DevBench averages 57.6 LOC per instance, with 117.9 prefix tokens and 108.2 completion tokens, maintaining a more balanced context-to-completion structure than CrossCodeEval.
  • Quality control: Generation produced 2,052 candidates, which were reduced to 1,800 instances after syntax or execution failures, near-duplicate filtering, and blinded human review.
  • Quality control: Each instance combines prefix, golden completion, suffix, and hidden assertions, while independent reviewers assess usefulness, realism, category alignment, and complexity authenticity.

A.3 Generator bias and contamination analysis

DevBench addresses generator-family bias and distributional overlap through multiple generator families, shared prompts, telemetry-derived task structures, and diversity checks. Its prompts also enforce realistic, context-dependent completion and hidden functional validation.

  • Generator bias: Three generator families produced 684 candidates each, enabling analysis of whether generator family predicts evaluated-model pass or fail outcomes.
  • Contamination analysis: Generation instantiates task structures derived from over one billion developer interactions rather than relying solely on code patterns produced by language models.
  • Diversity analysis: Instances sharing a prompt had mean pairwise TF-IDF cosine similarity of 0.094, indicating high token-based diversity within prompt groups.
  • Prompt design: Each language–category cell uses a shared prompt encoding task definitions, schemas, hidden-assertion formats, difficulty requirements, and rejection criteria across generator models.
  • Task realism: Generated tasks require models to resolve contextual constraints such as ownership, error paths, parameter ordering, boundary conditions, or helper-method consistency.
  • Task realism: The completion structure requires code at the cursor to remain inferable from prefix and suffix without duplicating the suffix or revealing the answer.
  • Validation: Hidden assertions validate functional behavior, include an edge case, and remain separate from the prefix and suffix so tests do not leak the golden completion.
  • Quality control: Prompts reject ambiguous, externally dependent, near-duplicate, keyword-solvable, boilerplate-dominated, invalid, or overly simplified instances.

B.2 Functional Correctness Evaluation

DevBench evaluates code completions in isolated, reproducible environments across six languages, combining standardized prompts with executable assertions to measure functional correctness and practical code-understanding ability.

  • Evaluation infrastructure: Each evaluation instance combines a context prefix, completion, context suffix, and assertions into an executable program.The pipeline applies language-specific safeguards and dependency management.
  • Evaluation infrastructure: Cross-language safeguards use isolated temporary directories, configurable timeouts, cleanup, error reporting, and local dependency installation.Timeouts range from 30–60 seconds according to language compilation requirements.
  • Evaluation infrastructure: The infrastructure produces human-readable reports and structured JSON for analysis across languages and task categories.This supports reproducible functional-correctness comparisons across diverse programming paradigms.
  • Prompt design: A structured prompt standardizes #TODO: You Code Here markers, indentation, replacement behavior, and avoidance of duplicated code.The design targets common formatting-related completion failures.
  • Prompt design: Standardized prompting makes model differences more indicative of code-completion capability than ambiguous prompt interpretation.The format is intended to reflect real-world contextual formatting requirements.

B.4 LLM-judge calibration and failure analysis

LLM-judge scores capture relevance and helpfulness rather than executable correctness, producing systematic divergences from Pass@1 that calibration and failure analysis make explicit.

  • Calibration: 11% of 450 validation completions disagreed by more than 2 points between averaged human scores and Gemini 2.5 Flash judge scores.The disagreements were manually categorized into two failure directions.
  • Failure modes: 31 of 50 judge-overestimation cases involved plausible, syntactically valid completions with subtle logical errors that humans detected but the non-executing judge missed.These errors included boundary conditions, operator precedence, and incorrect variable references.
  • Failure modes: 19 of 50 judge-underestimation cases occurred because the judge penalized terse accurate documentation or unconventional but functionally correct idioms.Thirteen cases involved Code2NL/NL2Code documentation standards, and six involved unconventional idioms.
  • Interpretation: Pass@1 remains necessary because judge scoring lacks execution feedback and therefore cannot directly verify functional correctness.The benchmark reports both metrics to expose their complementary information.
  • Cross-language diagnosis: Cross-language Pass@1 reveals capability gaps that aggregate or Python-only evaluations can conceal.Claude Opus 4.7 ranges from 9.9% to 54.1% across languages, while DeepSeek V4 Pro spans 36.1–45.5%.

C.3 Category correlation analysis

Category correlations provide an empirical diagnostic of relationships among benchmark task types, while examples show that successful completion requires structural and semantic awareness beyond local syntax.

  • Correlation analysis: Table 9 reports pairwise Spearman correlations of Pass@1 across 54 model–language combinations.The analysis uses aggregate category Pass@1 scores.
  • Interpretation: The correlation analysis is an empirical diagnostic of category relationships, not a formal latent-factor model of code-generation ability.The reported relationships should therefore not be interpreted as proof of underlying capabilities.
  • Correlation findings: API Usage correlates weakly with Code Purpose Understanding (ρ = 0.16) and Pattern Matching (ρ = 0.17).These relationships indicate empirical separation among category performance patterns.
  • Correlation findings: Pattern Matching and Code Purpose Understanding correlate more highly (ρ = 0.74), although they can still diverge for individual models.The passage associates both categories with semantic reasoning.
  • Example coverage: The examples span API usage, Code2NL/NL2Code, low-context JavaScript, and C++ syntax completion, illustrating distinct completion demands.The supplied examples include C# slicing, a Java singleton, JavaScript Proxy behavior, and C++ template specialization.
  • Example diagnosis: A C++ syntax example shows that completing an immediate fold-expression body is insufficient when the suffix requires an additional tuple_sig definition.All eight non-GPT-5.5 models missed the required surrounding definition, causing an undeclared-identifier error.

D.2.2 TypeScript Code Purpose: Claude Opus 4.7 vs. DeepSeek V4 Pro

The TypeScript transfer task tests whether models treat interfaces and surrounding behavioral rules as specifications; Claude Opus 4.7 does so more completely than DeepSeek V4 Pro in this example.

  • Behavioral specification: The task requires transfer() to represent failure with status: 'failed' and a separate reason field.The interface defines status and optional reason as distinct fields.
  • DeepSeek V4 Pro: DeepSeek V4 Pro places 'daily_limit_exceeded' in status and omits logging failed transfers.This conflicts with the interface’s two-field pattern and the required audit behavior.
  • Claude Opus 4.7: Claude Opus 4.7 correctly returns distinct status and reason values while implementing account, limit, fee, balance, and audit-log checks.Its completion includes the required transfer branches and successful-transfer bookkeeping.
  • Interpretation: The example indicates that Code Purpose Understanding requires reading type definitions as behavioral specifications, not merely syntactic constraints.The model must connect the interface structure to the expected runtime outcomes.
  • Validation: The hidden tests exercise normal and large transfers, including minimum-fee and balance updates.The examples check a 5-unit fee for a 100-unit transfer and a 6-unit fee for a 300-unit transfer.

D.3.2 Failed execution, similar completion

DevBench illustrates how functionally incorrect completions can remain structurally similar to the intended solution. Small logical differences, especially in edge-case handling, can determine whether hidden tests pass.

  • DeepEqual comparison: A single array-type guard separates GPT-5.5’s correct deepEqual implementation from DeepSeek V4 Pro’s failure on mixed array/object inputs.GPT-5.5 uses an OR-based guard followed by a mixed-type rejection, whereas DeepSeek enters the array branch only when both arguments are arrays.
  • DeepEqual comparison: The missing mixed-type guard lets DeepSeek’s object-comparison path treat an empty array and empty object as equivalent.Because both have zero enumerable keys, deepEqual([], {}) incorrectly returns true.
  • DeepEqual comparison: 5/5 versus 1/5 hidden-test performance results from one logical difference: using || rather than && in the array guard.The example shows that structurally similar completions can diverge sharply on targeted edge cases.
  • Other failure patterns: The slugify example requires coordinated handling of non-alphanumeric characters, camelCase transitions, and letter-digit boundaries.The Java implementation performs these transformations in a character-by-character StringBuilder loop.

E.1 Limitations and future directions

DevBench identifies limitations in metric interpretation, evaluation scope, and benchmark coverage. It also outlines extensions to broader development activities and languages while noting societal and security considerations.

  • Evaluation framework: Similarity, Pass@1, and LLM-judge metrics can diverge, motivating composite metrics that better capture multiple dimensions of code quality.Higher syntactic similarity does not always correlate with functional correctness.
  • Scope and future work: DevBench currently covers six programming languages and six code-completion categories, leaving refactoring, debugging, architecture design, and additional languages for future work.Suggested language extensions include Rust, Go, and Swift.
  • Evaluation assumptions: Results use fill-in-the-middle prompts and specific sampling settings, so scores may differ under chat-style, agentic, or alternative sampling configurations.The paper cautions that completion scores should not be interpreted as general coding ability outside this paradigm.
  • Evaluation assumptions: Functional correctness relies on finite hidden assertions, so passing all tests does not guarantee that a completion is free of latent bugs.The benchmark therefore combines functional, similarity-based, and judge-based assessments.
  • Broader impacts: Benchmark coverage spans only a limited subset of programming languages, which may produce uneven improvements for developers using languages outside the benchmark.The paper also notes potential employment effects from increasingly capable code-completion systems.
  • Broader impacts: More capable code synthesis may facilitate malicious-code generation or vulnerability exploitation, despite the benchmark’s emphasis on security patterns and code quality.The authors make the benchmark and methodology public to support scrutiny, validation, and improvement.
Loading 2601.11895v3…