PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov, Mikhail Evtikhiev, Ana Terna, Rastislav Rabatin, Timur Kudashev, Timofey Bryksin, Arina Puchkova, Patrik Bartak, Egor Bogomolov, Sergey Titov

arXiv:2608.13566v1cs.LGcs.AIcs.SE

TL;DR

Coding benchmark scores are often treated as evidence of general coding ability, although narrow benchmark evaluation may not distinguish broad improvement from task-specific optimization. Using a Django suite spanning multiple software-engineering modalities, the paper finds that SWE-bench-oriented post-training generally shows little cross-task transfer, motivating diversified evaluation.

  • Problem

    Small coding-benchmark evaluations provide limited evidence about whether reported gains reflect general coding capability or task-specific performance, affecting research and deployment decisions.

  • Method

    The paper builds a single-repository Django benchmark spanning method generation, method completion, and program repair to assess cross-task transfer.

  • Results

    SWE-bench-oriented checkpoints and task-specific fine-tuning generally improve trained or benchmark-aligned tasks but show little consistent transfer to other Django modalities or LiveCodeBench.

  • Takeaways & Limitations

    Evaluating coding models requires complementary benchmark suites, human-in-the-loop studies, and holistic assessments rather than relying on single or limited benchmark scores.

  • Takeaways & Limitations

    Using benchmark suites as proxies for general coding capability depends on a clear and correct taxonomy of the coding skills and behaviors they measure.

Abstract

from arXiv · show

Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.g., SWE-bench and LiveCodeBench) as evidence of broad coding capability, both for research artifacts and user-facing systems. We argue that optimization for these benchmarks leads to measuring task-specific performance, creating a meaning gap between measured scores and claims of general coding ability. We examine this gap with a Django-based case study benchmark suite we create. Evaluating foundation models and checkpoints post-trained on SWE-bench trajectories, we find that benchmark rankings frequently fail to generalize. Post-trained checkpoints show little cross-task transfer, and SWE-bench optimization yields limited or no gains on our tasks or on LiveCodeBench. Similarly, fine-tuning on individual Django modalities fails to transfer. We conclude that a small number of benchmarks is insufficient for evaluating diverse models under benchmark optimization pressure. We encourage the community to use differentiated evaluation - holistic assessment for frontier models, multi-task suites for research, and human-in-the-loop studies for narrow task applications. Finally, we argue for creating a capability taxonomy and sustained benchmark maintenance, rather than one-off benchmark releases. Without reliable evaluation standards, engineers and researchers using LLMs and agents have to rely on insufficient evidence to make research, development, and deployment decisions.

1. Introduction

The coding community often treats performance on a narrow set of benchmarks, especially SWE-bench, as evidence of general coding capability, creating an ambiguous construct-validity signal. The paper argues that SWE-bench-oriented post-training does not consistently transfer across coding tasks and calls for evaluation frameworks that test transfer and distinguish broad improvement from narrow specialization.

  • Evaluation paradigm: SWE-bench leaderboard rankings are frequently interpreted as proxies for general coding capability, despite the benchmark’s specific task focus.The field has converged on a narrow paradigm spanning convenient algorithmic tasks and SWE-bench as a “real-world” coding standard.
  • Evaluation paradigm: Post-training papers often report improvements primarily on SWE-bench or similar benchmarks, providing a limited and ambiguous signal about generalization.The passage notes that such improvements may not reveal changes in underlying capabilities.
  • Evidence of limited transfer: SWE-bench post-training gains do not consistently transfer to other code tasks, including tasks within the same repository.The authors support this interpretation with a Django benchmark covering code editing, generation, and completion, evaluated on community checkpoints and fine-tuned models.
  • Construct validity: Treating a small number of benchmarks as proxies for general coding capability creates a construct-validity problem because claims exceed what the measurements support.The paper argues that this misinterpretation can affect engineers and users of LLMs and coding agents.
  • Recommended evaluation shift: Evaluation frameworks should test transfer across task modalities and distinguish genuine coding improvement from benchmark-specific specialization.The proposed shift aims to prevent equating task performance with general coding capability and to provide a more complete picture.

2. Background

Coding benchmarks comprise self-contained and repository-level task families that measure different capabilities, so scores from either family may not reliably indicate general coding ability. Evaluation constraints and contribution-aligned reporting further make single-benchmark assessment common, motivating direct tests of cross-task transfer.

  • Benchmark families: Self-contained benchmarks evaluate short-form code generation from prompts, commonly reporting pass@k; HumanEval contains 164 hand-crafted Python function-generation problems.LiveCodeBench is cited as a broader example of this benchmark family.
  • Benchmark families: Repository-level benchmarks require producing tested patches within codebases; SWE-bench Verified contains 500 issue-resolution instances from 12 popular Python repositories.The passage notes that issue resolution is important but does not represent the whole coding space.
  • Benchmark families: The two benchmark families measure different capabilities, and performance on self-contained tasks may fail to predict repository-level success or cover general coding capability.Repository-level tasks require codebase navigation, dependency understanding, and targeted edits.
  • Evaluation constraints: Foundation-model reports summarize coding with few representative benchmarks, while post-training papers face page limits, evaluation costs, and harness overhead that make single-benchmark evaluation pragmatic.The GPT-5 model card is given as an example reporting only two coding benchmarks.
  • Evaluation constraints: Post-training studies commonly evaluate benchmarks aligned with their intended contribution, motivating cross-task transfer tests across multiple software-engineering modalities.Repository-level software-engineering works commonly report only SWE-bench, while instruction tuning can improve algorithmic benchmark scores without commensurate behavioral gains.

3. Benchmark measurements and capability claims

Single coding benchmarks provide noisy, partial measurements, yet their narrow scores are often generalized into broad claims of coding capability. The paper frames this meaning gap as consequential and tests it through a multi-modality suite assessing cross-task transfer.

  • Benchmark measurements: A single benchmark can provide only a noisy, partial measurement of latent general coding capability.Large, consistent gaps can still reasonably signal capability differences when models are far apart in overall competence.
  • Capability claims: Narrow benchmark achievements may inflate into broad assertions of general coding capability, creating a systematic meaning gap.The paper distinguishes practical benchmark reporting from how scores are commonly interpreted.
  • Capability claims: Model and checkpoint creators generalize benchmark scores into capability statements, which scientific and technical communication can further amplify.The study focuses on the relationship between benchmark scores and capability statements, while broader audience amplification lies beyond its scope.
  • Benchmark measurements: A high SWE-bench Verified score cannot be standardized into evidence of transfer, task-family capability, format optimization, or contamination.The benchmark remains valuable, but these explanations are not distinguished by the score alone.
  • Benchmark measurements: The study tests whether SWE-bench Verified predicts broader coding capability by measuring performance across multiple in-repository task modalities and cross-task transfer.If higher-scoring models perform worse on the suite, SWE-bench Verified is insufficient as a standalone proxy.

4. Evidence from a Django case study

The Django case study tests whether SWE-bench optimization transfers across three repository-level software-engineering modalities and to LiveCodeBench. Rankings frequently change across tasks, while SWE-bench-oriented checkpoints and task-specific fine-tunes show weak cross-task transfer despite within-task gains.

  • Benchmark design: The suite uses Django 4.0.4 to evaluate method generation, method completion, and program repair within a single repository.Django was selected because it constitutes 46% of SWE-bench Verified, enabling a close-distribution test of transfer within the same codebase.
  • Benchmark design: Evaluation reports pass@1 from greedy decoding at T = 0 on the Django suite and 341 post-October 1, 2024 LiveCodeBench tasks.LiveCodeBench provides a more distant comparison on self-contained algorithmic problems, whereas Django measures repository-level understanding and task execution.
  • Results: Foundation-model rankings can flip across modalities: Qwen-2.5-32B-Coder-Instruct outperforms Qwen3-32B on code completion but has worse SWE-bench scores.This demonstrates that a model’s relative performance on one benchmark may not represent its full capability profile.
  • Results: Across 28 out-of-domain checkpoint–benchmark comparisons, SWE-bench-oriented checkpoints show 18 degradations and 10 improvements.Five of seven checkpoints degrade on most benchmarks, none improve on all four, and only DeepSWE-Preview shows more improvements than degradations.
  • Results: Task-specific fine-tuning improves the trained modality in all six settings, with six improvements and no degradations, but cross-task transfer shows five improvements and 13 degradations.Low scores for Qwen2.5-Coder-7B-Instruct fine-tuned on code completion reflect overfitting to the output format.
  • Results: Within-task evaluation can make task-specific specialization appear to be a capability jump, while excluding output-format failures does not change cross-task transfer between Django benchmarks and LiveCodeBench.The comparison holds despite the two benchmark families having widely different task distributions.

5. Discussion

The experiments support a meaning gap: performance on popular coding benchmarks can fail to predict performance on related tasks, so narrow leaderboards can overstate broad coding capability. The paper therefore advocates differentiated evaluation, sustained benchmark maintenance, and a capability taxonomy.

  • Meaning gap: Benchmark performance can fail to predict related coding-task performance, making broad capability claims based on narrow leaderboards often overstated.The authors describe this as the meaning gap.
  • Meaning gap: Qwen3-32B scores higher on SWE-bench yet performs substantially worse than Qwen2.5-Coder-32B-Instruct models on the code completion benchmark.The authors characterize this limited comparison as suggestive rather than definitive evidence of a rank reversal.
  • Meaning gap: SWE-bench-optimized checkpoints show little to no consistent gains on the Django suite and LiveCodeBench, with some underperforming their base models across multiple benchmarks.Failures can remain hidden because some checkpoints follow instructions and formats reasonably well while still lacking out-of-distribution transfer.
  • Evaluation approaches: Single-score benchmarks support fast iteration, whereas benchmark suites, human-in-the-loop studies, and holistic evaluations offer broader or more realistic evidence with distinct tradeoffs.Holistic assessments and carefully designed human studies are recommended for frontier models; suites are practical for incremental research, and single scores should not be decisive evidence.
  • Benchmark suites and taxonomy: Reliable benchmark suites require a clear coding-skill taxonomy, coverage across tasks and domains, interpretable aggregation, and sustained curation, maintenance, contamination monitoring, and refresh.The proposed taxonomy is substantial research and is not ready for use; human evaluation also faces expense, scalability, consistency, and style-over-substance risks.

6. Alternative views

The paper advocates differentiated evaluation combining holistic, multi-task, and task-specific assessment, while treating scientific testing and qualitative analysis as complementary methods. It rejects the view that benchmark ecosystems automatically self-correct, warning that benchmark gains can mask transfer failures and induce Goodhart-like selection pressure.

  • Evaluation approach: The proposed evaluation approach combines holistic assessment of frontier models, multi-task benchmarks for academic research, and task-specific user studies for narrow practical tasks.These three methods are presented as differentiated evaluation strategies for different model-assessment contexts.
  • Scientific testing: Scientific testing can clarify how algorithms work, but benchmarking performance outside controlled experiments remains necessary and should complement rather than replace it.This position responds to the argument that rigorous benchmarking may be prohibitively expensive.
  • Qualitative evaluation: Qualitative evaluation complements quantitative metrics by revealing capability boundaries and failure patterns hidden by summary statistics.Examples include failures involving long-range dependencies, specification understanding, and semantic correctness despite syntactic validity.
  • Self-correction: Benchmark ecosystems are not necessarily self-correcting: benchmark gains can be interpreted as capability gains while masking transfer failures and creating Goodhart’s law-like selection pressure.SWE-bench and LiveCodeBench addressed limitations associated with HumanEval, but the paper remains less optimistic about automatic correction.

7. Conclusions: what should be done … B.3. Documentation Synthesis

The paper recommends differentiated evaluation rather than a one-size-fits-all benchmark approach, while illustrating taxonomy design and documenting a Django benchmark suite with controlled splits and standardized documentation. The taxonomy remains illustrative, and the suite preserves source comments while synthesizing structured docstrings for uniform evaluation.

  • 7. Conclusions: what should be done: Model evaluation should combine single-score benchmarks, benchmark suites, human-in-the-loop studies, and holistic open-ended evaluations.These approaches target iteration, capability profiling, narrow deployments, and frontier models, respectively.
  • A. Coding capability taxonomy illustration: The sample coding-task taxonomy is an illustration, not a research-grounded framework, and should not be used unchanged.It is intended to help benchmark creators consider construct validity and benchmark limitations.
  • B. Django Benchmark Suite Details: The benchmark-suite construction is described in detail as an expansion of the methodology introduced earlier.The suite details cover repository selection, splitting, and documentation synthesis.
  • B.1. Repository Selection and Leakage Control: The suite uses a frozen Django 4.0.4 snapshot because Django constitutes 46% of SWE-bench Verified, enabling a stronger test of whether SWE-bench performance transfers to Django issue resolution.The snapshot is commit 89807fbde8b7b17d00434bc4695535855e96fe77, dated 11th Apr 2022.
  • B.2. Train-Test Split: The split keeps each class with its ancestors and descendants, assigns non-class code to one file, and excludes tests from training and inference.This prevents contamination through inherited methods, mixins, abstract base classes, and duplicated code.
  • B.2. Train-Test Split: The training dataset is approximately 9× larger than the testing dataset, while similar average method lengths indicate a consistent complexity distribution.Inheritance-chain counts reflect the strategy of keeping related classes together to prevent cross-split contamination.
  • B.3. Documentation Synthesis: Because Django documentation is uneven, the suite uses DeepSeek-R1 to synthesize standardized docstrings for every class method.Each docstring covers purpose and behavior, inputs and constraints, outputs, side effects, and expected failure modes; original comments are merged when applicable.

B.4. Test Suite and Method-Test Alignment · C. Additional evidence for SWE-bench limitations

The benchmark uses Django 4.0.4 tests to identify method-specific correctness checks systematically. Evaluation runs only the mapped tests, and methods without test-detectable failures are excluded.

  • B.4. Test Suite and Method-Test Alignment: The benchmark runs the official Django 4.0.4 test suite in a dedicated test environment.This establishes the testing basis for the benchmark.
  • B.4. Test Suite and Method-Test Alignment: Each target method is first replaced with a stub implementation that returns None.The stub is used to identify tests whose failures depend on the target method.
  • B.4. Test Suite and Method-Test Alignment: All Django tests are run, and failures caused by the stubbed method are recorded.These failures provide the basis for aligning methods with correctness tests.
  • B.4. Test Suite and Method-Test Alignment: Methods with no failures under the return-None stub are excluded because their correctness cannot be verified through testing.The benchmark therefore retains only methods with test-detectable behavior.
  • B.4. Test Suite and Method-Test Alignment: At evaluation time, each generated method is checked only against its mapped tests and must pass all of them to be considered correct.This avoids running the entire Django test suite for every evaluation and focuses on functionality directly related to the target method.

C.1. Error-flow analysis … D.3. Program Repair

The paper’s Django suite exposes diverse failure modes and limited transfer from benchmark-optimized checkpoints, while defining three task-specific evaluations with explicit inputs, outputs, and pass@1 testing protocols.

  • C.1. Error-flow analysis: Across 7 539 examples, fine-tuning reduced the OK rate from 52.4% to 43.1%, with 47.2% of regressions caused by format or parsing errors.The dominant regression was OK → NOT RUN because agent-style output could not be parsed by the harness.
  • C.1. Error-flow analysis: Conceptual errors caused 52.8% of regressions, showing that benchmark evaluation can obscure numerous long-tailed failure modes and cross-task transfer problems.Failure modes included AssertionError and SyntaxError, not only format non-adherence.
  • C.1. Error-flow analysis: SWE-agent-LM-7B’s Method Generation OK rate fell from 52.9% to 5.8%, whereas SWE-agent-LM-32B’s rose from 64.1% to 66.6%.The 7B model’s largest regression was OK → NOT RUN across 113 examples; the 32B model fixed 11 previously failing cases.
  • C.2. Impact of agentic scaffolding: With identical agentic scaffolding, fine-tuned SWE-agent checkpoints outperformed raw models, but removing scaffolding further improved both raw models and SWE-agent-LM-32B in the ten-step setup.The study also tested SWE-agent-LM-32B with a 75-step limit to assess whether ten steps were insufficient.
  • D. Task Descriptions: The Django repository snapshot yields three distinct benchmarks, each paired with an instruction-tuning dataset using the same task format.The suite specifies task-specific input-output formats and evaluation protocols.
  • D.1. Method Generation: Method Generation supplies a specification, signature, synthesized docstring, and source context, then evaluates greedy pass@1 by running mapped Django tests.The expected output is the full method code, including its signature, inside a Python code block.
  • D.2. Method Completion: Method Completion provides approximately half of the target method and surrounding source context, requiring the model to infer behavior and generate the remaining lines.The completed method is validated against mapped Django tests using pass@1.
  • D.3. Program Repair: Program Repair presents a broken method, complete source, and Django stack trace, requiring a corrected implementation evaluated by mapped tests with pass@1 at temperature T = 0.Buggy methods were created from model failures and token or candidate shuffling, and success required all mapped tests to pass.

D.4. Benchmark Verification with Proprietary Models · E. Fine-tuning Setup · E.1. LoRA Configuration

Proprietary-model results indicate that the benchmark suite is solvable, discriminates expected capabilities, and exposes distinct task strengths. The following sections describe practical task-specific fine-tuning and its LoRA configuration.

  • D.4. Benchmark Verification with Proprietary Models: Top-performing proprietary models exceed 80% pass@1 on method generation and method completion, and nearly reach 80% on program repair.These results indicate that all three tasks are well-posed and solvable by contemporary high-capability models.
  • D.4. Benchmark Verification with Proprietary Models: GPT-5 outperforms GPT-4.1 across all tasks, while Claude and Gemini results mostly follow expected within-family capability orderings.Claude Opus 4.5 beats Sonnet 4.5 on two of three tasks, Sonnet beats Haiku 4.5 on all three, and Gemini 3 Pro and Flash beat Gemini 2.5 Pro on code generation and completion.
  • D.4. Benchmark Verification with Proprietary Models: Method generation is generally easier than method completion and program repair across proprietary models.This ordering matches observations for open-weights models and post-trained checkpoints.
  • D.4. Benchmark Verification with Proprietary Models: Frontier models show uneven capability profiles, with GPT-5 stronger than Gemini 3 Pro on code completion but not on code generation or program repair.The differing strengths and weaknesses support evaluating distinct coding tasks rather than assuming uniform capability.
  • D.4. Benchmark Verification with Proprietary Models: The benchmarks pass sanity checks and test connected but distinguishable aspects of LLM coding capabilities.The authors use them diagnostically to examine the meaning gap and risks of cross-task transfer.
  • E. Fine-tuning Setup: Qwen2.5-Coder-7B-Instruct and Qwen2.5-Coder-32B-Instruct are fine-tuned separately on the three task modalities.These bases enable direct comparison because most evaluated external checkpoints build on one of them.
  • E. Fine-tuning Setup: The fine-tuning setup prioritizes a practical, standard cross-task evaluation over exhaustive repository-data exploitation or state-of-the-art hyperparameter optimization.Its purpose is to reveal how task-specific training affects cross-task generalization.
  • E.1. LoRA Configuration: LoRA targets all attention and MLP projection layers to increase adaptation capacity while keeping trainable parameters manageable.Targeted modules include q proj, k proj, v proj, o proj, gate proj, up proj, and down proj.

E.2. Training Hyperparameters

Training uses standard AdamW and cosine-annealing defaults rather than extensive hyperparameter search. The setup evaluates last-epoch checkpoints and is intended to show that single-task fine-tuning can improve in-distribution performance while providing limited cross-task transfer.

  • Training uses AdamW with weight decay 0.01 and a cosine-annealing learning-rate scheduler.
  • The maximum gradient norm is 1.0, and training runs for 3 epochs.
  • Qwen2.5-Coder-7B-Instruct uses 1 example per device with gradient accumulation over 8 steps, yielding an effective batch size of 8.
  • Qwen2.5-Coder-32B-Instruct uses 4 examples per device with gradient accumulation over 8 steps, yielding an effective batch size of 32.
  • The last epoch checkpoint is evaluated in every case, without extensive hyperparameter search or careful tuning.
  • Single-task fine-tuning improves in-distribution performance but produces limited cross-task transfer, supporting concerns about benchmark evaluation’s meaning gap.

F. Prompt Templates

The section specifies task-specific prompt templates that constrain outputs, preserve surrounding Python structure, and provide task context through replaceable placeholders. It also illustrates synthesized Django documentation for methods with and without original docstrings.

  • F. Prompt Templates: Each task combines a system prompt defining the assistant’s role and output constraints with a user template containing inference-time placeholders such as {file content}.Curly-braced placeholders are replaced with actual values during inference.
  • F.1. Method Generation Prompts System prompt.: Method-generation prompts require only the target method in one python code block, preserving its exact declaration and using existing imports or nested helpers.They also prohibit the class wrapper, new imports, top-level definitions, and explanatory prose.
  • F.2. Method Completion Prompts System prompt.: Method-completion prompts generate only the continuation of the open method, preserving indentation and prior lines so the resulting file remains valid.The continuation must not repeat the existing body or declaration and should avoid new top-level code.
  • F.3. Program Repair Prompts (With Test Output): Program-repair prompts identify the single method most likely responsible for failing tests and return its repaired implementation with the original signature unchanged.Test output guides the repair, and the response excludes wrappers, imports, other code, and prose.
  • F.4. Examples of Synthesized Documentation: The documentation examples cover both methods without original developer documentation and methods whose original docstrings were incorporated.The section presents these as examples of documentation synthesized by DeepSeek-R1 for Django methods.
  • Example 1: Article.save: For Article.save, generated documentation states that the method sets a new instance’s created date, forwards *args and **kwargs, and saves the model.It also identifies updating created as a side effect for new instances.
  • Example 2: CustomErrorMessage.clean: For CustomErrorMessage.clean, generated documentation explains field-specific or global ValidationError outcomes triggered by forbidden name1 values.The documented cases include FORBIDDEN_VALUE, FORBIDDEN_VALUE2, and GLOBAL_ERROR.
  • F.4.2. EXAMPLES WITH ORIGINAL DJANGO DOCUMENTATION Example 1: StartsWithRelation.field: With original Django documentation, generated examples describe StartsWithRelation.field as exposing remote_field for bidirectional descriptor functionality.The accessor is documented as taking no inputs and returning the remote_field reference.
Loading 2608.13566v1…