PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-IF: Aligning Code Evaluation with Human Preference

Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun

arXiv:2510.07315v2cs.CLcs.AIcs.LGcs.SE

TL;DR

Code evaluation commonly measures functional correctness while overlooking non-functional instructions that users apply when judging code. The paper introduces VERICODE and SWE-IF to measure instruction following alongside functionality, finding that their combination better tracks human preference than either metric alone.

  • Problem

    Code evaluation remains centered on pass@k and functional correctness, leaving non-functional expectations such as project conventions, clear documentation, targeted edits, and preserving prior intent unmeasured.

    Source: §1 · p. 1

  • Method

    The authors create VERICODE, a 30-instruction taxonomy with deterministic verifiers, and augment established benchmarks with instruction constraints and functional tests in SWE-IF.

    Source: §1 · pp. 1–2

  • Results

    A composite of functional correctness and instruction following correlates better with human ratings than either metric alone, and IF is the key differentiator among advanced models on real-world programming tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that code evaluation and training should account for both functional correctness and non-functional instruction following to better align with human preference.

    Source: §6 · p. 8

    The evaluation is limited to Python function-level tasks and deterministic verifiers, though the framework is designed to generalize beyond these choices.

    Source: E. Limitations · p. 36

Abstract

from arXiv · show

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models' code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.

1. Introduction

The introduction argues that vibe checks include non-functional expectations alongside correctness, while code evaluation largely measures only functionality. It introduces VERICODE and SWE-IF, then reports that instruction following both challenges models and helps explain human preference.

1. Introduction

  • Vibe coding involves iterative natural-language code refinement, and its vibe check includes whether solutions read cleanly, preserve intent, and remain functional—not just whether they pass tests.The figure highlights coding style and logic patterns as factors in human preference; its example is simplified and not part of the taxonomy.Source: §1 · p. 1, Figure 1 · p. 2
  • The paper presents instruction following as an under-measured component of code evaluation and proposes VERICODE and SWE-IF as tools for measuring it alongside functionality.The authors describe this as a path toward evaluating and developing models against a more human-aligned notion of code quality.Source: §1 · pp. 1–2
  • VERICODE pairs 30 curated code instructions with deterministic verifiers, while SWE-IF combines those checks with functional unit tests across single-turn and multi-turn settings.The benchmarks augment BigCodeBench and LiveCodeBench, and the evaluation covers 31 LLMs.Source: §1 · p. 2
  • Under five instructions, average pass@1 drops by 5.85% and 6.61% on the two augmented benchmarks, despite the instructions not targeting functionality.The introduction presents this as functional regression from adding non-functional instructions.Source: §1 · p. 2
  • Under five instructions, the strongest model reaches only 46.75% and 40.95% success rates on Big-SWE-IF and Live-SWE-IF, respectively.Models also follow mid-position instructions less reliably than those at the beginning or end.Source: §1 · p. 2
  • Single-turn generation better preserves functionality but follows fewer instructions than multi-turn editing, which achieves higher IF with more functional regressions.The comparison is made under the same tasks.Source: §1 · p. 2
  • A composite of functional correctness and instruction following correlates better with coding-subset LMArena ratings than either metric alone, with IF the key differentiator among advanced models.The comparison concerns real-world programming tasks.Source: §1 · p. 2

2. VERICODE: A Taxonomy of Verifiable Code Instructions

VERICODE is a curated taxonomy of practical, verifiable code instructions, designed for deterministic evaluation and scalable extension. Its construction filters real-world rules and pairs the resulting instructions with automated checks.

2.1. Design Principles

  • VERICODE is designed for objective, scalable verification, practice grounding, broad non-functional coverage, and constraints that remain challenging for advanced models.The covered aspects include style, logic patterns, documentation, error handling, and library constraints.Source: §2.1 · pp. 2–3

2. VERICODE: A Taxonomy of Verifiable Code Instructions

  • The taxonomy is built through candidate sourcing, staged filtering, expert review, and verifier implementation.Its current instantiation focuses on Python function-level tasks, while the framework is intended to generalize beyond those choices.Source: §2.2 · p. 3

2.2. Taxonomy Construction Process

  • The authors source candidate rules mainly from Ruff and add response-level documentation instructions that static code analysis cannot assess.Ruff aggregates more than 800 rules from popular tools.Source: §2.2 · p. 3
  • Filtering consolidates overlapping rules, favors broadly applicable instructions, and removes constraints with above-90% success and no pass@1 degradation on the screening setup.Gemini 2.5 Flash is tested on BigCodeBench-Hard; borderline cases receive manual review.Source: §2.2 · p. 3
  • Experts review the final instruction set, and verifiers use linter-backed checks or deterministic AST and regular-expression tests where direct rules are unavailable.Each verifier returns a binary pass/fail result through a common interface.Source: §2.2 · p. 3

2.3. Resulting VERICODE Taxonomy

  • The resulting VERICODE taxonomy contains 30 instructions across five categories, organized hierarchically from the overall concept to individual instructions.The categories cover coding style and conventions, logic and code patterns, documentation, error handling, and library or API constraints.Source: §2.3 · p. 3
  • Each instruction specifies its category, description, prompts for both interaction settings, configurable parameters, and verification code.The full instruction set is provided in Appendix A.2.Source: §2.3 · p. 3
  • Parameters such as line length and maximum branches let one core instruction produce variants at different difficulty levels, expanding 30 instructions into hundreds of checkable constraints.The framework is designed to extend beyond its current Python instantiation using standard linters.Source: §2.3 · p. 3

3. SWE-IF: A New Testbed for Code Evaluation

SWE-IF augments established code benchmarks with verifiable instructions to assess both functional correctness and instruction following. Its augmentation and evaluation protocols cover diverse coding tasks and simulate single-turn generation and multi-turn editing.

3. SWE-IF: A New Testbed for Code Evaluation

  • SWE-IF extends standard code benchmarks with explicit, verifiable instructions and evaluates both functional correctness and instruction following.Source: §3 · p. 3

3.1. Benchmark Augmentation

  • Big-SWE-IF adapts BigCodeBench for real-world programming, while Live-SWE-IF adapts LiveCodeBench for algorithmic and contest problems.Together, these benchmarks cover diverse coding challenges.Source: §3.1 · p. 3
  • An LLM selector keeps query-relevant, non-conflicting instructions, then assigns parameters that undergo rule-based validation.Invalid parameter keys are removed and invalid values revert to predefined defaults.Source: §3.1 · p. 4

3.2. Evaluation Protocol

  • Single-turn generation presents all instructions at once, while multi-turn editing reveals them sequentially and evaluates the final code.Both settings are designed to mirror real-world usage.Source: §3.2 · p. 4, Figure 2 · p. 5
  • Evaluation measures functional regression against the original-prompt score and instruction following through per-instruction averages and all-instructions task scores.A task is a benchmark problem together with its selected instruction set.Source: §3.2 · p. 4

4. Experiments

Across the experiments, adding non-functional instructions challenges functionality and instruction following, while the best alignment with human preference comes from combining both measures. Results also differ by interaction mode, instruction position, and programming-task type.

4.1. Experimental Setup

  • The evaluation covers 31 LLMs from 10 model families and augments BigCodeBench and LiveCodeBench with five VeriCode instructions per instance.The benchmarks contain 1,140 and 1,055 instances, respectively, yielding over 10,000 instruction-level evaluations.Source: §4.1 · p. 5

4.2. Results for Functionality

  • With five instructions, multi-turn regression exceeds 5% for nearly every model on Big-SWE-IF and frequently on Live-SWE-IF.On Live-SWE-IF, o4 mini and Kimi K2 exceed 10% regression in more than half of test configurations; Table 2 reports regression relative to original-query pass@1.Source: §4.2 · p. 5, Table 2 · p. 6
  • Single-turn generation preserves functionality better, with Big-SWE-IF average regression rising from 2.48% to 5.76%, versus 3.18% to 9.31% for multi-turn.The values span one to five instructions; the single-turn advantage grows with instruction count.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3. Results for Instruction Following

  • With three or more instructions, most advanced models score below 50 on task-level IF across both benchmarks; on Live-SWE-IF, five of seven leading models score below 30 in single-turn.Task-level success requires satisfying all constraints simultaneously; Table 3 reports task-level IF scores.Source: §4.3 · p. 6, Table 3 · p. 7
  • Multi-turn editing has higher task-level IF than single-turn generation, by 3%–4.5% on Big-SWE-IF and around 8% on Live-SWE-IF.The paper suggests iterative targeted revisions may help satisfy newly introduced instructions, while single-turn integrates all constraints in one pass.Source: §4.3 · p. 6

4.4. Instruction Position Analysis

  • Instruction following varies by position: Big-SWE-IF has a U-shaped pattern, with single-turn favoring the first instruction and multi-turn the last.The U-shape does not generalize to Live-SWE-IF, but the first-position preference for single-turn and last-position preference for multi-turn remain.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Correlating with Human Preference

  • A mixture of instruction following and functionality correlates best with human preference across both benchmarks.The optimal blend outperforms either metric alone; Big-SWE-IF's Pearson optimum weights IF at 40%, and its Spearman optimum at 70%.Source: §4.5 · p. 7, Figure 5 · p. 8

Which single factor users value depends on the coding

  • For real-world programming, pure IF correlates over 0.1 points higher with human preference than pure functionality on Spearman correlation, while Live-SWE-IF favors pure functionality.The paper associates the difference with users valuing non-functional instruction adherence in practical coding and functional correctness in competitive programming.Source: Which single factor users value… · p. 8

5. Related Work

Code instruction-following research has focused on synthetic constraints, while practical coding instructions remain sparsely studied and often lack verifiability. SWE-IF addresses this gap with verifiable instructions and analyzes functional correctness alongside instruction following.

5. Related Work

  • Practical coding instructions concern software-development needs such as logic patterns, style, and library usage, but prior benchmarks often lack verifiability.Prior work has used DiffBLEU comparisons or LLM and human judgments, which the authors describe as unreliable and hard to scale.Source: §5 · p. 8
  • General instruction-following research commonly uses synthetic constraints and deterministic checkers or LLM judges, while coding instructions reflect practical software development.The paper introduces a taxonomy of verifiable code instructions, each paired with a verifier, for scalable evaluation and training.Source: §5 · p. 8
  • Existing code evaluation centers on unit-test functionality, while non-functional requirements such as runtime efficiency, maintainability, and security receive less attention.The paper positions SWE-IF as a way to analyze the trade-off between functionality and instruction following and examine how both relate to human preference.Source: §5 · p. 8

6. Conclusion

The paper presents VERICODE and SWE-IF as tools for evaluating non-functional code instructions alongside functionality, and reports that their combination predicts human preference better than either alone. The appendices document verification, benchmark construction, instruction selection, prompts, and evaluated models.

6. Conclusion

  • Across 31 LLMs, a composite of functional correctness and instruction following predicts human preference substantially better than either metric alone.The conclusion calls for moving beyond pass@k and optimizing both functional and non-functional qualities in future coding research.Source: §6 · p. 8

A.1. Verification Code with Ruff

  • 27 of VERICODE’s 30 verifiers use Python linter Ruff, and the appendix presents a helper function for running Ruff checks.Figure 6 shows the helper function’s implementation.Source: §A.1 · p. 14, Figure 6 · p. 14

A.2. Case Studies from VERICODE

  • The appendix presents full versions of five VERICODE instructions from Table 1 in Figures 7–11.The examples cover style, logic, documentation, error handling, and library constraints.Source: §A.2 · p. 15, Figure 7 · p. 15, Figure 8 · p. 16

B.1. Instruction Category Distributions

  • Coding Logic, Coding Style, and Documentation are the most frequent instruction categories across both augmented benchmarks.Live-SWE-IF has more Coding Logic instructions (42.3% vs. 35.9%), while Big-SWE-IF has more Error Management (6.3% vs. 0.9%) and Library Constraint instructions (2.2% vs. 0.1%).Source: §B.1 · p. 20, Figure 12 · p. 20

B.2. Accuracy of LLM-based Instruction Selection

  • A manual review of 500 selected instructions from 100 sampled instances found 100% validity for both relevance and non-conflict.Two authors independently reviewed 50 instances from each benchmark.Source: §B.2 · p. 20
  • Parameter selection, rather than selection logic, is the main challenge observed; invalid template values occur at approximately 1% to 3%.The pipeline addresses this issue with rule-based validation.Source: §B.2 · p. 20

B.3. Evaluation Prompts

  • The appendix provides the Big-SWE-IF system instruction and Live-SWE-IF prompts, with “complete, runnable code” changed to “complete Python functions” for Live-SWE-IF.BigCodeBench’s original instruct prompt is used without an additional evaluation prompt.Source: §B.3 · p. 21, Figure 13 · p. 21, Figure 14 · p. 22

C.1. Details of Evaluated Models

  • Table 4 lists the 31 evaluated models, their LMArena designations, and Elo ratings from September 18, 2025; Live-SWE-IF analysis excludes models with error rates above 10%.After exclusions, the Live-SWE-IF analysis covers 24 LLMs; persistent provider or token-limit failures are recorded as errors.Source: §C.1 · p. 23, Table 4 · p. 23

C.2. Detailed Results for Functionality

The appendix provides detailed functionality, instruction-following, and instruction-position results for both benchmarks. It also reports that human preference correlates best with a mixture of instruction following and functionality.

C.2. Detailed Results for Functionality

  • Tables 5 and 6 report functionality on both benchmarks, with pass@1 for the original query and functional regression rates relative to that baseline.Lower regression is better, and negative values indicate improvement.Source: §C.2 · p. 24, Table 5 · p. 24, Table 6 · p. 25

C.3. Detailed Results for Instruction Following

  • Tables 7–10 provide instruction-level and task-level instruction-following scores for Big-SWE-IF and Live-SWE-IF.The table captions specify that higher scores are better.Source: §C.3 · p. 26, Table 7 · p. 26, Table 8 · p. 27

D.1. Instruction Position Analysis

  • Tables 11 and 12 compare instruction-level scores at positions 1–5 for single-turn instruction lists and multi-turn instruction sequences.The captions define a position by list order in single-turn generation and by the turn an instruction is introduced in multi-turn editing.Source: §D.1 · p. 30, Table 11 · p. 30, Table 12 · p. 31

D.2. Correlation Analysis

  • Figure 15 shows that the peak correlation with LMArena coding Elo consistently occurs for a mixture of instruction following and functionality across the tested settings.The analysis varies the IF weight α, correlation type, and style-control setting.Source: Figure 15 · p. 32

D.3. Error Analysis via Category-wise IF Profiles

Category-wise results show different instruction-following strengths across models: error handling and library use are handled well, while documentation remains a weak point. The reported profiles are largely consistent across the two benchmarks.

D.3. Error Analysis via Category-wise IF Profiles

  • Error Handling achieves near-perfect instruction following across models, and Library use on Big-SWE-IF is also close to 100%.The authors associate these strengths with constraints that have explicit, localized code signatures, such as guard clauses or mandated library use.Source: §D.3 · p. 33, Table 13 · p. 33
  • Documentation is the lowest-performing category across both benchmarks, and remains error-prone even for strong models.The category-wise results are summarized in Tables 13 and 14.Source: §D.3 · p. 33, Table 13 · p. 33, Table 14 · p. 33
  • Category strengths vary by model: Gemini 2.5 Pro leads Big-SWE-IF Coding Style, GPT 5 leads Logic & Code Patterns, and o4 mini leads documentation on both benchmarks.Claude 4 Opus has a more balanced profile without an obvious weak category, consistent with its strong overall instruction following.Source: §D.3 · p. 33
  • The category patterns are largely consistent between real-world Big-SWE-IF and algorithmic Live-SWE-IF, and the authors suggest using model-specific strengths to guide selection.Examples include Gemini for style-strict projects and GPT-5 for logic-heavy tasks.Source: §D.3 · p. 33

D.4. Trending Analysis across Model Families

Across model families, increasing the number of injected instructions worsens functional regression and lowers instruction following in both interaction modes. Multi-turn editing typically improves instruction following, while single-turn generation tends to preserve functionality better.

D.4. Trending Analysis across Model Families

  • The family-level analysis plots functional regression and instruction-following scores against injected-instruction counts for both interaction modes.Source: §D.4 · p. 34, Figure 16 · p. 34, Figure 17 · p. 34
  • Across model families, more injected instructions consistently increase functional regression and decrease instruction following.Figures 16 and 17 show these trajectories for single-turn generation and multi-turn editing.Source: Figure 16 · p. 34, Figure 17 · p. 34, §D.4 · p. 34
  • Multi-turn editing typically yields higher instruction following, while single-turn generation tends to preserve functionality better.Source: §D.4 · p. 34

D.5. Statistical Significance and Result Stability

Repeated runs show stable pass@1 regression trends, with small standard deviations across instruction counts and interaction modes. Multi-turn editing often has lower variance than single-turn generation.

D.5. Statistical Significance and Result Stability

  • The stability analysis uses five repeated runs on Gemini models for Big-SWE-IF and reports mean pass@1 and standard deviation.The broader evaluation covers over 2,000 instances, 10,000 instructions, and 31 models at temperatures 0.0 and 0.2.Source: §D.5 · p. 35, Table 15 · p. 35
  • Across instruction counts and both interaction modes, standard deviations are consistently small, mostly below 0.35, suggesting regression trends are unlikely to reflect run-to-run randomness.Source: §D.5 · p. 35
  • Multi-turn editing often has lower variance than single-turn generation, consistent with the richer context from prior rounds.Source: §D.5 · p. 35

E. Limitations

The evaluation focuses on Python function-level tasks and deterministic verifiers, while the framework is designed to generalize beyond those choices. The limitations discuss language choice, task scope, and what deterministic checks can capture.

E. Limitations

  • The evaluation uses Python function-level tasks with deterministic verifiers, while the framework is designed to generalize beyond these choices.Source: E. Limitations · p. 36

E.1. Experimental Language Choice

  • Python benchmarks support comparison with prior work, while the framework’s rule-sourcing and verifier pipeline is designed to transfer across languages.The paper gives ESLint for JavaScript and Checkstyle for Java as examples.Source: §E.1 · p. 36

E.2. Why We Focus on Function-level Benchmarks

  • Function-level tasks remain challenging for instruction following, match snippet-heavy human-preference data, and can be extended through the plug-in method to repository benchmarks.SWE-Bench is given as an example of a repository-level setting.Source: §E.2 · p. 36

E.3. Scope of Deterministic Verifiers

  • Deterministic checks provide reproducible pass/fail evaluation but omit open-ended requirements such as API design, maintainability, and algorithmic elegance.The paper identifies LLM-as-a-judge verification as a direction and notes rare static-analysis false positives or negatives in complex cases; checkers can be replaced.Source: §E.3 · p. 36
Loading 2510.07315v2…