LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

1 Introduction

• Existing benchmarks do not jointly evaluate functionality preservation, alert reduction, new-alert avoidance, and edit minimality.Linter-guided file refactoring offers a controlled setting for testing whether models resolve context-sensitive code-quality issues beyond fixed rule-based transformations.Source: §1 · p. 1
• LintEval covers 13,061 files across C++, Python, and Java, with 171,662 lint alerts and high-coverage test oracles for functionality preservation.The files come from CodeContests+; the algorithmic tasks support test coverage rather than mimicking all production settings.Source: §1 · p. 1
• The benchmark evaluates model revisions across four dimensions, with quality metrics conditional on functionality preservation.The framework assesses alert reduction, new-alert avoidance, and edit minimality alongside preservation.Source: Figure 1 · p. 2
• Closed-source models prioritize alert reduction at a functionality cost, with regressions on over 31% of samples; DeepSeek-v3.2 reaches 76.6% Reduction@P on C++ versus 63.7% on Java.Refactoring-oriented supervision raises Qwen2.5-Coder-7B from 24.7% to 76.0% Reduction@P.Source: §1 · p. 2
• Refactoring-oriented data formatting substantially narrows the gap between open-source and closed-source models, whereas generation-oriented training does not reliably improve this task.The controlled study raises Qwen2.5-Coder-7B Reduction@P from 24.7% to 76.0%.Source: §1 · p. 2

2 LintEval Construction

• LintEval starts with initially correct source files verified by high-coverage test oracles, then runs static analyzers across C++, Python, and Java.The construction pipeline normalizes heterogeneous lint alerts into a unified taxonomy.Source: §2 · p. 2

2.1 Data Sourcing and Functional Validation

• Each problem has 50–200 test cases, which establish initial correctness and detect functionality-preservation failures after refactoring.The source files and test suites come from CodeContests+.Source: §2.1 · p. 2

2.2 Multi-Language Alert Extraction

• Version-pinned clang-tidy, Pylint, and PMD extract alerts from files that pass the provided test suites.The specified versions are clang-tidy 22.1.0, Pylint 4.0.5, and PMD 7.12.0.Source: §2.2 · p. 3

2.3 Benchmark Instance Construction

• Each benchmark instance combines source code and extracted alerts in a standardized prompt asking the model to reduce alerts while preserving functionality.The full prompt is provided in Appendix A.4.Source: §2.3 · p. 3

2.4 Dataset Characteristics and Statistics

• LintEval contains 13,061 samples across C++ (6,165), Python (4,221), and Java (2,675), averaging 61.0 LOC and 13.1 alerts per sample.Style accounts for 67.3% of alerts, while Error-prone, Performance, and Refactor together account for 31.6%.Source: §2.4 · p. 3, Figure 2 · p. 3

2.5 Taxonomy Normalization

• Alerts are normalized into Error-prone, Performance, Refactor, Style, and Other categories, and matched by rule identifier and sanitized text rather than line number.Excluding line numbers prevents code shifts from interfering with alert matching.Source: §2.5 · p. 3

3.1 Task Definition

• Each instance pairs a functional source file with its lint alerts and passing tests; refactoring must preserve functionality as a prerequisite, not recover it as an objective.The four evaluation dimensions are functionality preservation, lint alert reduction, new alert avoidance, and edit minimality.Source: §3.1 · p. 3

3.2 Evaluation Metrics

• Fixed_i counts resolved original alerts, New_i counts newly introduced alerts, and Lines_i is the absolute sum of added and deleted lines.Alerts are counted by rule identifier and sanitized text after excluding line numbers; quality metrics with suffix @P are computed only over functionally correct instances.Source: §3.2 · pp. 3–4
• Pass@1 is the fraction of refactored files that pass the same test suite as the initial file.Source: §3.2 · p. 4
• Reduction@P measures the net fraction of original alerts eliminated across functionally correct refactorings, using a corpus-level denominator.Corpus-level aggregation prevents files with few initial alerts from biasing the mean.Source: §3.2 · p. 4
• A negative Reduction@P means the model introduces more new alerts than it resolves.Source: §3.2 · p. 4
• New@P measures introduced alerts relative to the baseline alert volume; lower values indicate stronger new-alert avoidance.It shares the corpus-level denominator with Reduction@P.Source: §3.2 · p. 4
• ΔLine@P measures the average number of added and deleted lines per functionally correct refactoring instance.Source: §3.2 · p. 4
• LinePerFix@P measures amortized edit cost per successfully resolved original alert.Source: §3.2 · p. 4

4.2 Main Results

• 76.3% Pass@1 for rule-based baselines contrasts with 25.9% alert reduction; GPT-5.4-mini reaches 90.4% Reduction@P with commercial-model Pass@1 at 67.1%–68.3%.Several open-source models show low or negative reduction, including −1.2% for Qwen2.5-Coder-0.5B; no method jointly optimizes all four dimensions.Source: Table 3 · p. 5, §4.2 · p. 5
• Rule-based baselines make the largest edits (∆Line@P = 71.4), while GPT-5.4-mini has 14.2 LinePerFix@P and Qwen2.5-Coder-0.5B has 9.5 LinePerFix@P.LinePerFix@P is meaningful only alongside positive Reduction@P, not as an independent ranking signal.Source: §4.2 · p. 5
• On failed instances, Qwen2.5-Coder-7B reduction falls from 24.7% to −88.2%; fewer edits can still introduce more alerts while functionality fails.The paper characterizes open-source failures as weak target selection, while commercial models face a functionality-preservation bottleneck after substantial alert reduction.Source: §4.2 · pp. 5–6

4.3 Influencing Factors

• C++ is the most stable language, while 7B Reduction@P is 19.0% on Python and 21.6% on Java; Error-prone and Style alerts pose particular challenges.The paper suggests language differences may reflect rule sets, alert distributions, and dataset composition; Performance alerts tend to map to localized idioms, while Error-prone alerts require semantic judgment.Source: §4.3 · p. 6
• For GPT-5.4-mini, files with more than 10 alerts have 92.1% Reduction@P but 63.9% Pass@1, versus 62.5% and 82.1% on single-alert files.The 18.2-point Pass@1 decline accompanies the broader modification tendency on denser files.Source: §4.3 · p. 7
• Removing explicit alerts improves Pass@1 by 4.9–27.1 points but weakens reduction for alert-conditioned models; three feedback rounds raise conditional Reduction@P while slightly lowering Pass@1.The feedback differences are descriptive because each round’s @P values condition on that round’s passing outputs.Source: §4.3 · p. 7

4.4 Impact of Instruction Tuning

• Refactoring-oriented A+L→A′ supervision raises Qwen2.5-Coder-3B Reduction@P from 9.5% to 85.0%, but produces broader edits and does not ensure functionality preservation or minimality.Its Reduction@P advantage over Base appears in all nine language-size cells, while Pass@1 changes are mixed and the 0.5B model’s Pass@1 falls from 48.8% to 37.8%.Source: §4.4 · p. 8
• Q→A reaches 68.3% Pass@1 but 15.8% Reduction@P on 7B; Q→A′ reaches 47.9% Reduction@P on 7B, while its 3B New@P rises to 22.2%.Q→A has the strongest functionality preservation and lowest New@P among fine-tuned variants; Q→A′ omits the original file and alerts from training input.Source: §4.4 · pp. 8–9

Bug Repair Benchmarks

• Automated repair benchmarks target debugging or code generation rather than code-quality improvement under strict functionality preservation.Examples include Defects4J, DebugBench (Tian et al., 2024), and SWE-bench.Source: §5 · p. 9

Benchmarks Related to Linter-Guided File Refactoring

• RefactorBench (Gautam et al., 2025) evaluates free-form refactoring, while MetaLint and LintLLM (Fang et al., 2025) focus on domain-specific alert detection or localization.These benchmarks do not evaluate active transformation under structured lint alerts.Source: §5 · p. 9

Static Analysis

• Prior static-analysis and LLM-feedback approaches lack a standard benchmark jointly assessing functionality preservation, alert reduction, new-alert avoidance, and edit minimality.LintEval is presented as filling this benchmark gap.Source: §5 · p. 9

6 Conclusion

• LintEval is a multilingual benchmark for linter-guided file refactoring on programs with correct initial functionality.It evaluates under high-coverage test oracles.Source: §6 · p. 9
• Rule-based baselines preserve functionality but reduce few alerts, commercial models reduce more alerts at a functionality cost, and open-source weaknesses are partly mitigated by refactoring-oriented fine-tuning.Joint optimization of functionality preservation, alert reduction, new-alert avoidance, and edit minimality remains open.Source: §6 · p. 9

Limitations

• The controlled CodeContests+ setting omits production cross-file dependencies and multi-turn developer workflows.Functionality is measured dynamically with Pass@1 rather than formal semantic equivalence; fine-tuning is bounded in scale and primarily uses the Qwen2.5-Coder architecture.Source: Limitations · p. 9

• The work follows the ACL Ethics Policy.Source: Ethical Considerations · p. 10
• The benchmark uses publicly available CodeContests+ data under its original open-source licensing, without personal, private, or sensitive user data.Source: Ethical Considerations · p. 10
• Generative AI assistance was limited to language polishing and grammatical refinement; authors produced and verified the experimental design, analysis, and claims.Source: Ethical Considerations · p. 10

• The appendix provides reproducibility details that complement the main text.Source: A Reproducibility Details and… · p. 13
• It includes extended quantitative results and qualitative examples for replication and closer inspection.Source: A Reproducibility Details and… · p. 13
• The paper presents all central claims, methods, and conclusions in the main body.Source: A Reproducibility Details and… · p. 13

• Across three rounds, conditional Reduction@P rose +10.0 points for DeepSeek-v3.2, +9.0 for GPT-5.4-mini, and +1.9 for Qwen2.5-Coder-7B, while Pass@1 fell 1.2–3.8 points.The passing subset can change between rounds, so these observed trajectories are not causal gains on identical successful outputs.Source: Table 9 · p. 13, §A.1 · p. 13
• Wrong answers dominate non-passing revisions, and only 15.7%–22.0% of wrong-answer cases pass at least 90% of tests.Successful compilation is therefore not a reliable proxy for functionality.Source: Table 10 · p. 13, §A.1 · p. 13
• The selected rule-based tool/analyzer pairings fix 77.0% of C++ alerts, 36.8% of Python alerts, and 5.2% of Java alerts.These shares use all original alerts and are not an arithmetic decomposition of conditional Reduction@P.Source: §A.1 · p. 13
• Refactoring-oriented A+L→A′ supervision improves Reduction@P over Base in all nine language–size cells, while Pass@1 changes are mixed.The effect also survives full-parameter Qwen2.5-Coder-3B training and a DeepSeek-Coder-6.7B backbone; for the latter, A+L→A′ raises Pass@1 from 25.3% to 61.2% and Reduction@P from 19.0% to 79.0%.Source: Table 11 · p. 14, §A.1 · p. 14, Table 12 · p. 14
• Alert-category labels group rules by concern, while additional-analyzer audits flag 16.0% of C++ files, 32.0% of Python files, and 49.0% of Java files only under the matched alternative analyzers.The audit establishes analyzer dependence but does not imply that model rankings transfer.Source: §A.1 · p. 14
• Across the diagnostics, only refactoring-oriented transfer improves functionality and alert reduction together; iteration alone slightly lowers Pass@1.Many regressions are semantic rather than merely syntactic, so the authors recommend joint reporting rather than a single cleanup score.Source: §A.1 · p. 14

A.2 Cross-Language Taxonomy

• LintEval groups clang-tidy, Pylint, and PMD outputs into five semantic categories, with Style alerts making up 67.3% of the dataset.Original analyzer and rule identifiers remain available for per-rule inspection and regrouping.Source: §A.2 · p. 15

A.3 Execution Environment

• Analyzer and execution environments are pinned because changing either can alter the passing subset and conditional @P metrics.Runs have a 10-second timeout and 1GB memory limit across languages.Source: §A.3 · p. 15

A.4 Prompt Template; A.5 Model Decoding Configuration

• The prompt supplies code, lint alerts, and explicit requirements for functionality, alert reduction, new-alert avoidance, and minimal edits; greedy decoding makes each model’s output deterministic.This setup focuses on single-attempt conditional editing rather than candidate search or reranking.Source: §A.4 · p. 16, §A.5 · p. 16

A.6 Full Language-wise Performance Results

• C++ is the most stable setting, while open-source models lose ground on Python and Java, where robust refactoring remains the clearest cross-language bottleneck.Qwen2.5-Coder-7B has 67.6% Pass@1 on C++ versus 35.5% on Python; GPT-5.4-mini records the lowest LinePerFix@P on Python (9.8) and Java (13.4).Source: §A.6 · p. 16, Table 16 · p. 17, Table 17 · p. 17

A.7 Category-wise Performance

• No alert category is uniformly easy: semantic Error-prone fixes and numerous local Style alerts require different forms of control.GPT-5.2 reaches 83.0% Reduction@P on Error-prone with 63.3% Pass@1, while GPT-5.4-mini reaches 90.3% Reduction@P on Performance with 66.7% Pass@1.Source: §A.7 · pp. 17–18, Table 18 · p. 18, Table 19 · p. 18

A.8 Alert Count Analysis

• As alert density rises, stronger models trade functionality preservation for greater reduction, while several small models approach zero or negative Reduction@P in the densest bin.GPT-5.4-mini’s Pass@1 falls from 82.1% on single-alert files to 63.9% on files with >10 alerts, while its Reduction@P rises from 62.5% to 92.1%.Source: §A.8 · pp. 18–19, Table 20 · p. 19, Table 21 · p. 19

A.9 SFT Training Details

• On the shared evaluation subset, A+L→A′ fixes 2,453 alerts with 189 new alerts, a 13.0× Fix/New Ratio and the strongest ratio among the reported formats.The raw-count view complements, rather than replaces, percentage metrics; the corpus uses separate problem instances and approximately matches LintEval’s language proportions and alert density.Source: §A.9 · pp. 19–20, Table 22 · p. 20

A.10 Qualitative Examples

• The qualitative examples distinguish minimal valid repairs from over-editing and plausible-looking changes that fail to fix the underlying computation.Together they motivate evaluating alert removal, collateral lint damage, edit efficiency, and functionality rather than treating reduction alone as quality.Source: §A.10 · pp. 20–21