Large Language Model Critics for Execution-Free Evaluation of Code Changes

• Complex, interdependent repository edits can introduce syntactic, semantic, or logical errors across multiple locations, prompting repeated build-and-fix cycles.Edits may span multiple functions, classes, or files.Source: §1 · p. 1
• Existing build-status and log metrics require test execution, offer sparse evidence about patch quality, and can demand substantial manual interpretation.Build status alone does not show functional correctness or performance across conditions.Source: §1 · p. 1
• LLM-based intermediate, step-level proxies are intended to assess task progress and compare agentic workflows without relying on compilation or execution.These proxies are particularly relevant when partial failures leave traditional metrics unavailable to an agent.Source: §1 · p. 2
• The framework uses the gold test patch’s unseen tests to assess each candidate patch against each test, then aggregates predictions into build-status estimates.The approach assumes access to a solution patch, including its gold test patch, as a reference.Source: §1 · p. 2
• 91.6% F1 for test-oracle prediction and 82.1% for build-outcome prediction show the test-aware critics’ performance on SWE-bench-Lite.For factory-code-droid patches, the framework also outperforms reference-free metrics by 65.5%–75.5% and other reference-aware metrics by 38.9%–68.3% on build outcomes.Source: §1 · p. 2

2.1 A PRIMER ON USING LLMS FOR CODE EVALUATION

• Reference-aware code metrics compare generated code with a ground-truth implementation, but repository tasks can admit non-unique edits that limit their direct use for agent-generated patches.The section distinguishes these from reference-free metrics, which assess code against the task intent.Source: §2.1 · p. 3

2.1.1 PROBLEM FORMULATION

• The evaluation score assesses candidate patches against a task and assumes access to a ground-truth patch, with the assumption relaxed in Section 4.3.The section evaluates both success discrimination and progress monotonicity.Source: §2.1.1 · p. 3

2.2 LLM CRITICS FOR EXECUTION-FREE PATCH EVALUATION

• Test-centric critics use unseen tests from the gold test patch to predict each test’s pass or fail, enabling fine-grained tracking without execution.The framework aggregates predictions into build status: all tests predicted to pass means success, while any predicted failure means build failure.Source: §2.2 · p. 4

2.2.1 UNCERTAINTY QUANTIFICATION WITH BLACK-BOX CONFIDENCE MEASURES

• The critics elicit a 0–100 confidence estimate for each unseen-test pass/fail prediction, with inference parameters assumed unknown.The approach combines verbalized confidence with chain-of-thought prompting.Source: §2.2.1 · p. 4

3.1 DATASET

• SWE-bench-Lite contains 300 Python task instances from 11 projects, with gold change patches limited to three edits in one file.Each instance pairs a GitHub issue with a pull request and its validating tests.Source: §3.1 · p. 4

3.2 MODELS AND AGENTIC PATCHES

• The main experiments use claude-3-opus to assess patches from four agentic workflows, while the framework is described as workflow- and LLM-agnostic.The evaluated workflows are factory-code-droid, sweagent-gpt4, Gru, and codestory-aide-mixed.Source: §3.2 · pp. 4–5
• Figure 3 relates verbalized confidence in isolated, test-aware critic predictions to correctness and test character length as a complexity proxy.The highlighted region marks the confidence threshold.Source: Figure 3 · p. 5
• The project provides an open-source library for reuse with other agentic workflows or benchmarks.The authors also state that the library supports reproducibility of their results.Source: §3.2 · p. 5

4.1 LLM CRITICS FOR MICRO-EVALUATION OF PATCHES

• Context-enhanced patches perform best on test-oracle prediction, beating other baselines by 7.4% to 12.7%, while source-code patches identify more failing tests.Context-enhanced patches help identify around 25% more passing tests; the class imbalance also makes a frequency-weighted random baseline strong on this task.Source: §4.1 · p. 5
• For complex tests longer than 50 characters, treating critic confidence of 65% or less as failure improves specificity by 24.1%.The authors report that verbalized confidence is more often high for correct than incorrect predictions, particularly for lower-complexity tests.Source: §4.1 · p. 6

4.2.1 BASELINES

• The reference-aware baselines include random aggregation, patch similarity, change-aware critics using gold code changes, and holistic test-aware critics using all reference tests together.The test-centric method uses unseen gold tests as its reference, while change-aware critics compare candidate and gold change patches.Source: §4.2.1 · p. 6

4.2.2 EXPERIMENTAL RESULTS

• 82.1% F1 is achieved for build-status prediction by aggregating isolated, test-aware critic predictions, outperforming reference-aware LLM critics by 38.9% to 68.2%.The aggregation predicts failure if any unseen test is predicted to fail.Source: §4.2.2 · p. 6, §4.2 · p. 6, Table 2 · p. 7
• Compared with holistic test-aware critics, aggregating isolated critics improves F1 by 52.2% with source code and 68.2% with context-enhanced patches.The authors note that test complexity may partly explain why holistic test-aware critics compare more favorably with change-aware critics.Source: §4.2.2 · p. 7

4.3 REFERENCE HELPS, BUT IS NOT ALWAYS AVAILABLE!

• The test-centric framework outperforms the two reference-free baselines by 72.5% and 65.5%, respectively; adding hints yields negligible improvement over the problem statement alone.The authors report inconsistent trends from adding patch context to these baselines.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 COMPARING AGENTIC PATCHES WITH TEST-AWARE LLM CRITICS

• For 68.1% of task instances, workflow rankings based on predicted test pass rates perfectly match true rankings, with Spearman ρ = 1.The compared workflows show pass-rate patterns consistent with their reported resolution rates; the edit-distance baseline has higher counts at ρ < 0.Source: §4.4 · pp. 8–9, Figure 4 · p. 9

4.5 HARNESSING DIFFERENT LLMS IN TEST-AWARE EVALUATION

• Claude-3-opus achieves an 81.8% F1-score for build-outcome prediction, while claude-3-opus and claude-3-5-sonnet have comparable test-oracle performance.Both models outperform claude-3-sonnet on test-oracle prediction by 6.8%.Source: §4.5 · p. 9, Table 5 · p. 9

Benchmarks for Repository-Level Coding Tasks

• Repository-level coding benchmarks include SWE-bench for real-world patch-generation tasks, while SWE-bench is limited to Python and SWE-bench-java extends it to Java.The paper evaluates its critics on Python task instances and describes extension to other languages and repositories as possible.Source: §5 · p. 10

Static and change-based evaluation

• Static software-quality measures and repository differencing can characterize code or change impact, but do not account for patch correctness.Source: §5 · p. 10

Automated Evaluation of Large Language Models in Coding Tasks

• Reference-matching metrics such as BLEU, ROUGE, CodeBLEU, and CodeBERT are limited to source code and do not capture program semantics well or correlate efficiently with human judgment.Source: §5 · p. 10
• LLM evaluation proxies include reference-free approaches and reference-aware approaches using developer-written code or tests, but prior approaches are not extensible to patches and remain limited in efficiency.Source: §5 · p. 10

6 CONCLUSION

• With gold tests as reference, the critics predict editing-location executability at 91.5% accuracy and build status in 82.1% of SWE-bench instances.They outperform other reference-free and reference-aware LLM critics by 38.9% to 72.5%.Source: §6 · p. 10
• Future work includes evaluating other programming languages, relaxing reference awareness when incorporating execution scores into agents, and using LLM inputs to design better tests.Source: §6 · p. 10