LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead

1 Introduction

• LLM-generated artifacts are proliferating, and their complexity and open-endedness make evaluation a substantial challenge.The introduction frames the central question as how to evaluate their quality comprehensively and scalably.Source: §1 · p. 2
• 84% of surveyed SE researchers said human evaluation is problematic, citing time constraints, specialized knowledge, and cost.Evaluators may also experience fatigue and reduced concentration.Source: §1 · p. 2
• Traditional metrics offer scalable, lower-cost evaluation but can depend on high-quality references and misalign with human judgment on nuanced qualities.The cited examples include naturalness, usefulness, and adherence to best practices.Source: §1 · p. 2
• LLMs are presented as candidates for automated evaluation because they combine coding abilities and human-like reasoning with RLHF-informed alignment and no fatigue.The paper notes these traits as reasons LLMs may sustain evaluation over prolonged work.Source: §1 · p. 2
• The paper envisions reliable, robust, scalable judges for consistent, multi-faceted artifact evaluation and offers a roadmap toward that vision.The roadmap identifies research directions and potential solutions for the SE community.Source: §1 · p. 3
• The paper contributes a systematic review of 42 primary SE studies and an analysis of current limitations and research gaps.The review summarizes applications of LLM-as-a-Judge in software engineering.Source: §1 · p. 3

2 Definition of LLM-as-a-Judge

• Evaluation type T may be point-wise, pair-wise, or list-wise, determining whether one candidate is assessed, two are compared, or multiple candidates are ranked.Point-wise evaluation often produces a score or category; pair-wise evaluation compares relative quality.Source: §2 · p. 3
• The evaluation setup can specify criteria C and an item X, which may be one candidate or a set, with an optional reference R.A coding-task reference could be an implementation that fulfills the task requirements.Source: §2 · p. 3
• The result Y may assign candidates numerical or categorical grades, rank them, or select the best candidate.These are the three listed forms of evaluation result.Source: §2 · p. 3
• An evaluation may optionally include an explanation justifying the judgment or feedback suggesting improvements to the evaluated item.Source: §2 · p. 3
• The paper frames LLM-as-a-Judge as a human-evaluation surrogate for nuanced assessments, potentially including justifications or feedback.The defined function is an LLM-powered mechanism for evaluating software artifacts against criteria.Source: §2 · pp. 3–4

3 Review Methodology

• The review methodology combines a defined scope and selection criteria with literature searching, snowballing, and quality assessment.The section introduces these procedures before reporting the selected publications.Source: §3 · p. 4

3.1 Survey Scope

• The review covers LLM-as-a-Judge studies on software engineering tasks and excludes surveys, non-SE work, studies without experiments, duplicates, and papers under five pages.Included papers had to be in English, have accessible full text, and apply LLM-as-a-Judge to at least one SE task.Source: §3.1 · p. 4

3.2 Paper Source

• DBLP was the primary source, chosen for broad coverage of academic literature and arXiv preprints.The authors describe DBLP as broadly subsuming IEEE Xplore and the ACM Digital Library, while indexing recent preprints.Source: §3.2 · p. 4

3.3 Search String Formulation

• The authors derived search terms from seed papers, refined them with synonyms, and combined LLM-as-a-Judge and software-engineering terms in 42 DBLP searches.The final query used Agent/LLM and Judg*/Evaluat* terms alongside software-related terms.Source: §3.3 · pp. 4–5

3.4 Snowballing

• Because the initial DBLP search was title-only, backward and forward snowballing continued until no additional relevant studies were found.The procedure ran August 7–12, 2025, using references and Google Scholar citations.Source: §3.4 · p. 5

3.5 Quality Assessment

• Three quality criteria assess contribution, methodological clarity and reproducibility, and experimental rigor; three authors applied the checklist independently and resolved discrepancies by discussion.The criteria examine contribution, replicability of methods, and the appropriateness of datasets, baselines, and metrics.Source: §3.5 · p. 5

3.6 Results

• 26 publications were cataloged by August 2025, following 15 in 2024; the review selected 42 relevant publications overall.Figure 1 shows the annual distribution; formal acceptance year was used for published works, which may differ from initial preprint posting.Source: §3.6 · pp. 5–6, Figure 1 · p. 6

• The review covers requirements engineering, coding assistance, software maintenance, and quality assurance.Source: §4 · p. 6
• The four categories emerged from the reviewed literature rather than being imposed in advance.Source: §4 · p. 6
• The categories correspond to fundamental stages of the software development lifecycle.Source: §4 · p. 6

4.1 Requirements Engineering

• The reviewed applications assess requirement artifacts ranging from user stories to technical system specifications before development begins.Source: §4.1 · p. 6

4.1.1 Requirements Documents.

• Lubos et al. [55] use Llama 2 (70B) to assess requirements against ISO 29148’s nine quality characteristics, while Ahmed et al. [1] classify sentences for causal links.Lubos et al. report identifying most quality flaws and providing reliable explanations; Ahmed et al. use a binary judgment.Source: §4.1.1 · p. 7

4.1.2 System Specifications.

• Reinpold et al. [70] report that GPT-4o and Claude 3.5 Sonnet can assess whether a system specification fulfills its corresponding requirements.Source: §4.1.2 · p. 7

4.1.3 User Stories.

• Quattrocchi et al. [69] assess user-story semantic quality using five criteria, including feature specificity, rationale clarity, problem orientation, language clarity, and internal consistency.Source: §4.1.3 · p. 8

4.2.1 Generated Code.

• Execution tests do not necessarily indicate good code quality, since test cases may not cover every edge; constructing tests can be challenging for certain tasks, and reference-based metrics frequently misalign with humans [15] [83].Source: §4.2.1 · p. 8
• LLM-as-a-Judge can assess code without requiring execution or reference code and evaluate intrinsic qualities such as readability and usefulness, with actionable feedback.These properties distinguish it from execution-based and reference-based metrics.Source: §4.2.1 · p. 8
• Generated-code evaluation criteria fall into functionality and quality, covering behavior and error handling alongside intrinsic attributes such as readability and best-practice adherence.The section also lists complexity, helpfulness, stylistic consistency, reference similarity, and minor anomalies among quality criteria.Source: §4.2.1 · pp. 8, 10
• Judging methods include structured prompts such as ICE-Score [127] and multi-step approaches that decompose code or evaluation, including HuCoSC [100] and MCTS-Judge [90].Structured prompts specify task, criteria, scoring, and output; decomposition can use an abstract syntax tree or pseudocode.Source: §4.2.1 · p. 10
• Adding code execution raised agreement with ground truth from below 42% to approximately 72% in Findeis et al.’s evaluation of Python solutions [20].The study tested whether judges could distinguish correct from incorrect solutions.Source: §4.2.1 · p. 10
• Multiple-judge methods combine specialist evaluations or consensus, and AIME [63] theoretically establishes benefits over a single evaluator.Other approaches give judges tool access, while SWE-Judge [123] aggregates scores from a dynamically assembled team.Source: §4.2.1 · pp. 10–11
• CODE-DITING [104] distilled a 671B reasoning model into 1.5B and 7B judges, with the 7B version surpassing GPT-4o.The framework uses reasoning-focused data distillation to produce compact judges.Source: §4.2.1 · p. 11
• Kim et al. used an LLM-as-a-Judge to close the multi-shot success (Pass@5) gap without requiring ground-truth, while selecting the top-ranked sample significantly improved Pass@1.Vo et al. [91] used judge feedback to align CodeLlama-7B with coding preferences, improving coding-style adherence and functional correctness. Zhou et al. [125] found generic, unstructured critiques largely ineffective for code refinement.Source: §4.2.1 · p. 11

4.2.2 Code Summary.

• In subjective code-summary evaluation, LLM judges’ assessments align more closely with human judgment than traditional metrics [1] [11] [97] [123]; Crupi et al. report moderate-to-high agreement for GPT-4.CODERPE [97] instructs an LLM judge to adopt personas such as code reviewer, author, or system analyst to assess summaries from multiple perspectives.Source: §4.2.2 · pp. 11–12

4.2.3 Translated Code.

• LLM-as-a-Judge correlated 0.81 with human scores on CodeTransOcean, versus 0.34 for ChrF++.The passages attribute this advantage to judging functional and semantic equivalence, including for syntactically different but correct translations.Source: §4.2.3 · p. 12

4.2.4 Answers to Software Question.

• GPT-4 achieved over 80% agreement with human preferences on complex queries, comparable to inter-human agreement.For Databricks documentation answers, weighting Critical, Supporting, and Trivial Facts improved human alignment by an average of 6%.Source: §4.2.4 · p. 12

4.3.1 Bug Report.

• GPT-4o matched or exceeded human accuracy on bug-report titles and summaries, while its performance stayed stable as human evaluators fatigued [40].The study presents LLM-as-a-Judge as a scalable, potentially more reliable option for large-scale QA evaluation.Source: §4.3.1 · p. 12

4.3.2 Tests.

• Compiler-test-suite judges assess correctness from compiler outputs and execution results for OpenMP and OpenACC tests [79].This is an agent-based approach to evaluating tests for parallel-programming compiler implementations.Source: §4.3.2 · p. 12
• RUM assigns syntax and naming checks to rule-based systems and reserves LLM assessment for subjective test readability and coverage, achieving high correlation with human-expert scores [89].Its evaluation covers test case documents, test script code, and test reports.Source: §4.3.2 · p. 13

4.3.3 Vulnerability Explanation.

• LLM judges can review vulnerability explanations for logical consistency, detail, and clarity rather than making the ultimate security judgment [92].This meta-evaluation role is distinguished from direct, high-stakes vulnerability judgments [122] [124].Source: §4.3.3 · p. 13

4.4.1 Commit Message.

• With Chain-of-Thought and few-shot prompting, GPT-4 achieved human-expert-comparable commit-message evaluation and outperformed reference metrics such as BLEU and ROUGE [113].The evaluation considered whether messages explained what a change does and why it was made.Source: §4.4.1 · p. 13
• The commit-message evaluator also showed acceptable reproducibility and robustness [113].Source: §4.4.1 · p. 13

4.4.2 Code Patch.

• Ahmed et al. [1] use LLMs to judge static-analyzer-warning fixes, while an execution-free approach predicts whether patches pass unseen tests.The static-warning task requires deep reasoning.Source: §4.4.2 · pp. 13–14
• CleanVul uses LLM heuristics to identify security-related patches and build a high-quality dataset, improving downstream vulnerability detectors’ correctness and generalizability [46].Source: §4.4.2 · p. 14
• SWE-Judge combines direct assessment, semantic-equivalence checking, and test generation to better align patch evaluations with human judgments [123].Source: §4.4.2 · p. 14
• Reviewed code-patch work extends beyond correctness assessment to dataset curation and evaluation protocols for program repair and vulnerability mitigation.Source: §4.4.2 · p. 14

5 The Road Ahead

• The foundational phase calls for expert-annotated benchmarks with nuanced criteria and distribution-aware evaluation, plus systematic cross-task evaluations of judge designs and cognitive biases.Benchmarks should cover qualities such as readability and helpfulness and preserve the distribution of expert opinions; empirical studies should examine prompting strategies and LLM configurations.Source: §5 · p. 14
• The first long-term track would strengthen judges’ software-engineering expertise through domain-specific training, capture of experts’ procedural knowledge, and multimodal assessment of artifacts such as UML diagrams and GUI mockups.Source: §5 · pp. 14–15
• A second track would connect judges to external software-engineering tools, including formal verification, and refine them through a continuous feedback loop with human experts.Source: §5 · p. 15
• A third track would test vulnerabilities through advanced adversarial testing and harden judges against sophisticated semantics-preserving manipulations.Source: §5 · p. 15

5.1 Developing Larger-Scale Benchmarks and Rigorous Evaluation Frameworks

• Current studies lack large-scale human-annotated benchmarks for nuanced qualities, while small datasets limit generalizability and make a gold standard difficult to define.Existing benchmarks for code correctness do not adequately assess qualities such as helpfulness, readability, or alignment with human judgment.Source: §5.1 · pp. 15–16
• Benchmarks often erase legitimate disagreement by aggregating subjective ratings into one gold label, while traditional agreement metrics assume a single answer and interchangeable raters.Such metrics also struggle with systematic, reproducible LLM errors and offer no clear threshold for deciding whether a judge can replace a human evaluator.Source: §5.1 · pp. 16–17
• Empirical findings conflict across studies, and evidence on cognitive bias remains limited, motivating standardized large-scale comparisons and systematic bias analysis.For code summarization, Wang et al. [87] found traditional metrics outperformed LLM judges, whereas Wu et al. [97] reported the opposite.Source: §5.1 · p. 17
• Future benchmarks should combine expert-annotated samples across varied SE tasks, difficulty levels, programming topics, languages, and artifact types, while reflecting evolving human preferences.The proposed annotations should capture nuanced quality dimensions beyond code correctness, such as readability, helpfulness, and maintainability.Source: §5.1 · p. 17
• Evaluation protocols should treat the full distribution of expert judgments as ground truth rather than collapsing subjective ratings into a single gold label.This approach accounts for indeterminate criteria such as readability and makes evaluator preferences explicit.Source: §5.1 · pp. 17–18
• Protocols should compare LLM and human judgment distributions and report performance separately on high- and low-human-agreement subsets.Distribution-aware metrics, including Earth Mover’s Distance, can assess whether LLM judgments mirror the range of expert assessments.Source: §5.1 · p. 18
• Comprehensive empirical evaluations should compare judges across diverse SE tasks and systematically test how design choices affect judgment.The proposed scope extends beyond code generation to software design documents, API usability, and system requirements.Source: §5.1 · p. 18
• Empirical studies should probe auxiliary-information biases, including reference-driven solution fixation, and produce task-specific model and prompting recommendations with reproducible baselines.The roadmap also calls for investigating whether bias severity varies with task complexity or model type.Source: §5.1 · p. 18

5.2 Better Judgment From Internal Intelligence

• LLM judges need stronger SE expertise because coding capability does not ensure reliable evaluation of complex domains or alternative implementations.The roadmap proposes richer, higher-quality SE training data and broader assessment of artifacts such as Docker files, UML diagrams, and formal specifications.Source: §5.2 · p. 19
• Multimodal assessment is an opportunity because text- and code-focused judges miss visual SE artifacts, and textual descriptions can lose critical structural information.The passage introduces multimodal LLMs [68] [96] as capable of supporting this assessment.Source: §5.2 · p. 19
• Researchers can capture the tacit and procedural knowledge that human experts develop over years of experience using structured interviews, think-aloud protocols, and cognitive task analyses to systematically extract procedural knowledge.Source: §5.2 · p. 19
• MLLM judges can support holistic, cross-modal evaluations that better mimic expert workflows and verify relationships across requirements, designs, and code.These capabilities extend beyond text-only evaluation.Source: §5.2 · p. 19

5.3 Augmenting Judgment with External Insights

• The roadmap proposes LLM judges that orchestrate diverse SE tools, extending early integrations with linters and execution environments to formal verification, model checking, and performance profiling.Profilers could assess execution time and memory usage; tool augmentation remains in its early stages and is not yet widespread.Source: §5.3 · p. 20
• For evaluations that cannot be reliably automated, low-confidence judgments should trigger human review while expert corrections feed a fine-tuning loop.This division lets LLMs handle routine checks and directs human expertise toward complex evaluations, progressively aligning model reasoning with experts.Source: §5.3 · p. 20

5.4 Building Robust Judges

• Because semantics-preserving manipulations can evade standard defenses, the roadmap calls for rigorous adversarial testing, including optimization-based attacks and persistent backdoors.These stealthy attacks can exploit poisoned training data or subtly alter artifacts to elicit biased judgments.Source: §5.4 · pp. 20–21
• The roadmap pairs dynamic external defenses with stronger intrinsic robustness, including anomaly detection, data sanitization, adversarial fine-tuning, and self-correction.Together, these measures aim to counter sophisticated attacks and reduce reliance on external defenses.Source: §5.4 · p. 21

6.1 Calibrated Expert Judgment

• The LLM Judge is envisioned as a cost-effective expert surrogate offering nuanced judgments on maintainability, code smells, and security, with real-time feedback for iterative refinement.The paper contrasts this vision with current models’ limited precision for reliable vulnerability detection.Source: §6.1 · p. 21

6.2 Preference-Aware, Multi-Perspective Feedback

• Distinct evaluation personas would emulate different expert preferences, enabling multi-perspective feedback tailored to project needs rather than a single generic gold standard.The paper says this directly addresses Rating Indeterminacy and Evaluator Preferences.Source: §6.2 · p. 22

6.3 Cross-Modal Holistic Assessment

• By 2030, a multimodal judge is envisioned to assess diverse artifact coherence, such as UML/code discrepancies, while evaluating software quality, security, performance, and internal consistency.The paper describes UML class diagrams and corresponding code as an example of cross-modal verification that current text-only models cannot perform.Source: §6.3 · p. 22

7 Conclusion

• The paper conducts a systematic review of 42 primary studies and maps LLM-as-a-Judge applications across the software development lifecycle.Source: §7 · p. 22
• The review identifies limitations and associated opportunities, presenting a future vision for adoption of LLM-as-a-Judge in the SE community.Source: §7 · p. 22
• The proposed roadmap outlines specific paths and potential solutions, with short- and long-term goals for the research community to pursue.The paper aims to encourage broader participation in the research journey.Source: §7 · p. 22