PaperlayerView exact revision on arXiv

Source-linked AI summary

From Code to Courtroom: LLMs as the New Software Judges

Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, David Lo

arXiv:2503.02246v1cs.SE

TL;DR

Comprehensively and scalably evaluating LLM-generated software remains a central challenge. The paper sets out a forward-looking vision and research roadmap for LLM-as-a-Judge in SE, while identifying inconsistent empirical findings in existing studies.

  • Problem

    The paper frames comprehensive and scalable evaluation of LLM-generated software artifacts as a central open question.

    Source: §1 · pp. 1–2

  • Method

    The paper develops a forward-looking vision for LLM-as-a-Judge in SE and proposes a research roadmap for advancing it.

    Source: §1 · p. 3, §5 · p. 9

  • Results

    The paper identifies inconsistent empirical findings; for code summarization, ROUGE, METEOR, and ChrF++ significantly outperformed LLM-as-a-Judge methods in alignment with human judgment.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The roadmap is intended to guide future LLM-as-a-Judge development and adoption in the SE community by 2030 and beyond.

    Source: §5 · p. 9

    The paper flags small-scale experiments and inconsistent empirical findings as limitations of the current evidence.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have been increasingly used to automate SE tasks such as code generation and summarization. However, evaluating the quality of LLM-generated software artifacts remains challenging. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. Given that LLMs are typically trained to align with human judgment and possess strong coding abilities and reasoning skills, they hold promise as cost-effective and scalable surrogates for human evaluators. Nevertheless, LLM-as-a-Judge research in the SE community is still in its early stages, with many breakthroughs needed. This forward-looking SE 2030 paper aims to steer the research community toward advancing LLM-as-a-Judge for evaluating LLMgenerated software artifacts, while also sharing potential research paths to achieve this goal. We provide a literature review of existing SE studies on LLM-as-a-Judge and envision these frameworks as reliable, robust, and scalable human surrogates capable of evaluating software artifacts with consistent, multi-faceted assessments by 2030 and beyond. To validate this vision, we analyze the limitations of current studies, identify key research gaps, and outline a detailed roadmap to guide future developments of LLM-as-a-Judge in software engineering. While not intended to be a definitive guide, our work aims to foster further research and adoption of LLM-as-a-Judge frameworks within the SE community, ultimately improving the effectiveness and scalability of software artifact evaluation methods.

1 INTRODUCTION

The introduction frames scalable, comprehensive evaluation of LLM-generated software as unresolved: human review is costly, and traditional metrics miss important quality dimensions. It presents LLM-as-a-Judge as promising but early-stage, and positions the paper as a review, gap analysis, vision, and roadmap.

1 INTRODUCTION

  • 84% of surveyed SE researchers agreed human evaluation is problematic, citing time constraints, specialized practical knowledge, cost, fatigue, and reduced focus.These challenges can compromise evaluation quality and delay the process.Source: §1 · p. 2
  • Reference-dependent similarity and embedding metrics miss naturalness, usefulness, and best-practice adherence, and studies report misalignment with human judgment [30].These metrics rely heavily on high-quality references typically annotated by human experts.Source: §1 · p. 2
  • LLMs’ coding and reasoning abilities, alignment with expert judgments, and lack of fatigue make them promising cost-effective, scalable surrogates for human evaluators.The paper presents these attributes as reasons LLMs may suit content evaluation.Source: §1 · p. 2
  • LLM-as-a-Judge research in SE remains at an early stage, with only initial progress and substantial work needed to map the field.The introduction points to a few empirical studies and emerging methodologies, such as [1] [65] [68].Source: §1 · p. 2
  • The 2030-and-beyond vision is for reliable, robust, scalable human surrogates that assess diverse software artifacts with consistent, multi-faceted judgments.The roadmap identifies research directions and potential solutions for the SE community.Source: §1 · p. 2
  • The paper reviews 16 primary SE studies, analyzes existing research limitations and gaps, and proposes a detailed roadmap for future exploration.It aims to redirect attention toward advancing LLM-as-a-Judge in SE.Source: §1 · p. 3

2 DEFINITION OF LLM-AS-A-JUDGE

The paper defines LLM-as-a-Judge as an LLM-powered evaluator intended to act as a human-evaluation surrogate, while distinguishing it from broader embedding-based evaluation. Its framework describes evaluation inputs and possible outputs.

Inputs

  • The framework’s inputs include evaluation type, criteria, item, and optional reference information.Evaluation types are point-wise, pair-wise, or list-wise; an item may be one candidate or a set.Source: §2 · p. 3

Outputs

  • Evaluation outputs may include a score or category, candidate ranking, or best-choice selection, with optional explanations and feedback.The paper says explanations can justify an evaluation and feedback can suggest improvements.Source: §2 · p. 3

Definition and scope

  • The authors adopt a stricter definition than Wang et al. [65], excluding embedding-based similarity approaches such as BERTScore [82] and CodeBERTScore [87].They characterize those approaches as LLM-based evaluation rather than LLM-as-a-Judge.Source: §2 · p. 3
  • The proposed judge should assess content independently of a strict reference and support nuanced assessments, justifications, or constructive feedback.The paper excludes embedding-based similarity methods from this stricter definition because they lack these capabilities.Source: §2 · pp. 3–4

3 LITERATURE REVIEW

The literature review maps LLM-as-a-Judge research across software engineering tasks and summarizes how studies evaluate generated code, code changes, documentation, and other artifacts.

3 LITERATURE REVIEW

  • The review maps prior studies to SE tasks and identifies code generation, code change, software documentation summarization, and other SE tasks as application areas.Table 1 provides a structured map of studies to their corresponding SE tasks.Source: §3 · p. 4

3.1 Code Generation

  • For code generation, LLM-as-a-Judge can evaluate code without execution or a required reference and assess qualities such as readability and usefulness.Providing reference code does not generally improve code-evaluation performance [88], while test cases may miss edge cases and reference-based metrics depend on reference quality.Source: §3.1 · p. 4

3.2 Code Change

  • In code-change evaluation, studies use LLMs to judge whether patches resolve static-analysis warnings or vulnerabilities and to predict patch test outcomes.The cited examples are Ahmed et al. [1], Li et al. [35], and Yadavally et al. [71].Source: §3.2 · p. 5

3.3 Sofware Documentation Summarization

  • For software documentation, LLM judges assess code summaries and bug-report titles and summaries, including their language and content quality.CODERPE assigns multiple LLMs roles such as code reviewer, author, editor, and system analyst [68].Source: §3.3 · pp. 5–6

3.4 Other SE Tasks

  • Other explored tasks include programming-related question answering, code translation, and evaluating causal relationships extracted from natural-language requirements.These applications are reported by Zheng et al. [85], Wang et al. [65], and Ahmed et al. [1], respectively.Source: §3.4 · p. 6

4 THE ROAD AHEAD

The roadmap identifies limitations in current LLM-as-a-Judge research and proposes research opportunities spanning evaluation evidence, model expertise, external support, and security.

4.1 More Empirical Evaluation and Benchmarking

  • Existing evaluations often rely on small datasets, including studies measuring human alignment with only a few hundred samples.The review describes this reliance as a limitation of existing SE studies.Source: §4.1 · p. 6
  • Empirical comparisons conflict: Wang et al. [65] found traditional metrics outperformed LLM judges for code summarization, while Wu et al. [68] reported the reverse.The paper attributes variation in findings to differences in dataset selection, prompting strategies, and LLM configurations, and calls for standardized large-scale comparisons.Source: §4.1 · p. 6
  • Thorough empirical investigation of bias in SE LLM judges remains lacking, despite potential bias in software artifacts and data.The paper names position, verbosity, and egocentric biases as examples.Source: §4.1 · p. 7
  • The authors propose diverse, large-scale benchmarks and systematic comparisons with traditional metrics, including investigation of prompting, model configuration, bias, and fairness.The proposed benchmarks should capture real-world SE complexity and multiple dimensions.Source: §4.1 · p. 7

4.2 Beter Judgment From Internal Factors

  • Code-generation ability does not guarantee the domain expertise needed to judge alternative implementations; proposed directions include stronger SE expertise and broader artifact coverage.Examples of broader artifacts include Docker files, UML diagrams, and formal specifications.Source: §4.2 · pp. 7–8
  • The roadmap proposes capturing experts’ tacit procedural knowledge through interviews, think-aloud protocols, or cognitive task analyses, then integrating it into LLM training.Suggested training approaches include reinforcement learning and neuro-symbolic methods.Source: §4.2 · p. 8

4.3 Beter Judgment Through External Factors

  • External support could combine LLM judges with SE analysis tools and human oversight for tasks that cannot be reliably automated.The paper also identifies the optimal LLM-to-human evaluator ratio and judges’ self-confidence assessment as challenges.Source: §4.3 · p. 8

4.4 Addressing The Security Issues of Using LLM-as-a-Judge

  • Adversaries may manipulate artifacts or their surrounding text to alter evaluation outcomes, a threat the paper says remains under-explored in SE.Suggested future work includes adversarial testing and defensive mechanisms.Source: §4.4 · p. 8

5 CONCLUSION

The conclusion presents a forward-looking vision for LLM-as-a-Judge in software engineering and a roadmap of research paths and potential solutions toward adoption by 2030 and beyond.

5 CONCLUSION

  • The paper outlines research paths and potential solutions that the community can pursue to realize its vision for LLM-as-a-Judge by 2030 and beyond.The authors aim to encourage broader participation in this research.Source: §5 · p. 9

Related papers

Loading 2503.02246v1…