LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

• The paradigm addresses evaluation needs created by LLMs’ generalization, adaptability, and highly generative outputs across unseen tasks and diverse domains.Standardized measures such as BLEU and ROUGE often fail to capture important qualities of natural language generation.Source: §1 · p. 2
• LLMs-as-judges replace fixed metrics and human evaluators with task-adaptive, interpretable, scalable assessments for complex and open-ended AI outputs.Traditional metrics can miss fluency, coherence, creativity, robustness, fairness, and interpretability, while human evaluation is time-consuming and resource-intensive.Source: §1 · p. 2
• LLM judges can produce biased or inconsistent assessments because of prompt templates, inherited biases, and difficulty adapting criteria across tasks and domains.These challenges affect the fairness and reliability of their evaluations.Source: §1 · p. 3
• The survey systematically analyzes LLMs-as-judges through five perspectives: Functionality, Methodology, Application, Meta-evaluation, and Limitation.It aims to clarify current research, key challenges, and guidelines for future development.Source: §1 · p. 3
• The paper contributes a comprehensive survey, a structured analysis of how and why LLMs serve as evaluators, and a forward-looking discussion of challenges and research opportunities.It also provides an open-source resource repository to support collaboration and best practices.Source: §1 · p. 3

2 PRELIMINARIES

• LLMs-as-judges evaluate output quality, relevance, and effectiveness using LLM knowledge and contextual understanding across diverse NLP and machine-learning tasks.The paradigm formalizes evaluation scenarios within a unified perspective and presents an overview of the system.Source: §2 · pp. 3–4

2.1 Evaluation Function 𝐸

• The evaluation function E accepts type T, criteria C, item X, and optional references R, producing result Y, explanation E, and feedback F.Different input-output configurations correspond to distinct evaluation methods and objectives.Source: §2 · p. 5, §2.2 · p. 6, §2.3 · p. 8
• Single-LLM systems are simple and scalable, multi-LLM systems broaden assessment but cost more, and human-AI systems improve nuance and reliability while reducing scalability.Human collaboration can mitigate model bias, but coordination, consistency, cost, and time remain challenges.Source: §2.1 · pp. 5–6

2.2.1 Evaluation Type T.

• Evaluation type T includes pointwise scoring, pairwise comparison, and listwise ranking, each suited to different assessment goals.Pointwise evaluation is simple, pairwise evaluation emphasizes relative preferences, and listwise evaluation supports holistic ranking across candidates.Source: §2.2.1 · p. 6
• Pointwise, pairwise, and listwise judgments cannot always be reliably transformed into one another because LLM judges may show bias and violate transitivity.For example, separate scores can conflict with direct comparisons, and z_i > z_j plus z_j > z_k does not guarantee z_i > z_k.Source: §2.2.1 · p. 6

2.2.2 Evaluation Criteria C.

• Evaluation criteria C may assess linguistic quality, content accuracy, and task-specific attributes such as informativeness, completeness, diversity, structure, and logical clarity.Well-structured examples can further align evaluations with user expectations for complex or ambiguous tasks.Source: §2.2.2 · p. 7

2.2.3 Evaluation References R.

• Reference-based evaluation enables benchmarking against established data but depends on reference quality, whereas reference-free evaluation is flexible but difficult when judges lack domain knowledge.References are optional, allowing evaluation to use either external standards or intrinsic quality and source-context alignment.Source: §2.2.3 · p. 8

2.3 Evaluation Output

• Evaluation outputs include result Y, explanation E, and feedback F; explanations improve transparency and human alignment, while actionable feedback requires greater model capability.Results can be scores, rankings, labels, or qualitative assessments, and the three outputs may be combined according to task requirements.Source: §2.3 · pp. 8–9

3.1 Performance Evaluation

• LLMs-as-judges evaluate both individual responses and overall model capabilities, providing task-specific or holistic insights into quality, relevance, reasoning, and other skills.Response evaluation targets attributes such as accuracy, relevance, coherence, and fluency, while model evaluation examines capabilities across tasks and domains.Source: §3 · p. 9, §3.1 · p. 9, §3.1.1 · p. 9

3.1.1 Responses Evaluation.

• LLM judges assess text responses across dimensions such as content, grammar, relevance, and appropriateness, but paper review remains unreliable when tasks require deep reasoning or knowledge.Response evaluation can also analyze retrieval decisions, document relevance, faithfulness, and explanation quality, including in RAG and recommendation settings.Source: §3.1.1 · p. 10

3.1.2 Model Evaluation.

• Dynamic, interactive evaluation frameworks use LLMs to generate questions, conduct multi-turn assessments, and rank models, addressing data leakage and evaluation bias in static benchmarks.Static benchmark averages remain common because they are simple and comparable, but newer systems provide more granular assessments through model-to-model interaction.Source: §3.1.2 · p. 10

3.2.1 Reward Modeling During Training.

• During training, LLM judges provide reward signals that align outputs with human-defined criteria, enable self-optimization, and supply dense intermediate feedback to mitigate sparse rewards.SRLMs [274], OAIF [74], and RLAIF make LLMs their own reward models, while RELC uses a Critic LM for step-level evaluation.Source: §3.2.1 · p. 11
• Reward-modeling systems address bias and conflicting objectives through cross-iteration consistency constraints and task-specific Mixed Judges, improving reward reliability and alignment.Using the same LLM as policy and reward model can accumulate bias and preference-data noise; CREAM [239] and CGPO [257] introduce corrective mechanisms.Source: §3.2.1 · p. 11

3.2.2 Acting as Verifier During Inference.

• During inference, LLM judges verify and select among candidate outputs, guiding best-of-N sampling, self-consistency, tree or graph search, and stepwise reasoning validation.These methods use comparative or self-assessment signals to improve stability, path selection, logical consistency, factual grounding, and creative diversity [102] [209] [235] [15] [137] [250] [105].Source: §3.2.2 · pp. 11–12

3.2.3 Feedback for Refinement.

• LLM-generated feedback enables iterative response and reasoning refinement without additional supervision, but intrinsic self-correction can fail without reliable external validation.SELF-REFINE [155] and SELF-DEBUGGING [32] improve outputs through feedback, while REFINER [176] and multi-agent review [261] target reasoning; Self-Correct [93] [223] and Valmeekam et al. [224] identify limitations.Source: §3.2.3 · p. 12

3.3 Data Construction

• For data construction, LLM judges automate annotation and synthesize diverse, preference-aligned, instructional, and reasoning-focused data, reducing human effort and reliance on sensitive real-world data.GPT-4 achieved 83.6% annotation accuracy versus 81.5% for MTurk workers [85], while ChatGPT outperformed crowdsourced workers in several classification tasks [71] and ChatGPT-4 matched or improved human accuracy and reliability with comparable or lower bias [217].Source: §3.3 · p. 12, §3.3.1 · pp. 12–13, §3.3.2 · p. 13

4 METHODOLOGY

• The survey categorizes LLM-as-judge methodology into Single-LLM, Multi-LLM, and Human-AI Collaboration systems.Figure 5 presents this three-way methodological overview.Source: §4 · p. 14, Figure 5 · p. 14

4.1 Single-LLM System

• Single-LLM systems rely on one model and comprise Prompt Engineering, Tuning, and Post-processing, whose effectiveness depends on model capabilities and input-processing strategies.Source: §4.1 · p. 14

4.1.1 Prompt-based.

• Prompt engineering can reduce extensive training needs by eliciting clear, relevant, consistent, and goal-aligned judgments through structured task-specific instructions [189].Source: §4.1.1 · p. 14
• Prompt-based methods adapt evaluation through in-context examples, stepwise decomposition, explanation-rating, enhanced definitions, external knowledge, and multi-turn optimization.ICL methods such as GPTScore [68], LLM-EVAL [141], and TALEC [280] support flexible, training-free or customizable evaluation, while stepwise methods structure complex judgments; ICL example selection can introduce bias [62] [78] [296].Source: §4.1.1 · pp. 14–16
• Definition augmentation enriches evaluation criteria, principles, roles, or retrieved knowledge to produce more context-sensitive judgments and reduce hallucinations.AUTOCALIBRATE [146] aligns evaluators with human preferences, while BiasAlert [61] combines human knowledge with model reasoning to detect and mitigate social bias.Source: §4.1.1 · pp. 15–16

4.1.2 Tuning-based.

• Tuning adapts pretrained LLMs to specialized judgment tasks using score-based, preference-based, synthetic, or combined evaluation data.Fine-tuning can align judgments with human decision-making, transfer scoring to lighter models, support self-improvement, and improve generalization across held-out tasks; LLaVA-Critic [253] provides reliable multimodal scores and reward signals.Source: §4.1.2 · pp. 16–18

4.1.3 Post-processing.

• Post-processing refines judgments by calibrating probabilities, consolidating multiple evaluations, reprocessing text, and transforming tasks between open-ended and multiple-choice formats.These procedures target bias, subjectivity, reliability, ranking, labeling, and score calibration, including multi-round aggregation, divide-and-conquer critique consolidation, and response-adapted references.Source: §4.1.3 · pp. 18–19

4.2 Multi-LLM System

• Multi-LLM evaluation combines inter-model communication or independent output aggregation to mitigate bias, exploit complementary strengths, and produce more reliable judgments.These systems can also refine decision-making precision and support nuanced understanding of complex judgments.Source: §4.2 · p. 19

4.2.1 Communication.

• Communication enables cooperation through shared solutions, critiques, revisions, and specialized agent roles, but similar models may reinforce one another’s biases through groupthink.Examples include WideDeep, peer-review-style collaboration, and ABSEval’s synthesis, critique, execution, and commonsense agents.Source: §4.2.1 · pp. 19–20
• Competitive communication uses centralized orchestration or decentralized direct interaction, with debates and committees evaluating candidates’ outputs and, in some frameworks, domain knowledge and inconsistencies.Centralized systems include Auto-Arena, MORE, and SAMRE; decentralized systems include automated benchmarking with debate-based assessment.Source: §4.2.1 · p. 20
• ChatEval improves evaluation performance over single-agent methods, while PRD better aligns evaluations with human judgments and supports fairer capability ranking.ChatEval uses diverse role prompts and communication strategies; PRD uses peer rank and discussion to address self-enhancement and positional bias.Source: §4.2.1 · p. 20

4.2.2 Aggregation.

• Aggregation independently combines model judgments through majority voting, weighted scores, confidence-based selection, specialized criteria, Bayesian calibration, preference graphs, or LLM-based fusion.Fusion-Eval integrates specialized assistant evaluators to improve correlation between evaluation scores and human judgments.Source: §4.2.2 · pp. 20–21
• Cascade frameworks use weaker models first and stronger models only when needed, reducing evaluation cost while preserving agreement with human judgments or compensating for weaker judge models.Cascaded Selective Evaluation targets high human agreement with cheaper models, while CascadedEval combines proprietary models to address fine-tuned judge limitations.Source: §4.2.2 · p. 21

Human-AI Collaboration System

• Human oversight bridges automated LLM judgments and high-stakes requirements by serving as final decision-making or verification and refinement of model outputs.This hybrid approach aims to improve reliability, ethical alignment, and continuous model improvement through feedback loops.Source: §4.3 · p. 21
• During evaluation, COEVAL[131] has LLMs generate criteria and evaluations that human evaluators review and refine for open-ended natural-language tasks.Wang et al. [230] address positional bias through multiple evidence, balanced position, and human-in-the-loop calibration, while EvalGen[192] iteratively incorporates human feedback to mitigate criteria drift.Source: §4.3 · pp. 21–22
• After LLM evaluation, EvaluLLM[170] and a university-course LLM-TA workflow use human intervention to verify, adjust, and finalize results for trust, bias control, fairness, and academic consistency.In the course setting, the teaching team reviews LLM-generated feedback and finalizes assignment evaluations.Source: §4.3 · p. 22

5.1 General

• 0.962 system-level Kendall-Tau and 0.744 turn-level Spearman correlations show Fusion-Eval’s alignment with human judgments for summary and dialogue evaluation, respectively [198].Fusion-Eval integrates multiple assistant evaluators, while ACTIVE-CRITIC dynamically infers tasks and criteria and optimizes prompts for story-generation evaluation [256].Source: §5.1 · p. 23

5.2 Multimodal

• Multimodal judges evaluate images, audio, and video despite heterogeneous representations, using specialized models, unified frameworks, and self-evaluation or self-reward methods [253] [24] [117] [46] [299].In autonomous-driving evaluation, text-only LLM judges aligned more closely with human preferences than LVLMs [29].Source: §5.2 · p. 23

5.3 Medical

• Medical judges assess generation, clinical-note consistency, question-answering quality, counseling, and information filtering, requiring domain terminology, diverse data interpretation, accuracy, and ethical compliance [17] [114] [126] [95].They evaluate completeness, conciseness, attribution, medical adherence, terminology accuracy, clarity, relevance, empathy, and emotional-needs recognition.Source: §5.3 · pp. 23–24

5.4 Legal

• Legal judges evaluate Law LLM performance and retrieved-document relevance using jurisdiction-aware reasoning, criteria such as accuracy, completeness, and clarity, and scalable retrieval assessment [275] [185] [154].Interpretability, transparency, bias, and fairness are essential because legal evaluations affect judicial fairness [128].Source: §5.4 · p. 24

5.5 Financial

• Financial judges support investment-risk assessment, credit scoring, and benchmark construction, processing structured and unstructured data while addressing real-time adaptation and multifactor market complexity [248].FinMA improves risk-assessment accuracy and efficiency, while automated benchmarks support updating and compliance checks.Source: §5.5 · pp. 24–25

5.6 Education

• Educational judges grade assignments, essays, mathematical reasoning, debates, and academic papers, but paper-reviewing studies find they cannot yet replace humans for highly technical or innovative work [35] [228].ReasonEval assesses logical errors and unnecessary reasoning steps beyond final-answer correctness, while LLM TA generates scores and feedback using prompt-defined criteria [246] [35].Source: §5.6 · pp. 25–26

5.7 Information Retrieval

• Information-retrieval judges automate relevance judgments, pairwise and reasoning-intensive reranking, explanation assessment, and RAG evaluation across relevance, accuracy, coverage, fluency, and coherence [183] [180] [166] [188].These methods aim to reduce dependence on costly manual annotations and improve scalability.Source: §5.7 · p. 26

5.8 Others

• Other applications extend LLM judging to software-code and bug-report evaluation, biomedical query-focused summarization, democratic discussions, and social-agent intelligence assessment [175] [243] [115] [298].Across domains, judges provide scalable feedback but remain limited by domain knowledge, reasoning ability, and diverse evaluation criteria.Source: §5.8.1 · p. 26, §5.8.3 · p. 27, §5.8.2 · p. 27

• Meta-evaluation assesses LLM judges’ reliability, consistency, and validity, making it crucial for determining whether their evaluations can be trusted.Source: §6 · p. 27

6.1 Benchmarks

• Benchmark-based meta-evaluation uses datasets and task-specific assessment methods to examine LLM-based evaluation quality across diverse applications.Source: §6 · p. 27

6.2 Metrics

• Metric-based meta-evaluation develops measures tailored to different evaluation tasks and objectives while considering the advantages and limitations of existing approaches.Source: §6 · p. 27

6.1 Benchmarks

• The survey collects 40 benchmarks covering language understanding, factual accuracy, coherence, creativity, fairness, and other evaluation criteria across application domains.These benchmarks are intended to compare LLM judges against human preferences, which are treated as the gold standard for quality and reliability.Source: §6.1 · p. 27

6.1.1 Code Generation

• Code-generation benchmarks range from HumanEval’s 164 algorithmic tasks to SWEBench’s 2,294 complex software tasks, while DevAI, CrossCodeEval, and CodeUltraFeedback target iterative development, cross-language translation, and programming-preference alignment.CodeUltraFeedback contains 10,000 instructions with four responses from 14 LLMs, scored by GPT-3.5 for preferences including readability, efficiency, and adherence to user specifications.Source: §6.1.1 · pp. 28–30

6.1.2 Machine Translation

• Machine-translation benchmarks evaluate accuracy, fluency, contextual relevance, and multidimensional quality across language pairs, with WMT, Literary Translation Comparisons, and MQM emphasizing increasingly broad evaluation contexts.WMT provides human-annotated datasets but focuses mainly on high-resource languages; Literary Translation Comparisons covers 18 language pairs, while MQM assesses accuracy, fluency, terminology, style, and locale.Source: §6.1.2 · p. 30

6.1.3 Text Summarization / 6.1.4 Dialogue Generation

• Summarization benchmarks measure coherence, consistency, fluency, relevance, and factuality, while dialogue benchmarks assess naturalness, coherence, engagement, groundedness, understandability, overall quality, and toxicity control.SummEval contains outputs from 16 models over 100 news articles, annotated by five crowd workers and three experts; FRANK focuses on categorized factual errors, and OpinsummEval evaluates opinion summaries across four dimensions.Source: §6.1.3 · pp. 30–31, §6.1.4 · p. 31

6.1.5 Automatic Story Generation

• Story-generation benchmarks assess narrative quality through relevance, coherence, engagement, surprise, character development, and related criteria using human ratings, preference rankings, and personalized evaluations.HANNA includes 1,056 stories from 10 systems and 96 prompts; StoryER combines a 100k story-ranking dataset with 46,000 aspect-rating and reasoning entries, while PERSER includes 7,000 paired stories in Per-DOC.Source: §6.1.5 · pp. 31–32

6.1.6 Values Alignment

• Values-alignment benchmarks test whether LLM behavior reflects human ethical standards, including harmlessness, usefulness, honesty, safety, and responsibility.PKU-SafeRLHF contains 83.4K preference entries, HHH targets honesty, helpfulness, and harmlessness, and CVALUES evaluates safety and responsibility for Chinese LLMs.Source: §6.1.6 · p. 32

6.1.7 Recommendation / 6.1.8 Search

• Recommendation and search benchmarks extend evaluation beyond accuracy to explanation quality, user satisfaction, relevance, faithfulness, and domain-specific criteria.MovieLens explanations are rated for persuasiveness, transparency, accuracy, and satisfaction; TREC DL21 and DL22 use 0–3 relevance judgments over MS MARCO v2, while LeCaRDv2 adds characterization, penalty, and procedure to legal relevance.Source: §6.1.7 · pp. 32–33, §6.1.8 · p. 33

6.2 Metric

• Evaluation centers on how closely LLM judgments align with human evaluations, treating human ratings as the benchmark for quality.Because many evaluation tasks are complex and subjective, high agreement with humans is a key performance indicator.Source: §6.2 · p. 35

6.2.1 Accuracy.

• Accuracy measures the proportion of LLM judgments that match human judgments, but may miss quality differences in nuanced or continuous evaluations.It is simple and intuitive, making it suitable for classification tasks.Source: §6.2.1 · p. 35

6.2.2 Pearson Correlation Coefficient.

• Pearson correlation measures the linear relationship between continuous scores assigned by the LLM and human evaluators, with values ranging from −1 to 1.The metric compares LLM and human scores relative to their respective means.Source: §6.2.2 · p. 35

6.2.3 Spearman’s Rank Correlation Coefficient.

• Spearman’s rank correlation measures monotonic association using ranked scores, making it robust to outliers and non-linear relationships in preference or ranking tasks.It is most appropriate when the relative order of scores matters more than their exact values.Source: §6.2.3 · p. 35

6.2.4 Kendall’s Tau.

• Kendall’s Tau measures ordinal agreement between ranked lists and is useful for ranking consistency, particularly when datasets contain many ties.It compares concordant and discordant pairs and can provide a more nuanced agreement measure than Spearman’s ρ.Source: §6.2.4 · p. 36

6.2.5 Cohen’s Kappa.

• Cohen’s Kappa measures agreement between LLM and human raters beyond chance, making it more robust than accuracy for categorical classification.It accounts for observed agreement and expected agreement by chance.Source: §6.2.5 · p. 36
• The Intraclass Correlation Coefficient assesses rating reliability and consistency across multiple LLMs, human annotators, or data subsets.ICC uses variance components from one-way or two-way ANOVA models to provide a broader view of evaluator reliability.Source: §6.2.6 · p. 36

7 LIMITATION

• LLMs-as-judges have limitations that can affect their effectiveness, reliability, and fairness [208] [215].These limitations arise from inherent characteristics of LLMs.Source: §7 · p. 36

7.1 Biases

• Biases are examined as a key limitation of LLMs-as-judges.Source: §7 · p. 36

7.2 Adversarial Attacks

• Adversarial attacks are examined as a key limitation of LLMs-as-judges.Source: §7 · p. 36

7.3 Inherent Weaknesses

• Inherent weaknesses are examined as a key limitation of LLMs-as-judges.Source: §7 · p. 36

7.1 Biases

• Across these categories, biases inherited from training data can significantly affect evaluation results, creating challenges for fair, objective, and reliable judgments across tasks.The survey calls for systematic identification, quantification, and mitigation using balanced datasets, external feedback, fairness-aware prompting, and domain-specific solutions.Source: §7.1 · pp. 36–37, §7.1.4 · p. 42

7.1.3 Content-Related Biases.

• Content-related biases make judgments sensitive to emotional tone, token frequency, and application context rather than response quality or correctness.Examples include rewarding positive sentiment, favoring frequent pre-training tokens, and reproducing domain or contextual biases in healthcare, finance, and selected evaluation examples.Source: §7.1 · p. 37, §7.1.3 · p. 40

7.1.1 Presentation-Related Biases.

• Presentation-related biases make judgments depend on answer order or length rather than substantive quality, with position and verbosity bias producing skewed evaluations.Position bias can favor earlier-listed answers, while verbosity bias can favor longer responses regardless of content value; swapping, alignment, debate, controlled perturbation, and contrastive methods are proposed mitigations.Source: §7.1 · p. 37, §7.1.1 · pp. 38–39

7.1.2 Social-Related Biases.

• Social-related biases cause judges to privilege authority, majority opinion, anonymization cues, or identity markers, potentially producing unfair judgments.Prompt anonymization can reduce bandwagon effects, while standardized criteria and fairness-driven frameworks are suggested for compassion-fade and diversity biases; authority-bias solutions remain under exploration.Source: §7.1.2 · pp. 39–40

7.1.4 Cognitive-Related Biases.

• Cognitive-related biases include overconfidence, self-enhancement, refinement-aware, distraction, and fallacy-oversight biases, undermining the reliability and objectivity of judgments.Peer review and discussion, model diversity, reference-guided verdicts, external feedback, input sanitization, and stricter prompts are discussed, although effective mitigation of distraction bias remains underexplored.Source: §7.1.4 · pp. 40–42

7.2 Adversarial Attacks

• Adversarial attacks subtly modify inputs or context, causing inaccurate ratings, especially on complex or high-risk evaluation tasks [100] [193] [305].Manipulations include wording changes, word additions or removals, reordered words, and ambiguous phrasing.Source: §7.2 · p. 42

7.2.1 Adversarial Attacks on LLMs.

• General LLM attacks comprise text-level manipulations, structural or semantic distortions, and optimization-based attacks targeting model vulnerabilities.These categories respectively alter surface text, linguistic structure or meaning-preserving critical tokens, and influential features through gradient- or population-based optimization [57] [100] [18] [177] [260] [210] [211] [120].Source: §7.2.1 · pp. 42–43
• Automatic benchmarks such as MT-Bench can be deceived into producing artificially high scores, allowing malicious inputs to undermine evaluation reliability [293] [51].Zheng et al. [293] and Doddapaneni et al. [51] reported these vulnerabilities in LLMs-as-judges.Source: §7.2.2 · p. 43

7.2.2 Adversarial Attacks on LLMs-as-judges.

• Universal adversarial phrases can inflate LLM judge scores to their maximum regardless of quality and transfer from FlanT5-xl to GPT-3.5 and Llama2 [218] [184].Raina et al. [184] showed that short, carefully crafted appended phrases can manipulate scores across models.Source: §7.2.2 · p. 43
• JudgeDeceiver uses optimization-based prompt injection to generate adversarial sequences that make LLMs-as-judges select biased or incorrect candidate responses [196].Unlike handcrafted attacks, it formulates an explicit optimization objective for efficient sequence generation; this area remains largely underexplored [184] [196].Source: §7.2.2 · p. 43

7.3 Inherent Weaknesses

• Addressing these inherent weaknesses through retrieval, enhanced training, fact-checking, and knowledge graphs is crucial for reliable, accurate, and trustworthy evaluations across diverse applications.These limitations motivate continued research and innovation in LLMs-as-judges.Source: §7.3 · p. 43, §7.3.3 · p. 44

7.3.1 Knowledge Recency.

• Static training data can make LLM judges rely on obsolete information, while retraining risks catastrophic forgetting [152] and may produce erroneous evaluations of time-sensitive scenarios.Retrieval-augmented generation [69] [125], periodic fine-tuning, continual learning [244], and fact-checking can improve temporal reliability.Source: §7.3.1 · p. 44

7.3.2 Hallucination.

• Hallucinations may cause LLM judges to invent precedents, misinterpret facts, or fabricate sources, undermining judgment reliability and potentially producing unfair or harmful outcomes.Cross-verifying judgments against trusted databases and external knowledge sources can identify and correct hallucinated information [97] [216].Source: §7.3.2 · p. 44

7.3.3 Domain-Specific Knowledge Gaps.

• Despite broad generalization, LLM judges may lack the depth needed for specialized domains such as law, leading to shallow or incorrect judgments [55] [63] [171] [212].Domain-specific knowledge graphs [63] [171] and retrieval-augmented generation [69] can improve context awareness and specialized knowledge access.Source: §7.3.3 · p. 44

8.1.1 Automated Construction of Evaluation Criteria and Tasks.

• Future LLM judges should automatically adapt evaluation criteria and dynamically construct tasks to reduce manual prompt design and mitigate contamination in static datasets.Criteria could be tailored to task types, audiences, and domain knowledge while continuously optimizing evaluation processes for greater practicality and precision.Source: §8.1.1 · p. 45

8.1.2 Scalable Evaluation Systems.

• Modular frameworks, efficient candidate selection, and streamlined multi-LLM communication could improve cross-domain scalability while reducing evaluation costs in resource-constrained settings.These approaches target limited adaptability across tasks, the growing cost of pairwise comparisons, and the computational demands of multi-round multi-LLM evaluation.Source: §8.1.2 · p. 45, §8.1.3 · p. 45

8.2.1 Integration of Reasoning and Judge Capabilities.

• Future systems should integrate reasoning with evaluation and use collaborative multi-agent or ensemble mechanisms to produce more comprehensive, stable, and reliable judgments.For complex legal tasks, a model could infer relevant provisions before assessing case relevance; multiple models could offset individual biases and combine complementary reasoning and knowledge [22] [39].Source: §8.2.1 · pp. 45–46, §8.2.2 · p. 46

8.2.2 Establishing a Collective Judgment Mechanism.

• LLM judges should incorporate evolving domain knowledge and support cross-domain, cross-language, and multimodal transfer to handle specialized and complex evaluation scenarios.Proposed methods include knowledge graphs, expert-feedback fine-tuning, dynamic updates to statutes and case law [185], transfer from English to German [77] [202] [241], and unified processing of text, images, audio, and video [24].Source: §8.2.3 · p. 46, §8.2.4 · p. 46, §8.2.5 · p. 46

8.3.1 Enhancing Interpretability and Transparency.

• More reliable LLM judges should provide transparent explanations and validation grounded in logical frameworks, while using debiasing and fairness constraints to reduce unfair judgments across social, cultural, and legal contexts [147] [127].Interpretability is especially important in high-stakes domains, where users need to understand the basis of decisions.Source: §8.3.1 · pp. 46–47, §8.3.2 · p. 47

8.3.2 Mitigating Bias and Ensuring Fairness.

• Robustness research should address noise, incompleteness, ambiguity, and uncertainty through techniques such as bias detection, adversarial debiasing, and diverse data augmentation [58] [196].These methods aim to stabilize evaluation results and improve reliability in complex real-world environments.Source: §8.3.3 · p. 47, §8.3.2 · p. 47

9 CONCLUSION

• The survey integrates the framework’s advantages, limitations, implementations, applications, and evaluation methods across five dimensions.It also provides foundational resources and plans to maintain relevant content to support continued field development.Source: §9 · p. 47
• Future research should develop LLM judges that are more efficient, effective, and reliable.Source: §9 · p. 47