Source-linked AI summary
DA-RAC: Distance-Aware Calibration of LLM Judges for Trustworthy AI Auditing
Cheng Wu, Vishal Anand, Jaya Krishna Mandivarapu, Xiya Liu, Rui Zhuang
TL;DR
LLM judges can be miscalibrated by irrelevant reference examples, making context selection a central evaluation concern. DA-RAC retrieves and distance-weights relevant precedents, improving calibration relative to zero-shot, rubric-based, and static-anchor baselines while making judgments more inspectable and contestable.
Problem
LLM judges face context-induced miscalibration when irrelevant or mismatched reference examples distort artifact evaluation.
Method
DA-RAC retrieves semantically and structurally proximate precedents, weights them by distance, and exposes neighborhood difficulty for review.
Results
Benchmark results show that distance-aware anchoring improves calibration relative to zero-shot, rubric-based, and static-anchor baselines, while irrelevant references can substantially distort judgments.
Takeaways & Limitations
LLM evaluation should be context-sensitive, inspectable, and contestable rather than treated as fully automated cultural authority.
Takeaways & Limitations
The empirical results use LLM-judge benchmarks rather than community-specific cultural datasets, and precedent pools may encode canon or dominant-cultural bias.
Abstract
from arXiv · showhide
Generative AI systems are increasingly producing real-world artifacts, however their efficacy and validity are often evaluated via context-free LLM-scoring. These judges can be miscalibrated by irrelevant in-context reference examples, creating false confidence and allowing low-quality or harmful outputs to pass evaluation. We study this failure mode as context-induced miscalibration and introduce DA-RAC, a distance-aware reference-anchored calibration method for LLM judges. DA-RAC retrieves semantically and structurally similar labeled anchors for each judgement scenario, weights them by distance, and exposes neighborhood difficulty as a calibration and triage signal. On multi-run LLM-judge evaluation benchmarks, it improves calibration and reduces false-pass risk relative to zero-shot, chain-of-thought evaluation, and static-anchor baselines. Mechanistic analysis shows that judge scores vary systematically with anchor distance, while static references can induce misleading decision boundaries. Thus LLM-judgement requires not only better models, but also calibrated, auditable reference selection, especially when automated evaluation is used to support high-impact AI generated artifacts. Judgments should be grounded in relevant, inspectable, and contestable interpretive artifacts.
1. Introduction
Generative AI artifacts require context-sensitive evaluation because quality and value depend on genre, audience, community, precedent, and supported values. LLM judges can become miscalibrated when irrelevant reference examples shift their decision boundaries, motivating distance-aware, auditable precedent selection.
- Context-sensitive evaluation: Generative AI artifacts include stories, explanations, advice, critiques, and design proposals whose evaluation extends beyond correctness or harm avoidance.Their success may depend on genre, audience, community context, historical precedent, aesthetic stance, and supported values.
- LLM judges: LLM judges serve as scalable interpretive evaluators whose judgments about meaning, relevance, quality, and value are influenced by prompts, examples, and implicit norms.In few-shot evaluation, reference examples function as precedents.
- Context-induced miscalibration: Context-induced miscalibration occurs when irrelevant or mismatched reference examples shift an LLM judge’s decision boundary.This can make judges reward generic fluency, impose dominant norms on niche artifacts, or misread works from different interpretive backgrounds.
- Context-induced miscalibration: 52% accuracy is achieved by random precedents, eighteen points below the zero-shot baseline, revealing degraded accuracy or calibration in benchmark settings.The result is reported as evidence that irrelevant precedents can materially distort evaluation.
2. Contributions
The paper identifies context-induced miscalibration as a failure mode in which mismatched precedents distort LLM judges’ interpretive frames, and proposes DA-RAC to calibrate judgments using distance-aware, relevant precedents. It further argues that evaluators should make their precedents and uncertainty inspectable while routing contested cases to humans.
- Context-induced miscalibration occurs when irrelevant or mismatched precedents distort how judges interpret an artifact’s genre, audience, community, or value.
- DA-RAC retrieves semantically and structurally similar labelled precedents, weights them by distance, and exposes neighbourhood difficulty for human review.It replaces arbitrary few-shot context with distance-aware precedents to improve calibration.
- Evaluators should reveal which precedents shaped their judgments, flag cases far from known examples, and route contested cases back to humans.The approach supports contextual sensitivity and human agency rather than replacing critics, artists, communities, or domain experts.
3. Method
DA-RAC treats few-shot examples as distance-aware interpretive precedents, retrieving relevant anchors through instruction and structural similarity rather than using uniform references. It separates binary preference prediction from a weighted neighbourhood-difficulty score used to assess calibration and misrecognition.
- Interpretive precedents: DA-RAC treats few-shot examples as interpretive precedents whose relevance determines the genres, audiences, and criteria used to situate a target artifact.Mismatched precedents can shift evaluation toward the wrong interpretive frame, whereas relevant precedents help situate the target.
- Distance-aware retrieval: Distance-derived softmax weights make nearer precedents more influential while retaining contributions from more distant precedents; λ controls sharpness and k controls neighbourhood size.At λ = 1.0, distance 0.2 receives e^0.4 ≈1.49× the unnormalised weight of distance 0.6.
- Distance-aware retrieval: The method retrieves the k nearest precedents using instruction embeddings and combines surface-vector distances with sentence-level logic-graph distance.Graph Edit Distance captures causal, contrastive, and conditional logical divergence that surface embeddings may miss; retrieval excludes candidate responses and target labels.
- Inference and calibration: At inference, retrieved labelled precedents support a binary judge preference for accuracy, while their weighted vanilla-judge agreement produces a continuous neighbourhood-difficulty estimate.The difficulty estimate serves as the predicted probability for Expected Calibration Error (ECE) and Mean Squared Error (Brier).
- Inference and calibration: Evaluation uses Brier, ECE (10 bin), and interpretive misrecognition rate, with DA-RAC’s weighted score serving as the predicted probability against llm label.Static ablations include random, centroid, and diverse precedent selection.
4. Experimental Setup
The experiments use LLMEval2 as a controlled probe of reference dependence in LLM judging, with dynamically matched, static, and irrelevant precedents. The setup evaluates gpt-4o-mini and gpt-5.1 across repeated runs with fixed retrieval parameters.
- Experimental Probe: The controlled probe isolates how LLM judgments change when supplied precedents are irrelevant, static, or dynamically matched.This targets reference dependence as a mechanism central to artifact evaluation.
- Dataset and Models: LLMEval2 contains 1,600 samples split into 577 easy precedents, 223 hard precedents, and 480 test samples.The split is 70:30 between precedents and test data.
- Dataset and Models: Instruction embeddings use all-MiniLM-L6-v2 with 384 dimensions, while judgments use gpt-4o-mini and gpt-5.1.Each judge is run three times; variance is attributed to judge stochasticity under deterministic precedent selection.
- Experimental Configuration: Experiments use k = 5, λ = 1.0, and Order strategy, with an n=50 subset for multi-run judge-stochasticity experiments.The subset serves as a proof of concept and minimizes cost.
5. Probe: Reference Dependence in Judging
The probe shows that LLM judgments depend strongly on the relevance and structure of reference precedents, not merely on judge capability or rubric quality. DA-RAC improves accuracy, grounds scores in precedent distance, and exposes ambiguous cases for human review.
- Accuracy and reference dependence: 91.3% ± 2.5% accuracy: DA-RAC outperforms Vanilla zero-shot and CoT Rubric on multi-run evaluation, while static precedent strategies plateau at 52–67%.Vanilla zero-shot reaches 70.0%, and CoT Rubric reaches 74.7% ± 0.9%.
- Accuracy and reference dependence: 52.0% ± 0.0% accuracy: static-random precedents fall eighteen points below Vanilla zero-shot, showing that irrelevant context can mislead rather than help.The results make precedent relevance central to evaluative interpretation.
- Model capability and precedent selection: 84.7% ± 1.9% accuracy: DA-RAC improves GPT-5.1 over Vanilla by +12.0%, Static-centroid by +16.0%, and CoT by +20.7%.Against Static-diverse, the gap narrows to +4.0% (p = 0.149), indicating stronger models reduce but do not eliminate reference dependence.
- Distance-grounded judgment: ρ = −0.681 (hard, p < 0.001) and ρ = +0.356 (easy, p < 0.001): DA-RAC scores vary with precedent distance, unlike static methods at ρ = 0.000.This supports the claim that DA-RAC judgments are grounded in their precedent context.
- Structural relevance: 91 examples (19%): hybrid distance identifies surface-similar but logically divergent cases, while structural matching helps with contrastive, causal, and conditional reasoning.Embedding distance alone suffices for the 81% where embedding and structural distances agree.
- Calibration and triage: 92.0% judge accuracy: DA-RAC beats static-centroid’s 52.0% in the ambiguous zone, where neighborhood difficulty can flag cases for human review.Across zones, DA-RAC achieves ECE 0.084 and Brier 0.173; static-centroid reaches ECE 0.666 in the easy zone.
6. From Calibration to Reliable Evaluation Design
DA-RAC extends beyond calibration into a design pattern for culturally situated evaluation, combining community-curated precedents, inspectable interpretive lineage, contestable uncertainty signals, and deliberate human authority. This approach makes evaluation more context-sensitive and reviewable without treating cultural judgment as something AI should autonomously control.
- Evaluation design: DA-RAC’s culturally situated evaluation workflow uses community- and expert-curated precedent pools to encode successful performance for specific genres, audiences, or values.Relevant curators include communities, domain experts, artists, critics, educators, and practitioners.
- Evaluation design: For each judgment, DA-RAC exposes retrieved precedents and weights so reviewers can inspect and contest the context shaping the verdict.This creates an inspectable interpretive lineage for the judge’s effective context.
- Evaluation design: Distant or internally discordant neighborhoods signal interpretive uncertainty that may warrant community review rather than being treated as noise.Signals include high nearest-precedent distance, mixed neighborhood labels, and disagreement between semantic and structural distance; GED is proposed as a structural distance method.
- Evaluation design: DA-RAC supports human-AI ensembles by providing relevant precedents and provisional judgments while humans retain authority over ambiguous, novel, or contested cases.The method is not intended to automate cultural authority.
- Evaluation design: In community-facing cultural explanation, evaluation becomes more context-sensitive and reviewable by flagging distant or contested precedents for the people whose fields or materials are at stake.A generic evaluator may reward polished encyclopedic prose even when local terminology, contested histories, and avoidance of institutional flattening matter more.
7. Discussion
The discussion frames LLM evaluation as situated interpretation: judgments should use relevant, inspectable precedents and remain open to contestation rather than serving as universal or final arbiters. DA-RAC operationalizes this through distance-aware contextualization, while its benchmark evidence remains a technical probe with clear deployment and research limitations.
- Evaluation as situated interpretation: LLM evaluations are highly sensitive to framing examples, so cultural artifacts should be interpreted through precedents, genres, audiences, and values rather than universal criteria alone.
- Value add: contextual sensitivity: Contextual sensitivity requires evaluators to ground judgments in relevant precedents, expose them, and recognize when a case exceeds available context.
- Why distance matters: DA-RAC makes precedent selection local and distance-aware by combining structural and semantic similarity, addressing misleading frames from static or random references.Structural distance distinguishes artifacts that are topically similar but differ in argument, genre, contrast, causality, or conditional structure.
- Contestability rather than automation: DA-RAC aims to make judgments more contestable, logging retrieved precedents and exposing neighborhood difficulty so human reviewers can question relevance, representation, and ambiguity.
- Limitations: The empirical results are based on LLM-judge benchmarks, so they constitute a technical probe of reference dependence rather than evidence that DA-RAC solves cultural or artifact evaluation.The benchmark retrieves precedents using instruction embeddings to avoid target-label leakage; deployments should also incorporate nonlabel-bearing metadata such as genre and audience.
- Limitations: Future work may use community-curated precedent pools and evaluate DA-RAC for creative-writing feedback, museum labels, culturally specific health communication, public-memory projects, and multilingual civic explanation.
8. Conclusion
The conclusion frames LLM judges as interpretive technologies whose judgments depend on relevant context, and presents DA-RAC as distance-aware anchoring that improves calibration while supporting inspectable, contestable human review.
- 8. Conclusion: LLM judges depend on relevant precedents, genre, audience, and value rather than context-free criteria.DA-RAC operationalizes this interpretive view by retrieving semantically and structurally proximate precedents and weighting them by distance.
- 8. Conclusion: Distance-aware anchoring improves calibration over zero-shot, rubric-based, and static-anchor baselines, while less relevant references can substantially distort judgments.The method also exposes neighborhood difficulty as a signal for human review.
- 8. Conclusion: Evaluation systems should help humans inspect how judgments are made, identify missing context, and preserve areas where interpretation remains open.This vision treats evaluation as context-sensitive, inspectable, and contestable rather than as automated cultural authority.
Impact Statement
DA-RAC is intended to support culturally situated evaluation without automating cultural authority. Its approach emphasizes inspectable, contestable references, community curation, transparent reporting, and human review for difficult cases.
- Cultural evaluation: DA-RAC aims to reduce the risk that irrelevant or dominant-culture precedents flatten differences across genres, communities, and interpretive traditions.The method makes reference selection distance-aware, inspectable, and contestable.
- Safeguards: The paper recommends community-curated precedent pools, logging retrieved precedents, and reporting performance across cultural domains.
- Human agency: High-distance or high-disagreement cases should be routed to human review so AI evaluation helps humans interpret and contest cultural judgments.The stated goal is contextual sensitivity with human agency rather than silent automation.