PaperlayerView exact revision on arXiv

Source-linked AI summary

Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation

Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, Kyomin Jung

arXiv:2410.20774v2cs.CL

TL;DR

Epistemic markers are increasingly used to express uncertainty in LLM outputs, but their effect on automated evaluation has been overlooked. EMBER tests LLM-judges in single and pairwise settings and finds that all tested judges are influenced by these markers, with stronger bias against uncertainty-expressing weakeners. The results indicate that LLM-judges may penalize honest uncertainty rather than judge correctness alone.

  • Problem

    The impact of epistemic markers on LLM-judge evaluations has been largely overlooked, despite growing efforts to train LLMs to use them.

    Source: §1 · p. 1

  • Method

    EMBER evaluates LLM-judge robustness to epistemic markers on question answering and pairwise instruction-following tasks.

    Source: §1 · p. 1, §3.3 · p. 4, §4.1 · p. 5

  • Results

    All five tested LLM-judges were influenced by epistemic markers, with stronger bias against weakeners expressing uncertainty; adding weakeners caused over 30% of preferences to switch from stronger-model outputs to weaker-model outputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    LLM-judges do not reliably focus solely on output correctness when epistemic markers are present, potentially undermining their use for evaluating honest LLMs.

    Source: §1 · p. 2, §6 · p. 9

    The benchmark covers open-ended question answering and instruction following, leaving effects across other tasks such as dialogue response evaluation unresolved.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In line with the principle of honesty, there has been a growing effort to train large language models (LLMs) to generate outputs containing epistemic markers. However, evaluation in the presence of epistemic markers has been largely overlooked, raising a critical question: Could the use of epistemic markers in LLM-generated outputs lead to unintended negative consequences? To address this, we present EMBER, a benchmark designed to assess the robustness of LLM-judges to epistemic markers in both single and pairwise evaluation settings. Our findings, based on evaluations using EMBER, reveal that all tested LLM-judges, including GPT-4o, show a notable lack of robustness in the presence of epistemic markers. Specifically, we observe a negative bias toward epistemic markers, with a stronger bias against markers expressing uncertainty. This suggests that LLM-judges are influenced by the presence of these markers and do not focus solely on the correctness of the content.

1 Introduction

EMBER examines whether LLM-judges can evaluate outputs containing epistemic markers without being influenced by expressions of certainty or uncertainty. Across two evaluation tasks, the experiments find systematic bias against these markers, especially weakeners, with real-world consequences for model comparisons.

Benchmark

  • EMBER is a benchmark testing whether LLM-judges make correct verdicts without being influenced by epistemic markers.It covers question answering and instruction following evaluation.Source: §1 · pp. 1–2

Benchmark tasks

  • EMBERQA contains 2,000 instances for judging output correctness, while EMBERIF contains 823 instances for selecting the correct output in pairwise instruction-following comparisons.Source: §1 · p. 2

Benchmark construction

  • The evaluated outputs were augmented with GPT-4o-generated epistemic markers sampled according to their distribution in real-world usage.Source: §1 · p. 2

Findings

  • Five LLM-judges were heavily influenced by epistemic markers, generally showing stronger bias against weakeners that express uncertainty.The tested judges were GPT-3.5-turbo, GPT-4-turbo, GPT-4o, Llama-3-8B-Instruct, and Llama-3-70B-Instruct.Source: §1 · p. 2

Real-world implications

  • Human judgments remained based on output correctness, whereas LLM-judge preferences switched by over 30% toward a weaker model after weakeners were added to a stronger model’s output.These findings challenge whether LLM-judges reliably mimic human judges when epistemic markers are present.Source: §1 · p. 2

Contributions

  • The study contributes EMBER, analyses of bias patterns, and experiments examining the implications of LLM-judges’ biases against epistemic markers.Source: §1 · p. 2

2 Related Works

Prior work studies confidence expression and the use of LLMs as scalable evaluators, but the effect of epistemic markers on LLM evaluation remains unexplored. This paper positions that effect as a distinct evaluation bias.

Honesty Alignment

  • Research on honesty alignment has examined token probabilities, consistency across sampled outputs, explicit confidence prompting, and naturally expressed confidence levels.Source: §2 · p. 2

Epistemic markers

  • Epistemic markers verbally signal certainty, influence user trust, and are associated with overconfidence in current LLMs.Source: §2 · p. 2

LLM-as-a-judge

  • LLM-as-a-judge research reports strong human alignment and advantages in scalability, explainability, and reproducibility, alongside documented evaluation biases.Source: §2 · p. 2

Research gap

  • Known LLM-judge biases include position bias, self-enhancement bias, and beauty bias, while epistemic-marker effects had not been examined.The paper presents itself as the first study of epistemic markers in LLM evaluation.Source: §2 · p. 3

3 EMBER: Epistemic Marker Benchmark for Evaluator Robustness

EMBER is a benchmark for testing LLM-judge robustness to epistemic markers across single-answer QA and pairwise instruction-following evaluation. It defines marker categories, constructs filtered benchmark instances, and measures accuracy changes and verdict switching relative to unmarked evaluations.

3 EMBER: Epistemic Marker Benchmark for Evaluator Robustness

  • EMBER evaluates robustness in single-evaluation QA and pairwise instruction-following settings, with benchmark construction details and examples spanning both tasks.EMBERQA targets question-answer correctness, while EMBERIF compares two outputs for a given instruction.Source: §3 · p. 3, Table 1 · p. 4, §3.3 · p. 4

3.1 Epistemic Markers

  • Strengtheners convey certainty and weakeners convey uncertainty; EMBER samples the 20 most frequent markers of each type using frequency-weighted selection.The marker set is intended to reflect real-world usage distributions.Source: §3.1 · p. 3

3.2 EMBERQA

  • EMBERQA contains 2,000 QA instances augmented into strengthener, neutral, and weakener groups, with human assessment and refinement of output correctness and marker naturalness.The data are sourced from evaluations covering Natural Questions and TriviaQA, with 1,000 samples from each dataset.Source: §3.2 · p. 3

3.3 EMBERIF

  • EMBERIF creates nine marker combinations by augmenting both outputs in pairwise instruction-following comparisons, then filters instances to preserve distinguishable correctness and natural marker integration.The default setup places the correct output first, while marker combinations leave output veracity unchanged.Source: §3.3 · p. 4

3.4 Evaluation Metrics

  • Accuracy measures agreement with ground-truth labels, while Verdict Switch Rate counts verdict changes from unmarked evaluations and separates Correct-to-Incorrect from Incorrect-to-Correct switches.Accuracy changes relative to neutral groups indicate bias direction; robustness would ideally correspond to zero change, and VSR captures sensitivity regardless of direction.Source: §3.4 · pp. 4–5, Figure 2 · p. 5

4 Experiments

EMBER evaluates whether LLM-judges remain robust to epistemic markers in single QA and pairwise instruction-following evaluations. Across both tasks, judges are influenced by these markers, especially weakeners, although larger models generally show improved robustness.

4.1 Experimental Setting

  • The experiments evaluate five judges across reference-guided QA and instruction-following pairwise settings, averaging two orderings to reduce positional bias.The evaluated models are GPT-3.5-turbo, GPT-4-turbo, GPT-4o, Llama-3-8B-Instruct, and Llama-3-70B-Instruct.Source: §4.1 · p. 5

4.2 Results & Analysis

  • All tested LLM-judges are influenced by epistemic markers in both EMBERQA and EMBERIF, indicating a lack of robustness.Robust judges should show minimal accuracy shifts and near-zero verdict switch rates.Source: §4.2 · p. 5, §4.2.1 · p. 6, §4.2.2 · p. 6

4.2.1 Results on EMBERQA

  • Most models prefer neutral expressions over strengtheners and weakeners, with the strongest bias against weakeners: neutral (N) > strengtheners (S) > weakeners (W).GPT-4-turbo deviates in QA by frequently categorizing marker-containing outputs as correct.Source: §4.2 · p. 5, §4.2.1 · p. 6, §4.2.2 · p. 6
  • As judge capacity increases, robustness to epistemic markers improves, but GPT-4o remains significantly vulnerable to weakeners in QA.In instruction following, GPT-4o instead exhibits the greatest robustness against both marker types, showing task-dependent variation.Source: §4.2.1 · p. 6, §4.2.2 · p. 7

4.2.2 Results on EMBERIF

  • For Llama-3-8B-Instruct, accuracy rises 7.1% with strengtheners and 17.1% with weakeners in selected EMBERIF comparisons, reflecting bias toward the correct output and neutral expressions.The preference ranking remains neutral (N) > strengtheners (S) > weakeners (W).Source: §4.2.2 · p. 6
  • When identical markers appear in both pairwise outputs, judgments may still shift, but accuracy changes remain minimal and show no clear directional bias.Some prompting methods improve robustness, but none fully resolves unfair evaluation with epistemic markers.Source: §4.2.2 · p. 7

5 Real-Life Implications

Human judges largely prioritize output correctness over epistemic markers, unlike LLM-judges. In a realistic pairwise scenario, weakeners can reverse which model an LLM-judge favors despite leaving the stronger model’s objective content unchanged.

5.1 Do Human-Judges Exhibit Biases against Epistemic Markers?

  • Human-judge accuracy is identical for strengthener and neutral groups at 87.3, while weakener accuracy is 87.0, a negligible difference.The study reports substantial inter-annotator agreement across participants.Source: §5.1 · p. 8
  • Human judges prioritize correctness over epistemic markers, whereas LLM-judges may fail to mimic human evaluation when such markers are present.The human experiment uses reference-guided QA because instruction-following judgments can vary with annotators’ knowledge and common sense.Source: §5.1 · p. 8

5.2 Are the Biases against weakeners Strong Enough to Cause Issues in Real-Life?

  • The real-life analysis compares pairwise outputs where both, one, or neither may be correct, better reflecting the range of outcomes in practical settings.The evaluation is conducted twice and averaged, as in the earlier experiments.Source: §5.2 · p. 8
  • When neither output contains markers, Llama-3-70B-Instruct favors GPT-4o over GPT-3.5-turbo 58.6% to 41.4%; adding weakeners to GPT-4o’s output reverses this to 22.8% versus 77.2%.The weakeners do not change the objective content, yet the LLM-judge disproportionately disfavors the marked output.Source: §5.2 · p. 8

6 Conclusion

The paper introduces EMBER to quantify how epistemic markers distract LLM-judges. Its experiments find insufficient robustness, raising concerns about fairness and accurate alignment in model evaluation.

6 Conclusion

  • EMBER is a meta-evaluation benchmark for testing LLM-judge robustness when evaluating outputs containing epistemic markers.The benchmark targets evaluation processes that may be distracted by markers rather than focused on content correctness.Source: §6 · p. 9
  • Experiments across various LLM-judges reveal vulnerabilities in their evaluation processes, highlighting the importance of fairness and accurate alignment.Source: §6 · p. 9

Limitations

The study’s scope is constrained by its evaluated tasks, marker types, language, and text-only modality, leaving several extensions for future research.

  • The evaluation covers open-ended question answering and instruction following, leaving other tasks such as dialogue response evaluation unexamined.Source: Limitations · p. 9
  • EMBER focuses on the 20 most frequent strengtheners and 20 most frequent weakeners, leaving less frequent and other marker types for further analysis.Source: Limitations · p. 9
  • The study is monolingual and English-only, so marker use and interpretation across languages and cultures remain open questions.Source: Limitations · p. 9
  • The work is limited to text, leaving multimodal LLM-as-a-judge settings as a future investigation avenue.Source: Limitations · p. 9

Ethics Statement

The paper documents its dataset construction, marker sampling, verification, and research resources, alongside ethics and reproducibility practices.

Ethics Statement

  • The study uses publicly available EVOUNA and MixInstruct datasets, sourced from established question-answering and instruction-following benchmarks.Source: Ethics Statement · p. 9
  • Models and resources were obtained through official or publicly accessible platforms, and sentence-level AI assistance was used during paper writing.Source: Ethics Statement · p. 9
  • The study reports fair compensation, withdrawal rights, and screening for offensive, sexist, racist, toxic, and sexual content among human annotation procedures.Source: Ethics Statement · p. 9

A Details of EMBER creation

  • GPT-4o augments question-answering outputs and generates correct and incorrect instruction-following outputs before marker augmentation.Source: A Details of EMBER creation · p. 14
  • Manual verification refined output correctness and marker naturalness, requiring revisions for approximately 2% of EMBERQA and 4% of EMBERIF outputs.Source: A Details of EMBER creation · p. 14

A.1 Epistemic Markers Statistics

A.2 Dataset Statistics

The paper directs readers to Table 11 for EMBER’s dataset statistics.

  • EMBER’s dataset statistics are reported in Table 11.Source: §A.2 · p. 14

B Details of Experimental Setting

The experimental setup provides reproducibility resources, specifies computing and model versions, and documents the evaluation prompts.

B.1 Dataset and Source Code

  • The source code, configurations, generated datasets, and pretrained weights are intended for public release to support reproducibility.Source: §B.1 · p. 14

B.2 Computing Resources

  • Experiments used two NVIDIA Tesla A100 GPUs with 80GB of memory each, Python 3.7.13, and PyTorch 1.10.1.Source: §B.2 · p. 14

B.3 Versions of the LLMs

  • The evaluated GPT models were GPT-3.5-TURBO-0125, GPT-4-TURBO-2024-04-09, GPT-4O-MINI-2024-07-18, and GPT-4O-2024-08-06, accessed through OpenAI’s official platform, while the Llama models were LLAMA-3-8B-INSTRUCT and LLAMA-3-70B-INSTRUCT, obtained from Hugging Face’s official repository.Source: §B.3 · p. 14

B.4 Prompts for LLM Evaluation

  • Question-answering and instruction-following prompt templates are provided in Tables 12 and 13, with the latter largely based on Zeng et al. (2023).Source: §B.4 · p. 14

C Additional experimental result in EMBERQA

The additional experiments examine task-specific prompting as a way to improve LLM-judge robustness to epistemic markers, alongside further QA results and human-annotation details. Both tested prompting approaches improve robustness, but neither fully eliminates unfair evaluation of genuine epistemic markers.

D Task Specific Prompting

  • The study tests an added instruction highlighting epistemic markers and chain-of-thought prompting to address the judges’ lack of robustness.The added instruction warns that epistemic markers may be deceiving, while chain-of-thought prompting generates a reasoning chain alongside the evaluation.Source: D Task Specific Prompting · p. 14
  • Both chain-of-thought and task-specific prompting improve robustness, but neither fully mitigates unfair evaluation when genuine epistemic markers are present.The results are reported for QA and instruction-following tasks in Tables 16–19.Source: D Task Specific Prompting · p. 15, Table 16 · p. 23, Table 17 · p. 23

E Details of human annotation

  • Nine English-proficient crowd workers were divided across QAN, QAS, and QAW evaluations, with substantial inter-annotator agreement assessed using Cohen’s kappa.Workers received task descriptions, guidelines, examples, and compensation meeting a minimum hourly wage of $13 or more.Source: E Details of human annotation · p. 15
Loading 2410.20774v2…