PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze, Daniel Khashabi

arXiv:2610.12360v1cs.AIcs.CL

TL;DR

Outcome-focused evaluation offers limited insight into whether agents handle uncertainty during execution. The paper measures epistemic humility across matched knowledge-conflict tasks and finds that accuracy does not ensure uncertainty is preserved or communicated; prompting often raises escalation while lowering accuracy.

  • Problem

    Existing agent evaluations primarily grade task success, providing limited evidence about whether agents recognize, act on, and communicate uncertainty when evidence conflicts with prior beliefs.

    Source: §1 · pp. 1–2

  • Method

    The paper scores Identify, Solve, and Escalate from agent trajectories across controlled and naturally occurring knowledge conflicts with matched no-conflict controls.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Higher task accuracy does not necessarily correspond to greater epistemic humility, and system-prompt interventions often increase escalation while reducing accuracy.

    Source: §1 · p. 2, §3.4 · p. 8

  • Takeaways & Limitations

    The findings motivate evaluating agent behavior at the system level and scoring whether uncertainty is preserved and communicated, not accuracy alone.

    Source: §5 · p. 9

    ISE covers processing knowledge conflicts, not other facets such as resisting user pressure, declining tasks for missing tools, or calibrated confidence over continuous quantities.

    Source: Limitations · p. 9

Abstract

from arXiv · show

When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.

1 Introduction

The paper argues that outcome-based evaluation misses whether agents recognize, act on, and communicate uncertainty during execution. It evaluates epistemic humility through knowledge conflicts and finds that accuracy does not guarantee agents preserve or communicate uncertainty.

1 Introduction

  • Outcome-based evaluation grades final-answer correctness, not whether an agent flagged uncertainty that arose along the way.The paper notes that outcome-based rewards provide little incentive to surface uncertainty, and a harness may drop a conflict signal across a multi-step trajectory.Source: §1 · pp. 1–2
  • Epistemic humility is the willingness to recognize, act on, and communicate uncertainty, operationalized as Identify, Solve, and Escalate behaviors.The evaluation measures these behaviors throughout task execution, including escalation when uncertainty remains.Source: §1 · p. 2
  • The evaluation compares controlled conflicts with naturally occurring conflicts during multi-step tasks, pairing each setting with matched no-conflict controls.Conflicts can arise between parametric beliefs and retrieved evidence or between contextual sources.Source: §1 · p. 2
  • Higher task accuracy does not necessarily imply greater epistemic humility: agents may recognize conflicts but omit unresolved uncertainty from incorrect final answers.Conflict-relevant answer mentions peak in the first 10% of execution, while prompting often raises escalation and reduces accuracy.Source: §1 · p. 2

2 Evaluating Epistemic Humility via Knowledge Conflict

The evaluation constructs matched conflict and no-conflict tasks, records full agent trajectories, and scores three observable dimensions of epistemic humility. Its design covers controlled and naturally occurring conflicts across multiple agent and backbone configurations.

2 Evaluating Epistemic Humility via Knowledge Conflict

  • The framework evaluates trajectories on Identify, Solve, and Escalate using matched conflict and no-conflict tasks in two conflict settings.A GPT-5 judge scores the trajectory for recognizing a gap, following up with a tool call, and communicating residual uncertainty in the final turn.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Conflict Identification Pipeline

  • Controlled-conflict tasks contrast a model’s parametric answer with conflicting contextual evidence and pair each question with a same-question, same-passage-count no-conflict control.Instances are retained when the backbone’s free-form and multiple-choice answers agree and fall within the candidate set.Source: §2.1 · p. 3
  • Naturally occurring conflicts arise during multi-step tasks when tool results or the model’s own outputs contradict prior beliefs or other trajectory evidence.The control set draws from instances where the backbone is confidently correct or has no strong prior; comparisons across these questions are observational and may reflect task composition.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Evaluation Protocol

  • Identify scores the F1 of a judge’s intermediate-turn gap label against conflict status, while keeping final-turn hedging for Escalate.This prevents Identify from borrowing credit from final-turn hedging; the protocol records when recognition first appears in the trajectory.Source: §2.2 · p. 4
  • Solve measures bounded resolution: after flagging a gap, the agent must make a new, nonduplicate tool call and finish cleanly.It measures an attempt to resolve the conflict, not whether the attempt yields the correct answer; task accuracy is reported separately.Source: §2.2 · pp. 4–5
  • Escalate measures whether the final answer acknowledges residual uncertainty when a conflict-split answer is incorrect, whether or not the agent first identified the gap.This permits single-turn escalation to count and avoids conditioning the metric on intermediate recognition.Source: §2.2 · p. 5

2.3 Experimental Design

  • The study evaluates four harnesses across five fixed harness–backbone configurations, with each agent run on matched conflict and control pairs.Runs cover controlled and naturally occurring conflicts, with a maximum of 50 iterations per agent.Source: §2.3 · p. 5
  • Three model-family judges show agreement of 0.93–0.97 AC1 on correctness and 0.82–0.92 on Identify and Escalate; human annotators agree on 76.0% of Identify turns and 85.3% of Escalate answers.The human validation used three annotators labeling 50 Identify turns and 50 Escalate final answers.Source: §2.3 · p. 5

3 Findings

Across the findings, knowledge conflict can reduce task accuracy, and high accuracy does not ensure agents act on or communicate recognized uncertainty. Conflicts often surface early, while a prompting intervention usually raises escalation at an accuracy cost.

3.1 Knowledge conflict impacts task performance.

  • All reported TMD configurations lose more than 30 percentage points of accuracy under controlled conflict, while naturally occurring-conflict results vary across configurations and datasets.Some configurations score significantly lower on conflict than control, but BrowseComp and part of GAIA show higher accuracy on instances with false knowledge; the splits may differ in task difficulty.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 The Higher the Performance, the Less Likely the Agent to Elicit the Uncertainty.

  • Across agent-dataset cells, Solve falls from about 45% to under 10% and Escalate from about 38% to under 10% as accuracy rises, while Identify shows no clear monotone relationship.Claude Code reaches 90.0% Identify F1 on BrowseComp, yet its incorrect final answers often omit unresolved uncertainty.Source: §3.2 · pp. 6–7, Figure 3 · p. 6

3.3 Agents identify the issue early, but do not follow up.

  • Conflict-relevant answer mentions peak in the first 10% of trajectories, but early exposure is not consistently followed by resolution or escalation.The mentions then fall sharply into a long tail of later steps.Source: §3.3 · p. 7, Figure 4 · p. 7

3.4 Prompting improves EH but does not consistently improve accuracy.

  • On MoNaCo, Escalate rises from 1.6 to 60.7 for GPT-5 while its accuracy falls from 30.1 to 23.2; most prompted configurations trade higher escalation for lower accuracy.On BrowseComp, GPT-5 gains 28 points of Escalate but loses 6 points of accuracy.Source: §3.4 · p. 8

4 Related Work

Prior work studies knowledge conflict and uncertainty in single-turn language-model settings, while agent evaluations commonly emphasize final-answer correctness. This paper situates its contribution in measuring uncertainty-related behavior across agent trajectories.

Knowledge Conflict

  • Xu et al. (2024) organize conflicts into context-memory, inter-context, and intra-memory families; this work adopts the first two for elicitation, reusing Xie et al.’s ConflictQA pipeline for controlled conflict and using Hou et al.’s WikiContradict inter-context benchmark.Source: §4 · p. 8

Uncertainty and the incentive to commit.

  • Prior work on LLM self-knowledge and verbalized confidence studies whether models communicate what they know, with calibrated self-knowledge suppressed by post-training and confidence recoverable through prompting (Kadavath et al., 2022; Lin et al., 2022; Tian et al., 2023a).HumbleBench measures false-option rejection in multimodal LLMs, confined to a single-turn multiple-choice setting (Tong et al., 2025).Source: §4 · p. 8

Agentic systems and trajectory-level evaluation.

  • Agent benchmarks typically score final-answer correctness, while trajectory diagnostics largely locate errors rather than measure behaviors orthogonal to correctness, such as epistemic humility.The paper frames its focus as how such behaviors propagate or fail across steps.Source: §4 · p. 8

5 Conclusion

The conclusion argues that benchmark accuracy alone misses important differences in how agents handle uncertainty. It calls for trajectory-level evaluation that rewards recognizing, addressing, and communicating uncertainty.

5 Conclusion

  • Higher task accuracy is associated with lower escalation rates across configurations, and some high-accuracy agents leave unresolved uncertainty unacknowledged in incorrect final answers.The paper argues that evaluating the full agent system should account for the interaction of model, harness, and benchmark.Source: §5 · p. 9
  • Conflict-relevant answer mentions peak in the first 10% of execution, while follow-up and final acknowledgment remain limited in several configurations.The conclusion connects early conflict recognition with the need to preserve the conflict through later tool calls and the final answer.Source: §5 · p. 9
  • A system-prompt nudge often raises Escalate while lowering task accuracy.The findings motivate evaluating epistemic humility in the full agent system rather than isolating the model, harness, or benchmark.Source: §5 · p. 9
  • The authors recommend explicit abstain-or-escalate actions, uncertainty-aware benchmark scoring, and trajectory-step supervision.Per-turn ISE judgments are proposed as a signal for training beyond final-answer supervision.Source: §5 · p. 9
  • Agents with the same benchmark accuracy can differ substantially in whether they notice, try to resolve, and communicate uncertainty.The conclusion calls for scoring intermediate turns alongside final answers and crediting honest abstention.Source: §5 · p. 9

Limitations

The study’s scope is limited by its coupled backbone–harness configurations, its conflict-centered operationalization of epistemic humility, and its minimal prompt-only intervention.

Limitations

  • The study did not run a full backbone × harness combination, so its prompting intervention does not isolate the causal contributions of model, harness, and benchmark.The authors cite resource constraints and leave a factorial comparison to future work.Source: Limitations · p. 9
  • ISE captures conflict processing, not other facets such as resisting user pressure, declining tasks without suitable tools, or calibrated confidence over continuous quantities.The authors say the findings should not be read as a verdict on all aspects of agent honesty.Source: Limitations · p. 9
  • The evaluation elicits humility through conflicts between parametric beliefs and retrieved evidence or between retrieved sources; other situations are out of scope.Examples include unknowable answers with available tools, ambiguous instructions, and tasks beyond the agent’s competence.Source: Limitations · p. 9
  • The intervention tests a single system-prompt clause, not harness-level or training-level changes such as a first-class abstain action or process-supervised fine-tuning.The authors say its mixed results do not rule out a backbone-level explanation.Source: Limitations · p. 10

A Method details

The appendix supplies additional details on conflict selection, judges and diagnostics, and the evaluated backbones. Its procedures define which instances enter each setting and how trajectory-level behaviors are scored.

A Method details

  • The appendix provides further method details, data statistics, judge agreement and human-comparison results, verbatim judge prompts, and the intervention prompt.These materials are deferred from the main text to appendices A, D, and E.Source: A Method details · p. 13

A.1 Conflict-level taxonomy for the controlled-conflict setting

  • Only Level 3 controlled-conflict instances proceed: free-form and multiple-choice elicitation agree on an answer in the candidate set, providing a stable identifiable belief.Conflict levels are computed per backbone, so a question may qualify for one model but not another.Source: §A.1 · p. 13

A.2 Selection procedure for the naturally occurring-conflict setting

  • The naturally occurring-conflict set selects instances where the backbone confidently holds a wrong belief, with controls for confidently correct answers and no strong belief.Conflict instances require a non-refusal parametric answer, self-verification agreement, and disagreement with ground truth.Source: §A.2 · p. 13

A.3 EH judge prompts, think-block policy, and secondary diagnostics

  • Identify is judged from intermediate assistant turns, while Escalate is judged from the final turn for substantive acknowledgment of conflict or uncertainty.The judges require more than generic uncertainty language; Escalate criteria include source disagreement, low confidence, missing information, or a suggestion to verify.Source: §A.3 · pp. 13–14
  • Think blocks are stripped for correctness and secondary diagnostics but retained for Identify because mid-trajectory recognition can appear in reasoning traces.Stripping them for Identify would systematically under-count recognition by reasoning models.Source: §A.3 · p. 14
  • Secondary diagnostics track trajectory-wide conflict acknowledgment, self-inconsistency, final-answer source faithfulness, and tool-call counts.Conflict acknowledgment is the union of mid-trajectory Identify and final-turn Escalate; tool-call counts are compared across matched conflict and control runs.Source: §A.3 · p. 14

A.4 Backbone roster

  • The five evaluated harness–backbone pairs span four harnesses and are used in both conflict settings and prompting-intervention runs.Sample sizes vary because conflict identification is per pair, and the results tables report them alongside each number.Source: §A.4 · p. 14

B Data statistics

The data-statistics tables report counts across the conflict-identification pipeline, including the matched controls.

B Data statistics

  • Tables 3 and 4 report dataset counts from the original split through eligibility, conflict selection, and matched controls; counts vary by backbone.The predicates are applied per question–backbone pair, and the tables use the configurations from the main results.Source: B Data statistics · p. 14

C Epistemic-humility metrics

The metrics score epistemic humility across Identify, Solve, and Escalate, with each dimension using a distinct definition and denominator.

C Epistemic-humility metrics

  • Table 3 organizes controlled-conflict counts by parametric-knowledge level and scenario, with Conflict and Control splits for RAG and PCK.CB has only a closed-book split; the knowledge levels range from no knowledge to agreed knowledge.Source: Table 3 · p. 15
  • Table 4 reports naturally occurring-conflict counts for GAIA, MoNaCo, and BrowseComp, including a filter funnel and conflict/control totals.The control set includes instances where the model agrees with ground truth and instances where it has no strong belief.Source: Table 4 · p. 15
  • Identify is F1 against the conflict/control label, while Solve and Escalate are rates reported only on the conflict split.Identify is computed jointly over both splits; Solve and Escalate are defined on their own denominators.Source: C Epistemic-humility metrics · pp. 14–15

D.1 Agreement across judge models

The authors compare judge-model labels to assess whether their results depend on GPT-5. Agreement is high under Gwet’s AC1, while alternate judges are stricter on the two epistemic-humility labels.

D.1 Agreement across judge models

  • Correctness, Identify, and Escalate are judge-assigned; the comparison re-scores every instance with two additional judges from different model families.The same prompts and trajectory text are used across judges; Table 6 reports raw agreement, κ, and AC1 by judge pair and conflict setting.Source: §D.1 · p. 15, Table 6 · p. 16
  • 0.82–0.92 AC1 agreement on Identify and Escalate, and 0.93–0.97 on correctness, indicates high agreement across judge models.Agreement is higher in controlled conflict than in naturally occurring conflict.Source: §D.1 · p. 15, Table 6 · p. 16
  • Gwet’s AC1 is reported alongside Cohen’s κ because rare ISE labels can depress κ even when raters agree.The ISE behaviors occur at rates of 11 to 14%.Source: §D.1 · p. 15
  • Alternate judges label more instances as not identifying or escalating, so the reported low Solve and Escalate rates would be lower under either alternate judge.The authors conclude that agents’ infrequent user-facing conflict acknowledgment does not depend on the GPT-5 judge.Source: §D.1 · p. 16

D.2 Human validation of the EH judges

Human annotations provide a second check on the EH judges, while the appendices document the evaluation prompts and the humility-prompt intervention. The intervention’s reported tables enable row-by-row comparison with the corresponding main-text results.

D.2 Human validation of the EH judges

  • 76.0% Identify and 85.3% Escalate human–human agreement show that these judgments are difficult even for annotators reading the full turn.The GPT-5 judge agrees with the human majority on 70.0% of Identify and 72.0% of Escalate items.Source: §D.2 · p. 18

D.3 Prompts

  • The appendix records all four LLM-judge prompts, sent to GPT-5 without a max_tokens cap at temperature 1.Each prompt is a system message; instance-specific references, predictions, or trajectory turns are supplied separately as user content.Source: §D.3 · p. 18, Figure 9 · p. 19

E.1 Prompt Content

  • The humility-prompt clause is appended identically across agent–backbone pairs and changes neither tools, the planning loop, nor backbone weights.Figure 10 reproduces the clause used in the intervention.Source: §E.1 · p. 18, Figure 10 · p. 19

E.2 Performance and epistemic-humility scores under the humility prompt

  • Tables 8 and 9 report accuracy and EH metrics under the humility prompt, allowing direct row-by-row comparison with the corresponding main-text tables.Table 8 reports dataset-specific accuracy metrics; Table 9 reports Identify, Solve, and Escalate on the conflict split.Source: §E.2 · p. 18

Related papers

Loading 2610.12360v1…