PaperlayerView exact revision on arXiv

Source-linked AI summary

What Counts as Evidence? Epistemic Cue Utilization in Large Language Models

Jinchang Zhang, Kaijun Jiang, Guoyu Lu

arXiv:2610.12788v1cs.CY

TL;DR

The paper asks which cues shape LLM judgments when sources conflict. It measures cue identification and influence separately in a controlled benchmark, finding that near-identical reliability recognition can accompany history-swap effects ranging from negligible to complete choice reversal.

  • Problem

    Answer changes alone conflate whether models identify relevant cues, use them, or respond more strongly to competing features such as event time and presentation order.

    Source: §1 · p. 1

  • Method

    A controlled benchmark and interventions measure cue identification separately from the influence of controlled cue changes on final judgments.

    Source: §1 · pp. 2–3

  • Results

    Across 13 LLMs, Qwen3.8-27B pairs 99.5% recognition accuracy with a 0.0-point history-swap effect, while GPT-6 Sol pairs 100% with 100.0 points.

    Source: §2.5 · p. 5

  • Takeaways & Limitations

    Evaluating LLM judgment under uncertainty requires distinguishing what models identify, what influences their judgments, and whether that influence can be changed.

    Source: §5 · p. 8

    The controlled textual scenarios do not fully capture naturally occurring evidence, long interactive contexts, or multimodal inputs.

    Source: §6 · p. 9

Abstract

from arXiv · show

Which information actually shapes a language model's judgment when sources disagree? We study epistemic cue utilization through a controlled benchmark of 500 scenarios and five complementary tracks separating cue identification, cue influence, and competition between source, temporal, and presentation cues. Across 13 LLMs, nearly identical recognition of relative source reliability can accompany effects ranging from negligible to complete choice reversal when source histories are exchanged. The resulting profiles distinguish models that rely on source history from those more responsive to evidence lineage, event time, or presentation order. We then intervene on three Qwen checkpoints using matched sham material, fact restatement, structured evidence records, and an additional weighting principle. Restatement increases reliability sensitivity in the two 27B models, and adding the principle increases it by a further 39.2 and 47.2 percentage points relative to the ledger. The 9B model instead changes its sensitivity to evidence lineage without a comparable reliability response. Verified-state accuracy remains 100\% throughout these interventions. Together, the benchmark and interventions show that identifying an epistemic cue and allowing it to influence judgment are distinct, experimentally measurable aspects of model behavior.

1 Introduction

The paper asks which cues shape LLM judgments when information conflicts, distinguishing whether a model can identify a cue from whether it influences the model’s final judgment. It introduces a controlled benchmark and interventions to measure and reshape cue use while preserving performance when trusted verification makes the true state explicit.

1 Introduction

  • Existing answer-change measures conflate failures to identify a cue, behavioral insensitivity to it, and stronger responses to other features such as event time or presentation order.The paper draws on source monitoring, credibility, epistemic vigilance, and advice-taking research to motivate separating report content from source judgments.Source: §1 · p. 1
  • Cue identification and cue influence are separate behavioral questions, with source identity, historical reliability, evidence quality, independence, and event time treated as epistemic cues.The authors use cognitive-science distinctions as experimental measures, not as claims that LLMs share human psychological mechanisms.Source: §1 · p. 2
  • Cue availability measures explicit cue identification, while cue influence measures how much a controlled cue change alters final judgment.Their divergence is termed a cue availability–influence dissociation; availability is an operational measure, not a claim about internal representations.Source: §1 · p. 2
  • The 500-scenario benchmark separates cue identification from influence and tests competition among source, evidence, temporal, and presentation cues.The benchmark’s five tracks measure cue representation, integration, reliability utilization, cue competition, and verified reasoning.Source: §1 · pp. 2–3
  • Across 13 LLMs, models that recognize the same evidence can rely on it differently, and controlled presentation, organization, or weighting changes can reshape cue use.These interventions preserve strong performance when trusted verification makes the true state explicit.Source: §1 · p. 3

2 Benchmark

The benchmark measures cue identification and influence separately across controlled scenarios, then compares how models respond to competing cues and trusted verification. Across systems, reliability recognition can be similar even when the effect of exchanging source histories differs substantially.

2.1 Measuring Epistemic Cue Utilization

  • Unverified support-choice requests have no unique gold state, so the benchmark measures changes across matched conditions rather than scoring a single correct choice.Identification probes and final judgments are distinct requests in the evaluation design.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Scenario Bank and Evaluation Flow

  • The 500-world benchmark includes 120 Structural Generalization and 80 Controlled Naturalized worlds; Controlled Core uses tightly matched comparisons, while Structural Generalization varies five structural axes.Each world contributes 48 independent requests, yielding 24,000 requests per system and 312,000 model–request cells across 13 evaluated systems.Source: §2.2 · p. 3
  • Worlds encode event and evidence provenance structures; source history reflects prior agreement with trusted verification, while evidence lineage tracks a record’s path to its final report.Current evidence quality is varied through target identifiability, not by labeling reports true or false.Source: §2.2 · p. 3

2.3 Diagnostic Tracks

  • The five tracks test cue representation, integration, reliability utilization, cue competition, and verified reasoning, with trusted target-time evidence resolving the disputed state.The overview spans 500 evaluation worlds and supports cross-model profiles and intervention analysis.Source: §2.3 · p. 4, Figure 1 · p. 4

2.4 Evaluation Measures

  • Cue influence is the signed percentage-point change in state-B choices across matched conditions, while verified reasoning scores accuracy against trusted target-time evidence.The reliability-swap contrast compares LH and HL conditions; positive values mean the stronger history assigned to O2 increases selection of B.Source: §2.4 · p. 4, Table 2 · p. 5

2.5 Benchmark Results

  • Qwen3.8-27B has 99.5% recognition accuracy and a 0.0-point history-swap effect, versus GPT-6 Sol’s 100% and 100.0 points; profiles also differ across other cues.Figure 3 presents distinct contrasts on a common percentage-point scale, not as a shared latent weight.Source: §2.5 · p. 5, Table 2 · p. 5, Figure 3 · p. 6

3 Experiments

Across the benchmark diagnosis and targeted interventions, source recognition often diverged from its influence on choices, while interventions shifted cue sensitivities differently by model. The experiments distinguish source history, evidence relations, event time, and presentation order.

3.1 Cognitive Analysis and Diagnosis

  • The three 27B Qwen models achieve approximately 99–100% unequal-history recognition on Core, yet their history-swap effects range from 0 to only 4.0 pp.For Qwen3.5-27B, conditioning on correct reliability-identification probes leaves the Core effect essentially unchanged: 0.68 pp versus 0.67 pp overall.Source: §3.1 · p. 6
  • Qwen3.5-27B and Qwen3.6-27B rank unequal histories correctly on 99.25% and 99.42% of trials, but recover both exact success counts on 54.5% and 58.7%.Qwen3.5-27B’s HH and LL recognition is 0% on Core, while Grok 4.3 correctly handles all four configurations.Source: §3.1 · p. 6
  • For Qwen3.5-27B, repeating one report changes choice by 2.0 pp, adding a speaker sharing that report by 5.7 pp, and replacing the shared root with a separate root by 58.7 pp.The root effect is 75.0 pp for GPT-5.4 and 42.3 pp for GPT-6 Sol; the latter contrasts with its 0.7 pp factorial lineage effect.Source: §3.1 · p. 6
  • Qwen3.5-27B responds by 52.3 pp to event time and −43.7 pp to text order; Qwen3.6-27B’s corresponding effects are 69.5 and −28.1 pp.GPT-6 Sol combines a 99.8 pp time effect with a −0.2 pp order effect.Source: §3.1 · pp. 6–7

3.2 Theory-Driven Intervention

  • Flat raises the history-swap effect from 1.0 to 25.6 pp in Qwen3.5-27B and from 4.6 to 13.6 pp in Qwen3.6-27B.Both changes are significant in the pre-specified comparison family.Source: §3.2 · p. 7
  • +Rule raises history sensitivity by 39.2 and 47.2 pp relative to Ledger, reaching 64.4 and 65.0 pp in the two 27B models.Verified accuracy remains 100% for every model and treatment.Source: §3.2 · pp. 7–8
  • Sham reduces absolute order effects from 41.8 to 16.0 pp and from 26.7 to 11.7 pp in the two 27B models; Flat reduces them further to 6.6 and 3.3 pp.Ledger adds a corrected 4.5 pp reduction over Flat in Qwen3.5-27B, but not in Qwen3.6-27B.Source: §3.2 · p. 7
  • Qwen3.5-9B retains only a 0.2–0.6 pp history-swap effect across treatments, while +Rule raises its lineage effect from 36.2 to 58.2 pp relative to Ledger.The 27B models also shift across cues: Qwen3.5-27B’s time effect falls by 11.8 pp versus Ledger but remains above Base (70.5 versus 52.0 pp); Qwen3.6-27B retains a 90.8 pp time effect.Source: §3.2 · p. 8

4 Related Work

Related work examines conflicting information, source credibility, multi-cue judgment, and presentation effects. This paper distinguishes cue identification from the influence of matched cue changes on decisions.

4 Related Work

5 Conclusion

The paper’s benchmark separates cue identification from cue influence and finds that these behaviors can diverge. Controlled interventions reshape which cues influence decisions while preserving strong factual performance when the true state is explicit.

5 Conclusion

  • The paper introduces a benchmark that separates whether models identify judgment-relevant cues from how strongly those cues influence final choices.Across models, these behaviors often diverge and produce distinct patterns of cue use.Source: §5 · p. 8
  • Controlled changes to evidence presentation and use reshape which cues influence decisions while preserving strong factual performance when the true state is explicit.The conclusion frames evaluation under uncertainty as requiring measures of identification, influence, and whether influence can change.Source: §5 · p. 8

6 Limitations

The benchmark’s controlled, researcher-constructed text scenarios do not capture the full complexity of real-world evidence environments, and the interventions cover only three Qwen checkpoints.

6 Limitations

  • The 500-world benchmark uses controlled textual scenarios, so it does not fully represent naturally occurring information, long interactive contexts, or multimodal inputs.Even its Controlled Naturalized subset uses researcher-controlled materials rather than independently authored real-world cases.Source: §6 · p. 9
  • The intervention study tests prompt-level changes only on three Qwen checkpoints, leaving generalization to other model architectures unestablished.The interventions involve fact restatement, information organization, and evidence-weighting rules.Source: §6 · p. 9

7 Ethical Considerations

The benchmark uses fictional, text-based scenarios and measures responses to conflicting evidence, not factual correctness or guaranteed trustworthiness. Its results may inform research but do not replace independent verification in consequential decisions.

7 Ethical Considerations

  • The benchmark uses fictional, text-based scenarios rather than real individuals’ private records.Cue sensitivity is not a measure of factual correctness or a guarantee of trustworthy judgment.Source: §7 · p. 9
  • Stronger reliance on a particular cue does not necessarily imply better decisions in real-world settings.The authors caution against treating the benchmark as a substitute for independent evidence verification in consequential decisions.Source: §7 · p. 9
  • Selective evidence presentation or misleading source descriptions could be used to steer model judgments.The authors frame the benchmark as a tool for evaluating and improving model behavior.Source: §7 · p. 9

A Experiments details

The appendices specify the benchmark’s diagnostic tracks, request universes, model scopes, and analysis conventions, while keeping recognition, choice effects, and response coverage distinct.

A.1 Diagnostic Tracks setting

  • The benchmark’s five tracks test cue representation, cue integration, reliability utilization, cue competition, and verified reasoning.The tracks distinguish identifying a source’s reliability from the effect of swapping source histories on final choice, and separate event time from presentation order and report multiplicity from evidence lineage.Source: §A.1 · p. 11

A.5 Cue Competition and the Verified Boundary

  • The complete report-structure conditions are provided in Appendix C.7.The controls distinguish message count, relays, and underlying evidence records; answer-mapping controls and response coverage matter particularly for smaller models.Source: §A.1 · p. 11, §A.5 · p. 12

A.2 Protocols, Denominators, and Source Alignment

  • The unified benchmark evaluates 13 LLMs with 24,000 requests each, while separate diagnostic and intervention studies use different request universes.The diagnostic study evaluates six Qwen models with 41,020 requests each; the intervention evaluates three models with 50,000 requests each.Source: §A.2 · p. 11
  • Each scenario generates 48 requests, and the six-Qwen benchmark runs reuse 19,120 observations per model while adding 4,880 new requests.Identification and verified accuracies use planned denominators; choice contrasts require valid paired responses.Source: §A.2 · p. 11
  • The appendices distinguish eight local and four API systems and specify separate interval procedures and probability-scale contrast directions.The unified benchmark uses 5,000 whole-world bootstrap intervals, the diagnostic study uses 2,000 resamples, and the intervention uses at least 5,000 resamples with a fixed 48-comparison Holm family.Source: §A.2 · pp. 11–12

A.3 Full Benchmark Profiles

  • The full-benchmark profiles report valid-response, recognition, verified-accuracy, and signed cue-effect measures, with subset-specific valid-pair denominators.Table 4 covers completion and response validity; its non-valid response flags are not added again to the planned denominator.Source: §A.3 · p. 12, Table 4 · p. 12, Table 5 · p. 12

A.4 Source Identification and Historical Comparisons

  • Recognition tables separate unequal-history recognition, equal-history controls, initial-state identification, provenance, count extraction, and joint exact-count diagnostics.These scores answer different questions and should not be treated as interchangeable.Source: §A.4 · p. 12

A.5 Cue Competition and the Verified Boundary

  • Cue-competition tables separate repetition, speaker count, and evidence-root count, and retain order effects at each event time to expose possible cancellation.The Structural 120 and Naturalistic 80 profiles retain their own valid-pair denominators.Source: §A.5 · p. 12, Table 6 · p. 12

A.6 Factorial Interactions and Predictive Checks

The pilot quality–independence interaction did not hold in the frozen six-Qwen Core test, while the intervention comparisons preserve verified-state accuracy across all conditions. The appendices also document intervention components and robustness checks.

A.6 Factorial Interactions and Predictive Checks

  • All five estimable Holm-adjusted tests of the quality–independence interaction have p = 1.0, and the 2B test is not estimable.In the dedicated study, adding interaction terms changes heldout log loss by less than 0.0013 in every estimable Core comparison, distinguishing robust cue-profile differences from a universal interaction.Source: §A.6 · pp. 12–13

A.7 Intervention: Complete Component Comparisons

  • All 150,000 formal intervention requests are valid, with five conditions using the same underlying scenario evidence.Flat and Ledger share an identical fact set, +Rule retains the ledger verbatim, and Sham differs from Ledger by no more than 10% in token length under the three model tokenizers.Source: §A.7 · p. 13
  • The primary intervention comparison family covers three models, two scopes, four contrasts, and two endpoints.Component effects on time, repetition, speakers, and lineage are reported separately with pointwise intervals.Source: §A.7 · p. 13
  • 100% of verified decisions are correct and format-valid across all 15 model–intervention cells.The descriptive preservation margin was a decrease of no more than two percentage points.Source: §A.7 · p. 13

A.8 Additional Coverage and Robustness Tables

  • Additional tables report contributing Core worlds and descriptive relation-level history-swap estimates, while retaining the studies’ own resampling units.Requests are not pooled as independent observations for the diagnostic and intervention cluster checks.Source: §A.8 · p. 13

A.9 Coverage of Supplementary Results

The supplementary results retain additional diagnostics alongside the main-text cross-system profiles, cue contrasts, controls, and pre-specified intervention effects. Tables 8–10 identify selected Core effects and historical-source identification results.

A.9 Coverage of Supplementary Results

  • The main text reports cross-system profiles, identification–influence contrasts, source and temporal controls, and pre-specified intervention effects; supplementary material retains additional checks.Retained analyses include equal-history judgments, exact-count reconstruction, verified conditional denominators, null factorial findings, predictive checks, sham changes, conditional order contrasts, and baseline rerun consistency.Source: §A.9 · p. 13
  • Tables 8 and 9 cover Core history and factorial effects, plus temporal, presentation, and evidence-lineage controls.The supplied captions identify the estimates and intervals but do not provide their values.Source: Table 8 · p. 14, Table 9 · p. 14
  • Table 10 reports historical-source identification on Core, distinguishing equal-history cases from cases requiring source ordering.The caption defines H as 7/8 verified reports and L as 4/8.Source: Table 10 · p. 14

B Dataset Statistics

The benchmark uses a frozen 500-world evaluation set organized into three strata and varied across domains, discourse formats, evidence quality, and structural features. Its request design and counterbalancing support controlled comparisons across these materials.

B.1 Evaluation Set and Overall Scale

  • The frozen evaluation set contains 500 fictional worlds: 300 Controlled Core, 120 Structural Generalization, and 80 Controlled Naturalized worlds, all expressed as English text.The same world bank is used throughout; evidence such as photographs or logs is described textually, with no image, audio, or video inputs.Source: §B.1 · p. 14

B.2 Relation Families and Domains

  • The worlds span 12 relation families and 25 domains, with 20 worlds per domain; accompanying tables cover relation distributions and selected representation and report-structure diagnostics.The table captions identify their scope but do not supply numerical results.Source: §B.2 · pp. 14–15, Table 11 · p. 15, Table 12 · p. 15

B.3 Discourse Formats

  • Ten discourse formats vary surface organization independently of the underlying symbolic records, with each format appearing in 50 worlds.Source: §B.3 · p. 15

B.4 Evidence-Quality Manipulation

  • Evidence quality varies through target identifiability: high-quality records provide the complete target identifier, while limited-quality records omit part of it but retain the reported state.Each of the four textual realizations is assigned to 125 worlds; the manipulation does not directly declare a report true or false.Source: §B.4 · p. 15

B.5 Background Events and Structural Generalization

  • Ten background-event structures diversify contexts without trusted target-time verification, while five structural-generalization axes each contain 24 worlds.The five axes vary source count, evidence-graph topology, dependency patterns, relay-chain length, and role configuration.Source: §B.5 · pp. 15–16

B.6 Benchmark Request Composition

  • Each world yields 48 requests, producing 24,000 requests per model: 15,000 with an objective target and 9,000 unverified support-choice requests without a unique gold state.The benchmark includes 20,000 multiple-choice JSON requests and 4,000 integer-count requests per model.Source: §B.6 · p. 16

B.7 Counterbalancing

  • Candidate-option order and the source providing the shared evidence root are each balanced 250/250; Core tables also cover interaction estimates and a five-fold predictive comparison.For the predictive comparison, the caption states that lower is better and both models use the same folds and regularization.Source: §B.7 · p. 16, Table 16 · p. 17, Table 17 · p. 17

C Supplementary Prompts and Evaluation Protocol

The supplementary protocol separates identification probes from final decisions, manipulates competing evidence cues in controlled conditions, and specifies response validation. It also documents the intervention prompts and the scope of supplementary request-count and diagnostic tables.

C.1 Independent Context Branches / C.2 System Prompts

  • Each probe uses an independent context, so identification and final-choice questions are separate calls; prompts use only the fictional case and provide no few-shot answer demonstrations.Multiple-choice probes request a one-key JSON answer, while exact-count probes request one integer from 0 to 8.Source: §C.1 · p. 17, §C.2 · p. 17

C.3 Identification and Decision Probes

  • The probe suite separates initial-state identification, provenance and evidence-quality identification, source-history counts and comparison, and the final unverified support judgment.Ordinary P4 questions have no unique gold label unless the world is in a verified branch.Source: §C.3 · pp. 17–18

C.4 Factorial Integration Conditions

  • The 2 × 2 × 2 design crosses historical reliability, current evidence quality, and evidence-record ancestry while holding the initial state and event/query timing structure fixed.The R, Q, and I manipulations change source histories, target identifiability, and whether reports trace to distinct or shared records, respectively.Source: §C.4 · pp. 18–19

C.5 Reliability-Swap Conditions

  • The history-swap comparison changes which disagreeing source has the stronger verified history while holding current claims fixed, with limited-quality evidence and distinct record roots.The diagnostic uses four source-history profiles; the supplementary captions also identify sham and exact-count history conditions and a rerun baseline.Source: §C.5 · pp. 19–20, Table 24 · p. 20, Table 28 · p. 21

C.6 Text Order and Event Time / C.7 Report Multiplicity and Evidence Lineage

  • Temporal controls vary event time and presentation order independently, while report-structure controls distinguish message count, relay count, and independent evidence roots.Reordering paragraphs leaves timestamps unchanged; the report-structure table specifies contrasts that separate repetition, relay multiplicity, and independent roots.Source: Table 21 · p. 19, §C.6 · p. 20, §C.7 · p. 20

C.8 Verified Reasoning / C.10 Output Parsing

  • Verified branches append a trusted target-time record and have an objective answer; response parsing separately defines valid multiple-choice JSON and integer-count outputs.Multiple-choice format validity requires exactly one key with a legal option letter; exact-count validity requires one integer character from 0 to 8.Source: §C.8 · p. 20, §C.11 · p. 20, §C.10 · p. 20

C.11 Intervention Prompt Additions

  • The intervention preserves the original scenario and question, compares matched fact-set organizations, and appends a general cue-weighting principle in the +Rule condition.The principle gives no candidate-state answer; verified branches differ only in the target-time value and are scored by accuracy.Source: Table 18 · p. 18, Table 19 · p. 18, §C.11 · p. 20

C.9 Probe-Level Request Counts

  • Supplementary tables document probe counts, Core coverage and relation-stratified history swaps, plus sham/count diagnostics and factorial sensitivity tests.Other captions identify renderer-resampled intervention contrasts and predefined Structural and Naturalistic subset profiles; their captions alone do not establish outcome values.Source: §C.9 · p. 20, Table 25 · p. 21, Table 26 · p. 21
Loading 2610.12788v1…