Source-linked AI summary
Truth for Believable AI: Expressed Doubt, Provenance, and Belief Revision as an Engineerable Stance
Sebastian Cochinescu
TL;DR
Conversational agents can answer unknown questions in the same confident register as known ones. This paper builds a behavior layer for expressed uncertainty, provenance-aware assertion, and persistent belief revision over a fixed model, then evaluates it on a constructed benchmark. The evaluation supports an audit guarantee and partial correction discrimination, while expression checks fail for the prespecified configuration and the later consistency-gated result remains selection-dependent.
Problem
Conversational agents can use the same fluent, unhedged register for answers they cannot answer and answers they can.
Source: §1 · p. 1Method
Results
Acknowledgment soundness holds by construction, true corrections are accepted more often than false ones on held beliefs (0.444 vs. 0.145), and prespecified expression-fidelity and provenance margins fail.
Source: §1 · p. 2, §7.2 · p. 8, §8 · p. 13Takeaways & Limitations
The supported conclusions are a by-construction audit guarantee, store-dependent partial correction discrimination, and a benchmark- and model-specific failure of token-probability gating.
Source: §8 · p. 13, §9 · p. 13The benchmark is constructed, and fact-clustered uncertainty leaves both conversation-level criteria unresolved at 60 facts, so the fact base must expand before human evaluation.
Source: §9 · p. 13, §8 · p. 13
Abstract
from arXiv · showhide
Conversational agents often express answers in a uniformly confident register. We test whether expressed uncertainty, provenance-aware assertion, and explicit belief revision can be implemented as a behavior layer over a fixed language model; we do not test believability or trust. The layer combines three epistemic states, per-claim confidence and typed provenance, a provenance-gated expression rule, and a persistent revision store with auditable acknowledgments and partial resistance to false corrections. We evaluate it on a constructed, mechanically scored multi-session benchmark using a synthetic model and Qwen2.5-0.5B-Instruct. The synthetic instrument passes all five checks. On the real model, acknowledgment soundness, a by-construction guarantee, holds in 100% of cases, and true corrections are accepted more often than false ones (0.44 vs. 0.15 on held beliefs; 0.875 vs. 0.420 including rule-accepted corrections of unheld facts), but the pre-specified expression-fidelity, contradiction-separation, and provenance margins fail. A disclosed post hoc analysis shows that expression gated on mean answer-token probability ranks correctness below chance end to end (AUC 0.41, conversation-clustered), whereas gating on sampling consistency discriminates (AUC 0.66). A consistency-gated configuration selected from this finding and evaluated under a separately committed protocol meets the conversation-level manipulation and capability-equivalence criteria and replicates on a redrawn conversation set. The manipulation result is selection-dependent, and both criteria remain unresolved when uncertainty is clustered over the 60 facts. The supported conclusions are limited to the by-construction audit guarantee, store-dependent partial correction discrimination, and a benchmark- and model-specific failure of token-probability gating; scaling the fact base is required before human evaluation.
1 Introduction
The paper targets a behavior layer that makes uncertainty, sources, and belief revision visible and consistent, rather than treating accuracy alone as the goal. On the real model, several prespecified checks fail, while a separately committed consistency-gated configuration passes conversation-level checks with important caveats.
1 Introduction
- The motivation is that fluent, unhedged answers fail to distinguish knowledge from guesses, retrieval from inference, or user-provided facts from trained knowledge.Accuracy-only benchmarks can reward confident guessing and penalize “I don’t know”.Source: §1 · p. 1
- The study measures an engineered behavior layer, not believability, trust, or perceived mind, which are deferred to a perception study.Its evidence is synthetic, seeded, single-machine measurement mechanically scored against generator-injected ground truth.Source: §1 · pp. 1–2
- The behavior layer combines three-state expression, typed provenance, persistent belief revision, and auditable acknowledgments, extending beyond single-turn linguistic calibration.Each mechanism is varied by a dedicated ablation arm; the arm contrasts are single realizations and do not causally isolate each mechanism.Source: §1 · p. 2
- The real model fails three prespecified margins, while a separately committed consistency-gated configuration passes conversation-level checks but remains selection-dependent.Both criteria remain unresolved when uncertainty is clustered over the 60 facts, so the fact base must be expanded before human evaluation.Source: §1 · p. 2
2 The truth stance and its measurement
The paper treats the truth stance as visible behavior, distinct from accuracy or calibration. It uses evidence-sensitive expression fidelity as a manipulation check, while noting that the chosen check has limitations.
2 The truth stance and its measurement
- The truth stance is a disposition to expose epistemic state, not statistical calibration or answer accuracy.Keeping capability fixed makes the stance manipulable and makes ablations over the same base model meaningful.Source: §2 · p. 2
- Expression fidelity tests whether expressed-confidence categories track accuracy, distinguishing evidence-sensitive doubt from blanket hedging.Expression-ECE operationalizes evidence congruence, but evidence congruence and accuracy tracking are close rather than identical; anchor-ECE is structurally in tension with provenance-based confidence caps.Source: §2 · p. 2
3 A formal expression-and-revision layer
The layer assigns each claim confidence and an operational provenance tag, then uses these to govern expression and persistent belief revision. Its audit guarantee links public acknowledgments to logged accepted changes, not to the truth of revised values.
3 A formal expression-and-revision layer
- Each claim carries extractor-specific confidence and a pipeline-path provenance tag; the combined confidence takes the minimum across extractors.Provenance comes from instrumentation rather than a classifier, and the boundary between parametric and inferred content is acknowledged as ambiguous.Source: §3 · pp. 2–3
- Three expressible states— not-knowing, unsure, and confident—map to decline, hedge, and assertion behavior, while wrong-revising is triggered only by revision.The thresholds θfloor = 0.15 and θhigh = 0.75 were specified before evaluation; in the released implementation, wrong-revising is logged rather than returned by the state assigner.Source: §3 · p. 3
- The expression gate maps provenance and confidence band to decline, hedge, or assert; inferred claims never bare-assert and told claims are not asserted as the agent’s own knowledge.Table 1 lays out provenance class by combined-confidence band; its bottom row is the no-provenance ablation’s row-independent map.Source: §3 · p. 3, Table 1 · p. 3
- The persistent store accepts corrections to held beliefs when confidence is weak or retrieval corroborates them, while stronger uncorroborated beliefs resist correction.Corrections to absent claims are accepted unconditionally; the weak-belief rule leaves an adversarial user able to rewrite weak beliefs without corroboration.Source: §3 · pp. 3–4
- Every public error acknowledgment must link to an accepted revision-log entry, but this guarantee establishes neither the utterance’s wording accuracy nor the revised value’s truth.Corroboration consults the retrieval store, never ground truth.Source: §3 · p. 4
- The self-model term is a role-play-bounded design element: reports about the agent’s own store are generated behavior, not privileged access.The paper excludes introspection claims because its states are computed pipeline quantities.Source: §3 · p. 4
- In the worked example, a strong uncorroborated correction is resisted, a corroborated correction is accepted and acknowledged, and an unanswerable query is declined.A re-ask after accepted revision is not scored as a contradiction because the change follows an accepted revision-log entry.Source: §3 · p. 4
4 Related work
Related work treats confidence estimation, expression, abstention, attribution, and belief revision largely as separate components. The paper positions its contribution as their combination in a persistent, user-facing epistemic interface.
4 Related work
- Prior work addresses layer components separately; the paper combines them into a behavioral epistemic interface with auditable correction acknowledgment and ablation-compared mechanisms.The authors state that, to their knowledge, prior work does not combine these elements.Source: §4 · p. 4
- Linguistic calibration maps correctness estimates to hedging, but the cited work lacks persistence, provenance typing, and revision.The discrete confidence levels themselves are not claimed as this paper’s contribution.Source: §4 · p. 4
- Abstention work is often a static answer-or-refuse boundary, whereas this layer makes abstention one endpoint of a graded, provenance-typed policy over persistent state.Attribution work annotates or gates generation on source support but does not impose a graded cap on assertion strength [2] [4] [9] [10].Source: §4 · p. 5
- The store-level revision is situated in the AGM tradition, contrasting with weight-level knowledge editing [24] [25] [42] and user-facing communication-focused revision evaluation.The cited AGM operators are lineage, not implemented logic; existing behavioral revision work evaluates internal correctness rather than user-facing epistemic communication [37].Source: §4 · p. 5
- Unlike dialogue-consistency and long-horizon-memory benchmarks, this benchmark injects contradictions and corrections so revision behavior is mechanically scorable [27] [35].The paper treats hedging as an epistemic semantic device, not a politeness strategy.Source: §4 · p. 5
5 Implementation
The implementation layers confidence extraction, provenance-aware expression, and persistent belief revision over a fixed model. It fixes resolution and provenance handling for reproducibility, while the prototype remains an instrument rather than a deployed product.
5 Implementation
- Answer resolution follows a fixed order—store, parametric query, retrieval, then one-step derivation—and the path taken determines the provenance tag.Greedy decoding was frozen for determinism; retrieval uses a small document store designed to allow conflicts.Source: §5 · p. 5
- Two extractors provide confidence signals: mean generated-token probability and agreement between sampled answers and the greedy answer.Consistency extraction uses k=8 sampled generations; the realized retrieval store contains 34 of 60 subjects, including one stale subject.Source: §5 · pp. 5–6
- The instrument is reproducible from one command per stage, with version-controlled protocol constants and SHA-256 checks that reject drift in frozen files.It has no UI or serving stack.Source: §5 · p. 5
- 0.08 ms median orchestration overhead per 24-turn conversation, versus 0.005 ms for a bare cached lookup; model calls dominate end-to-end cost.The overhead was measured over cached outputs on one machine; consistency sampling adds k=8 generations per new fact, amortized by a per-fact cache.Source: §5 · pp. 5–6
6 Evaluation protocol
The evaluation uses a seeded synthetic stage and a pre-specified grid on a pinned real model, with fixed configurations and conversation-level metrics. A separately committed Stage-C protocol tests the consistency-gated configuration identified by the Stage-B mechanism finding.
6 Evaluation protocol
- Stage A tests the instrument against a seeded synthetic world with injected errors; Stage B uses Qwen2.5-0.5B-Instruct on a 60-fact geography world.Synthetic ground truth is knowable by construction; the real model has no injected corruption, and ambiguous capitals are excluded under an administrative-capital convention.Source: §6 · p. 6
- The seven configurations vary the three-state policy, provenance, acknowledgment, confidence thresholds, and persistent storage.Controls include always asserting, always hedging, silent revision, no provenance, and stateless operation.Source: §6 · p. 6
- The benchmark comprises 120 seeded three-session conversations of 24 turns, with half of statements and corrections made false by injection.Labels come from the generator’s injection record, and versioned instances and a language-agnostic scorer are provided.Source: §6 · p. 6
- Stage B’s pre-specified margins cover expression fidelity, revision behavior, provenance, and capability equivalence; Table 3 reports the Stage-B margin verdicts.The margins were set by design judgment without prospective power analysis; achieved precision is reported instead.Source: §6 · p. 6, Table 3 · p. 7
- Stage C evaluates the consistency-extractor-only configuration under its own freeze, committed before that grid ran, with previously known Stage-B values recorded.Stage-B protocol freeze and results first appear in the same repository commit, so the authors call those tests pre-specified, not preregistered.Source: §6 · pp. 6–7
7 Results
The synthetic instrument passes all five checks, but the real-model evaluation misses several pre-specified margins. A separately committed consistency-gated configuration passes conversation-level manipulation and capability-equivalence criteria, though those results remain unresolved under fact-level clustering.
7.1 Stage A: the instrument behaves as designed
- The synthetic instrument passes all five checks, including expression-ECE, provenance gating, acknowledgment auditability, and correction discrimination.These Stage-A results validate the instrument and are not the paper’s headline results.Source: §7.1 · p. 7
7.2 Stage B: pre-specified verdicts on a real model
- All 42 acknowledgments trace to accepted revision-log entries, a soundness check guaranteed by the emission rule rather than a behavioral discovery.Only 31 of the 42 flags render an admission utterance; the other 11 flag retrieval conflicts at response time.Source: §7.2 · p. 7
- The rule accepts more true than false corrections: 0.444 versus 0.145 for held beliefs, and 0.875 versus 0.420 across the scored set.The held-belief contrast depends on the constructed retrieval store’s coverage and freshness; scored-set rates also include unconditional acceptance of unheld claims.Source: §7.2 · pp. 7–8
- Capability equivalence passes the specified criterion, with answered accuracy 92.4% for full versus 90.6% for uniform, at lower coverage (0.833 versus 0.925).Accuracy is conditional on each arm’s answered subset, so the comparison does not establish unconditional task performance.Source: §7.2 · p. 8
- The pre-specified expression-fidelity, contradiction, and provenance margins fail on the real model for distinct metric, decoding, and attribution reasons.Anchor-based ECE favors uniform assertion near the 0.9 anchor; deterministic greedy decoding leaves stateless contradiction at 0.029 against a 0.03 margin; the provenance gap is 0.0123 against 0.02.Source: §7.2 · p. 8, Figure 1 · p. 10, Figure 2 · p. 11
- Consistency-gated expression discriminates correctness (AUC 0.661), while logit-gated expression ranks correctness below chance (AUC 0.410) under conversation clustering.The logit-gated result is unresolved under fact-level resampling; the consistency-gated Stage-C manipulation result was selected after its Stage-B value was known.Source: §7.2 · p. 9, Figure 1 · p. 10
- A separately committed consistency-gated configuration passes the conversation-level manipulation and capability-equivalence criteria, but both remain unresolved under fact-level clustering.On a redrawn conversation set, manipulation AUC is 0.677 (95% CI [0.635, 0.726]) and capability equivalence is ∆= +0.023 (95% CI [0.009, 0.036]); fact-level intervals include chance and exceed the equivalence bound.Source: §7.2 · pp. 9–10
8 Ablation handoff and evidential scope
Stage B does not qualify a condition pair because logit-gated expression fails the evidence-sensitivity criteria. Stage C’s consistency-gated configuration passes conversation-level criteria, but fact-clustered uncertainty and selection dependence limit the result.
8 Ablation handoff and evidential scope
- Stage B fails both the original ECE criterion and the post hoc AUC criterion because logit-gated expression is anti-discriminative on this model.Capability equivalence passes, but Stage B does not provide a qualifying condition pair for the downstream perception study.Source: §8 · p. 13
- Stage C passes both conversation-level criteria and replicates on a redrawn conversation set, but fact-clustered intervals leave both criteria unresolved at 60 facts.The selected configuration’s AUC is not independent confirmation because it motivated the selection; the fact base must be expanded before human evaluation.Source: §8 · p. 13
- Acknowledgment soundness is a by-construction guarantee verified in all audited events, while held-belief corrections are accepted more often when true than false (0.44 vs. 0.15).The broader 0.875 vs. 0.420 rates include unconditional acceptance of corrections to unheld claims; the correction effect depends on retrieval-store coverage and freshness.Source: §8 · p. 13
9 Limitations and scope
The layer engineers expressed epistemic behavior rather than knowledge, and the paper’s findings are bounded by its model, benchmark, and lack of human-subjects evidence.
9 Limitations and scope
- The layer is designed to produce calibrated expression, not knowledge; when the base model is wrong, it can express the wrong answer fluently.The benchmark tests mechanically scored, injected cases rather than open-domain consistency, and the paper makes no claim of access to real internal states.Source: §9 · p. 13
- The demonstrated results are scoped to one 0.5B model, one 60-fact world, and a consistency extractor; larger or better-calibrated models may place the extractor-dependence boundary elsewhere.The paper identifies the unnormalized mean-probability feature’s length bias as a rival explanation and leaves sampled-decoding replication of the contradiction metric for future work.Source: §9 · p. 13
- Whether expressed doubt, visible revision, and source-aware caution change perceived trustworthiness or mind-likeness remains a deferred human-subjects question.The paper reports no human-subjects data and does not claim trust, believability, or perceived mind.Source: §9 · p. 13
10 Series context
This paper contributes the truth dimension to a research program on believability as dimensional completeness, alongside work on temporal continuity and structured variation. Its methods and claims are self-contained, and the project materials are publicly available.
10 Series context
- The paper operationalizes the truth dimension in a research program that also includes temporal continuity and structured variation.A future preregistered perception study is intended to test the resulting ablation pairs; the series links provide motivation, while this paper’s methods and claims are self-contained.Source: §10 · p. 14
- The public companion repository includes the layer, versioned benchmark, frozen protocol, results files, and reproduction scripts.The manuscript snapshot and benchmark are also archived separately; no human-subjects or personal data were collected.Source: §10 · p. 14
- The archived benchmark is independently citable and released under CC BY 4.0.The full project snapshot is archived under an MIT license.Source: §10 · p. 14