PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Sharath M Shankaranarayana, Davor Runje, Jan Jannink

arXiv:2610.01006v1cs.AIcs.LG

TL;DR

Calibration alone does not establish whether a decision model’s uncertainty reflects chance or missing knowledge. This audit uses paired information interventions and targeted questions to test Jev, finding that confidence works on familiar closed-choice tasks but does not reveal missing knowledge, while questions about the case give sharper readouts than questions about Jev’s own knowledge.

  • Problem

    Calibration does not establish whether low confidence reflects chance or missing knowledge, nor whether confidence falls when a model moves beyond its knowledge.

    Source: §1 · p. 1

  • Method

    The audit varies supplied information for fixed items and asks separate questions about whether outcomes are settled, evidence suffices, or Jev knows the answer.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Jev’s confidence is calibrated on familiar closed-choice tasks but does not identify missing knowledge, whereas questions about the case give direct or sharper readouts on tested contrasts.

    Source: §1 · p. 1, §5 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    For tested familiar closed-choice questions, confidence can support abstention; add settled or enough questions for undetermined outcomes or incomplete context, and do not rely on known to detect memory gaps.

    Source: §5 · p. 8

    This is a case study of one version of one closed model, and no other system was run through the full protocol.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Decision models return probabilities intended for routing, abstention and automated action. Calibration makes those probabilities useful on average, but does not establish whether low confidence reflects chance or missing knowledge, nor whether confidence falls when a model moves beyond what it knows. We audit this distinction in Jev, a decision model, with over 15 public datasets and 6 generated task families, with paired interventions that vary the information supplied for a fixed item. Jev's confidence is calibrated on familiar closed-choice tasks but fails as an indicator of missing knowledge: with no answer-relevant information it assigns up to 0.80 to a salient option, and on news beyond an observed knowledge boundary it exceeds accuracy by 0.21--0.33, a gap that recalibration on earlier months does not close. Targeted yes/no questions give sharper readouts of the case: whether an outcome is settled (AUROC 1.00) and whether the evidence suffices (0.95, against 0.85 for confidence on the same items). Asking whether Jev knows the answer appears to flag fabricated entities and post-boundary news (0.91), but with realistic names or with dates removed it shows no advantage over answer uncertainty. Black-box knowledge audits therefore need explicit controls for surface cues. Code: https://github.com/Syntheme/beyond-answer-confidence.

1 Introduction

Decision models use probabilities to support consequential actions, but calibration alone cannot distinguish uncertainty from chance from missing knowledge. This audit tests whether Jev’s confidence responds to missing information and whether targeted questions reveal what its confidence does not.

1 Introduction

  • Decision-model probabilities are intended to support choices such as accepting an answer, abstaining, routing a request, or triggering human review.Their usefulness depends on whether they remain informative when the model lacks the knowledge needed for a case.Source: §1 · p. 1
  • Calibration alone cannot identify whether uncertainty reflects chance or missing knowledge, because a single predictive distribution does not determine that distinction [2] [6].A calibrated probability of 0.5 could describe either a fair coin or an unknown fact.Source: §1 · p. 1
  • Jev is calibrated on familiar closed-choice tasks, yet favors salient answers without answer-relevant information and becomes overconfident beyond an observed knowledge boundary.Recalibration on earlier months leaves the post-boundary confidence gap large.Source: §1 · p. 2
  • Questions about whether an outcome is settled or evidence suffices give direct readouts of case gaps, while asking whether Jev knows the answer mostly reads surface cues.The first two questions hold on tested contrasts; the familiarity question provides no evidence of memory-gap detection beyond confidence after measured cues are controlled.Source: §1 · pp. 1–2
  • The audit holds items fixed while varying supplied information and asks separately whether an outcome is settled, evidence suffices, or Jev knows the answer.It combines paired interventions, generated tasks with known ideal distributions, and cue-controlled follow-up variants across about 575,000 calls, 15 public datasets, and 6 generated task families.Source: §1 · pp. 1–2

2 Related work

Prior work studies uncertainty decomposition, black-box knowledge tests, and confidence as a predictor of correctness. This paper distinguishes its audit by testing responses to controlled missing information and by separating what follow-up questions measure.

2 Related work

  • A single predictive distribution does not determine the split between aleatoric and epistemic uncertainty [2] [6], and decomposition methods face supervision and disentanglement limits [6] [2] [17] [30].The distinction concerns intrinsic randomness versus the learner’s lack of knowledge of p*(y|x).Source: §2 · p. 2
  • Black-box tests have probed irrelevant context [36], knowledge boundaries [28], fabricated or unanswerable questions [22], temporal cutoffs [10], and self-evaluation [21] [37] [47].Related approaches also include iterative prompting and human label distributions [5].Source: §2 · p. 2
  • Prior probability studies examined coin and dice contexts [27] [46], disagreement between token and verbalized probabilities [44], and Jev’s fair-draw and benchmark calibration behavior.Concurrent work tests fabricated entities and post-cutoff events across many LLMs [35].Source: §2 · p. 2
  • Unlike studies asking whether confidence predicts correctness, this audit holds aleatoric uncertainty fixed by design and separates the targets of different follow-up questions.Other Jev work examines confidence for routing to a stronger model [25] and semantic choices in scientific pipelines [11].Source: §2 · p. 2

3 Setup

The setup audits Jev through paired information changes, generated tasks with known ideal distributions, and separate questions about the case and Jev’s familiarity. It defines operational checks and metrics, while noting that the checks rely on controlled task constructions.

3 Setup

  • Confidence is Jev’s chosen-answer probability, pmax; the study reports top-label SmoothECE, error or case AUROC, normalized entropy, and total variation to an ideal distribution.Confidence is called calibrated when SmoothECE is at most 0.05 on a task’s closed-choice form, which does not establish classwise or subgroup calibration.Source: §3 · p. 2
  • The audit checks reducibility, honest ignorance, invariance to non-evidence, calibration at each knowledge level, and a separate readout of missing knowledge.The checks assume controlled symmetric tasks and form an audit checklist, not proof that a model knows what it does not know.Source: §3 · pp. 2–3
  • The studies vary intent information, test knowledge-versus-chance scenarios with known ideal distributions, and probe boundaries using popularity, fabricated entities, and dated news.Intent labels are shown as opaque codes or with examples and names; generated scenarios include stated answers, unknown answers, and future random draws.Source: §3 · p. 3
  • Evidence-in-context tasks include HotpotQA paragraphs, Quizbowl clues, and generated policy cases with facts edited, deleted, contradicted, or joined by irrelevant information.Random devices and answer formats include dice, coins, cards, and stated odds asked as a Choice or one yes/no question per outcome.Source: §3 · p. 3
  • The study also uses standard benchmarks, out-of-scope detection, an open decision model, and a deep ensemble; most free-text benchmarks become four-option Choice tasks.Paired input changes support causal claims about Jev’s response, while comparisons across popularity, dates, and benchmarks are observational.Source: §3 · p. 3
  • Each call asks whether the answer is settled, whether the supplied paragraphs suffice for certainty, and whether Jev knows the answer rather than guessing.For dated news, the familiarity question asks whether Jev knows for certain how the event turned out.Source: §3 · p. 3
  • The intent dose–response table compares accuracy, mean confidence, and SmoothECE across what each intent code contains, scoring swapped examples against original labels.The top-probability table compares cases with no answer-identifying input against an ideal uniform distribution over options not ruled out; false-premise cases use 0.25 only as a forced-choice reference.Source: Table 2 · p. 4, Table 3 · p. 4

4 Results

Jev’s confidence is calibrated on familiar closed-choice tasks but remains a poor indicator of missing knowledge, including beyond an observed accuracy boundary. Targeted questions can more directly assess whether an outcome is settled or evidence suffices, while apparent signals of Jev’s own familiarity weaken under surface-cue controls.

4.1 Confidence: trustworthy when familiar, blind at the edge

  • On familiar tasks, Jev is calibrated (CLINC150 SmoothECE 0.015–0.025; PopQA 0.020–0.033) and follows swapped example mappings on 90% / 98% of items.At a 𝑝max threshold of 0.9, it answers 8,132 of 9,960 TriviaQA questions with 20 errors.Source: §4.1 · p. 4
  • With no answer-relevant information, accuracy is at chance but Jev puts a third of its mass on the first numbered code and 0.80 on “one” for a fair die.An open decision-model comparison is almost uniform on the same inputs, while an uninformative “made with no information” cue raises the guessed option by 0.65.Source: §4.1 · p. 4
  • After the late-2024 accuracy break, confidence rises as accuracy falls; Jev says “no” on 92% versus 61% before, although true “no” shares stay near 51–52%.Recalibration on earlier months leaves post-boundary questions 0.19–0.23 overconfident; removing dates changes answers but can also remove event identity.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 What targeted questions reveal, and what they do not

  • For fixed-but-unknown facts versus future random draws, the settled question separates cases (mean P(yes) 0.87 vs. 0.13); confidence’s separation reflects mode-seeking miscalibration.Answer uncertainty has AUROC 0.96 overall but 0.89 even in uniform-chance scenarios, where the ideal answer distributions are identical.Source: §4.2 · p. 5
  • On the same HotpotQA conditions, P(enough) separates complete from incomplete evidence at AUROC 0.95 versus 0.85 for confidence, and falls from 0.83 to 0.10 when a policy fact is deleted.Within 10-word length strata, enough retains AUROC 0.95 while length falls to 0.48.Source: §4.2 · pp. 5–6
  • Known scores 0.91 on original fabricated-entity and post-boundary contrasts, but with realistic names (0.74 vs. 0.76 for answer uncertainty) or dates removed (0.58 vs. 0.56), it shows no demonstrated advantage.The figure contrasts original cue-rich conditions with realistic-name and date-masked conditions.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3 Putting the questions to use

  • Combining P(known) with confidence raises dated-news error-detection AUROC from 0.61 to 0.66; a not-known option beats confidence thresholding by +0.05 at 50% post-boundary prevalence.At the date-panel’s 27% share, the difference is +0.01 [−0.00, 0.04]; saying context may be irrelevant restores HotpotQA accuracy from 0.75 to 0.84.Source: §4.3 · p. 6, Figure 4 · p. 7

5 Discussion

Jev’s confidence and targeted follow-up questions serve different roles: case-level gaps are better queried directly, while the familiarity question does not demonstrate detection of the model’s own knowledge gaps after measured cues are controlled. Practical use is limited to the tested settings and depends on wording, input mix, and costs.

5 Discussion

  • On familiar, in-scope closed-choice questions like those tested, confidence can support abstention, but targeted questions more directly assess whether an outcome is settled or evidence is sufficient.The paper did not evaluate deployment utility under other prevalences, costs, or shifts.Source: §5 · p. 7
  • The familiarity question shows no demonstrated advantage over confidence after measured cues are controlled, and the follow-up labels are proxies rather than a general epistemic state.Each question captures only the gap it asks about.Source: §5 · pp. 7–8
  • Evaluate a “not known” option at the expected input mix and error costs; its matched-coverage accuracy benefit appeared only when later-period questions were common.At the panel’s mix the result was inconclusive, and a simple rule skipping questions with the most recent dates came within 0.01–0.02 of it.Source: §5 · p. 8

6 Conclusion

Jev’s confidence tracks correctness on the familiar closed-choice benchmarks tested, but does not reveal missing knowledge. Targeted questions better assess particular gaps in the case, while the familiarity question remains vulnerable to surface cues.

6 Conclusion

  • Confidence tracks answer correctness on the familiar closed-choice benchmarks tested, but does not show what Jev does not know.With no answer-relevant information it backs a salient guess, and beyond its knowledge boundary confidence rises in a way earlier-month recalibration does not repair.Source: §6 · p. 8
  • Asking whether an outcome is settled or whether enough information is supplied gives sharper case-level readouts than answer confidence.Asking whether Jev knows the answer mostly reads surface cues.Source: §6 · p. 8
  • Gaps in the case can be queried, but gaps in the model’s knowledge remain hard to see from outside.The conclusion calls for explicit controls that remove readable surface cues when auditing knowledge gaps.Source: §6 · p. 8

Limitations

The paper’s limitations constrain how broadly its findings and follow-up signals can be interpreted. The appendix also includes an example Jev response with answer probabilities and a settled judgment.

Limitations

  • This is a case study of one version of one closed model; no other system received the full protocol, so the findings are not claimed to generalize to decision models or LLMs.The temporal knowledge boundary is an accuracy change point in dated news, not a verified training cutoff, and topic or style may change with it.Source: Limitations · p. 8
  • Follow-up questions are evaluated on curated contrasts, and the known, text-only controls do not certify that no surface cue remains.The settled judgment concerns the input; evidence sufficiency was checked against text length and length features only.Source: Limitations · p. 8
  • Removing dates can change a question’s proposition or event reference, so those results report response changes rather than validated accuracy.The provisional labels for 240 pairs came from a single annotator following a written protocol.Source: Limitations · p. 8

A Example request and response

  • In the example response, Jev chooses red with probability 0.60 and confidence 0.46, while its settled response is no with probability 0.89.The accompanying text says a second follow-up asked whether the facts determine the answer with certainty.Source: A Example request and response · p. 11

B Statistical methods and robustness

The robustness analyses examine single-call stability, interval coverage, the dated-news change point, and the limits of confidence-only recalibration. They find that single-call estimates are close to three-call estimates, but earlier-month recalibration leaves substantial post-change overconfidence.

B Statistical methods and robustness

  • The statistical methods use 95% item-clustered bootstrap intervals and resample paired AUROC comparisons by original question or generated scenario.Effect sizes and intervals are reported rather than pass/fail tests.Source: B Statistical methods and robustness · p. 11

B.1 A single call

  • Across 500 random single-call draws, SmoothECE stays within 0.010 and accuracy within 0.01 of three-call values in all 36 calibration sets.For SimpleQA Verified, the three-call SmoothECE is 0.051 and falls below 0.05 in 11% of draws.Source: §B.1 · p. 11, Table 5 · p. 11

B.2 Coverage of the SmoothECE intervals

  • At a true SmoothECE of exactly 0.05, the percentile bootstrap upper bound falls below 0.05 in 1–6% of datasets, near the nominal 5% rate.The estimator is biased upward by 0.010–0.025 for perfectly calibrated data, and only one overconfidence shape was simulated.Source: §B.2 · p. 12

B.3 The knowledge change point

  • After the dated-news change point, confidence exceeds accuracy by 0.21–0.33, while confidence rises by +0.02 to +0.07.Every fixed cutoff from 2024-06 to 2025-03 gives a confidence–accuracy gap of 0.26–0.32.Source: §B.3 · p. 12

B.4 Recalibration at matched confidence values

  • Forward-in-time confidence-only recalibration leaves post-change overconfidence of +0.19 with an expanding window and +0.23 with a map frozen at the change point.A period-aware isotonic map closes both gaps; the comparison is a post-hoc result for this news panel.Source: §B.4 · p. 12, Table 6 · p. 12

C.1 Surface-cue controls for the follow-up questions

Surface-cue controls show that apparent self-knowledge largely disappears with realistic names or date removal. Date masking also changes some questions’ meaning, so the reported comparison is a response change rather than validated accuracy.

C.1 Surface-cue controls for the follow-up questions

  • Removing dates (which changed 6,274 of 6,320 news questions) reduced post-change “no” answers from 92% to 67%; accuracy is reported only for answer-safe subsets.Source: §C.1 · p. 13
  • With realistic names the known-minus-answer-uncertainty difference is −0.020, and with dates removed it is +0.023; dated news retains a +0.526 difference.The table reports AUROC for known contrasts with and without surface cues, with rows using made-up/real names or after/before items.Source: §C.1 · p. 13, Table 8 · p. 13

C.2 Date removal and answer labels

  • Among judged answer-preserving post-change pairs, “no” responses fall from 92% to 65%, a drop of 0.27 [0.14, 0.43].Thirty of 112 “yes” pairs lost event identity, so not all “yes” questions were answer-safe.Source: §C.2 · p. 13
  • The same answer-preserving comparison before the change point falls from 39% to 35%, a drop of 0.04 [−0.04, 0.13].Setting today’s date just after each event leaves post-change accuracy at 0.514 and “no” responses at 91%.Source: §C.2 · p. 13

C.3 Proper scoring rules

Across the reported tasks, Jev’s confidence and answer distributions often fail to reflect stated chance, missing evidence, or scope boundaries. Separate readouts can detect some case properties, but their performance varies with the contrast and surface cues.

C.3 Proper scoring rules

  • Brier/log loss are 0.919/2.36 for the future draw, 1.321/2.98 for a fair die, and 1.000/4.61 for a loaded die, above their ideal scores.The respective ideal Brier/log-loss pairs are 0.608/1.08, 0.833/1.79, and 0.700/1.50; Table 10’s log-loss floor bounds loss on exact-zero probabilities.Source: §C.3 · p. 14, Table 10 · p. 14
  • GLiNER2.5-Decide is nearly uniform with no information but underconfident when informed; a five-member MiniLM ensemble remains cautious with one example while Jev is confident.The ensemble’s one-example confidence is 0.43–0.54 versus Jev’s 0.83–0.90; GLiNER2.5-Decide has pmax 0.34 at accuracy 0.66 on CLINC150 with names.Source: §C.3 · p. 14
  • TriviaQA, SimpleQA Verified, and TruthfulQA binary accuracy is 0.955, 0.753, and 0.909 respectively; P(answerable) detects SelfAware unanswerability at AUROC 0.882.P(single answer) only weakly flags AmbigQA ambiguity (0.661), and ChaosNLI entropy rank-correlates with human-label entropy at ρ=0.35 but is strongly compressed.Source: §C.3 · p. 14
  • An out-of-scope option flags 89% of off-topic CLINC150 requests at 3.5% false alarms (AUROC 0.982), but flags only 36% of held-out Banking77 intents with close schema relatives.Source: §C.3 · p. 15
  • On random devices, Jev assigns 0.80 to “one” for a fair die and 1.00 to the 50% face of a loaded die; averaging option orders leaves TV at 0.632 vs. 0.634.With tied chances, the choice follows the order in which chances are mentioned 99.6% of the time, not display order.Source: §C.3 · p. 15
  • For made-up events, P(settled) separates past from future with AUROC 1.000 from tense alone, 0.996 from dates with today’s date, and 0.898 from dates alone.Mean P(settled) is 0.73 vs. 0.14 using tense alone; the passage concludes Jev does not place itself in time unless told.Source: §C.3 · p. 15
  • For dated news, the “not known” option nearly eliminates confident errors—6 of 1,191 answers versus 330—but the count compares three-way and two-way formats.Its advantage over a confidence threshold varies by the share of post-change questions; the comparison is coverage-matched.Source: §C.3 · p. 15
  • Supporting paragraphs raise HotpotQA accuracy and confidence together, while non-supporting paragraphs reduce accuracy without a corresponding confidence drop; P(enough) rises only with complete evidence.Quizbowl confidence and accuracy rise together clue by clue.Source: Figure 7 · p. 16

D Experiment ledger

The experiment ledger counts paid calls from cached records and separates analysed calls from superseded or stray calls.

D Experiment ledger

  • Of 575,442 paid calls, 573,234 feed reported analyses and 2,208 are superseded or stray calls included in cost.Each paid call is one cached record attributed to the first experiment using it; dates are 2026 (UTC).Source: D Experiment ledger · p. 16
  • The intent study’s 27,240 units comprise a 2,270-item main arm plus two robustness arms on 681 items.Table 13 lists items, replicates, and paid calls per experiment; its asterisk marks a plan committed after the run and not externally verifiable.Source: D Experiment ledger · p. 16, Table 13 · p. 16

E Datasets

The dataset inventory combines converted public benchmarks with six generated task families.

E Datasets

  • Free-text benchmarks were converted to four-option Choice using dataset-specific distractors, while options were seeded-random except for MMLU’s original order.ChaosNLI and SelfAware mirrors were checked against independent copies.Source: E Datasets · p. 17
  • The six generated sets cover knowledge and chance, fabricated entities, random devices, policy cases, made-up events, and adversarial wording tests.They include 480 knowledge×chance scenarios, fabricated-entity items with pseudo-word or look-alike names, and 360 made-up events.Source: E Datasets · p. 17
Loading 2610.01006v1…