PaperlayerView exact revision on arXiv

Source-linked AI summary

LADDER: Diagnosing Domain-Level Self-Assessment Under Noisy Feedback

Zhenshuai Yin, Jianshu Liang, Xuyue Xiong, Yongxin Guo, Mingda Zhang, Zhongze Luo, Haoyue Liu, Lianmin Zhang, Xiaoying Tang

arXiv:2610.34737v1cs.CL

TL;DR

Language models need reliable capability estimates, but interaction feedback can be missing or incorrect. LADDER evaluates domain-level self-assessment across information conditions and finds that accurate reporting of supplied statistics does not ensure trustworthy reconstruction from feedback or predictive value on new tasks.

  • Problem

    Because interaction feedback may be missing or incorrect, it is unclear whether models’ domain-level self-assessments agree with observed task performance, a prerequisite for judging when verification is needed.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    LADDER compares domain-capability estimates across five information conditions, from priors and noisy feedback to verified outcomes and supplied statistics.

    Source: §1 · p. 2

  • Results

    Across models, overall calibration and domain discrimination are distinct, and accurate reconstruction need not imply predictive benefit on new tasks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Trustworthy self-assessment should be evaluated against observed task performance and tested for predictive value on new tasks.

    Source: §3 · p. 3, §5 · p. 8

    LADDER focuses primarily on structured question answering, and its feedback comes from simulated users rather than real human collaboration.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Knowing when they are reliable is essential for language models to support human-AI collaboration. Yet interaction feedback is not always complete or correct, and may mislead models' understanding of their own capabilities. We introduce LADDER to study how language models assess themselves under unreliable feedback. Our findings show that models can accurately report supplied performance information while still struggling to form trustworthy self-assessments from interaction experience. Apparently accurate assessments can obscure capability differences and do not necessarily predict future task performance. These findings suggest that trustworthy self-assessment should be evaluated against observed task performance and its predictive value. Our study provides empirical evidence of the limits of model self-knowledge and a basis for evaluating capability awareness in reliable human-AI collaboration.

1 Introduction

LADDER asks whether language models can reconstruct domain-level performance from question-answering histories when user feedback may be missing or incorrect. Its diagnostic ladder separates information conditions and tests whether self-assessments match observed outcomes.

1 Introduction

  • Interaction histories can mislead capability estimates because users may accept incorrect answers, question correct ones, or provide no usable judgment.The study first tests whether profiles summarize observed performance before asking how profiles might affect decisions.Source: §1 · p. 1
  • The diagnostic contrasts test reporting supplied statistics, prior domain estimates, and performance reconstruction from observed feedback against controls with different information access.The study distinguishes errors associated with available information and its aggregation rather than inferring a unique internal cause.Source: §1 · pp. 1–2
  • Across 18 models, global feedback means outperform domain-specific Count, Beta, DS, and GLAD-style estimators on 14 of 18 models, while overall accuracy and domain discrimination remain distinct.LSA reaches CBF .989, .995, and .835 for GPT-5.5, DeepSeek-V4-Pro, and Qwen2.5-7B, respectively, while all three reproduce supplied statistics exactly.Source: §1 · p. 2
  • LADDER contributes a five-level diagnostic protocol, domain-level CBF with complementary error measures, and an 18-model study with matched-input, noise, partition, and cross-task controls.The controls examine when feedback improves estimated capability profiles.Source: §1 · p. 2

2 Related Work

Related work studies self-knowledge, calibration, interactive agents, memory, feedback learning, and noisy-label estimation. LADDER focuses on whether repeated interaction with imperfect feedback yields domain-capability estimates that match recorded performance.

Model self-knowledge and calibration

  • Self-knowledge work examines answer confidence (Kadavath et al., 2022), activation monitoring, and behavioral self-prediction (Binder et al., 2025), while SAD (Laine et al., 2024) and SelfAware assess self-knowledge without accumulated feedback history.Knowledge-boundary benchmarks characterize scored-question capabilities, while selective prediction and confidence calibration connect confidence to query-level judgments; LADDER extends these questions to profiles from repeated interactions with imperfect feedback.Source: §2 · pp. 2–3

Interactive agents and memory

  • Interactive-agent benchmarks examine progress, task completion, or predicted success, whereas LADDER compares estimated capability profiles with observed performance under imperfect feedback.Memory systems study retaining and using prior interactions, while RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2023) use feedback to update model parameters; LADDER evaluates estimates from interaction records with fixed model parameters.Source: §2 · p. 3

Imperfect feedback and statistical estimation

  • Dawid–Skene models annotator confusion and GLAD (Whitehill et al., 2009) relates annotator ability to item difficulty; LADDER adapts estimation ideas to domain performance and compares them with direct counting and a global feedback mean.The study examines how feedback quality and aggregation affect the resulting capability profile.Source: §2 · p. 3

3 LADDER

LADDER evaluates whether domain-level capability estimates derived under different information conditions agree with empirical performance from scored answers. Its protocol compares prior estimates, noisy-feedback learners, verified outcomes, and reporting supplied statistics.

3 LADDER

  • LADDER compares capability estimates against recorded answer outcomes because user approval or criticism alone cannot establish whether a model understands its capabilities.Its design examines whether prior assessments are accurate, feedback improves them, and models can report known performance.Source: §3 · p. 3
  • The benchmark records questions, answers, confidence, and natural-language feedback, then evaluates domain-accuracy profiles against empirical accuracy from scored outcomes.Estimators receive records permitted by their information condition after the session; answering models see the current question, and domain identifiers are supplied to estimators.Source: §3 · pp. 3–4
  • The main benchmark uses 50 sessions of 50 turns per model over 20 MMLU subjects, with three additional tasks used to test cross-task generalisation.The extension suite comprises GSM8K, HumanEval, and BFCL tool-calling.Source: §3 · p. 4
  • Capability Boundary Fidelity (CBF) measures how closely estimated per-domain accuracy tracks the truth, while ECE uses individual confidence–outcome pairs and CBF uses domain-profile pairs.A constant profile can score highly when domain accuracies are near its value, so CBF measures profile error rather than task competence.Source: §3 · p. 4, Figure 3 · p. 4
  • Baseline-adjusted CBF measures the fraction of the oracle gap closed relative to the constant no-memory baseline; positive b denotes overestimation, and lower eshape indicates better recovery of domain contrasts.Signed and centered errors complement absolute error rather than forming an additive decomposition.Source: §3 · pp. 4–5
  • The five information levels range from a constant profile and zero-shot domain priors through noisy-feedback estimators and verified-outcome counting to reporting supplied empirical statistics.L2 LSA reads question and feedback history; algorithmic estimators use a strictly decoded positive/negative label stream, with ambiguous labels treated as missing. The matched-input control fixes supplied labels to examine aggregation directly.Source: §3 · p. 5

4 Diagnostic Results: Where Does Agent Self-Knowledge Fail?

Across 18 models, the diagnostic results distinguish reporting supplied statistics from reconstructing performance through noisy feedback. They also show that overall accuracy, domain-profile accuracy, and prediction on held-out tasks are distinct measures.

4.1 Setup

  • The evaluation compares five information levels across 18 models using shared scored answers and feedback, with session means on common interaction histories and pooled scores combining histories.Each session profile covers domains with at least two scored answers.Source: §4.1 · p. 5

4.2 Main Results

  • Global leads five matched algorithmic controls on 14/18 session-level rows, but its high CBF measures overall-level recovery rather than domain differentiation.On GPT-5.5, Global scores 0.800 versus 0.603 for Beta and 0.609 for GLAD-style; centered shape error tests whether a profile captures domain differences.Source: §4.2 · p. 5
  • All 18 complete L4 evaluations reproduce supplied statistics at CBF 1.000, while clean-label LSA reaches .835 on Qwen2.5-7B.The contrast separates reporting supplied numbers from accurately aggregating a history.Source: §4.2 · pp. 5–6
  • L1 improves on constant L0 for 14 of 18 models, but prior error varies; Qwen2.5-7B scores 0.670 at L1 versus 0.740 at L0.Signed bias and centered shape error distinguish a misplaced overall confidence level from incorrect domain differences.Source: §4.2 · p. 6
  • Correcting observed labels raises mean pooled Count CBF from 0.852 to 0.886, while complete correct labels reach 1.000; the L2–L3 gap also reflects missing feedback and estimator behavior.The comparisons use frozen feedback rules; session and pooled fits use different evidence budgets.Source: §4.2 · p. 6

4.3 Diagnosis via the Ladder Visualization

  • Figure 4 shows mean session CBF across five levels; Figure 5 contrasts answer accuracy with profile CBF, whose model rankings differ from task-accuracy rankings.The level contrasts separate reporting, prior error, history use, and reconstruction; high profile CBF alone need not indicate domain discrimination.Source: §4.3 · pp. 6–7, Figure 4 · p. 7
  • R4 descriptively selects ZS when its CBF exceeds NoMem and LSA otherwise; because selection uses a scored target, held-out testing is needed to assess prediction on new tasks.The rule measures potential selection value on observed tasks, not generalization by itself.Source: §4.3 · p. 7

4.4 Cross-Task Heterogeneity and Profile Transfer

  • Across six ReAct configurations and 960 records, empirical Count profiles improve on a global training-accuracy predictor in two configurations, while zero-shot profiles improve in one.Estimator rankings also change across domain partitions, so domain structure remains a candidate prediction signal whose held-out benefit must be established.Source: §4.4 · p. 8

5 Conclusion

LADDER separates overall calibration from domain discrimination and numerical reporting from history aggregation. Its held-out analysis finds that accurate reconstruction does not necessarily translate into predictive benefit.

5 Conclusion

  • Across 18 models, LADDER separates overall calibration from domain discrimination and numerical reporting from history aggregation.Paired level and shape metrics show whether domain-specific estimates add useful information.Source: §5 · p. 8
  • Matched-input controls distinguish feedback quality, coverage, and aggregation as separate sources of profile error.Source: §5 · p. 8
  • Held-out ReAct results show that accurate reconstruction need not imply predictive benefit; the conclusion calls for larger agentic task sets and documented human-feedback studies.Source: §5 · p. 8

Limitations

LADDER’s conclusions are bounded by its structured-question-answering setting and simulated feedback; broader interactive settings require further evaluation.

Limitations

  • LADDER studies self-assessment primarily through structured question answering, so broader task settings require evaluation.The paper names multilingual, open-ended, and long-horizon interactive tasks as settings for broader evaluation.Source: Limitations · p. 9
  • Feedback comes from simulated users, while sustained human–agent interaction with changing goals and implicit judgments remains to be tested.The paper identifies testing whether self-assessments improve decisions in sustained interaction as future work.Source: Limitations · p. 9

Ethical considerations

The study uses public benchmark questions and LLM-simulated feedback, with no human participants; consequential use requires validation on the intended task and user population.

Ethical considerations

  • Domain-profile estimates may identify mismatches between reported and observed performance, but consequential use requires validation on the intended task and user population.The study uses public benchmark questions and LLM-simulated feedback, with no human participants; benchmark materials remain subject to their respective licenses.Source: Ethical considerations · p. 9

A Datasets

  • The primary pool has 500 questions across 20 MMLU subjects, and each model completes 50 sessions of 50 attempted turns.Each subject contributes 25 questions; interaction records include questions, answers, confidence, simulated feedback, and reference-based correctness labels.Source: A Datasets · p. 12
  • MMLU-Pro and BBH provide held-out subject and task comparisons, while GSM8K, HumanEval, BFCL, and ReAct support additional task-level comparisons.ReAct records support prediction on held-out questions; dataset-specific scoring and sample sizes accompany the corresponding results.Source: A Datasets · p. 12

B.1 Benchmark Details: Task, Protocol, Question Pool, Personas, Metrics

The benchmark estimates domain-level performance from structured interaction records and compares methods across an oracle-bracketed baseline ladder. Its setup also specifies subject tiers, simulated feedback, and auxiliary confidence measures.

B.1 Benchmark Details: Task, Protocol, Question Pool, Personas, Metrics

  • Each turn records a question, answer, confidence, feedback, domain, and outcome; outcomes score the target profile but remain hidden from noisy-feedback estimators.The answering prompt contains the question, while feedback may confirm, correct, or leave validity unclear.Source: §B.1 · p. 12
  • Each model completes 50 sessions of 50 attempted turns, with profile estimators consuming saved records rather than conditioning the next answer on their estimates.At each turn, the answering model reports an answer and confidence before the simulated user responds.Source: §B.1 · p. 12
  • The pool covers 20 MMLU subjects tiered per model into 6 strong, 7 medium, and 7 weak subjects; threshold and random-partition analyses test tier sensitivity.The additional 232 numerical-answer turns in the math-word-problems domain of each Mistral/Llama archive are excluded from the 20-subject MCQ target.Source: §B.1 · p. 12
  • Auxiliary measures include hallucination rate and expected calibration error, alongside end-of-session CBF.The supplied ECE specification uses 10 equal-width bins; Figure 5 summarizes task accuracy and profile-estimation performance.Source: §B.1 · p. 12

B.2 MMLU Subject Selection

  • Across the 20 subjects, there are 13 STEM subjects, four in MMLU’s “Other” category, two in humanities, and one in social sciences; the following table illustrates model-relative tiers.Source: §B.2 · p. 12

B.3 User Persona Details

  • Personas vary expertise, response style, and tolerance; feedback labels are rule-decoded, and the figures show baseline coverage and confidence reliability.Figure 6 presents the full model-by-baseline CBF matrix across all 18 models and five levels; Figure 7 plots bin-level mean confidence and answer accuracy, with confidence 1 in the last bin.Source: Figure 6 · p. 13, Figure 7 · p. 13, §B.3 · p. 14

C.1 Baseline Implementations

  • The five-level ladder ranges from a constant profile and zero-shot prior through noisy-feedback estimators to exact verified-outcome counts and supplied-statistic reporting.L1–L3 measures prior error, L2–L3 reconstruction from feedback, and L4–L3 numerical reporting.Source: §C.1 · p. 14

C.2 Baseline Implementation Details

The appendix specifies how baseline profiles are constructed and how comparisons handle shared records, uncertainty, and limits on detecting small predictive differences.

C.2 Baseline Implementation Details

  • OraclePDC uses verified correct/total counts on the scored domain and achieves CBF 1 exactly, while NoMemory returns a constant .5 profile.Recorded answer confidence is separate from NoMemory’s domain self-assessment.Source: §C.2 · p. 14
  • User-Feedback PDC decodes explicit positive and negative feedback, treating uncertain and neutral responses as unobserved; LSA estimates domain accuracy from session summaries.LSA returns one probability per domain at session end, and its valid profile is not selected by CBF.Source: §C.2 · p. 14
  • EM-Joint estimates domain prevalence and expertise-state reliability from decoded labels, while GT-SelfStats reports supplied empirical domain accuracies.EM-Joint uses a symmetric reliability model and retains .5 for domains with missing feedback; GT-SelfStats supplies the current session’s eligible-domain statistics.Source: §C.2 · pp. 14, 16
  • Qwen-Max’s low confirmation rate includes decoder misses: 59 neutral replies saying “that was accurate” were not captured by fixed rules.The table reports 58/293 in-expertise correct answers with positive labels and 235/293 with neutral labels; its correction count is 60/78.Source: Table 5 · p. 15

D.1 Sample Size and Paired Comparisons

  • Model-level comparisons pair methods on shared records, while uncertainty estimates resample sessions or question clusters; the 18-model inventory is fixed.Table 6 presents paired pooled CBF differences on shared labels and domains; question-clustered intervals appear in Table 13.Source: §D.1 · p. 16, Table 6 · p. 16
  • Sparse per-domain outcomes limit session-profile precision, and larger held-out task sets are needed to resolve small predictive differences.The paper calls for larger held-out sets rather than post-hoc power estimates from observed effects.Source: §D.1 · p. 16

D.2 Statistical Robustness

The appendix qualifies cross-model CBF comparisons, reports uncertainty conditional on retained data, and distinguishes confidence-error summaries from broader measures of profile quality.

D.2 Statistical Robustness

  • Session-bootstrap standard errors use 2,000 draws for all 18 models, but uncertainty is conditional on the retained session pool.Paired differences require resampling the paired differences themselves, not applying a threshold based on one method’s standard error.Source: §D.2 · p. 16, Table 7 · p. 16
  • Cross-model CBF averages summarize different empirical targets, and raw CBF can be higher when a target profile is closer to the constant reference.These contrasts are not an ordering of model competence; normalized CBF instead reports error reduction relative to the .5 profile, with NoMem at 0 and exact L3 counting at 1.Source: §D.2 · pp. 16–17, Table 8 · p. 16, Table 9 · p. 16

D.3 Normalized Hallucination Rate

  • The normalized hallucination rate is the fraction of errors assigned high confidence, but it does not by itself measure confidence discrimination.Comparing confidence on correct answers is also necessary; Figure 7 provides a complementary reliability-curve view.Source: §D.3 · p. 17

D.4 CBF vs. Rank Correlation

  • Kendall’s τ is undefined for constant predictions such as NoMemory’s 0.5 estimates, while CBF applies through absolute deviation; report both when applicable.Figure 9 visualizes MMLU accuracy, recorded-confidence ECE, and NoMemory CBF across the n=18 cross-baseline panel.Source: §D.4 · p. 17

D.5 Cross-Baseline Bootstrap Significance

  • Matched pooled comparisons use paired question-cluster intervals with estimators refitted in each of 400 resamples.Intervals are conditional on recorded questions, and no blanket significance threshold is assigned across models, metrics, and baselines.Source: §D.5 · p. 17

E.1 Additional Robustness Ablations

This section examines the robustness of self-assessment findings to feedback decoding and availability, domain definitions, simulator choice, and outcome-informed adaptive selection. It also documents task variation in prior-profile errors, limits of short-session evidence about convergence, and scoring caveats for cross-task comparisons.

E.1 Additional Robustness Ablations

  • 0.870 MMLU-Pro and 0.825 BBH mean CBFs mask category- and task-level prior errors, including Claude-Sonnet-4.6 estimating .80 on word sorting despite .00 accuracy.Across the eight models, errors vary with task content; benchmark differences do not establish whether tasks were seen during training.Source: §E.1 · p. 17, Table 14 · p. 20, Table 15 · p. 21
  • Optimization raises clean-label GLAD-style CBF from .854–.938 to 0.920–0.986, while DS improves likelihood but retains substantial calibration error.The authors caution that fixed-budget score gains after adding noise do not establish a beneficial noise effect; optimization and identifiability matter beyond session length.Source: §E.1 · p. 22, Table 18 · p. 23
  • 0.890/0.224 frozen-rule precision/recall rises to 0.992/0.728 under expanded rules, while Global’s session-level lead falls from 14/18 to 11/18.For Qwen-Max, correct answers are less likely than incorrect answers to yield decoded labels, creating a low-accuracy observed subset that label correction alone cannot fix; decoder choice also changes estimator rankings.Source: §E.1 · p. 22, Table 20 · p. 24, Table 21 · p. 24

E.2 Learning Dynamics / E.3 Prefix Estimation Curves

  • Clean-label Count reaches CBF 1 by construction, while Beta approaches that ceiling with more observations and latent-reliability fits remain sensitive to optimization and identifiability.Across 50-turn prefixes, estimates are scored against a fixed final-session target; these curves do not establish unlimited-data convergence, and sparse informative feedback can leave estimates near initialization.Source: §E.4 · p. 25, §E.3 · p. 25, §E.2 · p. 25

E.5 Feedback Controls on the Common Pool

  • With complete clean labels, LSA reaches .989 on GPT-5.5 and .995 on DeepSeek-V4-Pro but .835 on Qwen2.5-7B, while Count is 1.000 for all three.The common-pool comparison shows that reconstruction accuracy depends on how exact feedback history is aggregated.Source: §E.5 · p. 25, Table 24 · p. 26

E.6 Domain-Partition Sensitivity

  • Semantic and random domain partitions change method rankings: Global leads four of five models in the four-category partition, while Qwen2.5-7B favors DS.Grouping changes target dispersion, so higher CBF on coarse groups does not demonstrate better fine-domain discrimination.Source: §E.6 · p. 25, Table 25 · p. 27

E.7 Noise and Coverage: Complete Grid

  • On GPT-5.5 with clean labels, GLAD-style exceeds Global at full coverage (.971 vs. .885 session CBF) but trails it at 12.5% coverage (.686 vs. .821).Thus sparse feedback can change estimator rankings even without label errors; session and pooled scores show the effect at different aggregation scales.Source: §E.7 · p. 25, Table 27 · p. 28

E.8 Dual-Role Bias / E.9 Within-Family Simulator Comparison

  • The Qwen-Max pairing yields few explicit corrections, and its comparison with Qwen-Turbo cannot separate simulator-family effects from model-specific response style.Hedged feedback may evade a strict binary decoder; the displayed runs differ by .6 percentage points in task accuracy and .005 in CBF, with different answer samples.Source: §E.8 · pp. 25, 27, §E.9 · p. 27

E.9 Within-Family Simulator Comparison

  • The simulator comparison is descriptive: similar raw answer confidence does not establish that feedback is equally informative.The accompanying CBF comparisons use 50% and 12.5% random label coverage, with the latter including direct Count, to test sparse feedback.Source: Table 28 · p. 28, §E.9 · p. 29, Table 29 · p. 29

E.10 R4 Adaptive Selection and Second-Order Estimators: Full Details

  • R4’s held-out ReAct comparison forms profiles only from each subject’s training half, unlike a deployment rule that would choose without test outcomes.The reliability-aware estimators are also compared at matched session and pooled scoring units.Source: §E.10 · p. 29, Figure 11 · p. 30
  • Mean CBF was 0.762 for the best-of-two observed-score reference, while R4 selected ZS on 14 of 18 models and averaged 0.744 versus 0.747 for always-ZS.The best-of-two value is a selection reference, not an implementable predictor without outcome information; R4 is also outcome-informed because its branch score depends on empirical domain outcomes.Source: §E.10 · p. 29

F.1 Cross-Task Heterogeneity: Full Per-Task Comparison

  • Archived per-item flags score Qwen3.6-Plus at 182/300 on GSM8K and 101/164 on HumanEval, but these lower scores alone do not identify an API or parser failure.Extension tasks use fixed archived denominators, and missing full responses prevent consistent separation of failed or unparsed cases from other outcomes.Source: §F.1 · pp. 29–30

F.2 Recorded Cross-Task Scores

The cross-task analyses show that recorded confidence gaps can vary in sign across tasks, finite-budget summaries agree without establishing convergence, and held-out ReAct prediction varies by configuration.

F.2 Recorded Cross-Task Scores

  • 12 of 16 rows with available confidence gaps have opposite signs across the three extension tasks.The table records task-dependent confidence–score gaps; unavailable gaps are unscored.Source: §F.2 · p. 30, Table 32 · p. 31

F.3 Lite protocol: T=100 vs T=500 rankings

  • The 100-versus-500-record correlations show agreement between finite-budget summaries, not convergence of the 18-model session protocol.The analysis covers 23 recorded run configurations, including repeated model variants.Source: §F.3 · p. 31, Table 33 · p. 31

F.4 Held-Out ReAct Prediction

  • Global has the lowest MAE in three of six held-out ReAct configurations, while ReAct Count leads in two and ReAct Beta in one.For Qwen3-14B, ReAct Count improves on Global (.171 vs. .206); for Qwen2.5-7B, Global leads (.237 vs. .285).Source: §F.4 · p. 32, Table 34 · p. 32

G.1 Implementation Note: Algorithmic-Baseline self_assess Specification

This appendix specifies how algorithmic baselines produce domain-level estimates and situates LADDER among work on self-knowledge, calibration, memory, and feedback. It also documents the model panel, evaluation materials, sample interactions, failure modes, and prompt and feedback-decoding procedures.

G.1 Implementation Note: Algorithmic-Baseline self_assess Specification

  • UF-PDC, Platt-Online, and HistBin-Online return domain label fractions, Bayes-Domain returns a Beta posterior mean, and EM-Joint returns a clipped fitted latent-correctness rate.UF-PDC, Platt, and HistBin share endpoint counts on matched labels, but their query-confidence mechanisms differ; Bayes adds smoothing.Source: §G.1 · p. 32

G.2 Extended Related Work Discussion

  • LADDER targets stable per-domain self-models accumulated from noisy interaction feedback, unlike prior work on one-shot probing or clean execution traces.Related work also distinguishes LADDER from static calibration, training-time feedback, and query-level uncertainty approaches.Source: §G.2 · pp. 32–33

G.3 Model Configurations

  • The evaluation panel contains 18 models across seven families, with local inference for three named models and API configurations for the others.LSA and L4 return one domain-indexed JSON profile at the session endpoint; local decoding is greedy and API requests use temperature zero.Source: §G.3 · p. 33

G.4 Extended Results

  • Table 35 compares pooled CBF for the constant profile, prior, and Beta estimate across empirical subject tiers for Llama-3-8B, Qwen-Turbo, and Qwen-Max.The supplied passage gives the table's scope but no cell values.Source: §G.4 · p. 33, Table 35 · p. 34

G.5 Reproducibility Index

  • The reproducibility materials link principal outputs to analysis inputs and provide scoring code, analysis inputs, and repeated draws.The index identifies tables and figures associated with analyses, including same-decoded-label records and fixed-answer noise repetitions.Source: §G.5 · pp. 33–34

H.1 Reproducibility Checklist

  • The primary interaction study uses simulated personas, and the benchmark measures domain-estimation performance rather than certifying safe deployment.Stored traces support deterministic estimator comparisons, while new LLM calls can vary with sampling and service versions.Source: §H.1 · p. 34

I.1 Sample session trajectories

  • Sample trajectories show that polite acknowledgment can follow a wrong answer, while only utterances satisfying decoder rules supply algorithmic labels.Unparsed answers are excluded from valid-answer accuracy, and the failure catalog also notes saturated confidence, ambiguous feedback, and cases where history estimates trail zero-shot profiles.Source: §I.1 · p. 35, §I.2 · p. 35

I.3 Prompt templates

  • The prompt templates elicit answers with confidence, simulated-user feedback, and domain profiles; only positive and negative rule-decoded feedback enters estimators.These labels measure decoded stance, not human-annotated semantic correctness.Source: §I.3 · pp. 35–36, L1 JSON response format · p. 36

Related papers

Loading 2610.34737v1…