Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs

Preetum Nakkiran; Arwen Bradley; Adam Goliński; Eugene Ndiaye; Michael Kirchhof; Sinead Williamson

Source-linked AI summary

Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs

Preetum Nakkiran, Arwen Bradley, Adam Goliński, Eugene Ndiaye, Michael Kirchhof, Sinead Williamson

arXiv:2511.04869v1Nov 6, 2025cs.CLcs.LGstat.ML

TL;DR

LLMs are known to calibrate next-token probabilities, but it is unclear whether they can assess confidence in the meaning of open-ended answers. This paper defines B-calibration, links it to local loss optimality, and finds that base models are often semantically calibrated while instruction-tuning and chain-of-thought can break calibration.

Problem
It is unclear whether LLM confidence remains meaningful for semantic answers beyond token-level calibration, especially across tasks and inference or training settings.
Method
The paper collapses generations into equivalence classes, defines B-calibration, and derives a local-loss-optimality mechanism predicting when semantic calibration should emerge.
Results
Base LLMs are semantically calibrated across tested question-answering settings, whereas post-training and chain-of-thought configurations are not reliably calibrated.
Takeaways & Limitations
Semantic calibration can emerge implicitly from token-level pretraining when models can predict their distribution over semantic answer classes before generation.
Takeaways & Limitations
The paper studies a specific sampling-based notion of calibration and leaves other forms, such as verbalized calibration, out of scope.

Abstract

from arXiv · show

Large Language Models (LLMs) often lack meaningful confidence estimates for their outputs. While base LLMs are known to exhibit next-token calibration, it remains unclear whether they can assess confidence in the actual meaning of their responses beyond the token level. We find that, when using a certain sampling-based notion of semantic calibration, base LLMs are remarkably well-calibrated: they can meaningfully assess confidence in open-domain question-answering tasks, despite not being explicitly trained to do so. Our main theoretical contribution establishes a mechanism for why semantic calibration emerges as a byproduct of next-token prediction, leveraging a recent connection between calibration and local loss optimality. The theory relies on a general definition of "B-calibration," which is a notion of calibration parameterized by a choice of equivalence classes (semantic or otherwise). This theoretical mechanism leads to a testable prediction: base LLMs will be semantically calibrated when they can easily predict their own distribution over semantic answer classes before generating a response. We state three implications of this prediction, which we validate through experiments: (1) Base LLMs are semantically calibrated across question-answering tasks, (2) RL instruction-tuning systematically breaks this calibration, and (3) chain-of-thought reasoning breaks calibration. To our knowledge, our work provides the first principled explanation of when and why semantic calibration emerges in LLMs.

1 Introduction

The paper asks whether LLMs can express calibrated confidence in the meaning of open-ended answers, beyond next-token probabilities. It proposes a mechanism for sampling-based semantic calibration and tests predictions about pretraining, post-training, and reasoning.

1 Introduction: Semantic calibration extends uncertainty estimation from next-token probabilities to confidence in the semantic class of an open-ended response.Next-token calibration is useful when one token determines the answer, but does not directly capture equivalent long-form answers.
1 Introduction: The paper defines semantic confidence by sampling generations, collapsing equivalent answers into semantic classes, and calibrating the resulting classifier’s predicted class confidence.Figure 1 illustrates this process using multiple temperature T =1 generations and a collapsing function B.
1 Introduction: The proposed mechanism adapts calibration–local-loss-optimality connections to explain how semantic calibration can emerge from maximum-likelihood pretraining.The model is treated as a multiclass classifier after outputs are collapsed by semantic meaning.
1 Introduction: The paper’s stated contribution is a theoretically motivated account of when semantic calibration emerges, supported by empirical tests of its practical predictions.The work distinguishes calibrated settings from settings where calibration need not hold.
1 Introduction: The theory predicts calibration when a base model can easily predict its probability of landing in each semantic class before generating an answer.The relevant prediction should be easy for the model to learn, such as through a LoRA adaptation.
1 Introduction: Experiments evaluate semantic calibration across Qwen, Gemini, Mistral, and Llama-family models from 0.5B to 70B on four datasets and three response styles.Configurations include base and instruction-tuned models with concise, sentence, or chain-of-thought responses.

2 Semantic Calibration and B-Calibration

The framework turns an LLM into a classifier over equivalence classes by collapsing generated outputs with B. Calibration then compares the induced class confidence with ground-truth class accuracy over an evaluation distribution.

2 Semantic Calibration and B-Calibration: Sampling generations and applying B produces a distribution over classes, whose most-likely class defines semantic prediction and whose probability defines semantic confidence.The semantic accuracy is whether that predicted class matches the ground-truth semantic class.
2.1 Notation and Setup: The LLM maps prompts to distributions over generated strings, while the evaluation data supplies a ground-truth distribution over prompt-completion pairs.Generated strings and dataset completions are represented separately in the notation.
2.1 Notation and Setup: A collapsing function B maps each prompt-completion pair to one of K classes, including semantic classes that group differently phrased answers.The framework permits arbitrary B, while semantic collapsing functions are the main focus.
2.2 Confidence Calibration: The evaluation distribution is central to the definition, with examples including TriviaQA and GSM8k.Calibration is defined over prediction-label pairs induced by the selected distribution D.
2.2 Confidence Calibration: B-confidence-calibration means that the induced class confidence is calibrated against the corresponding ground-truth class over the chosen evaluation distribution.The framework can be calibrated for some collapsing functions B but not others.

3 Theoretical Mechanism

The proposed mechanism links semantic calibration to local loss optimality and predicts calibration when a base LLM can represent its own semantic-class distribution during generation. The theory develops this link from B-calibration through autoregressive perturbations, while noting assumptions and varying levels of evidence.

3.1 Conjectured Mechanism: Overview: The conjecture combines a formal calibration–local-loss-optimality equivalence with partially supported assumptions about base-model optimality and perturbation representability.Figure 4 distinguishes a formally proved implication, weaker analogous proofs, and experimental support for the full chain.
3.1 Conjectured Mechanism: Overview: The framework assumes base LLMs are nearly locally loss-optimal for perturbations that are easy to learn, while requiring this property on each evaluation distribution.The evaluation-distribution assumption is treated as plausible when those distributions are reasonably sized sub-distributions of pretraining data.
3 Theoretical Mechanism: The mechanism predicts B-confidence calibration when a base LLM can easily represent the distribution of B-classes for its own answers.For semantic collapsing functions, this means predicting the model’s semantic answer distribution before or during generation.
3.2 B-calibration and local loss optimality: For cross-entropy, sequence-level loss decomposes into autoregressive next-token log-loss, motivating exponential tilting and perturbations of the model’s generative distribution.The perturbation takes the prompt and original generative distribution as inputs and defines a prompt-specific modified distribution.
3.2 B-calibration and local loss optimality: Theorem 7 equates B-confidence calibration with local loss optimality for a corresponding family of B-dependent perturbations.These perturbations remap B-class confidence, such as reducing the probability mass of the top semantic class when confidence is excessive.
3.3 Which Perturbations are Easy to Learn Autoregressively?: Theorem 10 shows that if intermediate B-confidences have simple representations, every perturbation in W_B can be implemented with only a small additional circuit.This reduces sequence-level perturbation difficulty to representing the model’s probability mass on its top B-class given the prompt and generated prefix.

4 Experimental Predictions: When are LLMs calibrated?

The paper predicts semantic calibration when a pretrained model can learn its own distribution over semantic answers from the question alone. This yields three testable implications: pretraining can produce calibration, post-training can disrupt it, and chain-of-thought can disrupt it when the final answer is not known initially.

Core mechanism: The theory predicts calibration for base models when semantic answer classes are learnable before generation begins.This condition concerns the model’s own output distribution rather than whether its answers are correct.
Core mechanism: Semantic calibration is predicted when a cross-entropy-trained LLM can easily learn its own semantic-answer distribution from the question.The proposed probe estimates whether the model can predict Bx♯px without ground-truth labels.
Prediction 1: Standard pretraining is expected to produce semantic calibration across many settings because the model may implicitly know likely semantic answer classes.The prediction is presented as an emergent consequence rather than an explicitly trained objective.
Prediction 2: Post-training methods using objectives other than cross-entropy, including RLHF, DPO, or RLVR, are not theoretically expected to preserve calibration, although the theory does not rule it out.Cross-entropy is required for the connection between calibration and local loss optimality.
Prediction 3: Chain-of-thought can break calibration because difficult reasoning may determine the final semantic answer only after intermediate computation.The same extra computation that can improve answers immediately unavailable to the model can prevent the proposed pre-generation calibration mechanism.
Contribution: The authors frame their contribution as a unified theoretical explanation supported by more conclusive experiments than prior versions of these predictions.The predictions themselves are not claimed to be entirely novel.

5 Experiments

Experiments evaluate semantic calibration across open-ended math and factual question-answering tasks, response styles, model families, and scales. Base non-CoT models are generally calibrated, whereas post-training and chain-of-thought often produce miscalibration; learnability of the model’s own semantic distribution tracks calibration error.

5.1 Experimental Results: Nearly all evaluated base concise and sentence configurations are semantically calibrated, including models at or below 1B parameters and across response styles.The result supports calibration based on semantic-class knowledge rather than answer phrasing or model size.
5.1 Experimental Results: Post-training often breaks calibration: DPO is significantly miscalibrated, while the base and SFT-only models are better calibrated.The comparison uses Mistral-7B-v0.1, zephyr-7b-sft-full, and zephyr-7b-dpo-full with sentence responses.
5.1 Experimental Results: Chain-of-thought produces poor calibration in math settings, with base-CoT underconfident and instruct-CoT varying between underconfidence and overconfidence across datasets.Base models are calibrated when answering immediately but become miscalibrated when allowed to reason.
5.1 Experimental Results: Low KL gap in a LoRA probe predicting the model’s semantic class distribution is associated with small calibration error, while high-KL configurations are chiefly chain-of-thought experiments.The probe evaluates Qwen2.5 models from 0.5B to 14B across concise, sentence, and CoT styles on GSM8K.
5.1 Experimental Results: For base models without chain-of-thought, semantic accuracy shows no correlation with calibration error, and scaling does not improve semantic calibration.This differs from reported scale-related improvements for other calibration notions in multiple-choice settings.
5.1 Experimental Results: In overconfident and underconfident configurations, calibration error generally shows little relation to semantic accuracy, except that underconfident models tend to improve near perfect accuracy.The authors caution that the high-accuracy underconfidence pattern may not be robust.
5.2 Discussion: Calibration in LLMs vs other deep networks: The discussion attributes the strong calibration of base LLMs to training practices that keep test loss near a local optimum, unlike classifiers often trained beyond increasing test loss.This connects local-loss optimality with calibration differences between LLMs and other deep networks.

6 Conclusion

The paper explains semantic calibration as a consequence of next-token training, while documenting scope limits and unresolved theoretical and practical questions.

6 Conclusion: Base LLMs can be calibrated with respect to sequence-level semantics despite token-level training, through a mechanism connecting calibration and loss optimality.The framework uses B-calibration to quantify alignment between the model’s output distribution and the ground-truth pretraining distribution.
6.1 Limitations: The paper studies sampling-based B-confidence-calibration, leaving verbalized and other calibration notions outside its scope.The authors identify this as a specific limitation rather than claiming that other calibration types cannot emerge.
6.1 Limitations: Practical adoption is limited because semantic confidence requires multiple samples per question, and computational efficiency is not evaluated.The authors leave translation of the scientific results into real-world improvements for future work.
6.1 Limitations: The experiments cover four selected open-ended datasets, so other datasets may exhibit different calibration behavior.The selection spans domains and difficulty levels from world knowledge to mathematical reasoning, while excluding already-studied multiple-choice calibration settings.
6.1 Limitations: TruthfulQA may behave differently because its common-misconception examples do not satisfy the theory’s in-distribution requirement.The authors therefore consider possible miscalibration on TruthfulQA consistent with their theory.
6.1 Limitations: The proposed theoretical mechanism remains partly conjectural because several steps lack formal definitions and proofs.The authors identify formalizing these steps in meaningful and tractable ways as an open question.
A Additional Related Works: The paper distinguishes its parameterized B-calibration framework from prior sampling-based confidence measures, including semantic entropy.Parameterizing calibration by the collapsing function B allows theoretical analysis beyond a fixed notion of semantics.
B.1 Potential Extensions: The theory may extend to weighted-calibration properties beyond confidence calibration, including top-label calibration and conformal-prediction guarantees.The broader message is that log-loss training should produce calibration for easy-to-learn perturbations of the output distribution.

B.2 Technical Remarks

The technical remarks clarify distributional assumptions, extend the theory across evaluation distributions, and explain why confidence calibration is weaker than full B-calibration.

Distribution assumptions: The theory formally assumes calibration data match the model’s training distribution, while practical reasoning relies on evaluation prompts being reasonably in-distribution.Modern pretrained models may behave as if trained on evaluation distributions when those distributions are sufficiently represented in broad pretraining data.
Multicalibration: A single model can be calibrated across multiple evaluation distributions, which the authors frame as a multicalibration property.The formal treatment models the pretraining distribution as a mixture of disjoint sub-distributions.
Multicalibration: Simultaneous B-confidence-calibration across sub-distributions is equivalent to local-loss-optimality under an expanded perturbation class.The expansion incorporates indicators identifying membership in each sub-distribution.
Multicalibration: The mechanism predicts simultaneous calibration when the model can estimate its B-class distribution and identify which evaluation distribution generated each sample.The latter condition is considered plausible because the experimental datasets are distinct and easy to identify.
Full calibration: Full B-calibration is generally too strong because the implication from local-loss optimality to calibration fails for its overly large weight family.When the number of B-classes is large, full B-calibration can also be computationally intractable to estimate.
Local loss optimality: The broader mechanism links easy-to-learn perturbations under log-loss training to local loss optimality and weighted calibration.This is stated as an informal assumption about perturbations that can be learned through LoRA fine-tuning.
Full calibration: Confidence calibration remains theoretically accessible because its perturbation class is learnable, whereas full calibration requires a class too large to learn efficiently from samples.The authors suggest that intermediate calibration notions may be possible future extensions.

C Additional Experimental Results

Figure 8 compares calibration error for pretrained-only, instruction-SFT, and DPO models based on Mistral-7B-v0.1, with several response-style and termination exclusions.

C Additional Experimental Results: Figure 8 compares calibration error across pretrained-only, instruction-SFT, and DPO models based on Mistral-7B-v0.1.The figure caption does not provide cell values or identify a winning model.

D Additional Experimental Details

The experiments evaluate semantic calibration across diverse open-ended QA datasets and models using controlled prompting, semantic collapsing, and repeated sampling.

Datasets and Models: The evaluation covers GSM8K, OpenMathInstruct-2, TriviaQA, and SimpleQA across Qwen, Gemini, Mistral, and Llama models from 0.5B to 72B.The study focuses on open-ended QA because multiple-choice calibration is already well studied.
Prompt format: Three prompt styles—sentence, concise, and chain-of-thought—control response format, with chain-of-thought used only for math datasets.All experiments use 5-shot examples, and instruct models additionally receive explicit formatting instructions.
Prompt format: Responses that unexpectedly contain long chain-of-thought text under concise math prompts are excluded using a 15-character pre-newline heuristic.This filtering addresses models, especially Qwen models, that sometimes ignore the requested concise format.
The semantic collapsing function: The semantic collapsing function first extracts and normalizes canonical answers, then clusters non-math responses by semantic equivalence.Math responses are directly compared after canonicalization, while non-math responses use an LLM judge to handle irrelevant wording differences.
Measuring calibration: Calibration is measured from 50 temperature-1 samples per question, using plurality semantic frequency as confidence and plurality correctness as accuracy.The resulting confidence-accuracy pairs are evaluated with SmoothECE over up to 10K questions per dataset.

D.1 Calibration Metric: SmoothECE

The paper measures calibration with SmoothECE and visualizes it using kernel-smoothed reliability diagrams that relate semantic confidence to semantic accuracy.

D.1 Calibration Metric: SmoothECE: SmoothECE is a kernel-smoothed version of Expected Calibration Error with improved theoretical properties.
D.1 Calibration Metric: SmoothECE: The evaluation fixes the kernel bandwidth at σ = 0.05, improving interpretability and comparability while slightly weakening theoretical guarantees.
D.2 Visualizing calibration: reliability diagrams: Reliability diagrams plot a regression estimate of semantic accuracy conditional on semantic confidence, alongside average confidence, accuracy, and confidence-density histograms.
D.2 Visualizing calibration: reliability diagrams: The diagrams use kernel bandwidth σ = 0.05 for the regression line and 15 constant-width histogram bins for confidence densities.

D.3 LoRA Fine-Tuning

The appendix trains a rank-8 LoRA model to imitate the base model’s distribution over canonical semantic answers, then tests whether that distribution is learned and linked to calibration.

D.3 LoRA Fine-Tuning: The LoRA adapter pϕ is trained on canonical answers sampled from the base model pθ, treating the collapsed outputs as ordinary autoregressive targets.
D.3 LoRA Fine-Tuning: Each training question is expanded into 50 question–canonical-answer pairs by sampling the original model and applying the collapsing function B.
D.3 LoRA Fine-Tuning: The expanded pairs are used with a standard autoregressive objective, and the procedure is described as similar to training P(IK) in Kadavath et al. (2022).
D.3 LoRA Fine-Tuning: The KL gap between the learned and target semantic distributions is evaluated through the difference between pϕ’s unnormalized negative log-loss and pθ’s semantic entropy.
D.3 LoRA Fine-Tuning: Across 15 configurations, low KL gaps coincide with small calibration errors, while chain-of-thought settings and the 0.5B sentence model have high KL and calibration error.

D.4 LLMs evaluated

The paper evaluates pretrained and instruction-tuned language models under concise, sentence, and chain-of-thought prompting, while the appendix formalizes weighted calibration and its relation to semantic confidence calibration.

D.4 LLMs evaluated: The evaluated model set includes Llama-3.1-8B, Llama-3.1-70B, and Phi-4, with pretrained-only and instruction-tuned model groups listed separately.
D.5 Prompts: Prompts use concise, complete-sentence, and chain-of-thought formats with five few-shot examples, while chain-of-thought is excluded for non-math datasets.
D.5 Prompts: The sentence prompts vary syntactic form without intentionally increasing semantic complexity, allowing the theory’s limits to be tested under more complex response phrasing.
D.5 Prompts: Canonicalization prompts extract core answers by removing filler, expanding abbreviations, standardizing numbers and dates, and preserving meaning without inference.
E.1 Quick Reference: The appendix extends calibration results beyond cross-entropy to proper losses and derives quantitative bounds linking post-processing with calibration gaps.
E.2 Weighted Calibration: Weighted calibration allows prompt-dependent weight functions and can express multiple calibration notions, including forms of multi-calibration.

E.3 Equivalence between Weighted Calibration and Local Loss Optimality

This section develops equivalences connecting weighted and B-calibration with local loss optimality, including extensions to multi-class proper losses. It also constructs circuits for B-confidence perturbations and B-perturbations, providing the proof connections underlying the main theorems.

E.3 Equivalence between Weighted Calibration and Local Loss Optimality: Weighted calibration is equivalent to local loss optimality under perturbations in the weight class.The proof uses the primal-dual mapping z = log f, translating multiplicative probability reweighting into additive logit perturbations.
E.3 Equivalence between Weighted Calibration and Local Loss Optimality: Under log-loss, calibration holds exactly when prediction error f − y is orthogonal to every systematic logit perturbation w.
E.4 Equivalence between B-confidence-calibration and weighted calibration: B-confidence-calibration is equivalent to weighted calibration over perturbations induced by the semantic collapsing function B.The induced categorical distribution is obtained by pushing the model’s completion distribution forward through B.
E.5 Proof of Thm. 7: Thm. 7 follows by substituting the equivalence between B-confidence-calibration and its corresponding weighted-calibration condition.
E.6 Proof of Thm. 10: A Simple Circuit for B-Confidence-Perturbations: B-confidence perturbations can be implemented autoregressively with a small arithmetic circuit using top-category confidence and prefix-conditioned probability of eventually reaching that category.The perturbed next-token probability is obtained by scaling the original conditional probability through a ratio of lookahead expectations.
E.7 Full calibration: Full B-calibration requires the induced distribution over semantic categories to have zero conditional expected prediction error across all classes.The semantic push-forward can be represented by a collapsing matrix B_x, which maps completion probabilities to category probabilities.
E.7.1 Equivalence between B-calibration and weighted calibration: Perfect B-calibration is equivalent to perfect weighted calibration for the full-calibration perturbation class W^(full).The perturbation assigns each sequence a weight based on its semantic category and adapts it to the model’s categorical prediction.
E.7.2 A Simple Circuit for B-Perturbations: B-perturbations reweight each conditional next-token probability by the inner product between exponentiated initial B-confidence and conditional B-confidence.The proportionality factor is independent of the current token, and the resulting circuit has constant depth and width linear in K.
E.8 Quantitative Bounds on Multi-Class Calibration and Post-Processing Gap for Proper Losses: For multi-class proper losses, Theorem 43 states that being close to W-weighted-calibrated is essentially equivalent to being close to W-loss-optimal.The result generalizes Theorem E.3 in Błasiok et al. (2023b) to the multi-class setting and provides a robust version of Theorem 25.
E.8 Quantitative Bounds on Multi-Class Calibration and Post-Processing Gap for Proper Losses: The proper-loss framework represents losses through a convex potential and its convex-conjugate dual parameterization.The section introduces generalized dual calibration and post-processing gaps for function classes W and predictors represented in dual form.

Define for shorthand

This material connects calibration and post-processing through dual parametrization, specializes the relationship to cross-entropy, and applies weighted calibration to conformal prediction. It also organizes disaggregated reliability diagrams by task and model type.

Define for shorthand: Theorem 43 relates calibration and post-processing for differentiable smooth losses, showing that closeness to W-loss-optimality implies closeness to W-weighted-calibration.The theorem generalizes an earlier result to the multi-class setting and provides a robust version of a prior theorem.
Define for shorthand: Under strong convexity, the relationship between post-processing gap and calibration error has a tighter exponent.
E.8.1 Specialization to cross-entropy loss: For cross-entropy, the dual representation uses logits z = log f, turning multiplicative probability reweighting into additive logit perturbation.The perturbed distribution is f ⋆w = softmax(log(f) + w), and the primal and dual losses are equal under this mapping.
E.9 Conformal Prediction via Weighted Calibration: Conformal prediction guarantees can be expressed as weighted calibration for a particular family of weight functions.This connects set-valued coverage guarantees with the paper’s calibration framework.
E.9.1 Conformal Prediction from Full Calibration: A perfectly calibrated predictor induces a valid conformal predictor, while full calibration is stronger than typically practical requirements.The induced set contains the highest-probability labels whose predicted probabilities sum to at least 1 − α.
E.9.2 Conformal Prediction from Weighted Calibration: Weighted calibration is sufficient for validity at every coverage level, and local loss optimality with the corresponding weight class yields the same guarantee.The induced conformal predictor Fα is valid for all α ∈ [0, 1].
F Disaggregated Reliability Diagram Results: The disaggregated reliability diagrams compare corresponding base and instruction-tuned models across GSM8K, OpenMathInstruct, TriviaQA, and SimpleQA.The right three columns show instruct models and the left three columns show corresponding base models; TriviaQA and SimpleQA omit chain-of-thought evaluations.

F.1 GSM8K

This section presents semantic-confidence plots across GSM8K, OpenMathInstruct, TriviaQA, and SimpleQA, covering multiple model families and base or instruction variants.

F.1 GSM8K: GSM8K evaluations include the Qwen/Qwen2.5, Qwen3, google/gemma-2, google/gemma-3, mistralai/Mistral, and Qwen/Qwen2.5-Math model families.One cited entry names Qwen3-0.6B-Base.
F.2 OpenMathInstruct: OpenMathInstruct includes semantic-confidence plots for Qwen/Qwen2.5, Qwen3, and Gemma model families, including base and instruction variants.The passages identify the evaluated families and variants but do not report numerical results or conclusions from the plots.
F.2 OpenMathInstruct: OpenMathInstruct includes semantic-confidence plots for Mistral, alignment-handbook/zephyr-7b, and Meta-Llama-3 model families, with base and instruct variants shown.
F.3 TriviaQA: The cited TriviaQA entries cover the Qwen2.5, Qwen3, Gemma-2, Gemma-3, Mistral, and Llama-3 model families.
F.4 SimpleQA: SimpleQA includes semantic-confidence plots for Qwen2.5, Qwen3, Gemma-2, and Gemma-3 families, including base-sentence and instruction-oriented variants.
F.4 SimpleQA: The listed plots use Semantic Confidence as the displayed axis or measure.The supplied passages repeat the 0.0, 0.5, and 1.0 Semantic Confidence labels without accompanying curves or values.
F.4 SimpleQA: The passages identify SimpleQA evaluations for Mistral, alignment-handbook/zephyr-7b, meta-llama/Llama-3, and microsof/phi-4 model families.These entries provide experiment labels but no reported performance comparisons or numerical results.
F.4 SimpleQA: The annotations distinguish base, instruct, concise, sentence, and chain-of-thought variants, including named Mistral and Llama models.The passages list Mistral-7B-v0.1, Mistral-Small-24B-Base-2501, Mistral-7B-Instruct-v0.1, Ministral-8B-Instruct-2410, Mistral-Small-24B-Instruct-2501, and Llama-3.1-8B-Instruct.