Source-linked AI summary
How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models
Christos Petridis, Konstantinos Pelechrinis, Zoran Obradovic
TL;DR
Because people ask AI systems questions whose answers are uncertain, understanding how models communicate probabilistic language matters. The paper evaluates bidirectional consistency in probability-language mappings and finds human-like structure alongside systematic calibration distortions and model differences in self-consistency.
Problem
People ask AI systems questions whose answers are uncertain, making reliable interpretation of probabilistic language important in human-AI communication.
Method
The paper introduces a roundtrip error and corrected baselines to distinguish bidirectional inconsistency from errors caused by models expressing only a small number of probability values.
Results
LLMs reproduce human-like word ordering, anchor fidelity, and ambiguity around possible, but systematically overrate non-anchor expressions and vary in bidirectional self-consistency.
Takeaways & Limitations
Models communicate uncertainty in recognizably human-like ways but carry measurable distortions that matter wherever hedged language is trusted at face value.
Takeaways & Limitations
Default temperatures are not standardized across providers, and each model-word-condition triplet uses only N=10 samples, confounding variance and limiting precision of individual means.
Abstract
from arXiv · showhide
Large language models increasingly produce and interpret verbal probability expressions, yet whether these expressions carry consistent meaning across models (or match human perceptions of uncertainty) remains unknown. We present a systematic cross-model evaluation using a word-to-number mapping task grounded in established human benchmarks. Eleven uncertainty expressions were presented to 19 models under two conditions, forced single-number response and explanation elicitation, alongside a novel bidirectional roundtrip test of internal consistency. LLMs track the human benchmark with surprising fidelity: word ordering is preserved, three anchor points are recovered, and ``possible'' shows the highest variance and cross-model disagreement of any expression tested, consistent with its documented bimodal interpretation in humans. However, models show a systematic upward bias for negative expressions such as ``unlikely'' and ``improbable.'' Explanation elicitation reduces within-model variance while increasing between-model divergence, stabilizing individual models at the cost of inter-model consensus, and the roundtrip experiment reveals clear stratification, with frontier models maintaining coherent bidirectional representations. LLMs thus reproduce the structure of human verbal probability cognition, including its biases, while diverging systematically at the negative end---with implications for any setting where humans and models exchange probabilistic language.
1 INTRODUCTION
The paper asks whether generative AI systems interpret verbal probability expressions consistently and align with human interpretations. It evaluates this question across models and prompting conditions, finding broadly human-like structure alongside systematic calibration deviations.
- Motivation: AI systems commonly communicate uncertainty with verbal expressions, making shared interpretations important for human-AI decision making.Without shared interpretations, users may overestimate or underestimate the confidence an AI system intends to convey.
- Approach: The study evaluates 11 probability expressions across 19 models using forced-number and explanation-elicitation prompts, grounded in a benchmark aggregating 20 human studies.A bidirectional roundtrip experiment additionally tests whether models’ word-to-number and number-to-word mappings are coherent.
- Findings: LLMs preserve word ordering and recover the human anchors “impossible,” “even chance,” and “certain” with near-zero variance.These patterns indicate broad structural fidelity to human verbal probability perception.
- Findings: “Possible” has the greatest variance and cross-model disagreement, consistent with its documented instability in human interpretations.The expression is also identified as the clearest departure from the human pattern.
- Findings: Models generally rate non-anchor expressions as more probable than human norms, with marginally higher inflation for negative expressions under forced single-number responses.The three anchor expressions are exceptions to the upward-bias pattern.
- Findings: Explanation elicitation reduces within-model variance but increases divergence across models, while roundtrip results show coherent mappings for most models and chance-level mappings for a small subset.The findings reveal a trade-off between individual-model stability and cross-model agreement.
2 RELATED WORK
Prior research shows that people assign variable numerical meanings to probability words, especially “possible,” while LLM studies find broadly human-like but context-sensitive uncertainty mappings. This paper isolates lexical semantics and compares models directly with human norms using broader evaluation conditions.
- LLM uncertainty research: Recent LLM research distinguishes interpreting verbal probability language from calibrating expressed confidence against actual accuracy or reasoning.These are related but distinct questions in the existing literature.
- LLM uncertainty research: Belem et al. found broadly human-like verbal-to-numerical mappings that are systematically distorted by models’ prior beliefs about the associated statement.The same expression received a higher probability when paired with different statements.
- Positioning: This study isolates lexical semantics independently of speaker or statement context and compares mappings with an established human benchmark across more models and prompting conditions.It also adds a bidirectional consistency check.
3 EXPERIMENTAL SETUP
The experimental setup measures verbal probability perception through direct word-to-number mappings, explanation elicitation, and bidirectional roundtrips. It spans eleven expressions and models across proprietary and open-weight families, with repeated sampling and explicit probability anchors.
- Stimuli and models: The study tests 11 expressions spanning subjective certainty from “impossible” to “certain,” selected to match the Mosteller and Youtz human benchmark.The set includes “possible,” “even chance,” and several positive and negative hedges.
- Prompt conditions: Each model receives a basic forced-number prompt asking for a 0–100 probability, with 0 and 100 explicitly anchored to never and always happening.The explicit direction addresses observed cases where models inverted the scale.
- Prompt conditions: The explanation condition adds step-by-step reasoning to the same word-to-number task, allowing comparison of changes in central tendency and variance.It tests whether externalizing an explanation is associated with changes in a model’s response distribution.
- Prompt conditions: The roundtrip condition samples numbers from 0 to 100, elicits associated probability words, and then maps those words back to numbers.Roundtrip error is the absolute difference between the sampled number and the recovered number.
- Stimuli and models: The evaluation covers proprietary and open-weight models ranging from lightweight deployment variants to large frontier-scale systems.This enables assessment of whether verbal probability perception varies with model size and access tier.
- Sampling: Prompts 1 and 2 are repeated 10 times per model, while roundtrip testing uses 30 sampled numbers and five full repetitions per number, yielding 150 trials per model.Default provider temperatures are not standardized across models.
- Analysis: Figure 1 compares the aggregate model probability scale with the human benchmark and shows cross-model variability using shaded ±1 standard deviation regions.The largest divergence occurs at “possible.”
4 RESULTS
Across 19 models, verbal probability mappings largely follow human structure, while negative expressions and explanation prompts reveal systematic deviations and model-specific instability. The roundtrip test further separates models by bidirectional coherence.
- 4.1 LLMs track the human benchmark closely: The aggregate LLM mapping preserves human word ordering and recovers the anchors impossible, even chance, and certain with near-zero variance.These anchors closely match the corresponding human benchmark points.
- 4.1 LLMs track the human benchmark closely: Possible is the clearest departure, with an aggregate LLM mean of ≈51 versus a human mean of ≈37 and the widest variance.Its instability is consistent with the word’s documented bimodal interpretation in human studies.
- 4.1 LLMs track the human benchmark closely: Both LLM and human antonym pairs are sub-additive, but LLM pairs are consistently closer to perfect complementarity than human pairs.The paired shifts are similar in magnitude, although individual models show substantial heterogeneity.
- 4.2 Per-model variation & negative-word bias: LLMs assign higher probabilities than humans to both negative and positive expressions, with the negative counterparts unlikely and improbable showing larger gaps.The negative/positive asymmetry under Prompt 1 is about 2.1 percentage points on average, but the authors treat it as suggestive because the tests are underpowered and Prompt 2 differences are not significant.
- 4.3 Explanation elicitation does not uniformly reduce variance: Explanation elicitation generally reduces within-model variance without eliminating the upward shift, and can increase divergence from humans at possible and probable.The effect is especially pronounced for smaller open-weight models such as qwen_14b and llama3_8b.
- 4.3 Explanation elicitation does not uniformly reduce variance: Explanation elicitation can also produce task-specific failures: GPT-5.4-nano shows scale-direction instability at impossible, while gpt-oss-20b sometimes evaluates the prompt text instead of the target word.For several frontier models, increased variance at possible instead reflects latent disagreement consistent with human ambiguity, though parallel bimodality in models is not established.
- 4.4 Roundtrip consistency: The roundtrip analysis uses sampled numbers, model-selected words, and independently recovered numbers to measure bidirectional mapping error.The corrected references include a best-constant baseline of 25 and a quantization floor of 25/k.
- 4.4 Roundtrip consistency: Only four models fail to beat the constant baseline of 25, while Qwen-14b and GPT-5.4 nano remain distinguishable from chance after Bonferroni correction.Mistral-7B is not distinguishable from chance under any correction, and llama3-8b is only suggestive under the stricter comparisons.
5 DISCUSSION AND LIMITATIONS
LLMs reproduce the structure of human verbal probability perception but systematically diverge in calibration, while explanation elicitation improves within-model consistency without reliably correcting bias. The findings also expose practical communication risks and scope limitations involving model settings, prompts, benchmarks, vocabulary, and roundtrip data volume.
- Discussion: LLMs reproduce human-like word ordering and anchor fidelity, but systematically rate both negatively and positively worded expressions as more probable than human norms suggest.The trend toward larger inflation for negative expressions is marginal and non-robust.
- Discussion: Explanation elicitation narrows response variance within models but does not correct calibration bias and can increase divergence from human norms across some models.This separates internal consistency from calibration accuracy as distinct, only loosely related properties.
- Discussion: Roundtrip results show that most models maintain coherent bidirectional probability mappings, while a small subset performs at chance; within-family comparisons suggest scale contributes to this gap.The pattern spans both proprietary and open-weight families and appears independent of architecture or training pipeline.
- Human-AI communication: Negative wording may cause humans to underestimate the probability a model assigns, regardless of whether the described event is desirable or undesirable.The practical consequence follows from bias operating on grammatical polarity rather than event desirability.
- Limitations: Cross-model variance may be confounded by nonstandardized default temperatures, and each model-word-condition estimate uses only N=10 samples.Individual per-word means, especially for smaller open-weight models with wider variance, therefore require appropriate uncertainty.
- Limitations: The study does not test robustness to paraphrasing its single anchoring instruction template, so some observed bias may be wording-sensitive.The authors cannot establish that the bias is fully prompt-invariant.
- Limitations: The human benchmark predates current LLMs, and the findings are limited to eleven expressions, three prompt conditions, and the evaluated models.Future work should broaden expressions, prompt phrasings, models, and comparative or contextual probability language.
- Limitations: Entropy-based word-choice consistency and mutual-information roundtrip metrics would require thousands rather than hundreds of trials per model.These metrics are identified as data-intensive directions for strengthening the results.
6 CONCLUSIONS
LLMs inherit the structure of human verbal probability perception without fully inheriting its calibration, producing recognizable but measurable distortions in uncertainty language.
- 6 CONCLUSIONS: LLMs reproduce word ordering, anchor fidelity, and rich probability vocabulary in frontier models, but not human calibration.Both negative and positive expressions are overrated relative to human norms, while explanation elicitation and scale do not reliably fix inflation.
A QUANTIZATION FLOOR DERIVATION
The quantization-floor derivation isolates error caused solely by representing continuous probability inputs with k discrete values. It provides a lower bound against which excess roundtrip error can be attributed to inefficient anchors or genuine inconsistency.
- A QUANTIZATION FLOOR DERIVATION: A model restricted to k distinct values behaves like a quantizer mapping continuous inputs to the nearest of k discrete representatives.The derivation partitions [0, 100] into k equal-width bins and reports each bin midpoint for uniformly distributed inputs.
- A QUANTIZATION FLOOR DERIVATION: By symmetry, the within-bin mean absolute error equals the overall mean absolute error across the full [0, 100] range.The result follows because every equal-width bin contributes the same error under the uniform input distribution.
- A QUANTIZATION FLOOR DERIVATION: The quantization floor is the best achievable MARE for a k-level representation under uniform inputs, not a prediction of observed model error.Observed excess error may reflect nonoptimal representative placement or genuine mapping inconsistency.