Source-linked AI summary
Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
Samer Awad, Javier Conde, Carlos Arriaga, Tairan Fu, Javier Coronado-Blázquez, Pedro Reviriego
TL;DR
LLMs often produce homogeneous text, but limited evidence isolates how decoding mechanics constrain lexical diversity. This paper introduces WCS to audit vocabulary reachability under sampling filters and finds that default decoding systematically erases meaningful lexical paths in the 10k–40k frequency band.
Problem
Existing research emphasizes model knowledge and training data, leaving decoding mechanics’ role in suppressing LLM linguistic diversity insufficiently isolated.
Method
The Word Coverage Score audits whether contextually appropriate human words survive Top-p, Top-k, and Min-p filters through a forced-path, token-level reachability analysis.
Results
Typical decoding filters systematically erase meaningful lexical paths in the 10k–40k word-frequency band across several open-weight models.
Takeaways & Limitations
Preserving lexical breadth may require semantic-guided decoding and treating vocabulary diversity as a core optimization objective.
Takeaways & Limitations
The evaluation covers only open-weight models with fewer than 20 billion parameters and cannot extend to proprietary models lacking complete step-wise logits.
Abstract
from arXiv · showhide
Modern Large Language Models (LLMs) are often criticized for producing repetitive and homogeneous text, despite possessing vast latent vocabularies. While previous research has focused on model knowledge and training data, we investigate the role of decoding mechanics in suppressing linguistic diversity. We introduce the Word Coverage Score (WCS), a metric that quantifies the extent to which contextually appropriate human vocabulary is mathematically pruned by standard sampling filters (e.g., Top-$p$, Top-$k$, and Min-$p$). Rather than assessing static knowledge, the WCS measures the lexical survival rate of low-frequency, high-information human words as a function of sampling parameters. By auditing open-weight models on human-authored corpus fragments, we identify which logical lexical choices are rendered unreachable by the decoder, even when they reside within the probability space. Our results provide quantitative evidence that industry-standard sampling defaults act as unintended censorship mechanisms, smoothing the unique textures of human expression into a homogenized discourse. The WCS offers a rigorous framework for optimizing the trade-off between text coherence and lexical richness, providing a diagnostic tool for preserving the diversity of human language in generative models.
1 Introduction
The introduction frames LLM text as homogenized and lexically narrower than human discourse, with rare-token probabilities decaying unusually quickly. It attributes this diversity loss to decoding mechanics and introduces the Word Coverage Score (WCS) to measure vocabulary pruned by standard sampling filters.
- Motivation: LLM outputs converge toward narrow phrasing and structural patterns, becoming more similar across model families than human-authored texts.The introduction characterizes this homogenization as a structural collapse of the probability distribution, often termed mode collapse.
- Motivation: Frontier LLMs exhibit an artificially steep Zipfian tail, causing rare-token probabilities to decay faster than in natural human corpora.The passage links this distributional pattern to constraints on the depth of vocabulary models can use.
- Contribution: The Word Coverage Score (WCS) measures how much contextually appropriate human vocabulary is mathematically pruned by standard sampling filters.The work shifts attention from model knowledge and training data toward decoding mechanics as a source of linguistic suppression.
- Method: The WCS audit selects Middle-Long Tail words, pairs them with human-authored contexts, tests their survival under Top-p, Top-k, or Min-p, and aggregates reachability.These stages are described as lexical selection, contextual pairing, forced-path auditing, and WCS aggregation.
2 Token Sampling and LLM Homogeneity
LLMs generate predictions over vast vocabularies, but Top-p, Top-k, and Min-p filters aggressively prune low-probability candidates before token selection. This creates a trade-off in which prioritizing local coherence reduces global linguistic and lexical diversity.
- Token prediction space: LLMs compute logits for all vocabulary tokens, creating tens of thousands of candidates and a vast space of potential trajectories.The passage characterizes this space as a “Garden of Forking Paths.”
- Sampling filters: Top-p, Top-k, and Min-p filters truncate the vocabulary distribution before selection, eliminating low-probability branches and retaining only surviving candidates.Their explicit objective is to prune the volatile long tail of the predictive distribution.
- Coherence–diversity trade-off: Sampling filters prioritize high-probability predictability to reduce incoherent generation, trading local text coherence against global linguistic and lexical diversity.The dilemma arises because autoregressive systems select tokens sequentially while constraining the long-tail distribution.
3 Methodology
The methodology defines WCS as a dynamic measure of whether human-authored lexical choices remain reachable under decoding constraints. It combines forced-path token audits, multi-token survival criteria, parameterized aggregate scores, and corpus-based sampling of moderately frequent words in naturalistic contexts.
- Word Coverage Score: WCS measures whether a contextually appropriate human-authored word remains mathematically reachable during sequential autoregressive generation.It distinguishes generative accessibility under decoding constraints from static lexical knowledge or isolated comprehension.
- Forced-Path Audit: The Forced-Path Audit supplies the model’s ground-truth prefix and deterministically traverses a target word’s sub-word token path.At each transition, the method records the ground-truth token’s rank and scalar probability from the full conditional distribution rather than sampling.
- Reachability Criteria: A multi-token word is covered only when every constituent token survives its sampling filter at the corresponding generation step.Reachability is binary, with R = 1 indicating that the token remains in the filter’s active vocabulary set.
- Sampling Filters: Top-k, Top-p, and Min-p define survival respectively by rank, cumulative-probability nucleus membership, and probability relative to the maximum token.Min-p is included to assess diversity preservation against traditional rank-based pruning methods.
- Aggregation and Analysis: WCS aggregates mean reachability across target words and contexts for parameter configurations θ ∈ {k, p, m}, producing lexical decay curves across parameter ranges.Per-word WCS analysis identifies words that are unreachable in all evaluated contexts and therefore removed by the sampler.
- Evaluation Data: The evaluation samples moderately frequent, non-artifactual words from the Google Web Trillion Word Corpus and tests them in naturalistic PG-19 contexts.Each context uses a preceding L = 256-token prefix, with random-entry search distributing samples across the corpus.
4 Experiments
The experiments trace target words from latent model distributions to final sequences by auditing token survivability under sampling filters. They compare base and aligned open-weight models across standardized word-context trials, decoder settings, and temperatures.
- Experimental framework: The framework comprises model selection, step-by-step token-survivability tracking, and systematic evaluation of word reachability.These phases trace how a word moves from a model’s latent internal distribution to its final generation sequence.
- Model selection: The study compares Base and Instruct/It variants of leading open-weight architectures with fewer than 20 billion parameters.Base models represent fundamental linguistic distributions, whereas aligned variants isolate the impact of alignment and distillation on lexical diversity.
- Reachability protocol: Each model pair uses Nw = 100 words and Nc = 10 contexts per word, with success requiring the complete multi-token target sequence to remain in Vθ.Reachability is defined as R = 1 for a given sampling parameter θ only when the full target sequence remains within the valid sampling set.
- Sampling conditions: Decoder sweeps cover nucleus p values from 0.7 to 0.99, Top-k values from 1 to 20, and Min-p values from 0.01 to 0.1.Min-p is evaluated for its capacity to preserve distributional diversity relative to rank-based pruning.
- Sampling conditions: Experiments use temperatures T = 0.7, 1, and 1.5, with T = 1.5 representing an aggressive high-temperature condition.The first two settings are common in chat or text-writing applications, while prior narrative-generation work associates higher temperature with weak novelty gains and reduced coherence.
5 Results
Across evaluated models, standard decoding filters substantially reduce lexical reachability, with severe word erosion persisting even at permissive Top-p settings and documented defaults. Top-k and Min-p improve coverage for most models, but model-family and alignment effects remain heterogeneous, including pronounced degradation for Gemma-4-E4B-it.
- Top-p sampling: At T = 0.7, Top-p = 0.95 still leaves most models with a significant fraction of words unreachable in every context, while p = 0.99 also excludes many words.These results indicate poor lexical reachability under commonly used nucleus-sampling settings.
- Model-family effects: Gemma alignment has opposite effects across generations: it reduces word erosion in Gemma-3-12B-it but exacerbates it in Gemma-4-E4B-it, eliminating the majority of evaluated words from reachability.Qwen and Llama families show relatively lower overall absolute word erosion than the contrasting Gemma-4-E4B-it pattern.
- Base versus instruct models: Mean word-level reachability decreases from 0.740 for base models to 0.728 for instruct models, while WCS decreases from 0.290 to 0.277.This aggregate reduction is not universal: Gemma-3-12B-it improves over its pretrained counterpart, but most matched families decline after instruction tuning.
- Documented decoding settings: 22% to 57% of words are unreachable in any context under documented default or recommended decoding settings.Table 2 evaluates practical configurations that may combine Top-p, Top-k, and temperature, rather than varying a single sampler parameter.
- Top-k and Min-p sampling: Top-k and Min-p improve word coverage for most models, yet some models still lose a significant fraction of words under aggressive settings.Gemma-4-E4B-it retains substantially lower reachability than Gemma-4-E4B across both samplers.
6 Discussion
The discussion frames WCS as a measure of complete lexical exclusion while acknowledging that sampling can also cause softer probability dampening. It links restrictive decoding to lexical narrowing and proposes semantic-guided sampling and training objectives that explicitly preserve vocabulary diversity.
- WCS scope and limitation: WCS detects binary exclusion by identifying target words that become mathematically impossible (R = 0) under a context and sampling configuration.The metric isolates total elimination of linguistic paths rather than partial reductions in probability.
- WCS scope and limitation: Words may remain in the truncated candidate pool (Vθ) yet undergo soft erosion when sampling substantially reduces their probability below levels observed in human discourse.Such words remain technically reachable, but their generation frequency declines; WCS does not capture this probabilistic erosion.
- Decoding trade-offs: Recovering sophisticated words in the 10k–40k frequency band often requires p near unity or temperatures above typical defaults, yet many words remain unreachable even at T = 1.5.The findings describe a tension between lexical reachability and structural coherence rather than a complete decoding solution.
- Broader implications: Restrictive sampling mechanics compress the active human lexicon to a high-probability subset, producing systemic lexical erosion with implications for language evolution.The discussion connects this narrowing to the increasing presence of model-generated text in public repositories, education, and digital publications.
- Future directions: Mitigation proposals include alternative or optimized samplers, semantic-guided decoders, lexical reachability penalties, and preference objectives that reward stylistic and lexical variation.The discussion ultimately treats vocabulary diversity as an architectural objective alongside factual correctness and syntactic coherence.
7 Conclusion
The paper introduces WCS as a framework for isolating inference-time token truncation and auditing token survivability, revealing a gap between latent vocabulary knowledge and empirical use. Results show default decoding filters restrict meaningful lexical paths, motivating semantic-guided decoding and objectives that treat lexical breadth as fundamental.
- 7 Conclusion: WCS isolates the direct impact of inference-time token truncation on vocabulary use through a naturalistic, long-context Forced-Path Audit.The audit tracks token survivability step by step and exposes divergence between latent vocabulary knowledge and empirical use.
- 7 Conclusion: Typical decoding filters act as pruning mechanisms that fundamentally restrict vocabulary use across several open-weight models.In the 10k–40k word frequency band, default samplers systematically erase meaningful lexical paths and omit a significant fraction of words.
- 7 Conclusion: Mitigating word erosion requires semantic-guided decoders and training or preference objectives that treat lexical breadth as a first-class loss penalty.The paper frames vocabulary diversity as a core architectural objective alongside factual correctness and syntactic grammar.
8 Limitations
The WCS analysis is limited by its model, language, corpus, frequency-band, metric, context-window, and tokenizer choices. These constraints restrict generalizability and leave several forms of lexical behavior unexplored.
- Model Architecture and Parameter Constraints: The evaluation covers only open-weight architectures below 20 billion parameters, leaving frontier-model behavior unexplored and requiring complete step-wise logit distributions.The threshold was chosen for computational accessibility and reproducibility among independent researchers.
- Linguistic and Historical Bias: The English-only pipeline uses English-centric corpora and pre-1919 PG-19 books, introducing historical bias despite avoiding modern web-text stylistic homogenization.The context selection therefore does not represent languages or post-1919 writing.
- Boundary Bounds of Frequency Bands: The frequency-based token selection examines only ranks 10,000–40,000, so effects on words outside this band remain unexplored.This window avoids ubiquitous functional words and extreme low-frequency noise.
- Binary Trait of the WCS: Because WCS defines reachability as binary, it detects absolute exclusion but misses substantial probability suppression when tokens remain inside the valid sampling set.The metric uses R ∈ {0, 1} and does not quantify soft reductions in probability mass.
- Context and Tokenization Constraints: The 256-token forced-path window omits long-range dependencies, while model-specific sub-word segmentation makes reachability comparisons across models imperfect for some words.Different token fragment counts require different numbers of filtering steps for a target word to remain reachable.
A Appendix: Selected Word List and Aggregate Reachability
The appendix evaluates 100 target words from the 10k–40k frequency-rank band across 1,000 forced-path audit samples and aggregates their reachability across evaluation conditions. Word-level reachability varies substantially, with source frequency explaining only a limited share of the variation (r = 0.29).
- Selected Word List: 100 target words from the 10k–40k frequency-rank band were paired with 10 PG-19 contexts each, yielding 1,000 forced-path audit samples.The targets span a selected frequency band, with more common words earlier and rarer words later in the list.
- Aggregate Reachability: Mean per-word reachability averages the fraction of evaluated model, sampler, context, and temperature conditions in which each target remained reachable.This provides a word-level view of targets consistently preserved or frequently pruned by decoding.
- Frequency Relationship: r = 0.29: Log-transformed source frequency has only a weak positive Pearson correlation with mean word-level WCS.The result indicates that corpus frequency alone explains only a limited portion of lexical-reachability variation.
- Low-Reachability Words: The hardest targets included supposedly (0.076), exceptionally (0.107), acknowledges (0.119), sylvan (0.079), saddened (0.082), and precipitated (0.087) in mean reachability.These low-reachability words span middle, upper, and lower parts of the sampled frequency range rather than representing only the rarest targets.
- High-Reachability Words: The easiest targets included profitable (0.537), disadvantage (0.526), bedside (0.501), offenders (0.494), volley (0.446), feces (0.444), bridle (0.434), and workmen (0.389).Both relatively frequent and lower-frequency words appear among the comparatively reachable targets.