Cross-Entropy Risk Estimation for Language Models: Inconsistency Must Be Dense, and the Holdout Method Is No Exception

Hanti Lin

Source-linked AI summary

Cross-Entropy Risk Estimation for Language Models: Inconsistency Must Be Dense, and the Holdout Method Is No Exception

Hanti Lin

arXiv:2608.15798v1Aug 16, 2026cs.LGstat.ML

TL;DR

The paper asks whether held-out per-token cross-entropy risk can be consistently estimated, despite its central role in comparing language models and fitting scaling laws. It formalizes consistency over data-generating distributions and trained models, proving dense impossibility results while identifying two conditional ways to restore consistency.

Problem
Held-out per-token cross-entropy risk underpins model comparison and scaling laws, but whether it can be consistently estimated across possible data-generating distributions and trained models remains unresolved.
Method
The paper defines pointwise consistency over pairs of data-generating distributions and models, then analyzes estimability under unrestricted and restricted classes.
Results
Inconsistency persists even with bounded expected sequence length and full-support models, and the states where it occurs are dense in that restricted class.
Takeaways & Limitations
Consistency can be recovered either by flooring model probabilities or by reporting risk only below a fixed threshold, but both approaches revise the estimation problem or add an untestable assumption.
Takeaways & Limitations
The probability floor does not control risk magnitude and requires finite expected sequence length, an assumption no amount of data can verify.

Abstract

from arXiv · show

Language models are compared by their held-out per-token cross-entropy risk---the quantity scaling laws are fitted to. We show that it cannot be consistently estimated. Consistency, or convergence to the estimand, is defined relative to a \emph{possible state of the world}: a pair consisting of a data-generating distribution and a model we turn out to train. Quantifying over models as well as data-generating mechanisms is essential, because what decides whether a model's risk is estimable is a tail property of the distribution its weights induce, which no sample reveals. The per-token cross-entropy risk is hard to estimate because of a topological fact: among the possible states, finite risk and infinite risk each lie arbitrarily close to every instance of the other. Consequently no estimator---not merely the holdout average---is consistent at every state at which the risk is defined. Worse, inconsistent estimation persists under both bounding the expected sequence length and restricting to full-support models; and in that restricted setting the states at which inconsistency occurs are even dense. Two interesting ways out are identified, and neither is free. Way out 1: using a bounded context window, we can floor a model's next-token probabilities, making its risk finite exactly when the data-generating distribution has finite expected sequence length---a new, statistical rationale for a choice that was made on computational grounds, though the assumption it substitutes is itself beyond the reach of any test. Way out 2: reporting the risk only when it falls below a threshold fixed in advance restores consistency, at no cost to what model selection actually requires---but we need to recognize that the goal of estimation is revised.

1 Introduction

Held-out per-token cross-entropy risk guides language-model comparison and scaling laws, but cannot be consistently estimated at every defined state when consistency is indexed by both data distribution and evaluated model. The impossibility persists under bounded expected sequence length and full-support models, while bounded contexts and thresholded reporting provide two trade-off-laden ways out.

Motivation: Held-out per-token cross-entropy risk determines checkpoint selection, summarizes model quality, and is the quantity fitted by scaling laws.It is estimated from average negative log probabilities on text withheld from training.
Problem: Per-token cross-entropy risk may be infinite because language models assign positive probability to countably many sequences, producing an infinite sum that can diverge.Infinite risk is not restricted to contrived cases.
Conceptual refinement: Consistency is defined at a possible state of the world, the pair (p, q) comprising the data-generating distribution and the model being evaluated.This refinement makes the estimand’s dependence on both arguments explicit.
Impossibility results: No estimator is consistent at every state where the risk exists, including finite and infinite-risk states, even after bounding expected sequence length and restricting models to full support.The impossibility applies to every estimation procedure, not only the holdout average.
Impossibility results: On the restricted class, inconsistent states are dense: every neighborhood of every candidate state contains a state where the estimator fails.Neighborhood-based assumptions therefore cannot certify consistency.
Way out 1: A bounded context window over a finite vocabulary floors next-token probabilities above β > 0, making risk finite exactly when expected sequence length is finite.The resulting loss grows only linearly in sequence length.
Way out 2: Reporting risk only below a fixed threshold r, while otherwise reporting that it is at least r, restores consistency whenever the risk is defined.This changes the estimand but requires no assumption beyond the law of large numbers.
Statistical learning perspective: Pretraining is a two-stage procedure: many candidate models are trained, then sifted by estimated held-out risks, with the survivor becoming the output.Risk estimation therefore supports model selection as well as model evaluation.

2 Related Work

The paper situates its impossibility result within statistical work on untestability, emphasizing that its obstruction is the finite-versus-infinite nature of the risk estimand. It also connects the result to entropy estimation, conditional-independence testing, topology-based testability, and other countable-alphabet functionals.

Classical impossibility results: Bahadur and Savage showed that no nontrivial test of a mean hypothesis is available over a sufficiently rich distribution class.Donoho further showed that one-sided inference can remain possible when two-sided inference fails.
Distinct obstruction: The present obstruction differs because the risk estimand may be non-real, with finite- and infinite-risk states dense in each other’s territory.Unlike Bahadur–Savage, the key boundary is between finite and infinite risk rather than between hypotheses about an always-defined real parameter.
Entropy estimation: Antos and Kontoyiannis established universal consistency for entropy and related additive functionals over finite-entropy distributions, while obtaining no convergence rate without further assumptions.The paper argues that finite-entropy restriction does the essential work because corresponding class membership cannot be determined from data.
Topological untestability: Conditional-independence testing exhibits a related mechanism: when the conditioning variable is continuous, the null is dense among alternatives, making valid tests powerless against every alternative.This parallels the paper’s use of topological denseness as a source of untestability.
Topology and distributional functionals: Other related work links statistical testability to topology and shows that missing mass is not distribution-free learnable, requiring restrictions such as tail assumptions.Genin and Kelly, Ermakov, and subsequent work characterize testability through topological conditions on hypotheses; Mossel and Ohannessian identify heavy-tail assumptions as necessary for rare-event prediction.

3 Setting

The paper studies per-token cross-entropy risk for autoregressive models over a countably infinite space of finite sequences. It formalizes holdout estimation and consistency at each possible state of the world, where the data distribution and trained model are both quantified.

3 Setting: The sample space contains all finite sequences ending in eos, and is countably infinite because sequence length is unbounded.The vocabulary is finite, and V0 is assumed nonempty.
3 Setting: Per-token risk normalizes sequence-level cross-entropy by expected sequence length m(p), requiring m(p) < ∞.Its exponential is the usual perplexity.
3 Setting: Finite and infinite per-token risk occur arbitrarily close to every state of the other kind under arbitrarily small modifications of p, q, or both.Thus finiteness is not settled by any amount of care in specifying the data-generating distribution and model.
3 Setting: The holdout estimator averages total observed loss over total observed tokens using complete sequences drawn independently from p and independently of q.The model q is consulted during evaluation, as required by the procedure.
3 Setting: When ρ(p, q) = ∞ but q assigns positive probability to every sequence, the holdout estimate is finite at every sample size and converges almost surely to infinity without reporting ∞.The strong law applies to the nonnegative losses even when their mean is infinite.
3 Setting: Consistency is defined at a state (p, q), and pointwise consistency requires an estimator to be consistent at every state with m(p) < ∞.A state pairs the data-generating distribution p with the trained model q being evaluated.

4 Main Results

The per-token risk cannot be consistently estimated across all states where it is defined, even after restricting sequence lengths and models to full support. This failure is dense, because estimability depends on tail properties of the model-induced distribution that model weights do not reveal.

Main impossibility result: Finite expected sequence length is necessary for the risk to exist, but every estimator fails pointwise consistency on P<∞ × Q.P<∞ × Q is the largest state space on which the risk is everywhere defined.
Restricted state spaces: Even restricting expected sequence length below any fixed L > 1 and requiring full-support models, every estimator remains pointwise inconsistent.Full support requires every conditional token probability, including eos, to be positive; softmax models satisfy this automatically.
Dense inconsistency: For every L > 1, inconsistent states are dense in P<L × Q>0, so no arbitrarily small neighborhood guarantees local reliability.Dense inconsistency means failures occur arbitrarily close to every state in the restricted space.
Why states include models: Consistency must quantify over model q as well as data-generating distribution p because the holdout method is applied across many models, not one fixed model.This matters for scaling laws, which fit curves through held-out risks for a family of models.
Why models remain uncertain: Knowing every model weight does not reveal whether risk is estimable: success requires −log q(x) to grow linearly with sequence length, whereas faster growth causes failure.The relevant tail property is a mathematical fact about the induced distribution, and the impossibility results hold even with complete access to q.

5 Two Possible Ways Out

Two ways out restore useful held-out evaluation: impose a probability floor, often via bounded context, or report risks only relative to a fixed threshold. Each changes the guarantees or assumptions, and neither resolves every limitation of unrestricted risk estimation.

Way out 1: floors: On P<∞ × floored models, the holdout method is pointwise consistent because a floor makes per-token risk finite.If q has floor β, then ρ(p, q) ≤ log(1/β) < ∞.
Way out 1: floors: A finite context window supplies a floor by limiting models to finitely many strictly positive next-token distributions.The model can distinguish only finitely many conditioning contexts, so their positive probabilities have a positive minimum.
Way out 1: floors: When the window is at least every training-sequence length, post-training windowing preserves empirical-risk minimization while making held-out risk consistently estimable.If any training sequence exceeds the window, truncation can increase empirical risk and invalidate the minimizer guarantee.
Way out 1: floors: The floor guarantee is conditional and quantitatively weak: real β values around e−30 make ρ ≤ log(1/β) finite but useless, while p ∈ P<∞ remains untestable.The statistical rationale is for securing a floor, not specifically for context windows; other mechanisms, such as logit caps, can also suffice.
Way out 2: thresholded reporting: A fixed threshold r enables consistent screening: eventually report a value when ρ(p, q) < r and verdict “ρ ≥ r” when ρ(p, q) > r, including infinity.The procedure may vacillate when ρ(p, q) = r, and it revises the goal from estimating a level to comparison and veto decisions.

6 More Possible Ways Out

This section maps the remaining ways out of the impossibility result by identifying which assumptions must be abandoned. It finds that bounding sequence length is the only listed exit yielding a positive theorem outright, while other escapes involve substantial costs or limitations.

Conditions on the table: The impossibility result jointly assumes an unbounded sequence space, per-token cross-entropy risk, infinite estimation error as a possibility, and consistency or even no dense inconsistency.The state space combines data-generating distributions with autoregressive models, and dense inconsistency is evaluated in total variation.
Conditions on the table: The analysis also restricts distributions to bounded expected sequence length, models to full support, and samples to IID holdouts.The expected-length bound is a fixed constant L > 1, while both bounded expected length and full support are stronger restrictions rather than weakenings.
Remaining exits: The evaluative standard cannot be abandoned, and abandoning IID sampling would make matters worse because dependence can only slow the analysis.Pointwise consistency demands nothing at finite sample sizes, while avoiding dense inconsistency is already a weakened requirement.
Where the above two exits sat: The Section 5 floor modifies the model class, while screened reporting modifies the estimand, respectively imposing an architectural commitment and withholding information about how bad a bad model is.The floor narrows Q>0 to models with a floor; screened reporting replaces per-token risk with a coarser target.
What the remaining exits come to: Bounding sequence length works completely and is the only exit yielding a positive theorem outright, but it combines two other conditions and provides no document-length bound.The passage states that neither of those constituent conditions suffices alone.
What the remaining exits come to: Other alternatives trade away meaning or scope: Brier scores are estimable at Hoeffding rates but weakly informative, KL divergence is unavailable to holdout procedures, finite error changes the estimand, and finer topology only partly helps.Finer topology can defeat denseness in one KL direction, but leaves Theorems 1 and 2 untouched.

7 Closing · A Proofs · A.1 Reduction to a Testing Problem

The paper argues that held-out per-token risk estimation requires assumptions about sequence length and model loss that practice leaves implicit. It identifies flooring and threshold screening as consistent alternatives, and reduces impossibility of risk estimation to an unsolvable finiteness-testing problem.

7 Closing: Held-out evaluation is only meaningful when the data-generating distribution has finite expected sequence length and the model’s loss does not outrun that length.These conditions determine whether per-token risk exists and is finite, and neither is settled by any finite sample.
7 Closing: For models with a probability floor, including bounded-context-window models, finite expected sequence length is also sufficient for finite per-token risk.The floor is supplied by the finite context window, though the required expected-length assumption remains untestable.
7 Closing: Threshold screening restores consistency for candidate selection, but reports no information about how poor a rejected model is.That limitation is immaterial when the risk estimate is used only to select among candidates.
7 Closing: Risk estimation is central to pretraining theory because the deployed algorithm combines empirical risk minimization on training data with model selection by held-out risk estimation.Flooring a minimization-trained model can provide consistent estimation, and when the context window is at least the training length it need not increase empirical risk.
A Proofs: The appendix reduces estimation of per-token risk ρ to testing whether sequence risk R is finite, then proves that testing problem unsolvable for a suitable model q and every admissible L.The final assembly obtains density in the state space from density in each component.
A.1 Reduction to a Testing Problem: The reduction partitions P<∞ into H0, where R(p, q) < ∞, and H1, where R(p, q) = ∞.A test outputs “finite” or “infinite” and is consistent when its verdict probabilities converge to the corresponding truth at each p.
A.1 Reduction to a Testing Problem: Any estimator ˆρ consistent at (p, q) induces a test consistent at p, so dense inconsistency of the test implies dense inconsistency of the estimator.Thus it suffices to exhibit, for a suitable q, a testing problem that no test solves.

A.2 Four Lemmas

The appendix establishes continuity of finite-sample events and constructs dense witness models whose superlinear loss enables finite- and infinite-risk perturbations. Consequently, finite and infinite cross-entropy-risk states are both dense, even among full-support models.

Continuity and witnesses: For any sample size n, event probabilities are Lipschitz in the data distribution with total-variation constant n, uniformly over events.Every p′ within δ/n of p changes an event probability by less than δ.
Continuity and witnesses: Witness models have full support, finite expected length, and sequence loss growing at least quadratically; they are dense in Q and therefore in Q>0.No witness has a probability floor.
Dense risk regimes: For every p with finite expected length and every ϵ > 0, a nearby p′ has finite risk while preserving or reducing expected sequence length.The construction truncates the tail and moves its mass to a shortest sequence.
Dense risk regimes: For every finite-length p and all ϵ, η > 0, a nearby p′′ has infinite risk while increasing expected length by at most η.The perturbation assigns small mass across long sequences: expected length remains cheap, but quadratic witness loss makes risk diverge.
Dense risk regimes: In P<∞ × Q, both infinite-risk and finite-risk states are dense, and both remain dense when models are restricted to Q>0.The result follows by combining the two perturbation lemmas with witness-model density.

A.3 No Test Succeeds on P<L · A.4 Proofs of the Theorems · B Two Details of the Setting

The paper proves that every test is densely inconsistent for distinguishing finite expected length from its complement, and uses this result to establish dense inconsistency of the cross-entropy risk estimator. The proofs rely on alternating nearby hypotheses and a convergent nested-ball construction.

A.3 No Test Succeeds on P<L: The budget keeps every constructed distribution in P<L despite the class not being closed under total-variation limits.Only even stages spend budget, while completeness and the uniform expected-length bound place the limit inside P<L.
A.3 No Test Succeeds on P<L: Every test of H0 against H1 is densely inconsistent with respect to P<L.For any neighborhood in P<L, the construction finds a distribution at which the test is inconsistent.
A.3 No Test Succeeds on P<L: The construction alternates hypotheses inside shrinking nested balls, forcing test verdicts above 0.9 at even stages and below 0.1 at odd stages.The resulting sequence of test probabilities has no limit, contradicting consistency at the limit distribution.
A.3 No Test Succeeds on P<L: Every test of finite expected length is densely inconsistent on P.Finite- and infinite-expected-length distributions are both dense: truncation supplies the former, while arbitrarily small perturbations supply the latter.
A.4 Proofs of the Theorems: Theorem 3 follows by fixing a dense witness q, inducing a test from the estimator, and applying Proposition 2 to find inconsistency in every nonempty relatively open subset of P<L × Q>0.Thus the estimator’s inconsistent states are dense in the product state space.
A.4 Proofs of the Theorems: Theorem 2 follows because P<L × Q>0 is nonempty, while Theorem 1 follows by inclusion into P<∞ × Q.An estimator inconsistent at one state is not pointwise consistent on the corresponding space.

B.1 The Per-Token Risk from Autoregressive Primitives · B.2 Training by Empirical Risk Minimization · C Ways Out: Details

The appendix reformulates per-token risk directly over autoregressive prediction events, showing this construction exists exactly when the data-generating process has finite expected sequence length and agrees with the quotient definition. It also identifies empirical risk minimization with maximum likelihood and introduces the detailed treatment of the six conditions behind the impossibility result.

B.1 The Per-Token Risk from Autoregressive Primitives: [B.1] When m(p) < ∞, direct expected next-token loss under prediction events equals the quotient definition of per-token risk for every model q.The identity holds in [0, ∞], with both sides infinite together.
B.1 The Per-Token Risk from Autoregressive Primitives: [B.1] Prediction events pair a finite context with its next token, and their valid distributions are characterized by kernel agreement and prefix consistency.Kernel agreement assigns the next-token conditional, while prefix consistency links reaching an extended context to emitting its preceding token.
B.1 The Per-Token Risk from Autoregressive Primitives: [B.1] A prediction-event distribution satisfying kernel agreement and prefix consistency exists exactly when m(p) < ∞, and is then unique.Its normalization forces the context marginal at the empty string to equal 1/m(p).
B.1 The Per-Token Risk from Autoregressive Primitives: [B.1] Almost-sure termination does not ensure finite expected sequence length: a process can halt almost surely while m(p) = ∞.For p(a | a^{k−1}) = k/(k + 1), Pp(|X| > k) = 1/(k + 1), yielding m(p) = ∑_{k≥0} 1/(k + 1) = ∞.
B.2 Training by Empirical Risk Minimization: [B.2] Under the i.i.d. idealization, minimizing empirical logarithmic risk over a parameterized model family is equivalent to maximum likelihood.Autoregressive factorization turns the objective into the familiar next-token cross-entropy objective.
B.2 Training by Empirical Risk Minimization: [B.2] Averaging training loss over observed tokens instead of sequences leaves the minimizers unchanged because the denominator is fixed independently of model parameters.The distinction between sequence and token denominators matters only during evaluation, where the denominator is random.
C Ways Out: Details: [C Ways Out: Details] The appendix treats six conditions that jointly produce the impossibility result, as listed in Section 6.This appendix proceeds through those conditions one at a time.

C.1 Sequence Space: Bound the Length

Capping sequence length at N makes the cross-entropy risk finite and the holdout estimator uniformly consistent at Hoeffding rates. But this works only by jointly restricting both the model and distribution, while treating N as an advance decision rather than a discovered fact.

C.1 Sequence Space: Bound the Length: Capping sequences at N makes X finite and yields finite risk for every distribution and model on the truncated space.The bound is R(p, q) ≤ max|x|≤N(−log q(x)) < ∞.
C.1 Sequence Space: Bound the Length: The holdout estimator becomes uniformly consistent across all truncated-space distributions, with Hoeffding-rate guarantees.This is a positive theorem, not merely a proof failure.
C.1 Sequence Space: Bound the Length: The cap combines two restrictions: models may vanish outside the capped set, while data-generating distributions are limited to bounded sequence lengths.Neither restriction alone suffices: model zeros can be contradicted by the truth, and bounded expected length can be defeated by excessive model loss.
C.1 Sequence Space: Bound the Length: No data-derived fact supplies N; choosing it means declaring in advance that sequences beyond that length do not occur.The cap discards the distribution’s tail rather than modeling it.

C.2 Estimand: Estimate Something Else

The section considers alternative estimands that avoid absolute held-out cross-entropy’s impossibility. Bounded proper scores are estimable but lose resolution and language-model diagnostics; divergence differences preserve ranking, while finite-threshold screening restores consistency at a declared cost.

Bounded proper scores: Locality and strict propriety force an unbounded loss, making logarithmic loss useful for training but fatal for evaluation.A bounded local loss is flat on a tail and therefore not strictly proper.
Bounded proper scores: Replacing evaluation with a bounded proper score such as Brier makes risk uniformly consistent, but sequence-level Brier scores collapse toward 1 and lose resolution.The Brier score lies in [0, 2], and both risk components become nearly zero for language-model sequence spaces.
Bounded proper scores: Bounded scores also forfeit the chain rule, per-token diagnostics, bits-based interpretation, and comparability with perplexity and scaling-law results.Whether small-outcome-space evaluations can replace held-out cross entropy remains an open empirical question.
Estimating the divergence: KL divergence is not holdout-estimable because its entropy term requires evaluating the unknown data-generating probability, although differences between models cancel that common term.The cancellation preserves model ranking when the relevant risks are finite.
Screening the estimand: Screening risk at any finite threshold restores consistency, whereas the unscreened infinite-risk question remains impossible; the threshold converts an implicit finiteness assumption into a declared parameter.The cost is non-uniformity in the threshold, which cannot be driven to zero.

C.3 Estimation Error: Redefine How Error Is Measured · C.4 Distribution Class: Assume Finite Expected Length · C.5 Model Class: More Than Full Support

The proposed exits either redefine the estimand or restrict distributions in ways that alter or relocate the original assumption. A bounded context window supports consistency under finite expected length, but that condition is equivalent to finite risk for floored models and is untestable.

C.3 Estimation Error: Redefine How Error Is Measured: Treating every estimation error as finite requires replacing infinite risk with a real-valued estimand.If ρ(p, q) = ∞ and the estimate is finite, the proposed finite penalty changes what is being estimated.
C.3 Estimation Error: Redefine How Error Is Measured: The proposed error function can equal distance from an estimand only when f(x) = |r′ −x| for one real number r′.Thus, the finite-error proposal is mathematically equivalent to selecting a different real-valued estimand.
C.3 Estimation Error: Redefine How Error Is Measured: That revised quantity is no longer per-token risk and carries the burden of showing that it is worth knowing.The strategy may be appropriate, but it inherits the costs of redefining the target.
C.4 Distribution Class: Assume Finite Expected Length: Restricting data-generating distributions to finite expected length makes holdout estimation consistent when a bounded context window floors next-token probabilities.This is the exit developed in Section 5.1.
C.4 Distribution Class: Assume Finite Expected Length: ρ(p, q) ≤ log(1/β) whenever m(p) < ∞, while ρ(p, q) is undefined when m(p) = ∞ for a model with floor β.The floor bounds risk under finite expected length and excludes infinite expected length from the estimand’s domain.
C.4 Distribution Class: Assume Finite Expected Length: Finite expected length does not independently guarantee finite risk: with a probability floor, it is equivalent to finite risk and is untestable.The exit relocates rather than discharges the assumption at issue.
C.4 Distribution Class: Assume Finite Expected Length: Moment assumptions rescue additive-functional estimation on countable alphabets, and held-out language-model evaluation implicitly assumes finite expected length.The displayed equivalence identifies which moment assumption the evaluation practice makes.
C.5 Model Class: More Than Full Support: Restricting the model class is another strategy discussed in Section 5.1.This subsection records the model-class restriction as a separate route from distributional restriction.

C.6 Topology: Use a Finer Topology

A topology finer than total variation is the only possible route to confining dense inconsistency, but divergence-based topologies escape only in one direction. The discrete topology removes the impossibility by abolishing meaningful small perturbations rather than refining them.

Finer-topology requirement: The weak topology cannot help: on countable discrete spaces it coincides with total variation, while Hellinger topology is topologically equivalent to it.Weak convergence is coordinatewise convergence of mass functions and implies total variation convergence; the Hellinger metrics are topologically equivalent.
Divergence topology: Kullback–Leibler divergence generates a strictly finer topology because small divergence implies small total variation, but not conversely.Small total variation changes can move mass onto sequences assigned minuscule probability by q, which is cheap in variation and expensive in divergence.
Divergence topology: The divergence-based escape depends on direction: neighborhoods defined by D(p′∥p) can avoid the perturbation, whereas D(p∥p′) preserves dense inconsistency.With the reverse direction, the perturbation only adds mass, and the relevant divergence tends to zero with µ.
Limiting case: The discrete topology makes the impossibility evaporate because no proper subset is dense and distinct distributions are never close.This abolishes the notion of small perturbation rather than refining it, so any intermediate proposal must explain what prevents that endpoint.