Evaluating Distributional Distortion in Neural Language Modeling

1 INTRODUCTION

• Natural language’s heavy-tail contains many individually improbable events that collectively account for substantial probability mass.The passage links this tail to speakers’ frequent production and comprehension of novel expressions.Source: §1 · p. 1
• Instance-level performance on rare sequences remains poorly understood because prior work has not explored it directly.The cited branches study distributional tendencies, modes, and rare-event performance, but not instance-level LM performance on rare sequences.Source: §1 · p. 2
• The evaluation trains LSTM and Transformer LMs on samples from Transformer-generated artificial languages with exactly computable sequence probabilities.Controlling the generative model’s conditional entropy creates target languages with varying distributional properties.Source: §1 · p. 2
• LSTM and Transformer LMs underestimate target-language sequence probabilities more severely for rare sequences, while tending to overestimate rare perturbed sequences.More training data weakens but does not eliminate underestimation, and lower-entropy target distributions exacerbate it.Source: §1 · p. 2

2 BACKGROUND

• Productivity means finite linguistic experience can support an unbounded number of grammatically acceptable utterances, yielding a heavy-tail of rare sequences.Such distributions place non-zero probability on unseen events at practical sample sizes.Source: §2 · p. 2
• The LNRE zone comprises sample sizes at which the probability of drawing a novel event remains positive, complicating straightforward maximum-likelihood estimation.The passage describes natural language’s LNRE zone as very large and novelty accounting as a central language-modeling challenge.Source: §2 · pp. 2–3
• A locally normalized autoregressive LM defines sequence probabilities through a chain-rule decomposition of token probabilities.Its neural network computes a score for each token conditioned on the preceding sequence.Source: §2 · p. 3
• Unseen sequences can have very different probabilities, even when both have never appeared in an English sample.The passage contrasts a plausible novel sentence with a nonsensical sequence as examples.Source: §2 · p. 3
• Mean holdout perplexity measures average likelihood on unseen instances, but does not establish instance-level alignment or performance on rare, idiosyncratic events.The true English distribution and its generative process are unknown, complicating direct evaluation of sequence probabilities.Source: §2 · p. 3

3 LANGUAGE MODEL EVALUATION IN THE LNRE ZONE

• Training language models on samples from pL, the evaluation compares each sequence's estimated probability pM(x) with its target probability pL(x).Table 1 summarizes the components of the scheme.Source: Table 1 · p. 3, §3 · pp. 3–4

3.1 ARTIFICAL LANGUAGES AS TARGET DISTRIBUTIONS

• The artificial language pL is induced by ancestral sampling from a GPT2-medium trained on 1.5M OpenWebText sentences, using softmax temperature T = 0.85.Table 2 gives sample sequences from this distribution.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 SEQUENCE PROBABILITY ESTIMATION IN THE LNRE ZONE

• Model estimation error is the difference between a model's and the target's sequence log probabilities; negative values indicate underestimation.Sequence probabilities are computed by chain-rule decomposition, with target probabilities accounting for softmax tempering.Source: §3.2 · p. 4

3.3 NEURAL LANGUAGE MODELS

• The experiments evaluate Transformer and LSTM models; Figure 2 presents joint probability-estimate histograms and mean estimation error by target sequence probability.Transformer variants include GPT2-small, GPT2-medium, and GPT2-large; model selection uses withheld validation cross-entropy unless otherwise noted.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

4.1 ESTIMATION ERROR WITH FIXED DATA

• LSTM and GPT2 models consistently underestimate probabilities of sequences sampled from pL, with more severe error for rarer sequences.GPT2-medium performs only slightly better than GPT2-small, typically within the bootstrapped 95% confidence intervals.Source: §4.1 · p. 5

4.2 ESTIMATION ERROR ACROSS TRAINING TIME

• By about epoch five, GPT2-medium nearly recovers training-sequence probabilities across rarity levels, but held-out sequences remain underestimated with persistent rarity-dependent gaps.Training error converges to zero almost simultaneously across sequences; test error does not converge to zero.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 ESTIMATION ERROR BY AMOUNT OF TRAINING DATA

• Figure 4 shows that added training data initially lowers error before improvement asymptotes; Transformers reduce mean estimation error more efficiently than LSTMs.The inset relative-change curves eventually fluctuate around zero for most of the distribution.Source: Figure 4 · p. 6, §4.3 · p. 7

4.4 WHERE DID THE PROBABILITY MASS GO?

• As perturbations increase, sequences shift from well-estimated to overestimated, suggesting the model favors sequences outside the rare-event tail defined by pL.Figure 5 maps estimation error against sequence rarity and perturbation degree.Source: Figure 5 · p. 7, §4.4 · p. 8

4.5 MODULATING THE SHAPE OF THE TARGET DISTRIBUTION

• GPT2-medium has less severe estimation error on higher-entropy target languages across nearly all target probabilities, approximating more uniformly spread distributions more accurately.Increasing temperature spreads probability mass across more sequences in Σ∗.Source: §4.5 · p. 8, Figure 6 · p. 8

• Prior LM-distribution work examined broad statistical tendencies, including Zipf’s law, and frameworks for characterizing them (Takahashi & Tanaka-Ishii, 2019; 2017; Meister & Cotterell, 2021).Source: §5 · pp. 8–9
• This work uses instance-level evaluation and controlled artificial corpora, alongside related work using generative-model corpora to study mode recovery (Kulikov et al., 2021).Source: §5 · p. 9
• Rare-event research spans under-resourced languages, under-represented domains, sequential and syntactic novelty, and low-frequency words; earlier n-gram approaches used smoothing and back-off (Oren et al., 2019; McCoy et al., 2021).Source: §5 · p. 9

• The evaluation compares LSTM and Transformer probability estimates with exact sequence probabilities in artificial languages generated by models trained on natural-language corpora.Source: §6 · p. 9
• Neural LMs under-represent target-tail sequences even with more training data, overestimate atypical sequences rather than the head, and approximate higher-entropy targets more accurately.Source: §6 · p. 9
• Together, these results indicate that autoregressive neural LMs tend to spread probability mass too uniformly across possible sequences.Source: §6 · p. 9
• The structural difference between Transformer-generated ground-truth distributions and actual natural-language distributions is unknown, and may affect the results.Source: §6 · p. 9

• The ground-truth distribution is a pretrained GPT2-medium model fine-tuned on 1.5M OpenWebText sequences, sampled with softmax T = 1.00.Source: §A.1.1 · p. 13
• The LM underestimates the probability of most test sequences, more severely for less-probable sequences; its test perplexity is 67.97.Source: §A.1.1 · p. 13, Figure 7 · p. 13
• The model overestimates the probability of most randomly sampled ill-formed sequences, extending that finding to a pretrained-model ground truth.Source: §A.1.1 · p. 13, Figure 7 · p. 13

• The artificial language uses ancestral sampling with T = 1.00, rather than the T = 0.85 setting used in Sections 4.1–4.4.Source: §A.1.2 · p. 13
• GPT2-medium and GPT2-large are trained on a total of 30M sequences sampled from that artificial language.Source: §A.1.2 · p. 13
• Model estimation error is computed on withheld test sequences at each training iteration.Source: §A.1.2 · p. 13

A.1.3 PRE-TRAINED MODEL ESTIMATION ERROR

• Pretrained GPT-2 models underestimate probabilities for most target-language test sequences, with more severe errors for rarer sequences.The section reports this pattern for GPT2-small, -medium, and -large fine-tuned on 1M target-language sequences.Source: §A.1.3 · p. 13, Figure 9 · p. 14
• The experiments fine-tune pretrained GPT2-small, -medium, and -large models on target-language sequences and evaluate on unseen test sequences.The 1M-sequence fine-tuning set is used for the initial comparison; the larger-data experiment uses GPT2-medium.Source: §A.1.3 · p. 13, Figure 8 · p. 14
• More fine-tuning data lessens but does not eliminate underestimation in GPT2-medium trained on 30M target-language sequences.Figure 10 tracks mean and relative estimation error across fine-tuning data amounts, with darker lines representing less probable sequences.Source: §A.1.3 · p. 14, Figure 10 · p. 14

A.1.4 ALTERNATIVE PERTURBATIONS

• GPT2-medium overestimates randomly sampled strings across true sequence probabilities, extending the alternative-perturbation tests to sequences sampled from Σ*.The model and target are GPT2-medium variants, with the target using temperature 0.85 and the model trained on 30M ancestrally sampled sequences.Source: §A.1.4 · p. 15, Figure 11 · p. 15

A.1.5 ESTIMATION ERROR BY SEQUENCE LENGTH

• As sentence length increases, GPT2-small and -medium show observed estimation error above n times average token error, so compounding alone does not account for the pattern.Expected error is computed as n times average token-level estimation error; the figure reports 95% bootstrapped confidence intervals for observed error.Source: §A.1.5 · p. 15, Figure 12 · p. 15

A.2 EMPIRICALLY MEASURING THE LNRE ZONE

• The appendix introduces an empirical estimate of potential productivity to test whether a sample lies in the LNRE zone.It uses the Good-Turing estimate to estimate probability assigned to unseen events from a sample.Source: §A.2 · p. 16, §A.2.1 · p. 16

A.2.1 ESTIMATING THE POTENTIAL PRODUCTIVITY

• Setting m = 0 in the Good-Turing estimate makes potential productivity equal the probability mass allocated to unseen events, estimated from singleton types.The method estimates event probability from observed frequency and the counts of types at each frequency.Source: §A.2.1 · p. 16

A.2.2 THE LNRE ZONE IN OPENWEBTEXT

• For OpenWebText n-grams with n ≥3, the estimated probability of a novel event remains significant after more than 10,000,000 samples.Figure 13 plots the estimate as sample size increases.Source: §A.2.2 · p. 16, Figure 13 · p. 16

A.3 MODEL PERPLEXITY VALUES

• The appendix reports mean perplexity on each model’s artificial-language validation set and on OpenWebText, without claiming the target models define English.Tables 4 and 5 identify the validation and English perplexity columns for the reported models.Source: §A.3 · p. 16, Table 4 · p. 17, Table 5 · p. 17

A.4 LANGUAGE SAMPLES

• The language-sample tables list sequences ancestrally sampled from GPT2-medium artificial languages at softmax temperatures 0.70, 0.85, 1.00, and 1.15.Source: Table 6 · p. 17, Table 7 · p. 18, Table 8 · p. 18