PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Evaluation Beyond Perplexity

Clara Meister, Ryan Cotterell

arXiv:2106.00085v3cs.CL

TL;DR

Perplexity alone cannot show which macro-level statistical tendencies of human language models reproduce. The paper tests generated text against these tendencies and finds that fit varies by architecture and generation strategy, with models capturing only a subset of natural-language distributions.

  • Problem

    Perplexity alone may not reveal whether language models capture macro-level statistical tendencies of human language.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates generated text against empirical language trends using an experimental design and accompanying hypothesis tests.

    Source: §1 · pp. 1–2

  • Results

    Neural language models capture only a subset of natural-language distributions, with fit depending on architecture and generation strategy; transformers with nucleus sampling match the type–token trend most closely.

    Source: §5.2 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    The framework offers a more fine-grained evaluation than perplexity alone and can help reveal which linguistic properties models have learned.

    Source: §7 · p. 9

    The results are limited to a single English-language corpus.

    Source: §7 · p. 9

Abstract

from arXiv · show

We propose an alternate approach to quantifying how well language models learn natural language: we ask how well they match the statistical tendencies of natural language. To answer this question, we analyze whether text generated from language models exhibits the statistical tendencies present in the human-generated text on which they were trained. We provide a framework--paired with significance tests--for evaluating the fit of language models to these trends. We find that neural language models appear to learn only a subset of the tendencies considered, but align much more closely with empirical trends than proposed theoretical distributions (when present). Further, the fit to different distributions is highly-dependent on both model architecture and generation strategy. As concrete examples, text generated under the nucleus sampling scheme adheres more closely to the type--token relationship of natural language than text produced using standard ancestral sampling; text from LSTMs reflects the natural language distributions over length, stopwords, and symbols surprisingly well.

1 Introduction

The paper evaluates language models by asking whether their generated text reflects macro-level statistical tendencies in human language, beyond what perplexity alone reveals. Its framework uses empirical trends as a baseline and finds that fit varies by model architecture and generation strategy.

1 Introduction

  • Empirical tendencies provide a baseline when no theoretical law is established or a proposed law may not fit the language domain.This supports analysis of corpus-dependent distributions such as document length and reduces bias from assumptions about language.Source: §1 · pp. 1–2
  • The paper shifts attention from sentence-level linguistic phenomena to macro-level patterns across a corpus, such as token rank–frequency.The authors ask whether neural language models exhibit human language’s statistical tendencies.Source: §1 · p. 1
  • The framework tests whether generated text follows empirical human-language trends, offering a view of model behavior beyond perplexity.It pairs an experimental design with hypothesis tests to assess fit.Source: §1 · p. 2
  • Fit to natural-language tendencies varies with both model architecture and generation strategy.Figure 1 compares average unique words by document length in sampled text with model test-set values.Source: Figure 1 · p. 1, §1 · p. 2

2 Language Models

The section defines language models as probability distributions over sentences and describes how their support and probabilities are represented. It also explains why held-out perplexity, though a measure of fit, does not show which linguistic attributes a model has learned.

2 Language Models

  • A language model is a probability distribution over natural-language sentences, with its support formed from vocabulary strings bounded by beginning and end tokens.The paper calls word vocabularies closed and BPE-token vocabularies open.Source: §2 · p. 2
  • For locally normalized models, a document’s probability is the product of its successive conditional token probabilities.The section also describes parameter estimation by optimizing log-likelihood over a training corpus.Source: §2 · p. 2
  • Held-out perplexity reflects some fit to natural language but does not reveal which linguistic attributes a model has learned.It is a monotonic function of average per-token log-likelihood on held-out data.Source: §2 · p. 2

3 Statistical Tendencies of Language

The section reviews established statistical laws and other corpus-level distributions that can be compared with language-model output. It describes both theoretical forms and empirical ways to characterize tendencies that may depend on the corpus.

3.1 Classical Laws

  • Zipf’s rank–frequency law models word frequency as a power-law decay with rank, and its exponent is estimated by maximum likelihood.The exponent is typically close to 1 in natural-language text; the discrete power-law MLE is solved numerically.Source: §3.1 · p. 2
  • Heaps’ law models the expected number of unique types as a sublinear power of document length, and the paper represents it with a non-homogeneous Poisson process.The rate is α · l(y)^β, with parameters α and β estimated by maximum likelihood.Source: §3.1 · pp. 2–3

3.2 Other Tendencies

  • For corpus-dependent tendencies without established laws, the paper uses empirical distributions, including document length and unigram frequencies.It also computes summary statistics such as the sample mean of document lengths.Source: §3.2 · p. 3
  • The section also characterizes the proportions of stopwords and symbols in strings as probability distributions.Symbols include numbers and punctuation; stopwords include common words serving primarily syntactic functions.Source: §3.2 · p. 3

4 Statistical Distances

This section uses probability-distance metrics and nonparametric tests to assess whether generated text matches natural-language distributions. It moves from expectation-based comparisons to metrics that compare distributions more fully.

4 Statistical Distances

  • The framework measures distributional fit with probability metrics and pairs each metric with a nonparametric significance test.Some metrics are formally pseudo-distances because they need not be symmetric.Source: §4 · p. 3, §4.1 · p. 3

4.1 Primary Metrics

  • Primary metrics estimate differences in expectations, such as means or variances, and test whether the observed difference is significantly different from zero.Because the underlying distributions are typically unknown, the framework uses resampling, including permutation tests, to construct the test-statistic sampling distribution.Source: §4.1 · pp. 3–4

4.2 Simple Metrics

  • Because primary metrics depend on a single statistic, the section turns to distribution-based metrics: KS for cumulative distributions and TVD for categorical distributions.The paper favors TVD over Chi-square distance for categorical distributions because Chi-square values can be hard to interpret.Source: §4.2 · p. 4
  • KS and TVD support one-sample tests against a theoretical distribution and two-sample tests of whether empirical samples share a distribution.In the two-sample case, the shared distribution need not be known.Source: §4.2 · p. 4
  • The KS test uses the maximum CDF difference and rejects its null when that statistic exceeds a sample-size- and significance-level-dependent critical value.Under the null, the statistic converges to zero as the sample size grows.Source: §4.2 · p. 4
  • For TVD, the paper constructs the null sampling distribution by permuting the observed samples because no universal theoretical limiting distribution is available.This resampling procedure applies when both samples are drawn from the same, potentially unknown distribution.Source: §4.2 · pp. 4–5

5 Experiments

Experiments compare generated samples with held-out corpus text across architectures and decoding schemes, finding that distributional fit depends on both and varies by linguistic distribution.

5 Experiments

  • On English Wikipedia, transformer, LSTM, and CNN samples from three decoding schemes are compared with a reserved test set; million-sample tests are expected to reject nearly always.The experiments also include a trigram model.Source: §5 · pp. 5–6

5.1 Rank–Frequency

  • Models poorly match standard Zipf, as does the test set (Dˆp = 0.148), yet track empirical rank–frequency trends well, warning against Zipf-only evaluation.The authors consider only the first 10,000 ranks because power-law behavior falls off at higher ranks.Source: §5.1 · pp. 6–7

5.2 Type–Token

  • Transformer samples, especially under nucleus sampling, match the empirical type–token trend most closely, but all models diverge more as documents lengthen.Both transformer models also show a visible fall-off for longer sequences.Source: §5.2 · p. 7

5.3 Unigram Distribution

  • Models generally capture empirical unigram distributions well, but closed-vocabulary Transformer AS consistently performs worse; BPE-like vocabularies may fit better.The authors note that TVD is generally robust to tail probabilities.Source: §5.3 · p. 8

5.4 Length, Stopwords and Symbols

  • Random- and nucleus-sampled models generally mimic length, stopword, and symbol distributions; LSTM random sampling is closest across all three, while beam sampling diverges strongly.The length results also suggest models fit the corpus distribution closely, contrary to claims that local normalization favors shorter sequences.Source: §5.4 · p. 8

5.5 Consistent Trends

  • Nucleus sampling often aligns more closely with natural language than other strategies, while empirical and parametric reference distributions can yield substantially different results.The authors caution that assumed distributional forms can significantly affect findings.Source: §5.5 · p. 8

6 Related Work

Research beyond perplexity uses task-based and model-based probes, but this work is closest to studies that visually inspect statistical tendencies in generated text. It adds quantitative distribution-fit measures and significance tests to that line of work.

6 Related Work

  • Task-based probes test linguistic knowledge through designed tasks, while model-based probes train auxiliary models on representations from the model under study.The section notes that auxiliary models can introduce confounding factors, and auxiliary-task design requires substantial manual effort and can encode implicit assumptions about linguistic phenomena.Source: §6 · p. 9
  • Takahashi and Tanaka-Ishii (2017, 2019) visually analyze whether generated text reflects statistical tendencies; this work adds quantitative fit measures and significance tests.It also compares models directly with a test set, and examines multiple architectures, generation strategies, and empirical distributions.Source: §6 · p. 9
  • Dror et al. (2018) likewise present statistical tests for NLP, a point of similarity with this work.The section identifies the work of Takahashi and Tanaka-Ishii (2017, 2019) as the closest prior work.Source: §6 · p. 9

7 Conclusion and Future Directions

The paper presents statistical-trend analysis as a finer-grained complement to perplexity and finds that models capture only a subset of natural-language distributions, with fit depending on architecture and generation strategy. Its results are limited to one English corpus, leaving cross-linguistic and corpus-size analyses for future work.

7 Conclusion and Future Directions

  • Neural language models capture only a subset of natural-language distributions, and no configuration captures them all.The subset captured depends strongly on model architecture and generation strategy.Source: §7 · p. 9
  • The analysis is limited to a single English corpus; cross-linguistic comparisons and analyses across corpus sizes remain future work.The paper also identifies correlations with other performance measures, such as perplexity or sentence-level phenomena, as future work.Source: §7 · p. 9

A Maximum Likelihood Estimates

The appendix gives likelihood formulations for observed corpus statistics and pseudocode for an unpaired two-tailed permutation test. It defines corpus word-count quantities used in the likelihood expressions.

A Maximum Likelihood Estimates

  • The appendix sets up a log-likelihood for observed rank–frequency data under the power-law paradigm and defines corpus word-count quantities used in likelihood calculations.The text defines c(w, C) as the count of occurrences of w in C and |C|_w as the total word count of C.Source: A Maximum Likelihood Estimates · p. 13
  • It also introduces the log-likelihood of observed unique versus total tokens under Eq. (5).The supplied passage does not state the full likelihood expression.Source: A Maximum Likelihood Estimates · p. 13

B Permutation Test Pseudocode

  • The unpaired two-tailed permutation-test pseudocode takes a function φ and two samples, S1 and S2, as inputs.The algorithm description identifies P([k]) as the power set of the integers 1 through k.Source: B Permutation Test Pseudocode · p. 13

C Chi-square Goodness-of-fit Test

The section describes the chi-square test as a way to assess whether observed and expected category frequencies differ. It cautions that large samples can make small differences significant and that the statistic has no upper bound.

C Chi-square Goodness-of-fit Test

  • A large chi-square value suggests that two samples do not come from the same distribution, based on differences between observed and expected category frequencies.The test uses a function of those frequency differences across one or more categories.Source: C Chi-square Goodness-of-fit Test · p. 13
  • At sample sizes of 500 or more, even small differences may appear statistically significant, and the chi-square statistic is not bounded above.These properties make its magnitude difficult to interpret and its significance sensitive to sample size.Source: C Chi-square Goodness-of-fit Test · p. 13
  • The accompanying test-set statistics are for one million Wikipedia Dumps samples, with standard deviations estimated from random training-set samples of the same size.The caption identifies the purple numbers as the standard deviations.Source: Table 4 · p. 14

D Additional Results

Additional results compare rank–frequency patterns and Zipf’s and Heap’s coefficients across generated samples, and summarize length, stopword, and symbol distributions against the test set.

D Additional Results

  • Rank–frequency distributions for the different samples follow a remarkably similar trend.Figure 6 presents this comparison.Source: Figure 6 · p. 14
  • The coefficient table compares Zipf’s and Heap’s coefficients across models and generation schemes, with test-set references s = 1.120 and β = 0.841.The caption gives the reference values but no model-specific coefficients.Source: Table 5 · p. 16
  • The length, stopword, and symbol comparisons include mean differences from the test set and permutation distributions for those differences.The test-set distributions are repeated in the boxplots for reference; the significance key marks p-values below 0.005 and 0.05.Source: Figure 7 · p. 14, Figure 8 · p. 15, Table 6 · p. 16

E Sample Generated Text

This section presents generated-text examples; the table caption specifies that the first two samples from each set are shown.

E Sample Generated Text

  • Some samples combine unrelated place descriptions, biographical details, and other claims within the same passage.Examples include the Moton aqueduct and Victoria, and separate material about Hawley and Barnard College.Source: E Sample Generated Text · p. 16
  • Other displayed outputs are short topic fragments or brief statements, including “short track speed skating” and “socialist youth union.”The examples also include a species description and a statement about aesthetic significance.Source: E Sample Generated Text · pp. 16–17
  • Table 7 presents the first two generated-text samples from each set.The examples include snippets about people, places, and other topics.Source: Table 7 · p. 17
Loading 2106.00085v3…