PaperlayerView exact revision on arXiv

Source-linked AI summary

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, Asli Celikyilmaz

arXiv:2111.09509v1cs.CL

TL;DR

The paper asks whether language models generalize linguistic structure or mainly copy training data. It introduces RAVEN to measure novelty in sequential and syntactic structure, finding productive generation alongside reduced local novelty and occasional long-range copying.

  • Problem

    Language models can generate high-quality text, but text quality alone does not reveal whether outputs were constructed by the model or copied from training data.

  • Method

    RAVEN evaluates generated-text novelty in n-gram and syntactic structure across LSTM, Transformer, Transformer-XL, and GPT-2 models against human test-set baselines.

  • Results

    Models produce novel structures but are less novel than human baselines for local structure, more novel for n-grams larger than 5-grams, and sometimes copy passages over 1,000 words long.

  • Takeaways & Limitations

    Novelty evaluation reveals both productive linguistic recombination and copying behavior that quality-based NLG evaluation can miss.

  • Takeaways & Limitations

    The syntactic-novelty numerical results are reported for only 4 of 7 analyzed attributes because parser outputs were not accurate enough for the others.

Abstract

from arXiv · show

Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure - e.g., individual dependencies - model-generated text is substantially less novel than our baseline of human-generated text from each model's test set. For larger-scale structure - e.g., overall sentence structure - model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis showing that GPT-2's novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).

1 Introduction

The paper asks whether language models generate text through generalizable linguistic structure or by copying training data. RAVEN evaluates novelty across sequential and syntactic structure, finding both productive generation and substantial copying.

  • Approach: RAVEN evaluates generated-text novelty through sequential structure, including n-grams, and syntactic structure across several language-model architectures.The study includes an LSTM, Transformer, Transformer-XL, and four GPT-2 sizes, with multiple generation methods and prompt lengths.
  • Findings: Human-generated baseline text contains 1.4 to 3.3 times as many novel bigrams as model-generated text, showing reduced novelty for local structure.This comparison uses human-generated text from each model’s test set as the baseline.
  • Findings: For n-grams larger than 5-grams, models are more novel than the baseline but sometimes duplicate training passages exceeding 1,000 words.The results therefore combine large-scale structural novelty with occasional extensive copying.

2 Background

The paper distinguishes novelty from diversity and argues that open-ended NLG evaluation should assess both generated-text quality and whether outputs duplicate training data. Its analyses focus on surface-form novelty rather than meaning.

  • Evaluation: Quality scores alone can overstate model abilities when a copied passage is fluent, coherent, or factually accurate.A copied paragraph may receive a high coherence score because copying reproduces coherent text without demonstrating learned coherence.
  • Evaluation: The paper argues that open-ended NLG evaluation should emphasize novelty alongside generated-text quality.Without novelty, quality may reveal copying rather than the targeted capability.
  • Novelty and diversity: Novelty concerns how generated text differs from the training set, whereas diversity concerns how generated text differs from other generated text.A model can be diverse but non-novel by copying varied training sentences, or novel but non-diverse by repeating one new sentence.
  • Scope: The main analyses test novelty of surface text and formal abstractions such as syntax, not language understanding or meaning.This defines an explicit scope boundary for the evaluation.

3 Motivation and approach

The paper motivates novelty as evidence of generalizable linguistic ability and defines a comparison against in-distribution human text. RAVEN marks structures as duplicated when they occur in training data or the prior context.

  • Motivation: Novel text is treated as necessary evidence when investigating whether NLG models possess generalizable linguistic abilities.The paper connects novelty to handling unfamiliar situations, while noting that novelty may not matter for every practical use case.
  • Approach: RAVEN evaluates novelty by generating many samples and checking n-gram and syntactic structures against prior text.A structure is duplicated if it appears in the training set or the concatenated prompt and already-generated context; otherwise it is novel.
  • Baseline: Human-generated text from each model’s in-distribution test set provides a baseline for expected duplication within the training domain.The paper treats model novelty at least as high as this baseline as evidence against excessive copying.

4 Experimental details

The experiments compare language-model architectures under controlled and larger-scale settings, using distinct corpora and tokenization schemes. Generation uses test-set prompts and top-40 sampling to balance novelty with text quality.

  • Models: LSTM, Transformer, and Transformer-XL are compared on the same 103-million-word Wikitext-103 training set.The comparison holds the dataset constant while contrasting recurrence, self-attention, and their combination.
  • Models: GPT-2 is evaluated as a larger-scale Transformer trained on the 7.7-billion-word WebText corpus with subword tokenization.Unlike the other models, GPT-2 uses WebText and subword rather than word-level tokenization.
  • Prompts: Models generate continuations from prompts drawn from their own test sets, using 1,000 prompts per setting.Wikitext-103 prompts contain 512 words followed by 1,000 generated words; WebText prompts contain 564 subword tokens followed by 1,100 generated tokens.
  • Decoding: Top-40 sampling selects the next token from the 40 highest-probability candidates to support high-quality generation during novelty evaluation.The same decoding scheme is used for the LSTM and Transformer, with other decoding methods analyzed separately.

5 N-gram novelty

N-gram novelty depends on scale: models are conservative for small n-grams but generally novel at larger scales, while decoding exposes a quality–novelty tradeoff and occasional extreme copying.

  • 5.1 How often are generated n-grams novel for various values of n?: For all models, the majority of n-grams of size 5 or larger are novel, and n-grams larger than 6 are almost always novel.
  • 5.1 How often are generated n-grams novel for various values of n?: For n < 6, models are less novel than test-set baselines, while for n > 6 they are more novel.On Wikitext-103, bigram novelty is 2%–3% for models versus 6% for the baseline, and trigram novelty is 17%–22% versus 31%.
  • 5.1 How often are generated n-grams novel for various values of n?: The LSTM is least novel for small n-grams, the Transformer is most novel, and Transformer-XL falls between them.The authors conjecture that recurrence increases recency bias, whereas self-attention distributes conditioning across recent and distant tokens.
  • 5.2 Do models ever duplicate large n-grams?: All models occasionally duplicate passages at least 100 words long, including GPT-2 passages exceeding 1,000 words.The paper calls duplication of an n-gram of size 100 or larger supercopying.
  • 5.3 How is novelty related to the decoding scheme and the generated text’s quality?: Increasing top-k, top-p, or temperature increases novelty, but every such modification decreases text quality.Quality is assessed using perplexity as a proxy, with lower perplexity interpreted as higher quality.
  • 5.3 How is novelty related to the decoding scheme and the generated text’s quality?: GPT-2 can match the baseline quality–novelty tradeoff under top-p decoding with p = 0.95.This setting achieves perplexity 93.7 versus baseline 89.4 and truncated pointwise duplication score 4.41 versus baseline 4.47.

6 Syntactic novelty

Syntactic novelty is high for global sentence structure but lower for local dependencies relative to human-generated test-set baselines. Parser imperfections constrain which syntactic attributes can support numerical conclusions.

  • 6 Syntactic novelty: The analysis evaluates novelty across seven syntactic aspects using constituency and dependency parses.
  • 6 Syntactic novelty: Manual checks found parsers accurate enough for numerical reporting on only four of the seven analyzed attributes.Novel cases are especially likely to confuse imperfect parsers, so the authors manually estimated parser error rates.
  • 6 Syntactic novelty: Most generated sentences have POS sequences and parse structures unseen in training, with little difference from the baselines.
  • 6 Syntactic novelty: For dependency arcs and dependency relations, human-generated baselines are far more novel than the models.This local-structure pattern resembles the n-gram results.

7 Analysis

GPT-2’s novel words are usually morphologically well-formed and syntactically appropriate, with evidence of productive composition and analogy, but semantic suitability is weaker.

  • Morphological well-formedness: 96% of GPT-2’s novel words are well-formed, compared with 99% in the baseline.
  • Morphological well-formedness: GPT-2 correctly forms 72 of 74 novel plurals and 135 of 136 novel possessives.
  • Morphological well-formedness: GPT-2’s novel acronyms are suitable in 28% of cases, below the baseline’s 81%.
  • Syntactic usage: 94% of GPT-2’s novel words occur in grammatically correct contexts, although errors remain more frequent than in the baseline.
  • Syntactic usage: GPT-2 maintains agreement for novel plural words, including across intervening singular distractors.
  • Contextual and semantic usage: GPT-2 appears to learn incrementing, ordering, and quotation-mark associations for novel words, while semantic errors include numerical inconsistencies and self-contradictory meanings.
  • Generalization: The examples suggest GPT-2 uses both compositional and analogy-based generalization, although the authors cannot always distinguish which process produced a word.

8 Discussion

RAVEN finds that language models combine familiar linguistic parts into novel larger-scale structures while remaining less novel locally and occasionally copying long passages. The authors recommend evaluating novelty against human test-set baselines and note that training-set scale complicates assumptions about what is novel.

  • Findings: Models generate novel n-grams, morphological combinations, and syntactic structures, but their local structure is less novel than the human baseline.This pattern supports productive combination without implying uniformly human-like generalization.
  • Findings: All models show compositional generalization, but their scores are lower than the baseline and GPT-2 sometimes generalizes too freely.The authors give “752th” as an example of overgeneralization and argue that models must learn rules alongside exceptions.
  • Evaluation: Novelty should be evaluated against in-distribution human test-set text because matching or exceeding that baseline helps rule out excessive copying.The recommendation targets open-ended generation, where quality alone cannot establish that outputs are constructed rather than copied.
  • Evaluation: Large training sets make it unsafe to assume that forms absent from ordinary experience are absent from the training data.GPT-2’s training set contains incorrect regular forms for all 92 basic English irregular verbs.
  • Improving novelty: Sampling penalties, training-data deduplication, and compositional mechanisms are proposed as routes to reduce copying while preserving deeper novelty.The authors specifically connect supercopying to repetition in the training set and cite evidence that deduplication can reduce copying.

9 Conclusion

The paper concludes that RAVEN reveals a mixed picture: models are often novel for larger-scale structure but rarely novel locally, while occasionally copying very long passages. It also documents the evaluation setup and supporting analyses used to establish these conclusions.

  • Conclusion: RAVEN covers sequential and syntactic structure and compares generated text with human-generated text from each model’s test set.The experiments include LSTM, Transformer, Transformer-XL, four GPT-2 sizes, 12 generation methods, and four prompt lengths.
  • Conclusion: Models are rarely novel for local structure but often novel for larger-scale structure, while occasionally copying passages over 1,000 words long.Supercopying is operationalized elsewhere as duplication of n-grams of at least 100 tokens.
  • Baseline: The baseline uses text following prompts in the test set, with overlap constraints differing between Wikitext-103 and WebText.Wikitext-103 continuations can overlap partially, whereas WebText continuations do not overlap because the dataset is larger.
  • Analyses: The analyses include examples of novel bigrams, supercopied passages, truncated and untruncated duplication scores, and decoding-parameter effects.These analyses connect qualitative examples with pointwise duplication measurements.

G.1 Evaluating overlap between Wikitext-103 and WebText

The overlap checks support using cross-dataset models for perplexity evaluation, while the appendix examines how dataset overlap, model size, prompts, decoding, and supercopying relate to novelty. The evidence shows no consistent size effect and highlights extreme copying outliers.

  • Dataset overlap: 0 of 1,000 sampled WebText 20-grams appeared in Wikitext-103, while 12 of 1,000 Wikitext-103 20-grams appeared in WebText.The authors characterize the resulting overlap proportion as very small and retain the cross-dataset perplexity comparison.
  • Model size: GPT-2 XL is the most novel across various n-gram sizes, but GPT-2 Medium is more novel than GPT-2 Large, so model size has no consistent effect.All four GPT-2 sizes use top-40 sampling in this comparison.
  • Supercopying: A supercopied GPT-2 passage occurred 176,424 times in its training set, illustrating the extreme frequency of some copied outliers.Figure 9 compares training-set counts for supercopied 100-grams with random 100-grams.
  • Prompt length: Longer prompts have little overall effect on novelty, with only slight increases in duplication for Transformer and GPT-2.Prompt-length comparisons exclude duplication from the context so longer prompts do not receive a trivial advantage.
  • Decoding: Increasing decoding parameters can increase novelty while the decoding schemes that improve quality can decrease novelty.The figures describe novelty through duplication on the y-axis and quality through lower perplexity on the x-axis.

J Position in generated text

Novelty changes little with position for the baselines, LSTM, and Transformer, but GPT-2 and Transformer-XL show slightly more duplication later in generation. The broader decoding-parameter comparison indicates that parameter increases can reduce duplication.

  • Method: The positional analysis excludes context duplication and uses pointwise duplication scores truncated at 10.This controls for later positions having more generated context available to copy from.
  • Position effects: GPT-2 and Transformer-XL show greater duplication at later output positions, while the baselines, LSTM, and Transformer show little positional effect.The increase is small: the pointwise duplication score rises by about 0.2 from the start to the end.
  • Decoding: Increasing top-k, top-p, or temperature increases novelty, meaning duplication decreases on the figure’s y-axis.The figure treats a higher y-axis value as less novel, so the reported direction is expressed through decreasing duplication.

K Duplication from the training set vs. the context

Models almost never copied content from the context that was not also present in the training set, so duplication from the training set closely tracks duplication from both sources.

  • Models almost never copied content from the context that was not also in the training set.

L Vetting syntax

The authors manually vetted seven syntactic attributes because parser errors are especially problematic for novel cases. They report numerical results only for attributes whose reliability was sufficient.

  • Parser errors are especially concerning because novel cases are particularly likely to confuse parsers.
  • POS tags: Most generated words identified with novel POS tags actually appeared with that POS in training, so those quantitative results were not trusted.
  • CFG rules: Every inspected novel CFG rule resulted from a parser error, so numerical results for CFG rules were not trusted.
  • POS sequences: POS-sequence checks were generally accurate, and the authors judged their quantitative results approximately correct.
  • Dependency argument structure: Dependency argument structure was not manually analyzed, so the authors provide no numerical results for it.

M Example of a mismatch between local and global novelty

Local and global novelty can diverge: sentences may share global novelty while differing in the novelty of their smaller n-grams or dependency arcs.

  • LMs are less novel than the baseline for local structure but more novel for larger-scale structure.
  • Sequential structure: Both generated examples are novel trigrams, but only one also contains novel bigrams.
  • Syntactic structure: Both generated sentences have novel overall parse structures, despite differing in the novelty of their individual dependency arcs.

N Examples of syntactic novelty

Manual analyses provide verified examples of productive syntactic generalization, while also showing important limits and frequent surface-level errors in generated text.

  • Manual candidate screening prioritizes precision: verified examples are certain syntactic novelties, but the method has no recall guarantee.
  • Dependency generalization: Models generated nouns with novel determiners and words used in subject or direct-object roles absent from training.
  • Argument structure: Transformer-XL used suffuses intransitively despite training examples containing only transitive uses.
  • Argument structure: No confirmed novel transitivity or dative alternations were observed in models or baselines.
  • Morphology categorization: GPT-2 novel-unigram analysis was manually performed because automatic methods are unreliable for rare long-tail phenomena.
  • Typographical errors: The generated text contained 754 typographical errors, including 706 caused by spacing or punctuation problems.
  • Novel words: Novel technology-related terms were about twice as frequent in generated text as in the baseline.

O.4 Number

GPT-2 generates novel numerical forms, names, compounds, affixes, and other word structures, often applying productive morphological patterns. However, some novel formations remain inconsistent or ill-formed.

  • Number: 672 novel real numbers were generally well-formed, while 190 novel phone numbers mostly followed North American formatting conventions.GPT-2 also generated 49 possible dates and 75 number-unit combinations.
  • Names: GPT-2 generated 1,499 novel names across usernames, personal names, places, languages, and other categories.The categories included 655 online usernames, 120 place names, and 220 last names.
  • Word formation: Novel words used compounds, inflectional endings, derivational affixes, character manipulation, and possible portmanteaus.Examples include dephrasal and coordinative compounds, diminutive and augmentative affixes, and blends such as Disqus-plus-etiquette.
  • Morphological errors: Some formations showed incorrect stem changes, inconsistent morphology, plural nouns in compounds, or overregularization.The paper notes that plural-first compounds are sometimes acceptable in standard usage, making some cases unclear as errors.

P.5 Syntactic errors

GPT-2’s novel words usually fit their syntactic contexts and sometimes follow productive patterns, but semantic and numerical errors remain frequent. The analyses also show that apparent novelty can reflect training-set exposure.

  • Syntactic errors: Most novel words fit their syntactic context, although GPT-2 sometimes misuses parts of speech, compounds, agreement, or phrase structure.Examples include using bat-washer as a verb, plural nouns in compounds, and mismatched forms such as look anti-Tunisian.
  • Structured generation: GPT-2 can continue structured sequences, including ordinal numbers, variable-name letters, and alphabetical company lists, while producing novel words.The model extended sequences from Firstly to Fourteenthly and from multiplyx to multiplyz.
  • Semantic fit: Novel words can be well-suited to their semantic contexts, but other examples have clearly incorrect meanings.Reported failures include inappropriate derivational meanings and unsuitable uses of names or adjectives.
  • Numerical reasoning: GPT-2 generally mishandles numerical meaning, producing implausible physical quantities and incorrect unit conversions.Examples include 1240 pounds equaling 735 kilograms and 975 milliliters equaling 2.2 gallons.
  • Novelty and training exposure: Only 57% of 1,339 tested words had both singular and plural forms in training, while all had singular forms because the training data listed all three-letter sequences.Therefore, regular-looking novel forms may reflect memorized training material rather than productive overregularization.
Loading 2111.09509v1…