PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Linguistic Diversity of Large Language Models

Yanzhu Guo, Guokan Shang, Chloé Clavel

arXiv:2412.10271v2cs.CL

TL;DR

LLM evaluation has often prioritized task performance over whether generated language reflects human linguistic diversity. This paper proposes a framework spanning lexical, syntactic, and semantic diversity and benchmarks models and influencing factors. It reports a gap in reproducing human linguistic richness for creative tasks, alongside differing diversity patterns across aspects and quality.

  • Problem

    LLM evaluation has paid relatively little attention to linguistic diversity, despite natural language generation's aim to produce outputs that are accurate and diverse.

    Source: §1 · p. 1

  • Method

    The paper proposes a corpus-level framework for lexical, syntactic, and semantic diversity, benchmarks six LLMs across five NLG tasks, and examines development and deployment factors.

    Source: §1 · p. 2

  • Results

    The authors report a significant gap in replicating human linguistic richness for creative tasks such as story generation, and find that diversity factors influence the three diversity dimensions.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Because quality and diversity relationships vary by task and aspect, the authors recommend reporting diversity metrics alongside quality metrics for comprehensive evaluation.

    Source: §5.1 · p. 7

    The study examines linguistic diversity in English, though its methodology could extend to other languages when suitable NLP tools exist.

    Source: §1 · p. 2

Abstract

from arXiv · show

The development and evaluation of Large Language Models (LLMs) has primarily focused on their task-solving capabilities, with recent models even surpassing human performance in some areas. However, this focus often neglects whether machine-generated language matches the human level of diversity, in terms of vocabulary choice, syntactic construction, and expression of meaning, raising questions about whether the fundamentals of language generation have been fully addressed. This paper emphasizes the importance of examining the preservation of human linguistic richness by language models, given the concerning surge in online content produced or aided by LLMs. We propose a comprehensive framework for evaluating LLMs from various linguistic diversity perspectives including lexical, syntactic, and semantic dimensions. Using this framework, we benchmark several state-of-the-art LLMs across all diversity dimensions, and conduct an in-depth case study for syntactic diversity. Finally, we analyze how different development and deployment choices impact the linguistic diversity of LLM outputs.

1 Introduction

LLM evaluation has largely prioritized task abilities, while linguistic diversity has received comparatively little attention despite its importance to natural language generation. This paper introduces a framework and benchmarks diversity across models, tasks, and development and deployment choices.

1 Introduction

  • LLM evaluations predominantly focus on task abilities, while linguistic diversity remains less studied despite generation's aim to produce outputs that are accurate and diverse.Prior work also reports that models can struggle to reflect variation in human expression and that synthetic-text training can further reduce diversity.Source: §1 · p. 1
  • The paper addresses the lack of comprehensive diversity evaluation: prior work often examines one aspect or task, while state-of-the-art LLMs had not been benchmarked across aspects and domains.The effects of choices such as model scale, quantization, decoding, prompting, and successive pretraining checkpoints were also little understood.Source: §1 · pp. 1–2
  • The authors establish a corpus-level framework, benchmark six LLMs on five NLG tasks across lexical, syntactic, and semantic diversity, and analyze development and deployment factors.They give particular attention to story generation and compare human and model dependency-tree distributions in a syntactic-diversity case study.Source: §1 · p. 2
  • The study focuses on English, but its evaluation methodology could extend to other languages when suitable NLP tools, such as parsers and sentence-embedding models, exist.The factor-analysis approach could also be applied to dimensions such as linguistic naturalness.Source: §1 · p. 2

2 Related Work

Prior work developed human-language measures and generated-text diversity metrics, while studies of LLMs and human writing examined how model-generated content relates to diversity. This paper situates its broader analysis within those lines of research.

2 Related Work

  • Linguistic diversity is framed as natural variation in vocabulary, grammatical structures, and semantic nuances, distinct from socio-linguistic diversity, which is outside this study's scope.Earlier human-language metrics often targeted lexical diversity or required costly manual annotation, limiting scalability.Source: §2 · p. 2, §2.1 · p. 2

2.1 Evaluation of Human Language

  • Human-language diversity metrics were developed for language acquisition, disorder detection, and essay-quality research, but many were limited to human text or narrow, manually annotated features.Interest in model-generated text has grown alongside its online prevalence, though diversity assessment is meaningful only when outputs meet basic quality standards.Source: §2.1 · pp. 2–3

2.2 Evaluation of Generated Language

  • Generated-language work introduced pairwise-similarity diversity metrics and later semantic and syntactic measures, while divergence metrics compare human and machine text distributions without directly measuring diversity.Tevet and Berant (2021) noted a disparity between automatic metrics and human judgment, and difficulty of human evaluation for sets larger than ten sentences.Source: §2.2 · p. 3

2.3 Impact of LLMs on Linguistic Diversity

  • Studies report diversity declines in synthetic-data training and human writing with LLMs; this paper broadens factor analysis and finds preference-tuning effects differ from Kirk et al. (2024), likely across task domains.The comparison underscores the importance of contextualizing conclusions about tuning and diversity.Source: §2.3 · p. 3

3 Metrics for Linguistic Diversity

The study evaluates lexical, syntactic, and semantic diversity as distinct dimensions, using across-input comparisons to characterize broad patterns in generated language. Its metrics quantify vocabulary variety, dependency-tree structure, and semantic embedding dispersion.

3 Metrics for Linguistic Diversity

  • The framework distinguishes lexical and syntactic form diversity from semantic content diversity, and measures across-input variation to capture general patterns across generations.The protocol could be extended to per-input diversity, but that analysis is beyond the study's scope.Source: §3 · pp. 3–4

3.1 Lexical Diversity

  • Lexical diversity uses global Unique-n, the ratio of unique n-grams to all n-grams, averaged over unigrams, bigrams, and trigrams.The method samples 40K n-grams for each n to mitigate output-length effects; global measurement targets repetitions across inputs.Source: §3.1 · p. 4

3.1.1 Syntactic Diversity

  • Syntactic diversity is measured by average pairwise distance between dependency trees parsed from sentences and compared with the Weisfeiler-Lehman graph kernel.Tree nodes represent words and are labeled with part-of-speech tags; edges represent syntactic dependencies.Source: §3.1.1 · p. 4

3.1.2 Semantic Diversity

  • Semantic diversity is measured as the average pairwise cosine distance among Sentence-BERT sentence embeddings, scaled to [0, 1].The paper notes that semantic diversity should be assessed only after text meets a basic quality standard, since noisy or irrelevant content can also yield high diversity.Source: §3.1.2 · p. 5

4 Settings for Diversity Benchmarking

The benchmark uses five NLG tasks spanning increasing creativity levels and compares six model families selected near 7 billion parameters. It also specifies sampling, decoding, and dataset choices that bound how to interpret the comparisons.

4 Settings for Diversity Benchmarking

  • Outputs for all tasks and models use nucleus sampling (t=0.6) combined with topk sampling (k=0.9).The study separately analyzes how decoding parameters affect diversity.Source: §4 · p. 5

4.1 Generation Tasks

  • Five tasks span increasing creativity levels, with minimal instructions intended to avoid overly influencing responses through prompt design.The tasks are language modeling, machine translation, summarization, next utterance generation, and automatic story generation.Source: §4.1 · p. 5
  • Each task uses 10K randomly selected samples from one representative dataset, so conclusions should be interpreted within those datasets.Source: §4.1 · p. 5
  • Language modeling uses Wikitext-2 and is classified as least creative; French-to-English WMT-14 translation and English XLSUM summarization are low-creativity tasks.Summarization allows slightly more flexibility than translation because the model chooses which information to prioritize.Source: §4.1 · pp. 5–6
  • DailyDialog next-utterance generation is creative but constrained by everyday conversation, while WritingPrompts story generation is the most creative task.Stories allow maximal expressive freedom because prompts impose few constraints on narrative structure and content.Source: §4.1 · p. 6

4.2 Language Models

  • The benchmark covers Llama, Mistral, OLMo, Gemma, Qwen, and Falcon, selecting each family’s latest version closest to 7 billion parameters.Language modeling uses base models; the other tasks use instruction-tuned versions.Source: §4.2 · p. 6

5 Results of Diversity Benchmarking

LLMs’ diversity relative to human outputs depends on task: they lag across all three dimensions in story generation but exceed human references on some next-utterance dimensions. Diversity metrics also relate inconsistently to one another and to quality, while the syntactic case study finds limited coverage of human structures.

5 Results of Diversity Benchmarking

  • French-input translation uses multilingual SentenceBERT for semantic diversity and a French-specific parser for syntax, so those scores may not be directly comparable with English scores.Source: §5 · p. 7

5.1 Correlation Study

  • Quality correlates positively with lexical and semantic diversity but often negatively with syntactic diversity, so their relationships vary across tasks and aspects.The study uses task-specific automatic quality metrics; it cautions that these measures can correlate weakly with human judgments.Source: §5.1 · p. 7, Figure 2 · p. 7
  • Syntactic and semantic diversity correlate moderately (0.55), while lexical diversity correlates weakly with syntax (0.13) and negatively with semantics (-0.14).Source: §5.1 · p. 7, Figure 3 · p. 7

5.3 Comparing Syntactic Diversity Between Humans and Models

  • For story generation, all models show near-perfect syntactic precision but substantially lower recall, indicating limited coverage of human syntactic diversity.Precision measures generated trees within the human support; recall measures human trees within the model-generated distribution.Source: Table 2 · p. 8, §5.3 · p. 9

5.2 Comparison Across Tasks and Models

  • Human diversity varies by task: dialogue has the lowest lexical and highest syntactic diversity, summaries combine limited form variation with high semantic diversity, and stories vary across all three dimensions.Source: §5.2 · p. 8
  • In story generation, LLMs lag humans in lexical, syntactic, and semantic diversity, but in next-utterance generation they exceed human references in lexical and semantic diversity.Source: §5.2 · p. 8
  • Models pretrained on fewer tokens show lower lexical diversity, while Qwen2.5 is more diverse in story generation than the extensively filtered Llama-3.1, Gemma-2, and OLMo.The lower lexical-diversity pattern is not observed for syntactic or semantic diversity; the authors describe the relation between data filtration and diversity as a comparison.Source: §5.2 · p. 8

5.3 Comparing Syntactic Diversity Between Humans and Models

  • The syntactic-pattern comparison identifies human-frequent patterns missing from model outputs and patterns models over-generate; the authors suggest pretraining–task mismatch as a possible source.For OLMo, over 80% of pretraining data comes from Common Crawl and less than 0.3% from Project Gutenberg, a source potentially aligned with narrative style.Source: Table 3 · p. 9, §5.3 · pp. 9–10

6 Factors Influencing LLM Diversity

This section tests how training stages and design or deployment choices relate to LLM diversity, using story generation because creativity is central; its conclusions are specific to that task.

6.1 Impact of Training Stages

  • During pretraining, lexical diversity rises after an early drop, while syntactic diversity quickly stabilizes and semantic diversity rises before saturating.The authors suggest increasing training data generally improves lexical diversity, while syntactic and semantic diversity may require alternative strategies.Source: §6.1 · p. 11
  • Across models, instruction tuning raises lexical but lowers syntactic and semantic diversity versus base models, with larger syntactic declines; OLMo SFT has minimal impact, while DPO shifts syntax down and lexical diversity up.The OLMo DPO changes may reflect characteristics of the SFT and DPO datasets.Source: §6.1 · p. 11

6.2 Impact of Decoding Parameters

  • Higher temperature increases lexical diversity with minor relevance loss; syntactic effects vary by model, and semantic diversity shows no consistent trend.Most models retain their relative diversity rankings across temperatures; the authors use 0.6 for other experiments, where relevance remains relatively high and diversity improves over lower temperatures.Source: §6.2 · p. 11

6.3 Impact of Prompt Formulation

  • Prompting models to be creative has minimal impact on diversity across all three evaluated aspects, making prompt engineering alone a challenging route to greater diversity.The comparison uses a standard story-continuation instruction and a version explicitly asking for creativity.Source: §6.3 · p. 12

6.4 Impact of Model Scale and Quantization

  • In Qwen2.5 models up to 32B parameters, lexical diversity increases with scale, semantic diversity stays stable, and syntactic diversity peaks at 7B before declining.The scale study is limited to models up to 32B because of computational resource constraints.Source: §6.4 · p. 12
  • 4-bit quantization leaves semantic diversity unchanged but reduces lexical and syntactic diversity, with stronger lexical effects in smaller models and syntactic effects in larger ones.The original models were run with bf16.Source: §6.4 · p. 12

7 Conclusion

The conclusion argues that LLMs still fall short of human linguistic richness in creative generation and that diversity varies with model development and deployment choices.

7 Conclusion

  • Despite coherent and plausible generation, current LLMs show a significant gap from human linguistic richness on creative tasks such as story generation.The study evaluates lexical, syntactic, and semantic diversity.Source: §7 · p. 12
  • Instruction tuning improves lexical diversity while constraining syntactic and semantic diversity, suggesting narrower expressive flexibility.The conclusion identifies pretraining data volume, instruction tuning, decoding strategies, model scale, and quantization as factors that influence diversity metrics.Source: §7 · p. 12
  • As LLM-assisted content creation becomes more prevalent, outputs may trend toward homogenization and risk losing linguistic richness.Source: §7 · p. 12

A Comparison of Benchmarked LLMs

Table 4 is a comparison of the LLMs included in the benchmark.

A Comparison of Benchmarked LLMs

B Detailed Results of Linguistic Diversity Benchmarking

The appendix presents the detailed linguistic-diversity benchmark results and identifies the highest model scores by task and diversity type.

B Detailed Results of Linguistic Diversity Benchmarking

Loading 2412.10271v2…