Source-linked AI summary
Contrasting Linguistic Patterns in Human and LLM-Generated News Text
Alberto Muñoz-Ortiz, Carlos Gómez-Rodríguez, David Vilares
TL;DR
The paper asks how closely LLM-generated English news matches human linguistic patterns, an important question given the models’ fluency and their use in text generation. It quantitatively compares human news with outputs from six LLMs across multiple linguistic dimensions. The results identify systematic differences in sentence length, vocabulary, syntax, emotions, and gender bias, with LLMs generally differing more from humans than from one another.
Problem
The paper investigates whether fluent LLM-generated news follows the linguistic patterns of human-generated news across multiple measurable dimensions.
Method
The study compares English New York Times news with outputs from six LLMs spanning three families and four sizes, analyzing lexical, morphosyntactic, psychometric, and sociolinguistic features.
Results
LLM outputs differ from human texts in sentence-length distributions, lexical diversity, syntax, emotions, and gender bias, with all models except Falcon 7B exacerbating human male-pronoun bias.
Takeaways & Limitations
The findings show that highly fluent LLM news retains measurable linguistic differences from human news and reproduces, usually amplifying, existing gender bias.
Takeaways & Limitations
The study is limited to English news and uses prompts based on human lead paragraphs that were written from full articles rather than headlines.
Abstract
from arXiv · showhide
We conduct a quantitative analysis contrasting human-written English news text with comparable large language model (LLM) output from six different LLMs that cover three different families and four sizes in total. Our analysis spans several measurable linguistic dimensions, including morphological, syntactic, psychometric, and sociolinguistic aspects. The results reveal various measurable differences between human and AI-generated texts. Human texts exhibit more scattered sentence length distributions, more variety of vocabulary, a distinct use of dependency and constituent types, shorter constituents, and more optimized dependency distances. Humans tend to exhibit stronger negative emotions (such as fear and disgust) and less joy compared to text generated by LLMs, with the toxicity of these models increasing as their size grows. LLM outputs use more numbers, symbols and auxiliaries (suggesting objective language) than human texts, as well as more pronouns. The sexist bias prevalent in human text is also expressed by LLMs, and even magnified in all of them but one. Differences between LLMs and humans are larger than between LLMs.
1 Introduction
LLMs produce fluent, human-like text, but their linguistic patterns may differ from those found in human-generated news. This study quantitatively compares vocabulary, morphosyntax, semantics, psychometrics, and sociolinguistic features across human and LLM-generated English news.
- The study asks whether LLM storytelling strategies align with linguistic patterns in human-generated texts.The questions concern vocabulary, voices, syntactic structures, and psychometric dimensions.
- Contrasting human and model text is difficult because training data are incompletely documented and linguistic preferences vary across cultures, societies, and geographic regions.
- The analysis compares human news with output from six LLMs spanning three families and four sizes.The models include four LLaMa variants, Falcon 7B, and Mistral 7B.
- The study examines vocabulary, sentence structure, part-of-speech tags, syntactic structures, psychometric features, and gender bias.The authors also investigate whether disparities vary across model sizes and families.
2 Related work
Related work studies the properties, detection, attribution, and uses of machine-generated text. It also highlights that synthetic data can reproduce biases present in source datasets.
- Research examines machine-generated text through linguistic analysis, human-machine discrimination, and LLM-based annotation or data generation.
- Modern detection work includes statistical, syntactic, feature-based, and neural approaches to identifying synthetic text.
- More complex sampling can make generated text harder for humans to recognize but easier for machines to detect.
- Synthetic text has been used to create task-specific data, silver annotations, and training data for smaller language models.
- LLM-generated datasets can reproduce racial, religious, and gender stereotypes present in scraped web data.
3 Data preparation
The study builds a controlled English news dataset by pairing New York Times articles with model-generated continuations prompted by each article’s headline and opening words. It compares six foundation models while limiting the analysis to English and non-instruction-tuned systems.
- 3.1 Data: Articles published after each model’s release date were selected to reduce exposure to their headlines and content during pre-training.
- 3.1 Data: The dataset contains 13,371 New York Times articles collected through the Archive API between October 1, 2023, and January 24, 2024.The study retains headlines and lead paragraphs, discarding articles with empty leads; about 26% include reviews, editorials, obituaries, or other text types.
- 3.1 Data: The analysis focuses on English and excludes instruction-tuned variants to isolate effects associated with pretraining and model size.This choice provides depth but limits how broadly the results can be applied beyond English.
- 3.1.1 Rationale for methodological decisions and technical trade-offs: The prompting comparison is imperfect because human lead paragraphs were written from full articles rather than generated from headlines.The authors identify this as a methodological trade-off and note that the lead remains highly correlated with the headline.
- 3.2 Generation: Each synthetic article is generated from the human article’s headline and first three lead-paragraph words, with generation capped at 200 tokens.The prompt is intended to keep output aligned with the article’s domain and topic.
- 3.2 Generation: The experiments use six pre-trained models from three families and four sizes: LLaMa 7B, 13B, 30B, and 65B, plus Falcon 7B and Mistral 7B.Different sizes are included only for LLaMa because same-family results were considered similar and larger models require more computation.
4 Analysis of linguistic patterns
The analysis compares human- and machine-generated news across morphosyntactic and semantic dimensions. It first examines linguistic form and then turns to meaning-related characteristics.
- The study first analyzes morphosyntactic patterns before examining semantic aspects of human- and machine-generated news.
4.1 Morphosyntactic analysis
The analysis compares human and LLM-generated English news using linguistic representations, vocabulary-diversity measures, and morphosyntactic categories. Human texts show greater sentence-length variation and richer vocabulary, while LLMs differ in their grammatical-category usage.
- Stanza performs segmentation, tokenization, part-of-speech tagging, and dependency and constituent parsing for the linguistic analysis.
- Sentence length: Human news has more varied sentence lengths, whereas LLMs cluster more often between 10 and 30 tokens.Sentences longer than 80 tokens were excluded from the plotted distribution.
- Richness of vocabulary and lexical variation: Humans use richer vocabulary than the studied LLMs, with LLaMa closest to humans, Mistral next, and Falcon lowest in STTR.The results suggest language family matters more than model size for vocabulary richness.
- Part-of-speech tag distributions: UPOS frequencies differ between humans and LLMs: humans favor nouns, adjectives, and punctuation, while LLMs use more symbols, numbers, and pronouns.The authors connect greater symbol and number use with furnishing specific data to sound convincing.
- Richness of vocabulary and lexical variation: MTLD complements STTR by measuring lexical diversity through forward and reverse threshold-based TTR calculations.The default TTR threshold used here is 0.72.
4.1.3 Part‑of‑speech tag distributions
The study finds that human and LLM texts differ in dependency structure and dependency optimality, despite substantial similarity among most LLM outputs. Falcon is a notable syntactic outlier, while human dependencies are more optimized relative to sentence structure.
- Dependency arc lengths: Dependency lengths are nearly identical across LLMs except Falcon, whose longer dependencies more closely resemble human texts.Falcon nevertheless uses more leftward dependencies than humans and the other LLMs.
- Dependency arc lengths: Table 4 reports left- and right-arc proportions, average arc lengths, standard deviations, and sentence counts across sentence-length intervals.
- Optimality of dependencies: Dependency length minimization describes dependencies that are shorter than expected by chance and is commonly linked to working-memory constraints.
- Optimality of dependencies: Human texts have larger Ω values than LLM texts, indicating greater dependency-length optimization after controlling for tree topology.This refines the absolute dependency-length comparison because Ω measures ordering relative to random and optimal word orders.
- Dependency types: LLM texts use syntactic dependency types similarly to one another, whereas all differ from human texts and Falcon is the most distinct model.Larger LLaMa models are slightly closer to human dependency-type usage.
4.1.7 Dependency types
LLM-generated news differs from human news in dependency-type usage, with higher rates for several dependencies associated with numerals, auxiliaries, copulas, and nominal subjects.
- LLM texts use more numeric-modifier dependencies than human texts, consistent with their higher use of the NUM part-of-speech tag.
- LLM texts also show higher ratios of auxiliary, copula, and nominal-subject dependencies than human texts.
- Dependency-type usage is similar among LLMs, while Falcon is the most distinct model and all LLMs differ from human news.
- Larger LLaMa models are slightly closer to human dependency-type usage than smaller LLaMa models.
4.1.8 Constituents
Human and LLM texts differ in constituent lengths and constituent-type distributions, with humans favoring several phrase types and LLMs favoring verb phrases and subordinate clauses.
- Constituent lengths: Human sentences are longer on average, but LLMs have longer average sentence constituents.
- Constituent lengths: Falcon 7B shows the largest constituent-length differences among the language models.
- Constituent lengths: Within LLaMa, constituent lengths generally decrease with model size, but increase again across most bins for LLaMa 65B.
- Constituent types: Human text uses more noun, adjective, and prepositional phrases, while LLM text uses more verb phrases and subordinate clauses.
- Constituent types: Adverbial-phrase frequencies are mostly similar, except for Falcon 7B, whose divergence from human text is substantial.
4.2 Semantic analysis
Semantic analyses compare emotions, similarity, and gendered pronoun use in human and LLM-generated news. LLMs are more neutral and joyful overall, while their gender bias generally exceeds the human baseline.
- Emotions: Over 50% of texts in every model and human sample are classified as neutral, with LLMs slightly more neutral than humans.
- Emotions: Human texts contain more disgust and fear, whereas LLMs produce more surprise and especially joy; anger is roughly equal.
- Emotions: Fear increases with parameter count across LLaMa models from LLaMa 13B, making larger models more human-like on this emotion.
- Emotions: Mistral most closely matches human emotion expression, while Falcon produces more joy and less anger and surprise than other models.
- Text similarity: Smaller LLMs do not necessarily have lower sentence similarity to human text, and differences across model families are negligible.
- Gender bias: Human texts use male pronouns 1.71 times more often than female pronouns, and every model except Falcon 7B increases this bias.
- Gender bias: LLaMa models use around 15% more male pronouns relative to humans, while Mistral 7B increases the male–female ratio by 3%.
5 Conclusion
The study compares human and machine English news under controlled conditions using post-release articles to address memorization concerns. It finds persistent linguistic differences, including emotion and gender-bias patterns, while identifying clear scope for broader future comparisons.
- The study compares human and machine texts under controlled conditions using modern generative models and post-release news headlines.
- Mistral 7B produces emotion distributions more similar to humans than LLaMa and Falcon models.
- Negative emotions increase with model size, consistent with prior findings associating larger models with heightened toxicity.
- All models except Falcon 7B exacerbate the male-pronoun bias found in human text.
- Future work should extend the analysis across languages, instruction-based models, and domains.
Appendix A: Statistical analysis of metrics
The appendix tests whether human and LLM texts differ in sentence length, arc length, and standardized TTR. All human–LLM comparisons are significant, while several model-to-model differences are not.
- Statistical tests: A t-test compares human and LLM means for sentence length, arc length, and standardized TTR.The corresponding results are reported in Tables 10–12.
- Human–LLM differences: p-values < 0.05 occur for every human–LLM comparison across the three metrics.These comparisons are statistically significant for each LLM relative to human text.
- Model comparisons: Several model-to-model differences are not statistically significant, including LLaMa 7B versus 30B for STTR and most models for arc length.Falcon 7B versus LLaMa 65B is also non-significant for sentence length.
Declarations
The authors report no relevant conflicts of interest and make the article available under a Creative Commons Attribution 4.0 license.
- Conflict of interest: The authors declare no conflict of interest relevant to the paper.
- Open access: The article permits sharing, adaptation, and reproduction under a Creative Commons Attribution 4.0 International License with attribution and change notices.