Source-linked AI summary
Do LLMs write like humans? Variation in grammatical and rhetorical styles
Alex Reinhart, Ben Markey, Michael Laudenbach, Kachatad Pantusen, Ronald Yurko, Gordon Weinberg, David West Brown
TL;DR
Prior work had identified surface differences between human and LLM writing, but had not examined LLMs’ rhetorical styles. The study compares parallel human- and LLM-written corpora using lexical, grammatical, and rhetorical features. It finds systematic stylistic differences between humans and models, with instruction-tuned models’ writing more unlike human writing than base models’.
Problem
Prior research had examined surface features and LLM detection, but rhetorical styles of LLMs remained unstudied.
Source: Discussion · p. 6Method
The authors built parallel human- and LLM-written corpora from shared prompts and analyzed them using Biber’s linguistic feature set.
Source: Methods · p. 2Results
Linguistic features distinguish human texts from model texts and different models, while instruction tuning pushes Llama models to produce writing that reads unlike human writing.
Source: Discussion · p. 6Takeaways & Limitations
Biber’s tagset can reveal otherwise tacit linguistic distinctions that help identify machine-generated text.
Source: Discussion · p. 6Some word-overuse findings may reflect generation artifacts, including repeated text in individual model outputs.
Source: Generalization across corpora · p. 3
Abstract
from arXiv · showhide
Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.
Methods
The methods compare human and LLM writing using parallel texts drawn from multiple genres, quantify linguistic features, and check generalizability with academic abstracts.
Methods
- From 12,000 human-authored English texts, researchers extracted two consecutive roughly 500-word chunks; each LLM received the first for context and a style sample, then was prompted to write 500 words in the same style, tone, and diction.Source: Methods · p. 2
- They measured 66 lexical and grammatical categories using Douglas Biber’s tagset (19, 25, 26) and tested human–LLM differences with paired Wilcoxon signed-rank tests and Bonferroni correction.Features included word length, nominalizations, agentless passive voice, hedging, and clausal coordination.Source: Methods · p. 2
- For a generalizability check, they compared arXiv abstracts with GPT-3.5 abstracts generated from preprint titles (17).This tested consistency across a different model and a distinct genre.Source: Methods · p. 2
Results
Biber features distinguish LLMs from human writing, while stylistic differences and cross-corpus detection vary by model and instruction tuning.
Classifying text by source.
- A random forest classified seven HAP-E text sources with 66% test accuracy versus 14% expected, with most errors between versions of the same LLM.Only 4.2% of LLM texts were classified as human, and 9.8% of human texts as LLMs.Source: Classifying text by source · p. 2, Figure 2 · p. 3
Differences in style and vocabulary.
- Instruction-tuned Llama and GPT-4o models overused distinctive vocabulary; GPT-4o used “tapestry” in 23% of outputs and “amidst” in 27%.GPT-4o and 4o Mini used words such as “camaraderie,” “palpable,” “tapestry,” and “intricate” at more than 100 times human rates. The passage notes that this bias appears associated with instruction tuning, and that some overuse may reflect generation artifacts.Source: Classifying text by source · p. 3, Differences in style and vocabulary · p. 3
Distinguishing individual LLMs.
- Human-versus-LLM classifiers reached 93–98% accuracy across HAP-E and CAP, while Llama base models were less distinguishable.Lasso classifiers achieved around 75% accuracy for Llama base models, suggesting their distinguishing signals include interactions between Biber features; lower accuracy also suggests instruction tuning may make writing easier to distinguish from human text.Source: Distinguishing individual LLMs · p. 3
Generalization across corpora.
- On M4 arXiv preprints, classifiers trained on instruction-tuned LLMs detected GPT-3.5 output above chance, while Llama-base classifiers reached 50%, equal to chance.These results indicate shared features among instruction-tuned LLMs but difficult generalization across models or registers. Some apparent word overuse may be generation artifacts, including repeated text that drove Llama 3 8B’s use of “Deborah” to 52 times the human rate.Source: Generalization across corpora · p. 3
Differences in style and vocabulary.
- Instruction-tuned models diverged from human style more than Llama base models, and larger models did not appear to match human usage better.All four instruction-tuned models favored informationally dense features; GPT-4o used present participial clauses 5.3 times as often as humans. GPT-4o models avoided clausal coordination while Llama variants favored it; the models also differed in downtoner use.Source: Differences in style and vocabulary · p. 3, Figure 3 · p. 4
Discussion
The discussion argues that human and LLM texts differ in stylistic features, and that instruction tuning appears to contribute to those differences. It presents linguistic analysis as a way to reveal machine-text patterns and stresses the need to revise outputs for context, with different implications for experts and learners.
- Human and model texts, as well as outputs from different models, differ in features including participial clauses, subject “that” clauses, passive voice, and nominalizations.The discussion links these features to informationally dense prose.Source: Discussion · p. 6
- Comparisons of Llama base and tuned models suggest instruction tuning contributes to human-model style differences, though undocumented tuning prevents identifying the precise cause.The authors suggest the differences are not simply due to base-model training-text selection.Source: Discussion · p. 6
- Because LLMs do not adapt linguistic output to context like humans, experts can use them productively if they revise outputs using genre expertise.Prior experience helps experts judge what a document needs to succeed.Source: Discussion · p. 6
- For learners, using LLM text can displace practice shaping writing for audience and purpose; instructors should teach critical evaluation and revision.The discussion distinguishes using LLMs to learn concepts or generate ideas from using their output in a text.Source: Discussion · p. 6
- Biber’s tagset successfully modeled and classified text, showing that linguistic perspectives can reveal machine-generated patterns and inform LLM use and development.The authors frame comparative analysis as a way to identify teachable revision moments, not to build detectors or police student writing.Source: Discussion · p. 6