PaperlayerView exact revision on arXiv

Source-linked AI summary

Does AI-Generated Scientific Text Follow Human Argumentation Patterns? A CARS-Based Comparison of Research Article Introductions

Abdelrahman Sadallah, Narjes Sheikh Asadi, Lonneke van der Plas

arXiv:2610.01353v1cs.CL

TL;DR

Surface cues commonly used to compare human and machine-written text are fragile, while rhetorical structure offers a way to examine how introductions make their argument. The paper compares original linguistics-article introductions with LLM-generated counterparts using CARS, finding that generated sequences are narrower and more rigid, especially with explicit CARS guidance.

  • Problem

    Comparisons of human and machine-written text have relied on surface cues, which are fragile, while it matters how LLMs build arguments and position work relative to prior research.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The authors compare expert-annotated introductions from 76 linguistics papers with four models’ generated counterparts using a validated CARS annotator.

    Source: §3.1 · p. 3, §3.3 · p. 5, §6 · p. 8

  • Results

    Generated introductions are more rigid and use narrower move sequences across all four models; CARS guidance pushes them further from human structure, and a CARS-informed reader distinguished sources with 90% accuracy.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Rhetorical structure carries a usable trace of machine authorship in the argument, beyond surface style that paraphrasing can erase.

    Source: §6 · p. 8

    The models receive bibliographic metadata and abstracts rather than the full cited works, and inherit the author’s citation selection, so their context is a curated approximation of the author’s knowledge.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models are moving from helping write up research to helping do it, which makes it important to know how the scientific text they produce differs from human writing. Work on this question has stayed mostly at the surface, using lexical and stylistic cues that light paraphrasing erases. We look instead at rhetorical structure, the sequence of argumentative moves through which a text makes its case. We study research-article introductions under Swales' CARS model, and compare original introductions from published linguistics articles with generated counterparts of the same papers. We find that human-written introductions are more flexible in which moves they use and in what order, while the generated ones are more uniform. Giving the models the CARS definitions makes them more rigid.

1 Introduction

This study compares the rhetorical structure of human-written and LLM-generated research-article introductions, rather than relying on surface style. Using Swales’ CARS model, it asks how their move sequences differ and whether explicit CARS guidance changes generated introductions.

1 Introduction

  • The study examines rhetorical structure—the sequence of argumentative moves—and focuses on research-article introductions using Swales’ CARS model.CARS describes territory, niche, and study-occupying moves; moves can recur, appear in different orders, or be absent.Source: §1 · pp. 1–2
  • The framework reconstructs each paper’s content and cited works, generates introductions under basic and CARS-guided prompts, then annotates and compares them.It uses 76 published linguistics papers and four LLMs, producing 608 generated introductions.Source: Figure 2 · p. 2, §1 · p. 2
  • The research questions ask whether an LLM annotator can identify CARS moves reliably, how human and generated introductions differ, and what CARS guidance changes.Expert CARS annotation is slow and hard to scale, motivating an automatic annotator.Source: §1 · p. 2
  • In blind judgements, a human reader distinguished machine-generated from original research-article introductions in 90% of cases, relying almost solely on their rhetorical structures rather than surface-level features.Source: §1 · p. 2
  • The authors also present a validated LLM-based CARS annotator that segments and labels introductions for rhetorical-structure research at scale.The introduction reports 92% of human labelling accuracy.Source: §1 · p. 2
  • Generated introductions follow a narrower, more uniform range of move sequences than human-written introductions.The paper presents this as its first rhetorical-structure comparison of original and LLM-generated research-article introductions.Source: §1 · p. 2

2 Related Work

Prior work addresses machine-written scientific text through surface-based detection and studies rhetorical moves through genre analysis and annotation. This paper extends that work by comparing research-article introductions across multiple models and validating joint segmentation and classification against a human baseline.

2.1 LLMs for scientific writing and its detection

  • Prior machine-text detection uses surface signals, which the authors describe as fragile, motivating examination of rhetorical structure as a deeper possible signal.The cited approaches include token-probability rank, probability curvature, decoding artifacts, and watermarking.Source: §2.1 · p. 3

2.2 Rhetorical move structure and its analysis

  • Earlier rhetorical-structure research includes CARS move analysis, argumentative zoning, and EAP classifiers, while recent studies compare generated and human abstracts.Prior abstract comparisons focus on move frequency and ordering; Kim and Lu (2024) prompt ChatGPT to label CARS moves and steps in applied-linguistics introductions.Source: §2.2 · p. 3
  • This study compares introductions rather than abstracts, includes several models, reconstructs cited-literature context, and validates joint segmentation and classification against a human–human ceiling.The authors note that expert CARS annotation is slow and hard to scale.Source: §2.2 · p. 3

3 Materials and methods

The study uses expert-annotated linguistics articles, generates counterpart introductions under two prompt settings, and applies an LLM-based CARS annotator. The annotator approaches the human baseline on move and boundary measures and is used to compare the introductions.

3.1 Corpus and gold annotations

  • The final corpus contains 76 published Applied Linguistics research articles, with expert CARS annotations used as gold labels.The 76 were retained from 80 papers after four paywalled articles were discarded; the scheme has three moves and ten steps.Source: §3.1 · p. 3

3.2 Generating introductions

  • Each model receives the paper’s title and content plus metadata and abstracts for works cited in its original introduction, and generates an introduction targeted to the original’s length.The authors describe this as a reconstruction of the author’s information context; the cited works come from the original introduction.Source: §3.2 · pp. 3–4
  • Four commercial and open-weight LLMs write each of 76 introductions under basic and CARS-guided prompts, yielding 608 generated introductions.The CARS-guided prompt supplies move and step definitions and asks the model to plan a move sequence before writing.Source: §3.2 · p. 4

3.3 CARS annotation

  • The annotator jointly segments and labels the full introduction because boundary-only prompting under-segmented and sentence-level classification over-segmented in pilot tests.Joint segmentation and classification improved pilot boundary F1 from 0.50 to 0.71.Source: §3.3 · p. 4
  • Two graduate-student annotators independently labelled 32 introductions, and both their output and the automatic annotator’s output were scored against expert gold annotations.Each human annotated 16 papers; no introduction was labelled by both annotators, so the human–human ceiling is estimated from a subset.Source: §3.3 · p. 5
  • The annotator reaches 92% of human label accuracy (0.694 vs. 0.752; macro F1 0.664 vs. 0.699) and approaches the human baseline on move accuracy and boundary F1.Move accuracy is 0.901 vs. 0.911, and boundary F1 is 0.725 vs. 0.756.Source: §3.3 · p. 5

4 Comparing Original vs. Generated Introductions

Generated introductions differ from originals in both what they include and how they order it: they use the CARS scheme more uniformly, and their move sequences are less varied. CARS guidance pushes them further toward uniformity rather than toward the human profile.

4.1 General distributions and ordering

  • Humans devote 59.5% of segments to Move 1, 16.0% to Move 2, and 24.5% to Move 3; generated sets shift the balance from Move 1 toward Move 2.The Move-1 share falls below the human level in five of eight generated sets, reaching 46.8%.Source: §4.1 · p. 5, Table 1 · p. 6
  • All nine sources most often open with claiming centrality, while Move-2 counter-claiming is absent from human annotations and appears in only four automatic-annotator segments.The authors characterize indicating a gap as a softer framing than directly attacking prior claims.Source: §4.1 · p. 6
  • Humans open with a centrality claim in 46% of papers or directly with literature in 32%, while models almost never choose the latter and open with centrality in 58–84%.The human RR→MTG→RR pattern appears in a quarter of papers, while each model’s strongest signature appears in at least 33%.Source: §4.1 · p. 6
  • Two thirds of human introductions return to a previously left move, usually Move 1; basic ChatGPT, Claude, and Gemma match that rate, while Qwen cycles significantly more often.CARS-guided sets depart downward from the human rate.Source: §4.1 · p. 6
  • Humans use an average of 2.67 moves and 5.3 steps, while most generated sets use all three moves and 6.1–7.4 steps; 86–96% of counterparts add steps humans omitted.The generated counterparts often add a niche, gap, or centrality step absent from the original.Source: §4.1 · p. 6
  • CARS guidance lowers cyclical introductions across all models, raises the number of distinct steps, and makes three of four models’ move balance less human-like.The cyclical share falls for Claude from 74% to 51%, Gemma 67% to 50%, ChatGPT 70% to 63%, and Qwen 95% to 87%.Source: §4.1 · pp. 5, 7

4.2 Graph analysis of the move structures

  • No generated set exceeds 0.43 in step-level edit similarity to originals, and bag similarity is higher than edit similarity, indicating better matches in content than order.The closest generated set reproduces well under half of the original authors’ step order.Source: §4.2 · p. 7, Table 3 · p. 7
  • Originals realize 36 distinct move structures and have self-similarity of 0.33, versus 8–35 structures and 0.37–0.46 self-similarity for generated sets; CARS guidance reduces diversity further.Table 3 reports generated-to-original similarity on the left and within-source similarity and structure counts on the right.Source: §4.2 · p. 7, Table 3 · p. 7

5 Manual analysis

Close reading found generated introductions more rigid in coverage and ordering than their human originals. In a blind reading task, rhetorical structure—not writing style—was the main basis for distinguishing them, and CARS-guided generations were especially recognizable.

5.1 Close reading of paired introductions

  • All ten generated introductions in the close reading realize all three moves, while one of the five originals omits an explicit niche and moves from territory to the present study.When human authors leave the niche implicit or omit it, the models state it.Source: §5.1 · p. 7
  • Generated versions stay closer to the canonical Move 1–2–3 progression; 89% of their returns follow Move 2→Move 1, versus 57% for originals.No generated introduction opens with Move 3, compared with one of the five originals in the close reading.Source: §5.1 · pp. 7–8
  • The authors suggest positional bias: models may follow the order in which CARS presents moves, and CARS-guided prompts list them canonically despite saying they may appear in any order.The guided outputs follow canonical order more closely than basic-prompted outputs, contrary to the intended effect of the guidance.Source: §5.1 · p. 8
  • Four examined authors who demonstrably knew CARS showed no common structure; the authors report journal quartile, not framework knowledge, as the distinguishing factor.The close reading covered four of the 76 articles whose authors engaged with CARS or rhetorical move analysis.Source: §5.1 · p. 8

5.2 What does a human reader rely on to distinguish between original and generated introductions?

  • A blind reader classified 108 of 120 introductions correctly and cited argument structure for 118 items, compared with writing style for 15.The reader reported using relations among moves—including order and returns—as well as corpus-level tendencies provided beforehand.Source: §5.2 · p. 8
  • The reader identified all 30 CARS-guided generations as machine-written, compared with 24 of 30 basic generations.This aligns with the reported greater uniformity and rigidity of CARS-guided structures.Source: §5.2 · p. 8

6 Conclusion

The paper finds that LLM-written research-article introductions are more rigid than human originals and that CARS guidance pushes them further from human structure. A reader using CARS knowledge and a summary of model patterns distinguished original from generated introductions with 90% accuracy.

6 Conclusion

  • Across the four generation models, introductions are more rigid and settle into a narrower range of move sequences than human-written originals.Where human authors select some moves, the models cover most of them.Source: §6 · p. 8
  • Supplying CARS definitions pushes generated introductions further from human structure.The conclusion characterizes the guided introductions as more rigid in their ordering.Source: §6 · p. 8
  • A reader with CARS knowledge and a summary of model patterns distinguished original from generated introductions with 90% accuracy, performing better on CARS-guided generations.The paper concludes that rhetorical structure carries a usable trace of machine authorship.Source: §6 · p. 8

Limitations

The comparisons are bounded by an approximate reconstruction of authors’ information context, a single-discipline corpus, and limits in CARS depth and annotator validation.

Limitations

  • Models received cited works’ metadata and abstracts rather than full texts, and inherited the human authors’ finished citation selections.The authors describe this as a feasible, consistent approximation, not the same knowledge position as the human author.Source: Limitations · p. 9
  • The study covers 76 applied-linguistics papers; a larger, multi-domain corpus would strengthen statistical claims and test how far the patterns transfer.PDF extraction introduced noise, which the authors mitigated through manual quality control.Source: Limitations · p. 9
  • CARS move and step patterns vary across disciplines and sub-disciplines, so absolute distributions may not transfer beyond this corpus.The authors identify the human–AI gap, rather than absolute distributions, as the more transferable finding.Source: Limitations · p. 9
  • CARS labels rhetorical functions but does not analyze the internal reasoning links among claims, evidence, and warrants.The authors frame this as a first step and suggest layering finer-grained argumentation frameworks onto the analysis.Source: Limitations · p. 9
  • The human–human ceiling is estimated from 32 introductions, with no introduction labeled by both annotators.A larger double-annotated set would sharpen the ceiling estimate.Source: Limitations · p. 9

Ethics Statement

The section describes the study’s ethical handling and documents its annotation scheme, preprocessing pipeline, and model configuration.

Ethics Statement

  • Only publicly accessible published articles were included; four paywalled papers were dropped, and the articles were not redistributed.No unpublished or otherwise private material was collected or processed.Source: Ethics Statement · p. 9
  • Annotation came from paper authors, unpaid academic collaborators, and a salaried research assistant; all annotators knew how their annotations would be used.The assistant’s salary covered the work time spent annotating.Source: Ethics Statement · pp. 9–10
  • Locally served Gemma and Qwen inputs stayed on the researchers’ machines; API providers stated that submitted data would not train their models.The inputs were published papers, so the authors say nothing sent to an external service was confidential.Source: Ethics Statement · p. 10
  • Generated introductions were research artifacts, not intended for submission, and some journals and conferences restrict or forbid generated content.Source: Ethics Statement · p. 10

A The CARS annotation scheme

B Preprocessing pipeline

  • The preprocessing pipeline converted PDFs to structured text, parsed reference lists separately, and manually corrected remaining extraction errors.Docling was selected after inspection for preserving multicolumn layouts, headings, and tables; GROBID parsed reference lists.Source: B Preprocessing pipeline · p. 13
  • The team identified and verified introduction citations, retrieved bibliographic metadata and abstracts, and supplied titles and authors when abstracts were unavailable.The corpus included roughly 1,065 cited works, averaging 14 per introduction; abstracts were recovered for about 72%.Source: B Preprocessing pipeline · p. 13

C Model configuration

  • The study used four generation models and gpt-5.5 as the automatic annotator, with provider-default decoding settings and low reasoning effort.The four generation models were gpt-5.5, claude-opus-4-8, Gemma-4-31B-it, and Qwen3.5-27B.Source: C Model configuration · p. 13

D Annotator validation results

The validation results describe where automatic and human per-class F1 differ and where they are at parity across the CARS steps.

D Annotator validation results

  • The human–annotator gap is concentrated in claiming centrality, with smaller gaps in topic generalization and announcing principal findings.The two annotators are at parity on the two most frequent classes, M1-RR and M3-OP, and on indicating a gap; Table 6 reports per-class F1 on the 32-paper validation set.Source: D Annotator validation results · p. 14, Table 6 · p. 14

E.1 Annotator validation metrics

The section defines how the annotator is evaluated and how introductions are compared, then reports differences in rhetorical granularity and variation across journal quartiles. Generated introductions show less quartile-linked variation than the originals.

E.1 Annotator validation metrics

  • The evaluation separates boundary segmentation from classification: it scores boundary F1 within ±5 characters, plus sentence-level label accuracy and parent-move accuracy.WindowDiff and Pk are also reported as segmentation error scores; sentence-level classification accounts for differences in predicted and gold segment counts.Source: §E.1 · p. 14

E.2 Move-structure similarity measures

  • Edit similarity measures class order, while bag similarity compares word-share profiles; together they distinguish arrangement from content allocation.Both measures compare move-chain graphs at move and step levels. Word shares avoid making a rhetorical class appear more important merely because it was split into more segments.Source: §E.2 · pp. 14–15

F Significance testing

  • Comparisons pair each original with generated introductions for the same paper (n = 76), using Wilcoxon signed-rank tests except for binary cyclicity, tested with McNemar.A star marks q < 0.05; model comparisons use each paper’s step edit similarity to its own original.Source: F Significance testing · p. 15

G Descriptive statistics

  • Generated introductions have 9.9–13.9 segments per introduction versus humans’ 9.4, and usually shorter segments: 50.2–57.3 versus 65.1 words.CARS guidance moves Claude and Qwen closer to human granularity, including Claude’s 11.4 → 9.9 segments; ChatGPT and Gemma remain essentially unchanged, while guided Claude has 66.1 words per segment.Source: G Descriptive statistics · p. 15, Table 7 · p. 15

H Variation across journal quartiles

  • Originals vary by quartile: Q1 averages 2.88 distinct moves and 75% cyclicity, Q3 averages 2.50 moves and 55% cyclicity, while Q2 uses 5.7 distinct steps.Generated introductions use nearly all three moves in every quartile, more steps than originals, and have less variation in cyclicity.Source: H Variation across journal quartiles · p. 15
  • Generated introductions use 2.97–2.99 moves across quartiles and 6.1–7.2 steps versus humans’ 5.0–5.7, failing to reproduce the originals’ quartile differences.Originals range from 2.50 to 2.88 moves and 55% to 75% cyclicity across quartiles; generated introductions cycle at a nearly constant 66%–74%.Source: H Variation across journal quartiles · p. 15

I The Swales-awareness effect

Examining four authors who demonstrably knew CARS, the section finds that their introductions were not more schematic. Journal quartile tracked structure more closely than authors’ framework knowledge.

I The Swales-awareness effect

  • The four examined articles were selected because their authors engaged with CARS or rhetorical move analysis, with one article sampled from each journal quartile.The appendix provides readings summarized in the main-text discussion.Source: I The Swales-awareness effect · p. 15
  • None of four framework-aware authors followed the standard Move 1–2–3 funnel; two omitted an explicit niche, one folded it into Move 1, and one returned to Move 1.Their command of the model did not surface as a more schematic structure.Source: I The Swales-awareness effect · p. 16
  • Journal quartile is a stronger predictor of introduction structure than authors’ demonstrated knowledge of CARS.Higher-quartile introductions move among the three moves more freely and realize Move 3 more fully; lower-quartile ones lean on Move 1 and compress or omit later moves.Source: I The Swales-awareness effect · p. 16

J Human detection study

In a blind human detection study, a reader correctly classified 108 of 120 introductions, and all CARS-guided generations were identified as machine-written. The appendix also documents the generation prompts and automatic annotator instructions.

J Human detection study

  • The study sampled 60 papers across quartiles, pairing each original with one Claude generation and crossing basic versus CARS-guided prompting with labelled versus unlabelled display.Each of four cells contained 15 papers; the 120 items were presented in a fixed order with each paper’s versions at least 20 positions apart.Source: J Human detection study · p. 16
  • The reader correctly judged 108 of 120 items, with six generated introductions mistaken for human and six originals mistaken for generated.The judge was correct on 54 of 60 labelled items and 54 of 60 unlabelled items; move annotations made no difference.Source: J Human detection study · pp. 16–17
  • All 30 CARS-guided generations were identified as machine-written, compared with 24 of 30 basic generations; 63% followed the 1 →2 →3 order, versus 13% of basic generations and 15% of originals.The 30 CARS-guided introductions used only 6 distinct move structures, compared with 29 among the 60 originals.Source: J Human detection study · p. 17
  • The reader cited structure on 58 of 60 unlabelled items and reported doing the move analysis mentally, whether or not annotations were shown.The judge had read the CARS guidelines and a summary of corpus-level tendencies beforehand.Source: J Human detection study · pp. 16–17

K.1 Basic generation prompt

  • The basic generation prompt requests a fresh applied-linguistics introduction from article content and cited works, with a target length and JSON-only output.The original introduction is withheld; the prompt asks for author–year citations and permits other relevant works.Source: §K.1 · p. 18

K.2 CARS-guided generation prompt

  • The CARS-guided prompt adds move and step definitions and asks the model to plan their sequence internally before writing unlabeled introduction prose.The prompt explicitly permits moves to recur and appear in any order, rather than requiring the 1 →2 →3 sequence.Source: §K.2 · p. 18

K.3 Generation user message

  • For both generation settings, each paper’s input includes its title, article content, cited works, and target length; the original introduction is withheld.The per-paper prompt supplies the article body and references as separate inputs.Source: §K.3 · p. 18

K.4 Annotator prompt

  • The automatic annotator segments introductions by communicative function, labels move before step, and uses the annotation manual, operational clarifications, and worked examples.A new segment is warranted only when communicative function changes, not because of surface cues, connectors, sentence boundaries, or paragraph layout.Source: §K.4 · pp. 18–19
Loading 2610.01353v1…