Source-linked AI summary
Synthetic linguistic data
Giuseppe Samo
TL;DR
The paper addresses how synthetic linguistic data relate to evidence used to formulate and evaluate theories of language. It distinguishes corpus-like model outputs from controlled synthetic data, concluding that comparison with natural language can reveal shared patterns and divergences, while interpretation depends on generation conditions.
Problem
The relationship between language-model evidence and the evidence needed to formulate and evaluate linguistic theories remains a methodological question.
Source: §1 · p. 2Method
Results
Comparing synthetic and natural data can identify patterns shared by human language use and model behaviour, as well as divergences requiring further explanation.
Source: §4 · p. 7Takeaways & Limitations
Similarities between model outputs and natural language, or between model responses and human judgments, should not automatically be treated as evidence of equivalent underlying mechanisms.
Source: §4 · p. 6Interpreting synthetic data requires careful consideration of the conditions under which they are generated.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Synthetic linguistic data have become increasingly prominent in linguistic research, from the analysis of large collections of AI-generated texts to controlled experiments investigating grammatical and lexical preferences. This glossary entry extends Bresnan's (2016) distinction between the linguistic garden and the bush by introducing an artificial plant nursery as a metaphor for synthetic data. We distinguish between synthetic bush data, consisting primarily of corpus-like collections of model-generated language, and synthetic garden data, in which model outputs are elicited under controlled conditions to investigate specific linguistic hypotheses. We argue that synthetic data constitute a distinct object of linguistic investigation: comparing them with natural language data can reveal both shared patterns and divergences, but their interpretation requires careful consideration of the conditions under which they are generated.
1 Introduction
Linguistics has long debated controlled judgments and naturally occurring language data; this entry extends the garden-and-bush metaphor with an artificial nursery for synthetic data. It distinguishes corpus-like and controlled forms of model-generated language and treats synthetic and natural data as distinct objects of investigation.
1 Introduction
- The entry extends Bresnan’s (2016) garden-and-bush metaphor with an artificial plant nursery for synthetic linguistic data.The garden represents carefully controlled data, while the bush represents naturally occurring language data.Source: §1 · p. 1
- Synthetic data include corpus-like conversational-AI outputs and other model-generated materials elicited under controlled conditions.Examples include masked-language-model and fill-mask procedures used to augment curated evaluation datasets.Source: §1 · pp. 1–2
- Synthetic data are neither straightforwardly garden data nor direct samples from the bush, and may resemble natural specimens while differing from both categories.Source: §1 · p. 2
- Treating natural and synthetic data as distinct objects allows examination of what each reveals without reducing evidence from one to the other.The entry situates this distinction within debate over whether language-model predictive success provides evidence for theories of language.Source: §1 · p. 2
- Synthetic data are also used to evaluate computational models and train them, raising questions about whether generated examples reinforce existing biases or limitations.Source: §1 · p. 2
2 Synthetic bush data
Synthetic bush data are corpus-like collections of conversational-AI outputs, but their properties depend on prompts and other generation conditions. Research described here uses these outputs to examine generated language and model capabilities, while underscoring the need to document how they were produced.
2 Synthetic bush data
- Conversational-AI outputs can be collected as corpus-like datasets, but their properties depend on prompts and other generation parameters, including temperature.They can also be treated as elicited data, with each model serving as an informant or population of speakers.Source: §2 · pp. 2–3
- GPT-4o reproduced some surface-level features of Hemingway’s and Shelley’s styles, but its generated texts retained similarities to more generic outputs.In-context learning improved stylistic alignment.Source: §2 · p. 3
- Synthetic Italian texts show English fingerprints, and prompting instructions in English can affect their characteristics.The cited work links the fingerprints to English’s pivotal role in current large language models (Wendler et al. 2025).Source: §2 · p. 3
- ChatGPT followed instructions in a rule-generalisation task but failed to infer its underlying structure and consistently gave incorrect solutions, even with considerable prompting assistance.The task tested patterns such as subject–verb agreement.Source: §2 · p. 3
- ChatGPT labelled contemporary Italian topicalisation as archaic, struggled with object clitics, and may represent more-resourced languages at the expense of minor varieties.The passage reports that around one fifth of direct objects are fronted in Italian treebank data.Source: §2 · p. 3
- Conversational-AI sycophancy can extend to harmful or questionable user behaviour and influence users’ subsequent judgments; synthetic semantic profiles can also differ from natural-language sources.The piazza ‘square’ example differed across synthetic data and natural-language sources, including general-purpose corpora and specialised sources.Source: §2 · p. 4
- Tagalog disaster-scenario outputs were rated poorly by speakers, although both tested models rated the other model’s Tagalog outputs extremely highly.The outputs were assessed for linguistic properties, plausibility, and suitability by practitioners and native speakers.Source: §2 · p. 4
- Interpretable and replicable comparisons require documenting the model, prompt content and language, and possibly the generation date.Such documentation is presented as essential for comparing studies and generalising about synthetic language.Source: §2 · p. 4
3 Synthetic garden data
Synthetic garden data are generated under controlled conditions to test specific linguistic hypotheses, using methods such as model elicitation and numerical acceptability judgments. The studies described use these approaches to investigate grammatical expectations, representations, and judgments, while showing that results can depend on the models and linguistic structures examined.
3 Synthetic garden data
- Synthetic garden data are generated under controlled conditions to investigate specific linguistic hypotheses, extending experimental methods used in psycholinguistics.Source: §3 · p. 5
- Masked-language-model and fill-mask procedures can elicit responses to designed stimuli for studying lexical preferences and grammatical constraints, including subject–verb agreement.Surprisal measures the information-theoretic unexpectedness of a linguistic unit given its preceding context and can illuminate grammatical constraints alongside probabilistic preferences.Source: §3 · p. 5
- Surprisal patterns in French clefts mirror human-processing patterns, while small-scale transformers clearly mark degraded ratings for whex-ti configurations in Chinese.The cited work also applies surprisal to grammatical expectations in complex English constructions.Source: §3 · p. 5
- Pronominal-only sentences and sentences with noun phrases occupy different embedding regions, and their shared syntax does not emerge when either type is considered alone.Source: §3 · p. 5
- Subword segmentation can affect morphological-form representations in Turkish and Hebrew paradigms, particularly when language-specific morphology does not align with token boundaries.Source: §3 · p. 5
- Conversational-AI acceptability ratings show asymmetries across models and constructions; for the Romance phenomena investigated, smaller monolingual transformers performed better.Romansh Verb Second violations were partially reflected in conversational-AI ratings, while Neapolitan topic–subject agreement remained challenging.Source: §3 · p. 6
- Carefully constructed model evaluations can be compared with human judgments and linguistic-theory predictions using replicable methods, and can start in-depth research on human language.Source: §3 · p. 6
4. Conclusions
Synthetic bush and synthetic garden data offer complementary approaches to linguistic inquiry, treating synthetic data as objects of investigation alongside natural language. Their comparison can reveal shared patterns and divergences, but interpretation requires attention to generation conditions and does not establish equivalent underlying mechanisms.
4. Conclusions
- Synthetic bush data collect and analyze corpus-like model outputs, while synthetic garden data use controlled generation to investigate specific linguistic phenomena.Source: §4 · p. 6
- Similarities between synthetic and natural data, or between model responses and human judgments, do not by themselves show that their underlying mechanisms are equivalent.Source: §4 · p. 6
- Treating natural and synthetic data as distinct but potentially complementary objects allows comparison to identify shared patterns and divergences for further explanation.Source: §4 · pp. 6–7