Source-linked AI summary
The Impact of Synthetic Data Augmentation on Discourse-Pragmatic Function Classification
Sara Sorahi, Kevin Tang, Reza Kazemian
TL;DR
The paper asks whether synthetic-data augmentation works differently depending on where generated examples lie relative to real training data, a central issue for sparse pragmatic classification. It partitions Llama 3.1 examples by cosine distance in RoBERTa space and compares quantity-matched placement strategies. Proximal examples yield the largest macro-F gain, balanced placement the highest accuracy, and no augmentation improves AUC.
Problem
Synthetic augmentation research emphasizes quantity and diversity, leaving the geometric relationship between generated and real examples underexamined in sparse discourse-pragmatic classification.
Method
Using 410 manually annotated BNC instances of look, the study generates Llama 3.1 training examples, partitions them by RoBERTa-space cosine distance, and compares quantity-matched placement conditions.
Results
Core-proximal examples produce the largest macro-F gain, a balanced near–far mix achieves the highest accuracy, and no condition improves AUC.
Takeaways & Limitations
Synthetic-data effectiveness depends on representation-space geometry as well as generation quantity, supporting distance-aware augmentation for low-resource pragmatic classification.
Takeaways & Limitations
The study uses a small corpus of 410 instances, including only 34 Discourse Marker and 18 Interjection examples.
Abstract
from arXiv · showhide
Synthetic data augmentation has become a common strategy for addressing class imbalance in NLP, but most approaches focus on the quantity and diversity of generated examples rather than their geometric relationship to real training data. We investigate this question in the context of discourse pragmatic function classification, a task where data sparsity is a structural feature rather than a collection artefact. Using 410 manually annotated instances of the English word look drawn from the British National Corpus, spanning four functions: Attention Signal, Directive, Discourse Marker, and Interjection. We generate synthetic training examples with Llama 3.1 and partition them by their cosine distance from real training data in RoBERTa embedding space. We compare six training conditions that differ in the placement of synthetic examples relative to the empirical decision boundary, while holding augmentation quantity constant across conditions. All augmented conditions improve macro F and accuracy over the real only baseline, but core proximal examples (NEAR) yield the largest gains in macro F (0.113), while a distance balanced mix achieves the highest accuracy (0.748). No condition improves AUC, indicating that augmentation shifts the decision boundary rather than improving the model's underlying probability estimates. These findings suggest that where synthetic examples land in representation space matters as much as how many are generated, with implications for low resource pragmatic classification more broadly.
1 Introduction
Synthetic augmentation research has emphasized quantity and diversity, but discourse-pragmatic classification raises a sharper question: whether generated examples are geometrically compatible with real data. The study uses multifunctional look to test how synthetic-example placement affects classification.
- Synthetic augmentation has largely focused on generating more varied examples, while distant synthetic data may blur class boundaries or shift decision boundaries.
- Discourse-pragmatic classification is difficult because functions depend on context and communicative intent, while pragmatic categories are sparse and benchmark resources remain limited.
- Automatic classification of look remains an open problem despite its extensive scholarly study across languages.
- Look can realize Attention Signal, Directive, Discourse Marker, or Interjection functions that emerge from context, prosody, and pragmatic inference rather than the word form alone.
- The study compares synthetic examples by cosine distance from real training data in RoBERTa space to test whether placement affects classification performance.
2 Data and Annotation
The dataset comprises manually annotated British National Corpus instances of look, covering four discourse-pragmatic functions with a strongly skewed class distribution. Evaluation uses authentic corpus examples in stratified train–test splits.
- The corpus contains 410 BNC instances of look manually annotated by two expert annotators for four discourse-pragmatic functions.
- The classes are Attention Signal, Directive, Discourse Marker, and Interjection, with counts of 287, 71, 34, and 18 respectively.
- Annotation focused on contextual discourse-pragmatic role rather than syntactic form, with ambiguous cases resolved through discussion.
- Evaluation used five independent stratified 80/20 splits preserving the original class distribution, with synthetic data restricted to training.
3 Synthetic Data Augmentation
Translation-based augmentation failed to preserve look’s pragmatic functions reliably, so the study moved to prompt-based Llama 3.1 generation with iterative function-specific refinement and manual review.
- Pragmatic augmentation is difficult because small changes in wording, register, or surrounding discourse can change function.
- Back-translation often removed or replaced look, altering the utterance’s pragmatic category, informality, and interactional stance.
- Reinserting look after translation produced unnatural sentences because the surrounding context had already been reorganized around its absence.
- Cross-translation compounded semantic and pragmatic distortions, with look disappearing or being replaced and category distinctions evaporating.
- Prompt-based generation with Llama 3.1 was refined using function-specific definitions, naturalistic examples, anti-formulaic instructions, and contextual requirements.
- The generated outputs were manually reviewed for grammaticality, naturalness, and functional appropriateness, and ambiguous or implausible examples were discarded.
4 Classification Experiments
The experiments hold augmentation quantity constant while varying synthetic-example distance from real training data in frozen RoBERTa space. Examples are grouped into near, middle, and far regions for comparison across training conditions.
- 4. Classification Experiments: Sentences were represented with mean-pooled frozen RoBERTa embeddings, allowing performance differences to be attributed to augmentation rather than encoder changes.
- 4. Classification Experiments: Synthetic minority-class examples were partitioned into equal-sized NEAR, MIDDLE, and FAR groups according to distance from real training instances.
- 4. Classification Experiments: All conditions add the same number of synthetic examples, isolating geometric relationship to the empirical decision boundary from augmentation quantity.
- 4. Classification Experiments: The six conditions include a real-only baseline and five augmentations that scale minority classes to the majority-class size using different distance-based mixtures.
- 4. Classification Experiments: A multinomial logistic regression classifier was trained on frozen embeddings to keep the explanatory focus on representation and training data.
- 4. Classification Experiments: Performance was averaged across five stratified splits using macro-F, accuracy, and macro-AUC, with authentic BNC examples in every test set.
5 Results
Across five stratified splits, augmentation consistently improves macro-F and accuracy over the real-only baseline, with the strongest overall gains from distance-aware placement. AUC remains flat, while function-level benefits are largest for the most under-represented classes.
- Overall performance: Every augmented condition improves macro-F and accuracy over the real-only baseline.
- Overall performance: The NEAR condition reaches macro-F 0.499 ± 0.065, a +0.113 gain over the 0.386 ± 0.045 baseline.REAL+BALANCED achieves the highest mean accuracy at 0.748 ± 0.040, while NEAR reaches 0.746 ± 0.033.
- Overall performance: The distance-sensitive ranking broadly favors synthetic examples closer to the natural class boundary.MIDDLE, FAR, and RANDOM also improve over baseline, but by smaller margins.
- Robustness and statistical significance: AUC shows no significant improvement, remaining within 0.695–0.726 while the real-only baseline achieves the highest AUC of 0.726 ± 0.053.The divergence suggests augmentation shifts final label assignments at the decision boundary rather than improving probability estimates.
- Robustness and statistical significance: Every augmentation strategy beats the real-only baseline on macro-F and accuracy in all five splits.The relative ranking remains stable across splits, although absolute scores fluctuate and split 4 is a shared performance peak.
- Function-level analysis: Function-level gains are largest for under-represented classes, especially INTJ, while the majority class AS remains essentially flat.INTJ rises from 0.14 to 0.36 under NEAR, whereas AS receives no synthetic examples and changes little.
- Function-level analysis: DM improves from 0.52 to 0.58–0.60, while DIR rises from 0.80 to 0.83–0.85 across augmented conditions.REAL+BALANCED reaches the highest DM F1 of 0.60 and DIR F1 of 0.85.
6 Conclusion
The study shows that synthetic augmentation effectiveness depends on where examples lie in embedding space, not only on how many are generated. Core-proximal examples maximize macro-F gains, while balanced near-and-far examples maximize accuracy without improving AUC.
- Core-proximal examples produced the largest macro-F gains, while a balanced near-and-far mix achieved the highest accuracy.The conclusion attributes the difference to synthetic-example placement relative to the real class distribution in RoBERTa embedding space.
- None of the augmentation improvements increased AUC, indicating a shifted decision boundary rather than better probability estimates.
- Distance-aware synthetic-data geometry should be considered when designing augmentation for low-resource pragmatic classification.The paper argues that generating more examples alone is insufficient when data sparsity is structural.
7 Limitations
The study is limited by its small and narrow dataset, frozen and potentially imperfect representations, simple distance measure, limited resampling, and prompt-dependent generation. These constraints restrict how confidently the findings can be generalized across models, domains, languages, and tasks.
- The corpus contains only 410 instances, including 34 DM and 18 INTJ examples, so augmented training sets remain limited.
- Frozen RoBERTa representations were not adapted to discourse-pragmatic data, leaving encoder adaptation and geometry-sensitive augmentation unresolved.
- Mean cosine distance captures only part of embedding geometry, omitting density and cross-class proximity.
- With five stratified splits, p-values and effect sizes remain sensitive to the particular partitions.
- The experiments use one language and one word, so generalization to other items, languages, and pragmatic tasks remains unestablished.
- Function-specific prompt templates may confound distance effects with simpler input-length differences.
- Results may not transfer directly to other language models or domains without comparable prompt refinement.
8 Ethical Considerations
The study uses licensed, publicly available corpus data without new human-subjects data or personally identifiable information. Synthetic examples came from a public model, were manually reviewed, and pose no anticipated direct harms, though downstream applications warrant scrutiny.
- The corpus is publicly available under an academic-research licence and contains no personally identifiable information.
- Synthetic examples were generated with publicly released Llama 3.1, served locally, and manually reviewed before training use.
- The annotation involved two expert linguists rather than crowd-sourced or low-paid labor, with no vulnerable populations involved.
- The authors foresee no significant direct harms but recommend scrutiny for downstream dialogue and discourse-analysis applications.
A Annotation Details
The annotation framework classifies look across four pragmatic functions using positional, semantic, speech-act, and argument-structure cues. Two expert annotators applied and reconciled these parameters on naturally occurring BNC examples.
- The framework distinguishes Directive, Attention Signal, Discourse Marker, and Interjection functions.
- Turn and clause position are recorded as initial, medial, or final and provide disambiguating contextual cues.
- Perceptual meaning records whether look is preserved lexically, partially preserved, or pragmatically bleached.
- Speech-act coding separates directive uses, attention prompts, expressive interjections, and cases without those acts.
- Argument structure tracks speaker, hearer, and theme roles, with different valence patterns across functions.
- A primary annotator assigned parameters, while a secondary annotator reviewed disagreements until consensus was reached.
- A naturally occurring BNC excerpt illustrates how turn position, clause position, bleaching, speech act, and argument structure support annotation.
C Supplementary Results and Analysis
The supplementary materials document the annotation scheme, a worked annotation example, prompt templates, constraint violations, and the six training conditions with paired statistical comparisons.
- Annotation and prompts: The annotation scheme specifies parameters and values by function for the four discourse-pragmatic categories.
- Annotation and prompts: A worked example demonstrates how an instance is annotated under the scheme.
- Annotation and prompts: The supplementary analysis reports constraint violations observed during early prompt iterations.
- Annotation and prompts: Separate prompt templates cover discourse marker, interjection, and directive synthetic examples.
- Supplementary results: The results tables describe six training conditions, paired comparisons against the real-only baseline, and function-level F1 alongside overall macro-averaged AUC and accuracy.Only the F1 columns vary within each condition row; AUC and accuracy are repeated overall metrics.