PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthetic Data Characterization via Training Dynamics

Irene Lago, Ana Ezquerro, David Vilares

arXiv:2609.39447v1cs.CLcs.LG

TL;DR

The paper addresses limits in static, feature-based assessments of synthetic data by characterizing sample-level learnability through encoder training dynamics. It compares synthetic and human-written data across tasks, models, and prompting strategies, finding systematic differences across LLM–dataset pairs and different effects of data selection by source.

  • Problem

    Existing synthetic-data assessments use static dataset-level or feature-space comparisons and overlook individual samples’ behavior during training.

    Source: §1 · p. 1

  • Method

    The authors characterize synthetic and human-written datasets using encoder-derived confidence, variability, and correctness distributions, then compare models, prompts, encoders, and difficulty-based selection.

    Source: §1 · p. 1, §3.1 · pp. 3–4

  • Results

    LLM–dataset pairs differ systematically in confidence and variability, prompting has no consistent effect, encoder-based distributions are stable, and selection affects synthetic and human data differently.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Training-dynamics signals distinguish patterns in human and synthetic data and show that data selection affects the two sources differently.

    Source: §6 · p. 9

    Synthetic annotations may miss nuances in subjective tasks; the study instead uses relatively consistent benchmarks and does not aim to replicate human annotation behavior.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.

1 Introduction

Synthetic data is increasingly used to meet demand for training and evaluation data as high-quality human-written text becomes scarce. This work characterizes synthetic data through sample-level learnability, comparing LLMs and prompting strategies with human-written data and testing encoder robustness and difficulty-based selection.

1 Introduction

  • Synthetic data supports pre-training, instruction tuning, and fine-tuning as data demands rise and new high-quality human-written text remains scarce.Source: §1 · p. 1
  • Existing quality and similarity approaches treat datasets as static distributions and overlook individual samples’ downstream behavior during training.Source: §1 · p. 1
  • The paper characterizes synthetic data through sample-level learnability across classification, sequence labeling, and tree prediction, using optimization dynamics and reference benchmarks.Source: §1 · p. 1
  • The analysis compares learnability distributions across LLMs and prompting strategies and against human-written data.Source: §1 · p. 1
  • The study tests whether training dynamics remain stable across encoders and compares difficulty-based selection with baselines.Source: §1 · p. 1

2 Background

The background situates synthetic-data diagnosis between generation research and training-dynamics methods. It reviews concerns about distributional differences and notes that prior work has not generally used training dynamics as a tool for diagnosing synthetic text.

2.1 Synthetic data generation

  • Earlier augmentation methods used lexical perturbations, substitutions, back-translation, and subtree edits, but often produced low-diversity near-duplicates.Source: §2.1 · p. 2
  • LLMs support low-effort synthetic annotation and autonomous data generation across classification, question answering, information extraction, and reasoning tasks.Source: §2.1 · p. 2

2.2 Diagnosis of synthetic data

  • Synthetic text has been associated with bias amplification, linguistic differences, and model collapse, prompting work on diversity, detection, and verification methods.Source: §2.2 · p. 2
  • Data-driven diagnosis has examined similarity to human or reference data and sample replicability, while motivating evaluation beyond surface-level quality toward learning impact.Source: §2.2 · p. 2

2.3 Training dynamics

  • Training-dynamics research uses sample-specific optimization behavior to study efficiency, pruning, pretraining, and fine-tuning; data maps organize classification examples by confidence, variability, and correctness.Source: §2.3 · p. 2
  • Prior synthetic-data work either optimizes training dynamics to match human-data manifolds or studies pretraining quality, rather than using dynamics to diagnose text.Source: §2.3 · p. 2

3 Methodology

The methodology represents individual examples through training-dynamics metrics and compares their empirical distributions. It also generates synthetic data with prompts designed to target different learning signatures, enabling comparisons across generation processes and against human-authored data.

3 Methodology

  • The method adapts sample-level learnability metrics to classification, sequence labeling, and graph prediction tasks.Source: §3 · p. 2

3.1 Data maps for classification and beyond

  • Confidence averages expected accuracy across training, variability is its standard deviation, and correctness measures how often the model predicts the correct output.Source: §3.1 · p. 3
  • The metrics place examples in a three-dimensional space whose regions include easy, hard, and ambiguous samples; ambiguous examples tend to lie nearer the decision boundary.Source: §3.1 · p. 3
  • For multi-label classification, confidence averages individual label contributions and correctness uses averaged Jaccard score; sequence labeling can average token contributions or retain them individually.Source: §3.1 · p. 3
  • For graph prediction, the method factorizes the edge distribution and averages edge-level performance; dependency parsing is a special case that maps edges to tokens and averages correct-head predictions.Source: §3.1 · p. 3
  • The study compares empirical distributions in confidence, variability, and correctness using Wasserstein distance and Cliff’s delta without assuming a parametric distribution.Source: §3.1 · p. 4

3.2 Synthetic Data Generation

  • Synthetic and human datasets are compared under reference and arbitrary model parameterizations, with equal dataset sizes assumed unless stated otherwise.Source: §3.2 · p. 4
  • Prompting strategies range from Naive and Exemplified to Difficulty-aware and Ambiguous-targeted, steering generation toward examples associated with specified learning signatures.Source: §3.2 · p. 4

4 Experimental Framework

The experimental framework spans several NLP task types, LLM families, prompting strategies, and encoder backbones. It evaluates in- and out-of-distribution performance and addresses formatting and structural issues in generated outputs.

4 Experimental Framework

  • The experiments use Qwen3 8B, Llama-3.1 8B, Olmo-3 7B, and Gemma-3 models at 4B, 12B, and 27B scales across four prompting strategies.Source: §4 · p. 4
  • Tasks include SST-5 sentiment classification, SemEval emotion classification, and English EWT part-of-speech tagging and dependency parsing.Source: §4 · pp. 4–5
  • RoBERTa supplies the primary training dynamics, with BERT, DistilBERT, and LLM2Vec used to assess robustness across encoders.Source: §4 · p. 5
  • Structured prediction outputs required post-processing; JSON generation and heuristics were used to preserve structure and produce well-formed trees.Source: §4 · p. 5
  • The authors report similar behavior across Spanish and French multilingual evaluations, with full results in the appendix.Source: §4 · p. 5
  • For dependency parsing, heuristics enforce well-formed outputs: label corrections affect 1.95% ± 1.31 of dependency relations, while 28.36% ± 9.19 of modifications involve structural fixes to ensure valid trees.Source: §4 · p. 5
  • Evaluation uses accuracy for sentiment and PoS, Jaccard score for emotion classification, and LAS for parsing, with in-distribution and external out-of-distribution tests.Source: §4 · p. 5

5 Results and Analysis

The analysis compares training-dynamics maps across tasks, synthetic-data generators and prompts, encoder backbones, and data-selection strategies, using human-written data as a reference. Results vary by task and data source, with synthetic and human data responding differently to learnability-based selection.

5.1.1 Human vs. synthetic data maps

  • SST-5 and EWT maps show an inverted-C confidence–variability pattern, whereas EC concentrates variability at lower levels, likely because multi-label confidence smooths predictions.Unless otherwise specified, the qualitative maps use RoBERTa as encoder and Gemma 27B with the Difficulty-aware prompt for generation.Source: §5 · p. 5, §5.1.1 · p. 6
  • Human and Gemma 27B maps show similar easy, hard, and ambiguous regions for sentiment and PoS, but synthetic EC lacks hard-to-learn examples and parsing lacks high-confidence, low-variability examples.For parsing, the reported distributions remain similar when restricted to uncorrected samples.Source: §5.1.1 · p. 6, Figure 1 · p. 6

5.1.2 Training dynamics across LLMs

  • Prompting strategies produce confidence and variability fluctuations without a clear complexity-related pattern, suggesting that training dynamics are insensitive to the specific prompting strategy.The comparison uses Cliff’s delta relative to the Naive prompt across SST-5, EC, PoS, and dependency parsing.Source: §5.1.2 · p. 7, Figure 3 · p. 7
  • Training-dynamics distributions are more consistent across LLMs than across tasks; Qwen and Gemma models generally produce distributions closest to humans, although LLaMA’s SST-5 distribution is also highly aligned.On SST-5, LLaMA and Qwen yield highly similar distributions (W1 = 0.03); dependency parsing shows substantially larger divergences across models.Source: §5.1.2 · p. 7
  • Most synthetic datasets have positive correctness effect sizes versus human data on SST-5 and EC, whereas synthetic dependency parsing has strongly negative values and fails to reproduce human easy-to-learn regions.Among synthetic distributions, Gemma and Qwen have near-zero correctness effect sizes for PoS tagging.Source: §5.1.2 · p. 7, Figure 4 · p. 7

5.2 Dynamics robustness across encoders

  • Across encoders, sample positions remain broadly stable, but synthetic samples polarize across encoder comparisons, suggesting architecture-specific learnability patterns that may generalize less consistently than human annotations.The study compares RoBERTa with DistilBERT, BERT, and LLM2Vec on SST-5.Source: §5.2 · pp. 7–8, Figure 5 · p. 8

5.3 Training with selected data

  • High-correctness synthetic subsets generally perform more strongly and consistently, especially OOD, while human-data subsets remain informative even when selected for high confidence.Subsets dominated by difficult samples or by easy, high-confidence synthetic samples generally perform more weakly.Source: §5.3 · p. 8, Table 1 · p. 8

6 Conclusion

The paper characterizes LLM-generated data through encoder training dynamics and sample-level learnability. It reports systematic differences across LLM–dataset pairs, robustness across encoders, and different effects of difficulty-based selection for synthetic and human data.

6 Conclusion

  • The paper introduces a framework for characterizing LLM-generated data using training dynamics and learnability metrics from encoder-only models.Source: §6 · pp. 8–9
  • Difficulty-based data selection affects performance differently for synthetic data than for human data.The conclusion does not specify the direction or size of these differences.Source: §6 · p. 9
  • LLM–dataset pairs across families and scales show systematic differences in confidence and variability, while prompting has no consistent effect on these properties.Source: §6 · p. 9
  • The training-dynamics framework is stable across encoders and reveals distinct patterns between human and synthetic data.Source: §6 · p. 9

Limitations

The study reports practical hurdles in generating synthetic datasets and limits its human comparison to controlled, scalable annotation rather than full replication of human labeling. Its findings are tested across prompts and encoders, but residual sensitivity remains possible, and computational costs lead the authors to focus on open-weight models.

Limitations

  • Matching human-dataset size often required generating 3–25× more samples because filtering and lower generation diversity reduced the usable output.The lower-diversity condition is reported for PoS tagging and dependency parsing.Source: Limitations · p. 9
  • Large-scale generation also posed output-format and near-duplication challenges, which the authors addressed with similarity-based filtering.The filtering used Jaccard and BERT-based similarity measures.Source: Limitations · p. 9
  • Synthetic annotations may miss nuances of human labeling, especially on subjective tasks, and the authors instead study controlled, scalable annotation regimes.They use benchmarks with relatively high annotation consistency to reduce label ambiguity, while noting that equivalence is difficult to verify reliably.Source: Limitations · p. 9
  • Trends were consistent across prompt variants and encoder backbones, but residual sensitivity to prompt or model design is not excluded.The evidence supports robustness to moderate variations, not invariance to every configuration.Source: Limitations · p. 9
  • Computational costs and proprietary-model API expenses led the authors to focus on open-weight alternatives.The available infrastructure included local GPUs and queued access to a shared facility.Source: Limitations · p. 9

Ethical Considerations

The authors caution that model-generated annotations may reproduce or amplify model biases and may not align fully with human annotation standards. They recommend treating synthetic labels as approximations rather than ground truth; AI assistants were used to improve the paper’s writing.

Ethical Considerations

A Synthetic Data Generation

The appendix describes synthetic-data generation across four task types, with prompts tailored to dataset formats and four strategies varying in granularity. It also notes that generation parameters vary by model and task.

A Synthetic Data Generation

A.1 Prompt design

  • The generation procedure covers single-label and multi-label classification, token-level classification, and tree prediction, with prompts adapted to original dataset formats.SST-5 uses comma-separated tabular output; EC uses label-conditioned generation; PoS tagging and dependency parsing use JSON fields adapted to CoNLL-U terminology.Source: §A.1 · p. 15

A.2 Prompt strategies

  • Four prompting strategies range from basic task instructions to examples selected for difficulty or explicit instructions to generate ambiguous samples.The strategies are Naive, Exemplified, Difficulty-aware, and Ambiguous-targeted prompts.Source: §A.2 · p. 15

A.3 Generation parameters

  • Generation parameters vary by LLM and task; PoS tagging and dependency parsing require lower sampling values to produce valid UD labels and preserve output format.Table 2 is identified as reporting the selected LLM prompting configurations.Source: §A.3 · p. 15, Table 2 · p. 15

B Results on English datasets

The English results are presented through dataset cartographies and evaluation-set performance, with the main paper focused on SST-5. For dependency parsing, nearly identical maps before and after restricting to originally tree-preserving samples indicate that the training dynamics reflect data characteristics rather than correction artifacts.

B Results on English datasets

  • The framework’s complete results are reported in terms of dataset cartographies and performance on evaluation sets; due to space limitations, the main content of the paper focused on SST-5 (Section 5).Table 4 provides the distributional comparison between prompts using Wasserstein distance (W1) and Cliff’s delta (δ); Table 5 includes results using the full human-authored and synthetic datasets.Source: B Results on English datasets · pp. 15–16
  • Nearly identical dependency-parsing cartographies for corrected data and originally tree-preserving samples indicate that the dynamics reflect data characteristics, not correction artifacts.The comparison uses subsamples of 20,000 instances for visualization.Source: B Results on English datasets · pp. 15–16
  • Figure 6 presents an example from the synthetic SST-5 dataset generated with Gemma 3 27B.Source: Figure 6 · p. 16, B Results on English datasets · p. 16

C Multilingual benchmark

The multilingual benchmark extends the analyses to Spanish and French using translated SST-5 data and language-specific datasets for other tasks. Results report similar behaviors across languages, while prompt differences span a broader range than in English, including a substantial LLaMA multi-label contrast when examples are omitted.

C Multilingual benchmark

  • Spanish and French analyses report similar behaviors across languages, using translated SST-5, Spanish EC, and Spanish GSD and French Sequoia for token-level and tree tasks.Spanish GSD results use a 2,000-instance subset; no French EC version was available.Source: C Multilingual benchmark · pp. 16–17
  • Prompt differences span a broader range in Spanish and French than in English; for LLaMA on multi-label classification, exemplified and Difficulty-aware prompts yield higher confidence and variability than the naive prompt.The associated test results indicate substantial degradation for LLaMA multi-label data when no examples are provided.Source: C Multilingual benchmark · p. 18
  • The multilingual figures compare prompt strategies, human and synthetic distributions, encoder differences, and dataset cartographies for Spanish and French.Source: Figure 12 · p. 20, Figure 15 · p. 20, Figure 13 · p. 20
  • Tables 6 and 7 summarize multilingual benchmark performance, including out-of-distribution evaluation on translated Yelp and language-specific UD PUD datasets.Source: C Multilingual benchmark · p. 18, Table 6 · p. 21, Table 7 · p. 21
  • The Spanish and French cartographies cover all tasks, and the dependency-parsing comparison indicates that corrected and originally tree-preserving samples produce nearly identical maps.The comparison uses 20,000-instance subsamples for visualization.Source: Figure 11 · p. 18, Figure 19 · p. 23, Figure 20 · p. 24

Related papers

Loading 2609.39447v1…