Better Synthetic Data by Retrieving and Transforming Existing Datasets

1 Introduction

• Task-specific annotated data is often scarce; manual curation requires domain experts and resources, while synthetic datasets can struggle to achieve both high correctness and sufficient diversity.Models trained on synthetic data are typically worse than those trained on manually curated data when dataset size is standardized.Source: §1 · pp. 1–2
• DataTune retrieves public datasets and uses an LLM to transform them into target-aligned data, preserving source diversity while synthesizing task-relevant outputs.For code description, it uses code solutions to generate English descriptions; the introduction reports a 22-point improvement over baselines.Source: §1 · p. 2
• DataTune-enhanced models improved by an average of 5.2 points on five tasks over few-shot prompting, and combining DataTune with synthetic generation yielded an 8-point improvement over that baseline.The evaluation used Mistral-7B without fine-tuning for the few-shot baseline.Source: §1 · p. 2
• DataTune can complement synthetic data generation, and its generated examples were often more difficult and diverse without sacrificing correctness in a small-sample check.The correctness comparison was based on a small sample of data.Source: §1 · p. 2

2 Problem Setup

• Given a task prompt, a collection of labeled datasets, a promptable large language model, and a fine-tunable small model, the goal is to create task-aligned training data.The task prompt may include a few demonstration examples.Source: §2 · p. 2
• Progress is measured by evaluating the fine-tuned model against labeled data for the specified task.The generated dataset is intended to improve the model’s ability to satisfy the task specification.Source: §2 · p. 2

3 Methods

• DataTune selects and reranks existing datasets, then transforms entries to create a synthetic dataset aligned with the target task.Transformation can adjust input-output formats or generate fields absent from the source dataset.Source: §3 · p. 2, §3.2 · p. 3

3.1 Dataset Retrieval

• Retrieval first uses a bi-encoder to find candidates, then an LLM reranks them using dataset descriptions, schemas, and sample rows.Self-consistency decoding selects the most frequently returned dataset; the method also recognizes that some tasks may have no suitable dataset.Source: §3.1 · p. 3

3.2 Dataset Transformation

• A task-level plan specifies how to adapt source examples, and an LLM executes it on each row to produce the transformed dataset.Schema selection removes irrelevant columns and task expansion enriches the task description to support more detailed plans.Source: §3.2 · pp. 4–5

3.3 Using Multiple Datasets

• When filtering or small source datasets reduce output quantity, DataTune transforms multiple highly ranked datasets until it reaches the desired dataset size.A dataset is excluded if a considerable proportion of its transformations fail.Source: §3.3 · p. 5

4 Experimental Setup

• The evaluation covers six BIGBench tasks spanning logical reasoning, coding, math, causal reasoning, multilinguality, and domain-specific tasks, with 3,000 DataTune examples created per task.All models were evaluated for two-shot performance using the BIGBench testing suite.Source: §4 · p. 5
• Dataset strategies include retrieved data, synthetic data, Prompt2Model, DataTune, and DataTune combined with synthetic data, all used to fine-tune Mistral-7B.Few-shot Mistral-7B prompting and GPT-3.5-Turbo provide prompting baselines.Source: §4 · p. 5
• DataFinder retrieves 25 candidate datasets, up to four are transformed per task, and GPT-4-Turbo powers components except final execution, which uses GPT-3.5-Turbo.The final execution model choice was made for budgetary reasons because transformation requires an LLM query per instance.Source: §4 · p. 5
• The normalized aggregate score maps random-chance performance to 0 and human-expert performance to 100, with scores below 0 indicating worse-than-chance performance.Fine-tuning used Mistral-7B with QLoRA for three epochs.Source: §4 · pp. 5–6

5.2 DataTune Impact on Dataset Diversity

• On three of five tasks, DataTune effectively eliminates near-duplicates found in more than 50% of synthetically generated examples.The analysis defines an example as unique when its maximum ROUGE-L similarity to another example falls below a task-specific threshold.Source: §5.2 · p. 6, Figure 4 · p. 6
• DataTune increases unique bigrams per input example on three of five datasets, with diversity decreasing on Temporal Sequences and Medical Questions in Russian.Those two tasks are also where DataTune fails to improve over fully synthetic training.Source: §5.2 · p. 6, Table 3 · p. 7

5.1 Performance Comparison

• 8.3 points: DataTune plus synthetic generation improves over Prompt2Model on average and leads on five of six tasks.The comparison defines the combined system as Mistral-7B fine-tuned on both DataTune-created and synthetically generated data.Source: §5.1 · p. 6, Table 2 · p. 7

5.3 DataTune Generates Harder Examples

• DataTune examples are more difficult than synthetic examples on all evaluated datasets except Medical Questions in Russian.The authors attribute the exception to a transformation plan that generated data in the wrong language; difficulty was estimated with a task-specific LLM prompt on a 1-to-5 scale.Source: §5.3 · pp. 7–8, Figure 5 · p. 7

5.4 Does generating harder examples lead to lower label accuracy?

• For Code Line Description, manually annotated label accuracy was 88% for DataTune and 86.6% for synthetic generation.The comparison manually annotated a random sample of 300 data points from each method.Source: §5.4 · p. 8

5.5 Transformed Data Can Be Complementary to Synthetic Data

• Combining DataTune with synthetic generation outperforms other comparable Mistral-based baselines on Temporal Sequences and Elementary Math, where DataTune alone fails to beat synthetic data.The authors report that transformed and synthetic examples occupy largely disjoint embedding clusters for Elementary Math, supporting their hypothesis that the methods cover different target-task distributions.Source: §5.5 · p. 8, Figure 6 · p. 8

• DataTune significantly outperforms existing automatic dataset-generation techniques on several challenging BIG-Bench tasks.Source: §6 · p. 8
• Combined with existing synthetic-data methods, DataTune achieves superior performance to other few-shot learning methods on Mistral-7B.Source: §6 · p. 8
• DataTune’s transformed datasets are more diverse and accurate and increase the complexity of fine-tuning data.Source: §6 · p. 8
• Future work could test retrieving examples from the open web and generate code to execute transformation plans instead of querying an LLM for each instance.The authors identify these directions as ways to improve accessibility and scalability.Source: §6 · p. 8

• The authors identify four key limitations in the system.Source: Limitations · p. 8

1. LLM Query Cost: Our dataset transforma-

• Transforming data requires an LLM query for every row, so costs scale linearly with dataset size and may be prohibitive for very large datasets.The LLM requirement may also exclude researchers without consistent access.Source: §1 · p. 9
• The system depends on a planning LLM following prompts correctly; in the Russian medical-question task, the plan failed to translate generated pairs back to Russian.The resulting dataset was conceptually close but practically far from correct.Source: §1 · p. 9
• The transformation agent’s processing of non-English tasks is substantially compromised and may alter task examples instead of the data.The authors primarily attribute this deficiency to models such as GPT-3.5 being extensively trained on English data.Source: §1 · p. 9
• The execution component requires instruction-following LLMs, limiting the system to a narrow selection of models that follow instructions accurately.The authors report performance discrepancies between instruction-tailored and conversational LLMs.Source: §1 · p. 9

Ethical Considerations

• The authors acknowledge potential misuse and ethical concerns, but assess the risks as likely no greater than those of the underlying open-source LLMs.Source: Ethical Considerations · p. 9
• They hope the risks are balanced by making NLP models accessible to people who lack resources to collect labeled data manually.Source: Ethical Considerations · p. 9
• The platform may help people outside NLP use models in their work despite lacking specialized model-development knowledge.Open-sourcing DataTune also invites community contributions to improve and expand the tool.Source: Ethical Considerations · p. 9

A.1 Reranking Prompt

• The reranking prompt asks the model to select the dataset most relevant to the task, using task examples and dataset descriptions and tags.Source: §A.1 · p. 12

A.2 Schema Selection Prompt

• The schema-selection prompt classifies dataset columns as relevant input, relevant output, irrelevant, or ambiguous, allowing at most one output column.Source: §A.2 · p. 12

A.3 Planning Module Prompt

• The planning prompt requests concise sequential steps to convert dataset samples into the target task format, including generating, combining, selecting, or ignoring columns and samples.Source: §A.3 · p. 13

A.4 Execution Module Prompt

• The execution prompt directs the model to apply the transformation plan to a sample and return JSON with exactly two fields: input and output.Source: §A.4 · p. 13

A.5 Task Expansion Prompt

• The task-expansion prompt asks for a clearer general description of the task, emphasizing trends and the format of its input/output examples.Source: §A.5 · p. 14

A.6 Difficulty estimation prompt: Code Line Descriptions example

• The code-description difficulty prompt rates how hard code is to describe on a 1-to-5 scale and requires a single-number response.Source: §A.6 · p. 14