Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

• The approach responds to challenges in ensuring synthetic data fidelity and constructing high-quality examples for multi-step reasoning or structured-data tasks.Source: §1 · p. 1
• The method grounds generation in real-world data sources and conditions each example on a task-specific attribute or seed.Source: §1 · p. 1
• Source2Synth targets table-based question answering and document-based multi-hop question answering, producing synthetic examples with intermediate reasoning steps.Source: §1 · p. 1
• Source2Synth divides generated data into slices, trains an intermediate model on one slice, and uses it to curate the other through imputation and rejection filtering.Source: §1 · pp. 1–2
• The paper contributes a scalable real-source generation approach and a curation method intended to improve data quality and downstream task performance.Source: §1 · pp. 1–2

• Existing synthetic-data methods support pretraining, fine-tuning, and instruction tuning, including approaches that generate tasks from valid dataset fields.Source: §2 · p. 2
• General instruction-tuning recipes yield only a small fraction of multi-hop examples, which can be poor quality.Source: §2 · p. 2
• Source2Synth creates task-specific data for tabular and multi-hop question answering by using real data to select seeds that steer generation.Source: §2 · p. 2
• Unlike some related approaches, Source2Synth does not require back-translation or initial fine-tuning to generate the task-specific seed.Source: §2 · p. 2
• Tool-use research aims to let language models manipulate structured data, but composing SQL relevant to a database schema remains challenging.Source: §2 · p. 2

• Source2Synth supports document-based multi-hop question answering and table-based question answering through a three-stage generation, curation, and fine-tuning pipeline.Source: §3 · p. 2

Multi-hop question answering

• MHQA generation links two questions through a shared entity and produces a merged two-hop question with an associated reasoning chain.Source: §3 · p. 2, §3.1.3 · p. 3

Tabular question answering

• TQA generation prompts an LLM for SQL, executes valid statements with sqlite3, and discards invalid statements before curation.Source: §3 · p. 2, §3.1.3 · p. 3

Data source and seed selection

• Real data sources provide task-specific seeds that anchor examples, including entities from Wikipedia articles for MHQA and table-derived facts for TQA.Source: §3.1.1 · p. 2, §3.1.2 · p. 3

Dataset curation

• Curation fine-tunes LLMSynth on half the data, then improves the remainder through imputation and filtering based on answer agreement after k = 3 tries.Source: §3.2 · p. 3, §3.2.1 · pp. 3–4, §3.2.2 · p. 4
• In MHQA imputation reconstructs discarded question parts and retains examples when the reconstructed question yields the original answer, producing more natural and cohesive questions.Source: §3.2.2 · p. 4
• MHQA curation removes around 13% of generated questions, while TQA retains 27% of the original examples.Source: §3.2.2 · p. 4

4.1 Multi-Hop QA Experimental Setup

• HotpotQA contains 113,000 Wikipedia-based multi-hop question-answer pairs, and the test set has 7,405 bridge and comparison questions with hop length n = 2.Source2Synth generates only bridge questions, so 500 comparison questions from HPQA training data are added to the fine-tuning set.Source: §4.1 · p. 4

4 Experimental Setup

• Synthetic-data contamination checks found no overlap with the HPQA test questions, while WikiSQL train and test tables are mutually exclusive by design.Performance is measured with soft-EM for MHQA and with EM and soft-EM for TQA.Source: §4.1 · p. 4, §4.2 · p. 5

4.1 Multi-Hop QA Experimental Setup

• The MHQA model is Llama2 70B-Chat, trained with either 1,250 Source2Synth examples alone or alongside 500 HPQA examples.The synthetic examples are bridge questions generated from 50 randomly selected Wikipedia articles; LLMCurated-datamix denotes the mixed-data model.Source: §4.1 · p. 4

4 Experimental Setup

• The evaluation compares fine-tuned and synthetic-data baselines with instruction-tuned prompting baselines for both tasks.The listed MHQA baselines include fine-tuning on HPQA, synthetic data without curation, and mixtures of synthetic and HPQA data; TQA includes zero-shot and one-shot prompting variants.Source: §4.1 · pp. 4–5, §4.2 · p. 5

4.1 Multi-Hop QA Experimental Setup

• MHQA synthetic examples link two document-based subquestions through a shared entity and include a reasoning chain, while TQA examples pair table-derived facts with generated SQL and answers.The MHQA process samples related articles around an entity; the TQA process executes generated SQL to obtain the answer.Source: Figure 2 · p. 5, Figure 3 · p. 5

4.2 Tabular QA Experimental Setup

• WikiSQL contains 80,654 hand-annotated question, SQL, and table examples drawn from 24,241 Wikipedia tables, with 7,857 validation examples after filtering.For TQA, the study uses Starchat-beta, an instruction-tuned 16-billion-parameter coding assistant, and generates 10,000 SQL statements while retaining 8,000 examples per slice.Source: §4.2 · p. 5

Data composition and curation

• Combining HPQA examples with synthetic data outperforms using either source alone, while curation adds further gains over uncurated synthetic data.The data-mix comparison uses 500 HPQA examples and synthetic examples; the full Source2Synth pipeline improves over LLMSynth variants.Source: §5.1 · p. 6

Scaling performance

• As synthetic data increases, uncurated and curated models generally improve, with curated models consistently outperforming uncurated versions across data-mix sizes.The scaling study varies synthetic examples while retaining 500 HPQA examples; curation removes 7%, 8%, and 11% of samples at 500, 750, and 1250 examples.Source: §5.1 · pp. 6–7, Figure 5 · p. 6

Multi-hop question answering

• Source2Synth performs better across hard, medium, and easy bridge and comparison question types.The analysis compares the base model, HPQA-only fine-tuning, and Source2Synth fine-tuning by question difficulty.Source: §5.1 · p. 7
• +13.38% average absolute performance gain over the base LLM and +7.35% over HPQA fine-tuning are reported for multi-hop question answering.Gains span question types and difficulties, including +16.8% on hard bridge and +16.5% on comparison questions.Source: §5.1 · p. 7

Model size and data grounding

• Source2Synth remains effective for smaller LLMs, increasing accuracy by 23.06% for Llama3 8B-instruct and 36.90% for Llama4 17Bx16E.The smaller models are fine-tuned on 1250 synthetic examples generated with Llama2 70B-Chat.Source: §5.1 · p. 7
• Using ungrounded synthetic data causes an average accuracy loss of -7% for Llama3 8B-instruct and Llama2 70B-Chat.This comparison isolates the contribution of grounding synthetic data in real-world sources.Source: §5.1 · p. 8

Tabular question answering

• For TQA, enabling SQL use raises EM to 12.3%, and Source2Synth fine-tuning improves over uncurated fine-tuning and all other baselines.The WikiSQL evaluation reports very low performance without table context or tool use, while the curated model is strongest among the compared approaches.Source: Table 2 · p. 7, §5.2 · p. 8

Main results

• Source2Synth improves performance across multi-hop and tabular question answering, with curated models outperforming uncurated counterparts and other baselines.For TQA, the curated model outperforms uncurated fine-tuning; for MHQA, the full pipeline improves performance over uncurated data.Source: §5.1 · pp. 6–7, §5.2 · p. 8

Contribution

• Source2Synth generates high-quality synthetic data for table question answering and multi-hop question answering.The method grounds generation in real data sources and filters low-quality samples.Source: §6 · p. 8

Applications

• Source2Synth can transform unstructured sources such as legal and medical text into QA data without human annotators.The paper frames this as a potential use case where source documents are available but question-answer pairs are not.Source: §6 · p. 8

Conclusion

• Fine-tuning on Source2Synth examples consistently yields substantial improvements while using less data.The conclusion presents this as evidence for automatic data generation without human input.Source: §6 · p. 8

Two-hop MHQA

• MHQA is restricted to two hops, although the generation procedure can be looped to support more hops.Each subsequent generation step can consume the previous step's result.Source: A Limitations • · p. 13

Tabular scope

• TQA assumes one table per query and does not support multi-table aggregation such as SQL joins.Table retrieval is proposed for identifying relevant tables, but multi-table tool use remains unsupported.Source: A Limitations • · p. 13

Curation

• MHQA dataset construction relies on rejection sampling, with more sophisticated multihop retrieval suggested as a possible improvement.The paper specifically mentions the multihop retrieval approach of Xiong et al. (2020).Source: A Limitations • · p. 13

Task scope

• The study covers question answering with document and table sources, while extending Source2Synth to other tasks and skills remains future work.The authors note potential application to domains such as legal and medical data when suitable source types exist.Source: A Limitations • · p. 13

Data assumptions

• Source data is assumed to undergo initial consistency checks, although filtering may mitigate some inconsistencies through answerability checks.For TQA, nonexecutable SQL statements are discarded.Source: A Limitations • · p. 13

Performance by difficulty and type

• The analysis evaluates hard, medium, and easy bridge and comparison questions using the LLM-Curated-1250 model.Table 4 subdivides results by difficulty and question type.Source: §C.1 · p. 13
• Source2Synth outperforms both the base model and HotpotQA-fine-tuned model on hard questions of both bridge and comparison types.The comparison uses the full HotpotQA training set, labelled by difficulty and question type.Source: §C.1 · p. 13, Table 4 · p. 14

Experimental setup

• The Source2Synth model is trained with 1250 synthetic examples for this difficulty- and question-type analysis.The table caption states that these examples yield a model capable of handling hard questions of both types.Source: Table 4 · p. 14

Cross-model fine-tuning

• Source2Synth improves performance for both Llama3 8B-instruct and Llama4 17B×16E relative to their corresponding base LLMs.Llama2 70B-Chat generates the synthetic data for both target-model experiments.Source: §C.2 · p. 14

Evaluation setup

• Evaluation uses a zero-shot prompt and soft-EM, with Llama2 70B-Chat serving as the dataset-generation model.The target models differ from the model used to generate the synthetic training data.Source: §C.2 · p. 14

Ungrounded data results

• Ungrounded synthetic data reduces accuracy by 7.17% for Llama3 8B-instruct and 6.82% for Llama2 70B-Chat versus grounded data.The ungrounded questions also show repeated structural patterns that the authors hypothesize hinder generalization.Source: §C.3 · p. 14, Table 7 · p. 15, Table 8 · p. 15
• Ungrounded two-hop questions have lower perplexity than grounded questions, but the model trained on them performs worse.The ungrounded questions are bridge-type and can follow repetitive templates.Source: §C.3 · p. 14

Imputation

• Imputation lowers perplexity and reduces question unnaturalness for both grounded and ungrounded synthetic data.Figure 6 illustrates rephrasing toward more natural wording, while Table 9 reports average perplexity.Source: §C.4 · p. 15, Table 9 · p. 15, Figure 6 · p. 16

TQA construction

• For TQA, 800 seed SQL statements generated from 50 tables yield 658 executable statements after sqlite3 filtering.Nonexecutable SQL statements are discarded before retaining the synthetic examples.Source: D SQL non-executable code filtering · p. 15

Prompt configurations

• The appendix provides zero-shot, one-shot, and table-plus-SQL prompts for TQA, alongside multi-shot chain-of-thought and question-generation prompts for MHQA.These prompts cover seed induction, SQL generation, question generation, question merging, and evaluation.Source: Figure 7 · p. 16, Figure 8 · p. 16, Figure 9 · p. 16