PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction

Martin Josifoski, Marija Sakota, Maxime Peyrard, Robert West

arXiv:2303.04132v2cs.CLcs.AIcs.LG

TL;DR

Closed information extraction lacks large, clean datasets, while LLMs may struggle to generate its structured outputs. The paper reverses the task by sampling triplet structures and generating text, then uses 1.8M synthetic examples to train SynthIE models that outperform equal-size prior state of the art by 57 micro-F1 and 79 macro-F1 points.

  • Problem

    Closed information extraction data is expensive to collect, leaving datasets small or noisy, and LLMs may struggle to generate the task’s structured outputs.

    Source: §1 · p. 1

  • Method

    The pipeline samples coherent triplet sets from Wikidata and prompts an LLM to generate text expressing those triplets and no others.

    Source: §1 · p. 2

  • Results

    On Wiki-cIE Text, SynthIE beats the equal-size prior state of the art by 57 absolute points in micro-F1 and 79 points in macro-F1; human evaluation also finds the synthetic data higher quality than REBEL.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach makes a task not directly solvable by LLMs feasible for a small 220M-parameter model, and the authors offer the data, code, and models for reuse.

    Source: §1 · p. 2

    The SDG setup uses code-davinci-002, whose API is not public, though access may be possible through OpenAI’s Researcher Access Program.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) have great potential for synthetic data generation. This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in task difficulty makes it possible to produce large-scale, high-quality data for complex tasks. We demonstrate the effectiveness of this approach on closed information extraction, where collecting ground-truth data is challenging, and no satisfactory dataset exists to date. We synthetically generate a dataset of 1.8M data points, establish its superior quality compared to existing datasets in a human evaluation, and use it to finetune small models (220M and 770M parameters), termed SynthIE, that outperform the prior state of the art (with equal model size) by a substantial margin of 57 absolute points in micro-F1 and 79 points in macro-F1. Code, data, and models are available at https://github.com/epfl-dlab/SynthIE.

1 Introduction

The introduction frames closed information extraction as a data- and format-challenged task, then proposes generating text from sampled output structures and evaluates the resulting data and models.

1 Introduction

  • For structured-output tasks, LLMs may generate data poorly because pretraining does not equip them to produce the required output format, so the paper reverses the task and generates text from sampled structures.The authors propose that suitable in-context information can let an LLM generate meaningful text even when it cannot directly solve the original task.Source: §1 · p. 1
  • In cIE, the model extracts disambiguated subject-relation-object triplets from text, with entities and relations defined by a knowledge-base schema.The paper identifies Wikidata as an example knowledge base.Source: §1 · pp. 1–2
  • Collecting cIE data is expensive because annotation requires catalog-wide knowledge and reasoning, leaving existing datasets small or noisy, with REBEL also suffering omissions, incorrect facts, and relation skew.The skew can lead models to ignore much of the information when trained on the data.Source: §1 · p. 2
  • The pipeline samples coherent Wikidata triplet sets with more uniform relation coverage than REBEL, then prompts an LLM to generate text expressing only those triplets, yielding 1.8M data points.The dataset is used to replace noisy REBEL data.Source: §1 · p. 2
  • 70% of REBEL test text facts are absent from targets and 45% of target triplets are unexpressed, versus 15% and 22% on the highest-quality Wiki-cIE Text test set.Human evaluation finds the synthetic data significantly higher quality than REBEL and more uniformly distributed across relations.Source: §1 · p. 2
  • By 57 absolute points in micro-F1 and 79 points in macro-F1, SynthIE outperforms the prior state of the art with equal model size on Wiki-cIE Text.The models are finetuned Flan-T5 models.Source: §1 · p. 2

2 Background and Related Work

Prior synthetic-data work often uses pretrained language models to augment existing datasets or generate examples for conventional NLP tasks. This paper applies reverse-direction generation to closed information extraction, where skewed data and costly annotation constrain model performance.

2.1 Synthetic Data Generation

  • Unlike prior methods that generate labels or augment conventional tasks, the proposed approach prompts an LLM to generate input text for a structured output when that reverse task is easier.Earlier approaches include dataset-dependent augmentation and prompt-based generation for tasks such as classification, question answering, and natural language inference.Source: §2.1 · pp. 2–3

2.2 Closed Information Extraction

  • Closed information extraction (cIE) extracts exhaustive facts expressible under a knowledge base’s entity and relation constraints; GenIE’s macro-F1 remained below 35% despite outperforming its strongest baseline by more than 3x.GenIE’s authors identify the heavily skewed relation distribution as the main reason for its low macro score.Source: §2.2 · p. 3

2.3 Data as a Core Limitation

  • Relation-frequency imbalance leaves models effective mainly on frequent relations, while rare-relation annotations are difficult to obtain; synthetic data can broaden coverage beyond cIE to other structured tasks.The paper names entity linking, open information extraction, and abstract meaning representation parsing as possible applications.Source: §2.3 · p. 3
  • The generation flow filters Wikidata to a REBEL-comparable subset, samples coherent triplet sets for more uniform relation coverage, and prompts an LLM to generate text for each set.The proposed pipeline targets data scarcity by generating text from sampled output structures.Source: Figure 2 · p. 3, §2.3 · p. 3

3 Exploiting Asymmetry for Synthetic Data Generation

The paper builds synthetic cIE data by sampling coherent, broadly covered Wikidata triplet sets and generating text that expresses them. Human evaluation finds the generated text more precise and complete than REBEL’s, while the Code dataset has more balanced relation frequencies.

3 Exploiting Asymmetry for Synthetic Data Generation

  • The pipeline constructs a knowledge graph, samples coherent triplet sets with broad entity and relation coverage, then generates text expressing those triplets without adding others.The paper presents these as the pipeline’s three primary components.Source: §3 · p. 3

3.1 Knowledge Graph Construction

  • The Wikidata graph is filtered to 2.7M entities and 888 relations that appear at least once in REBEL’s training set.The filtering keeps the graph comparable to previous work.Source: §3.1 · p. 3

3.2 Sampling Triplet Sets

  • A random walk with backtracking favors triplets sharing entities, while reweighting and alternating entity- and relation-centric starting strategies promote coverage of rare entities and relations.The mixed strategy switches every K samples, when empirical entity and relation distributions are recomputed.Source: §3.2 · p. 4

3.3 Triplet-Set-to-Text Generation

  • The generation setup varies OpenAI model, prompting strategy, and sampling parameters; Wiki-cIE Code contains around 1.8M training samples, while Wiki-cIE Text uses the same triplet sets with another model.The two datasets also include 10K validation and 50K test samples each.Source: §3.3 · pp. 4–5

3.5 Human Evaluation

  • 84.8% precision and 78.5% micro-recall for Wiki-cIE Text exceed REBEL text’s 29.4% precision and 56% micro-recall; Wiki-cIE scores are also more consistent across relations.The evaluation compares human-annotated triplets expressed in each text with the corresponding target set.Source: Table 2 · p. 5, §3.5 · p. 5

3.4 Distributional Properties of Data

  • The rarest Wiki-cIE Code relations occur more often than REBEL’s median relation, and its training split contains 1,805,504 unique entities versus REBEL’s 1,715,922.The paper describes Wiki-cIE Code’s relation distribution as much more balanced than REBEL’s.Source: §3.4 · p. 5

4 Synthetic Data in Action

SynthIE trains on synthetic data generated for closed information extraction and performs strongly across relations, while REBEL’s annotations remain an unreliable evaluation reference. The section describes the model and evaluation setup, then reports results on REBEL Clean and the Wiki-cIE datasets.

4.1 Modeling and Inference

  • SynthIE autoregressively generates a linearized sequence of the exhaustive facts expressed in an input text, trained with conditional log-likelihood and regularization.The model uses FLAN-T5; output sets can be linearized with fully expanded or subject-collapsed formats.Source: §4.1 · pp. 5–6
  • At inference, constrained beam search combines schema-level prefix constraints with entity- and relation-ID validity constraints from a precomputed trie.Source: §4.1 · p. 6

4.2 Experimental Setup

  • The experiments compare synthetic and non-synthetic training while holding architecture constant, using Wiki-cIE Text for testing and reporting micro- and macro-precision, recall, and F1.The entity and relation catalogs exclude labels with unknown tokens, leaving around 2.6M entities and 888 relations.Source: §4.2 · p. 6

4.3.1 Human Evaluation on REBEL

  • 73.8 micro-F1 and 43.76 macro-F1: REBEL Gold achieves these scores against hand annotations, while SynthIE T5-large exceeds REBEL Gold in macro performance.The large micro–macro gap indicates that original annotation quality varies greatly across relations.Source: §4.3.1 · pp. 6–7
  • REBEL Clean retains high relation imbalance and texts containing entities that cannot be resolved; Wiki-cIE Text is designed to avoid these issues.Source: §4.3.1 · p. 7

4.3.2 Performance Evaluation

  • 91.4 micro and 93.1 macro-F1: SynthIE T5-base achieves these scores on Wiki-cIE Text, while T5-large improves performance by 2–4 absolute points on both datasets.Table 3 reports the main results across REBEL Clean, Wiki-cIE Code, and Wiki-cIE Text.Source: §4.3.2 · p. 7, Table 3 · p. 7

4.3.3 Performance by Relation Frequency

  • On Wiki-cIE Text, SynthIE performs well across relation-frequency buckets at around 93% F1, while GenIE is near zero on 46% of relations and peaks around 58% on the most frequent ones.The figure buckets relations by training-set frequency and plots per-bucket F1 with confidence intervals.Source: Figure 3 · p. 8, §4.3.3 · p. 8

5 Discussion

The discussion presents Wiki-cIE as a better evaluation resource than REBEL and reports that SynthIE performs across relations. It argues that exploiting task asymmetry may apply beyond closed information extraction.

5 Discussion

  • Wiki-cIE Text enables substantially more accurate cIE evaluation than REBEL, which suffers from false positives and false negatives.Source: §5 · p. 8
  • Wiki-cIE Code supports performance across relations; SynthIE’s macro predictions exceed REBEL’s gold annotations and reach 93% macro-F1 on Wiki-cIE Text, versus GenIE’s 14%.Source: §5 · p. 8
  • The authors argue asymmetry-based synthetic data generation may extend to other information-extraction and parsing tasks, and to problems with a more tractable inverse formulation.Source: §5 · pp. 8–9

Limitations

The section describes limits on exhaustive cIE and LLM availability alongside supplementary details on model failures and synthetic-data generation. The generation process samples from a filtered Wikidata graph and prompts an LLM to produce text for triplet sets.

Limitations

  • Exhaustive cIE performs well when all text information can be linked to the KB, but missing information or limited model expressivity can prevent similar performance.The authors note that future work must decide how important exhaustiveness is for cIE.Source: Limitations · p. 9
  • Relying on an LLM to generate training data opens the possibility that its biases enter the synthetic data.Source: Limitations · p. 9
  • The text-davinci-003 API is publicly available, whereas code-davinci-002 is not, though access to the latter is possible through OpenAI’s Researcher Access Program.Source: Limitations · p. 9

A LLM cIE failure cases

  • Text-davinci-003 can often identify core information but may fail to map names to the corresponding Wikidata entities or relations.The figure caption also notes failures to recognize all facts in a sentence.Source: A LLM cIE failure cases · p. 12, Figure 4 · p. 13

B Synthetic Data Generation

  • The supplementary section describes sampling and synthetic text generation, with example prompts and generated sentences presented in a figure and table.Source: B Synthetic Data Generation · p. 12

B.1 Details about the Knowledge Graph

  • The filtered Wikidata graph contains 2,715,483 entities and 888 relations, and is subsampled to remain comparable to prior research.The authors state the approach can also be applied to larger graph subsets or the full KG.Source: §B.1 · p. 12

B.2 Triplet Sampling

  • Triplet sets use a Poisson-distributed size with mean 3 and a bias factor of 7; the starting-point strategy uses d = 0.01 and K = 20K.The bias factor 7 was selected over 10 for greater diversity; d and K were chosen as a compromise across sampling diagnostics.Source: §B.2 · p. 12

B.3 Triplet Set to Text

  • The prompt examples compare the two LLMs, with code-davinci-002 using demonstrations; their training and instruction-tuning differ, and Wiki-cIE Text cost $223.55 to construct.The caption identifies green text as model output.Source: §B.3 · p. 12, Figure 5 · p. 13

B.4 Distributional Properties of the Data

This section describes relation-frequency distribution plots and the procedures used to compute performance metrics and annotate data. It outlines crowdworker tasks, quality checks, and a separate annotation procedure for estimating triplets expressed in text.

B.4 Distributional Properties of the Data

  • Figure 6 presents cumulative distributions of relation frequencies for REBEL, Wiki-cIE Code, and Wiki-cIE Text to compare their distributions.Source: §B.4 · p. 14

B.5.1 Computing Performance Metrics

  • Precision and recall estimation requires counts of target triplets, correctly predicted triplets, and triplets expressed in the text.Target counts come from the REBEL target sets; MTurk annotations estimate correctly predicted triplets, while the text-triplet count supports micro-precision estimation.Source: §B.5.1 · p. 14

B.5.2 Human Annotation Task

  • MTurk workers judged which candidate triplets appeared in each sentence, using instructions and examples to guide their decisions.Source: §B.5.2 · p. 14, Figure 7 · p. 15

B.5.3 Quality of annotations

  • Quality checks included experienced workers, honeypots, and three-worker majority votes; Fleiss’ kappa was 0.2239.Workers were recruited with at least 1,000 completed tasks and a 99% acceptance rate; each task included one or two honeypots.Source: §B.5.3 · pp. 14–15

B.5.4 Precision Annotation Procedure

  • For precision annotation, one annotator added triplets expressed outside the original set, then two others independently checked them, with conflicts resolved by discussion.Annotators were blinded to whether pairs came from REBEL or a synthetic dataset; six data points were excluded due to problems with original data.Source: §B.5.4 · p. 15

C Datasets

The section describes REBEL’s construction and its processing for closed information extraction, including entity linking, filtering, and the use of a processed version for training and testing.

C Datasets

  • REBEL was created from Wikipedia abstracts by aligning sentences, hyperlinks, and corresponding Wikidata entities and relations.Source: C Datasets · p. 15
  • For this work, REBEL was adapted by linking entities to Wikipedia pages and filtering entities without an associated unique name; the processed dataset was used for training and testing.Source: C Datasets · p. 15
  • Its alignment pipeline includes mention detection, coreference resolution, entity disambiguation, and mapping triplets to sentences.The pipeline expands on Elsahar et al. (2018), and REBEL further filtered false positives with a natural language inference model.Source: C Datasets · p. 15

D Performance Metrics

The section defines triplet-level precision, recall, and F1 and motivates reporting macro scores alongside micro scores for imbalanced data. It also identifies information in accompanying dataset and sample tables.

D Performance Metrics

  • A predicted fact counts as correct only when its relation and both corresponding entities are correct.The notation distinguishes each document’s predicted triplets from its gold triplets.Source: D Performance Metrics · p. 15
  • Micro scores weight samples equally, whereas macro scores weight classes equally, making macro scores informative when entities or relations are imbalanced.Source: D Performance Metrics · p. 15
  • Table 5 compares REBEL text with two synthetic generations and describes the synthetic samples as more fluent and better in recall, especially precision.Table 6 reports dataset statistics and specifies filtering for out-of-knowledge-base triplets and sequences longer than 256 tokens.Source: Table 5 · p. 16, Table 6 · p. 17

E Experiment Implementation Details

The experiments reused REBEL-tuned hyperparameters for SynthIE and ran training and inference on specified GPU setups.

Training

  • Hyperparameters tuned on REBEL were reused to train SynthIE because tuning all models was costly.The authors expected the advantage of better-optimized parameters for the baselines to be insignificant.Source: E Experiment Implementation Details · p. 17
  • Training used Adam at 3e-4 for 8,000 steps, with batch size 2,568, gradient clipping, weight decay, and a polynomial learning-rate schedule.The schedule used 1,000 warm-up steps and a final learning rate of 3e-05.Source: E Experiment Implementation Details · p. 17

Inference

  • Inference used 10-beam Constrained Beam Search, with length penalties of 0.8 for fully expanded and 0.6 for subject-collapsed outputs.The other inference parameters followed their default values.Source: E Experiment Implementation Details · p. 17

Infrastructure and compute time

F Output Linearization

The paper compares fully expanded and subject-collapsed output linearizations for representing triplet sets. Subject collapse shortens sequences, especially on the more exhaustive Wiki-cIE datasets.

F Output Linearization

  • Subject-collapsed linearization produces shorter output sequences, especially for the more exhaustive Wiki-cIE Code and Wiki-cIE Text triplet sets.For REBEL, where many target triplets are missing, the sequence-length difference is not substantial.Source: Figure 8 · p. 18, F Output Linearization · p. 18
  • Fully expanded linearization marks each triplet’s subject, relation, and object separately, which can repeat entities when several facts share a subject.It is the simpler representation and was used by GenIE.Source: F Output Linearization · p. 18
  • Subject-collapsed linearization groups triplets by subject, stating each subject once before listing its relations and objects.This shortens targets at the cost of more complex dependencies between output parts; Huguet Cabot and Navigli (2021) used this mapping.Source: F Output Linearization · p. 18
  • Training orders triplets by subject appearance in the text, breaking ties by object appearance, to align sequence order with the unordered target set.When entities cannot be linked to mentions, a heuristic uses the largest overlapping text span in the entity name, or the sentence beginning if there is no overlap.Source: F Output Linearization · p. 18

G.1 Human Evaluation of REBEL

The REBEL Clean evaluation uses human-reviewed candidate triplets to assess annotations and model predictions. Its precision estimates are designed to be accurate for selected models, while recall estimates depend on candidate coverage and, for macro-recall, an across-relations assumption.

G.1.1 Constructing REBEL Clean

  • The REBEL Clean sample excluded texts with little extractable information or literal central information, which GenIE and SynthIE do not support in this setting.The annotation used two Ph.D. and two MSc students, with a second Ph.D. student resolving disagreements.Source: §G.1.1 · pp. 18–19
  • Annotators judged a high-recall candidate set rather than exhaustively searching catalogs of about 2.6 million entities and 888 relations; triplets outside that set could be missed.Candidates combined REBEL targets with SynthIE T5-large predictions, and the set was filtered by human annotators.Source: §G.1.1 · p. 19

G.1.2 Precision on REBEL Clean

  • Because REBEL Clean candidates include REBELGold and SynthIE T5-large predictions, their estimated precision matches true precision up to human error.The authors expect good micro- and macro-precision estimates for GenIE and SynthIE T5-base as well, given their training and model-size relationships to those candidates.Source: §G.1.2 · p. 20

G.1.3 Recall on REBEL Clean

  • Recall estimates depend on candidate-set coverage; interpreting macro-recall additionally assumes REBEL Clean’s gold-annotation recall is approximately equal across relations.The measured recall is an upper bound with a constant multiplicative gap to true recall, determined by potentially missing triplets.Source: §G.1.1 · p. 19, §G.1.3 · p. 20

Related papers

Loading 2303.04132v2…