PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffIE: Diffusion-based Open Information Extraction

Konstantin Fedorov, Valentin Malykh

arXiv:2609.02315v1cs.CLcs.AI

TL;DR

OpenIE must produce multiple valid triplets from a sentence, but existing autoregressive and fixed-slot approaches trade flexibility against speed or train-time extraction limits. DIFFIE uses stochastic conditional discrete diffusion to sample and cluster candidate triplets, making both compute and output budgets adjustable at inference. Across four benchmarks, it achieves the strongest reported results on stricter CaRB (1-1) and BenchIE measures while remaining competitive elsewhere.

  • Problem

    OpenIE requires multiple valid triplets per sentence, while autoregressive systems are slow and redundant and fixed-slot systems tie extraction capacity to training.

  • Method

    DIFFIE samples independent reverse-diffusion trajectories over per-token role tags, then leniently clusters and ranks candidate triplets.

  • Results

    Across four benchmarks, DIFFIE achieves the best reported CaRB (1-1) F1 and AUC, beats the strongest rule-based system on BenchIE, and remains competitive on standard CaRB and WiRe57.

  • Takeaways & Limitations

    Diffusion stochasticity provides an effective mechanism for structured prediction with multiple valid outputs, with test-time compute and extraction count adjustable without retraining.

  • Takeaways & Limitations

    The four-tag scheme permits one role per token in each trajectory, so overlapping roles and discontinuous arguments cannot be represented within a single sample.

Abstract

from arXiv · show

A single sentence often expresses multiple valid relational triplets, which makes Open Information Extraction (OpenIE) fundamentally a multi-output task. Existing neural systems handle this by autoregressive generation, which is flexible but slow and prone to redundancy, or by fixed-slot prediction, which is efficient but couples the extraction budget to training. We introduce DIFFIE which instead treats the stochasticity of conditional discrete diffusion as the extraction mechanism itself: independent reverse-diffusion trajectories over per-token role tags produce a pool of candidate triplets, which are clustered under lenient matching and ranked to form the output. Both the pool size and the number of returned extractions are inference-time choices, decoupling the extraction budget from training and exposing test-time compute as a tunable axis. DIFFIE achieves the new state of the art in CaRB (1-1) both F1 and AUC, and outperforms the strongest rule-based system (ClausIE) in BenchIE; it also remains competitive in standard CaRB and WiRe57 evaluations, giving the best average score among systems that report all four benchmarks. Ablations show that uniform discrete diffusion outperforms absorbing state diffusion in our setting, and that a matched non-diffusion stochastic tagger does not reproduce its gains. Our results indicate that diffusion stochasticity is an effective mechanism for structured prediction tasks with multiple valid outputs.

1 Introduction

DIFFIE addresses OpenIE’s multi-output setting by using stochastic conditional discrete diffusion to generate and aggregate multiple candidate triplets. This decouples extraction capacity from training while targeting the redundancy and fixed-budget limitations of prior neural systems.

  • OpenIE sentences often express multiple valid schema-free relational triplets, making extraction a multi-output task.
  • Autoregressive generators are flexible but slow and redundant, whereas sequence-labeling models are efficient but typically produce one triplet per pass.
  • Fixed-slot prediction emits a predetermined number N of candidates but requires retraining when the target extraction cardinality changes.
  • DIFFIE samples independent reverse-diffusion trajectories over per-token role tags, then aggregates candidate triplets into a final set.
  • The inference process exposes both candidate-pool size and returned extraction count as inference-time choices, decoupling extraction budget from training.
  • DIFFIE reports best CaRB (1-1) F1 and AUC and BenchIE F1, outperforming prior neural systems and the strongest rule-based baseline on BenchIE.

2 Related Work

Prior OpenIE work spans token tagging, fixed-budget detection, autoregressive generation, low-resource curation, and several diffusion formulations. DIFFIE combines categorical discrete diffusion with inference-time sampling for schema-free multi-extraction.

  • Sequence-labeling OpenIE systems use BIO-style token tagging, while OpenIE6 uses a 2-D grid to represent discontinuous spans and overlapping relations.
  • DetIE emits a fixed number N of triplet candidates in one pass, whereas DIFFIE replaces the train-time fixed budget with inference-time sampling.
  • Autoregressive OpenIE systems generate extractions iteratively or with text-to-text models, offering flexibility but incurring corpus-scale cost and limited distributional control.
  • CycleOIE curates small GPT-annotated training sets and reports stronger results for few-shot example-guided curation; DIFFIE uses its LSOIE-examples subset.
  • Discrete diffusion operates directly over categorical vocabularies, with uniform and absorbing-state corruption among its principal formulations.
  • Uniform-noise diffusion can match or exceed absorbing-state diffusion for small vocabularies, motivating comparison in DIFFIE’s four-symbol tag space.
  • Unlike prior structured-prediction diffusion work, DIFFIE uses categorical discrete diffusion, multiple inference trajectories, and task-specific extractors for multi-reference OpenIE.
  • OpenIE benchmarks differ in reference construction and matching, spanning lenient CaRB, stricter CaRB (1-1), synset-level BenchIE, and high-precision WiRe57.

3 Method

DIFFIE formulates OpenIE as conditional discrete diffusion over four-role token sequences, samples multiple denoising trajectories, and clusters their triplets into ranked outputs. Sampling and output budgets are controlled at inference time.

  • Diffusion Formulation: DIFFIE uses categorical diffusion to reverse corruption of tag sequences, then draws independent trajectories to obtain candidate triplets for aggregation.
  • Problem Formulation: Each input token receives one of four roles—B, S, R, or O—and one tag sequence encodes exactly one Subject–Relation–Object triplet.
  • Inference: Unlike fixed-slot DetIE and autoregressive IMoJIE, DIFFIE produces one tag sequence per sample and recovers multiple extractions through repeated stochastic inference.
  • Architecture: The model combines a pretrained transformer encoder with a randomly initialized self-attention diffusion denoiser.
  • Training: Training expands each sentence with m gold triplets into m sentence–tag-sequence examples, teaching the denoiser a marginal distribution over single-triplet taggings.
  • Training: Uniform discrete diffusion corrupts ground-truth tags toward uniform draws from the four-symbol vocabulary and trains denoising with token-level crossentropy.
  • Sampling and Construction: At inference, n parallel reverse-diffusion trajectories generate candidate triplets; each role is surfaced using its longest contiguous predicted span.
  • Aggregation: Lenient matching clusters near-equivalent candidates, sums cluster frequencies, and returns the top-k clusters represented by their highest-frequency members.

4 Experimental Setup

The experiments train DIFFIE on curated LSOIE-examples configurations, evaluate it across four OpenIE benchmarks, and average results over independent random seeds. Hyperparameters and inference settings are selected using CaRB development data.

  • Training Data: Training uses LSOIE-examples, decomposing each sentence into one sentence–triplet instance and filtering instances through token-span alignment.
  • Training Data: Span-alignment filtering discards 38–40% of instances across configurations because curated arguments may paraphrase source spans.
  • Training Data: The study compares five curated subsets ranging from 50 to 4,901 sentences and uses LSOIE-EX-2.5K as its primary configuration.
  • Implementation: The denoiser uses a 6-layer transformer with dimension 512 and 8 attention heads, while diffusion runs for T = 16 steps with a cosine noise schedule.
  • Evaluation Protocol: Evaluation covers four OpenIE benchmarks, with comparisons emphasizing per-benchmark metrics because prior systems do not report identical metric sets.
  • Evaluation Protocol: Results are averaged over 10 independent random seeds, with n, k, and τ fixed globally from the CaRB development set.

5 Results

DIFFIE delivers strong results across OpenIE benchmarks, setting the best reported CaRB (1-1) F1 and AUC and outperforming prior neural systems on BenchIE. Its gains are strongest on stricter evaluations but do not transfer uniformly to WiRe57.

  • DIFFIE achieves the best reported CaRB (1-1) F1 and AUC among published OpenIE systems, with every seed exceeding the previous best F1 of 51.5.F1 variance across benchmarks is at most 0.5 points; Holm-corrected tests yield adjusted p = 0.0017 for CaRB (1-1) F1 and p < 10−7 for CaRB (1-1) AUC.
  • DIFFIE achieves the best average reported BenchIE F1, with 9 of 10 seeds exceeding ClausIE, although the gain over this strongest rule-based baseline is small.It substantially outperforms prior neural OpenIE systems on the stricter fact-level benchmark.
  • Despite training on only a 2.5K-sentence subset of CycleOIE’s LSOIE-examples resource, DIFFIE improves over CycleOIE on CaRB F1 and stricter CaRB (1-1) F1/AUC.CycleOIE retains higher standard CaRB AUC, while DIFFIE trails OpenIE6 on WiRe57.

6 Ablation Study

The ablations show that curated training data, inference-time sampling, uniform discrete diffusion, and reverse-diffusion candidate generation each materially shape DIFFIE’s behavior.

  • Effect of Training Data Size: CaRB (1-1) F1 peaks at 53.1 for LSOIE-EX-2.5K, exceeding LSOIE-EX-1.25K by 1.8 points and LSOIE-EX-FULL by 1.7.Adding raw LSOIE data reduces performance, with +LSOIE-FULL falling 5.6 CaRB F1 points and 7.4 CaRB (1-1) F1 points below LSOIE-EX-FULL.
  • Test-Time Compute Scaling: Larger denoising pools improve performance before saturation, while lenient matching consistently outperforms exact-frequency aggregation.The selected setting is n = 512, k = 4, τ = 0.9; it is within 2.0 F1 of the best BenchIE sweep point and matches the best WiRe57 point.
  • Test-Time Compute Scaling: Reducing n from 512 to 64 costs 1.5 F1 but raises throughput 6.6×, while reducing n to 16 costs 3.0 F1 and raises throughput 15×.DIFFIE remains slower than fixed-slot DetIE in absolute throughput: DetIE processes 686 sentences per second versus 14.2 at n = 16.
  • Comparison Caveats: The DetIE comparison is a same-evaluator reference rather than a controlled training comparison because the systems use different annotation targets.DetIE-LSOIE trains on original LSOIE token labels, whereas DIFFIE trains on CycleOIE annotations.
  • D3PM-uniform vs. Absorbing-State Diffusion: D3PM-uniform outperforms MDLM at both training sizes, by +2.2/+3.4 points on LSOIE-EX-2.5K and +4.1/+2.9 on LSOIE-EX-FULL.The four-symbol tag vocabulary (|V| = 4) places this comparison in the small-vocabulary regime.
  • Is Diffusion Necessary?: The matched MC-dropout tagger reaches 37.5 CaRB F1 versus 52.3 for DIFFIE and 37.3 CaRB (1-1) F1 versus 53.1.Its recall never exceeds 27.3 on CaRB or 27.9 on CaRB (1-1), whereas DIFFIE reaches 45.8 and 48.0, indicating candidate-pool limitation.

7 Discussion

DIFFIE uses stochastic reverse diffusion to generate diverse OpenIE candidates, then aggregates them under lenient matching. This creates an inference-time quality–cost trade-off while improving alignment with benchmarks that penalize over-extraction.

  • Multiple reverse-diffusion samples produce diverse valid outputs, matching OpenIE’s multi-output structure without fixed slots or autoregressive iteration.The sample count controls candidate-pool size and inference cost after training.
  • DIFFIE is slower than fixed-slot tagging but faster and more memory-efficient than direct LLM prompting, with quality traded against throughput after training.The sample count acts as a post-training cost-control knob.
  • Lenient-match aggregation recovers near-equivalent triplets with varying span boundaries more effectively than frequency aggregation.The benefit is attributed to clustering variations that exact-match aggregation would fragment.
  • Uniform-noise diffusion outperforms absorbing-state diffusion because non-mask tags can be revised as more context becomes available.This distinction matters because OpenIE role tags are jointly constrained, and the advantage persists after remasking strategies are swept.

8 Conclusion

DIFFIE treats OpenIE as conditional discrete diffusion over per-token role tags and uses reverse-process stochasticity to generate multiple candidate extractions. Across four benchmarks, it leads on CaRB (1-1) and BenchIE while remaining competitive elsewhere.

  • DIFFIE samples independent denoising trajectories and clusters their candidate triplets with a lenient-match extractor.Both candidate-pool size and returned-cluster count are inference-time choices.
  • DIFFIE achieves the best reported CaRB (1-1) F1 and AUC, beats the strongest rule-based system on BenchIE, and has the best average score across all four benchmarks.It remains competitive on standard CaRB and WiRe57.
  • A matched MC-dropout tagger does not reproduce DIFFIE’s gains, placing the observed improvement in candidate generation rather than stochastic aggregation alone.The control shares the encoder, labels, and aggregation procedure.
  • Uniform discrete diffusion outperforms absorbing-state diffusion in the four-tag setting, while supervision quality matters more than corpus size for sample-aggregation models.The authors attribute the diffusion result to the reverse process’s ability to revise earlier decisions.

Limitations

DIFFIE’s post-sampling construction and training supervision impose representational and data-scope limits. Sampling removes the output-count cap, but individual trajectories still cannot encode overlapping or discontinuous roles, and evaluation is limited to retained English extractive data.

  • The longest-span heuristic discards information when a role is discontinuous, although aggregation can recover the same fact from other trajectories.At n = 512, 28.9% of denoised samples contain at least one discontinuous role-tag run.
  • Each trajectory assigns one role per token, so overlapping roles and discontinuous arguments cannot be represented within a single sample.Alternative post-sampling rules may improve recall, but this remains future work.
  • All experiments use English supervision from CycleOIE’s curated LSOIE-EXAMPLES and do not evaluate other languages or out-of-domain corpora.The conclusions are therefore scoped to this training and evaluation setting.
  • The span-alignment filter removes roughly 40% of instances whose generated arguments do not match source spans verbatim.Because the discarded instances were not stratified, possible correlations with relation type, argument length, discontinuity, or syntax remain unresolved.

A Full Results with Standard Deviations

The reported evaluation tables summarize DIFFIE performance on CaRB, CaRB (1-1), BenchIE, and WiRe57 for the LSOIE-EX-2.5K configuration. Results are presented with means and standard deviations across samples where specified.

  • Each panel reports the mean and standard deviation across samples in its final row.The evaluation tables report precision, recall, F1, and AUC where applicable, with metrics shown in percentage points.
  • Table 7 reports DIFFIE results on CaRB and CaRB (1-1) for the LSOIE-EX-2.5K configuration.
  • Table 8 reports DIFFIE results on BenchIE and WiRe57 for the LSOIE-EX-2.5K configuration.

B Sensitivity Analyses

The sensitivity analyses vary τ, k, and n independently, selecting τ = 0.9, k = 4, and n = 512 as the operating point. They also show that lenient matching benefits from multiple samples and that increasing the sample count eventually yields diminishing returns.

  • Operating point: τ = 0.9, k = 4, and n = 512 are the selected operating-point values.The sweeps vary one parameter at a time while holding the others at their CaRB-dev-selected values.
  • Operating point: BenchIE and WiRe57 sweeps are diagnostic only because those benchmarks lack development splits.They therefore do not determine any reported test setting.
  • Parameter sensitivity: Table 9 measures F1 sensitivity to the clustering threshold τ and output budget k using one-at-a-time sweeps.Bold rows identify the CaRB-development-selected values used for reported results.
  • Aggregation and scaling: Lenient matching is consistently stronger than exact-frequency aggregation once multiple samples are available.Exact-frequency aggregation requires all three span boundaries to match, fragmenting posterior mass across boundary-shifted realizations of the same fact.
  • Aggregation and scaling: Larger n produces diminishing returns across all evaluated benchmarks.Table 10 reports F1 as the number of sampled trajectories increases, using lenient matching for BenchIE and WiRe57 and comparing lenient and exact-frequency aggregation on CaRB development data.
Loading 2609.02315v1…