PaperlayerView exact revision on arXiv

Source-linked AI summary

Ontology-Driven Structural Regularization for Document-Level Relation Extraction

Laura Menotti, Stefano Marchesin, Gianmaria Silvello

arXiv:2608.20856v1cs.CL

TL;DR

DocRE depends on costly manual data because distant supervision is noisy, including overlooked structural inconsistencies in relational triples. The paper introduces an ontology-driven framework and preprocessing strategy, finding that structurally consistent training data reduces prediction violations and improves generalization.

  • Problem

    Distant supervision provides large DocRE datasets but introduces semantic and structural noise, while structural inconsistencies remain an overlooked source of supervision noise.

  • Method

    The paper uses OWL-based ontology constraints to diagnose and enforce structural well-formedness in DocRE training data through model-agnostic preprocessing.

  • Results

    Training on structurally consistent data lowers structural violations in predictions and improves performance across nearly all metrics, with average gains of +6.36% in F1 and +7.62% in IGNF1.

  • Takeaways & Limitations

    Structural data quality is important for the reliability and generalization ability of DocRE systems.

  • Takeaways & Limitations

    The strategy removes affected entities rather than correcting all ill-formed triples, leaving 58,619 instances for which manual revision would be prohibitively expensive.

Abstract

from arXiv · show

Document-Level Relation Extraction (DocRE) relies heavily on costly manually annotated datasets, while large distant supervision resources such as DocRED distant remain underexploited due to noise. We show that a critical yet overlooked source of noise lies in structural inconsistencies within relational triples, including violations of ontology constraints and logical contradictions. We introduce an ontology-driven framework to quantify and enforce structural consistency in DocRE datasets. Our analysis reveals substantial structural noise in DocRED distant and demonstrates that such inconsistencies propagate to model predictions. Enforcing structural well-formedness during training significantly reduces logical contradictions and consistently improves generalization performance. These findings establish structural consistency as a missing axis of supervision in DocRE and highlight structural regularization as an effective strategy for leveraging distant data at scale.

1 Introduction

DocRE depends on reliable data to construct useful knowledge graphs, but distant supervision introduces substantial semantic and structural noise. This paper quantifies ontology-level inconsistencies and proposes preprocessing that enforces structural consistency before training.

  • Motivation: DocRE extracts relations between entity pairs across full documents to support knowledge-graph construction and downstream applications.Accurate knowledge graphs depend on reliable DocRE models trained on high-quality data.
  • Motivation: 101,873 documents comprise DocRED distant, compared with 5,053 documents in the manually curated DocRED split.The distant dataset aligns Wikidata triples with Wikipedia text through distant supervision.
  • Motivation: Distant supervision assumes that a knowledge-graph relation holds whenever two entities co-occur in a document, adding substantial noise to DocRE.Document-level co-occurrence is less likely than same-sentence co-occurrence to express a true relation.
  • Motivation: DocRE datasets also contain structural inconsistencies, including ontology violations, missing inverse relations, asymmetric contradictions, and cardinality conflicts.The paper defines structural consistency as RDF-triple well-formedness under ontology-imposed constraints, rather than full ontology satisfiability.
  • Approach: The framework uses OWL semantics to quantify structural violations and applies a lightweight, model-agnostic preprocessing pipeline before training.The pipeline cleans and augments data without changing model architectures, loss functions, or inference procedures.
  • Contribution: Enforcing structural well-formedness reduces logical contradictions in predictions and consistently improves generalization across state-of-the-art DocRE models.The framework is presented as structural regularization for existing sequence-based and graph-based systems.

2 Related Work

Prior DocRE denoising work mainly improves semantic reliability through model-driven refinement, whereas this paper targets structural well-formedness under ontology constraints as a complementary dimension.

  • Distant-supervision denoising: Distant supervision introduces noisy supervision through heuristic alignment between text entity pairs and reference knowledge-graph triples.
  • Distant-supervision denoising: Existing DocRE denoising methods use multi-task alignment, uncertainty-guided label filtering, or active learning for long-tail relations.
  • Research gap: These approaches primarily refine semantic reliability of distant labels through model-driven methods.
  • Logical reasoning: Logical-rule methods enhance relation prediction through latent-rule learning or secondary reasoning over model predictions.
  • Research gap: This work instead addresses structural well-formedness under ontology-imposed constraints independently of model architecture or inference strategy.
  • Constraint-based extraction: Constraint-based approaches in sentence-level relation extraction use entity types, descriptions, or constraint graphs to guide prediction under distant supervision.

3 Methodology

The methodology interprets DocRE predictions as RDF triples and evaluates their local well-formedness using OWL-derived type, inverse, asymmetry, and cardinality constraints.

  • DocRE formulation: DocRE predicts relations from R between ordered entity pairs, with na representing absence of a relation, and treats predictions as local knowledge-graph construction.This perspective exposes structural inconsistencies invisible to standard independent classification objectives.
  • DocRE formulation: Each entity mention receives one of six types, and the first mention represents the entity’s type for structural checking.
  • Invalid triples: A relation triple is valid when the subject type belongs to the relation domain and the object type belongs to its range.Relations are modeled as OWL object properties with permitted subject and object types.
  • Invalid triples: Relation domains and ranges combine Wikidata property constraints with popular subject and object types from ReDocRED, while every relation includes MISC.
  • Missing inverse relations: If a relation has a declared inverse, its reversed inverse triple should also appear under the document-level local closed-world assumption.For example, P1376 (capital of) and P36 (capital) form an inverse pair.
  • Asymmetric violations: For asymmetric relations, the presence of both directional triples constitutes a logical contradiction; P26 and P3373 are the only symmetric relations identified.
  • Cardinality violations: Cardinality violations occur when a subject has more distinct relation objects than the relation’s maximum threshold.Although 11 relations were initially expected to be functional, empirical evidence motivates relaxing some thresholds to 2.

4 Structural Consistency in DocRE

This section assesses structural consistency in DocRED and related datasets using ontology-based rules, statistically grounded error analysis, and manual annotation. It finds substantially more inconsistencies in DocRED distant than in manual datasets, while revised datasets retain smaller but non-negligible violations.

  • Datasets Consistency Assessment: The analysis applies structural-consistency rules to DocRED, ReDocRED, UGDRE, and DOREMI datasets, with DocRED distant as the primary focus.The rules assess invalid triples, missing inverse relations, asymmetric violations, and cardinality violations.
  • Datasets Consistency Assessment: The study reports absolute and percentage errors for each structural constraint across manual, distant, and revised dataset splits.Table 1 distinguishes manually annotated Train, Dev, and Test splits from the Train DS distant split.
  • Datasets Consistency Assessment: DocRED distant contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies than ReDocRED manual training data.Its cardinality-violation rate is only 0.01%, compared with 0.28%–0.65% in the other datasets.
  • Datasets Consistency Assessment: ReDocRED splits retain 0.71%–0.94% invalid triples, below 10% missing inverse relations, and 0.33%–0.48% asymmetric violations.Cardinality violations reach 0.65% in the ReDocRED test set, and ill-formed evaluation triples may degrade models generating well-formed outputs.
  • Invalid Triples Analysis: The invalid-triple analysis classifies errors as incorrect relation annotations or incorrect entity annotations using a manually annotated random sample and binomial proportion estimation.The sample contains nS = 400 invalid triples, and uncertainty is reported with a 95% HPD credible interval.
  • Invalid Triples Analysis: 69% of invalid triples stem from entity annotation errors, while 31% stem from incorrect relation annotations.Because the intervention does not refine entity extraction, the reported performance gains are conservative estimates.

5 Training Effect

Training on structurally corrected distant data reduces ill-formed predictions and improves DocRE generalization beyond random data removal.

  • Experimental Setup: The study trains ATLOP and DREEAM on DocRED distant data and a structurally corrected version, evaluating predictions and performance on ReDocRED.The corrected data removes ontology-violating triples, missing inverse relations, and asymmetric violations; random removal provides a comparison.
  • Structural Consistency in Predictions: Invalid triples in predictions range from 0.85% to 1.61%, while asymmetric violations remain between 0.53% and 0.79%.The prediction rates are lower than those observed in training data, but structural inconsistencies still propagate into model outputs.
  • Structural Consistency in Predictions: 62.92%: structurally consistent training reduces the average proportion of invalid predicted triples.Missing inverse relations decrease by 87.44% for ATLOP-RoBERTa, 86.97% for ATLOP-BERT, and 74.13% for DREEAM.
  • Generalization Performance: ATLOP gains average +1.65% precision, +3.10% recall, and +2.69% F1 after structural correction.Ignored-pair metrics also improve, with IGNPREC increasing by +3.83% and IGNF1 by +4.59%.
  • Generalization Performance: DREEAM improves in recall and F1 after correction, although precision decreases slightly by −0.45% on average.Average recall rises by +14.07% and F1 by +10.02%; random removal produces only marginal or sometimes negative changes.

6 Conclusions

The paper formalizes structural consistency in DocRE through ontology-based constraints and shows that correcting structural noise improves prediction well-formedness and generalization.

  • Conclusions: The framework models DocRE as knowledge-graph construction under RDF and uses OWL constraints to identify structural inconsistencies.It detects invalid triples, missing inverse relations, asymmetric violations, and cardinality violations.
  • Conclusions: DocRED distant contains three times more invalid triples, nearly five times more missing inverse relations, and twice as many asymmetric inconsistencies than ReDocRED manual training data.Almost 70% of invalid triples in a representative sample arise from named entity misannotations.
  • Conclusions: Structurally consistent training systematically lowers prediction violations and improves performance across nearly all metrics.Average gains reach +6.36% in F1 and +7.62% in IGNF1.

7 Limitations

The evaluation and mitigation strategy have important scope boundaries involving benchmark noise, entity removal, limited statistical analysis, and model coverage.

  • Evaluation Scope: The ReDocRED test set contains a limited number of logical inconsistencies, so it is not entirely free of structural noise.Re-annotating it could also raise concerns about annotation bias because modifications might appear tailored to favor the approach.
  • Data-M itigation Scope: The cleaning strategy removes affected entities instead of directly correcting ill-formed triples, discarding data to avoid large-scale manual revision.Manual correction of 58,619 instances would be prohibitively expensive; systematic re-annotation is proposed as future work.
  • Statistical Analysis: The error analysis is primarily empirical and does not quantify statistical significance across different structural violation sources.Future work could derive generalization bounds or analyze effects on hypothesis-class capacity.
  • Model Scope: Training-effect analysis is restricted to ATLOP and DREEAM, leaving graph-based and generative DocRE approaches for future investigation.The released code and structural rules are intended to support broader evaluation.
  • Qualitative Analysis: The qualitative error analysis uses nS = 400, yielding a 95% HPD interval of ±4.5% around the 0.69 entity-error proportion.Halving the interval width would require approximately 1,600 samples and four times the annotation cost.

8 Ethical Considerations

The paper reports limited anticipated ethical concerns, emphasizing open-source data and models while acknowledging privacy, licensing, and pretrained-model bias considerations.

  • Ethical Considerations: The authors anticipate no significant ethical concerns and discuss licenses, data privacy, hidden biases, and AI-assistant use.The section frames these as potential ethical aspects and risks of the analysis.
  • Licenses and Data Privacy: Open-sourced datasets and models are used to reduce privacy-leakage risk and support transparency and accessibility.The empirical models are trained and evaluated on MIT-licensed datasets.
  • Hidden Biases: ATLOP and DREEAM rely on pretrained language models that may encode hidden biases from their training data.The authors urge special caution for sensitive relation types or entities.
  • AI Assistants: ChatGPT, Perplexity, Writefull, and Grammarly were used only to improve language clarity, while the authors developed and reviewed the scientific content.The manuscript states that all authors approved the final work.

A.1 Consistency Assessment

Structural inconsistencies remain widespread across denoised distant datasets, including invalid triples, missing inverses, asymmetric inconsistencies, and cardinality violations.

  • Denoised distant datasets retain substantial structural errors across the analyzed rule categories.The analysis reports absolute error counts and percentages for each rule.
  • 3.01% of UGDRE-DocRED triples are invalid, nearly matching the 3.07% found in DocRED distant.
  • 75.13% of DOREMI-DocRED triples miss inverse relations, compared with 25.42% in DOREMI-ReDocRED.The corresponding UGDRE percentages are also lower for ReDocRED than DocRED.
  • 0.77% of UGDRE-DocRED triples show asymmetric inconsistencies, versus 0.36% for DOREMI-DocRED and 0.84% for DocRED distant.
  • State-of-the-art denoising fails to eliminate structural inconsistencies and can amplify cardinality violations.

A.2 Training effect

The training-effect analysis evaluates whether removing structural noise from denoised datasets reduces ill-formatted predictions.

  • Table 5 evaluates ATLOP and DREEAM predictions after training on denoised distant datasets and corrected versions.
  • The evaluation reports absolute and percentage errors for invalid triples, missing inverse triples, and asymmetric inconsistencies.

A.2.1 Models Predictions

Correcting structural noise reduces logical inconsistencies in model predictions and can improve performance, with the largest prediction gains concentrated in missing inverse relations.

  • −86.45% average reduction in missing inverse relations is achieved by corrected DOREMI-DocRED across both DocRE models and configurations.
  • Invalid-triple errors fall from 1.37% to 0.36% for DREEAM-RoBERTa trained on corrected UGDRE-DocRED, a −73.72% relative reduction.
  • Asymmetric inconsistencies drop from 1.32% to 0.13% in the same configuration, a −90.15% relative decrease.
  • Performance improves in 47 of 80 cases (59%) across models and configurations, although denoised datasets do not always benefit from correction.

B Individual Structural Constraint Impact

Cleaning individual structural constraints produces constraint-specific gains, while combining all corrections yields the strongest overall performance across ATLOP and DREEAM.

  • Individual constraint effects: Correcting invalid triples consistently improves ATLOP precision, while correcting missing inverse triples produces the largest recall increase.For ATLOP-BERT, invalid-triple correction raises precision by +0.35%, whereas inverse correction raises recall by +3.55%.
  • Individual constraint effects: +0.73% F1 and +2.08% IGNPREC result from correcting invalid triples for ATLOP-BERT on DocRED distant.
  • Combined constraints: All corrections produce ATLOP-BERT gains of +0.60% precision and +3.41% IGNF1, while ATLOP-RoBERTa achieves its best average improvement with all constraints.
  • Combined constraints: All structural constraints raise DREEAM recall by +15.52% for DREEAM-BERT and +7.05% for DREEAM-RoBERTa, with small precision degradation.
  • Combined constraints: Both models show that invalid-triple removal improves precision and inverse-relation correction increases recall, while combined constraints deliver the best overall performance.
  • Constraint inventory: The constraint inventory covers domain and range types, inverse relations, and bounded maximum cardinalities for DocRED relations.
Loading 2608.20856v1…