PaperlayerView exact revision on arXiv

Source-linked AI summary

HeRE: A Novel Benchmark for Hebrew Relation Extraction via LLM-Guided Denoising of Knowledge Graph Alignments

Ron Keinan, Amir David Nissan Cohen, Reut Tsarfaty

arXiv:2610.22542v1cs.CLcs.LG

TL;DR

Low-resource languages lack native RE benchmarks, while annotation from scratch is expensive and translation can introduce bias. This paper builds HeRE by denoising naturally aligned knowledge-graph and text data, then compares five strategies. Multilingual models lead in few-shot settings, with Hebrew encoders strongest in label-free cross-training; Gemma-4-31B-it is statistically indistinguishable from GPT-5.4 on validation.

  • Problem

    Low-resource languages lack standardized native RE benchmarks, limiting reliable evaluation of model competence and resources for Hebrew-specific RE fine-tuning.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper creates HeRE by selecting a denoising method on a gold validation set and applying it to noisy alignments, using strategies including NLI encoders and few-shot LLMs.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Multilingual LLMs lead in few-shot settings, while Hebrew encoders are strongest in label-free cross-training; Gemma-4-31B-it reaches F1 = 0.915 versus GPT-5.4 at F1 = 0.921.

    Source: §9 · p. 9

  • Takeaways & Limitations

    HeRE supplies 1.5 million silver instances and two 500-example gold sets for Hebrew RE evaluation and model fine-tuning.

    Source: §1 · p. 2

    The pipeline is evaluated only on Modern Hebrew, assumes adequate Wikidata coverage and capable models, and has not yet been tested on a second language.

    Source: §10 · p. 10

Abstract

from arXiv · show

While LLMs demonstrate strong capabilities in English Relation Extraction (RE) tasks, a significant gap remains in assessing their RE performance in low-resource languages, due to a lack of standardized benchmarks. Creating native, language-specific annotated RE benchmarks is expensive and requires human expertise, while relying on translation yields suboptimal, inaccurate datasets that introduce English-centric biases. To tackle this, we propose a scalable, language-agnostic methodology to create RE data by denoising naturally occurring yet noisy alignments of knowledge bases and corresponding texts. We define a proper alignment via a Natural Language Inference (NLI) task, where a relation triplet is entailed by an input text. We empirically assess five denoising strategies, and demonstrate that multilingual models outperform language-specific models in the few-shot LLM setting, while Hebrew-specific encoders remain competitive under NLI fine-tuning and are strongest in the label-free cross-training setting. We further find that prompting in the native language achieves performance on par with or slightly exceeding English prompts, confirming that the pipeline operates naturally in the target language. The best open-weights model, Gemma-4-31B-it (F1\,=\,0.915), matches the best API model, GPT-5.4 (0.921), and reaches F1\,=\,0.844 on a fully held-out, dual-annotated test set. Applying the optimized pipeline, we release HeRE: a novel, massive, 1.5-million-instance silver dataset for Hebrew RE, designed to support data-intensive experimentation and model fine-tuning. HeRE bridges a critical gap in standard Hebrew RE evaluation, while providing the NLP community with a reproducible, language-agnostic pipeline for RE dataset construction in other low-resource settings.

1 Introduction

The paper addresses the shortage of standardized, native-language Hebrew RE resources with a pipeline that denoises naturally aligned text and knowledge-graph data. It introduces HeRE and compares denoising strategies, including their performance on held-out data.

1 Introduction

  • NLI frames text-relation alignment as entailment: the text is the premise and the relational triplet is the hypothesis, enabling comparison of monolingual and multilingual strategies.The study compares transformer relation classifiers and LLMs with models fine-tuned for NLI.Source: §1 · p. 2
  • Multilingual LLMs outperform competing architectures in few-shot denoising, while monolingual encoders remain competitive with NLI fine-tuning and lead in label-free cross-training.Task-specific NLI fine-tuning narrows the multilingual advantage for decoder models.Source: §1 · p. 2
  • HeRE provides 1.5 million silver-standard instances and two 500-example gold sets, including a held-out test set never used for selection.The 500-example validation set was used to select the denoising method.Source: §1 · p. 2
  • F1 = 0.844 on a fully held-out, dual-annotated test set; Gemma-4-31B-it reaches F1 = 0.776 on unseen predicates and 0.944 on seen predicates.The 500-example test set was not used for tuning.Source: §1 · p. 2
  • The deployed Gemma-4-31B-it filter gains +0.131 F1 over tuned CROCODILE and +0.509 over its published configuration; consensus-denoised edges support better link prediction than random selections.The paper also presents its validated, language-agnostic generation pipeline as a reproducible blueprint for other low-resource languages.Source: §1 · pp. 2–3

2 The Challenge

The challenge is to evaluate Hebrew RE and support Hebrew-specific model training despite the lack of native benchmarks and large training corpora. Existing ways to create benchmarks are difficult to scale or poorly matched to native Hebrew context.

2 The Challenge

  • Low-resource languages lack native RE benchmarks, limiting reliable evaluation of whether models have acquired RE competence and restricting fine-tuning on Hebrew-specific relation patterns.The section identifies the absence of a large Hebrew RE training corpus as a further constraint.Source: §2 · p. 3
  • Creating gold-standard data from scratch requires expert annotators to label thousands of triples, making human annotation prohibitively expensive at scale.The burden is especially relevant across diverse document types.Source: §2 · p. 3
  • Translating English benchmarks can introduce cultural and encyclopedic mismatch with Hebrew Wikipedia, while morphological errors may corrupt entity spans.The paper proposes a third path to sidestep the limitations of both standard alternatives.Source: §2 · p. 3

3 The Proposal: Denoising Noisy Text-KG Alignments

The proposal builds candidate Hebrew RE data by aligning naturally linked Wikidata triples and Wikipedia text, then denoises those noisy alignments. It compares five strategies on a gold-standard evaluation set.

3 The Proposal: Denoising Noisy Text-KG Alignments

  • Hebrew Wikipedia hyperlinks ground entities to Wikidata IDs, allowing passages where two entities co-occur to be aligned with triples in which they jointly participate.This yields a scalable, annotation-free source of native-language relational data.Source: §3 · p. 3
  • Entity co-occurrence does not ensure that a passage expresses the aligned relation, so the candidate corpus contains false positives that denoising must remove.The task assigns each tuple a binary label according to whether the text explicitly or clearly implicitly asserts the relation.Source: §3 · p. 3
  • The pipeline cleans Wikipedia markup, generates three NLI hypothesis variants per triple, then applies and compares five strategies on a gold-standard evaluation set.Figure 1 depicts the HeRE construction pipeline.Source: §3 · p. 3, Figure 1 · p. 4

4 Denoising Strategies

The paper compares five binary strategies for validating knowledge-graph and text alignments, varying the training signal, model type, and use of prompting. Each predicts whether a passage supports a verbalized relation.

Strategy 1: NLI Encoders

  • NLI encoders are fine-tuned on Hebrew NLI data and score entailment between the text premise and triple hypothesis, with a threshold producing a binary prediction.This strategy is fast and GPU-efficient but requires an NLI training corpus transferable to the target language.Source: §4 · p. 4

Strategy 2: Fine-tuned LLM NLI

  • Fine-tuned LLM NLI adapts a decoder with LoRA on the NLI corpus and derives its binary prediction from first-token probabilities for entailment, neutral, and contradiction.It uses the same NLI-format inference framing as the encoder strategy.Source: §4 · p. 4

Strategy 3: Cross-trained Predicate Classifiers

  • Cross-trained predicate classifiers learn from raw distant-supervision data without gold labels or NLI data, using held-out-fold predicate predictions and majority vote.Each fold's model is trained on the other K−1 folds.Source: §4 · p. 4

Strategy 4: Open-source LLMs (Few-shot)

  • Open-source instruction-tuned LLMs answer a binary yes/no relation question without fine-tuning; prompt language and triple verbalization are evaluated as independent choices.In-context examples calibrate the prompt.Source: §4 · p. 4

Strategy 5: Foundation LLMs (Few-shot)

  • Commercial foundation LLMs use the same binary prompt format via API, with hard-label predictions rather than token probabilities.The paper describes them as a practical performance ceiling but cost-prohibitive for large-scale silver annotation.Source: §4 · p. 4

5 Experimental Settings

The experiments compare five denoising strategies on a shared gold benchmark, then test the selected pipeline on knowledge-graph link prediction. They cover candidate data, hypothesis construction, model configurations, evaluation, and downstream setup.

5 Experimental Settings

  • The study evaluates all five denoising strategies as binary classifiers on a common 500-example gold-standard benchmark.Positive-class F1 is primary; macro-F1 is also reported because positive rates vary across predicates.Source: §5 · p. 4, §5.4 · p. 5

5.1 Data

  • Applying CROCODILE to an early-2025 Hebrew Wikipedia dump and matching Wikidata snapshot yields 3.1 million candidate tuples across 1,137 predicates, pairing section-level passages with triples whose subject and object are both mentioned.After entity-field normalization, passage lengths vary considerably, with a mean of 297 words after markup cleaning.Source: §5.1 · p. 4

5.2 Hypothesis Generation

  • Each triple receives basic concatenation, predicate-specific template, and LLM-generated Hebrew sentence hypotheses for testing relation entailment.Templates are grammatically natural and semantically precise.Source: §5.2 · pp. 4–5

5.3 Models

  • The model study spans fine-tuned NLI encoders and decoders, cross-trained predicate classifiers, few-shot models, and API LLMs.The evaluated sets include six encoders, nine fine-tuned decoder LLMs, four cross-trained encoders, five few-shot models, and three API models.Source: §5.3 · p. 5

5.4 Evaluation

  • F1 scores use bootstrap confidence intervals and paired-bootstrap model comparisons, with soft scores converted to binary labels using strategy-specific thresholds.The mean confidence-interval half-width is ±0.022; macro-F1 averages over predicates alongside the micro-averaged headline metric.Source: §5.4 · p. 5

5.5 Downstream Evaluation on HeRE-KG

  • Link prediction compares denoised, raw, size-matched random, and confidence-weighted graphs on shared held-out triples, with results averaged over three seeds.The shared test set scores triples whose entities and relation occur in every training graph; splitting is over unique triples to avoid mention leakage.Source: §5.5 · p. 5

6 Results

On the gold validation set, the denoising pipeline outperforms the CROCODILE filter, while the best open-weights model is not measurably worse than leading API models. Results also show differences across strategies, hypothesis formats, prompt languages, and downstream settings.

6 Results

  • 0.131 F1: deployed Gemma-4-31B-it exceeds tuned CROCODILE; the published filter has precision 0.830 and recall 0.269 on Hebrew.Changing hypothesis format alone improves CROCODILE more than retuning its threshold alone (+0.32 vs. +0.16 F1).Source: §6 · p. 6
  • GPT-5.4, Gemini 3 Flash, and Gemma-4-31B-it form one statistical cluster, with paired-bootstrap differences no larger than +0.007 (CI [−0.012, +0.027]).The best open-weights model is not measurably worse than the best commercial API model.Source: §6 · p. 6
  • Multilingual LLMs dominate few-shot prompting, but Hebrew encoders remain competitive under NLI fine-tuning and lead multilingual encoders in cross-training.Macro-F1 trails micro-F1 by 0.115 on average, and the ranking changes: Gemini 3 Flash is most balanced while Gemma-4-31B-it leads on micro-F1.Source: §6 · pp. 6–7
  • Structured hypotheses outperform plain triple concatenation for every encoder, by up to 0.10 F1 for NeoDictaBERT; templates lead for every fine-tuned decoder.Few-shot and API models are less sensitive to hypothesis form.Source: §6 · p. 7
  • Hebrew prompts are within 0.002 of English for Gemma-4-31B-it and Gemini 3 Flash, while Hebrew leads by 0.013 for GPT-5.4.The best Hebrew and English configurations are compared for each model.Source: §6 · p. 7
  • 24% relative MRR and 29% relative Hits@10: the denoised graph beats a size-matched random control, while the full raw graph scores highest in absolute terms.Confidence-weighted sampling does not improve on uniform sampling: MRR is 0.150 vs. 0.161.Source: §6 · p. 7, Table 3 · p. 7
  • Gemma-4-31B-it loses 0.07 F1 overall and 0.14 on 203 unseen-predicate examples in the held-out test, but shows no drop on seen predicates.The test set is a stress test, with 40% unseen predicates and passages 40% longer than validation passages.Source: §6 · p. 7, Table 4 · p. 7
  • Errors cluster around predicates without positive gold examples, low-prior geographic predicates such as borders, and a small set of annotation-ambiguous cases.Source: §6 · p. 7

7 HeRE: The Hebrew Relation Extraction Benchmark

HeRE combines human-annotated gold evaluation sets with a large silver dataset and a provenance-grounded knowledge graph. The released silver labels use unanimous agreement among three open-weights models.

Silver dataset

  • Table 5 reports held-out test-set F1 for each signal deployed on the silver corpus, with a threshold of 0.9 for marked signals.Ensemble results are reported separately in Table 16; the supplied passage does not give Table 5’s cell values.Source: Table 5 · p. 8

Gold datasets

  • Two 500-example gold sets provide validation and held-out evaluation, with dual native-speaker annotation and Cohen’s κ = 0.809 and 0.757, respectively.The held-out set includes 203 examples with predicates absent from validation and 200 drawn from the silver distribution.Source: §7 · p. 8

Silver dataset

  • 1,491,070 silver examples are released with an agree3 label that reaches precision 0.899, recall 0.934 and F1 = 0.916.The label requires unanimous agreement among Gemma-4-31B-it, Gemma-3-27B-it, and DictaLM-3.0-24B; the release also includes scores and hard predictions from ten signals and four consensus rules.Source: §7 · p. 8

HeRE-KG

  • HeRE-KG contains 1,080,605 evidence-linked edges over 424,216 entities and 917 relation types from 212,332 documents.Each edge carries its supporting Hebrew Wikipedia passage and the confidence scores behind its label; 95.6% of entities lie in one connected component.Source: §7 · pp. 8–9

8 Related Work and Future Directions

Prior work offers extensive English RE benchmarks and denoising methods, but native-language RE benchmarks remain scarce. HeRE’s pipeline and dataset suggest future applications while depending on linked Wikipedia coverage.

Distant Supervision and Denoising

  • HeRE extends CROCODILE’s single XLM-RoBERTa NLI filter into a five-architecture comparison, finding multilingual LLMs strongest in few-shot settings and monolingual NLI-fine-tuned encoders competitive.The comparison concerns denoising for low-resource RE benchmark construction.Source: §8 · p. 9

Multilingual Task Benchmarks

  • Native-language RE benchmarks for low-resource languages remain scarce, and HeRE is presented as the first native Hebrew RE benchmark.Existing English RE benchmarks span sentence-level, document-level, and large-scale corpora.Source: §8 · p. 9

Future Directions

  • The pipeline is directly applicable to languages with sufficient Wikidata-Wikipedia coverage, and its only data prerequisite is a Wikipedia edition linked to Wikidata.The authors identify Arabic, Persian, and other Semitic languages as natural next steps; passage-grounded HeRE-KG edges could also be proposed as candidate Wikidata statements.Source: §8 · p. 9

9 Conclusions

HeRE contributes a Hebrew RE benchmark and a language-agnostic pipeline for denoising Wikidata-Wikipedia alignments. Its results span model comparisons, native-language prompting, held-out evaluation, and knowledge-graph link prediction.

9 Conclusions

  • Gemma-4-31B-it reaches F1 = 0.915, statistically indistinguishable from GPT-5.4 at F1 = 0.921, while the held-out test F1 is 0.844 and 0.776 on unseen predicates.Hebrew prompting matches English; consensus denoising improves link prediction over a size-matched control.Source: §9 · p. 9
  • Across five denoising strategies, multilingual LLMs outperform Hebrew-specific models in few-shot open-source settings, while Hebrew-specific encoders lead in label-free cross-training.Monolingual encoders remain competitive under NLI fine-tuning.Source: §9 · p. 9
  • HeRE releases datasets, per-signal scores, and a reproducible pipeline for RE benchmark construction in languages with Wikidata-linked Wikipedia.The corpus also yields more than a million provenance-grounded knowledge-graph edges.Source: §9 · p. 9

10 Limitations

HeRE’s evidence comes from Hebrew Wikipedia aligned with Wikidata, and evaluation covers only a subset of predicates. The authors also report limits associated with passage length and language scope.

Domain scope

  • HeRE uses encyclopedic Hebrew Wikipedia passages, so transfer to social media, clinical, news, or legal Hebrew is not established.The authors identify this as a domain-scope limitation.Source: §10 · p. 10

Predicate coverage

  • Gold validation and test sets cover 97 and 124 of 1,137 predicates, respectively, leaving F1 on rare predicates indicative rather than definitive.The authors report F1 = 0.944 on seen versus 0.776 on unseen predicates.Source: §10 · p. 10

Passage length

  • Gold passages average 70 words versus 297 for silver; test F1 declines from 0.909 at 100–250 words to 0.818 at 500+ words, with overlapping bucket confidence intervals.The authors caution that length alone does not explain the validation-to-test gap.Source: §10 · p. 10

Language scope

  • Although designed to be language-agnostic, the pipeline is evaluated only on Modern Hebrew and presumes adequate Wikidata coverage and capable encoders or LLMs.Transfer to a second language is left for future work.Source: §10 · p. 10

11 Ethical Considerations

The supplied sections describe HeRE’s data and annotation safeguards, the rules for judging text–relation alignments, and the construction and evaluation of its Hebrew data pipeline. They also specify how denoising scores are converted into binary predictions.

11 Ethical Considerations

  • HeRE uses Hebrew Wikipedia and Wikidata, both freely licensed, and introduces no private or personally sensitive data.The two gold sets were independently annotated by paid expert annotators, with disagreements adjudicated.Source: §11 · p. 10

A Annotation Guidelines

  • A relation is positive when explicitly stated or unambiguously inferable within one or adjacent sentences; mere co-mention, a distinct relation, or inference requiring outside knowledge is negative.Annotators defaulted to negative in genuinely ambiguous cases and flagged unclear predicate definitions for adjudication.Source: A Annotation Guidelines · p. 14

B Experimental Setup

  • Few-shot prompt demonstrations are hand-written and disjoint from both gold sets, and Hebrew prompts are literal translations of the English versions.The prompts are included in the released code.Source: B Experimental Setup · p. 15
  • Threshold conversion selects hard labels by strategy: Strategy 1 uses fixed thresholds, Strategies 2 and 4 use swept soft-score thresholds, and Strategy 3 uses a 0.5 majority-vote threshold.Strategy 1 reports the best F1 over thresholds 0.6–0.9; Strategies 2 and 4 sweep 0.3–0.9, while Strategy 5 reports hard-label metrics only.Source: B Experimental Setup · p. 15

C CROCODILE Data Construction Pipeline

  • The CROCODILE-based pipeline extracts linked entities from Hebrew Wikipedia, retrieves Wikidata triples, aligns passages where both entities occur, and replaces the default NLI filter with the paper’s denoising study.The Hebrew corpus contains 3.1 million candidate tuples spanning 1,257 predicate strings before entity-field normalization and 1,137 afterward.Source: C CROCODILE Data Construction Pipeline · pp. 15–16

D HebNLI Fine-tuning

  • HebNLI supplies three-label entailment, contradiction, and neutral training data for the NLI backbones used in denoising, rather than direct RE evaluation.It contains 300,067 training examples, 1,999 validation examples, and 884 test examples; Table 6 reports macro-F1 for the fine-tuned models.Source: D HebNLI Fine-tuning · p. 16

E Error Analysis: Annotated Examples

The error analysis identifies false positives from unsupported geographic or administrative relations and from incidental naming cues, alongside cases that remain difficult for all models. The supplied results passages also describe the strategy-level and per-predicate analyses used to report denoising performance.

E Error Analysis: Annotated Examples

  • Three error patterns emerge: predicates with no positive gold examples, low-prior neighboring predicates that drive false positives, and residual cases misclassified by every model.The residual cases include ambiguity where a valid knowledge-graph triple is not expressed in its aligned passage.Source: E Error Analysis: Annotated Examples · p. 16
  • Borders and diplomatic relations have low positive rates (0.19 and 0.13); geographically adjacent entities can co-occur without the passage asserting either relation.The best configurations reach F1 0.857 for borders and 1.000 for diplomatic relations, while these predicates remain a major false-positive source for mid-tier models.Source: E Error Analysis: Annotated Examples · p. 16
  • All API prompt configurations reject the Serbia–Serbian official-language example, while 16 of 72 open-weights configurations answer yes to a parenthetical naming cue.Adjudicators labeled the case negative because determining the official language requires external knowledge; this predicate has no positive validation examples.Source: E Error Analysis: Annotated Examples · p. 17
  • Models treat “a district in France” as jurisdiction area, although the annotation requires a more specific administrative relation than geographic containment.The passage’s surface wording illustrates how finer-grained Wikidata predicates can be overpredicted from natural-language co-occurrence.Source: E Error Analysis: Annotated Examples · p. 17

F Full Results by Strategy

G Per-predicate, Length and Silver-Signal Analyses

  • The appendix tables cover predicate positive rates, per-predicate F1, passage-length F1, deployed-signal positive rates, and consensus-rule test F1.The captions identify these analyses but do not expose the underlying table values.Source: Table 12 · p. 18, Table 13 · p. 19, Table 14 · p. 19
  • Two deployed Strategy-3 signals predate a later grid re-analysis that found K immaterial; they abstain on 0.4% of rows with fewer than 20 silver examples.Aya-Expanse-32b and Gemma-2-9B were deployed at score threshold 0.9.Source: G Per-predicate, Length and… · p. 19

Related papers

Loading 2610.22542v1…