Source-linked AI summary
U4RASD at KnowledgeGraphEval-2026: Ontology-Constrained Encoder and LLM Systems for Arabic Relation Extraction
Ahmad Abbas, Hussein Mazeh, Chadi Abou Chakra, Aya Jouni, Mohammad Ballout, Fadi A. Zaraket
TL;DR
Directed Arabic relation extraction must distinguish among semantically similar relations, repeated mentions, and entity-type constraints. The paper develops encoder and QLoRA-adapted LLM systems with ontology-constrained prediction, and its ensemble ranks first in Subtask 2.
Problem
Directed Arabic relation extraction remains challenging when relations are semantically similar, entity mentions repeat, or entity types determine which relations are valid.
Source: §1 · p. 1Method
The paper combines typed-marker encoders and QLoRA-adapted Yehia and Qwen systems with shared occurrence resolution, entity typing, and ontology-based relation filtering.
Source: §1 · p. 1, §3.3.2 · p. 2Results
Takeaways & Limitations
The results highlight constrained prediction, entity-type compatibility, candidate resolution, and complementary modeling approaches for Arabic relation extraction.
Source: §6 · p. 4Training-level results use a single random seed, and the systems depend on hard entity-type predictions and exact-match occurrence recovery.
Source: Limitations · p. 4
Abstract
from arXiv · showhide
This paper details our submission to KnowledgeGraphEval 2026 Subtask 2 on directed Arabic relation extraction. We explored encoder-based and large language model approaches for predicting one of 40 relations or no_relation from an Arabic sentence and a subject-object pair. Our encoder systems use typed entity markers and separate relation-existence and relation-classification heads, while our LLM systems fine-tune Yehia and Qwen3.5 with QLoRA and ontology-constrained prediction. Our best system, an ensemble of Yehia-RE, Qwen-RE, and TypedPair, achieved a micro-F1 score of 94.06 and ranked 1st in Subtask 2. Yehia-RE (full-vocabulary) achieved 92.98, while the best encoder and Qwen-RE achieved 92.69 and 91.94 respectively. Our results highlight the importance of constrained prediction and complementary modeling approaches for Arabic relation extraction.
1 Introduction
The paper addresses directed Arabic relation extraction, where semantically similar relations, repeated entity mentions, and type constraints pose challenges. It presents encoder and LLM systems for the task and reports a first-place ensemble result.
1 Introduction
- Arabic relation extraction is challenging when relations are similar, mentions repeat, or entity types constrain valid relations.Source: §1 · p. 1
- The task predicts one of 40 positive relations or no_relation from an Arabic sentence and designated subject–object pair.The data are highly imbalanced, making positive-relation prediction and no_relation handling important.Source: §1 · p. 1
- The paper investigates TypedPair encoders and QLoRA-adapted Yehia-RE and Qwen-RE, with shared occurrence resolution, entity typing, and ontology-based filtering.TypedPair uses directional typed markers and separate existence and positive-relation classification heads; Yehia-RE uses constrained one-token choices, while Qwen-RE uses causal-LM prediction or sequence classification.Source: §1 · p. 1
- 94.06 micro-F1 ranked the Yehia-RE, Qwen-RE, and TypedPair ensemble 1st in Subtask 2.The strongest individual systems achieved 92.98 for Yehia-RE, 92.69 for TypedPair, and 91.94 for Qwen-RE.Source: §1 · p. 1
2 Background
The background situates the submission in Arabic relation extraction resources and prior marker-based, ontology-filtered, and generative approaches.
2 Background
- WojoodRelations supplies the task data, while WojoodOntology provides relation definitions and entity-type domain–range constraints [Aljabari et al., 2025a; Aljabari et al., 2025b].Source: §2 · p. 1
- Prior Arabic work introduced NLI-RE, which frames candidate relations as natural-language-inference hypotheses and filters them by compatible entity types [Aljabari et al., 2025b].Source: §2 · pp. 1–2
- Related work includes marker-based encoder methods [Baldini Soares et al., 2019; Ye et al., 2022; Zhou and Chen, 2022] and generative or prompted RE approaches [Rakan Al Mraikhat et al., 2024; Wan et al., 2023; Wadhwa et al., 2023; Zhang et al., 2023].The paper investigates encoder-based and constrained LLM approaches in the same ontology-aware Arabic RE setting.Source: §2 · pp. 1–2
3 System Overview
The system overview combines shared occurrence and type processing with encoder and LLM approaches, whose relation decisions are constrained by entity types and the ontology.
3 System Overview
- Figure 1 summarizes the shared pipeline and the two model families, with subsequent subsections describing components that distinguish the submitted systems.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Shared Pipeline
- The shared pipeline resolves exact subject and object occurrences, predicts their types, filters candidate relations by WojoodOntology, and applies a row-level decision strategy when multiple occurrence pairs exist.Source: §3.1 · p. 2
3.2 Encoder-Based Systems
- TypedPair uses typed directional markers, pair representations, and a 42-dimensional type vector, with separate heads for relation existence and positive-relation classification.Negative rows supervise only the existence head; inference masks positive relations by predicted type pair and consults that head only above a validation-selected threshold.Source: §3.2 · p. 2
3.3.1 Yehia-RE
- Yehia-RE poses each row as an Arabic multiple-choice task over ontology-compatible relations plus no_relation, using one-token codes for displayed options.The full-vocabulary variant trains against the complete vocabulary; the row-local variant normalizes over options displayed for that example.Source: §3.3.1 · p. 2
3.3.2 Qwen-RE
- Qwen-RE uses QLoRA prompts with marked sentences, entity strings and types, and ontology-compatible label meanings; causal-LM variants generate or score labels, while Qwen-RE-9B-Cls predicts with a masked 41-way head.The classifier uses a shorter prompt and avoids textual generation and parsing failures.Source: §3.3.2 · p. 2
4 Experimental Setup
The setup enriches and filters training data, uses model-specific imbalance interventions, and selects checkpoints on a development artifact before official inference.
4.1 Data and Preprocessing
- 17,381 raw relation rows were reduced to 15,686 training instances after conflict filtering, ontology compatibility filtering, and deduplication.The development artifact has 1,687 rows and the official blind input has 2,074; their class distributions differ, so their micro-F1 scores are not directly comparable.Source: §4.1 · p. 3
- 6,677 training examples are no_relation, followed by 4,269 Location.located_in and 1,684 PartOf.geopolitical_division examples.Encoder models use cross-entropy, Yehia-RE frequency-based sampling, Qwen causal-LM clipped inverse-frequency weighting, and the Qwen classifier square-root inverse-frequency sampling.Source: §4.1 · p. 3
4.2 Training, Decoding, and Selection
- Relation-extraction checkpoints are selected on the 1,687-row development artifact and frozen before official inference; TypePredictor is selected separately on its entity-typing validation split.Encoder models are fully fine-tuned, while Yehia and Qwen use 4-bit NF4 QLoRA with frozen base weights.Source: §4.2 · p. 3
- Approximately 43 minutes on an NVIDIA A40 trained AraBERTv02-Large TypedPair, versus 3.32 hours for Yehia-RE on an L40S and 12 hours for Qwen-RE-9B on an RTX PRO 4500.Because the measurements use different GPUs, they are not a controlled efficiency comparison, though they illustrate substantially higher LLM training cost.Source: §4.2 · p. 3
- Official evaluation uses micro-F1, while row-level decisions vary by configuration among voting, confidence or existence selection, max-positive, and nearest-candidate strategies.Source: §4.2 · pp. 3–4
5 Results
The official results show strong performance across encoder and LLM systems, with the three-system ensemble ranking first in Subtask 2. Error analysis identifies interacting challenges involving entity types, repeated mentions, similar relations, label imbalance, and canonical-label generation.
5.1 Encoder Results
- 92.69 was the strongest official encoder result, achieved by AraModernBERT; the complete configuration with typed text markers and a directional type vector led tested variants.It outperformed ARBERTv2 at 91.96 and the two reported AraBERTv02-Large aggregation strategies at 92.20–92.29.Source: §5.1 · p. 4
5.2 LLM Results
- 94.06 micro-F1 and 1st place in Subtask 2 went to the Yehia-RE, Qwen-RE, and TypedPair majority ensemble.Yehia-RE (full-vocabulary) scored 92.98, and Qwen-RE-9B-Cls scored 91.94.Source: §5.2 · p. 4
- 91.94 was Qwen-RE-9B-Cls’s official blind score, compared descriptively with 91.53 for Qwen-RE-4B and 91.76 for Qwen-RE-9B.These differences were not controlled ablations.Source: §5.2 · p. 4
5.3 Error Analysis
- The error analysis identifies type errors, repeated surface forms, semantically close relations under skewed labels, and invalid or partial generated labels as interacting failure modes.Type errors can remove the correct relation from the compatible set, while repeated mentions can associate predictions with the wrong evidence.Source: §5.3 · p. 4
6 Conclusion
The paper presents complementary encoder and LLM systems for Arabic relation extraction, with ontology-constrained decisions across the LLM family. Their ensemble ranked first in Subtask 2, and the authors identify future work on uncertainty, joint scoring, and confidence-weighted ensembles.
6 Conclusion
- 94.06 official micro-F1 and 1st place in Subtask 2 were achieved by an ensemble of Yehia-RE, Qwen-RE, and TypedPair.Yehia-RE (full-vocabulary) scored 92.98; the strongest encoder and Qwen-RE systems scored 92.69 and 91.94.Source: §6 · p. 4
- The encoder family combines typed directional markers with a two-head relation objective, while the LLM family constrains Yehia-RE and Qwen-RE to ontology-compatible decisions.The authors highlight constrained prediction, entity-type compatibility, candidate resolution, and complementary modeling approaches.Source: §6 · p. 4
- Future work will propagate type uncertainty, jointly score occurrence and relation hypotheses, and study confidence-weighted ensembles.Source: §6 · p. 4
Limitations
The appendix reports AraModernBERT’s strongest official encoder result and describes the preprocessing and modeling pipeline. It also notes limits from single-seed training results and possible sentence overlap in the mention-level TypePredictor split.
Limitations
- Training-level results use a single random seed, leaving run-to-run variation unquantified and making small configuration differences uncertain.The systems also rely on hard entity-type predictions and exact-match occurrence recovery, and remain challenging for rare or semantically adjacent relations.Source: Limitations · p. 4
A.4 Span and Entity-Type Recovery
- Training rows gain occurrence-specific spans and entity types through NER alignment, model predictions, and manual resolution of 240 remaining rows; blind rows rely entirely on inference.Processed training and development artifacts had zero span-slice errors after validation.Source: §A.4 · p. 7, §A.5 · p. 7
A.6 Conflict Removal and Deduplication
- Conflict removal reduced 17,381 raw rows to 17,208 non-conflicting rows; deduplication and ontology filtering yielded 15,686 final training instances.The filtering checks ordered subject–object entity types against relation domain–range constraints.Source: §A.6 · p. 7, §A.7 · p. 7
A.7 Ontology Compatibility and Operational Mapping
- The operational Location.located_in range was expanded to include ORG, FAC, LOC, and GPE entity types based on recurring labeled examples.This was an implementation-level adaptation to the relation data.Source: §A.7 · p. 7
B.1 Dataset Construction
- The mention-level TypePredictor split allows sentence overlap across splits, which may affect entity-type predictions used for typed markers and ontology filtering.TypePredictor uses entity-type labels rather than relation labels.Source: §B.1 · p. 8
C.2 Ontology Filtering
- At inference, predicted entity types inform typed markers and ontology-based relation filtering; systems use different strategies to resolve multiple occurrence pairs.For each type pair, incompatible positive relations are masked while no_relation remains available.Source: §C.2 · p. 9, §C.3 · p. 9
D.1 Typed Directional Input
- TypedPair combines directional subject/object type markers with a 42-coordinate type vector and separate heads for relation existence and positive-relation classification.Negative rows supervise existence but do not assign a positive-relation target.Source: §D.1 · p. 10, §D.2 · p. 11
D.5 Extended Official Results
- 92.69 was the highest reported official TypedPair F1 score, achieved by AraModernBERT.Table 13 summarizes the official TypedPair scores.Source: Table 13 · p. 12, §D.5 · p. 12
D.6 AraModernBERT Component Ablation
The AraModernBERT ablation found that the complete configuration performed best among four tested submissions. The appendices describe Yehia-RE’s row-specific multiple-choice design and its training, inference, and candidate-pair procedures.
D.6 AraModernBERT Component Ablation
- The complete AraModernBERT configuration achieved the strongest result among the four official blind-evaluation ablation submissions.The comparison tested configurations with and without typed text markers and the directional type vector; no score values are supplied here.Source: §D.6 · p. 12, Table 14 · p. 12
E.1 Row-Specific Arabic Multiple Choice
- Yehia-RE uses a row-specific Arabic multiple-choice task because full relation-phrase generation was vulnerable to explanatory text, punctuation, and lexical variation.The model sees Arabic natural-language templates rather than canonical relation identifiers as answer options.Source: §E.1 · p. 12
E.2 Full-Vocabulary and Row-Local Objectives
- The Yehia-RE full-vocabulary objective trains against the complete vocabulary, whereas the row-local objective normalizes over only the displayed codes.At inference, the full-vocabulary variant compares only codes displayed for the current row.Source: §E.2 · p. 13
E.5 Deterministic Transformation
- The deterministic Yehia-RE builder validates spans and types, retrieves compatible relations, renders Arabic templates, assigns codes, and constructs chat messages.Positive options are deterministically shuffled with seed 42, and no_relation is appended.Source: §E.5 · p. 14
E.6 QLoRA Configuration
- Yehia-RE applies RS-LoRA to internal transformer modules while keeping the language-model head frozen.The audit reports 225 supported linear/head modules, with adapters wrapping 224 internal modules.Source: §E.6 · p. 14
E.7 One-Token Masking Audit
- Yehia-RE supervises exactly one assistant code token per row and masks prompt, template, and padding tokens from the loss.Rows are rejected when the answer is not one token or the sequence exceeds 1,024 tokens.Source: §E.2 · p. 12, §E.7 · p. 14
E.8 Candidate-Pair Aggregation
- Candidate-pair aggregation can use majority vote, top confidence, or max positive, with specified tie-breaking rules for majority vote.Majority voting requires at least half of candidate pairs to predict a positive relation; otherwise the row receives no_relation.Source: §E.8 · p. 14
E.9 vLLM Inference
- The released Yehia-RE blind-test runner uses one GPU, generates one token, and restricts allowed tokens to codes displayed in each row.Requests are batched up to 512 prompts.Source: §E.9 · p. 14
E.10 Representative Official Results
The paper reports representative official Yehia-RE results and describes Qwen-RE’s ontology-aware causal-LM and classifier formulations. Its development results are explicitly distinguished from official blind scores because the split and evaluation pipeline differ.
E.10 Representative Official Results
- Table 16 summarizes official results for the two Yehia-RE training variants, but the supplied passage gives no scores for either variant.The table caption identifies the results as representative official Yehia-RE results.Source: §E.10 · p. 14, Table 16 · p. 15
F.1 Causal-LM Prompt Template
- Qwen-RE’s causal-LM prompt includes the marked Arabic sentence, entity strings and types, and only ontology-compatible relation labels with Arabic meanings and direction notes.The allowed label set can vary by input row because it is derived from the predicted directed entity-type pair.Source: §F.1 · pp. 15–16
F.3 Generation and Complete-Label Scoring
- Qwen-RE causal-LM inference can score complete candidate labels instead of relying on free-form generation that may produce explanations, punctuation, or incompatible continuations.The candidate set includes no_relation.Source: §F.3 · p. 16
- Normalizing scores over compatible labels does not change the highest-scoring label, although the normalized value depends on candidate-set size and composition..Source: §F.3 · p. 16
F.4 Sequence-Classifier Prompt and Objective
- Qwen-RE-9B-Cls uses a fixed 41-way classification head and masks ontology-incompatible logits before cross-entropy.The classifier and causal-LM formulations are summarized in Table 17.Source: §F.4 · pp. 16–17, Table 17 · p. 17
F.6 QLoRA Fine-Tuning
- Qwen3.5-4B and Qwen3.5-9B are adapted with 4-bit NF4 QLoRA, double quantization, and BF16 computation.LoRA adapters target attention and feed-forward projection modules.Source: §F.6 · p. 17, Table 18 · p. 17
F.7 Validation and Checkpoint Selection
- The best confirmed development micro-F1 was 0.6846 for Qwen-RE-4B and 0.6977 for Qwen-RE-9B.These development results are not directly comparable with official blind scores because they use a different split and evaluation pipeline.Source: §F.7 · p. 17
F.8 Extended Results
Official Qwen-RE scores differ by 0.18 points between Qwen-RE-9B and Qwen-RE-9B-Cls. The appendices report manual relation-disambiguation rules and an exploratory GEPA prompt-evolution experiment.
F.8 Extended Results
- 0.18 points separate Qwen-RE-9B and Qwen-RE-9B-Cls in the official scores.Source: §F.8 · p. 17
G.1 Disambiguation Rules
- The rules distinguish ordinary containment from headquarters and administrative-division relations, preferring specific alternatives only when their evidence appears.Source: §G.1 · p. 18
- The disambiguation rules did not eliminate confusion: headquarters examples could remain labeled as ordinary containment, and longer rules reduced validation performance.Source: §G.1 · p. 18
G.2 GEPA Prompt Evolution
- GEPA used error summaries and a reflective editor to revise prompts, then selected candidates with a Pareto-based procedure.Source: §G.2 · p. 18
- Positive F1 = 0.6011 and accuracy 0.5408 on the 196-example validation subset; no improvement was claimed because the pre-GEPA score was unrecorded.Source: §G.2 · p. 18
H.1 Entity-Type Errors
The discussion identifies risks from entity-type errors, repeated mentions, semantically close relations, and causal label generation. It also lists released artifacts and limits on reconstructing experiment environments.
H.1 Entity-Type Errors
- A hard type error can alter typed markers, the 42-dimensional type vector, and the ontology-compatible relation inventory, including for lower-support types and unseen-sentence mentions.Source: §H.1 · p. 18
H.2 Repeated Mentions
- Repeated surface forms create an occurrence-matching trade-off: nearest-candidate selection can be wrong, while scoring all pairs costs more and adds competing span hypotheses.Source: §H.2 · p. 18
H.3 Semantically Close Relations
- Training support for Location.located_in can favor that generic label when a sentence supports a more specific geographic relation.Source: §H.3 · p. 18
H.4 Causal Generation Errors
- Causal generation may emit extra prose, punctuation, aliases, or partial labels despite assigning probability mass to the correct relation, motivating complete-label scoring and direct classification.Source: §H.4 · p. 18
I.1 Released Resources
- Released resources include TypedPair checkpoints, Yehia models, annotated relation data, and entity-type prediction artifacts.Source: §I.1 · pp. 18–19
I.3 Software, Hardware, and Model Revisions
- The archive lacks a shared environment manifest, and exact Yehia Python/CUDA versions and Qwen's complete software/hardware environment are unrecoverable.Source: §I.3 · p. 19