PaperlayerView exact revision on arXiv

Source-linked AI summary

REGLAT at ArGuard Shared Tasks: An Ensemble-Based Model Integrating NER for Harmful Content Detection in Arabic

Ahmed M. Fetouh, Mariam Labib, Nsrin Ashraf, Omer Dawood, Hamada Nayel

arXiv:2610.37265v1cs.CL

TL;DR

Fine-grained harmful-content detection in Arabic is challenged by dialectal variation and overlapping categories. The paper combines GLiNER entity extraction with MARBERT classification, achieving 0.886 macro-F1 and 0.897 accuracy on the ArGuard development set.

  • Problem

    Arabic harmful-content classification is complicated by dialectal variation, semantic overlap among categories, and prompts whose surface wording may not make their sensitive category apparent.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The system appends GLiNER-extracted harm-related entities to each Arabic prompt and classifies the enriched text with fine-tuned MARBERT.

    Source: §3 · p. 2, §1 · p. 2

  • Results

    On the 2,405-prompt development set, the system achieves a macro-F1 score of 0.886 and an accuracy of 0.897.

    Source: §6.4 · p. 5

  • Takeaways & Limitations

    The results support the feasibility of combining generalist entity extraction with an Arabic pretrained transformer, while category overlap and target identification remain challenges.

    Source: §7 · p. 7

    Evaluation is based on the ArGuard development set; further testing is needed to assess generalization across sources and linguistic conditions.

    Source: §7 · p. 7

Abstract

from arXiv · show

Detecting harmful content in Arabic is challenging because of dialectal variations and the semantic overlap between different harm categories. This paper presents a two-stage approach for classifying Arabic user prompts into seven categories of harmful content. First, GLiNER is used to extract harm-related entities from each prompt. The extracted entities are then added to the original text and passed to a fine-tuned MARBERT classifier for final classification. The approach combines explicit entity information with the contextual understanding of a pretrained Arabic language model. The system was evaluated on the development set of the ArGuard 2026 shared task. The results achieved a macro-F1 score of 0.886 and an accuracy of 0.897, demonstrating promising performance in fine-grained harmful content detection. The system reported an accuracy of 0.793 and Macro-F score of 0.735 on a blind test set for subtask 1. For subtask 2, it achieved an accuracy of 0.770 and Macro-F score of 0.754. The analysis also showed that distinguishing between related categories remains challenging. These findings suggest that integrating NER with Arabic language models is a promising approach to harmful content detection.

1 Introduction

Arabic harmful-content detection is complicated by dialectal variation, overlapping categories, and context-dependent intent. The paper combines harm-related entity extraction with Arabic transformer classification and evaluates the system on ArGuard 2026.

1 Introduction

  • Dialectal variation, informal wording, overlapping category vocabulary, and context-dependent intent complicate fine-grained Arabic harmful-content classification.Class imbalance may also lead to uneven performance across categories.Source: §1 · p. 1
  • The system combines GLiNER entity extraction with fine-tuned MARBERT classification across seven harmful-content categories.Extracted entities are appended to the original prompt before classification.Source: §1 · p. 2
  • The approach makes harm-related entity cues explicit in a structured input alongside contextual language representations.The paper motivates entity information as complementary to contextual representations that may not capture sufficiently explicit entity-level cues.Source: §1 · pp. 1–2
  • Evaluation and error analysis on the ArGuard 2026 development set identify category-specific strengths and persistent confusion among semantically similar classes.The paper highlights confusion among harm to others, bullying, and harassment.Source: §1 · p. 2

2 Task and Data

ArGuard Subtask B2 asks systems to assign Arabic prompts to seven granular harm categories. Its training data is imbalanced, and prompts can be difficult to classify because of semantic overlap and dialectal or idiomatic wording.

2 Task and Data

  • The task assigns each Arabic prompt to one of seven granular harm categories, including adult_content, harm-to-others, self-harm, harassment, FraudandDeception, Bully, and hate_speech.Each sample also has a safe/unsafe label and a dialect tag.Source: §2 · p. 2
  • The training set contains 16,835 prompts, with adult_content the largest class at 4,592 samples (27.28%) and hate_speech the smallest at 943 (5.60%).The reported imbalance ratio is roughly 4.9:1.Source: §2 · p. 2
  • Classification is challenging because categories overlap semantically, harmful intent may appear in dialectal or idiomatic wording, and surface-benign prompts can belong to sensitive categories.Rows with missing values were removed; the labelled development set was used for model selection and evaluation, while the unlabeled test set was reserved for final prediction.Source: §2 · p. 2

3 System Overview

The system appends GLiNER-extracted harm-related entities to each prompt and classifies the enriched text with fine-tuned MARBERT. Its entity labels are organized around harm domains, and its input uses a fixed, explicit representation.

3 System Overview

  • GLiNER extracts harm-related entities, which are appended to the original prompt before a fine-tuned MARBERT classifier predicts a category.The same pipeline is used for training, development, and test data.Source: §3 · p. 2

3.1 Entity Extraction with GLiNER

  • GLiNER accepts arbitrary inference-time labels without NER-specific training data, and the system defines 34 labels aligned with seven harm domains.Detected entities use a confidence threshold of 0.5; duplicates within a label are collapsed by surface form.Source: §3.1 · p. 3

3.2 NER-Augmented Text Representation

  • The representation appends a fixed-order line for each of 34 labels, listing detected entities or “none” after the original prompt.This consistent format makes cue presence or absence visible to the classifier.Source: §3.2 · p. 3

3.3 MARBERT Classifier

  • MARBERT uses a seven-unit classification head and cross-entropy to predict only the granular label, with inputs capped at 256 tokens.The model is pretrained on Modern Standard Arabic and dialectal Arabic text; training uses no class weighting or resampling.Source: §3.3 · p. 3

3.4 Training Configuration

  • The model is evaluated on the development set after each epoch, and training restores the checkpoint with the highest macro-F1.Table 1 is identified as containing the training hyperparameters.Source: §3.4 · p. 4, Table 1 · p. 4

4 Evaluation and Inference

The evaluation uses macro-F1 as its official primary metric and applies the same extraction, text construction, tokenization, and prediction pipeline at test time.

4 Evaluation and Inference

  • Macro-F1 is the official primary metric because it weights all seven categories equally under class imbalance.Accuracy and macro-averaged precision and recall are also reported.Source: §4 · p. 4
  • Per-class precision, recall, F1, and raw and row-normalized confusion matrices are reported to analyze interclass confusion.Source: §4 · p. 4
  • At test time, prompts pass through the same GLiNER extraction, entity-augmented text construction, MARBERT tokenization, and selected-checkpoint prediction pipeline.Predicted identifiers are mapped back to granular label names with the training label encoder.Source: §4 · p. 4

5 Results

The system reaches strong overall development-set performance, but results vary across categories and errors cluster among semantically adjacent harms.

5 Results

  • Macro-F1 of 0.886 and accuracy of 0.897 summarize performance on the 2,405 development prompts.Macro-averaged precision and recall are 0.885 and 0.888, respectively.Source: §5 · p. 4, Table 2 · p. 4
  • Adult_content achieves F1 = 0.987 and hate_speech F1 = 0.923, while harm-to-others is weakest at F1 = 0.815.Bully and harassment reach F1 scores of 0.834 and 0.848, respectively.Source: §5 · p. 4

5.1 Error Analysis

  • Errors concentrate among related categories, especially harm-to-others and self-harm, which are confused in 28 and 27 prompts, respectively.Bully and harassment are also confused in both directions, while adult_content is rarely confused with other classes.Source: §5.1 · p. 4

6 Ablation Study

The development-set analysis reports strong overall scores but uneven performance across categories, with errors concentrated among semantically adjacent labels. Ablations are designed to test NER augmentation and pipeline choices, while the authors note that a direct prompt-only comparison is needed to establish the augmentation’s benefit.

6 Ablation Study

  • The ablations use three random seeds and compare the full system with prompt-only and entities-only inputs, as well as removing one entity-label domain at a time.Macro-F1 is the primary metric, and results are reported as mean and standard deviation.Source: §6 · p. 4, §6.1 · p. 5, §6.2 · p. 5

Serialization

  • Pipeline variants test listing all 34 labels with explicit none values versus a compact format, confidence thresholds of 0.3, 0.5, and 0.7, and maximum lengths of 128, 256, or 512 tokens.The compact format shortens input under the 256-token limit but removes the explicit absence signal.Source: Serialization · p. 5

6.4 Discussion

  • Macro-F1 was 0.886 and accuracy was 0.897 on the 2,405-prompt development set.Macro-averaged precision and recall were 0.885 and 0.888, respectively.Source: §6.4 · p. 5

6.5 Category-Specific Performance

  • adult_content had an F1-score of 0.987, while harm-to-others had 0.815, Bully 0.834, and harassment 0.848.hate_speech scored 0.923, despite having 135 development instances.Source: §6.5 · p. 6

6.6 Analysis of Classification Errors

  • The largest confusion was between harm-to-others and self-harm, with 28 and 27 cross-predictions, respectively.Bully and harassment also overlapped, with 12 bullying instances predicted as harassment and 14 harassment instances predicted as bullying.Source: §6.6 · p. 6
  • adult_content had only 8 of 657 instances assigned to other categories, while shared cues and context complicate distinctions among other labels.Entity extraction identifies relevant expressions, but does not by itself establish the intended target or communicative purpose.Source: §6.6 · p. 6

6.7 Implications of Integrating NER with MARBERT

  • GLiNER extracts harm-related entities, and MARBERT classifies the original prompt together with a structured representation of those entities.Keeping the prompt preserves context because isolated entities may not reveal intent.Source: §6.7 · pp. 6–7
  • The results alone do not establish that entity augmentation outperforms a MARBERT-only baseline; that requires a direct prompt-only comparison under the same conditions.The ablation study also examines entity-label domains, confidence thresholds, serialization formats, and maximum sequence length.Source: §6.7 · p. 7

7 Limitations and Future Directions

The results support the feasibility of combining generalist entity extraction with an Arabic pretrained transformer, but broader applicability remains to be established. The discussion identifies limits in evaluation scope, category discrimination, entity extraction, input length, and practical safety assessment.

7 Limitations and Future Directions

  • Evaluation on the ArGuard development set alone leaves generalization across sources and linguistic conditions unassessed.Further testing on the official test set and additional Arabic harmful-content datasets is needed to assess that generalization.Source: §7 · p. 7
  • Performance differences indicate continued sensitivity to semantic overlap, especially when distinguishing harm-to-others from self-harm and Bully from harassment.Future work could model target and intent, entity relationships, or category-specific training strategies.Source: §7 · p. 7
  • Enriched-input quality depends on GLiNER detections: missed entities can remove useful evidence, while incorrect extractions can add irrelevant information.The fixed serialization of 34 entity labels increases input length, and the representation is sensitive to the selected sequence-length limit.Source: §7 · p. 7
  • Classification performance is not a complete measure of practical safety because errors can carry different consequences across categories and applications.Future evaluations could examine category-specific error costs, calibration, dialectal robustness, and indirectly or ambiguously expressed harmful intent.Source: §7 · p. 7
  • The results support the feasibility of combining generalist entity extraction with an Arabic pretrained transformer, while broader applicability and planned ablation comparisons remain to be established.Error analysis identifies challenges involving target identification, contextual intent, and overlapping category definitions.Source: §7 · p. 7
Loading 2610.37265v1…