Source-linked AI summary
AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task
Mo El-Haj, Saad Ezzini, Shadi Abudalfa, Mustafa Jarrar, Nguyen Minh Chi, Nguyen Minh Quan
TL;DR
AraGenre addresses the challenge of generalising Arabic genre classification beyond limited annotated examples and fixed training labels. It uses natural-language definitions to evaluate broad and specific genre predictions across controlled and naturally occurring data; participating systems identified broad genres more reliably than specific ones.
Problem
Arabic genre classification faces limited, unevenly distributed annotated data, while systems must generalise beyond the classes and linguistic patterns observed during training.
Source: §1 · p. 1Method
Results
Across 17 final teams, mean Broad Macro F1 was 0.6509 and mean Specific Macro F1 was 0.3238, a gap of 0.3271.
Source: §6 · p. 4Takeaways & Limitations
Near-ceiling performance on controlled examples does not guarantee generalisation to unseen genres and noisy naturally occurring text.
Source: §6 · p. 4Development and final scores are not directly comparable, and the lack of standardised system descriptions limits controlled comparison of architectures, external resources, and computational cost.
Source: Limitations · p. 4
Abstract
from arXiv · showhide
AraGenre is a shared task on hierarchical, definition-guided Arabic genre classification, motivated by the limited availability of annotated data in Arabic and other low-resource languages. Systems assign each Arabic text segment both a broad communicative genre and a fine-grained specific genre. The released training and development sets contain limited, primarily synthetic and controlled examples, whereas the hidden final benchmark contains noisier naturally occurring text spanning Modern Standard Arabic, Classical Arabic, and multiple dialects. Participants received natural-language definitions for 74 previously unseen specific genres, creating a zero-shot label generalisation setting in which systems had to infer class semantics rather than memorise fixed label-feature associations. The task attracted 46 registrations and 373 submissions, with 17 teams completing the final evaluation. Thakaa ranked first with a Hierarchical Macro F1 of 0.7352, followed by HoangPhong (HP) with 0.7169 and NAMAA with 0.7013. The results show strong broad-genre recognition but a substantial gap in fine-grained classification under linguistic and domain variation.
1 Introduction
AraGenre frames Arabic genre classification as hierarchical semantic generalisation, using genre definitions to evaluate transfer to unseen specific genres. It pairs limited controlled supervision with a naturally occurring, distribution-shifted evaluation benchmark.
1 Introduction
- Arabic genre classification is challenging because diglossia, dialect variation, optional diacritisation, non-standard orthography, and diverse writing traditions coexist.These challenges matter in low-resource NLP, where annotated datasets are often small, domain-specific, and uneven across linguistic varieties.Source: §1 · p. 1
- AraGenre’s final evaluation tests transfer from class descriptions and limited demonstrations rather than memorised label–feature associations.It introduces naturally occurring text and variation in domain, dialect, register, and orthography.Source: §1 · p. 1
- The task combines a two-level Arabic genre taxonomy with definition-guided zero-shot generalisation to specific genres absent from labelled training data.Its evaluation pairs controlled low-resource training with a naturally occurring, distribution-shifted benchmark.Source: §1 · p. 1
2 Motivation and Related Work
Earlier Arabic genre resources provide supervised benchmarks, but typically assume a fixed label inventory known during training. AraGenre instead tests definition-guided transfer to 74 previously unseen specific genres.
2 Motivation and Related Work
- Ferhat et al. (2024) introduced the Arabic Functional Text Dimensions Corpus, with 3,400 documents across 17 categories, while earlier Arabic resources generally assume known training genres.El-Halees (2017) and Elnagar et al. (2020) classified Arabic documents using stylistic and linguistic features.Source: §2 · pp. 1–2
- AraGenre evaluates transfer to 74 specific genres represented only by their names, definitions, and positions within the broad taxonomy.This differs from fixed-label benchmarks and tests whether systems model communicative function rather than rely on lexical memorisation.Source: §2 · p. 2
- The task relates to label-description learning (Gao et al., 2023) and zero-shot hierarchical classification, while requiring alignment of Arabic text with English definitions.The benchmark also tests robustness under distribution shift.Source: §2 · p. 2
3 Data Collection and Creation
AraGenre assembles Arabic text segments from varied corpora and communicative settings, then supplies reviewed natural-language definitions for a hierarchical genre taxonomy. The final evaluation includes 74 specific genres absent as labelled classes from the released data.
3 Data Collection and Creation
- Each instance is an Arabic text segment, from a short fragment to a longer passage, and the hidden final benchmark uses noisier naturally occurring text than the controlled training and development sets.The data cover Modern Standard Arabic, Classical Arabic, and multiple dialects, with formal and informal writing and diacritised and undiacritised text.Source: §3 · p. 2
- The six broad genres are Informative, Creative, Interactive, Learning, Legal, and Religious, each containing finer-grained communicative genres.Source materials span news, finance, recruitment, literature, song lyrics, online discussion, dialectal writing, and summarisation.Source: §3 · p. 2
- Organisers drafted definitions with ChatGPT 5, then reviewed, corrected, and manually post-edited them to represent communicative function and distinguish neighbouring genres.Definitions were intended to remain informative for definition-guided classification.Source: §3 · p. 2
- The final evaluation adds 74 specific genres absent as labelled classes in released data; participants receive their English names and definitions but no labelled examples.Examples include Sudanese song lyrics and administrative and secretarial jobs, testing transfer across genres, domains, dialects, and writing conventions.Source: §3 · p. 2
4 Task Description and Evaluation
Systems predict both broad and specific genres, with the official Hierarchical Macro F1 weighting the two levels equally. The evaluation expands from limited controlled supervision to a large set of naturally occurring text and 74 unseen specific genres.
4 Task Description and Evaluation
- Each system outputs both a broad genre label and a specific genre label, but the scorer evaluates those predictions independently and does not enforce hierarchical consistency.Each specific genre belongs to one of six broad branches.Source: §4 · p. 3
- The final evaluation contains 27,972 naturally occurring Arabic text segments and 74 previously unseen specific genres, for which participants receive names and definitions but no labelled examples.The released training set has 140 examples across seven specific genres and five broad genres; development has 110 examples across six specific genres and all six broad genres.Source: §4 · p. 3
- The final inventory distributes its 74 specific genres unevenly: 42 are Informative, 12 Learning, seven Creative, six Religious, four Interactive, and three Legal.Examples span news, job advertisements, online comments, textbooks, constitutions, and religious texts.Source: §4 · p. 3
- Development and final scores do not estimate performance on the same distribution because the final phase changes genre inventory, domain, dialect, register, orthography, and dataset scale.Table 1 summarizes the number of instances and distinct genre labels in each split.Source: §4 · p. 3, Table 1 · p. 3
- The official Hierarchical Macro F1 gives equal weight to high-level communicative classification and fine-grained genre recognition.Performance is also calculated separately for broad and specific genres using Macro F1, Weighted F1, and accuracy.Source: §4 · p. 3
5 Baselines
Baseline 1 classifies Arabic text by comparing multilingual sentence embeddings of the input and genre definitions.
5 Baselines
- Baseline 1 embeds each input text and genre definition using a multilingual sentence encoder, then selects labels according to semantic similarity, following the sentence-embedding paradigm of Reimers and Gurevych.Source: §5 · p. 3
R Team Hier. Spec. Broad
The released baselines use semantic similarity and retrieval-augmented generation, and Baseline 1 outscored Baseline 2 on development data.
R Team Hier. Spec. Broad
- Table 2 reports official final results using Hierarchical, Specific, and Broad Macro F1 measures.The caption abbreviates these measures as Hier., Spec., and Broad.Source: Table 2 · p. 4
- Baseline 2 combines retrieved demonstrations and genre definitions in a prompt to an instruction-tuned multilingual language model.The approach uses retrieval-augmented generation, following Lewis et al. (2020).Source: R Team Hier. Spec. Broad · p. 4
- 0.4658 Hierarchical Macro F1 for Baseline 1 exceeded 0.3899 for Baseline 2 on development; Baseline 1 scored 0.2472 in the final phase.The final evaluation expanded from six development genres to 74 unseen specific genres, from 110 to 27,972 instances, and from controlled examples to noisier naturally occurring Arabic text.Source: R Team Hier. Spec. Broad · p. 4, Table 2 · p. 4
6 Participation and Results
Participating systems performed better on broad than specific genres, while development-to-final scores dropped sharply; the report notes that detailed error analysis requires additional system materials.
6 Participation and Results
- AraGenre received 46 registrations and 373 submissions, and 17 teams submitted final predictions.Verified official scores are reported without assigning unconfirmed architectures or external resources to teams because the exported leaderboard omitted mandatory fact sheets.Source: §6 · p. 4
- 0.6509 mean Broad Macro F1 versus 0.3238 mean Specific Macro F1 yielded an average gap of 0.3271 across teams.All 17 teams exceeded the organiser baseline; Thaka ranked first because its stronger specific-genre score outweighed HP’s slightly higher broad-genre score.Source: §6 · p. 4
- 0.9344 mean best development score fell to 0.5261 mean final score for 13 teams matched across phases, a reduction of 0.4082.The phases differ intentionally and should not be treated as samples from one distribution; the contrast shows that near-ceiling controlled-example performance does not guarantee generalisation to unseen genres and noisy natural text.Source: §6 · p. 4
- Neighbouring definitions, short context-poor segments, dialect- or domain-specific cues, and texts with multiple communicative functions are expected to be challenging.Confirmed instance-level error analysis requires participant predictions and system descriptions.Source: §6 · p. 4
7 Conclusion
AraGenre evaluates hierarchical, definition-guided Arabic genre classification across linguistic and distributional variation, with fine-grained semantic generalisation remaining challenging.
7 Conclusion
- Seventeen teams completed the final evaluation, and Thaka achieved the highest Hierarchical Macro F1 of 0.7352.The final systems were tested on 74 specific genres absent from released labelled data and represented only through English definitions.Source: §7 · p. 4
- AraGenre combines six broad genres with a changing fine-grained label space and evaluates systems under realistic linguistic and distributional variation.The final test uses 74 unseen specific genres represented only through English definitions.Source: §7 · p. 4
- The broad–specific performance gap and development-to-final reduction indicate that fine-grained semantic generalisation remains challenging.The conclusion calls for more naturally occurring labelled data, parallel Arabic and English definitions, hierarchy-aware consistency constraints, and systematic analysis across dialects, text lengths, and overlapping communicative functions.Source: §7 · p. 4
Limitations
The section flags limits on score comparability and architecture-level interpretation, then sketches participating teams’ varied strategies for hierarchical Arabic genre classification.
Limitations
- Development and final scores are not directly comparable as single-distribution estimates; architecture comparisons are also limited by nonstandard system descriptions, interpretative genre boundaries, and possible multi-function texts.The evaluation sets intentionally differ in naturalness and label coverage.Source: Limitations · p. 4
A.1 Thakaa / A.2 HP
- Thakaa pairs a family-restricted definition-guided LLM judge with agreement-gated correction and attractor-drain redistribution; HP pseudolabels with Gemini before fine-tuning MARBERTv2 and AR-BERTv2.Thakaa’s correction models view the full 74-genre taxonomy; HP trains on approximately 28,000 generated labels.Source: §A.1 · pp. 6–7, §A.2 · p. 7
A.3 NAMAA Community / A.4 GHADAI
- NAMAA uses zero-shot DeepSeek with definitions, self-consistency, and disagreement resolution; GHADAI retrieves with multilingual E5, then reranks candidates with Qwen2.5-32B and cyclic option rotation.NAMAA includes a Claude-assisted pass for disagreements.Source: §A.3 · p. 7, §A.4 · p. 7
A.5 Nasaq / A.8 MohannadHendi
- Nasaq retrieves GPT-4o-generated Arabic prototypes with BGE-M3 and selectively applies GPT-5; MohannadHendi contrastively aligns Arabic texts with deterministic English-definition views for nearest-definition inference.Source: §A.5 · p. 7, §A.8 · p. 7
A.9 WathiqTeam
- Wathiq combines multilingual E5, Arabic prototype retrieval, and BM25 in a broad-first pipeline with reciprocal-rank fusion, hierarchical gating, and selective Qwen3 reranking.Source: §A.9 · p. 7
A.10 Rabee Al-Qasem
- Rabee Al-Qasem combines multilingual MPNet and genre-definition representations with a Conformer projection, broad-genre supervision, synthetic augmentation, and seed-based ensembling.Source: §A.10 · p. 7
A.11 Northwestern
- Northwestern encodes Arabic texts and English genre definitions with multilingual BGE-M3, assigning labels by cosine similarity without task-specific fine-tuning or additional supervised training.Source: §A.11 · p. 7
B Examples of Broad and Specific Genre
The supplied material pairs broad and specific genre examples with shortened definitions and describes two approaches that use retrieval or refined genre representations.
B Examples of Broad and Specific Genre
- Phoenix combines multi-view multilingual retrieval and per-genre z-normalisation with calibrated Qwen3-8B reranking, score fusion, sibling reranking, metric-aware decoding, and candidate expansion.Source: B Examples of Broad and Specific Genre · p. 7
A.7 SAGE
- SAGE fine-tunes sentence encoders for genre similarity and adds a residual adapter to reduce the representational gap between English genre descriptions and Arabic text instances.Source: §A.7 · p. 7
- Table 3 presents illustrative broad–specific genre pairs and shortened definitions.Source: Table 3 · p. 7
- Table 4 gives broad and specific genres with definitions and illustrative Arabic samples.Source: Table 4 · p. 8