Source-linked AI summary
Instruction-Guided Poetry Generation in Arabic and Its Dialects
Abdelrahman Sadallah, Kareem Elozeiri, Mervat Abassy, Rania Elbadry, Mohamed Anwar, Abed Alhakim Freihat, Preslav Nakov, Fajri Koto
TL;DR
Arabic poetry LLM research has focused mainly on analysis, leaving controllable generation across constraints and dialects underdeveloped. The paper builds a large instruction dataset covering creative and analytical tasks, fine-tunes four LLMs, and reports improved alignment and poetry quality through automated and human evaluation.
Problem
Arabic poetry LLM research has largely emphasized analysis and classification, with limited controllable generation supporting style, meter, rhyme, and dialect constraints.
Method
The paper constructs an instruction-finetuning dataset with four task families, 54 subtasks, and 3,220 templates spanning MSA and four Arabic dialects, then fine-tunes four LLMs.
Results
Instruction fine-tuning significantly improves analytical accuracy and output quality across automatic metrics, LLM judging, and human evaluation.
Takeaways & Limitations
The dataset supports controllable Arabic poetry generation, continuation, revision, and analysis across structural, stylistic, and dialectal requirements.
Takeaways & Limitations
The approach uses LoRA rather than full fine-tuning, which may limit learning complex poetic structures and stylistic nuances.
Abstract
from arXiv · showhide
Poetry has long been a central art form for Arabic speakers, serving as a powerful medium of expression and cultural identity. While modern Arabic speakers continue to value poetry, existing research on Arabic poetry within Large Language Models (LLMs) has primarily focused on analysis tasks such as interpretation or metadata prediction, e.g., rhyme schemes and titles. In contrast, our work addresses the practical aspect of poetry creation in Arabic by introducing controllable generation capabilities to assist users in writing poetry. Specifically, we present a large-scale, carefully curated instruction-based dataset in Modern Standard Arabic (MSA) and various Arabic dialects. This dataset enables tasks such as writing, revising, and continuing poems based on predefined criteria, including style and rhyme, as well as performing poetry analysis. Our experiments show that fine-tuning LLMs on this dataset yields models that can effectively generate poetry that is aligned with user requirements, based on both automated metrics and human evaluation with native Arabic speakers. The data and the code are available at https://github.com/mbzuai-nlp/instructpoet-ar
1 Introduction
Arabic poetry is culturally central but underrepresented in LLM research, which has emphasized analysis rather than controllable creation. The paper addresses this gap with a large instruction dataset and evaluates fine-tuned models across Arabic varieties and poetic tasks.
- Arabic poetry preserves linguistic norms, collective memory, and social and emotional expression across classical and modern traditions.
- Existing Arabic-poetry LLM research mainly targets analysis and classification, while controllable generation for style, meter, rhyme, and dialect remains limited.
- The authors aggregate and standardize a large, diverse corpus spanning eras, genres, and linguistic varieties for consistent downstream use.
- The instruction dataset covers four tasks and 54 subtasks with 3,220 templates across MSA and four major dialects.
- The study fine-tunes four LLMs using joint and curriculum-based training regimes to assess poetic structure, stylistic constraints, and dialectal variation.
2 Related Work
Arabic poetry research spans analysis and generation, but generation has historically traded off fluency and controllability. Recent work introduces planning, user controls, and alternative modeling strategies across Arabic and other poetic traditions.
- Arabic poetry generation has largely remained uncontrolled and restricted to Modern Standard Arabic, overlooking dialectal diversity and user-specified constraints.
- Poetry-generation research has evolved from evolutionary algorithms to neural architectures, with continuing trade-offs between fluency and controllability.
- Planning-based architectures separate global thematic planning from line-level realization to address thematic drift and improve long-range coherence.
- Recent systems reduce reliance on rigid templates by allowing natural-language prompts or control codes, alongside Arabic rhythm-aware Transformer approaches.
- Other work explores token-free decoding and character-by-character generation for English, German, and classical Chinese poetry.
3 Dataset Construction
The dataset pipeline unifies and enriches Arabic poetry from public sources, removes duplicates and leakage, and converts the material into dialect-aware instruction tasks. Its four task families cover creative generation, continuation, revision, and metadata analysis.
- Dataset pipeline: Raw poetry sources are unified, normalized, enriched with metadata, deduplicated, and transformed into instruction templates for MSA and four dialects.
- Data sources: The corpus draws mainly from Arabic poetry websites and spans classical, medieval, and modern eras, with mostly MSA and some regional dialects.
- Data unification: Poems are standardized to one verse per line, while single-verse poems are removed because they provide limited learning value and introduce noise.
- Metadata enrichment: Metadata normalization includes automatic rhyme labeling when at least 70% of verses share the same ending, plus generated keywords and keyphrases.
- Data quality: Deduplication removes intra-source duplicates and training examples overlapping the FannOrFlop benchmark after normalized string matching.
- Task design: The four task families comprise constrained poem generation, partial-poem continuation, corruption repair, and multiple-choice metadata analysis.Continuation examples use random 10%–90% splits, while revision corruptions include altered wording, disrupted meter, missing verses, and syntactic errors.
- Dataset composition: Generation, continuation, and analysis each contain over 427K training samples, whereas revision contains 68K samples across task-specific subtasks.
- Template design: Templates express equivalent requests through varied phrasings, and the dataset contains 246 generation, 176 continuation, 214 analysis, and 8 revision templates.
4 Experimental Setup
The experiments adapt four instruction-tuned Arabic-centric and multilingual models with LoRA under joint or curriculum learning. Evaluation combines automated task assessments, an external LLM judge, and blind human ratings of generated poetry.
- Models and adaptation: LoRA fine-tuning compares two Arabic-centric and two multilingual instruction-tuned models to assess Arabic specialization and poetic control.
- Training regimes: Training uses a causal language-modeling objective with frozen base parameters and compares randomly shuffled joint training against fixed-order curriculum learning.
- Evaluation: The evaluation framework combines LLM-as-a-judge, lm-eval-harness, automated assessment, and human evaluation across subtasks.
- LLM evaluation: Gemini 2.5 Flash rates generation, continuation, and revision using task-specific prompts and four 1–5 dimensions: compliance, fluency, coherence, and poetic quality.
- Analysis evaluation: Analysis is evaluated as a completion task by selecting the answer choice with the highest likelihood in lm-eval-harness.
- Human evaluation: Human evaluation compares four base and fine-tuned model variants on 400 generated samples rated blindly by two Arabic-speaking poetry annotators.
5 Results
Fine-tuning improves Arabic poetry performance across dialects, models, and tasks, but results vary by task difficulty and evaluation criterion. Human evaluation shows substantial gains over base models, while poetic quality remains the main weakness.
- Automatic Metric Evaluation: Fine-tuning improves performance across MSA and dialects, without a dialect being consistently easiest or hardest.
- Automatic Metric Evaluation: Generation scores highest, continuation lower, and revision lowest across compliance, fluency, coherence, and poetic quality.Generation offers more freedom, whereas continuation and revision require preserving context, meaning, style, meter, or structural constraints.
- Analysis: Analysis accuracy improves substantially over base models across model families, with LLaMA-3-8B and Qwen3-8B gaining over 30 points.Random corruption usually slightly outperforms curriculum learning, although the differences are small.
- Human Evaluation: 0.97 points (+32%) raises ALLaM-7B from 3.02 to 3.99, while Qwen3-8B gains 1.42 points (+63%), from 2.24 to 3.66.All observed model differences are statistically significant under ANOVA, with p < 0.0001 for every criterion.
- Human Evaluation: 3.99/5.0 makes fine-tuned ALLaM-7B the top model across compliance, fluency, coherence, and poetic quality, ahead of fine-tuned Qwen3-8B at 3.66/5.0.Among base models, ALLaM-7B scores 3.02 versus 2.24 for Qwen3-8B.
- Human Evaluation: 3.82/5.0 is the best reported Poetic Quality score, while Fluency is consistently strongest and poetic creativity remains difficult.Human annotators showed substantial overall agreement: Pearson r = 0.58, Spearman ρ = 0.58, and quadratic weighted κ = 0.57.
6 Conclusion and Future Work
The work introduces an instruction-following framework for controllable Arabic poetry creation and understanding across MSA and four dialect groups. It reports improved analytical accuracy and output quality, while identifying broader literary coverage and training approaches for future work.
- The framework treats Arabic poetry creation and understanding as controllable, user-driven tasks.
- Its dataset covers generation, continuation, revision, and analysis across MSA and four major dialect groups.
- Instruction fine-tuning significantly improves analytical accuracy and output quality across automatic metrics, LLM-as-a-judge, and human evaluation.
- The dataset grounds instructions in metadata and curated templates for structural, stylistic, and dialectal features including meter, rhyme, genre, and era.
- Future Work: Future work will expand literary capabilities and coverage to contemporary poetry, free verse, and dialectal compositions.
- Future Work: The authors also plan to add modern and natively dialectal poetry and investigate full fine-tuning, larger models, and prosody-aware objectives.
Limitations
The study's limitations concern adaptation strategy, model-scale coverage, and the composition of its poetic data. These constraints leave open questions about complex poetic structures, scale effects, and modern linguistic diversity.
- LoRA-based adaptation may limit capture of complex poetic structures and stylistic nuances compared with full fine-tuning.
- The experiments use a small set of mid-sized models and omit smaller variants, limiting systematic analysis of model-size effects.
- Although instructions span dialects, the poetic content is largely historical and predominantly in MSA.
Ethics and Broader Impact
The dataset uses publicly available Arabic poetry without private or sensitive information, but source imbalance and output attribution risks require caution. Canonical sources may underrepresent contemporary, dialectal, and community poetry.
- Ethics: The dataset consists of publicly available Arabic poetry, with no private or sensitive information used.
- Ethics: Generated poems may be misinterpreted as human-authored or incorrectly attributed to real poets, requiring clear disclosure.
- Broader Impact: Public sources are skewed toward canonical forms and may underrepresent contemporary, dialectal, or community poetry.
- Broader Impact: Such coverage gaps may affect model behavior on underrepresented genres, motivating more diverse data and inclusive benchmarks.
- Broader Impact: The authors expect the work to support research, education, and creative exploration in Arabic language and literature.
A Additional Data Statistics
The appendix describes corpus-level and instruction-level dataset statistics, templates, annotation procedures, and detailed analysis-task results. It reports substantial fine-tuning gains, with structurally grounded tasks easier than semantically deeper identification tasks.
- Corpus Statistics: Corpus statistics summarize poetic meter, poet era, and genre, with Table 7 listing the top 10 values for each category.
- Instruction Statistics: Instruction statistics describe sample distributions across analysis, generation, continuation, and revision, with detailed subtask breakdowns in Tables 8–11.
- Templates and Samples: Table 12 provides one template per task, while Figure 3 shows task samples with their inputs and outputs.
- Annotation: Four native annotators, each representing a different regional Arabic dialect, contributed to template generation.
- Evaluation: Human evaluation used two native Arabic speakers familiar with Arabic poetry and literary texts.
- Analysis Results: Fine-tuning substantially improves all model families, approaching perfect performance on meter and rhyme prediction while genre and poet identification remain harder.
F Automatic Evaluation
Automatic evaluation measures analysis accuracy, semantic similarity, lexical overlap, and rhyme adherence across task and model variants. Fine-tuning improves rhyme adherence and generally preserves or improves semantic alignment, while some poetic attributes remain difficult to assess automatically.
- Evaluation Metrics: Table 16 evaluates analysis with multiple-choice accuracy and generation-related tasks with BERTScore, ROUGE-L, and rhyme adherence.Generation also includes additional key-phrase evaluation.
- Structural Fidelity: Fine-tuning substantially improves rhyme adherence across model families, indicating stronger compliance with structural constraints.
- Structural Fidelity: Generation tasks show particularly large rhyme-adherence gains because baseline models often struggle with rhyme constraints.
- Content Fidelity: Most models also improve in BERTScore, suggesting that stronger structural adherence does not reduce overall semantic quality.
- Evaluation Limitations: Meter and poet era are difficult to evaluate with rule-based methods, so comprehensive assessment relies primarily on human judgments and LLM-based evaluators.Automatic metrics provide complementary evidence for structural fidelity and content quality.
G Analysis
Fine-tuned models consistently outperform base models across generation, revision, continuation, and dialectal settings. Gains are strongest for concrete structural or identity constraints and for tasks requiring sustained poetic consistency.
- Cross-Task Results: Fine-tuning consistently improves LLM-as-a-judge performance across generation, revision, and continuation subtasks, with gains varying by task complexity and training strategy.
- Generation: The fine-tuned ALLAM-7B-INSTRUCT outperforms its base model across most generation settings involving genre, meter, rhyme, poet identity, and combinations.
- Generation: Fine-tuning is especially effective for concrete constraints such as poem titles, poet names, and rhyme, while gains are less visible for broad meter or genre cues.
- Revision: The randomly fine-tuned LLAMA-3-8B outperforms its base model across all revision corruption categories, with the largest improvements on rhyme-related corruptions.These include rhyme_content and rhyme_substitution.
- Continuation: The randomly fine-tuned QWEN-3-8B consistently outperforms its base model when continuing verses under meter, rhyme, genre, poet-era, and poet-identity conditions.
- Continuation: Continuation gains are especially notable when maintaining long-range meter, era, and rhyme consistency, supporting sustained global poetic structure.
- Dialectal Analysis: Fine-tuning improves performance across MSA and Gulf, North African, Levantine, and Nile Valley dialects, although gain magnitudes vary.
- Dialectal Analysis: ALLAM-7B-INSTRUCT and QWEN-3-8B lead among base models; after fine-tuning, ALLAM remains dominant while LLAMA surpasses QWEN.