Source-linked AI summary
Beyond Poetry: Can Large Language Models Generate Classical Arabic Maqamat?
AbdulRahman A. Morsy, Aya Zirikly
TL;DR
Classical maqama’s demands for saj, rhetorical ornament, and coherent narrative make it a challenging test of LLM literary competence, yet this form remains underexplored. This study evaluates model generation with complementary rubric-based and pairwise judgments, finding that success depends on genre-specific competence beyond fluency.
Problem
LLM generation of culturally grounded, stylistically constrained literary forms such as classical maqama remains underexplored, despite the genre’s demanding conventions.
Source: §1 · p. 1Method
Results
GPT-5.4-mini achieves the strongest results, followed by GPT-4o, while open-weight models lag; prompting particularly improves weaker models’ stylistic dimensions but has limited impact on stronger models.
Source: §7 · p. 7Takeaways & Limitations
Successful maqama generation requires genre-specific stylistic and rhetorical competence beyond grammatical fluency.
Source: §6 · p. 6, §7 · p. 7The evaluation covers short-form generations on 14 curated topics, not the longer narratives and richer discourse development of classical maqama literature.
Source: §8 · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) have shown strong performance in creative text generation, yet their ability to produce culturally grounded and stylistically constrained literary forms remains underexplored. Prior work has focused largely on modern language varieties and poetry, while classical prose traditions such as maqama remain largely unstudied. The maqama is a classical literary genre characterized by rhymed prose (saj), dense rhetorical ornamentation, and episodic narrative structure, making it a challenging testbed for evaluating whether LLMs can move beyond surface fluency toward deeper literary competence. In this paper, we present the first controlled evaluation study of maqama generation with LLMs, comparing five models under zero-shot, few-shot, and rule-based prompting, and evaluating outputs through both human annotation and an LLM-as-a-judge framework across dimensions such as rhetorical richness, saj density, structural coherence, and stylistic authenticity. Our results show that prompting strategy plays a strong role in stylistic quality: few-shot prompting most consistently improves saj density, while its effects on rhetoric and coherence vary by model, with the strongest models (GPT-4o and GPT-5.4-mini) benefiting most from rule-based prompting on these dimensions, though zero-shot prompting yields the highest aggregate scores across all five models. We further observe systematic differences between models in stylistic alignment with Arabic maqama conventions, and corroborate our findings with a second independent LLM judge, paired statistical significance testing, and non-LLM proxy measures of saj.
1 Introduction
Maqama generation tests whether LLMs can meet classical prose conventions that demand more than grammatical fluency. The study evaluates models and prompting strategies on this underexplored genre using human and LLM-based assessments.
1 Introduction
- Maqama generation requires controlled saj, rhetorical devices, and narrative coherence while sustaining a classical register.These demands make it a test of literary competence beyond surface fluency.Source: §1 · pp. 1–2
- Classical Arabic prose remains underexplored in prior work focused mainly on modern varieties and poetry generation.The maqama is a classical prose genre marked by rhymed prose, dense rhetorical ornamentation, lexical ingenuity, and episodic storytelling.Source: §1 · pp. 1–2
- The study presents a controlled evaluation of multiple LLMs and prompting strategies for reproducing maqama’s stylistic and structural properties.It combines human annotation with LLM-as-a-judge assessment across grammar, saj, rhetoric, coherence, creativity, relevance, and authenticity.Source: §1 · p. 2
- The study asks whether LLMs can generate coherent maqama-style prose, how prompts affect quality, which dimensions respond to prompting, and whether evaluation approaches agree.The questions cover genre properties, prompt sensitivity, model dependence, and agreement between rubric-based and pairwise judgments.Source: §1 · p. 2
- The paper contributes the first systematic maqama-generation evaluation, empirical evidence on prompt sensitivity, and released topics, prompts, and outputs for future comparisons.It also frames the study as offering insight into stylistic controllability limits in LLMs.Source: §1 · p. 2
2 Related Work
Prior work addresses controllable generation, Arabic language modeling, and text evaluation, but has not systematically studied LLM generation of classical Arabic rhetorical prose. The related work also notes limits in relying on LLM judges and in transferring general multilingual ability to culturally situated literary forms.
Arabic NLP and classical language modeling
- Arabic NLP work has focused on Modern Standard Arabic, dialects, and poetry, while classical Arabic prose and rhetorical genres remain relatively underexplored.Recent Arabic generation work also primarily addressed controllable poetry or poetic-understanding benchmarks (Sadallah et al., 2026; Al Ghallabi et al., 2025).Source: §2 · p. 2
Evaluation of generated text
- LLM-based evaluation may favor surface fluency and respond to prompt framing when used alone.LLM-as-a-judge frameworks offer scalable evaluation alternatives, but this related work identifies those risks of sole reliance (Zheng et al., 2023).Source: §2 · pp. 2–3
Cultural and stylistic gaps
- General multilingual competence does not necessarily entail cultural competence in historically situated language use.Maqama meaning depends on inherited rhetorical conventions, stylistic rhythm, and cultural-literary context (Bagheri Nezhad, 2026).Source: §2 · p. 3
Gap
- No prior work had systematically studied maqama generation or evaluated LLMs on classical Arabic rhetorical prose.Related work on controllable generation and narrative modeling focused on other settings and left this genre unaddressed.Source: §2 · pp. 2–3
3 Data and Experimental Setup
The paper constructs a controlled synthetic generation suite because no standardized dataset for classical Arabic maqamat is available. It samples five models across 14 thematic prompts and prompting conditions, with five independent generations per configuration.
3 Data and Experimental Setup
- The study uses five proprietary and open-weight LLMs under a common protocol, aiming to represent current systems rather than isolate model scale or training effects.All models use temperature 0.7 and a maximum generation length of 2,400 tokens.Source: §3 · p. 3
- The authors create 14 thematic prompts inspired by classical maqama scenarios, including travel, deception, rhetorical contests, satire, and culturally hybrid situations.The topics draw on themes discussed in Drory (2000) and Hämeen-Anttila (2002).Source: §3 · p. 3
- Each model–topic–prompting-condition configuration yields five independent samples in a synthetic dataset, and the topic list and outputs are released as supplementary material.The released materials are intended to support reproducibility.Source: §3 · p. 3
4 Methods
The methods compare three prompting approaches and combine structured LLM judging with a smaller human pairwise evaluation. The evaluations use absolute scoring and direct comparisons, with randomization and blinding procedures to address positional and cognitive bias.
4.1 Prompting Strategies
- The prompting study compares minimal topic-only instructions, exemplar-based few-shot imitation, and rule-based instructions enforcing maqama conventions.The rule-based prompts specify saj, rhetorical devices, and narrative progression.Source: §4.1 · p. 3
4.2 Automatic Evaluation
- Claude Sonnet evaluates outputs through constrained structured judgments, using both 1–5 absolute ratings across literary dimensions and pairwise comparisons.Pair order and candidate identity are randomized, and evaluations run at temperature 0 to mitigate positional bias.Source: §4.2 · p. 3
4.3 Human Evaluation
- Two native Arabic-speaking annotators with classical-language and literary-studies backgrounds assess a balanced sample of 60 maqamat in 30 pairwise comparisons.They compare different models and prompting conditions, choose a preferred output, and provide brief justifications; presentation is randomized and double-blind, with agreement measured by Cohen’s Kappa.Source: §4.3 · p. 4
5 Results
The results compare models and prompting strategies across absolute and pairwise judgments, human evaluation, and non-LLM proxies. GPT-5.4-mini leads overall, while prompting effects vary by model and dimension.
5.1 Absolute Evaluation
- GPT-5.4-mini outperforms all baselines on maqama-specific dimensions, while GPT-4o's stronger grammaticality and coherence do not extend to comparable authenticity.Arabic-specialized ALLaM also underperforms in most literary dimensions; GPT-5.4-mini has lower Overall variance than GPT-4o (0.38 vs. 0.53).Source: §5.1 · p. 4
5.2 Prompting Effects
- Across all five models, zero-shot prompting has the highest aggregate results on rhetoric and coherence, although GPT-4o and GPT-5.4-mini score best on those dimensions under rule-based or few-shot prompting.Few-shot prompting gives GPT-4o its largest gain (+0.54), while GPT-5.4-mini remains comparatively stable across conditions.Source: §5.2 · p. 4, Table 1 · p. 5
5.5 Human Evaluation
- Human annotators show high agreement and prefer few-shot and rule-based outputs, citing better saj’, rhetorical density, and narrative cohesion; they also find closed-source models outperform open-source alternatives.Some open-source outputs contain repetitive or structurally invalid passages and fabricated religious quotations.Source: §5.5 · pp. 5–6
5.3 Pairwise Comparison
- GPT-5.4-mini wins against every competing system in pairwise judgments, GPT-4o beats all non-GPT baselines, and ALLaM outperforms Qwen and LLaMA.LLaMA's 52.4% win rate over Qwen is not statistically distinguishable from chance (bootstrap 95% CI [0.381, 0.667]).Source: §5.3 · pp. 4–5, Table 4 · p. 6
5.4 Global Ranking
- The Elo ranking separates models into four performance tiers, with a +341 Elo gap between GPT-5.4-mini and GPT-4o.Model differences in Overall scores are significant after Holm-Bonferroni correction, and an independent judge produces strong agreement with the ranking, differing by one adjacent swap.Source: §5.4 · p. 5, Table 5 · p. 6
5.6 Non-LLM Validation of Saj’
- Rhyme-ending consistency correlates with LLM-judge saj’ scores (Spearman’s ρ = 0.328, p < 0.001), while the lightweight proxies are not intended to replace literary evaluation.A fuller computational assessment of saj’ would require richer linguistic modeling.Source: §5.6 · p. 6
5.7 Diacritization
- Few-shot prompting produces more diacritics across evaluated models; for Qwen, density rises from 30.6 to 77.3 marks per 100 Arabic letters.The authors attribute the pattern to imitation of diacritized examples, while evaluating correctness requires gold-standard references and dedicated metrics.Source: §5.7 · p. 6
6 Discussion
The discussion finds that maqama generation demands genre-specific skills beyond general language ability, and that prompting affects models and stylistic dimensions unevenly. It also cautions that model differences may partly reflect instruction-following and pretraining exposure.
6 Discussion
- Strong general language ability is necessary but insufficient for maqama, which requires genre-specific constraints including saj’, rhetorical ornamentation, and stylistic authenticity.Agreement between absolute and pairwise evaluation supports the robustness of the observed ranking across evaluation paradigms.Source: §6 · pp. 6–7
- GPT-5.4-mini performs most strongly and consistently, while GPT-4o’s fluency and coherence do not guarantee authenticity or saj’, and open-weight models perform substantially worse overall.The authors note that these differences may partly reflect instruction-following fidelity and exposure to classical Arabic registers, rather than generative capacity alone.Source: §6 · pp. 6–7
- Prompting benefits are uneven and concentrated in surface features: saj’ is most prompt-sensitive, while creativity and rhetoric remain relatively stable.ALLaM sometimes loses performance under prompting, possibly because imposed constraints interfere with its existing Arabic-generation priors.Source: §6 · p. 7
7 Conclusion
The study evaluates maqama-style Arabic generation through rubric scores and pairwise comparisons, finding a model hierarchy and prompting effects that depend on model strength. It concludes that literary performance requires more than grammatical fluency and identifies longer-form evaluation as future work.
7 Conclusion
- GPT-5.4-mini ranks strongest, followed by GPT-4o, while open-weight models remain significantly behind.The conclusion frames these results as a clear performance hierarchy.Source: §7 · p. 7
- Prompting can improve weaker models, particularly on saj’, but has limited impact on stronger models.The conclusion also notes that maqama generation requires stylistic and rhetorical competence beyond grammatical fluency, which remains difficult for most models.Source: §7 · p. 7
- Future work could evaluate longer maqamat with richer narrative structure and stronger alignment to classical exemplars.The proposed setting would assess both local stylistic fidelity and long-range literary coherence.Source: §7 · p. 7
8 Limitations
The study evaluates short-form generations on 14 curated topics, relies mainly on LLM judging with limited human evaluation, and estimates pairwise results from a sample rather than all possible pairs.
8 Limitations
- Short-form generations across 14 curated topics do not capture classical maqama’s longer narratives and richer discourse development.The authors identify this as a scope boundary for interpreting the evaluation.Source: §8 · p. 7
- LLM-judge results may reflect model-specific bias, and the human evaluation is too small to serve as a full reference standard.The evaluation relies mainly on LLM-as-a-judge, supported by pairwise comparisons and limited human evaluation.Source: §8 · p. 7
- Pairwise results use 420 stratified comparisons rather than all possible pairs, leaving potential sampling variance.The authors flag the sample-based computation as a limitation.Source: §8 · p. 7
Ethics Statement
The authors caution that generated culturally and historically grounded narratives may reproduce simplified or stereotypical cultural representations.
Ethics Statement
- Generated narratives may reproduce simplified or stereotypical cultural representations, and reflect statistical learning rather than historical understanding.The authors report observing no direct harmful outputs but urge care in interpreting the narratives.Source: Ethics Statement · p. 7
AI Usage Statement
This material documents manuscript-preparation assistance, presents a canonical maqama example, and provides prompting and evaluation materials alongside additional analyses.
AI Usage Statement
- ChatGPT assisted with manuscript writing support, language refinement, and verification of parts of the reported analyses; the authors reviewed and validated generated content.The authors state that they conceived and conducted the hypotheses, experimental design, implementation, and final scientific conclusions.Source: AI Usage Statement · p. 7
A Canonical Classical Maqama Example
- The al-Hamadhani excerpt illustrates framed narration, performed eloquence, saj’, and rhetorical embellishment as features informing evaluation dimensions.The passage connects these features to authenticity, rhetoric, coherence, and saj’ density.Source: A Canonical Classical Maqama Example · pp. 8–9
B Generation Prompt Templates
- The prompt templates are provided in Arabic and English for reproducibility.The appendix identifies them as the exact templates used for synthetic maqama generation.Source: B Generation Prompt Templates · p. 9
C Evaluation Protocol
- The evaluation protocol uses a two-stage LLM-as-a-judge pipeline implemented with Claude Sonnet 4.5 at temperature zero.Source: C Evaluation Protocol · p. 10
Arabic
- The Arabic subsection includes evaluation criteria and rubric material in Arabic.Source: Arabic · pp. 10–11
English Translation
- The English prompt asks for judgments against classical maqama conventions and criteria including saj’, narrative coherence, language, rhythm, and rhetoric.It directs the evaluator to record scores after delivering a literary judgment.Source: English Translation · p. 11
D Additional Evaluation Analyses
- The appendix’s additional analyses examine model behavior across stability, prompting strategies, pairwise judgments, and fine-grained literary dimensions.These analyses are described as complementing the main findings; this passage does not state their results.Source: D Additional Evaluation Analyses · p. 12
D.1 Stability Analysis
Across models, saj’ is the most prompt-sensitive dimension, while rhetoric and creativity vary minimally with prompting. Model-level analyses show differing stability and prompt responsiveness, alongside perfect agreement between absolute and pairwise rankings.
D.1 Stability Analysis
- GPT-5.4-mini achieves the highest mean among strong-performing models with relatively low variance; open-weight models show lower performance and reduced stability.The analysis measures mean and standard deviation of overall scores.Source: §D.1 · p. 12, Table 6 · p. 12
D.2 Prompt Sensitivity
- Prompt sensitivity varies by model: GPT-4o and Qwen benefit substantially from structured prompting, GPT-5.4-mini stays stable, and ALLaM can degrade under few-shot prompting.ALLaM shows only marginal gains overall from prompting.Source: §D.2 · p. 12, Table 7 · p. 12
D.3 Agreement Between Evaluation Protocols
- Absolute scoring and pairwise ranking have a perfect monotonic correlation, yielding fully consistent rankings across the two evaluation protocols.Agreement is measured with Spearman’s rank correlation coefficient.Source: §D.3 · p. 12, Table 8 · p. 12
D.4 Dimension Difficulty
- Grammar is relatively well handled across models, whereas authenticity and creativity remain significantly underdeveloped.Scores are averaged across models and prompting conditions.Source: §D.4 · p. 12, Table 9 · p. 12
D.5 Model-wise Prompt Dynamics
- GPT-5.4-mini is largely prompt-invariant but improves in saj’ and creativity with few-shot and rule prompting; GPT-4o gains in saj’ and rhetoric with few-shot prompting.ALLaM can degrade under few-shot prompting, while LLaMA and Qwen show unstable, limited consistent gains.Source: §D.5 · p. 13, Table 10 · p. 14
D.6 Dimension-level Prompt Sensitivity
- Saj’ is the most prompt-sensitive dimension, while rhetoric and creativity show minimal variation, indicating deeper literary competence is more model-dependent than prompt-driven.Grammar and coherence show moderate prompt sensitivity.Source: §D.6 · p. 13, Table 11 · p. 14