PaperlayerView exact revision on arXiv

Source-linked AI summary

POEMetric: The Last Stanza of Humanity

Bingru Li, Han Wang, Hazel Wilkinson

arXiv:2604.03695v1cs.CL

TL;DR

LLMs can generate poetry, but existing evaluations have not fully captured advanced poetic qualities. POEMetric evaluates these qualities across human and LLM poems, finding that top models follow forms and themes yet remain behind human poets in advanced creativity and expression.

  • Problem

    Existing poetry metrics emphasize meter, rhyme, formal diversity, fluency, or coherence while underassessing advanced poem-specific qualities such as creativity, emotion, imagery, and literary devices.

  • Method

    POEMetric evaluates instruction following, advanced creative abilities, and overall appraisal using annotated fixed-form human poems and poems generated by 30 LLMs.

  • Results

    Top LLMs achieved strong form accuracy, reaching 4.26 out of 5.00, but human poems excelled in creativity, emotional resonance, imagery, and literary devices.

  • Takeaways & Limitations

    Poetry generation remains a formidable challenge for LLMs despite their ability to write poems in particular styles and themes.

  • Takeaways & Limitations

    Human experts validated only a smaller representative sample because full-dataset literary annotation was resource-intensive and required scarce domain expertise.

Abstract

from arXiv · show

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

1 INTRODUCTION

The paper introduces POEMetric, a comprehensive framework for evaluating whether LLMs can move beyond formal poetry generation toward creativity, emotional resonance, and artistic expression. It also contributes an annotated human-poem dataset and experiments with 30 LLMs under matched forms and themes.

  • Motivation: LLM poetry generation remains challenging because formal accuracy does not guarantee creativity, diversity, artistic beauty, or expression of authorial intentions and emotions.The paper frames this gap as a limitation of existing evaluations focused primarily on meter and rhyme.
  • POEMetric: POEMetric evaluates basic instruction-following, advanced creative abilities, and general appraisal through 10 metrics.The metric set includes form accuracy, theme alignment, creativity, lexical diversity, idiosyncrasy, emotional resonance, literary devices, imagery, overall poem quality, and authorship estimation.
  • Data and evaluation: The authors curated 203 English poems spanning 7 fixed forms and annotated their forms, meter, rhyme patterns, themes, and imagery.The collection spans the past 200 years and includes canonical and less-known recent works.
  • Data and evaluation: Using the human data’s forms and themes, the study prompted 30 LLMs to generate poetry for comparison with human-authored poems.Evaluation combines rule-based analysis and LLM-as-a-judge, with results validated by human experts.
  • Findings: 4.26 and 4.99 out of 5.00 were Gemini-2.5-Pro’s top scores for form accuracy and theme alignment, respectively.These results indicate strong performance on basic instruction-following dimensions, while the paper investigates broader poetic abilities.
  • Reproducibility: The paper also provides an algorithm for automatically detecting formal poetic patterns and releases code and the public-domain human poem dataset for reproducibility.The implementation contribution complements POEMetric’s evaluation framework and curated dataset.

2 RELATED WORKS

Prior poetry-generation work has targeted formal constraints, language- and culture-specific styles, and evaluation through objective metrics and human or LLM judgments. However, existing metrics underrepresent advanced, poem-specific abilities such as creativity, emotion, authorial intention, and imagery.

  • Poetry generation with Language Models: Poetry-generation models have incorporated structural metrics such as rhyme and meter to satisfy formal constraints.ByGPT5, PoeLM, GPoet, and a GPT-2-based model integrated rhyme, meter, or related structural measures into generation.
  • Poetry generation with Language Models: Language-specific adaptations have produced high-quality poetry in low-resource languages and culturally nuanced forms.Examples include Pashto, Arabic, Vietnamese, Czech, and classical Chinese poetry.
  • Evaluation of poetic quality: Poetic quality evaluation has combined objective measures such as meter, rhyme, BLEU, and perplexity with human judgments, Turing-test-inspired frameworks, and LLM-as-a-judge.ProFTAP evaluates poetic indistinguishability from human works, while other work applies LLM-as-a-judge to LLM-generated poems.
  • Evaluation of poetic quality: Existing metrics focus mainly on meter, rhyme, formal diversity, fluency, and coherence, leaving advanced poem-specific abilities insufficiently evaluated.The paper identifies creativity, author intentions, emotions, and imagery as important aspects of poetry composition that require more nuanced metrics.

3 THE HUMAN POEM DATASET

The human poem dataset focuses on seven fixed poetry forms to create a rigorous, diagnostic benchmark with a quantifiable baseline. Poems from two online databases were filtered by detected meter and rhyme patterns, yielding 203 poems from an initial 1,309.

  • Dataset design: Seven fixed poetry forms establish a rigorous, diagnostic setting for creative evaluation and a quantifiable baseline.The paper uses constrained forms to support systematic development and validation of more subjective evaluations.
  • Data collection and filtering: 203 poems remained from 1,309 collected poems after algorithmic detection of meter and rhyme patterns.The poems were collected from the Poetry Foundation and Academy of American Poets, retaining only poems that followed a certain prosodic pattern.
  • Dataset annotations: Each retained poem is annotated with author, title, content, source, form, meter pattern, rhyme pattern, theme, and imagery.These annotations also support the generation prompts used for LLM poetry experiments.

4 POEMETRIC

POEMetric evaluates poetry across 10 dimensions spanning instruction following, advanced creative abilities, and general appraisal. It combines rule-based analysis, LLM-as-a-judge evaluation, and human expert validation to assess poems at scale and support robust judgments.

  • Framework dimensions: POEMetric covers 10 dimensions, ranging from form and theme instruction following to advanced creativity and general poem appraisal.Basic evaluation examines form features such as meter and rhyme where applicable, alongside thematic adherence.
  • Framework dimensions: The advanced metrics assess creativity, lexical diversity, idiosyncrasy, emotional resonance, literary devices, and imagery.These dimensions quantitatively operationalize elements traditionally emphasized in literary criticism.
  • Framework dimensions: General appraisal evaluates overall poem quality and whether evaluators can distinguish human-written from LLM-generated poems.The authorship task estimates whether each poem was written by a human or an LLM.
  • Evaluation methodology: The framework triangulates LLM-as-a-judge scoring with rule-based quantitative evaluation and human expert judgments.Rule-based algorithms detect meter and rhyme patterns, while lexical diversity uses MATTR averaged across poems for each author.
  • Evaluation methodology: Seven expert human judges validated evaluations of 58 anonymized poems written by humans and 7 representative LLMs.The validation was conducted with Institutional Review Board approval on a subset rather than the entire dataset because expert annotation is resource-intensive.

5 EXPERIMENTS

The experiments generated 6,090 poems from 30 LLMs using prompts based on 203 human poems and evaluated models under standardized prompting and sampling conditions. Gemini-2.5-Pro was selected as judge because it agreed more closely with human experts than DeepSeek-R1 and GPT-4o in a pilot study.

  • Experimental setup: All LLMs received the same text prompt, while open-source models were deployed locally with vLLM and default sampling parameters.System prompts also used each model’s default setting.
  • Judge selection: 0.662 vs. 0.548/0.438: Gemini-2.5-Pro showed higher agreement with human experts than DeepSeek-R1 and GPT-4o in the pilot study.The reported agreement measure was PAo.
  • Experimental setup: 30 models generated 6,090 poems from 203 prompts encoding each human poem’s form, rhyme, meter, and theme.The models represented 7 leading AI companies and used a simple prompting template.

6 RESULTS

Results show that LLMs followed poetic forms and themes reasonably well, but remained behind human poets in advanced creative abilities and overall quality. Gemini-2.5-Pro’s evaluations also showed strong agreement with expert human ratings.

  • Rule-based evaluation: 0.50 and 0.47 rule-based form accuracy were achieved by Gemini-2.5-Pro and Claude-3.7-Sonnet, while LLM poems had higher MATTR but more word-level repetition than human poems.The rule-based evaluation covered meter and rhyme patterns across seven representative LLMs.
  • Basic instruction-following abilities: 4.26 form accuracy and 4.99 theme alignment made Gemini-2.5-Pro the strongest LLM on basic instruction following.Llama-3.3-70B-Instruct ranked lowest, with scores of 2.29 and 4.91.
  • Advanced creative abilities: Human poems led LLM-generated poems in creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), imagery (4.49), and literary devices (4.67).Among representative LLMs, DeepSeek-R1 performed best and Llama-3.3-70B-Instruct worst.
  • General appraisal: 4.22 overall quality for human poems exceeded the LLM score, with DeepSeek-R1 ranking second among the evaluated authors.The evaluation attributed the human advantage to effective and idiosyncratic language use.
  • Evaluation validation: 0.662 Proportion Agreement, Observed indicated strong agreement between Gemini-2.5-Pro and expert human evaluators across 10 multiple-choice questions.Human evaluators and Gemini-2.5-Pro showed high similarity on form accuracy and theme alignment, although humans gave higher theme-alignment scores.

7 CONCLUSION AND LIMITATIONS

The paper introduces POEMetric as a comprehensive framework for evaluating poetry generation, supported by a curated human-poem dataset and experiments with 30 state-of-the-art LLMs. Although leading models can write poems in specified styles and themes, they remain behind humans in advanced creative abilities.

  • POEMetric is introduced as the most comprehensive evaluation framework for poetry generation so far.
  • The study curates 203 human poems spanning different poetic forms and themes.
  • The experiments evaluate 30 state-of-the-art LLMs for poetry generation.
  • Although top models write poems in certain styles and themes, they fall short in creativity, idiosyncrasy, emotional resonance, and skillful imagery use.

8 ETHICS STATEMENT

The research follows the ICLR Code of Ethics and emphasizes transparency and reproducibility through POEMetric, detailed methodological reporting, and released code and dataset. Human-participant work received IRB approval and informed consent.

  • Ethics and transparency: The study adheres to the ICLR Code of Ethics and promotes transparency by introducing POEMetric and documenting methods, model selection, and limitations.The paper presents POEMetric as a comprehensive framework and details its methodological choices throughout.
  • Reproducibility: The authors provide the code and dataset in supplementary materials to foster reproducible research.The released materials support replication of the reported research.
  • Human participants: All work involving human participants was conducted under Institutional Review Board approval with informed consent.The ethics statement explicitly identifies IRB oversight and participant consent.

9 REPRODUCIBILITY STATEMENT … C THE POEMETRIC-BASED LLM PROMPT AND HUMAN SURVEY

The paper supports reproducibility through publicly available evaluation materials and describes fixed poetic forms, rule-based form checking, and aligned human- and LLM-judge evaluation instruments. The supplements specify the forms, algorithmic tolerance, scoring criteria, and survey procedures used to assess poems.

  • 9 REPRODUCIBILITY STATEMENT: The evaluation framework, rule-based algorithm code, curated human poem dataset, 30-model list, and experimental details are publicly provided or documented.The framework and methodology appear in Section 4, while code and data are supplied anonymously in the supplementary materials.
  • A FIXED FORMS OF ENGLISH POETRY: The dataset covers ballads, ghazals, limericks, pantoums, sestinas, sonnets, and villanelles, each defined by characteristic stanza, rhyme, repetition, meter, or line structures.The described forms range from ballad quatrains and ghazal couplets to sonnets’ 14 lines and villanelles’ repeating lines.
  • B THE ALGORITHM OF RULE-BASED FORM ACCURACY: The rule-based form detector uses a 0.7 threshold to balance precision and recall, avoiding both rejection of stylistic variation and acceptance of poorly formed poems.The threshold is applied after testing on a human-poem subset to optimize the precision–recall trade-off.
  • B THE ALGORITHM OF RULE-BASED FORM ACCURACY: The POEMetric algorithm tokenizes poems, extracts stress and rhyme features, and validates meter and rhyme against target constraints with tolerance.Its inputs are poem text, target form, target meter, and target rhyme; it returns whether the poem satisfies the constraints.
  • C THE POEMETRIC-BASED LLM PROMPT AND HUMAN SURVEY: Human experts and LLM judges use the same POEMetric-based questions to evaluate poems without being told whether authorship is human or LLM.Both procedures ask judges to read the generation prompt and response before scoring the poem.

D LLM-AS-A-JUDGE JUSTIFICATION · E AN OVERVIEW OF THE 30 SELECTED LLMS

The paper selected Gemini-2.5-Pro as judge after pilot analyses found that averaging multiple LLM judges could reduce evaluation quality. It also summarizes the features of the 30 selected LLMs.

  • D LLM-AS-A-JUDGE JUSTIFICATION: Pilot analyses found that averaging scores from multiple LLMs would degrade evaluation quality.Other leading models showed low agreement with human experts and weak discrimination between poem qualities.
  • D LLM-AS-A-JUDGE JUSTIFICATION: Gemini-2.5-Pro achieved stronger agreement with human experts than GPT-4o and DeepSeek-R1.Observed Proportion Agreement was 0.662 for Gemini-2.5-Pro, compared with 0.548 for GPT-4o and 0.438 for DeepSeek-R1.
  • D LLM-AS-A-JUDGE JUSTIFICATION: 0.20 and 0.22 were the standard deviations for DeepSeek-R1 and GPT-4o, whose scores clustered at the high end.Their mean scores were 4.26 and 3.69, respectively, indicating limited discrimination between high- and low-quality poems.
  • D LLM-AS-A-JUDGE JUSTIFICATION: Table 2 reports human-versus-LLM-as-a-judge results for the overall poem quality dimension.The table is used in the judge-selection justification.
  • D LLM-AS-A-JUDGE JUSTIFICATION: The authors selected Gemini-2.5-Pro to ensure evaluation quality and validity.The selection followed comparisons of agreement with human experts and discrimination ability.
  • E AN OVERVIEW OF THE 30 SELECTED LLMS: Table 3 summarizes the features of the 30 selected LLMs.This overview constitutes the paper’s description of the selected model set.

F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET · G MORE SHOWCASES OF LLM AND HUMAN POEMS

The linguistic analysis shows that Claude 3.7 Sonnet most closely resembles human word usage, while LLMs and humans differ in opening-word preferences and imagery. Additional showcases compare Claude-3.7-Sonnet and Gemini-2.5-Pro poems with human poems using POEMetric scores judged by Gemini-2.5-Pro.

  • F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET: 0.602 cosine similarity makes Claude 3.7 Sonnet the LLM most similar to humans in the top-20 case-insensitive word distributions.The comparison removes stop words and covers seven state-of-the-art LLMs representing seven AI companies.
  • F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET: “Eyes,” “sun,” and “face” are common imagery for both LLMs and human poets, but author-specific preferences remain visible.Human poets frequently write about “water” and “god,” while DeepSeek R1 prefers “threads” and “bloom,” and QwQ-32B favors “thread.”
  • F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET: Figure 10 presents the top 20 words across the human and LLM poem datasets.The associated analysis compares case-insensitive words with stop words removed.
  • F LINGUISTIC FEATURES OF THE HUMAN-LLM POEM DATASET: Figure 11 presents the top opening words and top imagery across the human and LLM poem datasets.The analysis highlights both shared imagery and differences in opening-word and imagery preferences.
  • G MORE SHOWCASES OF LLM AND HUMAN POEMS: Figure 12 showcases Claude-3.7-Sonnet and human poems written in response to the same prompt.Bar charts report their POEMetric scores judged by Gemini-2.5-Pro.
  • G MORE SHOWCASES OF LLM AND HUMAN POEMS: Figure 13 showcases Gemini-2.5-Pro and human poems written in response to the same prompt.Bar charts report their POEMetric scores judged by Gemini-2.5-Pro.

H POEMETRIC SCORES OF HUMAN POETS AND ALL 30 LLMS

The section reports average POEMetric scores for basic instruction-following, advanced creative abilities, overall poem quality, and human-authorship estimation across human poets and 30 LLMs. Within model families, larger models generally performed better in poem generation.

  • Model scaling: Within the same model family series, models with more parameters generally performed better in poem generation.The passage describes this as an overall trend across model family series.
  • Basic instruction-following: Average basic instruction-following scores for all 30 LLMs are reported in Figure 14.The figure covers basic instruction-following abilities.
  • Advanced creative abilities: Average advanced creative-ability scores for human poets and LLMs are reported in Figure 15.The figure covers advanced creative abilities.
  • Overall poem quality: Average overall poem-quality scores are reported in Figure 16.The figure summarizes overall poem quality.
  • Human authorship estimation: Average human-authorship-estimation scores are reported in Figure 17.The figure covers human authorship estimation.
Loading 2604.03695v1…