Source-linked AI summary
Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?
Tian Lan, Shanshan Wang, Zehua Duo, Jiang Li, Guanglai Gao, Derek F. Wong, Xiangdong Su
TL;DR
Modern Chinese poetry’s implicit imagery and unconventional structure require holistic poetic-logic reasoning, a capability not systematically evaluated by existing paradigms. The paper introduces Peony, a four-task benchmark across stanza, line, and imagery levels, and evaluates mainstream LLMs under thinking and non-thinking configurations. The results show substantial limitations in recovering poetic logic, while thinking models generally outperform non-thinking counterparts but gain little advantage on SNI.
Problem
Existing evaluation paradigms lack a specialized framework for determining whether LLMs comprehend the underlying poetic logic of modern Chinese poetry.
Method
Peony evaluates modern Chinese poetry comprehension through four tasks across stanza, line, and imagery levels.
Results
All evaluated models show substantial limitations in recovering poetic logic across Peony’s four tasks, while thinking models generally outperform their counterparts.
Takeaways & Limitations
Thinking strategies are not uniformly applicable to poetic-logic comprehension, as their advantage is not pronounced on SNI despite stronger average performance.
Takeaways & Limitations
Peony covers 800 works by six contemporary poets and does not directly evaluate the full range of literary understanding or support generalization to other poetic traditions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks, yet their ability to understand literary texts, particularly modern Chinese poetry, remains largely unexplored. The unique literary characteristics of modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension. Unlike conventional texts that convey clear information, the unique "poetic logic" of modern Chinese poetry requires a holistic reasoning approach that goes beyond superficial semantic analysis to be understood. However, current evaluation paradigms largely ignore this critical dimension. To address this gap, we propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry. We define poetic logic as four tasks across three levels, namely stanza, line, and imagery, and systematically evaluate and analyze six mainstream LLMs based on Peony. We evaluate these models under both non-thinking and thinking configurations. The experimental results reveal the limitations of current LLMs in understanding the poetic logic of modern Chinese poetry and validate the effectiveness and necessity of Peony. Our data and code will be available.
1 Introduction
Modern Chinese poetry requires holistic reasoning over implicit imagery, emotional transitions, and unconventional expressions, yet existing evaluations largely overlook this poetic logic. Peony addresses this gap with a benchmark spanning four tasks and three structural levels, followed by systematic LLM evaluation.
- Motivation: Modern Chinese poetry has received relatively little attention despite requiring reasoning beyond superficial semantic analysis.Its freer forms, fragmented imagery, semantic ambiguity, emotional transitions, and unconventional syntax create distinct comprehension challenges.
- Poetic logic: Poetic meaning often emerges from implicit associations between images, requiring readers to reconstruct emotional and semantic connections across lines.Examples include linking seasons with a lotus’s blooming and falling to convey waiting, time, and melancholy.
- Evaluation gap: Current NLP evaluation paradigms lack a specialized framework for testing whether LLMs comprehend modern Chinese poetry’s underlying logical structures.This leaves unanswered whether current LLMs can genuinely comprehend poetic logic.
- Proposed benchmark: Peony is introduced as the first benchmark specifically designed to evaluate poetic logic in modern Chinese poetry.The benchmark contains 800 high-quality works by six contemporary poets and spans varied themes and styles.
- Benchmark design: Peony formalizes poetic logic through four tasks across line, stanza, and imagery levels for systematic LLM evaluation.The authors evaluate six mainstream models and provide detailed experimental analysis.
2 Related Works
Prior poetry research has emphasized generation, translation, and detection, while comprehension has received less systematic attention. Existing comprehension studies cover several languages and poetry traditions, but poetic logic in modern Chinese poetry remains underexplored.
- Existing research: Poetry research has primarily focused on generation, translation, and detection rather than comprehension.These directions address production, cross-language transfer, and identification of generated poetry.
- Poetic comprehension: Poetic comprehension is a prerequisite for downstream tasks such as poetry generation, translation, and detection.This motivates dedicated evaluation of how models interpret poetic meaning.
- Prior comprehension studies: Existing benchmarks and studies examine ancient Chinese, German, Arabic, and modern Chinese poetry across diverse linguistic and literary dimensions.These include literary nuances, meter, rhythm, figurative language, metaphor, cultural context, and interpretation.
- Research gap: The poetic logic underlying modern Chinese poetry remains underexplored despite these prior efforts.The paper identifies a need for more rigorous evaluation of modern Chinese poetry comprehension.
3 The Dataset
Peony is constructed from 800 human-written modern Chinese poems by six contemporary poets and targets poetic logic through four task types. Its curation emphasizes diversity, expert validation, and reduced data-leakage risk.
- Task construction: The benchmark constructs tasks, questions, and ground-truth answers focused on poetic logic from high-quality human-created poetry.Its four tasks address structural organization, imagery association, emotional progression, and contextual coherence.
- Corpus: Peony uses 800 modern Chinese poems by six contemporary poets curated from the AIGenPoetry dataset.The corpus includes varied poetic imagery, styles, themes, forms, and lengths.
- Data selection: Contemporary works were selected to mitigate data leakage because canonical poems may appear in LLM pretraining corpora.AIGenPoetry was released in November 2025, after the reported knowledge cutoffs of all evaluated models.
- Annotation pipeline: The construction pipeline selects compatible poems, applies task-specific shuffling or masking, and uses model-generated candidates followed by substantial professional-poet revision.Two professional poets and poetry-theory researchers further redesign options to remove contextual breaks, ambiguity, or misleading imagery overlap.
- Dataset diversity: The dataset covers diverse forms, including single- and multi-stanza poems, variable lengths, and poems spanning emotions, daily life, social issues, travel, philosophy, art, and artificial intelligence.It also includes works that retain rhythmic or structural echoes of classical Chinese poetry.
4 Evaluation Tasks
Peony evaluates poetic logic through four multiple-choice tasks spanning stanza, line, and imagery levels. The tasks test sequence reconstruction, contextual imagery selection, and inter-line inference while using accuracy and Perfect Rate where appropriate.
- Task Overview: Peony measures poetic logic with Stanza Ordering, Line Ordering, Imagery Filling, and Sentence Inference across line, stanza, and imagery levels.Each task uses accuracy scores ranging from 0 to 100, with higher scores indicating stronger comprehension.
- Stanza Ordering: Stanza Ordering tests whether models can reconstruct a poem’s original logical sequence from shuffled stanzas.The task treats each stanza as a relatively coherent semantic unit and scores performance with accuracy.
- Line Ordering: Line Ordering requires models to reorder shuffled sentences using contextual poetic text rather than relying on explicit logical markers.The context supplies imagery, mood, and intentional ambiguity, while modern Chinese poetry’s imagery associations and spatiotemporal discontinuity support the evaluation.
- Imagery Filling: Imagery Filling evaluates whether models select contextually appropriate imagery for masked slots in complete modern Chinese poems.The task uses multiple-choice questions focused on imagery within a given poetic framework.
- Metrics: Perfect Rate measures the proportion of poems for which all blanks are correctly filled, complementing accuracy with a poem-level consistency measure.It evaluates whether logical coherence is maintained across an entire poetic work.
- Sentence Inference: Sentence Inference tests inter-line coherence by selecting the original masked sentence from candidates inconsistent with the poem’s underlying poetic logic.The task covers relationships including causality, contrast, progression, resonance, and imagery leaps, and uses the same scoring criteria as Imagery Filling.
5 Experimental Setup
The experiments evaluate LLMs in non-thinking and thinking configurations using developer-recommended default parameters. Accuracy is reported for ordering tasks, while Imagery Filling and Sentence Inference additionally use Perfect Rate, with results averaged across three runs.
- Models and Settings: Models are evaluated under non-thinking and thinking settings, including DeepSeek-V4-Flash and Pro, GPT-5.1, Kimi-K2.5, GLM-5.1, and Gemini 3 Pro.Gemini 3 Pro is included in the thinking-mode group, while the other listed models are evaluated in both configurations where specified.
- Metrics: Accuracy is reported for Line Ordering and Stanza Ordering, while Imagery Filling and Sentence Inference use both Accuracy and Perfect Rate.All experiments are conducted three times, and average values are reported as final results.
6 Results and Analysis
Peony reveals substantial differences among LLMs in poetic-logic comprehension, with thinking configurations generally stronger but persistent difficulty coordinating poem-level structure and meaning.
- Imagery Filling: IMF accuracy exceeds poem-level PR: DeepSeek-V4-Pro records 72.93 accuracy but only 59.95 PR, showing difficulty coordinating multiple imagery units.
- Sentence Inference: 30.77 is the highest SNI PR, achieved by Gemini 3 Pro, indicating persistent difficulty maintaining cross-sentence poetic logic.
- Human Comparison: Experts achieve higher average performance than non-expert readers, although performance varies substantially across individual readers.
- Overall Performance: Gemini 3 Pro achieves the strongest overall performance at 74.01, exceeding Kimi-K2.5’s 64.27 among thinking models and 59.89 among non-thinking models.
- Overall Performance: Thinking configurations generally outperform their non-thinking counterparts, with DeepSeek-V4-Flash and DeepSeek-V4-Pro gaining 18.46 and 11.24 points, respectively.
- Task Difficulty: The SNI task is most challenging, followed by STO, while IMF is relatively easiest across the evaluated models.
- Line Ordering: Gemini 3 Pro leads LIO with 85.29 accuracy, while Kimi-K2.5 is the strongest non-thinking model at 63.00.
- Sentence Inference: Gemini 3 Pro leads SNI with 63.53 accuracy and 30.77 PR, surpassing other models in selecting lines consistent with poem-level themes and emotions.
7 Conclusion
The paper introduces Peony to evaluate poetic logic in modern Chinese poetry and finds substantial limitations across current LLMs, despite advantages for thinking configurations.
- Peony contains 800 modern Chinese poems and evaluates poetic logic through four tasks spanning stanza, line, and imagery levels.
- All evaluated models exhibit substantial limitations in recovering poetic logic across the four Peony tasks.
- Thinking models outperform their counterparts on average, but their SNI advantage is not pronounced, suggesting explicit reasoning is not always applicable.
- Models achieve relatively high IMF accuracy but significantly lower PR, indicating difficulty coordinating logical relationships among multiple imagery units.
Limitations
Peony’s coverage is constrained by its poet selection, task operationalization, and focus on modern Chinese poetry rather than other poetic traditions.
- The 800 works by six contemporary poets cannot represent the full historical and stylistic diversity of modern Chinese poetry.
- Peony’s four tasks do not directly evaluate cultural interpretation, open-ended metaphor analysis, or aesthetic judgment.
- Findings should not be generalized to classical Chinese poetry, haiku, or Arabic poetry without further study.
Ethics Statement
The study uses contemporary poetry for non-commercial scientific research and combines model-assisted curation with human review and task-specific prompts.
- The poetic samples come exclusively from contemporary poets, and all data are used strictly for non-commercial scientific research.
- Experts were fairly compensated at rates above the minimum wage, and the research was conducted in compliance with legal standards.
- DeepSeek-V4-Pro assisted imagery selection and distractor generation, while professional poets reviewed and substantially revised or replaced generated candidates.
- Task-specific prompts specify descriptions, inputs, candidate options, and output formats to support consistent and reproducible assessment.
C Supplementary Analyses and Benchmark Comparison
Supplementary analyses find no meaningful option-position bias and stable STO performance under random option orders. Peony is characterized by diverse imagery and uses four tasks to evaluate poetic logic, with human references indicating higher performance from professional poets than non-experts.
- Positional-bias analysis: Correct answers are approximately balanced across positions A–D in every task.
- Positional-bias analysis: STO performance remains stable across two additional random option orders for two representative non-thinking models.
- Benchmark comparison: Peony targets modern-poetry poetic-logic comprehension rather than generation detection or a single semantic classification task.
- Imagery analysis: The benchmark analyzes imagery frequencies across 800 poems, separating single-character and two-character imagery words.
- Imagery analysis: The most frequent short imagery word is “人(Human)” with 245 occurrences, while “自己(Self)” ranks highest among long imagery words with 145.
- Imagery analysis: The imagery vocabulary spans concrete sensory terms and abstract concepts, supporting Peony’s challenge for evaluating poetic-logic comprehension.
- Human reference: Professional poets achieve a mean score of 51.44 versus 41.28 for non-expert readers on a random sample of 40 poems.
G SNI Case Study
The SNI case study shows that a thinking-mode model can fail when poetic comprehension requires cross-line continuity, imagery interpretation, and correctly applied cultural knowledge. In “Lintie Ji” (Copying Calligraphy), DeepSeek-V4-Flash answered both blanks incorrectly through locally compatible imagery matching and incomplete reasoning.
- Case result: DeepSeek-V4-Flash in thinking mode answered both “Lintie Ji” SNI blanks incorrectly, selecting AA instead of the correct DC.The case identifies SNI as the most challenging task.
- Cross-line logic: The first error favored locally compatible ancient imagery but missed the action continuity from leaping onto horseback to passing through a rain of arrows.The correct option links the action in one line with the following line rather than merely matching imagery.
- Imagery interpretation: The second error failed to connect “one dot, one stroke, one hook” with calligraphic strokes, overlooking the option containing “xuan paper.”The error reflects a missed relation between the poem’s wording and the material context of calligraphy.
- Cultural knowledge: The reasoning recognized Wang Xizhi’s Preface to the Orchid Pavilion Collection but misused that knowledge by choosing Orchid Pavilion rather than Kuaiji Shanyin.Recognizing an allusion was insufficient because the relevant background had to be applied correctly.
- Error analysis: The case attributes the SNI errors jointly to surface imagery matching, failed cross-line logic, and incomplete use of cultural knowledge.These error sources operate together in the illustrated SNI example.