PaperlayerView exact revision on arXiv

Source-linked AI summary

Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku

Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese

arXiv:2609.15511v1cs.CLcs.AI

TL;DR

The paper asks whether readers can distinguish AI-generated from human-written Japanese haiku, and whether aesthetic impressions track true authorship. It evaluates mixed haiku with human readers and finds that aesthetic judgments align more with perceived authorship than correct recognition, which varies by model and poem.

  • Problem

    Evidence is limited on whether aesthetic cues associated with perceived humanness also support correct authorship attribution across contemporary models in constrained haiku.

    Source: §1 · p. 2

  • Method

    The study generated Japanese haiku with contemporary LLMs, mixed them with human-written poems, and collected human authorship judgments and aesthetic ratings.

    Source: §1 · p. 2

  • Results

    Recognition varied across models and individual poems, while impression ratings predicted perceived humanness more than correct authorship recognition.

    Source: §6 · p. 12, §7 · p. 13

  • Takeaways & Limitations

    Aesthetic judgments remain useful for understanding perceived quality but may be less reliable for identifying creative-text authorship as models produce fluent, plausible writing.

    Source: §7 · p. 13

    The findings concern few-shot generation of Japanese seasonal haiku and 144 university participants from a limited set of Japanese institutions.

    Source: §8 · p. 13

Abstract

from arXiv · show

This paper investigates the generation and human evaluation of Japanese haiku by contemporary Large Language Models (LLMs), focusing on authorship perception and aesthetic judgment within a constrained poetic form. Using a few-shot prompting strategy, Japanese haiku were generated across a heterogeneous set of large language models, including open- and closed-source systems, medium-scale and large-scale architectures, models with native or adapted Japanese support, and multilingual proprietary models. These AI-generated haiku were combined with human-written ones and presented in a questionnaire distributed to students at Japanese universities in Tokyo. The survey assessed whether respondents could distinguish between AI-generated and human-written haiku and which cues informed their judgments. Recognition accuracy varied across models. GPT-5, Gemini 2.5, and StableLM-7B performed at approximately chance level (approx 0.50), whereas LLM-JP, Gemma-2B, and LLaMA-2 showed moderate detectability (approx 0.59-0.67). However, recognition was strongly item-dependent. Ratings of fluency, coherence, poeticness, and related aesthetic dimensions predicted perceived humanness but not correct classification, indicating an attribution bias linked to aesthetic evaluation and revealing a dissociation between aesthetic evaluation and true authorship detection. The extended analysis additionally examines generation-constraint adherence, participant-level characteristics, and exploratory LLM-based evaluations of haiku authorship. Overall, the findings suggest that as LLMs improve, surface-level creative plausibility may reduce reliable human discrimination within constrained poetic settings.

1 Introduction

The paper studies how people judge creativity and authorship in AI poetry, using regular Japanese haiku as a constrained but creatively open testbed. It compares contemporary models and asks whether aesthetic impressions help readers identify true authorship.

Why haiku

  • Regular seasonal haiku offer a controlled 5-7-5-mora form that retains room for creativity, making them a testbed for authorship perception and aesthetic judgment.Haiku also combine seasonal reference, brevity, and a cutting moment.Source: §1 · p. 2

Research gap

  • Prior work has rarely compared linguistic quality across contemporary models while testing whether perceived creativity and aesthetic cues align with correct authorship attribution in constrained haiku.The study addresses whether cues associated with perceived humanness support accurate source recognition across generative systems.Source: §1 · p. 2

Study overview

  • The study generated Japanese haiku with six diverse models and asked 144 Japanese university students to identify authorship and rate each poem’s impressions.The questionnaire mixed AI and human poems; its authorship task compared perceived source with correct recognition rather than treating indistinguishability as proof of creativity.Source: §1 · p. 2
  • Recognition was near chance for some strong models and varied by item, while impression ratings tracked perceived source more closely than true authorship.The findings distinguish aesthetic impression from correct authorship identification.Source: §1 · p. 2

Scope

  • The conclusions concern constrained Japanese haiku generated with few-shot prompting, not unconstrained creative writing or creativity generally.The study’s scope is a specific poetic and prompting setting.Source: §1 · p. 2

2 State of the Art

Computational poetry research has moved from rule-based and neural generation toward LLMs, while evaluation has used automatic proxies, human ratings, and LLM judges. The paper identifies limited cross-model evidence connecting aesthetic impressions with correct authorship attribution in constrained haiku.

2.1 Computational Poetry and Haiku Generation

  • Early computational poetry systems emphasized handcrafted structural control, followed by neural approaches using iterative refinement, adversarial objectives, and autoregressive generation.Examples include Gabriel’s Inkwell, Yan’s iPoet, and SeqGAN-style haiku generation.Source: §2.1 · p. 3

2.2 Evaluating Haiku: Automatic Scoring, Human Ratings, and LLM-as-Judge

  • Haiku evaluation has used popularity proxies, human aesthetic ratings, and LLM evaluators, but these approaches often prioritize quality or preference over authorship attribution.Hitsuwari et al. (2023) reported that collaborative haiku received the highest beauty scores and participants could not accurately discriminate human from AI authorship.Source: §2.2 · p. 3
  • Few studies jointly compare contemporary generative models in seasonal fixed-form Japanese haiku and distinguish perceived humanness from correct authorship recognition.The related work motivates controlled designs mixing AI- and human-written haiku and collecting both aesthetic ratings and authorship judgments.Source: §2.2 · p. 3

3 Models and Generation Framework

The generation framework uses a curated Japanese haiku corpus and few-shot prompts to compare contemporary models under formal constraints. Its design prioritizes comparable, seasonally controlled outputs while balancing structural adherence with lexical variation.

3 Models and Generation Framework

  • The framework uses regular haiku as a constrained testbed and combines a curated corpus with few-shot prompting across contemporary language models.The design targets comparable outputs while preserving 5-7-5 mora structure and seasonal references.Source: §3 · p. 3

3.1 Dataset and Preprocessing

  • Preprocessing retained all 37,158 corpus poems, removed sparse auxiliary fields, labeled regularity by mora count, and separated seasonal metadata for prompting.The dataset’s native kigo field supplied seasonal terms; archaic kana were retained.Source: §3.1 · p. 4

3.2 Models

  • The six Japanese-capable models span open- and closed-source systems, different scales, and native, adapted, or multilingual Japanese support.Table 1 summarizes their technical and institutional profiles.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Prompt Design

  • Few-shot prompting was chosen over fine-tuning after fine-tuning attempts overfit, and prompts specified Japanese three-line 5-7-5 output with a seasonal word.Language-framed identity wording improved quality under identical decoding settings: StableLM-7B perplexity fell from approximately 52 to 34, and Gemma-2B from roughly 6975 to 530.Source: §3.3 · pp. 4–5

3.4 Decoding Strategy

  • Line-by-line stochastic sampling was preferred to beam search to reduce prompt copying and increase lexical diversity, prioritizing structural control despite occasional losses in inter-line coherence.Only outputs satisfying canonical 5-7-5 mora structure and seasonal-word inclusion entered the evaluation pool; adherence remained model-dependent.Source: §3.4 · pp. 5–6

4 Evaluation Methodology

The evaluation centers on human judgments of mixed AI- and human-written haiku, complemented by exploratory LLM assessments. Statistical models account for both participant and poem-level repeated measures.

4.1 Human Evaluation Design

  • The human evaluation used a pilot-refined questionnaire distributed to 144 participants at multiple Japanese universities, with background variables collected for participant-level analyses.Most participants were native Japanese speakers, with a small number of exchange students with varying Japanese proficiency.Source: §4.1 · p. 6
  • Participants rated seven impression dimensions and perceived authorship for each poem; AI and prize-winning human haiku were randomly mixed in short questionnaires.Human poems came from curated prize-winning contests, a choice intended to avoid low-quality comparisons but not to match all stylistic or aesthetic dimensions.Source: §4.1 · pp. 6–7

4.2 AI-based Evaluation

  • Exploratory AI-based assessments used GPT-5 for impression ratings and Grok for authorship classification, with justifications reviewed for recurring attribution cues.The analyses complemented the human questionnaire rather than replacing it.Source: §4 · p. 6, §4.2 · p. 7

4.3 Statistical Analysis

  • Regularized logistic regression and Bayesian binomial mixed-effects models analyzed recognition and impression patterns, accounting for respondent and haiku-item variation.The analysis used accuracy and cross-validated AUC; the authors caution that exploratory results from the limited participant sample warrant care.Source: §4.3 · pp. 7–8

5 Results

Recognition varied by model and by individual haiku, while impression ratings were generally positive and did not reliably identify true authorship. Aesthetic ratings tracked perceived authorship more strongly than correct recognition, and participant background alone had limited predictive power.

5.1 Recognition Accuracy

  • 0.493 accuracy for GPT-5 contrasted with 0.670 for LLaMA-2; GPT-5, Gemini 2.5, and StableLM-7B were near chance, while the other models showed moderate detectability.Gemini 2.5 reached 0.503, StableLM-7B 0.524, Gemma-2B 0.611, and LLM-JP 0.590.Source: Table 4 · p. 8, §5.1 · p. 8
  • Recognition varied widely across 84 unique haiku: many items were near chance, while some were recognized above or below chance, suggesting item-specific rather than stable model-level cues.Figure 1 shows the distribution; accuracy is the proportion of correct authorship judgments aggregated across participants, with 50% marking chance.Source: §5.1 · pp. 8–9, Figure 1 · p. 9

5.2 Impression Ratings

  • Ratings clustered around 3–4, with fluency, haiku-like wording, and coherence rated highest; AI- and human-written haiku received similar impressions, while confidence was often lower.Unexpectedness received the lowest average rating, nearer the scale midpoint.Source: §5.2 · pp. 9–10

5.3 Participant Background and Recognition Ability

  • Japanese proficiency showed the clearest participant-level association with recognition, but background models had near-chance AUCs; the STEM association was exploratory and potentially confounded.Cross-validated AUCs were 0.416 for the full model, 0.456 for the language-only model, and 0.432 for the no-language model.Source: §5.3 · p. 10

5.4 Perceived Authorship vs. Correct Recognition

  • 0.505 cross-validated AUC for correct recognition contrasted with 0.793 for perceived-human attribution, showing impression ratings were more predictive of perceived authorship than true source.Favourability, haiku-like wording, and theme relevance were among the strongest predictors of perceived authorship; impression cues had only weak associations with correct recognition.Source: §5.4 · p. 12

6 Discussion

The discussion distinguishes aesthetic evaluation from authorship attribution: participants used aesthetic expectations to judge whether haiku felt human- or AI-written, but those cues did not reliably reveal true source. It also considers stereotype-based expectations about AI text as one possible explanation for misclassification.

6 Discussion

  • Favourability, natural wording, and thematic appropriateness shaped whether poems felt human- or AI-generated, but did not reliably distinguish their true source.The discussion frames this as a distinction between creative evaluation and creative attribution, citing Colton and Wiggins.Source: §6 · p. 12
  • Near-chance results for some models and moderate detectability for others do not support a simple account in which participants systematically attributed lower-quality poems to AI.The discussion notes that attribution may also reflect expectations about AI and model-specific textual cues.Source: §6 · p. 12
  • One proposed explanation is that participants relied on stereotypical expectations of machine-generated language, despite contemporary LLMs producing fluent, stylistically appropriate haiku.Such expectations may guide authorship judgments toward perceived stylistic deviations rather than true source.Source: §6 · p. 13

7 Conclusions

The study finds a dissociation between aesthetic evaluation and correct authorship recognition: aesthetic impressions predict perceived humanness more than accurate identification. Recognition also varies by model and individual haiku, while the conclusion cautions that perceived quality may become less aligned with true authorship.

7 Conclusions

  • Recognition remained close to chance for several models, while other systems showed moderate detectability and results varied substantially across individual haiku.The conclusion characterizes detectability as strongly item-dependent.Source: §7 · p. 13
  • Aesthetic impressions were substantially more predictive of perceived human authorship than of correct authorship recognition.Favourability, haiku-like wording, and thematic relevance appear to function primarily as attribution heuristics rather than reliable signals of true authorship.Source: §7 · p. 13
  • As language models produce more fluent and stylistically plausible creative text, perceived quality and true authorship may become less aligned.Aesthetic judgments remain useful for evaluating perceived quality but may be less reliable for identifying creative artifacts’ sources.Source: §7 · p. 13
  • Future work should examine more diverse human-written corpora, additional models and decoding strategies, contextual framing, and collaborative human–AI creativity.Source: §7 · p. 13

8 Limitations

The study’s findings are bounded by its specific generation setting, participant and comparison samples, and six-model design. Its analyses are exploratory, and its appendix-based LLM-judge results are not confirmatory.

8 Limitations

  • The findings concern Japanese seasonal haiku generated with few-shot prompting and should not be generalized directly to unconstrained writing or other artistic domains.Source: §8 · p. 13
  • The human evaluation included 144 university participants from a limited set of Japanese institutions, so the sample does not represent the broader Japanese population.Source: §8 · p. 13
  • The human-written comparison set contained only prize-winning contemporary competition haiku, potentially selecting an especially successful subset of poems.Source: §8 · p. 13
  • The study compared six models under one prompting and decoding framework, and recognition patterns may change with other models, versions, prompts, or decoding strategies.Source: §8 · p. 13
  • The statistical analyses are exploratory, and appendix LLM-judge explanations are model-generated rationales rather than transparent evidence of internal decision processes.The analyses use a relatively limited participant sample; Grok’s explanations are not treated as transparent access to its internal decisions.Source: §8 · p. 13

A Constraint Adherence and Loss Rates

Table 5 reports constraint adherence, acceptance, and loss rates across models. Adherence requires both canonical 5-7-5 mora structure and an appropriate seasonal word.

A Constraint Adherence and Loss Rates

B Grok Qualitative Coding Protocol

The appendix describes an exploratory qualitative analysis of Grok’s haiku authorship justifications. It inductively identified attribution cues, grouped them into themes, and did not establish a formal coding taxonomy.

B Grok Qualitative Coding Protocol

  • Grok classified each haiku as human-written or AI-generated and supplied a short justification, which the authors reviewed through exploratory qualitative content analysis.Source: B Grok Qualitative Coding Protocol · p. 14
  • The analysis identified recurrent cues inductively in a first pass, then grouped similar cues into broader thematic categories in a second pass.Source: B Grok Qualitative Coding Protocol · p. 14
  • Six categories covered seasonality and kigo, formulaic phrasing, coherence, poetic depth, imagery and emotional resonance, and wording naturalness.The analysis was descriptive, not confirmatory, and was not intended to establish a formal coding taxonomy; no formal intercoder reliability was computed.Source: B Grok Qualitative Coding Protocol · p. 14

C Exploratory AI-based Evaluation

The exploratory AI evaluation used ChatGPT-5 for direct scoring and Grok for authorship classification. ChatGPT-5’s five-haiku pilot documented structured ratings but was not compared with human ratings.

C.1 Overview

  • The exploratory overview considered direct scoring with ChatGPT-5 and authorship classification with Grok as probes of automated haiku evaluation.Source: §C.1 · p. 14

C.2 ChatGPT-5 as a Judge

  • ChatGPT-5 rated five haiku across seven dimensions on a 1–5 Likert scale, with scores ranging from 2 to 5.Examples included scores of 4–5 across most dimensions for some haiku and around 2–3 in coherence and understandability for others.Source: §C.2 · pp. 14–15
  • Because the pilot covered only five haiku and was exploratory, its scores were not compared with human ratings.Source: §C.2 · p. 15

C.3 Grok as a Judge

Grok’s exploratory classifications varied across models, from chance-level performance on three models to higher accuracy on the others. Its justifications cited surface cues, while Figure 6 presents impression ratings pooled across models and authorship conditions.

C.3 Grok as a Judge

  • Grok achieved 50% accuracy on GPT-5, Gemini 2.5, and StableLM-7B, versus 83% for Gemma-2B, 71% for LLM-JP, and 79% for LLaMA-2.Source: Figure 4 · p. 15, Figure 5 · p. 15, §C.3 · pp. 15–16
  • Grok’s explanations associated AI classifications with unclear seasonal references, formulaic phrasing, weak imagery, and forced metaphors, while human classifications aligned with more coherent imagery and natural phrasing.Source: §C.3 · p. 16
  • The authors treat Grok’s model-level patterns as indicative rather than generalizable because the analysis was exploratory and had a limited sample size.Source: §C.3 · p. 16

D Overall Impression Ratings

Related papers

This paper cites

Loading 2609.15511v1…