PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Keunhyeung Park, Seunguk Yu, YoungBin Kim

arXiv:2608.28986v1cs.CL

TL;DR

LLMs often produce modern Korean poetry resembling line-broken prose, motivating a need to detect authorship and guide generation toward human-like form. The paper decomposes this gap into four dimensions and five interpretable features, then applies them to both tasks. Detection reaches 83.60 average AUC-ROC across seven unseen LLMs, while feature-guided generation improves expert judgments and shifts targeted statistics toward human poetry.

  • Problem

    LLM-generated Korean poetry often resembles “line-broken prose,” with safe, plain word choices replacing poetic tension or rhythm.

  • Method

    The paper decomposes human–LLM differences into four form-level dimensions and operationalizes them as five features for detection and prompt-level generation guidance.

  • Results

    83.60 average AUC-ROC across seven unseen LLMs is achieved by logistic regression using the five features, while guided generation improves expert judgments and shifts targeted properties toward human distributions.

  • Takeaways & Limitations

    Interpretable linguistic features can function as both diagnostic signals and intervention targets for Korean poetry generation.

  • Takeaways & Limitations

    The features are tailored to modern Korean, transfer to other languages is untested, and form-level analysis omits imagery, metaphor, and discourse coherence.

Abstract

from arXiv · show

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

1 Introduction

The paper frames modern Korean poetry generation as a form-level problem: LLM outputs often resemble line-broken prose, reflecting documented biases in length, diversity, rhythm, and orthography. It introduces five interpretable features across four dimensions for both zero-shot detection and generation guidance.

  • LLM-generated Korean poetry often resembles “line-broken prose,” favoring safe, plain word choices over poetic tension or rhythm.
  • The diagnostic stance decomposes the human–LLM gap instead of imposing a single rigid form constraint.This approach responds to modern Korean poetry’s reliance on lineation and meaning-driven cadence rather than fixed meter.
  • The framework characterizes human–LLM differences through Volume, Structure Variation, Rhythmic Irregularity, and Normative Adherence.These dimensions are motivated by documented biases involving prose-dominant training, reduced diversity, and regression toward the mean.
  • Five interpretable linguistic features operationalize the four dimensions as signals for zero-shot OOD detection and targets for generation guidance.
  • The features support detection across seven unseen LLMs and generation guidance that improves expert judgments while shifting form toward human poetry.The detection comparison and generation result are presented as coupled uses of the same feature set.

2 Related Work

Related work provides evidence that LLM text is statistically and stylistically uniform, while Korean detection has used punctuation and spacing cues. Poetry-generation research has mainly enforced explicit formal constraints, which fit templates better than modern Korean free verse.

  • LLM outputs show reduced variance, tight stylistic clustering, and limited syntactic construction compared with heterogeneous human writing.
  • Computational stylometry supplies broad feature inventories for authorship, genre, readability, and classification, including multilingual stylometric vectors.
  • KatFishNet detects Korean LLM text using punctuation- and spacing-oriented cues, whereas this paper develops a poetry-specific taxonomy for free-verse form.
  • Prior computational poetry systems primarily impose hard constraints such as meter, rhyme, acrostics, or exact character counts.
  • These constraint-based methods are less applicable to modern Korean poetry, whose rhythm typically emerges through lineation and meaning-driven cadence rather than fixed meters.

3 Linguistic Dimensions: Four Key Differences

The paper decomposes human–LLM differences in Korean poetic form into four diagnostic dimensions, operationalized as five quantitative features for analyzing length, endings, rhythm, and orthography.

  • Four Key Differences: Four dimensions—Volume, Structure Variation, Rhythmic Irregularity, and Normative Adherence—diagnostically differentiate human-authored from LLM-generated Korean poetry.The dimensions are motivated by documented regularities, not established as causal mechanisms.
  • Volume: Raw Volume measures poem length as the number of whitespace-tokenized elements.The feature is defined as the cardinality of τ(x), where τ denotes whitespace tokenization.
  • Structure Variation: Ending Type Diversity measures distinct normalized line-final forms, while Syntactic Connection measures the proportion of connective endings.Both features use line-level ending sequences; diversity captures repeated versus varied endings, and connection captures connective-ending use.
  • Empirical Use: The five features support zero-shot OOD detection, averaging 83.60 AUC-ROC versus 75.84 for the strongest KatFishNet configuration.The comparison is reported across target LLMs, with the method’s gain significant on four of seven targets after Holm correction.
  • Rhythmic Irregularity: Burstiness uses the coefficient of variation of content-character counts across lines to quantify rhythmic irregularity.Whitespace and punctuation are excluded from each line’s content-character count; low CV indicates more uniform line lengths.
  • Normative Adherence: Spacing Ratio measures normative rigidity as the ratio of space characters to total characters.The feature targets differences in adherence to standard spacing conventions between LLM outputs and human poetic practice.

4 Detection via Linguistic Features

The paper evaluates a five-feature logistic regression detector under zero-shot OOD conditions and finds stable, interpretable separation between human and LLM poetry, with generator-specific exceptions.

  • Zero-Shot OOD Detection: 83.60 average AUC-ROC across seven unseen LLMs exceeds KatFishNet’s 75.84 under the same zero-shot OOD protocol.This is a 10.23% relative improvement; gains are significant for four targets but not for EXAONE-3.5-7.8B or EEVE-10.8B.
  • Boundary Case: KatFishNet performs better on Qwen2-72B by capturing a generator-specific comma pattern outside the proposed feature taxonomy.This identifies a boundary case for the five-feature approach.
  • Feature Validity: Every linguistic-dimension ablation lowers mean AUC-ROC, with Structure Variation removal producing the largest drop.No single dimension group matches the full system in isolation.
  • Interpretability: Consistent coefficient signs across all seven target models indicate that the five features provide stable signals across generator changes.The coefficients come from separate per-target logistic regression classifiers.
  • Feature Validity: The observed MMD for every target exceeds all values from 1,000 label shuffles and remains significant after Holm correction.The distributional gap appears in both the five-feature representation and klue/roberta-large embeddings, tested separately.
  • Robustness: 80.44 cross-source average AUC-ROC versus 70.34 for KatFishNet and 95.63–98.12 across GPT-5.2 temperatures support robustness to training source and decoding setting.The temperature results use a classifier trained on GPT-4o without retraining.

5 Feature-Guided Generation

Feature-guided prompting adds five form-level instructions to a reference-conditioned baseline. On GPT-5.2, experts rank refined poems above baseline poems, while several form statistics shift toward human poetry and spacing remains resistant.

  • Generation Setup: Feature-guided prompting improves expert naturalness judgments relative to the unconstrained baseline in a blind GPT-5.2 evaluation.The evaluation uses five native Korean literature students ranking 40 HUMAN–BASELINE–REFINED triplets.
  • Prompt Guidance: The five instructions target reference length, non-uniform line lengths, fewer connective endings, varied sentence-final forms, and flexible spacing.They operationalize the paper’s four linguistic dimensions as prompt-level guidance.
  • Expert Evaluation: 45.0 mean Borda points for REFINED versus 26.2 for BASELINE, with both REFINED and HUMAN above BASELINE at p < 0.001.REFINED and HUMAN do not differ significantly, but this does not establish equivalence.
  • Linguistic Shifts: 0.357 to 0.248: mean fconn moves toward the HUMAN mean of 0.206 under REFINED prompting.The pooled statistic covers Gemini-3 and GPT-5.2 outputs.
  • Linguistic Shifts: REFINED fburst distributions move toward HUMAN and become less concentrated than BASELINE, consistent with the expert rankings.The figure aggregates Gemini-3 and GPT-5.2 outputs.
  • Linguistic Shifts: 72%–107% of the BASELINE-to-HUMAN gap closes for volume, burstiness, ending diversity, and connectivity, but spacing closes only 13%.Spacing density changes from 0.243 to 0.236 against a HUMAN mean of 0.189, while spacing variability rises toward HUMAN.
  • Detector Evaluation: 82.73 to 57.67 on Gemini-3 and 96.42 to 84.28 on GPT-5.2: the feature-based detector’s separability decreases under REFINED.The authors treat detector evaluation as corroborating evidence, not evidence of detector evasion, because general-purpose scores lack calibrated direction.

6 Conclusion

The paper presents five interpretable features as a shared basis for detecting and guiding Korean LLM poetry. Detection generalizes across models, while prompting improves naturalness and most targeted form statistics, with spacing remaining difficult to change.

  • Conclusion: 83.60 average AUC-ROC across seven unseen LLMs is achieved by logistic regression over five features and remains effective across training generators and temperatures.The features encode length, line endings, rhythmic variation, and orthographic regularity.
  • Conclusion: Feature-guided prompts improve expert naturalness judgments and move most targeted properties closer to the human distribution, lowering detector separability.Spacing remains hard to influence through prompting alone.
  • Implication: Interpretable linguistic features serve as both diagnostic signals and generation-guidance targets within Korean poetry.The paper suggests adapting language- and genre-specific features beyond this setting.

Limitations

The paper’s scope is limited by Korean-specific form features, incomplete coverage of poetic qualities, and constrained evaluation and comparison settings.

  • The features are tailored to modern Korean, and transfer to other languages remains untested.
  • The form-level features do not capture imagery, metaphor, or discourse coherence, which require expert annotation.
  • Automatic analyses cover GPT-5.2 and Gemini-3, while expert evaluation covers only GPT-5.2 with five evaluators.
  • A larger multi-generator expert evaluation remains future work.
  • The study does not compare feature-guided prompting with LoRA or hard meter- and rhyme-based constraints.

Ethics Statement

The paper uses public texts and AI-generated content within stated ethical guidelines, while framing feature-guided prompting as analysis rather than concealment.

  • The study uses publicly available texts and generates AI content within ethical guidelines respecting privacy and intellectual property rights.
  • Feature-guided prompting may reduce detector separability and potentially be misused for evasion, but concealment is not the stated goal.
  • Detector changes are treated as diagnostic evidence rather than a primary objective.
  • The paper recommends that Korean poetry detectors avoid relying solely on shallow form-level cues.

B Model Specifications and Screening Details

The appendix specifies the source and evaluation models, generation setup, and screening procedure used to construct the expanded poem collection.

  • Model Specifications: The base models are GPT-4o, Solar, Qwen2-72B-Instruct, and Llama-3.1-70B-Instruct from KatFishNet.
  • Model Specifications: The additional evaluation models are Gemini-3-Pro, GPT-5.2, EXAONE-3.5-7.8B-Instruct, and EEVE-Instruct-10.8B.
  • Generation Configuration: New poems use default decoding with temperature 1.0 and top-p 0.95, conditioned on a human reference poem and poet age group.
  • Generation Configuration: The REFINED condition appends five feature-guided instructions to the same base prompt.
  • Screening Details: Screening excludes instruction violations, severely degraded quality, and meta-commentary or refusal responses.

C Qwen2-72B Boundary-Case Analysis

Qwen2-72B is a generator-specific boundary case because frequent line-final commas fall outside the five-feature taxonomy, weakening detection on that target.

  • Boundary Case: 37.6% of Qwen2-72B’s non-empty lines end in commas, compared with 3.3% for human poems.
  • Boundary Case: The corresponding line-final comma shares are 2.4% for GPT-5.2 and 2.6% for Gemini-3.
  • Boundary Case: 96.9 AUC-ROC is achieved by the line-final-comma feature alone in exploratory univariate analysis.
  • Boundary Case: The detector can misclassify a Qwen2-72B poem as human with probability 0.90.
  • Taxonomy Gap: The five features omit punctuation, explaining their poor separation of Qwen2-72B relative to punctuation-based cues.
  • Fusion Analysis: Adding KatFishNet’s punctuation and spacing features raises Qwen2-72B AUC from 79.78 to 91.99 but lowers average transfer from 83.60 to 82.48.
  • Ablation Analysis: Combining all four dimensions outperforms every single dimension group across targets, indicating complementary detection signals.
  • Cross-Source Robustness: Cross-source average AUC standard deviation is 3.54, indicating consistent performance across training-source choices.

G Robustness under Decoding Temperature

Across decoding temperatures, the detector remains effective without retraining, while feature-guided generation shifts several form statistics toward human-authored distributions. Prompt paraphrases preserve the direction of the effect, but broader prompt robustness remains untested.

  • Robustness under Decoding Temperature: 95.63–98.12 AUC-ROC: the classifier stays above 95 across GPT-5.2 decoding temperatures without retraining.It is trained on default GPT-4o poems and evaluated at temperatures 0.5, 0.8, 1.0, and 1.2.
  • Human–LLM Form Differences: LLM poems are generally longer than human poems, with fewer extremely short outputs and higher central tendencies across models.Human poems place substantial probability mass on outputs of ≤30 tokens, whereas LLM outputs shift toward longer sequences.
  • Human–LLM Form Differences: LLM outputs show lower ending-type diversity and burstiness than human poems, reflecting reused line endings and more uniform line lengths.Human distributions are broader and have longer tails toward higher diversity and rhythmic irregularity.
  • Human–LLM Form Differences: Model outputs cluster at higher spacing ratios than human poems, with GPT-5.2 showing a particularly high central tendency.Values above approximately 0.275 appear only in model-generated texts.
  • Guided Generation: Prompt paraphrases reduce detection AUC relative to BASELINE and bracket the original refined score, while broader prompt robustness remains untested.All checks use the same 189 GPT-5.2 reference poems and alter only the guidance prompt.

J Statistical Significance of Human Evaluation

The paired expert evaluation finds significant ranking differences among HUMAN, REFINED, and BASELINE poems. REFINED ranks above BASELINE, but its difference from HUMAN is not significant and does not establish equivalence.

  • Statistical Testing: p < 0.001: rankings differ across HUMAN, REFINED, and BASELINE conditions.The analysis uses paired triplets and a Friedman test followed by corrected pairwise comparisons.
  • Pairwise Comparisons: p < 0.001: REFINED receives higher expert naturalness rankings than BASELINE.BASELINE is most often ranked last, while HUMAN and REFINED occupy higher ranks.
  • Pairwise Comparisons: p = 1.0: REFINED differs nonsignificantly from HUMAN after Bonferroni correction, without establishing statistical equivalence.The supported conclusion is directional: feature guidance narrows the observed gap in perceived naturalness.
  • Evaluation Reliability: Krippendorff’s α = 0.378, indicating low agreement among pooled expert rankings despite the directional REFINED-over-BASELINE comparison.The paper notes that disagreement is common in creative-text ranking.
Loading 2608.28986v1…