PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Diversity in LLM Short Story Generation

Zahra Solati Dehkordi, Vasileios Lampos

arXiv:2610.06729v2cs.CL

TL;DR

LLMs can show limited diversity across repeated prompts, a concern for creative writing that depends on exploring possible ideas. DIVLM combines continued pre-training on creative-writing data with diversity-focused reinforcement learning, and the paper reports significant diversity gains while maintaining response quality.

  • Problem

    LLMs often show limited diversity across repeated or similar prompts, constraining creative-writing workflows that use them to explore possible ideas.

    Source: §1 · p. 1

  • Method

    DIVLM combines continued pre-training on creative-writing data with weight residuals to restore instruction-following capabilities, then applies reinforcement learning with composite rewards for genre, style, tone, and named-entity diversity.

    Source: §1 · p. 2, §4.2 · p. 4, §6 · p. 8

  • Results

    DIVLM yields significant gains in diversity performance while maintaining response quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Optimizing story-level diversity also improves embedding-based semantic distance.

    Source: §1 · p. 2

    The study evaluates short-story generation, not longer or interactive creative-writing tasks, and treats diversity as only one component of creativity.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) can generate accurate responses, but these are void of diversity. We attempt to address this for the task of creative short story generation. Drawing on established writing conventions and known LLM limitations, we target variation in genre, tone, style, and named entities. To promote diversity across these dimensions, we introduce DivLM, an LLM post-training framework consisting of two phases. First, we perform continued pre-training on a creative writing corpus and restore instruction-following capabilities using weight residuals. We then apply reinforcement learning with a custom, composite reward function that jointly maximizes diversity across the targeted narrative dimensions while maintaining response quality. Our empirical results on two LLM families show that DivLM increases diversity metrics by more than 9% on average compared to alternative approaches, while preserving instruction following, overall response quality, and similarity to human outputs.

1 Introduction

LLMs can produce accurate, helpful responses yet show limited diversity across repeated prompts, constraining their use for creative exploration. DIVLM trains for variation across story-level dimensions while maintaining output quality.

1 Introduction

  • LLM generations can show limited diversity across similar prompts, with post-training narrowing outputs toward typical responses and potentially causing mode collapse.This is a concern for creative writing, where users rely on models to explore possible ideas.Source: §1 · p. 1, Figure 1 · p. 2
  • The paper defines diverse short-story generation as producing high-quality stories for one prompt that differ along salient narrative dimensions.Its group-level rewards target variation across multiple generations rather than optimizing only a single fluent, instruction-following response.Source: §1 · pp. 1–2
  • DIVLM yields significant diversity gains while maintaining response quality.The framework combines continued pre-training on creative-writing data with reinforcement learning and group-level rewards for genre, style, tone, and named entities.Source: §1 · p. 2

2 Related Work

Prior diversity methods intervene during inference or training, but common objectives capture surface variation or reduce diversity to semantic distance. The paper identifies a need to target dimensions central to creative writing.

2 Related Work

  • Inference-time decoding and prompt changes encourage varied outputs without directly modifying the model’s learned generation behavior.Training-time methods instead incorporate diversity as an auxiliary post-training objective.Source: §2 · p. 2
  • Training-time diversity objectives use lexical or token-level proxies, embedding-space distances, or diversity over selected options.Embedding-based approaches move beyond lexical variation but often reduce diversity to one semantic-distance or equivalence signal.Source: §2 · pp. 2–3
  • Existing objectives may miss creative-writing diversity because surface metrics capture lexical variation and embedding distances compress distinct creative dimensions.The paper also distinguishes its aim—improving diversity—from work focused on preserving base-model diversity during post-training.Source: §2 · p. 3

3 Problem Formulation

The problem is to optimize groups of stories generated for the same prompts, prioritizing diversity while treating quality as a minimum requirement.

3 Problem Formulation

  • The objective optimizes a policy that generates groups of stories for prompts, seeking high group diversity while maintaining acceptable quality.Each output receives a reward score in [−1,2], with reward derivation deferred to section 4.2.Source: §3 · p. 3
  • Quality is enforced as a minimum requirement through reward design rather than treated as the primary optimization objective.Diversity is the primary objective.Source: §3 · p. 3
  • The formulation uses the previous policy parameters and story reward values in its condensed optimization objective.The passage identifies θ′ as the previous policy parameters and Λ_i as sampled from R(o_i).Source: §3 · p. 3

4 Improving Diversity in Creative Writing with DIVLM

DIVLM combines continued pre-training on creative-writing data with reinforcement learning to increase story diversity while retaining core requirements such as prompt adherence and completeness.

4 Improving Diversity in Creative Writing with DIVLM

  • DIVLM pairs continued pre-training with reinforcement learning to improve short-story diversity while retaining core requirements.The two phases are continued pre-training (CPT) and reinforcement learning (RL).Source: §4 · p. 3

4.1 Continued Pre-training For Creative Writing (CPT Phase)

  • CPT exposes the model to curated creative writing intended to strengthen stylistic flexibility, narrative variety, and later reinforcement learning.The corpus is designed to support varied and imaginative outputs and improve the effectiveness of RL.Source: §4.1 · p. 3
  • After LoRA training on frozen base weights, a scaled instruction residual is added back to recover instruction following while preserving CPT diversity gains.The residual is the instruction-tuned weights minus the base weights, and its scaling parameter s controls the restoration trade-off.Source: §4.1 · p. 3

4.2 Enhancing Diversity with Reinforcement Learning (RL phase)

  • GRPO samples multiple outputs per prompt and uses group-level reward signals, avoiding a separately trained value model.The group-based setup matches the need to assess output diversity across multiple responses to a prompt.Source: §4.2 · pp. 3–4
  • Quality scoring checks prompt adherence, absence of meta-commentary, and completeness, with critically flawed outputs receiving a fixed penalty.Structural and linguistic criteria such as length and coherence help prevent reward hacking through trivially short or malformed text.Source: §4.2 · p. 4
  • The diversity reward targets genre, style, tone, and named entities, using subtype balance for the first three and cross-output entity frequency for the last.The genre, style, and tone score rewards proximity to a uniform subtype distribution; the named-entity reward is separately formulated for an open-ended category set.Source: §4.2 · pp. 4–5
  • Total reward averages the four diversity scores but adds diversity only after quality reaches the validation-calibrated threshold τ.Below τ, training still receives a graded quality-driven signal rather than a hard zero reward.Source: §4.2 · p. 5

5 Experimental Results

Experiments on Llama-3.1-8B and Gemma-2-9B show that DIVLM improves diversity while keeping quality close to the instruction-tuned models. The section also examines how residual scaling and the two training phases affect these outcomes.

5.1 Datasets

  • The CPT corpus contains 18M tokens of curated creative writing, while RL and evaluation use filtered, balanced WritingPrompts data.The corpus combines Project Gutenberg and BookCorpus; the WritingPrompts filtering removes highly similar prompts and balances implied genres, styles, and tones.Source: §5.1 · p. 6

5.2 Diversity Categories and Judge Validation

  • The evaluation uses nine genre categories, three narrative styles, and three tones; LLM-judge agreement with humans is 89.86%, near the 91.63% human-human rate.The styles are descriptive, action-focused, and dialogue-heavy; tones are humorous, somber, and neutral.Source: §5.2 · p. 6

5.3 Evaluation Metrics and Experimental Settings

  • Diversity is scored per output group and quality per output; final settings are s=0.6, τ=2/3, h=5, with 1,000 generations per model.The 1,000 generations come from 100 test prompts with 10 outputs per prompt.Source: §5.3 · p. 6

5.4 Model Performance

  • CPT models outperform INS across diversity metrics but lower completeness and average quality; DIVLM improves completeness and narrows the human–LLM semantic-variation gap.Llama DIVLM reaches 76.2% of the human reference’s observed semantic variation, compared with 42.1% for Llama INS.Source: §5.4 · p. 7
  • RL rewards increase 86.5% for Llama and 42% for Gemma over INS, with gains over CPT of 23.7% and 9.7%, respectively, while DIVLM quality stays close to INS.Against DQO, DIVLM improves quality by 0.71%, genre/tone/style coverage by 50.34%, and diversity rewards by 85.53%; gains against DDPO are 85.39%, 8.91%, and 10.50%, respectively.Source: §5.4 · p. 7

5.5 Instruction Residual Scaling Analysis

  • Increasing the instruction-residual share improves adherence and completeness but reduces diversity; RL sustains diversity training only for s values from 0.5 to 0.9.At s=1, the model is more constrained, leaving insufficient flexibility for diversity optimization.Source: §5.5 · pp. 7–8, Figure 3 · p. 7

5.6 Ablation Study

  • GRPO-only training raises average diversity reward by just 18.3% over INS, indicating a limited diversity gain when RL starts from the more constrained instruction model.The authors interpret CPT as expanding the available response set and giving GRPO a broader trajectory space to optimize.Source: §5.6 · p. 8
  • Replacing DIVLM’s custom reward with cosine distance yields a 28.6% average-diversity-reward gain over INS, but tone and style rewards fall substantially.DLMCOS’s semantic distance is only 1.7% higher than DIVLM’s, while DIVLM improves diversity across the broader target dimensions.Source: §5.6 · p. 8

6 Conclusion

The conclusion presents DIVLM as a two-phase approach to short-story diversity across genre, tone, style, and named entities. It emphasizes that CPT and RL play complementary roles in improving diversity while maintaining response quality.

6 Conclusion

  • DIVLM targets limited diversity in LLM-generated short stories across genre, tone, style, and named entities.The conclusion frames these as four narrative dimensions.Source: §6 · p. 8
  • DIVLM combines creative-writing continued pre-training, instruction-following weight residuals, and diversity-focused RL that maintains overall response quality.Its composite reward encourages variation across the four narrative dimensions.Source: §6 · p. 8
  • CPT broadens the model’s output distribution, giving RL a richer trajectory space to explore and optimize.The conclusion identifies the two training phases as complementary.Source: §6 · p. 9

Limitations

The study is limited to short-story generation, models of at most 9B parameters, and output diversity rather than creativity broadly. It also defines a compact set of narrative dimensions and describes how its writing and prompt datasets are curated.

Limitations

  • The study evaluates only short stories; long-form stories and extended role-playing, where continuity and long-term memory may interact with diversity, remain untested.The authors identify these settings as future work.Source: Limitations · p. 9
  • Experiments are limited by computational and memory constraints to models with at most 9B parameters.The authors state that evaluating larger models would better clarify effectiveness and generalizability.Source: Limitations · p. 9
  • The work addresses output diversity, one component of creative behavior, rather than creativity in its broader sense.The authors identify reliable comprehensive evaluation and optimization of creativity as future work.Source: Limitations · p. 9

A About Writing Styles

  • The paper uses dialogue, action, and description as explicit writing modes, while related taxonomies differ on categories such as thought, exposition, and background.These modes are also called narrative modes in creative-writing and stylistics literature.Source: A About Writing Styles · p. 13

B Construction of Genre, Style, and Tone Dimensions

  • Genre, style, and tone are designed as distinguishable, interpretable dimensions; stories receive one dominant style label, while tone is treated as orthogonal to genre.Genre categories were derived from Goodreads book genres; style is grouped into descriptive, action-focused, and dialogue-heavy modes, and tone into humorous, somber, and neutral.Source: B Construction of Genre, Style, and… · p. 13

C.1 Creative Writing Dataset

  • The continued-pretraining corpus uses 150 genre-balanced books, filtered and segmented into English story passages of 500–800 tokens.The pipeline removes non-narrative material and applies toxicity screening to BookCorpus passages, but not to Project Gutenberg books; deduplication removes 0.6% of passages.Source: §C.1 · pp. 13–14

C.2 Writing Prompts Dataset

  • WritingPrompts prompts are filtered for community engagement, deduplicated using embedding similarity, and sampled uniformly across identified genres, styles, and tones.Pairs with cosine similarity above 0.55 lose one prompt; the sampling pipeline also classifies constrained, implied, and open-ended prompts and excludes invalid ones.Source: §C.2 · p. 14

D Quality Metrics

The supplementary analyses define quality checks for generated stories and compare alternative diversity rewards. The reported comparisons favor Proximity + Penalty for genre, tone, and style rewards, and the clipped named-entity formulation for limiting repetition while maintaining entity counts near baseline.

D Quality Metrics

  • Quality metrics assess prompt adherence, meta-commentary, and completeness using a judge and rule-based checks for the latter two.Each metric is binary; outputs failing either the judge or corresponding rule-based check are marked as failing.Source: D Quality Metrics · pp. 14–15
  • Outputs are classified as malformed for fewer than 50 words, a unique-token ratio below 30%, formatting artifacts, or failed coherence or grammar scores.Formatting artifacts include special-token leakage and HTML or CSS fragments.Source: D Quality Metrics · p. 15

E Alternative Formulations for the Diversity Reward

F Named Entity Reward

  • The clipped named-entity reward is the only tested formulation reported to keep unique entities close to baseline while significantly reducing repeated entities.The original reward produces more than twice as many unique entities as baseline; the penalty variant mitigates but does not fully prevent this increase.Source: F Named Entity Reward · p. 16, Table S2 · p. 16
  • Named-entity reward variants assign neutral scores to outputs containing no entities; the penalty variant additionally applies its entity-count penalty.The penalty is based on mean and standard deviation of entity counts estimated from validation outputs of the INS model.Source: F Named Entity Reward · p. 16

G LLM-as-a-Judge Evaluation Based on Human Labels

The authors compare human annotations with an LLM judge on a 100-sample subset of the test set. Overall agreement rates overlap, while agreement patterns differ between quality and diversity metrics.

G LLM-as-a-Judge Evaluation Based on Human Labels

  • Human-human agreement is 91.6% and judge-human agreement is 89.9% on the annotated evaluation subset.The 100 samples constitute 10% of the 1,000-sample test set.Source: G LLM-as-a-Judge Evaluation Based on… · p. 16
  • Judge-human agreement is slightly below inter-human agreement for quality metrics but slightly above it for diversity metrics.The authors attribute the latter pattern as possibly reflecting agreement with aggregate human judgment on more subjective diversity labels.Source: G LLM-as-a-Judge Evaluation Based on… · p. 16

H Training Configuration

The training configuration describes DIVLM and comparison methods, while the sensitivity analyses report trade-offs from instruction-residual scaling and improvements from stronger named-entity repetition penalties. The final named-entity sensitivity setting is h=5.

H Training Configuration

  • DIVLM uses continued pre-training followed by GRPO; comparisons include PPO-based DQO and DDPO trained on paired human-written responses and preference pairs.DIVLM trains on 3,000 prompts with group size G = 10; the CPT phase runs one epoch and GRPO runs two epochs.Source: H Training Configuration · p. 17

I Instruction Residual Scaling Analysis

J Sensitivity Parameter (h) in Named Entity Diversity Reward

  • At h=5, named-entity reward improves 55.25% over its initial value and repeated entities per group over total entities per group decrease by 50.47%.The authors report that h=1 does not sustain reduced repetition, while larger h values penalize repeated entities more harshly; final evaluation uses h=5.Source: J Sensitivity Parameter (h) in Named… · p. 18, Table S5 · p. 18

K Decoding Temperature Analysis

Across the tested decoding temperatures, DIVLM maintains higher diversity than INS, and increasing temperature alone does not bring INS to comparable diversity. Higher temperatures slightly increase named-entity, style, and tone diversity for DIVLM, while genre diversity remains largely unchanged.

K Decoding Temperature Analysis

  • The analysis evaluates test-set diversity metrics for T ∈ {0.8, 0.9, 1.0, 1.1, 1.2} against INS.Temperature controls output randomness by scaling logits before softmax normalization.Source: K Decoding Temperature Analysis · p. 18
  • DIVLM maintains substantially higher mean diversity reward than INS at every tested temperature, and INS at T = 1.2 does not reach DIVLM at T = 0.8.The comparison also reports a consistent gap in cosine distance across the evaluated range.Source: K Decoding Temperature Analysis · p. 18, Figure S7 · p. 19
  • For DIVLM, higher temperatures slightly increase named-entity, style, and tone diversity, while genre diversity remains largely unchanged.The complete diversity evaluation across temperatures is shown in Figure S8.Source: K Decoding Temperature Analysis · pp. 18–19, Figure S8 · p. 19

L Group Size Analysis

Group-size changes leave DIVLM's cosine-distance and diversity-reward gap over INS broadly stable, while its diversity metrics shift slightly from G = 5 to G = 10 and then stabilize. For length control, the authors adopt a linear penalty because it preserves average output length relative to INS.

L Group Size Analysis

  • DIVLM's mean diversity reward maintains a consistent gap over INS across group sizes, cosine distance stays approximately constant, and all reported diversity metrics stabilize after G = 10.From G = 5 to G = 10, genre, tone, and style diversity increase slightly while named-entity diversity decreases slightly.Source: L Group Size Analysis · p. 19, Figure S10 · p. 19, Figure S9 · p. 19

M Length Penalty

  • The penalty targets an average output length matching INS because output length can affect evaluation metrics and LLM-based judge metrics cannot be reliably length-normalized.The authors note that output-length variation may itself be a desirable form of diversity.Source: M Length Penalty · p. 19
  • The linear length penalty preserves average output length relative to INS, so the authors adopt it over the tested alternatives.Table S8 reports mean output length for INS and DIVLM under different penalty settings.Source: Table S8 · p. 19, M Length Penalty · p. 20
Loading 2610.06729v2…