Source-linked AI summary
DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writing
Qian Cao, Yahui Liu, Wei Bi, Yi Zhao, Ruihua Song, Xiting Wang, Ruiming Tang, Guorui Zhou, Han Li
TL;DR
RL enhancement can collapse the diversity of LLM outputs, limiting creative writing applications. DPWriter structures generation around explicit planning, branches plans by diversity, and adds a group-aware diversity reward; experiments report improved diversity without sacrificing quality. The framework’s computational overhead and unresolved diversity–quality balance remain limitations.
Problem
RL-based enhancement can reduce LLM output diversity, while existing methods lack explicit mechanisms for controlling diverse exploration.
Method
DPWriter uses semi-structured long CoT with planning-stage Diverse Planning Branching and a group-aware diversity reward to guide diverse trajectories.
Results
DPWriter consistently outperforms baselines across creative writing benchmarks in diversity and quality, including 15% higher embedding-based diversity and 9.9% higher EAD than standard GRPO on WritingBench with Qwen3-4B.
Takeaways & Limitations
The framework provides a planning- and reward-based approach for promoting more expressive and versatile language generation in open-ended applications.
Takeaways & Limitations
The semi-structured CoT and DPB add computational overhead, the diversity–quality trade-off may remain unresolved, and benefits for creativity beyond quality are open.
Abstract
from arXiv · showhide
Reinforcement learning (RL)-based enhancement of large language models (LLMs) often leads to reduced output diversity, undermining their utility in open-ended tasks like creative writing. Current methods lack explicit mechanisms for guiding diverse exploration and instead prioritize optimization efficiency and performance over diversity. This paper proposes an RL framework structured around a semi-structured long Chain-of-Thought (CoT), in which the generation process is decomposed into explicitly planned intermediate steps. We introduce a Diverse Planning Branching method that strategically introduces divergence at the planning phase based on diversity variation, alongside a group-aware diversity reward to encourage distinct trajectories. Experimental results on creative writing benchmarks demonstrate that our approach significantly improves output diversity without compromising generation quality, consistently outperforming existing baselines.
1 Introduction
DPWriter addresses diversity loss in RL-enhanced LLMs by guiding generation through semi-structured planning and diversity-aware exploration. Its planning branching and group-aware reward improve diverse creative writing while maintaining quality.
- RL training can reduce the content diversity of LLM outputs, especially for creative writing and RLHF-trained models.
- Existing diversity methods modify rewards or branch rollouts, but often leave trajectory exploration weakly controlled or prioritize efficiency and performance.
- DPWriter decomposes generation into global planning, long CoT reasoning, and final response stages to guide diverse exploration.
- The framework is supported by a curated dataset containing 43K writing instructions with semi-structured long CoT and high-quality responses.
- Diverse Planning Branching selects diverse candidate plans during rollout, while a group-aware diversity reward evaluates each response by its contribution within its group.
- Experiments across creative writing benchmarks report higher diversity and quality than existing baselines, with DPB and the diversity reward working synergistically.
2 Related Work
Prior work addresses diversity through supervised or preference-based training and through RL exploration strategies. However, branching methods generally target efficiency or performance rather than explicit diversity control.
- SFT and preference optimization can reduce output diversity, motivating methods that diversify training data or modify optimization objectives.
- RL-based diversity methods have gained attention as RL improves model capabilities and performance.
- Existing branching approaches primarily seek sample efficiency or overall performance instead of explicitly promoting diversity.
- These methods often branch at high-entropy tokens or use fixed segment lengths, making rollout diversity less controllable.
3 Preliminaries
The paper formulates open-ended generation as producing a response through reasoning, then introduces an explicit planning stage before reasoning. Its RL objective maximizes expected reward using GRPO-style group-relative optimization.
- For an instruction q, the model generates a reasoning chain c and final response y, with the response conditioned on the reasoning chain.
- Existing long-CoT reasoning is unstructured and implicitly learned, lacking explicit planning representations.
- The semi-structured paradigm generates a global plan p, reasoning chain c conditioned on q and p, and response y conditioned on q, p, and c.
- The RL objective maximizes expected cumulative reward, where r(q, o) evaluates rollout quality for instruction q.
- GRPO estimates rollout advantages within a group and uses clipped policy updates with a reference-policy divergence penalty for stability.
4 Constructing Semi-structured CoT Data with Planning
The paper constructs semi-structured CoT data by adding explicit multi-aspect plans and plan-consistent reasoning to guide coherent, controllable generation.
- Multi-aspect Planning Generation: The data-construction method generates plans that guide both long CoT reasoning and final response generation.Plans cover goals and audience, information and perspective, structure and logic, and language and style.
- Planning Structure: The framework uses a global plan before reasoning, making intermediate planning explicit in the generation process.
- Plan-consistent CoT Generation: GPT-4.1 generates plans from instructions and responses, then revises original CoT into plan-consistent reasoning.This revision addresses potential inconsistency between directly inserted plans and the reasoning process.
- Structured Representation: Special tokens enclose individual plan aspects so the model can recognize and follow the structure.The resulting examples contain instruction, plan, revised CoT, and response fields.
5 Method
The method combines planning-level branching with quality- and group-aware diversity rewards to explore varied trajectories while retaining response quality.
- Overview: The method exploits planning controllability in semi-structured CoTs through Diverse Planning Branching and diversity-contribution rewards.
- Planning Capability Cold Start: Supervised fine-tuning on semi-structured CoT data cold-starts the model to learn planning formats and coherent reasoning.The paper reports comparable or better performance than other CoT datasets in Section 6.3.
- Diverse Planning Branching: At each planning segment, every candidate produces K continuations, forming a pool of size |C|×K before selecting G diverse candidates.Subsequent segments preserve branch diversity by selecting one continuation from each originating candidate group.
- Diversity Metrics: Branching diversity uses either distinct n-grams for lexical variety or average pairwise embedding cosine distance for semantic variation.
- Quality Reward: The quality reward uses a human-preference-trained reward model to assess generated responses.
- Diversity Contribution Reward: The Diversity Contribution Reward favors responses containing unique elements absent from the other responses in their group.
- Balancing Quality and Diversity: Diversity reward contribution is activated only when response quality exceeds threshold τ, preserving a minimum quality requirement.The combined reward uses λ to control the diversity component and sets λ = 0 below the threshold.
6 Experiments
Experiments evaluate DPWriter across creative-writing benchmarks, component ablations, and qualitative comparisons. The results report consistent gains in diversity and quality, with branching and diversity reward contributing jointly.
- Experimental Setup: Experiments use WritingBench, Creative Writing v3, ArenaHard v2.0, and NoveltyBench to assess quality and diversity.Table 1 reports quality and diversity metrics, while NoveltyBench provides an additional Distinct-based diversity assessment.
- Main Results: DPWriter consistently outperforms baselines on WritingBench and Creative Writing v3 across quality and diversity metrics.The comparison spans different model backbones and reports gains on both evaluation dimensions.
- Main Results: 15% embedding-based diversity and 9.9% EAD improvement over standard GRPO on WritingBench accompany DPWriter's highest overall score of 6.43.These results use the Qwen3-4B backbone.
- Main Results: DPWriter outperforms all baselines on NoveltyBench's Distinct metric, supporting gains in diverse content generation.The paper describes consistent quality-and-diversity improvements across multiple benchmarks.
- Ablation Studies: Removing the diversity reward causes a significant decline in all diversity metrics.This ablation highlights the role of explicitly rewarding diversity during model learning.
- Ablation Studies: Combining branching with diversity reward yields the highest diversity scores across branching factors K ∈ {16, 32, 64, 128}.Diversity metrics also improve as K increases.
- Ablation Studies: DPWriter produces five distinct responses spanning Harry Potter books, whereas GRPO repeats “Harry Potter and the Philosopher’s Stone” in four of five generations.The case study qualitatively compares five generations from each model on NoveltyBench.
7 Conclusion
DPWriter addresses diversity collapse in RL-enhanced language models with semi-structured long CoT reasoning, planning-phase branching, and group-aware diversity rewards. Across creative-writing tasks, the framework promotes output diversity without sacrificing quality.
- Conclusion: DPWriter combines semi-structured long CoT reasoning, strategic planning-phase branching, and group-aware diversity rewards.The framework explicitly guides diversity exploration during generation.
- Conclusion: Experiments across multiple creative-writing tasks show increased output diversity without sacrificing quality.The conclusion presents diverse planning and targeted reward signals as a pathway toward more expressive generation.
Limitations
DPWriter improves output diversity in creative writing, but its semi-structured CoT and branching strategy may add computational overhead. The balance between diversity and quality, and diversity’s effect on creativity, remain open questions.
- Semi-structured CoT and Diverse Planning Branching may introduce computational overhead that limits scalability for extremely large models or datasets.
- DPWriter improves diversity without compromising quality, but the trade-off between these aspects may not be fully resolved.
- Whether increased diversity benefits creativity beyond quality improvement remains an open question.
A Implementation Details
The implementation uses fixed RL training, sequence-length, branching, diversity-evaluation, and reward settings, with embedding-based plan similarity.
- RL training runs for 5 epochs with a maximum prompt length of 1024 and response length of 3072.
- The branching factor is K = 32, and an n-gram-based strategy evaluates diversity in diverse planning branching.
- The diversity reward uses weight λ = 0.6 and quality threshold τ = 10, while Qwen3-Embedding-0.6B extracts plan embeddings for similarity calculation.
B Additional Related Works
The related-work discussion distinguishes inference-time diversity methods from training-time approaches and describes prompt-based construction of semi-structured CoT data.
- Diversity methods for LLMs are broadly categorized into inference-time and training-time approaches.
- Inference-time methods modify next-token selection through temperature changes or token-distribution cutoff strategies.
- The paper uses separate prompts to generate multi-aspect plans, plan-consistent CoT, and long CoT for instruction-response data lacking existing reasoning traces.
D More Case Studies
Additional case studies illustrate DPWriter’s ability to produce high-quality, coherent, and creative writing while following generated plans and plan-consistent reasoning.
- WritingBench: DPWriter generates high-quality WritingBench responses by following multi-aspect plans and plan-consistent CoT.
- NoveltyBench: DPWriter produces creative and coherent NoveltyBench stories by adhering to generated plans and thought processes.
- A sample case study examines DPWriter generations for a narrative involving weather, preparation, encounter, shelter, and acceptance or epiphany.