PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

Yuhao Wu, Yushi Bai, Zhiqiang Hu, Juanzi Li, Roy Ka-Wei Lee

arXiv:2506.04180v1cs.CL

TL;DR

Long-form generation can lose coherence and logical consistency as sequences grow, and prior approaches often lack structured planning. SuperWriter-Agent adds staged planning, writing, and refinement with thinking-supervised training and hierarchical DPO; SuperWriter-LM outperforms same-size models and reports wins over 98% of user comparisons against top open-source baselines.

  • Problem

    Long-form generation remains challenging because coherence and logical consistency can degrade as sequences grow, while previous writing approaches often lack explicit planning and intermediate thinking.

    Source: §1 · p. 2

  • Method

    SuperWriter-Agent creates planning, writing, and refining supervision, while hierarchical DPO uses MCTS to propagate final-output assessments to generation steps.

    Source: §1 · p. 2

  • Results

    SuperWriter-LM outperforms all same-size models on WritingBench, exceeds 671B DeepSeek-R1 in key domains, and wins over 98% of real-user comparisons against top open-source baselines.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The authors conclude that multi-stage generation and structured preference learning are valuable for improving writing quality.

    Source: §6 · p. 10

    The alignment stage relies on offline DPO from static preference pairs and lacks online RLHF’s adaptivity.

    Source: §7 · p. 10

Abstract

from arXiv · show

Long-form text generation remains a significant challenge for large language models (LLMs), particularly in maintaining coherence, ensuring logical consistency, and preserving text quality as sequence length increases. To address these limitations, we propose SuperWriter-Agent, an agent-based framework designed to enhance the quality and consistency of long-form text generation. SuperWriter-Agent introduces explicit structured thinking-through planning and refinement stages into the generation pipeline, guiding the model to follow a more deliberate and cognitively grounded process akin to that of a professional writer. Based on this framework, we construct a supervised fine-tuning dataset to train a 7B SuperWriter-LM. We further develop a hierarchical Direct Preference Optimization (DPO) procedure that uses Monte Carlo Tree Search (MCTS) to propagate final quality assessments and optimize each generation step accordingly. Empirical results across diverse benchmarks demonstrate that SuperWriter-LM achieves state-of-the-art performance, surpassing even larger-scale baseline models in both automatic evaluation and human evaluation. Furthermore, comprehensive ablation studies demonstrate the effectiveness of hierarchical DPO and underscore the value of incorporating structured thinking steps to improve the quality of long-form text generation.

1 Introduction

Long-form generation can lose coherence and logical consistency as text grows, while prior single-pass approaches often lack planning and sustained coherence. SuperWriter-Agent adds structured thinking stages, and the paper introduces supervised training data and hierarchical DPO for generation.

1 Introduction

  • Prior long-form writing approaches often generate in one pass without structured thinking or explicit planning, and can develop contradictions over longer spans [6] [35] [21] [28].Source: §1 · p. 2
  • SuperWriter-Agent embeds intermediate cognitive steps in long-form generation through a structured writing process modeled on human planning, drafting, reviewing, and revision.The framework coordinates agents to support this process.Source: §1 · p. 2
  • The paper segments supervised fine-tuning data into planning, writing, and refining stages, then uses hierarchical DPO with MCTS to propagate final-response feedback to generation steps [39].Source: §1 · p. 2
  • On WritingBench, the authors report improvements in fluency, coherence, and logical consistency over existing methods, and use ablations to examine structured thinking data and hierarchical DPO [60].Source: §1 · p. 2

2 SuperWriter-Agent

SuperWriter-Agent addresses the scarcity of writing-process supervision with a three-stage framework for planning, paragraph-level generation, and refinement. Its stages provide an outline, use explicit thinking during writing, and review text for targeted revision.

2 SuperWriter-Agent

  • The framework generates thought-enriched writing supervision through planning, paragraph-level writing, and iterative refinement, addressing the limited representation of writing processes in training corpora.Source: §2 · pp. 2–3, Figure 2 · p. 3

2.1 Stage 1: Plan

  • Planning uses oral narration and iterative dialogue to develop themes, arguments, settings, paragraph structures, and word allocations into a detailed outline.Source: §2.1 · p. 3

2.2 Stage 2: Write

  • Writing follows a reasoning-oriented, two-stage “thinking before writing” approach; the thinker organizes paragraph ideas and structure before drafting.Source: §2.2 · p. 3
  • The writer uses the outline and preceding paragraph as context when generating each paragraph, supporting transitions and document-level logical flow.Source: §2.2 · p. 4

2.3 Stage 3: Refine

  • Refinement evaluates text quality, identifies paragraphs needing revision, and uses checker feedback to target issues such as inconsistencies, unclear expressions, and structural deviations.Source: §2.3 · p. 4

3 SuperWriter-LM

Section 3 asks whether a model can internalize SuperWriter-Agent’s structured writing process to improve long-form content with fewer inference steps. It describes staged SFT data and hierarchical DPO that propagates final-output evaluations to earlier generation decisions.

3 SuperWriter-LM

  • The training question is whether a model can internalize SuperWriter-Agent’s structured thinking and writing capabilities rather than relying on 30 to 40 agent calls per sample.The stated objective is to improve coherence, relevance, and depth, not merely extend output length.Source: §3 · p. 4

3.1 SFT-Training

  • From 4,000 filtered instructions, the agent produces 4,000 instances for each plan, write, and refine stage, totaling 12,000 training instances.At inference, the model runs the three stages sequentially to produce the final output.Source: §3.1 · p. 4
  • Stage-wise SFT supports review of intermediate outputs and keeps each sample within 32K tokens, although some complete agent outputs exceed 100K tokens.The stages map query to outline, outline to draft, and draft to final output.Source: §3.1 · p. 4

3.2 Hierarchical DPO for Multi-Stage Generation

  • Conventional DPO on final outputs alone misses preference signals from earlier steps in the sequential planning, drafting, and refinement process.The proposed hierarchical multi-stage DPO combines structured preference-data construction with systematic evaluation.Source: §3.2 · p. 5
  • MCTS explores plans, drafts, and refinements, then back-propagates leaf quality signals under assumptions that stronger plans support better drafts and refined drafts support better finals.The evaluation scores final outputs using a task-selected six-dimension rubric, repeats scoring three times, and averages the scores.Source: §3.2 · p. 5
  • The preference dataset combines pairs from each hierarchy level, and the resulting policy is optimized with the standard DPO loss.The paper reports continuing DPO training of the supervised fine-tuned model with a batch size of 32 and learning rate of 1 × 10−6.Source: §3.2 · pp. 6–7
  • Table 1 scopes WritingBench comparisons across six writing domains and three requirements—style, format, and length—using a 1–10 critic scale.The caption identifies category-specific scores but supplies no model scores or performance ranking.Source: Table 1 · p. 7

4 Experiment

SuperWriter-LM is evaluated on WritingBench and real-world user queries, with comparisons covering benchmark scores, pairwise win rates, and component ablations. It ranks second on WritingBench overall, leads same-size models in win-rate comparisons, and improves as structured stages and hierarchical DPO are added.

4.1 Experiment Setup

  • WritingBench assesses outputs across writing domains, while nearly 200 curated real-world instructions are used for pairwise comparisons against several baselines.The WritingBench critic scores style, format, and length; the Writing-Model-Qwen-7B baseline is an SFT model obtained via rejection sampling with that critic.Source: §4.1 · pp. 7–8

4.2 WritingBench Result

  • 8.51 average on WritingBench ranks SuperWriter-LM second behind DeepSeek-R1 [12]; its 8.6 Chinese and 8.5 English scores include a match with DeepSeek-R1 in English.The benchmark covers 1,200 prompts across six writing domains and 100 sub-domains.Source: §4.1 · p. 7, §4.2 · p. 8
  • 8.6 in Academic & Engineering and 8.7 in Finance & Business, Politics & Law, and Education are the highest domain scores, slightly exceeding DeepSeek-R1; length_C is the exception.The length_C discrepancy is attributed to agent-generated data tending to produce longer outputs even for short-text tasks; the passage says it does not affect long-form generations.Source: §4.2 · p. 8

4.3 Win-Rate Result

  • SuperWriter-LM leads same-size models substantially in win-rate comparisons and remains competitive with larger models, sometimes slightly outperforming state-of-the-art LLMs.The paper describes the comparisons as setting a new performance benchmark among 7B-scale models and challenging existing state-of-the-art systems.Source: §4.3 · p. 9
  • Human annotators also prefer SuperWriter-LM over Writing-Model-Qwen-7B, though frequent ties on subtle differences make the overall win rate appear slightly lower.The assessment covers approximately 200 real-world queries.Source: §4.3 · p. 9

4.4 Ablation Study

  • 8.21 for SuperWriter-final-answer rises to 8.47 with +Three-Stage and 8.51 with hierarchical DPO on WritingBench.The passage reports consistent improvement with each added component; the listed ZH/EN pairs are 8.3/8.2, 8.5/8.4, and 8.6/8.5, respectively.Source: §4.4 · p. 9

5 Related Work

Related work addresses long-context input capacity and long-form writing methods, while noting a gap in output capability and consistently coherent, high-quality long-form generation. SuperWriter responds with an agent framework and training methods for coherent extended text.

5 Related Work

  • Long-context research has focused on expanding input capacity through zero-shot methods or fine-tuning, while reported output lengths can remain far below input limits.Prior findings cited in the passage describe maximum input lengths over 100K tokens versus achievable output lengths of approximately 2K words.Source: §5 · p. 9
  • Long-form approaches include recursive prompting, task decomposition, personalization, and detailed question answering, but challenges in coherence and consistently high-quality generation remain.The passage also identifies limited theoretical grounding as a limitation of existing methods.Source: §5 · pp. 9–10
  • SuperWriter addresses these challenges with a theoretical agent framework for long-form writing and training methodologies targeting coherent, extended text.Source: §5 · p. 10

6 Conclusion

SuperWriter combines a planning, writing, and refining process with hierarchical DPO to train a model to align outputs across writing stages. Experiments report stronger WritingBench performance than same-size models, more than 98% wins against top open-source baselines, and advantages over the 671B DeepSeek-R1 in key domains.

6 Conclusion

  • SuperWriter uses a planning, writing, and refining process to teach the model to think before writing and produce high-quality supervision signals.Source: §6 · p. 10
  • Hierarchical DPO trains the model to align outputs across all writing stages.Source: §6 · p. 10
  • SuperWriter-LM outperforms same-size models on WritingBench, exceeds 671B DeepSeek-R1 [12] in key domains, and wins over 98% of real-user comparisons against top open-source baselines.The conclusion presents these results as evidence for multi-stage generation and structured preference learning.Source: §6 · p. 10

7 Limitation

The paper identifies inference latency, model scale, and reliance on offline alignment as limitations of SuperWriter-LM. These constrain deployment speed and may affect knowledge-intensive writing and adaptive refinement.

7 Limitation

  • Three sequential forward passes may increase user-perceived latency, although the workflow is more efficient than multi-round pipelines requiring 30–40 calls per output.The three stages are plan, write, and refine.Source: §7 · p. 10
  • The 7B Qwen2.5 backbone may limit internal world knowledge in specialized writing, where qualitative outputs showed shallow factual grounding or subtle reasoning errors.Examples include legal, medical, and scientific writing.Source: §7 · p. 10
  • The alignment stage uses offline DPO trained on static preference pairs, lacking online RLHF’s adaptive exploration; high rollout costs for reward models remain a bottleneck.The authors identify scalable, low-latency reward models or reward distillation as future research directions.Source: §7 · p. 10

A.1 Stage-1 Plan Prompt

The appendix overviews six Stage-1 Plan prompt modules and points to their detailed templates and usage notes.

A.1 Stage-1 Plan Prompt

  • BrainStorm develops an in-depth initial thinking process to analyze a task and form a preliminary writing plan.Source: §A.1 · p. 18
  • BrainStorm Review examines task design for potential flaws, ambiguities, or unclear requirements.Source: §A.1 · p. 18
  • BrainStorm Refine applies editorial judgment to reviewer feedback, revising or rewriting the task design.Source: §A.1 · p. 18
  • Outline structures the article plan with estimated word counts per paragraph and a logical framework.Source: §A.1 · p. 18
  • Check outline reviews logical structure and completeness, identifying gaps or missing elements against the intended objectives.Source: §A.1 · p. 18
  • Refine outline incorporates reviewer feedback to improve clarity, completeness, and alignment with writing objectives.Source: §A.1 · p. 18

Think template

The thinking template guides a writing plan from purpose and structure through genre-specific content and presentation requirements.

Think template

  • Writers first define the piece’s purpose and type, including its objective, style, genre, and stylistic characteristics.Source: Think template · p. 18
  • The template asks writers to identify key points and develop a coherent structure that maintains logical flow.Source: Think template · p. 18
  • For narrative writing, it calls for character traits and motivations alongside pivotal plot points and emotional cues.Source: Think template · p. 18
  • It also specifies formatting and genre- or task-specific elements, noting that summaries may use a simpler step-by-step approach.Source: Think template · p. 18

BrainStorm

The BrainStorm prompt asks a professional writer to develop an initial task design by thoroughly analyzing a supplied thinking template.

BrainStorm

  • The prompt frames BrainStorm as a professional writer’s initial task-design step.Source: BrainStorm · p. 18
  • It directs the writer to examine every template point and provide a comprehensive plan without vague information.Source: BrainStorm · p. 18
  • The requested thinking should be thorough while keeping the structure manageable, and it explicitly does not yet require an outline.Source: BrainStorm · p. 18

BrainStorm Review

This appendix presents prompt modules for planning, drafting, reviewing, revising, and evaluating long-form text, along with sample outputs and a method for estimating requested length. It describes prompts that estimate expected output length from task characteristics and return either an over-2,000-word judgment or a word-count category with a brief justification.

A.2 Stage-2 Write Prompt

  • The Stage-2 pipeline first plans each paragraph from the outline, prior text, and key point, then uses that plan to generate a coherent paragraph.The plan covers purpose, organization, transitions, supporting details, and style; the writing module returns only the paragraph, enclosed by delimiters.Source: §A.2 · pp. 19–20, Write-thinker · p. 20, Writer · p. 20

A.3 Stage-3 Refine Prompt

A.4 Evaluation Prompt for Hierarchical DPO

  • The hierarchical-DPO evaluation pipeline selects and query-customizes three general and three relevant special criteria, then scores each with analysis and a numeric score.Its structured, query-specific evaluations are designed to be interpretable, machine-parsable, and usable with DPO loss.Source: §A.4 · p. 21, prompt · p. 23

evaluation_criteria Prompt

SYSTEM_PROMPT for GPT-4.1 Win-Rate Evaluation

  • A GPT-4.1 judge compares two responses on relevance, writing quality, creativity, specificity, and tone, and returns a graded winner verdict or tie.The listed verdicts distinguish significant from slight advantages for either response.Source: SYSTEM_PROMPT for GPT-4.1 Win-Rate… · pp. 23–24

A.6 Case Study

  • The case-study sample is an academic-style response on digital inclusive finance that frames asset allocation, equity trade-offs, and risk mitigation as gaps in prior research.It describes a mixed-methods study proposal and argues that access-focused measures may miss effects on household behavior and risk exposure.Source: §A.6 · p. 24

A.7 Predicting User’s Length Requirement with DeepSeek-R1-distill-Qwen-32B

  • Length estimation uses few-shot R1-distill-Qwen-32B predictions in two steps: classify whether the request exceeds 2,000 words, then assign a length category.The second step chooses among 2K–4K, 4K–8K, 8K–16K, and 16K+ based on contextual hints and input characteristics.Source: §A.7 · p. 26
  • The length-estimation prompts assess depth, scope, structure, research requirements, and expected detail to judge how long a response should be.One version asks whether the response is likely to exceed 2000 words; another asks for a category ranging from less than 2000 to 16000 words, with examples and a brief justification.Source: §A.7 · pp. 26–27
Loading 2506.04180v1…