Source-linked AI summary
A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs
Yifan Song
TL;DR
Existing evaluation largely scores final writing rather than outlines, leaving outline quality difficult to assess independently. This paper compares seven frameworks across three granularities using a controlled benchmark and finds no stable global ranking, while outline- and writing-side rankings correlate only moderately.
Problem
Existing writing-side evaluation scores generated writing rather than outlines, conflating outline quality with writing quality and leaving independent outline evaluation unclear.
Method
The study constructs a unified head-to-head benchmark of 7 frameworks across 3 generation granularities, varying framework structure while holding the LLM backend, evaluator model, and metric family fixed.
Results
No single framework dominates across granularities; SuperWriter ranks first in length-constrained single-chapter mode with HANNA 6.83, but falls to HANNA 5.89 in whole-book mode, while outline- and writing-side rankings correlate only moderately.
Takeaways & Limitations
Framework performance depends on task granularity, and outline quality should be evaluated separately from the quality of the writing it produces.
Takeaways & Limitations
Representative frameworks including MoPS, Re3, DOC, and RecurrentGPT are not included, so external validity has not been verified on them.
Abstract
from arXiv · showhide
Long-form generation exposes fundamental limitations of large language models. Even 70B-parameter models exhibit length collapse at 16k-token outputs, and multi-chapter stories frequently trigger the attribute drift characteristic of the ``lost-in-the-middle'' effect. The ``outline-first, write-later'' paradigm has gained wide adoption, yet existing research evaluates the final writing rather than the outline itself, conflating two evaluation objects that should be decoupled. We construct a unified head-to-head benchmark covering 7 representative long-form generation frameworks across 3 generation granularities -- single-chapter, multi-chapter, and whole-book -- and propose an anchor-based LLM-as-a-judge protocol that directly assesses outlines against the source text on a 5-point anchored scale. Across 21 framework-granularity cells, no single framework dominates; performance depends on the match between a framework's intrinsic output form and the target granularity. SuperWriter ranks first in the length-constrained single-chapter mode, but this advantage degrades in whole-book mode. The outline-side ranking correlates only moderately with the writing-side ranking, supporting the outline--writing decoupling principle. Compute constraints limit the writing-side evaluation to a subset of cases; follow-up experiments will expand the sample size and add cross-model evaluators to enable stronger statistical inference.
1 Introduction
The paper addresses the unresolved problem of evaluating outlines independently from final long-form writing. It introduces a controlled benchmark and direct outline-judging protocol, finding that framework performance varies by granularity and that outline- and writing-side rankings only moderately agree.
- Problem: Long-form generation is constrained by structural coherence, long-range consistency, and length compliance, with lost-in-the-middle effects contributing to drift and contradictions.At 16k/32k-token outputs, 70B-parameter models achieve 11.5% on STIC-2, while models often stop near 2,000 characters when asked for 4,000.
- Problem: Existing writing-side evaluations assess final text rather than outlines, making framework-level outline comparisons indirect and conflating two evaluation objects.This gap obscures differences between outline quality and writing quality.
- Contributions: The benchmark compares 7 long-form generation frameworks across 3 granularities in 21 controlled framework-granularity cells.The comparison uses a unified LLM backend while controlling faithfulness to original-paper skeletons.
- Contributions: The anchor-based LLM-as-a-judge protocol evaluates outlines directly against source text on a 5-point anchored scale, while separating outline-side and writing-side tracks.The tracks share only the source text; the outline side contains approximately 89 cases, while the writing side uses a subset of samples.
- Findings: No single framework dominates across granularities, and outline-side and writing-side rankings correlate only moderately.These findings support the outline-writing decoupling principle.
- Limitation: Compute constraints limit writing-side evaluation to a subset of cases, with larger samples and additional evaluator models planned for follow-up experiments.The planned expansion is intended to support stronger statistical inference.
2 Related Work
Related work spans outline-driven planning, multi-agent collaboration, reflection-based revision, and increasingly fine-grained LLM-based evaluation. The paper positions its contribution as a controlled side-by-side benchmark that directly evaluates outlines rather than inferring their quality from writing outputs.
- Outline-driven generation: Outline-first, write-later generation separates planning from writing and has been extended through recursive reprompting, fine-grained outline control, and natural-language memory simulation.This paradigm originates in neural story generation and targets long-range coherence.
- Multi-agent and reflection-based revision: Multi-agent approaches coordinate planning and writing agents, encode procedures as prompt sequences, or iteratively refine outlines through dialogue.They model complex long-form generation as collaboration or process supervision.
- Evaluation methodology: Existing writing-side evaluation targets generated stories rather than outlines, leaving direct outline quality insufficiently measured.The paper identifies this as the methodological gap addressed by its protocol.
- Positioning: The paper differs from single-framework proposals by providing a controlled platform for evaluating representative frameworks side by side.Its unified protocol combines multiple narrative-quality, length, and query-relevant scoring components.
3 Method
The method compares seven long-form generation frameworks under a unified benchmark, varying generation granularity while separating outline-side evaluation from writing-side evaluation. It also documents controlled and uncontrolled factors, observational research questions, and limitations affecting interpretation and reproducibility.
- Experimental design: Seven frameworks are compared across three loading modes while holding the LLM backend, evaluator model, and metric family fixed.The modes are single, multi-chapter, and whole-book; metric dimensions activate according to mode.
- Limitations and controls: The comparison is not fully univariate because frameworks retain different sampling parameters, prompt-level biases, and residual placeholder defects affecting 12 cells.The evaluator and generator are also the same model, so self-preference bias is not mitigated and winning claims are directional rather than statistically significant.
- Research questions: The study asks descriptive questions about framework rankings, transfer from outline quality to writing quality, rank correlation, and metric co-variation.Causal attribution and broader cross-language-by-genre robustness are left for future work.
- Datasets and loading modes: The benchmark treats each sample as an existing source text paired with an explicitly declared target length.Single mode uses one chapter, whereas whole-book mode concatenates the entire book; target_chars contributes to length-compliance scoring.
- Limitations and controls: Writing-side evaluation is compute-constrained, while human calibration uses only four samples and serves as a directional reference.Future work will expand the sample and calibration set and add additional evaluator models.
- Evaluation protocol: Outline-side primary evidence evaluates outlines directly, whereas writing-side auxiliary evidence evaluates long texts expanded from those outlines.The two tracks share the source text, but the expansion step may introduce gains or losses in faithfulness.
4 Experiments
Across outline-side and writing-side experiments, framework performance varies by dimension and generation mode: structure-driven methods generally lead outline faithfulness, while SuperWriter’s advantages weaken in whole-book settings and do not uniformly transfer to writing quality.
- Outline-side main results: Structure-driven frameworks outperform non-outline baselines on faithfulness, with A2 rising from 3.42 for naive to 4.43 for SuperWriter and A3 from 3.39 to 4.71.The advantage holds across all three per-mode tables.
- Outline-side main results: SuperWriter ranks first on 6 of 8 outline dimensions and scores 4.36 overall, about 0.7 points above second-place CogWriter at 3.65.Dome leads SuperWriter only on B2 hierarchy, 4.24 versus 3.46.
- Outline-side main results: Naive ranks last or second-to-last across all 8 dimensions, while cot and selfrefine remain below structure-driven frameworks overall at 2.95–3.03 versus 3.55–4.36.The comparison indicates a broad outline-side disadvantage for direct-generation baselines.
- Outline-side main results: StoryWriter ranks second on C1 executability at 4.10, whereas CogWriter scores 2.86 on B2 hierarchy because its block-level planning is weak across layers.The CogWriter result is consistent with its disabled constraint set and degraded writing strategy.
- Output length: StoryWriter expands from 4,142 to 28,307 characters in wholebook mode, a 6.8× increase, while SuperWriter has the highest standard deviations across all modes.SuperWriter’s standard deviations are ±4,864, ±4,275, and ±16,516 for single, multichap, and wholebook.
- Writing-side profile: SuperWriter is the only framework in the top two on both writing-side rankings, scoring 6.50 on HANNA overall and 6.61 on WriteJudge overall.It ranks first on HANNA and second on WriteJudge.
- Per-mode writing-side analysis: SuperWriter leads the most length-constrained single mode with HANNA 6.83 and WriteJudge 6.94, but its HANNA score degrades to 5.89 in wholebook mode.Explicit paragraph budgets are less effective against whole-book long-range coherence challenges.
- Per-mode writing-side analysis: StoryWriter performs best for event-driven single-mode narrative, while Dome is most useful for whole-book macro-structure; their aggregated means mask these mode-specific strengths.StoryWriter’s WriteJudge score falls to 5.44 in multichap mode.
5 Discussion
The controlled comparison shows that framework strengths vary by task granularity and evaluation metric, while the study’s backward-extraction setting and evaluator biases constrain interpretation.
- Comparison with prior work: About 1 point separates SuperWriter’s A2 faithfulness from naive’s 3.42 to 4.43, while A3 consistency rises from 3.39 to 4.71.These gains are smaller than improvements reported in the original framework papers, partly because this study uses backward extraction and includes selfrefine.
- Granularity-dependent strengths: SuperWriter leads in length-constrained single-chapter evaluation with HANNA 6.83 but declines to 5.89 in whole-book mode.Whole-book generation exceeds 10,000 characters and replaces explicit paragraph budgets with long-range coherence challenges.
- Granularity-dependent strengths: Dome’s Campbell five-act structure best supports whole-book macro-structure, whereas StoryWriter’s event-driven design performs best in single mode but degrades in multichap.Dome scores 5.94 under single-chapter mode, while StoryWriter reaches HANNA 6.22 in single mode and WriteJudge 5.44 in multichap.
- Metric interpretation: Naive ranks first on WriteJudge overall at 6.70 but second on HANNA overall at 6.07, indicating that metric-system compatibility affects framework comparisons.The moderate correlation between HANNA and WriteJudge across frameworks means one metric system cannot fully characterize framework capabilities.
- Baseline interpretation: Naive’s relative position changes with task setting and metric choice, contrasting its anomalous WriteJudge lead here with weaker completion and length results in original studies.CogWriter reports a 0.46 completion rate for naive-class models, while StoryWriter reports 1,078-character average stories from GPT-4o-mini.
- Limitations: The backward long-text-to-outline task creates a ceiling effect on faithfulness and may not trigger framework-specific refinement or constraint mechanisms.Unmitigated evaluator–generator self-preference bias also makes winning claims directional patterns.
6 Conclusion
The study compares seven frameworks across three granularities and finds no stable global winner: framework performance depends on task mode, while outline-side and writing-side rankings are only moderately aligned.
- Benchmark scope: Seven frameworks were compared across single-chapter, multi-chapter, and whole-book modes using seven metric classes on one controlled platform.The benchmark covers 21 framework-granularity cells.
- Main findings: SuperWriter ranks first on HANNA overall at 6.83 in the most length-constrained single-chapter mode, but no stable global ranking holds across modes.Its single-mode result is jointly affected by disabled refinement and self-preference bias, and the claim disappears after excluding critical-deviation frameworks.
- Main findings: The outline-side and writing-side rankings correlate moderately positively, with Spearman ρ ≈0.45–0.58, supporting outline–writing decoupling.Writing-side metrics should not be used to backward-infer outline-side performance.
- Limitations and future work: Compute constraints limited the writing-side sample size, motivating follow-up experiments with larger samples and cross-model evaluators.
7 Ethical Statement
The study reports no ethical concerns because it uses public datasets and does not involve human subjects, personal information, or sensitive content.
- Data and participants: The research uses publicly available datasets, including a WritingPrompts subset and Chinese novels.
- Ethical scope: No human subjects, personally identifiable information, or sensitive content are involved.
- Model use: Deepseek-v4-flash is used as the generation backend and evaluation model, not as a content-creation agent.
8 Code and Data Availability
The authors plan to release the code, data, configurations, and complete test records after follow-up experiments and organization are completed.
- Release plan: Code, data, configuration files, and per-cell evaluation results will be released under the Apache 2.0 license after follow-up work is completed.
- Release scope: Pre-trained model checkpoints will not be redistributed.The experiments use publicly available models, including deepseek-v4-flash.
Appendix A Framework Reproduction Deviation List (Critical / Significant / Minor)
Appendix A catalogs reproduction deviations across the seven frameworks, classifying them by impact while directing readers to the technical report for complete details.
- The appendix classifies known framework deviations as critical, significant, or minor according to their impact on experimental conclusions.
- The main text references three representative critical deviations from the original papers or official code.
- The complete list contains 40+ deviations and is preserved in an accompanying technical report rather than fully reproduced here.
Appendix B Per-mode Per-dimension Mean Tables (B.1–B.4)
Appendix B covers per-mode and per-dimension outline-judge statistics, while related tables compare outline-side and writing-side rankings and report correlation caveats.
- Per-dimension means and standard deviations are reported for single, multichap, and wholebook modes, with overall-quality statistics aggregated by independent sample.
- 12 of 21 cells show dual-list agreement, defined as a ranking difference of at most 1 between HANNA overall and WriteJudge overall.
- Outline–writing Spearman correlations should be interpreted cautiously because self-preference bias is unmitigated and expansion fidelity is uncontrolled as an intermediate variable.
Appendix C Cell-level Per-cell Data
Appendix C preserves cell-level framework data across the three generation modes, including 63 reported rows and nine cells that were not run.
- The accompanying technical report preserves 63 cell-level rows spanning nine cells and seven frameworks.
- Single mode uses samples novel_ch_idx15, novel_en_idx4, and other_idx32.
- Multichap mode uses novel_ch_idx15, novel_ch_idx41, and novel_en_idx55, while wholebook mode uses novel_ch_idx15, novel_ch_idx41, and novel_en_idx4.
- The remaining nine cells were not run.
Appendix D Writing 6-sample Properties
Appendix D documents the six writing-side evaluation samples, their selection criteria, and the use of convenience rather than random sampling.
- The appendix lists detailed properties of the six writing-side evaluation samples for reviewer verification.
- The selection covers all four Chinese/English-by-novel/other cells, with each cell represented by at least one case.
- The two novel_ch cases include one long-context case, idx41, to examine the long-context scenario.
- Sampling is convenience sampling rather than random sampling.