PaperlayerView exact revision on arXiv

Source-linked AI summary

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

arXiv:2408.07055v1cs.CLcs.LG

TL;DR

Long-context LLMs process very long inputs but generally cannot generate beyond about 2,000 words, and the paper attributes this limitation to scarce long-output examples in SFT data. It introduces AgentWrite to construct extended training data, then uses LongWriter-6k to scale existing models beyond 10,000 words while maintaining quality and achieving strong benchmark performance.

  • Problem

    Long-context LLMs can process inputs exceeding 100,000 tokens but struggle to generate outputs beyond approximately 2,000 words because existing SFT datasets rarely contain long-output examples.

  • Method

    AgentWrite decomposes long-writing tasks into planned paragraph-level subtasks, generates them sequentially, and supports construction of LongWriter-6k long-output SFT data.

  • Results

    LongWriter-6k scales existing models to well-structured outputs exceeding 10,000 words, while the 9B model achieves state-of-the-art LongBench-Write performance and ties GPT-4o in human preference comparisons after DPO.

  • Takeaways & Limitations

    The results indicate that extending SFT output lengths can unlock substantially larger output windows in existing long-context LLMs.

  • Takeaways & Limitations

    LongWriter models currently reach only 10k–20k words, and longer outputs require more SFT data with long outputs.

Abstract

from arXiv · show

Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability. Our code & models are at: https://github.com/THUDM/LongWriter.

1 INTRODUCTION

Long-context LLMs can process very long inputs but generally fail to generate beyond about 2,000 words because their SFT data lacks sufficiently long outputs. The paper proposes AgentWrite and LongWriter-6k to construct and use longer-output training data, scaling generation beyond 10,000 words.

  • Long-context LLMs consistently fail to generate outputs beyond approximately 2,000 words despite processing histories exceeding 100,000 tokens.
  • SFT datasets primarily constrain models’ maximum generation length because their output examples rarely exceed 2,000 words.Datasets distilled from existing models can inherit this limitation.
  • AgentWrite decomposes long-writing tasks into planned paragraph-level subtasks and generates them sequentially to produce coherent outputs up to 20,000 words.
  • LongWriter-6k adds 6,000 long-output SFT examples, enabling existing models to generate well-structured outputs exceeding 10,000 words.
  • LongBench-Write evaluates writing instructions across output-length ranges, while DPO further improves long-instruction following and content quality.
  • The contributions identify SFT output length as the main constraint, introduce AgentWrite, and scale existing models to 10,000+ words without compromising output quality.

2 FINDING THE CAUSE OF THE BOUNDED GENERATION LENGTH LIMIT

The paper tests whether SFT data limits generation length by probing models with LongWrite-Ruler and controlling the maximum output length of GLM-4-9B’s training data. Results show that model output length rises with the SFT length ceiling, supporting insufficient long-output data as the limiting factor.

  • LongWrite-Ruler probes maximum output length using 48 Chinese and English prompts with requested lengths from 1,000 to 30,000 words.
  • The evaluation measures four open-source and four proprietary models while checking that API or inference token limits do not truncate outputs.
  • Figure 1 reports an approximately 2,000-word maximum output limitation across all tested models.
  • Figure 2 compares GLM-4-9B models trained on SFT datasets with different maximum output lengths.
  • Controlled experiments filter GLM-4-9B’s SFT data at 500, 1,000, and 2,000 words to test whether the training-output ceiling determines generation length.

3 AGENTWRITE: AUTOMATIC DATA CONSTRUCTION

AgentWrite automatically constructs long-output training data by decomposing complex writing tasks into paragraph-level subtasks and concatenating their sequentially generated outputs.

  • AgentWrite is a divide-and-conquer agent pipeline that breaks long writing tasks into multiple paragraph-level subtasks.
  • The model executes subtasks sequentially and concatenates their outputs to form the final long response.
  • The planning stage specifies each paragraph’s structure, content, and target word count before generation begins.
  • A 30,000-word Roman Empire article is used as an example of the long-writing task that AgentWrite plans and decomposes.

STEP I: Plan

Step I creates a detailed writing plan for the requested long-form article, assigning content and word-count requirements to individual paragraphs.

  • Paragraph 1 introduces the origins of the Roman Empire and has a 700-word requirement.
  • Paragraph 15 summarizes the Roman Empire’s history and has a 500-word requirement.
  • Paragraph 2 describes the founding of the Roman Empire and has an 800-word requirement.

STEP II: Write

AgentWrite uses sequential, plan-guided paragraph generation to extend off-the-shelf LLM outputs while evaluating length adherence and writing quality on long-form benchmarks.

  • STEP II: Write: AgentWrite sequentially generates each planned paragraph using previously written sections to preserve coherence.The pipeline avoids parallel generation because validation found sequential writing produced superior overall coherence and quality.
  • STEP II: Write: AgentWrite is evaluated on LongWrite-Ruler for generation length and LongBench-Write for instruction adherence and writing quality.LongBench-Write includes varied prompts with explicit length requirements, while evaluation uses separate length and quality measures.
  • STEP II: Write: The length score reaches 100 when actual output matches the requirement and decays to zero beyond four times or below one-third of the required length.The scoring function penalizes outputs that are too short more strongly than outputs that are too long.
  • STEP II: Write: GPT-4o’s maximum output length increases from 2k to approximately 20k words with AgentWrite.On LongBench-Write, AgentWrite improves length following especially for outputs in the [4k, 20k) range without compromising overall output quality.
  • STEP II: Write: AgentWrite improves Breadth and Depth by 5% but slightly reduces Coherence and Clarity by 2%.The outputs can also contain minor repetitions, while the parallel variant reduces Coherence by 6%.

4 LONGWRITER: TEACHING MODELS TO GENERATE ULTRA-LONG OUTPUT

LongWriter teaches models to generate ultra-long outputs by combining long-output SFT data with alignment and evaluation methods. LongWriter-6k enables outputs above 10,000 words, while DPO further improves quality and length adherence.

  • 4.1 DATA CONSTRUCTION: LongWriter-6k supplements scarce long-output SFT data, with outputs relatively evenly distributed between 2k and 10k words.The training set combines LongWriter-6k with 180k general SFT examples.
  • 4.3.1 MAIN RESULTS: LongWriter models consistently produce longer responses than previous models for prompts requiring more than 2,000 words.Previous models generally perform poorly in the [2k, 4k) range and fail to reach targets in the [4k, 20k) range.
  • 4.3.1 MAIN RESULTS: DPO improves LongWriter-9B’s length and quality scores by +4% and +3%, respectively, across all required-length ranges.Human annotators preferred the DPO-trained model over LongWriter-9B in 58% of cases.
  • 4.3.1 MAIN RESULTS: LongWriter models reach maximum generation lengths between 10k and 20k words, but longer outputs require additional long-output SFT data.Fewer than 100 training examples have output lengths of 20k words or greater.
  • 4.3.2 ABLATION STUDY: Adding LongWriter-6k improves output quality by +5%, including an 18% absolute gain in Breadth and Depth, without biasing responses toward greater length.The ablation compares LongWriter-9B with a model trained without LongWriter-6k.
  • 4.3.2 ABLATION STUDY: Instruction-backtranslated long-output data fails to improve length scores beyond general SFT and compromises generation quality.The authors attribute this result to low-quality selected long texts and related data-construction issues.

5 RELATED WORK

Related work addresses long-context memory and instruction alignment. These lines of research motivate LongWriter’s focus on teaching models to follow length and richness constraints.

  • Long context LLM: Long-context research expands LLM memory so models can process longer inputs.Methods include zero-shot context extension and fine-tuning on longer sequences.
  • Aligning LLM to follow constraints in instruction: LongWriter connects long-context modeling with alignment for longer, richer outputs.Its methodology relies on aligning LLMs to follow user instructions while increasing output length.
  • Aligning LLM to follow constraints in instruction: Alignment research uses supervised fine-tuning and reinforcement learning from human feedback to improve instruction following.Prior studies examine how alignment can teach models to prioritize instructions and follow length constraints.

6 CONCLUSION

The paper attributes current output limits to insufficient long-output alignment data and addresses them with AgentWrite and LongWriter-6k. It scales output beyond 10,000 words while identifying data quality, output length, and inference efficiency as future challenges.

  • 6 CONCLUSION: The paper identifies a 2,000-word generation limit in current LLMs and proposes adding long-output data during alignment.The conclusion frames this as increasing the models’ output window size.
  • 6 CONCLUSION: AgentWrite uses off-the-shelf LLMs to construct extended, coherent outputs for training data.The pipeline is described as agent-based and intended to generate long-output data automatically.
  • 6 CONCLUSION: LongWriter-6k scales current LLM output windows to 10,000+ words.The conclusion also reports extensive ablation studies supporting the approach.
  • 6 CONCLUSION: Future work includes constructing longer and higher-quality data and improving inference efficiency for long outputs.The paper notes that longer outputs create inference-efficiency challenges.

A MODEL CARDS

The model cards and evaluated-model details are reported in Table 5.

  • A MODEL CARDS: The paper lists details of the evaluated models in Table 5.
  • A MODEL CARDS: Table 5 is used as the location for model information accompanying the evaluation.
  • A MODEL CARDS: Table 5 provides the model cards for the evaluated models.

B LONGWRITE-RULER TEST

LongWrite-Ruler evaluates long-form generation across eight seed prompts, six target lengths, and 48 test prompts, using structured quality assessment and length-based request selection.

  • 8 seed prompts cover novels, historical articles, climate-economy papers, and China travel guides in English and Chinese.
  • 6 target lengths from 1000 to 30000 words produce 48 total test prompts.
  • Quality assessment scores relevance, accuracy, coherence, clarity, breadth and depth, and reading experience on a 1–5 scale.
  • A separate selector identifies user requests requiring articles longer than 2,000 words in English or 2,000 characters in Chinese.

D MORE EVALUATION RESULTS

The additional evaluation section reports AgentWrite strategy quality, English-sample results, and generation-length statistics on LongBench-Write.

  • Table 6 reports quality assessments of AgentWrite strategies on LongBench-Write.
  • Table 7 reports evaluation results for English samples in LongBench-Write.
  • Table 8 reports generation-length statistics in words for LongBench-Write.
Loading 2408.07055v1…