PaperlayerView exact revision on arXiv

Source-linked AI summary

Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

Qiankai Xu, Qiguang Chen, Zixin Su, Wenhao Huang, Yue Gao, Jiaheng Liu, Ge Zhang

arXiv:2608.25826v1cs.CLcs.AIcs.LG

TL;DR

Existing synthetic-data methods reconstruct only local thinking around short web passages, leaving whole-document structure underused. This paper unfolds scientific papers into multi-turn trajectories with a writing request, global plan, and section-level deliberations while keeping source text verbatim, and extends the same construction to SFT and PAW-Bench. The resulting CPT data improves writing broadly while preserving general reasoning and benefiting long-document reading, with gains persisting after dedicated writing SFT.

  • Problem

    Prior reconstruction methods operate on short web passages, recover local thoughts, and leave whole-document structure untouched.

  • Method

    The paper reconstructs each scientific paper’s writing request, global plan, and per-section deliberations around verbatim text, then derives SFT tasks and held-out-paper evaluation tasks backward.

  • Results

    Writing improves broadly across benchmarks, survives further SFT, and is accompanied by preserved general reasoning plus improved paper comprehension and long-context ability.

  • Takeaways & Limitations

    Whole-paper writing-process data supports continued pre-training, while the same reverse construction supplies instruction data and paper-anchored academic-writing evaluation.

  • Takeaways & Limitations

    CPT experiments cover only Qwen2.5-7B with one data mixture, so transfer to larger models and other ratios remains untested.

Abstract

from arXiv · show

A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web passages, recovers only local thoughts, and leaves the structure of whole documents untouched. Scientific papers are written to a clear and largely uniform structure and make a natural substrate for lifting this paradigm to the document level. We present a pipeline that unfolds each paper into a multi-turn generation trajectory in which a teacher model reconstructs the writing process of the whole paper: a writing request, a global plan, and pre-writing deliberation for each section. All section texts and the abstract are kept verbatim from the source paper. We apply the pipeline to quality-filtered arXiv papers and obtain a corpus for continued pre-training (CPT) that is roughly twice the size of the source text. The same reverse construction extends to instruction data and evaluation. Treating real paper text as the answer yields an SFT dataset. Anchoring tasks in held-out papers yields PAW-Bench, an academic-writing benchmark whose tasks carry their own rubrics and checklists. In controlled experiments CPT on our corpus followed by supervised fine-tuning on public datasets improves writing benchmarks broadly while preserving general reasoning and improving long-document reading. The writing gain persists even when every model is fine-tuned on a dedicated writing SFT dataset. Mixing our SFT data into that recipe lifts academic writing further.

1 Introduction

Existing synthetic-data methods reconstruct local thinking around short text, but scientific papers enable document-level recovery of global plans and section-specific writing processes. The paper introduces a reverse-construction pipeline that produces CPT, SFT, and evaluation data while preserving source text, and reports broad writing gains without harming reasoning.

  • Motivation: Existing approaches rewrite or annotate short web passages, leaving document structure and whole-document planning largely untouched.The limitations are mechanical segmentation and flat commentary attached to the original text.
  • Motivation: Scientific papers provide clear rhetorical sections and dense expert writing whose underlying judgments and global plan can be reconstructed.The paper treats introduction, method, experiments, and conclusion as structure that guides decomposition.
  • Pipeline: The pipeline unfolds each paper into a trajectory containing a writing request, global plan, per-section deliberations, verbatim section text, and a final abstract look-back.Across 1.8M arXiv papers, 30B source tokens become 57–60B trajectory tokens per generator, with median documents expanding from 11K to 28–29K tokens.
  • Results: Controlled experiments find writing improvements across benchmarks that survive further SFT, while general reasoning, paper comprehension, and long-context ability are preserved or improved.Plain-text controls do not show the same writing lift, and the experiments isolate the effect of unfolding.
  • Data products: The same reverse construction yields a 200K-sample SFT dataset and PAW-Bench with 2,940 academic-writing tasks, each carrying rubrics and checklists.Real paper text is fixed as the answer for SFT, while held-out papers anchor evaluation questions and grading materials.

2 Related Work

Prior synthetic-data work often alters or invents content, while related methods reconstruct reasoning, instructions, or scientific-text adaptations. This paper combines those ideas around the writing process of whole scientific papers and paper-anchored evaluation.

  • Synthetic data for model training: Synthetic-data methods span web-text rewriting, entity-relation expansion, instruction synthesis, and semi-synthetic editing, with many altering or inventing content.The paper also notes that recursive model-generated training can lose distributional tails.
  • Reconstructing training signal from real text: Related work reconstructs latent thoughts, stretches documents with equal-split reasoning, and derives prompts backward from known good answers.These approaches recover training signals from real text but generally do not unfold whole scientific papers.
  • Scientific literature: adaptation and evaluation: Scientific-text adaptation trains on cleaned prose, whereas PAW-Bench anchors each task in one paper and scores answers with a rubric plus a code-checkable checklist.Existing writing benchmarks either generate criteria per prompt or build tasks from arXiv at multiple abstraction levels.

3 Method

The paper uses one reverse-construction pipeline to derive three products from filtered arXiv papers: whole-paper CPT trajectories, answer-grounded SFT tasks, and held-out-paper evaluation. For CPT, papers are unfolded into structured multi-turn writing processes while source text remains fixed; the resulting trajectories roughly double token volume and preserve a common document structure across generators.

  • Shared construction: One reverse construction fixes real paper text as the final output and derives a writing process, an instruction task, or grading criteria backward.The CPT corpus fixes whole papers, SFT fixes passages, and PAW-Bench fixes held-out papers.
  • Data preparation: The shared preprocessing trunk merges source files, expands macros, retains body text with equations and tables, removes peripheral material, deduplicates papers, and applies quality filters.CPT and SFT use the 2006–January 2026 window, whereas PAW-Bench uses papers from February–June 2026 to maintain disjoint evaluation sources.
  • CPT data: CPT unfolds each paper into a writing request, global plan, per-section deliberations, section text, and abstract deliberation.Section summaries are created first, then used to derive the request, plan, section-level thoughts, and abstract deliberation.
  • CPT data: 57–60B trajectory tokens are produced per generator from approximately 30B source tokens across 1.8M papers, roughly a 2× expansion.The median source-body length is 11.2K tokens, while generator medians are 29.3K, 29.4K, and 28.3K.
  • CPT data: The three generators preserve similar document-level structure but produce distinct local deliberation styles, with 27B trajectories shorter by about 1K tokens.The lexical comparison uses 10,000 shared papers and counts synthesized words and phrases per 10K reasoning words.

4 Experiments

Controlled experiments compare unfolded paper trajectories with plain-paper and general-text CPT under fixed SFT recipes. Trajectory CPT broadly improves writing, preserves reasoning, and benefits paper comprehension and long-context performance, while corpus difficulty and generator scale reveal important training-dynamics effects.

  • Writing Results: Under the same DeepWriting recipe, trajectory-CPT models remain 1.6–2.4 Average points above the no-CPT baseline; adding the paper SFT mixture raises PAW-Bench to 64.13 and the table’s top score to 54.38.Every configuration including the authors’ CPT finishes above the best configuration without it, which scores 52.35.
  • Generator Analysis: The writing gains are attributed to the unfolding construction rather than generator strength: generator-trained models differ by only 0.86 Average points, versus up to 2.22 over Plain-Paper CPT.The smallest 4B generator’s data achieves the highest Average in this comparison.
  • Effect on Reasoning: Trajectory CPT does not hurt reasoning: replacing 30B FineWeb-Edu tokens restores post-SFT Average to 45.66–46.82, near direct SFT at 46.32.FineWeb-Edu CPT alone lowers the Average from 46.32 to 44.30, whereas the authors’ mixture mitigates this forgetting.
  • Paper Comprehension and Long Context: Unfolding improves paper reading and long-context performance: the 27B model gains over 3 Qasper points, while 9B and 27B beat the baseline in both LongBench v2 brackets.The gains are 2.6 and 2.2 points for 9B, and 6.0 and 3.3 for 27B; about a third of trajectories extend into 32–64K tokens.
  • Training Dynamics: The 4B corpus is hardest to fit but yields the best writing Average at 54.34 and reasoning Average at 46.82, while the 27B corpus has lower loss and more repetitive content.Within this corpus family, the authors caution that CPT loss should not determine the generator for writing data; the comparison is informative only within one corpus family.

5 Conclusion and Limitations

The paper concludes that unfolding scientific papers into reconstructed writing processes produces broad writing gains while preserving reasoning and benefiting paper comprehension and long-context ability. It also identifies scope, measurement, evaluation-size, and reconstruction-fidelity limitations.

  • Conclusion: The pipeline unfolds whole scientific papers into multi-turn trajectories containing a writing request, global plan, and per-section deliberations around verbatim paper text.Applied to 1.8M arXiv papers, it turns 30B tokens of paper text into a 60B-token CPT corpus.
  • Conclusion: The same reverse construction yields CPT trajectories, a 200K-sample SFT dataset, and the 2,940-task PAW-Bench.Each product holds real paper text fixed while deriving the surrounding process, task, or grading materials.
  • Results: On Qwen2.5-7B, trajectory CPT improves writing broadly, with gains persisting under two SFT recipes while general reasoning remains intact.The same papers as plain text provide neither the writing lift nor the long-context gain.
  • Limitations: CPT experiments cover only Qwen2.5-7B and one mixture, leaving transfer to larger models and other data ratios untested.The limitation is attributed to the expense of CPT experiments.
  • Limitations: Writing metrics rely on an LLM judge, and a gap from human preference cannot be ruled out without large-scale human grading.Automated checks, sampled human inspection, and code-checkable checklist items mitigate this concern.
  • Limitations: The paper-comprehension and long-context suites are small, so their gains are treated as corroborating evidence alongside writing and reasoning results.This limits how strongly those secondary gains can be interpreted.
  • Limitations: The trajectories are post-hoc reconstructions and need not match the authors’ actual deliberation.The method reconstructs a plausible writing process around the finished document.
  • Future direction: The construction is intended to extend beyond papers to any long text with clear structure.This is presented as the next direction for the work.

B.2 Deterministic Structural Filters

The appendix describes deterministic filtering and synthesis steps that prepare structured scientific papers and reconstruct trajectories in paper order. The process preserves source sections and abstracts verbatim while generating the surrounding request, plan, and deliberations.

  • Structural filters: The shared hard-filtering trunk removes malformed, duplicate, unsupported, out-of-range, and structurally defective papers before synthesis.It also checks section lengths, repetition, residual commands, environment balance, and layout-command density.
  • Structural filters: The SFT source applies explicit thresholds for abstract and body length, section count, introduction presence, character ratios, command density, repetition, and retained languages.Residual acknowledgments, references, image or citation commands, control characters, HTML fragments, and unclosed environments cause rejection.
  • Structural filters: About 356K papers survive an LLM quality threshold of 8 after stricter SFT filtering and deterministic balancing across coarse arXiv subject groups.The quality score covers technical soundness, novelty, clarity, organization, evidence, and writing quality.
  • Trajectory definition: The raw trajectory is q, p, c1, s1, c2, s2, …, cn, sn, ca, a, with generated context interleaved around verbatim sections and abstract.q is the writing request, p the whole-paper plan, ci section deliberation, and ca retrospective abstract deliberation.
  • Pipeline stages: The pipeline runs n summaries, one request, one plan, n section deliberations, and one abstract deliberation, totaling 2n + 3 model calls.Section deliberations use the current text plus surrounding section summaries, while abstract deliberation uses all summaries and the real abstract.
  • Pipeline stages: Prompt contracts require recoverable claims, ordered coverage of retained sections, prospective author reasoning, and discussion of rhetorical purpose without quoting sections.Structured labels and summaries preserve section-level organization during trajectory construction.

C.4 Generator Settings and Production Scale

Three Qwen3.5 generators produce large-scale paper-writing trajectories with similar document-level structure and different local deliberation styles. The resulting corpora are roughly twice as large as their source papers.

  • Generator settings: Three generators—Qwen3.5-4B, Qwen3.5-9B, and Qwen3.5-27B—produce the corpus variants.Generation uses the same broad sampling configuration across model scales.
  • Production scale: All three generators complete trajectories for over 99.5% of input papers, yielding 57–60B training tokens per generator from approximately 30B source tokens.The paired original papers contain approximately 30B tokens before unfolding.
  • Production scale: Median source-body length is 11.2K tokens, while median trajectory lengths are 29.3K, 29.4K, and 28.3K for the 4B, 9B, and 27B generators.The three generators produce similar document-level structure and distinct local deliberation styles.

C.5 Training Data Preparation

Training preparation interleaves generated trajectory context with the paper’s original sections and abstract, while masking the reconstructed request from loss. Documents are tokenized, truncated, shuffled, and packed for training.

  • Training sequence: The sequence begins with a reconstructed writing request, followed by the assistant-side plan, interleaved deliberations and original sections, and abstract deliberation followed by the original abstract.The writing request is masked from loss.
  • Training sequence: Documents are tokenized with the Qwen tokenizer, right-truncated to 65,536 tokens, shuffled, and greedily packed.

D SFT Data: Deriving Tasks Backwards from Paper Text

The SFT pipeline constructs source-grounded deliverables first, then derives the requests and deliberations that plausibly produce them. It samples diverse academic-writing, long-form, and paper-grounded tasks, and evaluates them with weighted rubrics and checklists.

  • Task construction: Source-grounded answers preserve the factual core while applying minimal cleaning, reorganization, or audience adaptation before the request and deliberation are inferred.The inferred request includes the length and format constraints realized by the answer.
  • Task construction: Three task families cover rewriting, paper-grounded question answering or extraction, and reorganization into functional artifacts such as decision memos or reading maps.Sampling prioritizes academic writing, followed by long-form writing and paper-grounded research tasks.
  • Evaluation design: PAW-Bench tasks receive independent weighted rubrics and pass/fail checklists, with deterministic checks used for many concrete format requirements.Rubric scores are normalized to 0–100, while checklist scores are weighted fractions passed.
  • Evaluation design: 14,848 rubric criteria and 15,105 checklist items comprise PAW-Bench, including 8,896 deterministic checklist items and 6,209 judge-scored items.The deterministic subset represents 58.9% of checklist items.
  • Quality control: Repeated automated audits, targeted corrections, and manual spot checks classified every finalized task as answerable and scoreable.The authors report that remaining minor issues do not affect intended task behavior.

E Abbreviated Prompts, Algorithms, and Examples

The appendix specifies prompt templates, production algorithms, quality gates, and examples for unfolding papers and reverse-constructing instruction data. It also illustrates how the resulting tasks encode structured academic-writing requirements and evaluation criteria.

  • E Abbreviated Prompts, Algorithms, and Examples: The appendix omits repeated formatting reminders and long paper contexts while retaining instructions that alter generated targets or acceptance decisions.This defines the scope of the abbreviated prompt section.
  • E.1 CPT Prompt Templates: CPT prompts infer an author-facing request, global plan, section-level deliberations, and abstract deliberation from titles, summaries, surrounding sections, and true text.The deliberations use a first-person author perspective and explain each section’s role in the paper’s argument.
  • E.2 SFT Prompt Templates: SFT reverse construction fixes a source-grounded answer first, then derives the natural request and a forward-looking deliberation using visible request and material information.Answer construction preserves source-supported claims, numbers, comparisons, citations, and conclusions while satisfying genre and format constraints.
  • E.2 SFT Prompt Templates: A semantic judge evaluates deliberation credibility, contract satisfaction, answer quality, factual faithfulness, audience fit, request naturalness, and deliverable usefulness.The judge returns a 1–10 quality score and explicit problems.
  • E.3 PAW-Bench Task-Generation Prompt: PAW-Bench task generation supplies task definitions, material regimes, audiences, length bands, and format complexity, requiring feasible academic-writing requests with explicit output contracts.Prompt-only tasks place scenario-specific facts directly in the request.
  • E.4 Production Pseudocode: The CPT algorithm cleans ordered paper sections, generates summaries and whole-paper planning, adds prospective section deliberations before verbatim sections, and appends abstract deliberation before the original abstract.Sequences are right-truncated and packed into 64K-token training examples after tokenization and shuffling.
  • E.4 Production Pseudocode: SFT samples three task types with varied material, audience, length, and format settings, then applies deterministic validation and retains semantic-judge outputs scoring at least 9 with no listed problems.The process supports stable resumption by flushing terminal results by paper and task identifier.
  • E.5 Abridged Examples: The examples instantiate paper-level requests, reverse deliberations, and tightly specified PAW-Bench outputs, while Table 6 records CPT hyperparameters.Examples include robust Bayesian modeling, claim checking, and a TL;DR-plus-contributions task with weighted rubric dimensions and checklist tests.

F Training Details

Training runs use fixed seeds, controlled token budgets, and standardized comparison recipes. The main CPT experiments compare an unfolded corpus variant with FineWeb-Edu and cleaned-paper controls before writing-oriented SFT.

  • Training setup: All project training runs use seed 42.
  • Training setup: CPT runs initialize from Qwen2.5-7B and use 30B synthetic-corpus tokens plus 20B FineWeb-Edu tokens, totaling 50B tokens.Controls use the same total budget with plain cleaned papers plus FineWeb-Edu or pure FineWeb-Edu.
  • Training setup: The five CPT runs’ loss curves, final 100-step means, and pairings between final losses and downstream results are analyzed in Section 4.5.
  • Writing SFT recipes: Every project-trained row follows the official DeepWriting recipe for three epochs at a constant learning rate of 2×10^-5.The official mixture combines 15K OpenThoughts examples with 20K DeepWriting examples; Our-Mixed SFT replaces 10K DeepWriting examples with 10K writing-oriented corpus examples.
  • Writing SFT recipes: The corpus portion of the mixed recipe prioritizes writing tasks while varying material, length, and output format, and the DeepWriting portion remains category-balanced with long-form examples.

F.4 Writing SFT Optimization Curves

Training loss under a shared DeepWriting recipe does not track downstream writing performance closely, whereas the Our-Data-27B initialization lowers loss and improves the downstream Average under Our-Mixed SFT. The authors explicitly treat these relationships as correlational.

  • DeepWriting SFT: Figure 8 compares six runs sharing the DeepWriting mixture and recipe over a common 1,134-step axis using 50-step moving averages.The lower panel enlarges steps 800–1,134, where curves are difficult to distinguish.
  • DeepWriting SFT: 0.854 is the final 50-step loss for both DeepWriting SFT and Our-Data-4B CPT, yet Our-Data-4B CPT performs substantially better on downstream writing Average.The authors distinguish in-distribution SFT fit from transferable downstream writing results.
  • DeepWriting SFT: FineWeb-Edu CPT has the highest loss through most training and the highest final mean, while also producing the lowest downstream Average among DeepWriting-tuned rows.Plain-Paper CPT lies between this control and the trajectory-CPT runs.
  • Interpretation: The reconstructed writing process provides the stronger transfer signal in downstream results, but the authors state that these observations are correlational rather than causal.
  • Our-Mixed SFT: 0.803 versus 0.816: Our-Data-27B CPT ends below direct Our-Mixed SFT, while downstream Average rises from 52.35 to 54.38.The direction of both loss and writing Average is described as stronger alignment with the mixed paper-writing objective.
  • Our-Mixed SFT: Each figure has its own absolute loss scale because the two SFT mixtures contain different examples and target distributions.
  • Controlled comparisons: CPT initialization is isolated in reasoning and reading comparisons because all checkpoints within each table receive the same downstream SFT data and recipe.General reasoning uses OpenThoughts SFT, while paper QA and long-context reading use SmolTalk2 SFT with thinking disabled.

G Writing Benchmark Protocols

The section defines four writing-benchmark protocols and explains how their scores are reported and combined. Each suite retains its own task-specific evaluation structure, while the final judge model is shared.

  • WritingBench: WritingBench reports an overall score and a separate Academic & Engineering subset using direct writing requests and five-criterion judging.
  • PAW-Bench: PAW-Bench reports both its 0–100 rubric score and 0–1 checklist pass rate.
  • HelloBench: HelloBench reports the Open-Ended QA and Heuristic Text Generation subsets, selected for their proximity to the paper’s writing setting.
  • LongBench-Write: LongBench-Write preserves original prompts and explicit length targets to probe sustained generation to a requested length.
  • Cross-benchmark scoring: GPT-5.5 with medium reasoning effort serves as the final judge across all four suites, while benchmark-specific rubrics remain distinct.
  • Cross-benchmark scoring: The equal-weight Average combines the HelloBench aggregate, WritingBench overall, PAW-Bench rubric, and LongBench-Write after averaging HelloBench’s two subsets.
Loading 2608.25826v1…