Source-linked AI summary
PEER: A Collaborative Language Model
Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, Sebastian Riedel
TL;DR
Existing language models are poorly suited to collaborative writing because they generate final text in one pass rather than revising, following plans, or explaining actions. PEER imitates the full plan–edit–explain–repeat process and trains infilling models for self-training. It performs strongly across editing tasks and domains, while evaluation remains limited in task, domain, language, and interaction-session coverage.
Problem
Existing language models cannot readily update text, are difficult to control, and lack reliable planning and explanation abilities needed for collaborative writing.
Method
PEER models collaborative writing as repeated planning, editing, explaining, and repetition, using infilling models and self-training to extend Wikipedia-based data.
Results
PEER clearly outperforms various baselines on text-editing tasks and performs edits across domains while improving instruction following and citation and quotation abilities.
Takeaways & Limitations
PEER provides a collaborative writing assistant that can follow plans and perform syntactic, stylistic, and meaning-changing edits with relevant documents.
Takeaways & Limitations
Evaluation covers only a small subset of editing tasks, few domains, and English, while collaborative potential is tested only through limited individual examples rather than full human–AI sessions.
Abstract
from arXiv · showhide
Textual content is often the output of a collaborative writing process: We start with an initial draft, ask for suggestions, and repeatedly make changes. Agnostic of this process, today's language models are trained to generate only the final result. As a consequence, they lack several abilities crucial for collaborative writing: They are unable to update existing texts, difficult to control and incapable of verbally planning or explaining their actions. To address these shortcomings, we introduce PEER, a collaborative language model that is trained to imitate the entire writing process itself: PEER can write drafts, add suggestions, propose edits and provide explanations for its actions. Crucially, we train multiple instances of PEER able to infill various parts of the writing process, enabling the use of self-training techniques for increasing the quality, amount and diversity of training data. This unlocks PEER's full potential by making it applicable in domains for which no edit histories are available and improving its ability to follow instructions, to write useful comments, and to explain its actions. We show that PEER achieves strong performance across various domains and editing tasks.
1 Introduction
PEER models collaborative writing as an iterative process of planning, editing, explaining, and repeating, addressing language models’ limited control, revision, and explanation abilities. It uses infilling and self-training to extend Wikipedia-derived training beyond a single domain, and outperforms baselines on text-editing tasks.
- Motivation: Existing language models generate outputs in one left-to-right pass, making them unable to retroactively revise text, difficult to control, and unable to explain intentions reliably.These limitations hinder collaboration on coherent, factual writing.
- Core contribution: PEER is a collaborative language model that plans and performs edits, explains them with text and sources, and repeats the process until no further updates are needed.The model is trained on edit histories to cover the writing process rather than only generating final text.
- Training approach: PEER is trained primarily on Wikipedia edit histories and infills missing parts of the writing process to support application beyond Wikipedia.The infilling approach also leverages self-training to improve data quality, amount, and diversity.
- Results: For different text-editing tasks, PEER clearly outperforms various baselines, while the authors analyze factors contributing to its performance.The paper also releases PEER models, training data, and code for further research.
2 Related Work
Related work spans text editing, instruction following and planning, collaborative writing, and self-training. PEER differs by combining iterative edits with plans, explanations, references, and broader domain applicability.
- Text Editing: Prior text-editing models address Wikipedia or specific tasks, but some restrict edits to single sentences or omit plans, explanations, and reference documents.These tasks include updating information, correcting grammar, and improving citations.
- Instruction Tuning and Planning: Instruction-tuning and planning research teaches models to follow human-written instructions or separates planning from generation in summarization, data-to-text, and story writing.PEER applies related ideas to iterative textual editing.
- Collaborative Writing: Collaborative-writing systems improve fluency, coherence, style, or continuations, but the cited approaches do not provide PEER’s natural-language control over edits.The comparison is with systems focused on syntactic edits or suggestions for continuations.
- Self-Training: PEER uses self-training to infill missing editing-process data, paralleling bootstrapping methods in word-sense disambiguation, machine translation, sequence generation, and few-shot learning.Related work also turns documents into dialogue sequences.
3 Plan, Edit, Explain, Repeat
PEER treats writing as repeated plan–edit–explain cycles that transform a text using supplied background documents. Wikipedia provides scalable edit histories, but infilling models are needed to address missing annotations and domain-specific limitations.
- 3 Plan, Edit, Explain, Repeat: Each PEER iteration formulates a verbalized plan, transforms x_t into x_t+1 through an edit, and optionally explains the edit using supplied documents D_t.The process can begin from an existing text or an empty sequence.
- 3 Plan, Edit, Explain, Repeat: The plan–edit–explain cycle repeats until the text stops changing or a manually defined halting criterion is reached.This iterative decomposition is intended to make writing more interpretable and controllable than standard left-to-right generation.
- 3.1 Overcoming Data Scarcity: Training this process is difficult because web-scale edit histories are scarce and available edits usually lack plans, documents, or explanations.This data scarcity limits direct scaling of the full writing-process objective.
- 3.1 Overcoming Data Scarcity: Wikipedia supplies large-scale edit histories, comments, diverse topics, and citations, but its noisy comments, missing citations, and domain-specific text create important limitations.Retrieval may also fail to find supporting background information for many edits.
- 3.1 Overcoming Data Scarcity: PEER trains multiple models to infill different parts of the editing process, then uses them to generate synthetic data for missing training components.Figure 2 schematizes this multi-model process.
4 Infilling Edit Histories with PEER
PEER addresses missing edit histories by training multiple model instances to infill plans, edits, explanations, and supporting documents, enabling synthetic collaborative-writing data.
- 4 Infilling Edit Histories with PEER: Multiple PEER instances infill missing parts of the writing process to generate synthetic data that completes or replaces training examples.The approach uses PEER-Edit, PEER-Undo, PEER-Explain, and PEER-Document for different missing components.
- Decomposing Texts into Edits: PEER-Undo decomposes plain texts into synthetic backward edits, allowing PEER-Edit to train on domains without recorded edit histories.The model iteratively predicts prior text states until reaching an empty sequence, then learns the reverse editing process.
- Generating Plans: PEER-Explain generates candidate explanations that become plans after selecting the candidate that makes the observed edit most likely.This addresses missing or low-quality comments while using edit likelihood to choose among generated candidates.
- Generating Documents: PEER-Document generates synthetic background documents only for PEER-Edit training, never as synthetic documents during inference.Sampled documents are ranked by how well PEER-Edit predicts the actual edit.
- Controlling Model Outputs: Control tokens regulate explanation length, instruction-like form, word overlap, document substrings, and undo-step size without imposing predefined editing controls on PEER-Edit.For PEER-Undo, difference in word counts helps ensure termination; PEER-Edit remains uncontrolled because user editing tasks are not specified in advance.
5 Training Data
PEER training data derives from Wikipedia edit histories, then filters, formats, and augments them with retrieved documents to represent edits, comments, and evidence.
- Data Sources: Wikipedia edit histories provide source-target texts, comments as proxies for plans and explanations, and identifiers for cited documents.The February 2022 English Wikipedia dump is used as the main natural source of commented edits.
- Filtering: Filtering removes reverted or bot-made edits, edits affecting more than two paragraphs, evaluation-page edits, and unresolved document identifiers.These heuristics target vandalism, automated changes, excessive edit scope, contamination, and unavailable sources.
- Retrieving Documents: Each edit receives a document set from citations in the edited paragraphs, augmented with Sphere documents when fewer than the required number of citations is available.The number of documents is controlled by a hyperparameter k.
- Formatting: Formatting removes unaffected paragraphs and Wikipedia-specific syntax while retaining titles, emphasis, lists, links, and citations needed for basic formatting and evidence attribution.Citations remain so PEER can cite and quote documents supporting generated text.
- Retrieving Documents: Documents are linearized with identifiers, titles, content, and domains, enabling PEER to reference and quote specific sources.When citations are insufficient, retrieved documents from the Sphere corpus augment the document set.
6 Experiments
Experiments test whether PEER can follow plans, edit across domains, use references, cite and quote evidence, and benefit from iterative generation. Across these tasks, PEER generally outperforms baselines, while performance depends on synthetic training, decoding strategy, and interaction mode.
- 6.2 Natural Edits: Plans and reference documents provide complementary information: removing both makes PEER substantially worse than removing either one alone.This experiment evaluates EM, EM-Diff, and SARI on the Wikipedia subset of Natural Edits.
- 6.3 Downstream Tasks: PEER (SP) achieves the strongest average zero-shot performance, with 42.0 SARI versus 24.1 for the strongest T5-based baseline across tasks.It also outperforms InstructGPT by 4.1 points on average, despite InstructGPT’s much larger size and human-annotated fine-tuning.
- 6.3 Downstream Tasks: Iterative decoding can drastically improve some tasks, such as ASSET, but substantially worsen others, such as WNC, making automatic strategy selection important.The compared alternatives use three greedy iterations or top-p sampling with p = 0.9 for three iterations.
- 6.4 Citations and Quotes: PEER places the correct citation correctly in 74.1% of cases without the position, and PEER (SP) reaches 88.9 versus 91.1 for supervised Side when given it.PEER also supports citing and quoting reference passages, which aids explainability and fact-checking.
- 6.5 Iterative Editing for Text Generation: Autonomous PEER trails WikiLM on Rouge, whereas manual PEER surpasses it by about one Rouge-1 point and all PEER variants achieve better QuestEval scores.The comparison evaluates autonomous, manual, and collaborative generation modes on Wikipedia introductions.
7 Analysis
PEER’s analysis illustrates how its editing-process components support self-correction, domain adaptation, synthetic-data generation, and controllable explanations. Across qualitative examples, the model can compose from documents, revise errors, and generate useful plans and references, though synthetic references may contain repetitions.
- Collaborative editing: PEER can compose information from multiple documents, correct a false claim through a model-generated plan, and modify document layout.The model can identify and remove an unsourced statement in a later editing step.
- Collaborative editing: After synthetic domain adaptation, PEER can follow StackExchange-style structure and propose plans that correct errors in earlier edits.This demonstrates transfer beyond Wikipedia to a domain with different document structure and style.
- Synthetic data: PEER-Undo decomposes an unseen scientific sentence into meaningful edits, while PEER-Explain reconstructs plans and handles differently formatted references.The models can generate a sequence ending in an empty text and rewrite plans in the opposite direction.
- Synthetic data: Controlling plan length and word overlap changes PEER-Explain’s specificity, with overlapping plans identifying the added reference and helping PEER-Edit most.Short controls produce brief actions, whereas longer controls capture multiple changes.
- Synthetic data: PEER-Document can generate a plausible reference supporting an edit, but sampled documents often contain substantial repetition.The generated document backs up the inserted claim while repeating phrases such as the number of Natural Edits.
8 Limitations
The paper identifies limitations in PEER’s approach and evaluation, including target-dependent retrieval, unsupported claims, inefficient paragraph-level editing, and narrow testing. These constraints limit direct applicability to real-world, long-document, and broad collaborative-writing settings.
- 8.2 Evaluation: Domain-adapted PEER clearly outperforms regular PEER on all Natural Edits subsets, but this result does not remove the broader approach and evaluation limitations.The reported comparison motivates the method’s promise while the surrounding limitations constrain its interpretation.
- 8.1 Approach: PEER assumes documents are available from a retrieval process that uses the target text, which is not possible in real-world applications.The authors suggest retrieval without target access or joint retriever-model training as future work.
- 8.1 Approach: Manual inspection found many false or unsupported claims despite reference documents and comparably high QuestEval scores.Citations and quotations improve checking but can also make unsupported generations appear more authoritative.
- 8.1 Approach: Rewriting entire paragraphs makes PEER less efficient than recent editing approaches and prevents processing entire documents.The authors consider full-document handling important for many real-world applications.
- 8.2 Evaluation: The evaluation covers only a small subset of editing tasks, a few domains, and English, while collaborative interaction is tested mainly through isolated examples.Broader sessions require better retrieval, evaluation of jointly authored texts, and greater efficiency.
9 Conclusion
The conclusion presents PEER as a writing assistant that performs diverse edits by following plans. It attributes broader-domain instruction following and improved citation and quotation abilities to training variants that infill parts of the editing process.
- 9 Conclusion: PEER follows plans to perform syntactic, stylistic, and meaning-changing edits, including removing, updating, or adding information.The conclusion frames these capabilities as writing-assistant behavior.
- 9 Conclusion: Training PEER variants to infill parts of the editing process enables edits in different domains, better instruction following, and improved citation and quotation.These are the principal capabilities highlighted by the authors’ experiments.
A Training Data
The training-data pipeline filters and restructures Wikipedia revisions, retrieves reference documents, applies input perturbations, and defines controls for generated outputs. These procedures support PEER’s edit, explanation, undo, and document-generation components.
- A Training Data: Reference documents are collected from cited identifiers and retrieved from CCNet or Sphere using target-text relevance and the Side reranker.Documents are chunked into 100-word segments, with additional retrieval when cited-document coverage is insufficient.
- A Training Data: Training examples are perturbed by removing titles, non-edited sentences, and some uncited documents to expose PEER to varied contexts.The pipeline also reserves fixed token budgets for document domains, titles, and contents.
- A Training Data: The resulting linearized input and target sequences define the format used for PEER-Edit training.An example of this representation is provided in Figure 7.
- A Training Data: PEER outputs are controlled through tokenized key=value sequences specifying type, length, overlap, word-count difference, or required substrings.These controls target PEER-Explain, PEER-Undo, and PEER-Document behaviors.
C Generating Synthetic Data
PEER generates synthetic edits, plans, and documents through sampling procedures designed to provide diverse training examples and retain valid document evidence.
- C Generating Synthetic Data: Synthetic edits sample one pair per target using top-p sampling and draw word-control values from a clipped normal distribution.The sampling uses p = 0.9, with control-token values having µ = −10 and σ = 8, clipped at −40 and 10.
- C Generating Synthetic Data: Synthetic plans generate 10 alternatives per document pair while varying plan length, instruction type, and overlap settings.Top-p sampling uses p = 0.9; lengths are sampled from {s, m, l, xl}, while type=instruction and overlap=false are each selected 80% of the time.
- C Generating Synthetic Data: Synthetic documents sample 10 candidates containing the cited quote, discard invalid candidates, and retain the document maximizing PEER-Edit’s probability for the actual edit.The document generator uses top-p sampling with p = 0.9 and filters candidates based on whether they contain the required quote.
D Training Details
PEER training starts from T5 and uses distributed-training infrastructure plus scheduled optimization and regularization settings.
- D Training Details: PEER training begins from the T5 implementation in the Transformers library and uses DeepSpeed for efficient multi-GPU training.The cited implementations are attributed to Wolf et al. (2020) and Rasley et al. (2020).
- D Training Details: Optimization uses a maximum learning rate of 10^-4, 2,000 warmup steps, and linear decay.These settings define the learning-rate schedule used during training.
- D Training Details: Training applies gradient clipping, weight decay, and dropout with maximum norm 1.0, weight decay 0.01, and dropout rate 0.1.These settings provide the reported regularization and stabilization choices.
E Downstream Tasks
The downstream-task setup specifies task plans, baseline formatting conventions, reference handling, and example layouts for PEER and comparison models.
- E Downstream Tasks: Plans for downstream tasks are listed in Table 8, with manually written instructions except for ITERATER, which uses Du et al. (2022b)’s definitions.The plans correspond to the downstream tasks considered in Section 6.3.
- E Downstream Tasks: Baseline models such as T0, GPT3, InstructGPT, and OPT wrap each plan for an input using the described formatting template.The supplied passage introduces the wrapping procedure, while the adjacent template passage indicates that formatting follows a specified structure.
- E Downstream Tasks: Inputs with titles place the title after the string “Title:” before the input, while Tk-Instruct changes “Task” to “Definition.”These conventions adapt the prompt format to the example type and model-specific interface.
- E Downstream Tasks: Tasks requiring references append all references after the input using the string “Reference:”, and Figure 7 illustrates a linearized PEER-Edit input and output.The figure’s newlines improve readability but are not part of the model input.