Source-linked AI summary
Text Editing by Command
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, Bill Dolan
TL;DR
One-shot generation does not readily accommodate dynamic user constraints, motivating an interactive editing approach. The paper proposes command-driven edits with grounding and trains an editor on Wikipedia edits; it reports outperforming parrot and GPT-2 baselines and competitive human-evaluation results against gold-standard edits.
Problem
One-shot generation offers limited scope for users to adjust requirements as they evolve during writing.
Source: §1 · pp. 1–2Method
Results
The trained model outperforms parrot and GPT-2 baselines and achieves competitive results against gold-standard edits in human evaluations.
Source: §1 · p. 2Takeaways & Limitations
Ablations show the importance of commands and grounding, and the model performs well on fluency edits as well as harder content edits.
Source: §8 · p. 9The evaluation focuses on single turns, and strong single-turn performance does not establish performance over multiple turns, where edits may have path dependencies.
Source: §6 · p. 7
Abstract
from arXiv · showhide
A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. The one-shot setting is inadequate, however, when the constraints the user wishes to impose on the generated text are dynamic, especially when authoring longer documents. We address this limitation with an interactive text generation setting in which the user interacts with the system by issuing commands to edit existing text. To this end, we propose a novel text editing task, and introduce WikiDocEdits, a dataset of single-sentence edits crawled from Wikipedia. We show that our Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations. We present empirical and qualitative analyses of this model's performance.
1 Introduction
One-shot generation cannot accommodate users’ changing constraints during longer writing, so the paper frames text editing as a step toward interactive document generation. It introduces command-driven, grounded edits and reports favorable automatic and human evaluations of its trained model.
1 Introduction
- The paper introduces controllable text editing, where users revise existing text to meet dynamic constraints through a stateful process rather than generating from scratch.The proposed edits can add or remove content or change meaning, extending beyond small paraphrases.Source: §1 · pp. 1–2
- The task asks a model to edit text in response to user commands while using grounding to help avoid hallucination.The paper releases WikiDocEdits, a dataset of sentence-level Wikipedia edits with editor comments used as commands and pre-retrieved grounding documents.Source: §1 · p. 2
- The transformer-based model trained on WikiDocEdits outperforms parrot and GPT-2 baselines and achieves competitive results against gold-standard edits in human evaluations.The paper also analyzes the roles of commands and grounding and the varying difficulty of edits.Source: §1 · p. 2
2 Text Editing Task
The task is to produce an edited document from the current document, a user command, and grounding. This formulation treats editing as controlled, stateful revision rather than recasting text from scratch.
2 Text Editing Task
- Given a document, command, and grounding, the model learns to produce an edited version of the document.The formulation defines a dataset of document, command, grounding, and edited-document tuples.Source: §2 · p. 2
- Commands specify what kind of edit to make, while grounding supplies external knowledge needed to make it.Prior text-editing work described here typically takes only the document as input.Source: §2 · p. 2
- The paper’s task instance edits one sentence at a time, with document-level edits treated as compositions of sentence-level edits.The source and edited documents differ only in a single sentence.Source: §2 · p. 2
- The task uses natural-language commands and text snippets as grounding, with dataset groundings retrieved by querying a commercial search engine.The paper notes that commands and grounding could alternatively use categorical variables and structured data, respectively.Source: §2 · p. 2
3 Data
WikiDocEdits contains nearly 12 million sentence-level edits extracted from Wikipedia revisions, with editor comments used as command proxies and retrieved snippets as grounding. Its analysis examines edit types and how well the inputs cover inserted content.
3 Data
- WikiDocEdits comprises nearly 12 million sentence-level edits from Wikipedia revision histories.Editor comments serve as proxies for user commands; only revisions with a single sentence-level edit are retained so the comment can be associated with that edit.Source: §3 · pp. 2–3
3.1 Grounding
- For each edit, grounding is retrieved by querying a commercial search engine with page and section titles plus target-sentence keywords; preview snippets are checked for overlap, reranked, and merged to 512 tokens.Snippets with more than 50% 4-gram overlap with the target sentence are removed; remaining snippets are reranked using an information extraction score.Source: §3.1 · p. 3
3.2 Data Analysis
- The dataset contains 11,850,786 extracted edits, with a 1% sample of 118,818 used for analysis.The edits come from the February 1st 2020 English Wikipedia dump.Source: §3 · p. 2, §3.2 · p. 3
- Automatically classified labels indicate fluency edits are the majority, while a large proportion are content edits.The edit-intention categories are grouped from a classifier’s labels, so the breakdown is subject to automatic-labeling caveats.Source: §3.2 · p. 3, Table 2 · p. 4
- Coverage analysis measures inserted-word coverage with BERTScore recall, using fuzzy embedding matches and inverse-document-frequency weights.The analysis asks whether information needed for an edit appears in the source, context, command, or grounding inputs.Source: §3.2 · pp. 3–4
- Grounding alone provides fairly high coverage of targets, and command-only coverage indicates that commands also provide grounding.Table 3 reports coverage for different input corpora on a 1% subsample.Source: §3.2 · p. 4, Table 3 · p. 4
- Examples illustrate elaborations, typo fixes, factual changes, and more complex insertion-and-deletion edits, indicating substantial variety in the dataset.The examples are presented in Table 4.Source: §3.2 · p. 4
4 Model
Interactive Editor models sentence editing as autoregressive sequence generation conditioned on the source sentence, its document context, a user command, and grounding text. It uses a T5-style encoder-decoder with pretrained weights and assumes the sentence to edit is supplied.
4 Model
- Because WikiDocEdits contains only single-sentence edits, the model assumes the sentence to edit is given as input.Inputs are tokenized, concatenated with separator tokens, then truncated or padded to fixed lengths; training uses cross-entropy loss.Source: §4 · pp. 4–5
- The model conditions each candidate edit on the source sentence, surrounding document context, user command, and grounding corpus.Its encoder performs full attention over these inputs.Source: §4 · pp. 4–5
- Interactive Editor uses a T5 encoder-decoder architecture with pretrained language-model weights to generate edited sentences autoregressively.The encoder attends over the source sentence, document context, command, and grounding corpus; the decoder generates the target sentence.Source: §4 · pp. 4–5
5 Experiments
The experiments evaluate Interactive Editor against baselines and ablations using automatic and human evaluations. The model outperforms both baselines across reported metrics, while human judgments also assess command fulfillment and grounding relevance.
5.1 Evaluation
- The model was finetuned on 1,000K training edits; evaluation used BLEU, word-edit precision/recall/F1, and exact-match accuracy against parrot and GPT-2 baselines.The training split was 1,000K/10K/10K edits for training, validation, and test.Source: §5 · p. 5, §5.1 · pp. 5–6
5.2 Results
- Interactive Editor beats both baselines across all reported metrics.Table 5 compares the model with the parrot and GPT-2 baselines, as well as ablations.Source: Table 5 · p. 6, §5.2 · p. 6
- Removing context, grounding, or commands hurts performance; command and grounding are each crucial and appear synergistic.Removing either command or grounding alone has limited effect, suggesting the two inputs work together.Source: §5.2 · p. 6
- The model performs better on fluency edits than content edits, yet its BLEU-over-source improvement is equal across both types.Content edits require retrieving and incorporating information from grounding; the edit-type labels come from an automatic classifier, not human annotation.Source: §5.2 · pp. 6–7
5.3 Human Evaluations
- 59% of command-task judgments rated Interactive Editor equal to or better than the reference, although judges clearly preferred the references overall.The comparison used each system’s top beam-search output over 200 test examples.Source: Table 7 · p. 7, §5.3 · p. 7
- 92.86% of judgments rated retrieved grounding as somewhat or very relevant to the context.Judges assessed grounding relevance in the context of the document.Source: §5.3 · p. 7
- About 65% of examples with at least one neutral judgment had majority-neutral judgments, suggesting some commands were difficult for judges to interpret.The authors note that other factors may also partly account for the high neutral rate.Source: §5.3 · p. 7
6 Discussion
The discussion illustrates a multi-turn use of the single-turn model, while noting that performance across multiple edits remains uncertain. The example uses grounding to make a false sentence about Geoff Hinton more factually correct.
6 Discussion
- The model is evaluated only on single turns, so success across multiple edits is uncertain because edits may have path dependencies.The multi-turn interaction is offered as an example, not evidence of multi-turn performance.Source: §6 · p. 7, Table 8 · p. 7
- A sequence of edits turns a false sentence about Geoff Hinton into a factually correct one by retrieving and inserting relevant grounding information.When asked to “fix profession,” the model changes the claim that Hinton is a tennis player to identify him as a computer scientist; the authors present this as an illustrative, not empirical, result.Source: §6 · pp. 7–8
7 Related Work
The related work situates command-driven editing among grounding, controllable generation, story generation, and prior editing tasks. The paper distinguishes its focus on generating content-changing edits under dynamic user constraints.
Grounded and Controllable Generation
- Unlike grounding work focused on factual consistency, this task uses a user command to retrieve specific information from grounding during generation.The paper also contrasts natural-language commands with lexical and attribute controls, noting that commands can express low- or high-level constraints.Source: §7 · p. 8
Controllable Generation
- The paper’s grounding and command-based task also connects to dialog and question-answering work, but those tasks do not address text generation for document creation.The paper treats natural-language commands as flexible controls over edits.Source: §7 · p. 8
Story Generation
- Unlike story-generation approaches that condition on plans or other constraints, this approach treats document creation as iterative, with constraints supplied dynamically on demand.The paper also frames this as allowing a human to generate a full document from scratch through repeated edits.Source: §7 · p. 8
Text Editing
- Prior text-editing work includes prototype-based generation, edit representation, and fact-based editing, whereas this paper focuses on generating edits in response to commands.The paper’s task also differs from paraphrasing, which usually preserves sentence meaning; its edits can add or modify content.Source: §7 · p. 8
8 Conclusion
The paper argues for interactive text generation through user-commanded editing, and reports encouraging evaluation results for a single-sentence editing model.
8 Conclusion
- The proposed task asks a system to edit a document in response to a user command.The authors frame this as a step toward interactive document generation.Source: §8 · p. 8
- A transformer-based model trained on WikiDocEdits, initialized with pretrained language model weights, obtains encouraging automatic and human evaluation results.Ablations show the user command and grounding play crucial roles, while the model performs on both fluency and harder content edits.Source: §8 · p. 9
- A multi-turn example suggests the single-turn editing model could feasibly be used to generate longer documents.Source: §8 · p. 9
A.1 Data Processing pipeline
The pipeline filters Wikipedia revisions, aligns sentences to identify edits, and removes some residual markup artifacts.
A.1 Data Processing pipeline
- Revisions are retained only when they affect one section and include an editor comment; talk and user pages are excluded.Source: §A.1 · p. 10
- Table 9 reports edit-intention prevalence in WikiDocEdits and a hand-labelled dataset, with percentages that may exceed 100% because edits can have multiple labels.Source: Table 9 · p. 11
- Markup stripping is not guaranteed to remove all markup, so edits containing markup punctuation are filtered out as a final cleanup step.The authors note that malformed markup can remain after stripping.Source: §A.1 · p. 12
- Source and target sentences are matched within a 20-sentence neighborhood to avoid comparing every possible pair.Unmatched sentences become edit candidates.Source: §A.1 · p. 12
- Revisions with multiple sentence-level edits are discarded, and paired unmatched sentences count as edits only when their BLEU score exceeds 0.1.An unmatched source or target sentence is treated as a deletion or insertion only when the total sentence counts differ by one.Source: §A.1 · p. 12
A.2 Grounding Search Query Construction
The appendix describes how the authors construct grounding searches, rerank retrieved snippets, and evaluate word-level edits.
A.2 Grounding Search Query Construction
- Grounding queries combine page and section titles with target-sentence keywords, defined as words appearing in fewer than 1% of a 500,000-article sample.Source: §A.2 · pp. 12–13
A.3 Grounding Document Reranking
- Retrieved grounding snippets are reranked with tf-idf information-extraction scores because their combined length exceeds model capacity.The authors merge ranked snippets and retain the first 512 tokens.Source: §A.3 · p. 13
A.4 Word Edit Metrics
- Word-level edit evaluation represents edits as insertion or deletion operations at positions, then compares the model's edits with the reference.The word tokenizer is from NLTK.Source: §A.4 · pp. 13–14