Source-linked AI summary
Improving Text Auto-Completion with Next Phrase Prediction
Dong-Ho Lee, Zhiqiang Hu, Roy Ka-Wei Lee
TL;DR
Text auto-completion models can require substantial training effort to adapt to expert domains. This paper introduces intermediate training with Next Phrase Prediction (NPP), which guides phrase-level completion; experiments report stronger auto-completion performance on email and academic-writing datasets than the base models.
Problem
Language models can generate syntactically sound completions but require substantial training effort to adapt auto-completion to specific writing domains.
Source: §1 · p. 1Method
The paper intermediately trains a pre-trained text-to-text transformer with NPP, which guides it to complete a partial query using an appropriate phrase.
Source: §1 · p. 2Results
NPP-intermediately trained models outperform the base models on email and academic-writing auto-completion, and NPP+T5 exceeds NSP+T5 by 0.2 to 0.3 BLEU/METEOR/SPICE score.
Source: §4.5 · p. 4Takeaways & Limitations
The approach outperforms baselines for email and academic-writing auto-completion with around 1.2B training tokens.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Language models such as GPT-2 have performed well on constructing syntactically sound sentences for text auto-completion task. However, such models often require considerable training effort to adapt to specific writing domains (e.g., medical). In this paper, we propose an intermediate training strategy to enhance pre-trained language models' performance in the text auto-completion task and fastly adapt them to specific domains. Our strategy includes a novel self-supervised training objective called Next Phrase Prediction (NPP), which encourages a language model to complete the partial query with enriched phrases and eventually improve the model's text auto-completion performance. Preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic writing domains.
1 Introduction
Text auto-completion is a key feature of natural-language interfaces, but adapting language models to expert domains can require substantial training data and effort. The paper proposes intermediate training with Next Phrase Prediction (NPP) and reports improved auto-completion and domain adaptation.
1 Introduction
- Deep neural auto-completion models require substantial everyday text and computing power, while expert-domain writing requires a large domain-specific training corpus.The introduction cites academic writing as a domain-specific example.Source: §1 · p. 1
- With 100K training instances per domain, Bi-LSTM perplexity for academic writing is almost double that for email.Table 1 compares the two domains under equal training-instance counts.Source: Table 1 · p. 1
- GPT-2 can generate syntactically sound, general sentences from partial queries but needs extensive fine-tuning for expert-domain-focused output.The figure caption illustrates this contrast with computer-science sentences.Source: §1 · p. 1, Figure 1 · p. 1
- The proposed intermediate training strategy uses NPP to improve text auto-completion and adapt pre-trained language models to expert domains.NPP guides the model to select a complete phrase of the same type as the target phrase, after phrase extraction by constituency parsing.Source: §1 · p. 2
- Experiments report that the approach improves text-to-text transformer performance on auto-completion and adapts models to expert domains.The authors describe this as the first intermediate-training strategy proposed for improving language-model performance on text auto-completion.Source: §1 · p. 2
2 Overview
The paper frames auto-completion as extending a partial query into a full sentence and evaluates output soundness and semantic similarity. Its workflow applies NPP to a pre-trained T5 model before fine-tuning for the target task.
2 Overview
- Section 2 introduces the formal auto-completion problem and the workflow for the intermediate-training strategy.The workflow proceeds from intermediate NPP training to target-task fine-tuning.Source: §2 · p. 2, §2.2 · p. 2
2.1 Problem Statement
- Auto-completion returns a full sentence in which the completion is a syntactic and semantic extension of the partial query.The model is evaluated for output soundness and semantic similarity to the ground truth.Source: §2.1 · p. 2
2.2 Workflow
- The workflow starts with pre-trained T5, applies NPP as intermediate-task training, and then fine-tunes on the target auto-completion task.Figure 2 depicts phrase groups serving as answer options for next-phrase selection.Source: §2.2 · p. 2, Figure 2 · p. 2
3 Next Phrase Prediction
NPP trains a text-to-text transformer to complete a partial query with an adequate phrase, using constituency-parsed phrases and a generative question-answering objective. The method extracts NP, VP, and PP phrases, then trains the model to identify a correct next phrase among same-type options.
3 Next Phrase Prediction
- NPP uses phrase-level completion because phrases can express meaning beyond word concatenation and commonly appear as units in text.The objective comprises phrase extraction followed by generative question answering, unlike objectives focused on the next word or text span.Source: §3 · pp. 2–3
3.1 Phrase Extraction
- Constituency parsing extracts noun, verb, and prepositional phrases, retaining only phrases without a child of the same type to avoid overlap.For “She wants to eat pie,” the method keeps “eat pie” as the verb phrase.Source: §3.1 · p. 3
3.2 Generative QA
- The generative QA task asks the model to predict a selected phrase from a partial query while providing phrases from its type group as answer choices.A phrase group is randomly selected only when it contains more than two phrases.Source: §3.2 · p. 3
4 Experiments
The experiments evaluate NPP intermediate training on email and academic-writing auto-completion, comparing it with untrained baselines and another intermediate objective. NPP+T5 outperforms the base models on both datasets and NSP+T5 across BLEU, METEOR, and SPICE.
4.1 Details for intermediate training
- NPP intermediate training samples 1M English Wikipedia sentences, about 1.2B tokens, fewer than the 34B tokens used for T5 and 10B for GPT-2.The sampled sentences are used as source data for NPP.Source: §4.1 · p. 3
4.2 Target Dataset
- The target datasets use Enron emails and English academic abstracts, and sentences are split at word boundaries into partial queries and remaining completions for fine-tuning.The academic abstracts are mainly from computer science; the source collections include DBLP and ACM.Source: §4.2 · p. 3
4.5 Experimental Results
- NPP+T5 outperforms the base models on both email and academic-writing datasets, and exceeds NSP+T5 by 0.2 to 0.3 BLEU/METEOR/SPICE score.Table 3 reports the experimental comparison; the paper also says NPP+T5 outperforms GPT-2 despite having less than half its parameters.Source: Table 3 · p. 4, §4.5 · p. 4
4.3 Base models
- The baselines include GPT-2 large and T5-base, while NSP+T5 and NPP+T5 are intermediately trained using next-sentence and next-phrase prediction, respectively.GPT-2 large has 774M parameters and T5-base has 220M parameters.Source: §4.3 · p. 4
4.4 Evaluation Metrics
- The evaluation uses BLEU, METEOR, CIDEr, and SPICE to assess generated sentences’ syntactic and semantic soundness through surface similarities and associations with original text.The metrics assess whether generations are semantically expert-domain focused.Source: §4.4 · p. 4
4.5 Experimental Results
- Table 4 compares T5 and NPP+T5 completions for the same academic-writing queries; NPP+T5 completions are generally more acceptable in semantic similarity to the originals.The caption identifies underlining as marking overlap words between original completions and generated completions.Source: §4.5 · p. 4, Table 4 · p. 4
5 Conclusion
The conclusion presents NPP as an intermediate training strategy that encourages enriched phrase completions. Preliminary experiments report baseline-beating performance in email and academic-writing auto-completion, with broader domain testing left for future work.
5 Conclusion
- The proposed intermediate training strategy uses Next Phrase Prediction to encourage enriched phrase completion and improve text auto-completion performance.The approach intermediately trains a language model with the NPP self-supervised objective.Source: §5 · pp. 4–5
- Preliminary experiments show NPP outperforming baselines for email and academic-writing auto-completion with around 1.2B training tokens.Source: §5 · p. 5
- Future work aims to test the approach in more writing domains and develop a demonstration system for text auto-completion.Source: §5 · p. 5