Source-linked AI summary
Expository Text Generation: Imitate, Retrieve, Paraphrase
Nishant Balepur, Jie Huang, Kevin Chen-Chuan Chang
TL;DR
Writing expository text requires planning, finding dispersed facts, and rephrasing them clearly and consistently. IRP iteratively plans sentences, retrieves supporting facts, and paraphrases them; experiments report factual, organized outputs across three datasets, while its runtime scales linearly with output sentence count.
Problem
Expository writing requires careful planning, retrieval of dispersed facts, and faithful rewording in a consistent style.
Source: §1 · pp. 1–2Method
IRP iteratively generates stylistic content plans, retrieves facts using those plans, and paraphrases the facts into sentences.
Source: §1 · p. 2Results
Across three diverse datasets, IRP produces more factual expository texts than existing models, and human judges find the best balance between style and factuality.
Source: §1 · p. 2Takeaways & Limitations
Ablations find that sentence-level generation and fine-grained search queries each improve IRP’s performance.
Source: §5.3 · p. 8, §6 · p. 9IRP is computationally more expensive than traditional RAG, with runtime scaling linearly with the number of output sentences.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Expository documents are vital resources for conveying complex information to readers. Despite their usefulness, writing expository text by hand is a challenging process that requires careful content planning, obtaining facts from multiple sources, and the ability to clearly synthesize these facts. To ease these burdens, we propose the task of expository text generation, which seeks to automatically generate an accurate and stylistically consistent expository text for a topic by intelligently searching a knowledge source. We solve our task by developing IRP, a framework that overcomes the limitations of retrieval-augmented models and iteratively performs content planning, fact retrieval, and rephrasing. Through experiments on three diverse, newly-collected datasets, we show that IRP produces factual and organized expository texts that accurately inform readers.
1 Introduction
Expository text generation aims to create accurate, organized, stylistically consistent documents from a topic and factual corpus. IRP addresses the planning, retrieval, and rephrasing demands of this task through iterative sentence-level generation.
1 Introduction
- Expository text generation creates multi-sentence documents from a topic and related factual sentences, aiming to provide accurate and organized information.The task requires up-to-date facts from the knowledge source and a consistent style learned from expository texts.Source: §1 · p. 1
- The task is challenging because relevant facts may be dispersed across a corpus and must be reworded in a learned domain style.Retrieval-augmented models may hallucinate facts, while generic topic queries can miss nuanced information.Source: §1 · pp. 1–2
- IRP iteratively plans, retrieves, and paraphrases one sentence at a time, using stylistic content plans as fine-grained queries for factual retrieval.The Imitator draws on expository style, the Retriever finds facts, and the Paraphraser rewords them to match the plan.Source: §1 · p. 2
- IRP produces more factual texts than existing models across three datasets, and human judges find its outputs have the best balance between style and factuality.The datasets cover university descriptions, medical drug information, and computer science history Wikipedia articles.Source: §1 · p. 2
2 Related Work
IRP differs from prior retrieval-augmented generation by using stylistic sentence plans as queries and generating iteratively. The paper situates expository generation among factuality, style, summarization, and Wikipedia-generation tasks.
2.1 Retrieval-Augmented Generation
- IRP uses learned, fine-grained stylistic content plans rather than document titles as retrieval queries, and generates sentence-by-sentence; ablations find both choices improve performance.The paper notes contemporaneous iterative retrieval methods but does not compare them because they require extra engineering to synthesize facts from multiple sentences in a consistent style.Source: §2.1 · p. 3
2.2 Factuality and Style in Text Generation
- Factuality methods address generation through constraints, reranking, modified training data, objectives, and post-editing, while style methods use stylistic exemplars and guidance.Related approaches include controllable paraphrase, data-to-text, and keys-to-text generation.Source: §2.2 · p. 3
2.3 Summarization
- Unlike summarization, which condenses input, expository generation synthesizes facts dispersed in a knowledge source in a consistent style learned from examples.Source: §2.3 · p. 3
2.4 Wikipedia Generation
- Wikipedia generation builds articles from titles and retrieved web text, often for specific domains; expository generation spans more domains, unified by factuality and stylistic consistency.Source: §2.4 · pp. 3–4
3 Method
IRP takes a topic, factual corpus, and optional prefix, then uses an Imitator, Retriever, and Paraphraser to produce a stylistically consistent factual document. Its modules are trained separately, with Paraphraser examples constructed from similar stylistic plans and retrieved facts.
3 Method
- IRP takes a topic, factual sentence corpus, and prefix, and aims to produce accurate facts from that corpus in a style learned from training documents.The corpus is fixed for model comparison but could be acquired in real time for up-to-date facts.Source: §3 · p. 4
3.1 Imitator
- The Imitator generates a stylistic plan for the next sentence from the existing output or initial prefix, learning expository content planning through language modeling.It generates one sentence plan at a time and stops when it emits the end-of-text token.Source: §3.1 · p. 4
3.2 Retriever
- The Retriever uses DistilBERT to rank corpus sentences against a content plan, seeking top-k facts while reducing the influence of hallucinated entities in the plan.It is trained to classify sentence positions in expository documents; relevance is computed by dot product of sentence and plan embeddings.Source: §3.2 · pp. 4–5
3.3 Paraphraser
- The Paraphraser generates a sentence from retrieved facts and a stylistic plan, synthesizing information across multiple fact sentences while following the plan’s style.BART receives the plan, retrieved facts, and document topic as input.Source: §3.3 · p. 5
3.4 The Iterative IRP Framework
- In each IRP iteration, the Imitator plans, the Retriever finds related facts, and the Paraphraser forms one sentence that is appended as context for the next iteration.Generation ends when the Imitator emits the end-of-text token.Source: §3.4 · p. 5
3.5 Training
- The Imitator and Retriever train on expository documents, while Paraphraser training examples pair a target sentence with a similar plan and retrieved facts.Retrieving facts using the target sentence rather than the plan makes the training facts more likely to contain information needed to reconstruct the target.Source: §3.5 · p. 5
4 Experimental Setup
The experiments use three newly collected datasets and compare IRP with several language-model and retrieval-based baselines. They assess output similarity and factuality, using corpora that are assumed to contain accurate, consistent facts.
4.1 Datasets
- The three datasets cover college descriptions, medication information, and computer-science history sections, with 70/10/20 train/validation/test splits.The datasets are U.S. News (433 descriptions), Medline (844 medications), and WikiCS (500 Wikipedia articles).Source: §4.1 · p. 6
- Each dataset has a without doc version using retrieved web sentences and a with doc version that adds the gold text to the corpus.The setup assumes corpora contain accurate, consistent facts; back-translation adds variation to with doc gold outputs while preserving their factual content and style.Source: §4.1 · p. 6
4.2 Baselines
- Baselines include LLaMA, LLaMA with retrieved sentences, LED, RAG, and BART, which uses the topic alone to test whether models use the factual corpus.LLaMA uses five representative training examples; RAG retrieves top-k facts, while LLaMA+Retr uses the top five retrieved sentences.Source: §4.2 · p. 6
4.3 Training Setup
- IRP uses GPT-2 Large, DistilBERT Base, and BART Large for its Imitator, Retriever, and Paraphraser, respectively.Generation uses dataset-specific prefixes selected from common training-set prefixes, and sentences deemed repetitive by Retriever embeddings are filtered.Source: §4.3 · p. 6
4.4 Quantitative Metrics
- Evaluation combines ROUGE-1, ROUGE-2, BLEU, and METEOR with factuality measures for hallucinated tokens, FactCC consistency, and entailment or contradiction against the true output.The entailment and contradiction estimates use a DistilBERT classifier trained on MNLI; all metrics are reported from a single run.Source: §4.4 · pp. 6–7, Table 1 · p. 7
5 Results
IRP combines iterative sentence-level generation with fine-grained retrieval to produce expository texts with strong factuality while maintaining domain style. Results cover benchmark comparisons, human ratings, ablations, factual-error patterns, Retriever behavior, and example outputs.
5.1 Performance Comparison
- IRP achieves the strongest results on 19 of 24 factuality metrics and, apart from Medline, outperforms baselines on almost all traditional metrics while maintaining expository style.The authors suggest this supports iteratively performing content planning, retrieval, and paraphrasing.Source: §5.1 · pp. 7–8
5.2 Human Evaluation
- IRP strikes the best factuality–style balance in 3 of 4 datasets and competes with ChatGPT on the fourth, despite having 1.2B versus 175B parameters.LLMs generally do better in factuality but worse in style, while seq2seq models show the opposite pattern.Source: §5.2 · p. 8, Table 2 · p. 8
5.3 Ablation Study
- Both stylistic content-plan queries and sentence-level generation improve IRP performance over topic queries and generating all at once.Table 3 compares IRP with these variants on WikiCS with documents.Source: §5.3 · p. 8, Table 3 · p. 8
5.4 Factual Error Investigation
- Many IRP factual errors arise when retrieved facts contain alternatives, rather than from weaknesses in retrieval, paraphrasing, or data collection.The authors give outdated university rankings as an example and raise fact verification as a future research question.Source: §5.4 · pp. 8–9, Figure 4 · p. 9
5.5 Retriever Embedding Analysis
- The Retriever assigns less weight to factually specific hallucinated entities and can focus on terms more useful for retrieving information.Figure 3 visualizes token attribution scores; darker blue indicates higher attribution.Source: §5.5 · p. 9, Figure 3 · p. 9
5.6 Sample Expository Document Outputs
- Examples of generated documents from the three datasets are provided in Appendix A.3.Source: §5.6 · p. 9
6 Conclusion
The paper introduces expository text generation and IRP, which iteratively plans content, selects facts, and paraphrases. Evaluations report factual accuracy and stylistic consistency, while ablations support sentence-level generation and fine-grained search queries.
6 Conclusion
- The paper introduces expository text generation and designs IRP to overcome limitations of existing models through iterative content planning, fact selection, and paraphrasing.The task aims to generate a factual and organized document that clearly informs readers about a specified topic.Source: §6 · p. 9
- Automatic and human evaluations on three newly collected datasets reveal that IRP preserves factual accuracy while maintaining stylistic consistency.The conclusion also reports that human judges found IRP outputs had the best balance between style and factuality.Source: §6 · p. 9
- Ablations confirm the importance of sentence-level generation and fine-grained search queries, and the paper examines Retriever visualizations and IRP factual errors.Source: §6 · p. 9
7 Limitations
The paper identifies training, computational, and knowledge-acquisition limitations for IRP and expository text generation. It points to live retrieval during inference as a possible direction for narrowing the with-doc performance gap.
7 Limitations
- IRP trains three separate components for each expository document domain and is not trained end-to-end.The authors chose separate components for interpretability and simpler error detection, and say this leaves room for improvement.Source: §7 · p. 9
- IRP is more computationally expensive than traditional RAG models, with runtime scaling linearly with the number of output sentences.Source: §7 · p. 9
- Most models perform much better on with-doc datasets, and the authors propose real-time web retrieval during inference as a promising next step.They suggest rewording stylistic content plans as search queries to retrieve up-to-date information.Source: §7 · pp. 9–10
8 Ethical Considerations
IRP may produce factual errors, and the paper describes a misuse risk involving a malicious Paraphraser. It identifies safeguards as a future research need.
8 Ethical Considerations
- Like other text-generation frameworks, IRP may produce factual errors despite its goal of generating factual content.The authors suggest fact verification, live retrieval, more knowledge sources, or source attribution as possible ways future models could ameliorate harms.Source: §8 · p. 10
- A malicious Paraphraser could be plugged into IRP to turn true information into misinformation or deceptive claims.The paper identifies safeguards for detecting and combating these potential risks as future research needs.Source: §8 · p. 10
A.1 Detailed Dataset Collection
The authors collected expository documents from public websites and constructed factual-sentence corpora using search results for document topics and sentences. They cleaned the extracted sentences, including removing detected personal information.
A.1 Detailed Dataset Collection
- The authors web-scraped university, medical, and computer-science-history documents; they found no specific research licenses but note the datasets are publicly available online.They also report that each dataset had been analyzed in previous NLP research papers.Source: §A.1 · p. 14
- For each document sentence, the authors queried the top five search results using the topic and sentence, excluded URLs containing the ground-truth document, and extracted unique sentences from paragraph tags.Source: §A.1 · p. 14
- The authors cleaned sentences to ASCII, reviewed samples for offensive language and personal information, and removed sentences flagged by Presidio above a 0.3 prediction score.The listed detected entities include phone numbers, email addresses, bank and identity numbers, cryptocurrency, and IP addresses.Source: §A.1 · p. 14
A.2 Detailed Training Setup
IRP uses separately trained Imitator, Retriever, and Paraphraser components, with model and training settings held constant across datasets apart from slight training-time differences.
A.2 Detailed Training Setup
- GPT-2 Large trains the Imitator for 3,000 steps with batch size 1 and learning rate 1e-3.The Imitator uses the aitextgen package; other parameters follow its defaults.Source: §A.2 · p. 14
- DistilBERT Base uncased trains the Retriever for one epoch with batch size 16, learning rate 2e-5, and weight decay 0.01.Training uses the AdamW optimizer on a single NVIDIA A40 GPU.Source: §A.2 · p. 14
- BART Large trains the Paraphraser for five epochs with batch size 32, learning rate 2e-5, weight decay 0.01, and eight gradient-accumulation steps.Training takes about two hours; the generator uses the same parameters as the Paraphraser.Source: §A.2 · p. 14
- At prediction time, GPT-2 uses temperature 0.7 and a 512-token maximum, DistilBERT retrieves 15 sentences, and BART has 512-token input and output limits.LED uses input size 16,384; RAG and BART baseline generators use the Paraphraser’s training parameters, and RAG retrieves 15 sentences.Source: §A.2 · p. 14
- LLaMA and LLaMA+Retr use five manually selected in-context examples, while ChatGPT uses five-shot prompting through its web interface.Baseline outputs receive the same quality-control check as IRP to filter semantically repetitive sentences and improve fluency.Source: §A.2 · p. 14
A.3 Qualitative Analysis
IRP’s generated examples are described as factually accurate while retaining fluency and domain style; the qualitative comparisons also identify dataset-specific weaknesses in baseline outputs. The accompanying evaluation materials describe human-rating instructions, prompts, and retriever and ablation comparisons.
A.3 Qualitative Analysis
- IRP’s sample documents show high factual accuracy without sacrificing fluency or the expository domain’s style.The examples cover U.S. News, Medline, and WikiCS, with and without document input.Source: §A.3 · pp. 14–15
- Baselines show dataset-specific weaknesses: factual errors on U.S. News and Medline, and limited structure and coherence on WikiCS.LLaMA struggles most to preserve domain style; LED is factually correct on WikiCS but gives insufficient detail or topic coverage in the cited examples.Source: §A.3 · p. 15, Table 12 · p. 24
A.4 Human Evaluation
- The human-evaluation materials include instructions for judging style and factual accuracy, and a WikiCS LLaMA prompt with a retrieval variant that adds concatenated DPR facts.Figure 5 shows the prompt setup; Figure 6 contains the evaluation instructions.Source: §A.4 · p. 15, Figure 5 · p. 16, Figure 6 · p. 21
- DistilBERT outperforms DPR on 5 of 6 datasets when retrieving top-five sentences, measured by average ROUGE-1 recall against the ground-truth output.The recall measures the proportion of output tokens covered by retrieved factual sentences.Source: Table 5 · p. 17
- The ablation comparison covers ROUGE, BLEU, METEOR, and factuality metrics; Gen All generates at once, while Topic Query searches with the topic instead of IRP’s stylistic content plans.The supplied table caption identifies the metrics and variant definitions but does not provide the outcome values.Source: Table 6 · p. 17