Source-linked AI summary
Teach LLMs to Personalize -- An Approach inspired by Writing Education
Cheng Li, Mingyang Zhang, Qiaozhu Mei, Yaqing Wang, Spurthi Amba Hombaiah, Yi Liang, Michael Bendersky
TL;DR
Personalized text generation research has often relied on domain-specific approaches, and cross-domain LLM personalization remains less explored. This paper proposes a writing-education-inspired multistage and multitask framework; it reports significant improvements over varied baselines across three public datasets.
Problem
Personalized generation has largely been studied with domain-specific approaches, while LLM personalization across domains without domain-specific or user-defined features is less explored.
Source: §2 · p. 2Method
Results
The framework significantly improves over a variety of baselines on three public datasets covering personal email, social media discussions, and product reviews.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that the multistage multitask framework is effective across representative domains.
Source: §8 · p. 9The synthesis step experiments with keyword extraction, while more sophisticated synthesis approaches are left for future work.
Source: §5.4 · p. 4
Abstract
from arXiv · showhide
Personalized text generation is an emerging research area that has attracted much attention in recent years. Most studies in this direction focus on a particular domain by designing bespoke features or models. In this work, we propose a general approach for personalized text generation using large language models (LLMs). Inspired by the practice of writing education, we develop a multistage and multitask framework to teach LLMs for personalized generation. In writing instruction, the task of writing from sources is often decomposed into multiple steps that involve finding, evaluating, summarizing, synthesizing, and integrating information. Analogously, our approach to personalized text generation consists of multiple stages: retrieval, ranking, summarization, synthesis, and generation. In addition, we introduce a multitask setting that helps the model improve its generation ability further, which is inspired by the observation in education that a student's reading proficiency and writing ability are often correlated. We evaluate our approach on three public datasets, each of which covers a different and representative domain. Our results show significant improvements over a variety of baselines.
1 INTRODUCTION
Personalized generation can tailor content to users’ prior writing, but most existing approaches target particular domains and few studies address personalization with LLMs. This work adapts writing-from-sources instruction into a multistage, multitask framework and reports improvements across three public datasets.
1 INTRODUCTION
- Personalized responses use auxiliary context, such as a user’s previously written documents, to address particular audiences, creation contexts, and information needs.The paper notes applications ranging from social posts and articles to communications and style transformations.Source: §1 · p. 1
- Prior personalized-generation research mostly uses domain-specific features or knowledge, while general approaches for LLM personalization remain underexplored.Existing work has addressed reviews, dialogue agents, and social networks; few studies have explored equipping LLMs to personalize.Source: §1 · p. 1
- The proposed approach adapts writing-from-sources instruction into retrieval, ranking, summarization, synthesis, and generation stages.The approach is inspired by writing education’s sequence of finding, evaluating, summarizing, synthesizing, and integrating information.Source: §1 · pp. 1–2
- An auxiliary authorship-attribution task is intended to improve LLM reading ability and, in turn, personalized generation, drawing on reported correlations between reading and writing proficiency.The authors also cite evidence that author-recognition tasks can measure an individual’s reading and correlate with reading proficiency.Source: §1 · p. 2
- The framework significantly improves over a variety of baselines on three public datasets covering personal email, social media discussions, and product reviews.The datasets represent three different domains.Source: §1 · p. 2
2 RELATED WORK
Earlier personalized-generation work largely targets particular domains, while LLM-based personalization across domains without domain-specific or user-defined features is less explored. The paper distinguishes its passage-length generation setting from related benchmarks and from controlled generation and style transfer.
2 RELATED WORK
- Prior personalized-generation studies address reviews, dialogue agents, and social networks, often using domain-specific features or knowledge.Examples include user reviews conditioned on product information and history, social-text generation guided by personalized features, and dialogue datasets built from personas or user posts.Source: §2 · p. 2
- LLM personalization across domains without domain-specific or user-defined features remains less explored; LaMP offers three classification and four text-generation tasks, with sentence-level generation.The paper instead considers passage-length generation and treats retrieval-based personalization as one component of its multistage framework.Source: §2 · p. 2
- Controlled text generation steers output toward predefined stylistic or semantic attributes, unlike personalization as framed here.The paper notes that controlled-generation methods can steer pretrained models at decoding time, whereas its setting lacks predefined attributes.Source: §2 · p. 2
- Text style transfer transforms given text while preserving content, whereas personalized generation does not assume an original text is given.The paper also distinguishes style-transfer paradigms based on whether parallel or non-parallel corpora are available.Source: §2 · p. 2
- The paper aligns its task decomposition with teaching LLMs to reason through chains of thought, while describing its training as going beyond prompt engineering [36] [37].Source: §2 · p. 2
3 PROBLEM FORMULATION
The task is to complete a user’s current document using its immediate context and the user’s personal context, aiming for output similar to the real document. The formulation treats the title and a short opening as immediate context and previously written documents as personal context.
3 PROBLEM FORMULATION
- The model completes a user’s current document from immediate and personal context, aiming to make its output similar to the real document.Source: §3 · p. 2
- Immediate context is the document title and a short start, while personal context comprises documents the user wrote before the current document.The paper presents these contexts as analogous to sources used in writing instruction.Source: §3 · p. 2
- Training uses examples pairing a user’s immediate context and past documents with the current document, and seeks to maximize similarity between generated and real documents.The formulation denotes the past-document set as D and the generated current document as d′.Source: §3 · p. 3
4 METHOD OVERVIEW
The method combines retrieval, ranking, summarization, synthesis, and generation over a user’s prior documents. It also trains the generation model on an author-distinction task using document pairs labeled by whether they share an author.
4 METHOD OVERVIEW
- Figure 1 presents an overview of the multistage multitask framework.Source: §4 · p. 3
- The retriever queries a user’s past documents with current-document context, and the ranker orders retrieved entries for downstream use.The ranked entries feed both summarization and synthesis models conditioned on the immediate context.Source: §4 · p. 3
- The generator uses immediate context, a summary, synthesized key elements, and ranked entries to produce the current document against its ground truth.Source: §4 · p. 3
- The auxiliary author-distinction task trains the model on document pairs labeled true for the same author and false for different authors.The labels are text because the generation model is sequence-to-sequence.Source: §4 · p. 3
5 PERSONALIZED TEXT GENERATION
The framework conditions personalized generation on retrieved personal documents through ranking, summarization, and synthesis, and adds author distinction as an auxiliary task.
5.1 Retrieval
- Retrieval queries past personal documents with the current document’s immediate context, using sparse BM25 or dense retrieval at document or snippet granularity.The immediate context uses the first 150 characters and includes the title when available.Source: §5.1 · p. 3
5.2 Ranking
- RankDocBySnpt retrieves snippets but ranks their containing documents by each document’s best snippet–context similarity, addressing long-document and redundant-snippet issues.Source: §5.2 · p. 3
5.3 Summarization
- Context-dependent summaries are weakly labeled by matching ground-truth-document snippets to similar ranked-entry snippets and joining selected matches.The paper also tests context-independent summarization using a T5-11B model fine-tuned on public summarization datasets.Source: §5.3 · p. 4
5.4 Synthesis
- Synthesis extracts shared key elements as keywords, with context-independent frequency ranking and context-dependent weak labels based on likely-useful retrieved words.Richer synthesis approaches are left for future work.Source: §5.4 · p. 4
5.5 Personalized Generation
- Personalized generation conditions on immediate context, ranked entries, summary, and synthesis, training against the current document; source-specific prefixes distinguish inputs.Source: §5.5 · p. 5
5.6 Multitask Learning
- Author distinction adds a same-author document-pair task alongside generation to train style understanding, motivated by the reading–writing association.Half of the pairs are same-user; the rest are cross-user, and auxiliary-task users are excluded from personalization validation and test data.Source: §5.6 · p. 5
6 EXPERIMENT SETUP
The experiments assess the approach on three user-grouped public datasets, compare baseline and component variants, and use overlap metrics with paired significance tests.
6.1 Datasets
- Experiments use Avocado emails, Amazon book reviews, and Reddit posts and comments; qualifying documents exceed 300 characters, and users contribute at least five examples.Documents are grouped by users and split by user into train, validation, and test sets at 85/5/10.Source: §6.1 · p. 5
6.2 Training details
- T5-11B is fine-tuned for summarization, synthesis, and generation; multitask training mixes generation and author-distinction examples 1:1.Source: §6.2 · p. 5
6.3 Competing methods
- Baselines include immediate-context-only generation, UserID-trained next-snippet generation, and zero-shot PaLM 2 prompted with the best configuration.Source: §6.3 · pp. 5–6
- Comparisons vary retrieval and ranking, context-independent or context-dependent summarization and synthesis, and AuthorPred multitask training.LaMP is the closest related retrieval-based LLM approach but targets sentence-length generation.Source: §6.3 · pp. 5–6
6.4 Metrics
- Generated documents are scored against references with BLEU, ROUGE-1, ROUGE-2, and ROUGE-L, with paired t-tests for significance.Source: §6.4 · p. 6
7 EXPERIMENTAL RESULTS
Across three dataset benchmarks, retrieval-augmented methods generally outperform non-retrieval baselines, and context-dependent summarization and synthesis add gains; multitask learning further helps on most datasets.
7.1 Overall performance
- Retrieval-augmented methods generally outperform baselines without retrieved information, while multitask learning further improves generation on most datasets.The overall results cover Avocado email, Amazon reviews, and Reddit.Source: Table 2 · p. 6, §7.1 · pp. 6–7, Table 3 · p. 7
- All retrieval-augmented methods outperform ImmedCtx; RankDocBySnpt matches or exceeds other retrieval methods, while BM25 is less effective on Amazon reviews.RankDocBM25 otherwise performs closely to RankDocBySnpt.Source: §7.1 · p. 7
- RecentDoc matches some similarity-based retrieval methods in many cases, but immediate-context ranking can improve performance by retrieving content relevant to the current topic.RecentDoc is especially competitive on Avocado email.Source: §7.1 · p. 7
- Context-dependent summaries and synthesis outperform their context-independent counterparts; synthesis improves Rouge-1 more than other metrics.Generic summaries add no gain over retrieval augmentation without summarization, whereas context-dependent summaries help guide topic and phrase choices.Source: §7.1 · pp. 7–8
- AuthorPred outperforms single-task SynCtx on Avocado email and Reddit and performs closely on Amazon; UserID's shorter outputs help on shorter-document datasets but impede generalization to new users.LLMZeroShot's reduced performance suggests finetuning is important for this task.Source: §7.1 · pp. 7–8
7.3 Case studies
- Table 6 provides illustrative model-output examples, with italicized words marking the immediate context.The examples are presented to illustrate the proposed method's guidance for personalized content generation.Source: Table 6 · p. 8
7.2 Formulation of input and output
- Removing ranked entries or the document start degrades performance, while moving immediate context to the end leaves performance close to AuthorPred.The authors suspect including the short document start in both input and target helps the model understand the task and converge faster.Source: §7.2 · pp. 8–9, Table 5 · p. 8
8 CONCLUSION
The paper concludes that its multistage multitask framework is effective for personalized text generation. It also identifies incorporating world knowledge, such as product information, as a promising direction for future work.
8 CONCLUSION
- The authors report that the multistage multitask framework is effective across evaluations on three publicly released datasets from representative domains.Source: §8 · p. 9
- The approach teaches LLMs personalized generation through retrieval, ranking, summarization, synthesis, and generation.Source: §8 · p. 9
- An authorship-attribution multitask setting aims to improve the model’s reading ability and further improves personalized text generation empirically.Source: §8 · p. 9
- Incorporating world knowledge, such as product information, is identified as a promising direction for future work.Source: §8 · p. 9