PaperlayerView exact revision on arXiv

Source-linked AI summary

Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers

Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Bahareh Sarrafzadeh, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, Tara Safavi

arXiv:2311.09180v2cs.CLcs.HCcs.IR

TL;DR

LLM writing assistants can generate overly generic text that does not reflect an author’s style, knowledge, or values. PEARL trains a retriever to select useful user-authored examples and calibrate scores to generation quality. Evaluations show it matches or outperforms strong baselines, and its scores can support selective revision.

  • Problem

    LLM writing assistants may produce overly generic text, limiting personalization to an author’s communication style, specialized knowledge, and values.

    Source: §1 · p. 1

  • Method

    PEARL selects personalizable requests and useful historical documents, then trains a retriever with a scale-calibrating objective so scores track downstream generation quality.

    Source: §1 · p. 2

  • Results

    PEARL consistently matches or outperforms strong baselines on workplace communications and Reddit, and selective revision improves or retains performance by 2-4% with PEARL on WORKSM.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    Retriever scores can predict low-quality retrieval and help identify generations for LLM revision.

    Source: §1 · p. 2, §6 · p. 9

    Personalized generation may exacerbate misuse harms, while the implications of AI-mediated communication remain under development.

    Source: §7 · p. 9

Abstract

from arXiv · show

Powerful large language models have facilitated the development of writing assistants that promise to significantly improve the quality and efficiency of composition and communication. However, a barrier to effective assistance is the lack of personalization in LLM outputs to the author's communication style, specialized knowledge, and values. In this paper, we address this challenge by proposing Pearl, a LLM writing assistant personalized with a retriever that is trained to be generation-calibrated for personalization. Generation calibration ensures that our retriever selects historic user authored documents to augment an LLM prompt such that they are likely to help an LLM generation better adhere to a users' preferences. We propose two key novelties for training such a retriever: (1) A training data selection method that identifies user requests likely to benefit from personalization and documents that provide that benefit; and (2) A scale-calibrating KL-divergence objective that ensures that our retriever scores remain proportional to the downstream generation quality from using the document for personalized generation. In a series of holistic evaluations, we demonstrate the effectiveness of Pearl in generating long-form texts on multiple social media datasets. Finally, we demonstrate how a generation-calibrated retriever can double as a performance predictor -- detecting low quality retrieval, and improving potentially under-performing outputs via revision with LLMs.

1 Introduction

PEARL addresses the challenge of personalizing LLM writing to preserve an author’s knowledge, style, and values. It trains a retriever to select useful historical documents and calibrate their scores to generation quality, and its evaluations match or outperform strong baselines.

1 Introduction

  • PEARL uses historic user-authored documents to augment an LLM prompt, personalizing outputs to retain the user’s style, knowledge, and values.The paper focuses on LLMs accessed through prompt-based APIs.Source: §1 · pp. 1–2
  • A difference-of-likelihoods method selects requests likely to benefit from personalization and the associated documents likely to help downstream generations.The motivation is that users have roughly hundreds of historical documents and their requests may diverge as preferences change.Source: §1 · p. 2
  • A scale-calibrating objective makes retriever scores proportional to the quality of LLM output generated with each request-document pair.The calibrated scores can predict retrieval performance and support revision of potentially low-quality generations.Source: §1 · p. 2
  • PEARL consistently matches or outperforms strong baselines across a private workplace-communications dataset and public Reddit dataset.The evaluations use davinci-003 and gpt-35-turbo and span intrinsic, extrinsic, and personalized LLM-as-judge measures.Source: §1 · p. 2

2 Related Work

Prior retriever work trains document selection while generally assuming retrieval can help every request. PEARL adds selection of requests that benefit from personalization and uses calibration to identify low-quality retrievals; its retrieval approach complements other personalized-writing methods.

2 Related Work

  • Earlier retriever approaches distill LLM likelihoods or reward-model feedback, or use reinforcement learning, but assume satisfactory predictions can be made for all requests.PEARL instead selects both useful training documents and requests that benefit from retrieval, which the paper describes as necessary with smaller per-user histories.Source: §2 · p. 2
  • PEARL’s training pipeline selects personalizable historical requests and documents, trains a retriever with a scale-calibrating loss, then retrieves examples for unseen requests.The resulting scores are proportional to the quality of LLM output.Source: Figure 2 · p. 3
  • Unlike prior personalized writing systems using nearest-neighbor models or group-level fine-tuning, PEARL explores retrieval-based in-context personalization with LLMs.Prompt rewriting is described as complementary to a trained retriever, and future systems may benefit from both.Source: §2 · p. 3

3 Problem Definition

The task assumes a user’s historical authored documents, a textual request, and access to an LLM through a prompt-based API. A retriever chooses historical documents as few-shot examples for generation.

3 Problem Definition

  • Each user is associated with historical documents, such as previously authored social-media posts or emails, and a textual request that may be authored or inferred from task context.The request can be explicit or implicit, such as an email prefix requiring completion.Source: §3 · p. 3
  • The system assumes access to a large language model through a prompt-based text-generation API.Source: §3 · p. 3
  • The retriever selects a subset of historical documents as few-shot examples, and the LLM generates a target text of up to 300 words reflecting the user’s style, knowledge, and values.Source: §3 · p. 3

4 Proposed Approach

PEARL personalizes LLM writing assistance through an offline retriever trained to select historical user documents that help generation reflect user preferences. Its approach selects useful training requests and documents, then calibrates retriever scores to generation benefit.

4 Proposed Approach

  • PEARL trains an offline retriever to select historical documents for personalized LLM generation, using data selection and a scale-calibrating objective to track generation benefit.Source: §4 · pp. 3–4

4.1 Training Data Setup

  • An auxiliary model scores historical documents by the target’s likelihood gain over using the request alone, identifying requests and documents that benefit personalization.The score is highest when the request alone is insufficient and the candidate document is particularly beneficial; the auxiliary model is assumed smaller than the deployed LLM.Source: §4.1 · p. 4

4.2 Training Data Selection

  • Training retains the top two-thirds of request-target pairs in practice, then selects each retained request’s top-scoring P documents as positives and samples negatives randomly.Requests may be excluded when histories lack suitable documents or requests are underspecified.Source: §4.2 · p. 4

4.3 Retriever Optimization

  • A pretrained-LM cross-encoder distills auxiliary-model scores; calibration adds an anchor example, using the median positive score to spread predictions and better reflect document utility.The anchor target score is tunable and its logit is set to zero.Source: §4.3 · pp. 4–5

4.4 System Details

  • At inference, the retriever selects top-k historical texts, which are formatted into a prompt for the LLM to generate a personalized target.Source: §4.4 · p. 5

5 Experiments

PEARL was evaluated on workplace posts and Reddit comments using multiple measures of personalization. It generally matched or outperformed baselines, and its calibrated scores were also used to select generations for revision.

5.1 Experimental Setup

  • The experiments cover personalized post writing on WORKSM and comment writing on AITA, using intrinsic, extrinsic, and personalized LLM-judge evaluations.WORKSM uses reference-text similarity measures; AITA also uses stance classification to evaluate whether generated comments match users’ values.Source: §5.1 · pp. 5–6

5.2 Generation Evaluation

  • PEARL matched or outperformed the best baselines across datasets and LLMs, beating all baselines by 1.5–5 Macro F1 points on AITA.The best baseline varied by dataset and inference LLM.Source: §5.2 · p. 7, Table 1 · p. 7
  • PEARL had a greater win-rate than BM25 in 3 of 4 LLM-judge settings, with lower agreement in the exception.Agreement was α = 0.41–0.56 in the three settings and α = 0.28 for WORKSM with gpt-35-turbo.Source: Table 2 · p. 7, §5.2 · p. 8

5.3 Selective Revision with PEARL

  • Selective revision improved or retained WORKSM downstream performance by 2–4% with PEARL and BM25, while BM25-based revision dropped on AITA.The reported WORKSM outcomes use downstream performance metrics.Source: §5.3 · p. 8, Table 3 · p. 8
  • PEARL selected 75.8% of WORKSM instances and 77.9% of AITA instances for editing, while revising every output consistently degraded performance.Macro F1 does not measure style changes that may occur during revision.Source: §5.3 · p. 8

6 Conclusion

The paper presents PEARL, a writing assistant personalized with generation-calibrated retrieval, and evaluates its performance on social-media communication tasks.

6 Conclusion

  • PEARL personalizes an LLM writing assistant using generation-calibrated retrievers.The retriever is trained with selected data and a scale-calibrated objective.Source: §6 · p. 9
  • Evaluations on social-media datasets show PEARL’s effectiveness compared with baselines and ablated models.Source: §6 · p. 9
  • The retriever’s calibration supports using it as a performance predictor to identify outputs that may benefit from LLM revision.Source: §6 · p. 9

7 Ethical and broader impact

The paper discusses potential harms from more personalized generation and notes that longer-term effects on communication remain uncertain. It also reports implementation details and how WORKSM evaluation requests were produced.

7 Ethical and broader impact

  • Personalized generation may exacerbate phishing, tailored misinformation, impersonation, fake profiles, and plagiarism; the effectiveness of proposed safeguards remains under investigation.Source: §7 · p. 9
  • Potential longer-term effects include shifts in language use, self-presentation, social evaluation, and feedback loops, while understanding of AI-mediated communication is still developing.Source: §7 · p. 9

A Model Details

  • PEARL uses a 110M-parameter MPNET retriever and two API-based generation models; FLANT5-XL supplies token likelihoods for training-data construction.The implementation uses three retrieved examples for WORKSM and four for AITA.Source: A Model Details · pp. 13–14

B.1 Evaluation Requests in WORKSM

  • For WORKSM, two authors manually wrote requests for 163 evaluation target posts; external request writers were not recruited because the data were private and highly regulated.Source: §B.1 · p. 14

B.2 Training Requests in WORKSM

The appendix describes synthetic WORKSM training requests, benchmark baselines and judge prompts, then provides additional results including a qualitative AITA example of PEARL personalization.

B.2 Training Requests in WORKSM

  • WORKSM synthetic requests are built by extracting salient post aspects and concatenating them; the aspects include an overview, proper nouns, contact details, links, and specialized knowledge or anecdotes.GPT-4 generates an explanation and salient aspects, but the explanation is not used elsewhere.Source: §B.2 · pp. 14–15

B.3 Baselines

  • Baselines include request-only and fixed-example non-personalized prompting, plus personalized retrieval methods such as BM25, MPNET-1B, and QL-FT5.The appendix also describes RelevanceCE, a cross-encoder trained for request-document relevance rather than personalized generation.Source: §B.3 · p. 15

B.4 Judge LLM prompts

  • For GPT-4o judge evaluations, the appendix presents prompts used to assess whether a generated post aligns with a user-authored reference.Source: §B.4 · p. 15

C Additional Results

C.1 Qualitative example from AITA

  • In the AITA example, PEARL retrieves a historical comment reflecting the user's tone and values and generates a comment described as highly similar to the gold comment.The example concerns sharing medical information and highlights phrases about individual liberties, self-care, and mental health.Source: Figure 4 · p. 16, §C.1 · p. 16

C.2 Ablating Training Strategies

Ablations indicate that scale calibration and selecting requests that benefit from personalization both matter, although the observed calibration effect varies by dataset and LLM.

C.2 Ablating Training Strategies

  • The ablations compare alternate fretr training strategies while keeping inference steps identical for gpt-35-turbo and davinci-003.Source: §C.2 · p. 16
  • Removing scale calibration degrades AITA performance and is comparable on WORKSM with gpt-35-turbo, while scale calibration consistently improves performance with davinci-003.The authors also report that scale calibration consistently improves correlation between retriever scores and task performance.Source: §C.2 · p. 16
  • Distilling paux(tu|du, qu) while assuming all requests benefit from personalization markedly lowers performance compared with PEARL's training strategy.The alternative selects documents that benefit personalization but does not select requests for which personalization is beneficial.Source: §C.2 · pp. 16–17

C.3 Calibration Evaluation

PEARL's retrieval scores are more correlated with downstream generation performance on both datasets, and removing scale calibration weakens calibration.

C.3 Calibration Evaluation

  • For WORKSM, calibration is measured by Pearson r between the top-1 document score and R2; for AITA, requests are binned by score before correlating bin starts with average Macro-F1.Source: §C.3 · p. 17
  • PEARL scores correlate better with downstream performance on WORKSM and AITA, while removing the scale-calibrated objective underperforms PEARL.Table 6 reports correlation between the top-1 document score and R2 for WORKSM or Macro-F1 for AITA.Source: §C.3 · p. 17, Table 6 · p. 17
  • BM25 and MPNET-1B are better calibrated with downstream performance than likelihood-based QL-FT5.Source: §C.3 · p. 17

C.4 Selective Revision with PEARL – Extended Results

Selective revision benefits medium-length requests and users with few historical posts in WORKSM, but the reported procedure has limits in model coverage and comparison.

C.4 Selective Revision with PEARL – Extended Results

  • Selective revision benefits medium-length requests and users with few historical posts in WORKSM.The authors hypothesize that very short requests may need more user input and very long requests may be harder to follow; users with larger profiles see a performance drop.Source: §C.4 · p. 18
  • The authors suggest that users with larger profiles may benefit from better calibration, since their revision performance drops.Source: §C.4 · p. 18
  • Selective-revision metrics in Table 3 are not directly comparable with Tables 1, 4, and 5 because they use different LLM runs and exclude 50 WORKSM and 200 AITA dev instances.The procedure for learning a threshold for selective revision found no davinci-003 threshold that improved dev-set performance.Source: §C.4 · p. 18

D Extended Related Work

Related work places PEARL among approaches to personalized text generation and writing assistance. It contrasts PEARL’s retrieval-based choices with work on prompt selection, context robustness, and calibrated ranking.

Dynamic prompts for LLMs

Prompt robustness in LLMs

  • Prior robustness methods filter retrieved contexts, fine-tune LLMs against irrelevant contexts, or modify decoding; PEARL instead assesses context quality with scale-calibrated retrieval scores.The cited filtering approaches use entity frequency or query entailment.Source: D Extended Related Work · p. 18

LLM chaining

Calibrated retrievers

  • PEARL applies scale calibration to personalized writing assistance, minimally modifying retriever training without extensive architecture changes or added inference costs.Unlike probabilistic uncertainty estimation, this approach does not require extensive training changes.Source: D Extended Related Work · p. 19

Writing assistants

  • Creative-writing assistants target diverse generations and long-range coherence rather than personalization; alternative interaction modes may require additional work to extend PEARL.Prior systems also use completion, infilling, tag-based control, and instruction-guided generation.Source: D Extended Related Work · p. 19

Personalized text generation

  • Author personalization aims to mimic an author’s style, interests, and values, whereas reader personalization targets engaging, relevant text and remains understudied.Reader personalization has also been explored for reviews, recipes, news headlines, dialogue, text simplification, and lay summaries.Source: D Extended Related Work · pp. 19–20

E Limitations

The authors identify limitations tied to model choice, evaluation design, and method design.

Choice of LLMs

  • Experiments use two closed API LLMs; effectiveness with other LLMs remains unestablished, while larger open models for faux add training-data costs.Using smaller models for the final LLM may reduce generation performance, and the tradeoff remains for future work.Source: E Limitations · p. 20

Evaluation setup

  • Private WORKSM data limits reproducibility, WORKSM and AITA cover limited scenarios, and online evaluation is needed to establish value across broader uses.The paper also describes personalization evaluation as an emerging, under-explored area.Source: E Limitations · p. 20

Method design

  • The training-data selection method may bias performance toward some users or requests, while expressive cross-encoders and large LLMs may constrain interactive latency.The authors leave bias examination and faster retrieval and generation models for future work.Source: E Limitations · p. 20
Loading 2311.09180v2…