PaperlayerView exact revision on arXiv

Source-linked AI summary

LongLaMP: A Benchmark for Personalized Long-form Text Generation

Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, Chien Van Nguyen, Thien Huu Nguyen, Hamed Zamani

arXiv:2407.11016v3cs.CLcs.LG

TL;DR

Personalized generation research has focused mainly on short text, despite the prevalence of applications that require longer outputs. LongLaMP introduces a four-task benchmark and evaluates retrieval-augmented personalization across tasks. The reported results show improvements over non-personalized baselines and highlight the importance of personalization for long-text generation.

  • Problem

    Personalized generation research has focused on short text, while long outputs pose challenges such as maintaining style and coherence and avoiding topic drift.

    Source: §1 · pp. 1–2

  • Method

    LongLaMP benchmarks four personalized long-text tasks and evaluates retrieval-augmented generation that incorporates retrieved user data into LLM prompts.

    Source: §1 · p. 2

  • Results

    Personalization is reported to improve results across a wide variety of long-text tasks; the framework improves various metrics by 5.7% to 128% over non-personalized baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark provides an extensible, publicly available framework for evaluating and developing personalized long-text generation techniques.

    Source: §1 · p. 2

    The standard evaluation metrics do not fully capture the complexities of long-text generation and personalization.

    Source: §7 · p. 9

Abstract

from arXiv · show

Long-text generation is seemingly ubiquitous in real-world applications of large language models such as generating an email or writing a review. Despite the fundamental importance and prevalence of long-text generation in many practical applications, existing work on personalized generation has focused on the generation of very short text. To overcome these limitations, we study the problem of personalized long-text generation, that is, generating long-text that is personalized for a specific user while being practically useful for the vast majority of real-world applications that naturally require the generation of longer text. In this work, we demonstrate the importance of user-specific personalization for long-text generation tasks and develop the Long-text Language Model Personalization (LongLaMP) Benchmark. LongLaMP provides a comprehensive and diverse evaluation framework for personalized long-text generation. Extensive experiments on LongLaMP for zero-shot and fine-tuned language tasks demonstrate the effectiveness of the proposed benchmark and its utility for developing and evaluating techniques for personalized long-text generation across a wide variety of long-text generation tasks. The results highlight the importance of personalization across a wide variety of long-text generation tasks. Finally, we release the benchmark for others to use for this important problem.

1 Introduction

LongLaMP addresses personalized long-text generation, a relatively unexplored problem despite the importance of long outputs in applications such as email, reviews, and content creation. It introduces a four-task benchmark and evaluates retrieval-augmented personalization, reporting gains over non-personalized baselines.

1 Introduction

  • Personalized long-text generation remains relatively unexplored compared with short-text personalization, although long outputs serve applications such as email, reviews, and content creation.The paper identifies maintaining style, coherence, and consistency while preventing topic drift as challenges for long outputs.Source: §1 · pp. 1–2
  • LongLaMP evaluates personalized long-text generation across email, abstract, review, and topic-writing tasks, with two settings per task.The user setting holds out new users, while the temporal setting evaluates adaptation for previously seen users over chronological splits.Source: §1 · p. 2
  • 5.7% to 128% improvement across various metrics over non-personalized baselines is reported for the proposed framework.The findings also highlight personalization’s importance for most applications requiring long-text generation.Source: §1 · p. 2
  • LongLaMP investigates retrieval-augmented generation, using retrieved user data in LLM prompts to personalize long-text while maintaining computational efficiency.The paper contrasts this with per-user fine-tuning and latent-space approaches, which it describes as costly, poorly scalable, and potentially risky for privacy.Source: §1 · p. 2
  • The paper formalizes personalized long-text generation as a problem relevant to outputs such as abstracts, product reviews, and emails.It frames personalization as important for tasks that require generating lengthy text.Source: §1 · p. 2
  • The benchmark is designed as an extensible, publicly available environment for adding models, tasks, and evaluation metrics.The paper presents it as a comprehensive framework for evaluating personalized long-text generation.Source: §1 · p. 2

2 LongLaMP Benchmark

LongLaMP represents personalized generation with an input, target output, and user profile, and covers four domains with distinct writing and content demands. Its user and temporal splits support evaluation of new-user generation and adaptation for known users.

2.1 Problem Formulation

  • Long-text generation produces cohesive, contextually relevant outputs spanning multiple sentences, paragraphs, or pages; personalization conditions them on the prompt and user profile.Each benchmark entry contains an input prompt, target output, and profile aggregating historical or static user data.Source: §2.1 · p. 2

2.2 The LongLaMP Benchmark

  • The four tasks cover personalized email completion, abstract generation, review writing, and topic writing, using prior user-authored material as profile data.The tasks differ in audience, purpose, style, content type, credibility requirements, length constraints, and structure.Source: §2.2 · p. 3
  • The email task uses a private email collection, and its illustrative Figure 3 example is synthetic to preserve confidentiality.Email input comprises a subject and partial content; the profile contains previously authored subject-email pairs.Source: §2.2 · p. 3

2.3 Dataset Splits and Evaluation

  • The user setting has non-overlapping training, validation, and test users, while the temporal setting evaluates linguistic and style changes for known users.For most tasks, the temporal split assigns recent documents to test, validation, and training chronologically, with earlier documents forming profiles; topic writing uses random assignments because temporal information is unavailable.Source: §2.3 · pp. 3–4
  • Evaluation uses ROUGE-1, ROUGE-L, and METEOR to compare generated text with reference outputs.The paper notes that these metrics have limitations for capturing long-text generation and personalization complexity.Source: §2.3 · p. 4

3 Framework

The framework retrieves relevant material from a user profile and adds it to an LLM prompt for personalized long-text generation. It tests BM25 and Contriever retrieval, then evaluates generated outputs against user-written targets.

3 Framework

  • The framework uses a query function, retriever, personalized-prompt function, and LLM to condition generation on selected user-profile documents.The query is formed from the user input; the retriever returns the top-k similar profile documents, which are combined with the input to form the personalized prompt.Source: §3 · p. 4
  • The retrieval comparison uses BM25 term matching and the pre-trained dense retriever Contriever.The query function uses non-templated parts of the task input, and the prompt concatenates the task instruction, input, and retrieved profile information.Source: §3 · p. 4
  • Models are fine-tuned and evaluated on personalized prompts paired with user-written target texts, with generated text compared against the target using evaluation metrics.The pairs are denoted (x̄_i, y_i), where x̄_i is the final personalized prompt and y_i is the ground-truth text.Source: §3 · p. 4

4 Experiments

Experiments evaluate personalized long-text generation across zero-shot and fine-tuned settings, using multiple tasks, models, and retrieved-profile counts.

4.2.1 Personalized Email Completion

  • Email completion’s user setting gained 169.40% in ROUGE-L with LLaMA2; temporal-setting gains exceeded 50% across all metrics.Contriever with four retrieved profiles performed best in both settings.Source: Table 2 · p. 5, Table 3 · p. 6, §4.2.1 · p. 7

4.2.2 Personalized Abstract Generation

  • Abstract generation favored personalization in both settings; user-setting ROUGE-L gains reached 12.09% across models, while temporal ROUGE-1 and ROUGE-L gains were 4.52% and 4.91%.Contriever with four profiles performed best in both settings.Source: Table 2 · p. 5, Table 3 · p. 6, §4.2.2 · p. 7

4.2.3 Personalized Review Writing

  • Review writing favored personalization across user-setting metrics; temporal results also improved except for a slight LLaMA2 ROUGE-L decline.METEOR gains were 10.12% in the user setting and 7.79% in the temporal setting.Source: Table 2 · p. 5, Table 3 · p. 6, §4.2.3 · p. 7

4.2.4 Personalized Topic Writing

  • Topic writing improved on all user-setting metrics with LLaMA2 and in the temporal setting; GPT-3.5 had slight ROUGE-L declines in both settings.METEOR gains were 28.92% in the user setting and 25.61% in the temporal setting.Source: Table 2 · p. 5, Table 3 · p. 6, §4.2.4 · p. 7

4.3 Fine-tuning Results

  • Fine-tuning improved personalized results over non-personalized results across both settings.The experiments used FLAN-T5-base; reported fine-tuning results cover all four tasks.Source: Table 4 · p. 6, §4.3 · p. 7
  • Fine-tuned email improved in both settings, with ROUGE-L gains of 101.8% for user-based separation and 86.2% for temporal separation.Contriever generally led on ROUGE-L, while BM25 performed better on ROUGE-1.Source: §4.3 · p. 7
  • Personalization improved review results across all metrics using one profile and topic results in both settings using two profiles; review METEOR gains were 6.89% and 9.89%, while topic peaks were 7.34% METEOR and 10.43% ROUGE-1.Review gains are for user and temporal settings, respectively; topic peaks are METEOR in the user setting and ROUGE-1 in the temporal setting.Source: §4.3 · pp. 7–8

4.4 Hyperparameter Sensitivity

  • Profile-count effects varied by task: more profiles helped abstract quality with diminishing returns, while ROUGE scores in LongLaMP-4 declined after k=1.The user setting in LongLaMP-3 benefited from more profiles, while the temporal setting plateaued or declined after a certain count.Source: Figure 2 · p. 5, §4.4 · p. 8

5 Related Work

Prior personalization work spans several domains and methods, but has largely targeted short-text generation. LongLaMP addresses a distinct setting: generating long text from limited context without a fixed starting point.

5 Related Work

6 Conclusion

LongLaMP introduces a benchmark for personalized long-text generation and evaluates retrieval-augmented generation across tasks and settings. The reported results show average ROUGE gains across tasks and underscore personalization’s importance for long-text applications.

6 Conclusion

  • 30.21% average improvement in ROUGE-1 and 47.5 % in ROUGE-L are reported across all tasks.Source: §6 · p. 8
  • LongLaMP is presented as the first benchmark for personalized long-text generation, with experiments spanning diverse tasks, settings, language models, and fine-tuned and zero-shot use.The study also varies retrieval methods and the number of user-profile documents retrieved.Source: §6 · p. 8
  • The authors conclude that personalization matters across most long-text generation applications and may help tailor generated language to individuals.Source: §6 · pp. 8–9

7 Limitations

The paper notes limits in its evaluation metrics and does not address privacy-risk mitigation for personalization. It also uses a private email dataset as a distinctive benchmark component.

7 Limitations

  • Standard evaluation metrics do not fully capture long-text generation and personalization, though they offer a consistent comparison framework and indicators of textual alignment and overlap.Source: §7 · p. 9
  • The paper does not address resolving privacy concerns from fine-tuning on user data, including memorization exposure and inference attacks.Source: §7 · p. 9
  • The benchmark includes the private Avocado email dataset to introduce a novel element and counter potential pre-training exposure.Source: §7 · p. 9

8 Ethics Statement

The ethics statement describes confidentiality controls for the Avocado Research Email collection and emphasizes privacy and responsible AI use. The collection is restricted to authorized access and inaccessible to the public.

8 Ethics Statement

  • The Avocado Research Email collection is managed under a confidentiality agreement with limited authorized access and is inaccessible to the public.Source: §8 · p. 9
  • The authors state that their research follows ethical standards focused on data privacy and responsible AI use.Source: §8 · p. 9
  • The ethics statement frames privacy protection and responsible technology use as commitments in advancing AI.Source: §8 · p. 9

A Benchmark Details

LongLaMP organizes personalized long-text generation around email completion, abstract generation, review writing, and topic writing, with task-specific data, profiles, and prompts. Its benchmark covers distinct writing contexts and user-history settings.

A Benchmark Details

  • LongLaMP covers email completion, scientific abstract generation, product-review writing, and Reddit topic writing, with task characteristics varying in audience, purpose, style, content, credibility, length, and structure.The benchmark section also points to task statistics and a comparison of task novelty.Source: §A.1 · p. 11, §A.2 · p. 12, §A.3 · p. 13

A.1 Personalized Email Completion

  • Email completion uses a subject and partial email as input, prior subject-email pairs as the profile, and Avocado emails filtered for substantive length and sender history.The user setting separates users across splits; the temporal setting sorts emails chronologically and uses remaining emails to enrich profiles.Source: §A.1 · pp. 11–12

A.2 Personalized Abstract Generation

  • Abstract generation conditions a scientific abstract on a paper title and selected keywords, while the author profile contains the author’s papers and abstracts.The dataset uses authors with at least 70 publications; its user setting has no author overlap, while its temporal setting assigns each author’s three most recent papers across train, validation, and test.Source: §A.2 · pp. 12–13

A.3 Personalized Review Writing

  • Review writing generates a review from the product description, rating, and experience summary, using the reviewer’s previous reviews and their associated product information as profile context.Filtering requires reviews of at least 120 words and users with at least 50 contributions; the temporal split assigns the three most recent reviews to test, validation, and training.Source: §A.3 · pp. 13, 15

A.4 Personalized Topic Writing

  • Topic writing generates Reddit post content from a user-created summary and a profile of that user’s previous summary-content pairs.The filtering requires content of at least 50 words and users with at least 16 posts; because the dataset lacks temporal data, its temporal setting assigns posts randomly to splits.Source: §A.4 · pp. 16–17

B Prompts Used for Adding User Profile to the Language Model’s Input

  • Prompt construction creates a per-profile-entry prompt and combines those prompts with the task input; abstract profiles use only the first 750 words of each abstract, and Llama review profiles use the first 100 review words.The abstract truncation was reported to perform better than using entire abstracts, possibly because longer profile entries added noise.Source: B Prompts Used for Adding User Profile… · pp. 17–18

C Experimental Setup

The experiments use zero-shot GPT-3.5 and Llama 7B, fine-tuned FlanT5-base, and task-specific prompt templates.

Experimental settings

  • GPT-3.5 zero-shot generation uses a 2048-token response limit and temperature 0.6; Llama 7B uses a 4096-token limit, temperature 0.8, and top_p 0.95.Figure 10 overviews the templates used to create each benchmark task.Source: C Experimental Setup · p. 18, Figure 10 · p. 18

Fine-tuning

  • FlanT5-base is fine-tuned for 20 epochs with AdamW, learning rate 5 x 10^-5, batch size 1, and gradient accumulation of 4.Its maximum output length is 1024 tokens.Source: C Experimental Setup · p. 18

Compute

D Performance on Validation & Gains

Validation results report gains for personalized generation across model settings, while the additional-baseline experiments include a recency retriever for temporal evaluation.

D Performance on Validation & Gains

E.1 Additional Baselines and Retriever

  • The recency retriever selects a user’s most recent profile entries as a temporal-setting baseline, on the premise that they may better reflect current interests, language, and context.Additional baseline tables report user- and temporal-setting results for GPT-3.5 and LLaMA2-7B.Source: §E.1 · pp. 21, 23, Table 12 · p. 21, Table 13 · p. 21

E.2 Summarization Experiments

Additional zero-shot experiments test summarization-only and stylistic-extraction approaches to using retrieved profiles. Both report improvement over generation without retrieved personalized documents, with summarization also improving review and topic generation relative to the earlier framework.

Experimental Approaches

  • Summarization-only prompting summarizes retrieved documents before combining the summaries with the original input; stylistic extraction derives user linguistic properties from retrieved texts for prompting.The summaries correspond to retrieved profile entries, and the stylistic approach generates linguistic properties for each user.Source: §E.2 · p. 23

Summarization Experiments

  • For summarization-only, results show a significant improvement over the baseline with no personalized documents retrieved from the user's profile, and significant improvements over the Section 3 framework on review and topic generation.Table 19 reports zero-shot personalized results using GPT-3.5 for user and temporal settings.Source: §E.2 · p. 23, Table 19 · p. 24

Stylistic Extraction Experiments

  • Style experiments report significant improvement over the baseline without retrieved user-profile documents across datasets.Table 20 presents GPT-3.5 zero-shot results using BM25 and Contriever with top k values of 1, 2, and 4.Source: §E.2 · p. 23, Table 20 · p. 24
Loading 2407.11016v3…