PaperlayerView exact revision on arXiv

Source-linked AI summary

NextQuill: Causal Preference Modeling for Enhancing LLM Personalization

Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng, Tat-Seng Chua

arXiv:2506.02368v3cs.IR

TL;DR

Personalization methods must account for users’ differing preferences, but existing approaches can fail to distinguish preference-bearing content from other prediction and training signals. NextQuill estimates causal preference effects in model predictions and ground-truth responses, then uses them to guide alignment. Experiments report improved generation quality and overall personalization performance across multiple personalization domains.

  • Problem

    General-purpose LLMs often neglect diverse user preferences, while personalization methods may not distinguish preference-bearing components of predictions and training data.

    Source: §1 · pp. 1–2

  • Method

    NextQuill estimates causal preference effects on model predictions and ground-truth responses, aligning those effects and emphasizing preference-driven ground-truth tokens during fitting.

    Source: §1 · pp. 2–3

  • Results

    Experiments across multiple personalization domains show significant improvements in generation quality and overall personalization performance.

    Source: §1 · p. 3

  • Takeaways & Limitations

    NextQuill provides targeted, interpretable supervision aligned with user preferences for personalized generation.

    Source: §6 · p. 10

    The formulation assumes each user has historical text that reflects their preferences for a query.

    Source: §2 · p. 3

Abstract

from arXiv · show

Personalizing large language models (LLMs) for individual users has become increasingly important as they are progressively integrated into real-world applications to support users' daily lives. However, existing personalization approaches often fail to distinguish which components of model predictions and training data truly reflect user preferences, leading to superficial personalization alignment. In this paper, we introduce NextQuill, a novel LLM personalization alignment framework grounded in causal preference modeling. We approach personalization from a causal perspective, treating both model predictions and ground-truth data generation as outcomes influenced by user preferences, along with other factors. We define the true preference effect as the causal impact of user history (which reflects preferences) on each token prediction or data generation instance, estimated through causal intervention techniques. Building on this insight, NextQuill introduces two complementary alignment strategies: (1) aligning model-internal causal preference effects on predictions with those reflected in ground-truth data, rather than indiscriminately fitting predictions, and (2) focusing on fitting preference-bearing tokens identified via ground-truth data preference effects, rather than treating all tokens uniformly. By integrating these strategies, NextQuill shifts the alignment process toward learning from causal preference effects, facilitating more effective and personalized adaptation. Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality, offering a principled, causal foundation for LLM personalization. Our codes are available on https://github.com/juntaoyou/NextQuill.

1 INTRODUCTION

Existing personalization methods can miss which predictions and response tokens actually reflect user preferences. NextQuill models preference effects on both sides of the response process and uses them to guide alignment; experiments report improved personalization performance.

1 INTRODUCTION

  • Existing tuning methods align all model predictions and ground-truth tokens uniformly, overlooking which components reflect user preferences.The paper argues this superficial treatment limits alignment quality.Source: §1 · p. 2
  • Causal Preference Modeling estimates preference effects separately for model predictions and ground-truth responses, distinguishing them from effects of context and other factors.The model-side effect is attributed to user history; the data-side effect measures how user characteristics drive ground-truth tokens.Source: §1 · p. 2
  • NextQuill aligns model-internal preference effects with those in ground truth and gives greater fitting attention to preference-driven ground-truth tokens.This replaces uniform alignment of predictions and uniform treatment of target tokens with two complementary strategies.Source: §1 · p. 3
  • Experiments across multiple personalization domains show significant improvements in generation quality and overall personalization performance.The paper presents these experiments as evidence for NextQuill’s effectiveness.Source: §1 · p. 3

2 PROBLEM FORMULATION

The paper formulates personalized text generation as using a user’s query and preference-reflecting history to produce a response tailored to that user. Its goals are to improve preference modeling and align generated responses with personalized information in the expected response.

2 PROBLEM FORMULATION

  • Personalized text generation aims to use user-specific information to tailor an LLM’s response to an individual user.The formulation represents the query as x and the personalized response as ŷ = Mθ(x, h).Source: §2 · p. 3
  • A user’s query specifies the item or objective of interest, while their historical text is assumed to reflect preferences for that query.The ground-truth response is text written by the user, and the query, history, and response form a sample.Source: §2 · p. 3
  • The formulation seeks both to improve the model’s use of user history and to align its generated response with personalized information in the expected response.Source: §2 · p. 3

3 METHODOLOGY

The methodology frames personalization as causal preference modeling across model predictions and user-written ground truth. NextQuill uses preference effects on both sides to guide alignment during training.

3.1 CAUSAL PREFERENCE MODELING

  • Causal graphs analyze model prediction and user ground-truth generation, with preference-related causal effects serving as the basis for preference modeling.Source: §3.1 · p. 3

3.1.1 MODEL-SIDE CAUSAL PREFERENCE MODELING

  • Model-side causal preference effects isolate how user history, rather than other context, influences each predicted token relative to a reference with no history.The effect is formulated as the difference between interventional token probabilities with history set to h and to a reference value.Source: §3.1.1 · p. 4

3.1.2 DATA-SIDE CAUSAL PREFERENCE MODELING

  • Data-side causal preference effects isolate how user characteristics influence ground-truth responses; tokens with stronger effects more strongly reflect user preferences.Source: §3.1.2 · p. 5

3.2 PERSONALIZATION ALIGNMENT BASED ON CAUSAL PREFERENCE MODELING

  • NextQuill aligns model-side and data-side causal preference effects and gives preference-driven ground-truth tokens greater weight during training.The method combines these strategies in an SFT-like fine-tuning objective.Source: §3.2 · p. 5, §3.2.2 · p. 6

3.2.1 PREFERENCE-DRIVEN COMPONENT IDENTIFICATION

  • The method identifies preference-driven prediction components using effects of history and classifies ground-truth tokens using estimated effects and a threshold.The data-side estimate uses a model trained on D and represents user characteristics with historical data; token weights distinguish preference-driven from other tokens.Source: §3.2.1 · pp. 5–6

3.2.2 TUNING STRATEGY

  • The tuning strategy combines preference-weighted next-token fitting with a loss aligning model-side and ground-truth preference effects.The combined objective fine-tunes the model similarly to SFT.Source: §3.2.2 · p. 6

4 EXPERIMENTS

Across three product-review datasets and a topic-writing dataset, NextQuill outperforms all baselines across datasets and metrics. Ablations, scaling results, hyperparameter analysis, and token-level comparisons further support its causal preference alignment design.

4.1 EXPERIMENTAL SETTINGS

  • Experiments cover three Amazon review datasets and personalized long-form Reddit writing, with user histories capped at 4096 input tokens.The study uses Qwen2.5-3B as the backbone and evaluates lexical, semantic, and LLM-judged generation quality.Source: §4.1 · pp. 7–8

4.2 OVERALL PERFORMANCE

  • NextQuill consistently achieves the best performance across all datasets and evaluation metrics, outperforming a wide range of baselines.Methods incorporating user information outperform the unpersonalized Qwen baseline, while PEFT-based methods generally surpass retrieval-only approaches.Source: §4.2 · p. 8

4.3 IN-DEPTH ANALYSIS

  • DCE, MCE, and their alignment each improve over SFT, while the full NextQuill model achieves the best overall performance.The ablation compares causal components and token-level preference weighting.Source: §4.3 · pp. 8–9, Table 2 · p. 8
  • NextQuill consistently outperforms ContextSFT across datasets and metrics at 1.5B, 3B, and 7B, with a widening performance gap as model size increases.The comparison covers three Qwen2.5 backbone sizes.Source: §4.3 · p. 9, Table 3 · p. 9
  • On CD Review, α = 0.05 consistently yields strong performance across multiple metrics and serves as a robust setting across datasets.The analysis isolates α by disabling token weighting; λ is analyzed separately in Appendix C.5.Source: §4.3 · p. 9, Figure 2 · p. 9

4.4 WORD-LEVEL PREFERENCE ANALYSIS.

  • In samples of 50 training instances per dataset, NextQuill produces larger token-level logit differences with versus without user history than ContextSFT.The authors interpret these differences as stronger preference representations.Source: §4.4 · p. 9, Figure 3 · p. 10

5 RELATED WORK

Prior LLM personalization work uses retrieval-based prompting or fine-tuning with user history, while causal methods have been applied to other LLM problems. The paper positions its contribution as a unified causal framework for modeling preference effects on both model and data sides.

LLM Personalization

  • Personalization methods mainly retrieve user-specific context for prompting or fine-tune model parameters using user histories.Related approaches include profiles, personalized embeddings, parameter-efficient methods, and methods modeling inter-user differences.Source: §5 · p. 10
  • Most existing personalization methods treat tokens uniformly, without identifying which input or output parts reflect user intent.The paper contrasts this with token-level causal-effect attribution for supervision over model-internal representations and output behavior.Source: §5 · p. 10

Causal Inference for LLMs

  • Although causal methods have been used for other LLM tasks, the authors say personalization has not been systematically addressed and present a unified model- and data-side framework.They describe their work as the first to model user preference effects from both sides for personalized text generation.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents NextQuill as a token-level causal preference framework that aligns model-side and data-side signals. Experiments across diverse benchmarks confirm improved personalized generation performance.

6 CONCLUSION

  • NextQuill attributes and aligns user-specific signals through token-level causal effect estimation.It models both model-side and data-side preference pathways.Source: §6 · p. 10
  • NextQuill aligns model-internal causal effects with preference-bearing supervision rather than uniformly fitting all predictions and target tokens.The approach is described as targeted, interpretable supervision for what to adapt and where to focus.Source: §6 · p. 10
  • Experiments across diverse benchmarks confirm improved personalized generation performance.Source: §6 · p. 10

ETHICS STATEMENT

The ethics statement describes use of public, anonymized user data and calls for stronger privacy safeguards in private-domain applications. The accompanying causal-graph discussion distinguishes model-side and data-side preference effects, while compute details report training overhead but no added inference computation.

ETHICS STATEMENT

  • The experiments use publicly disclosed, anonymized user data, and private-domain applications should include stronger privacy safeguards.Suggested safeguards include user control over data and local learning scenarios.Source: ETHICS STATEMENT · p. 11

A DETAILED CAUSAL GRAPHS AND CAUSAL PREFERENCE EFFECTS

  • The model-side effect isolates history’s contribution to predictions, while the data-side effect captures user attributes’ impact on ground-truth responses.The framework uses both effects to align how the model leverages history with how personalization appears in user data.Source: A DETAILED CAUSAL GRAPHS AND CAUSAL… · p. 16, Figure 4 · p. 16

B.1 COMPUTE RESOURCES

  • Causal effect estimation adds training overhead through factual and counterfactual passes, but NextQuill uses a single forward pass at inference without retrieval or dynamic adaptation.Training takes approximately 26 hours per dataset on four A100 GPUs.Source: §B.1 · p. 16

B.2 ABLATION VARIANT IMPLEMENTATION DETAILS

The ablation variants selectively add model-side causal preference effects, data-side effects, and their alignment. A random-token negative control tests whether the identified preference-driven tokens matter beyond weighting alone.

B.2 ABLATION VARIANT IMPLEMENTATION DETAILS

  • MCEOnly adds model-side causal preference effects but uses uniform weights in both loss terms, while Base Model (SFT) has no personalization mechanism.The SFT baseline disables MCE, DCE, and their alignment.Source: §B.2 · p. 16
  • Full NextQuill combines preference-weighted next-token loss with causal preference alignment; DCEOnly and MCE-DCE Alignment isolate parts of this combination.DCEOnly adds weighted fitting; MCE-DCE Alignment adds alignment while using uniform weights; Full NextQuill uses both objectives.Source: §B.2 · p. 17

C.1 NEGATIVE-CONTROL VALIDATION OF PREFERENCE-DRIVEN TOKEN IDENTIFICATION

  • Replacing causally identified preference-driven tokens with random tokens causes substantial performance drops across all datasets and metrics.Table 4 reports the negative-control comparison; the result argues against improvements arising from trivial weighting heuristics.Source: Table 4 · p. 17, §C.1 · p. 17

C.2 ROBUSTNESS TO POTENTIAL CONFOUNDERS: ITEM POPULARITY STRATIFICATION

The analyses test robustness to item popularity and DCE-estimator choices, then examine token clustering and the preference-token weighting parameter. NextQuill retains advantages across popularity groups, and increasing λ improves reported metrics.

C.2 ROBUSTNESS TO POTENTIAL CONFOUNDERS: ITEM POPULARITY STRATIFICATION

  • NextQuill consistently outperforms ContextSFT in both low- and high-popularity item groups.Models are trained and evaluated separately within the two popularity subsets.Source: Table 5 · p. 17, §C.2 · p. 18
  • Across Book Review, Movie Review, and CD Review, the 3B dynamic variant beats ContextSFT-3B; static weights from the 1.5B model also outperform the 1.5B dynamic model.The static-from-1.5B approach incurs extra precomputation cost, so dynamic DCE estimation is the default.Source: §C.2 · p. 18, Table 6 · p. 18

C.4 EXTENDED WORD-LEVEL PREFERENCE ANALYSIS.

  • K-means clustering of final-layer hidden states separates personalized, common, and auxiliary review tokens.The authors interpret the separation as evidence that the strategy organizes tokens by their relevance to personalization.Source: §C.4 · p. 18

C.5 HYPER-PARAMETER ANALYSIS OF λ

  • As λ increases across 0.7, 0.8, and 0.9, performance improves consistently across all evaluation metrics.The authors relate the trend to higher weighting of preference-bearing tokens.Source: §C.5 · p. 18

C.6 ANALYSIS OF MODEL PERFORMANCE ACROSS DIFFERENT USER HISTORY LENGTHS

The history-length analysis reports improved personalization with more user history and continued gains over ContextSFT when history is limited. The paper also reports that NextQuill leads the four-dataset LLM-as-a-judge evaluation and outlines its training procedure.

C.6 ANALYSIS OF MODEL PERFORMANCE ACROSS DIFFERENT USER HISTORY LENGTHS

  • Longer user histories improve ROUGE, METEOR, and BLEU scores, while NextQuill still outperforms ContextSFT with limited history.Table 7 compares performance at 1024 and 2048 history tokens; the supplied text does not give the individual scores.Source: §C.6 · pp. 18, 20, Table 7 · p. 20

C.7 COMPREHENSIVE RESULTS VIA LLM-AS-A-JUDGE EVALUATION

  • NextQuill achieves the best LLMScore performance across all four personalized text-generation datasets.The evaluation uses Qwen2.5-72B-Instruct to assess content relevance and stylistic alignment; Table 8 reports relative improvement over base Qwen.Source: Table 8 · p. 20, §C.7 · p. 21

C.8 DETAILED ALGORITHM DESCRIPTION

  • The training algorithm computes model-side and data-side causal effects per token, assigns preference weights, and jointly optimizes preference-weighted next-token and causal-alignment losses.The final parameters are updated by minimizing the combined losses; the accompanying description says inference adds no extra overhead.Source: §C.8 · p. 22

D A DETAILED DISCUSSION ON THE LIMITATIONS

NextQuill’s limitations include dependence on user-history quality, added training computation, and a single-session token-level scope. Its causal graph is based on domain knowledge, and unobserved confounders may require an extension.

D A DETAILED DISCUSSION ON THE LIMITATIONS

  • Sparse or low-quality user histories make it harder to distinguish true preferences from incidental patterns, especially in low-resource and cold-start settings.The authors suggest auxiliary signals such as demographic features or side information to improve robustness in sparse-data regimes.Source: D A DETAILED DISCUSSION ON THE… · p. 22
  • Evaluating factual and counterfactual predictions adds training overhead, but does not affect inference speed.The authors contrast this training-only overhead with baseline methods using repeated retrieval or dynamic generation-time adaptation.Source: D A DETAILED DISCUSSION ON THE… · p. 22
  • The method models token-level personalization within a single-session context, not longer-term user dynamics or cross-session behavior.The authors identify lifelong personalization, multi-turn dialogue, and recommendation-oriented generation as settings where those dynamics may matter.Source: D A DETAILED DISCUSSION ON THE… · p. 22

E CAUSAL GRAPH SELECTION

  • The causal graph is constructed from domain-level knowledge about model prediction and the user, item, and their interaction in response generation.The experimental results are described as indirectly supporting the plausibility of the data-side graph.Source: D A DETAILED DISCUSSION ON THE… · p. 22, E CAUSAL GRAPH SELECTION · pp. 22–23
  • The approach remains valid if unobserved confounders are absent; their presence may require an extension, which the authors leave to future work.The authors identify unknown confounders as beyond this work’s scope.Source: E CAUSAL GRAPH SELECTION · p. 23

F LLM USAGE

  • LLMs were used only to check grammar and improve phrasing, not for research ideation, experimental design, analysis, or technical content.The authors attribute the scientific contributions, results, and interpretations solely to themselves.Source: F LLM USAGE · p. 23

G CASE STUDY

Across Book, CD, and Movie Review examples, NextQuill’s generations align more strongly with users’ actual reviews than ContextSFT’s. Token-level probability heatmaps across the three datasets illustrate how DCE distinguishes preference-driven tokens from preference-agnostic content.

G CASE STUDY

  • The case studies compare NextQuill and ContextSFT on Book, CD, and Movie Review samples, with highlighted sentences marking alignment to the actual review.The comparisons appear in Figures 7, 8, and 9.Source: G CASE STUDY · p. 23, Figure 7 · p. 23, Figure 8 · p. 24
  • Across three review domains, NextQuill outputs show stronger alignment with users’ real preferences than ContextSFT, whose reviews tend to be generic or templated.The examples compare generations with users’ actual reviews and highlight aligned sentences.Source: G CASE STUDY · pp. 23–24, Figure 7 · p. 23, Figure 8 · p. 24
  • Across all three datasets, DCE heatmaps separate preference-driven tokens from contextually relevant but preference-agnostic content.The authors say these patterns validate the effectiveness of preference-driven token weighting.Source: Figure 10 · p. 25, Figure 11 · p. 25, G CASE STUDY · p. 26
Loading 2506.02368v3…