PaperlayerView exact revision on arXiv

Source-linked AI summary

When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs

Zhongxiang Sun, Yi Zhan, Chenglei Shen, Weijie Yu, Xiao Zhang, Ming He, Jun Xu

arXiv:2601.11000v1cs.CLcs.AI

TL;DR

Personalization can cause LLMs to answer factual questions in line with user history rather than objective truth, creating hallucinations through entanglement between personalized and factual representations. The paper introduces FPPS to mitigate these distortions and PFQABench to evaluate factual and personalized question answering jointly. Across models and personalization settings, FPPS improves factual accuracy while preserving personalized performance.

  • Problem

    Personalized LLMs may systematically distort factual reasoning by aligning answers with users’ prior statements, risking factual reliability and propagation of incorrect beliefs.

  • Method

    The paper proposes FPPS, a lightweight inference-time framework that detects personalization-sensitive factual distortions and selectively steers representations toward factual reasoning.

  • Results

    FPPS substantially improves factual accuracy while preserving performance on personalized queries across multiple models and personalization methods.

  • Takeaways & Limitations

    Personalization should be treated as a controlled signal constrained by factual consistency rather than as an unconditional bias in reasoning.

  • Takeaways & Limitations

    Experiments cover a limited set of open-weight backbones, and FPPS requires intermediate representations, making it inapplicable directly to closed-source API models.

Abstract

from arXiv · show

Personalized large language models (LLMs) adapt model behavior to individual users to enhance user satisfaction, yet personalization can inadvertently distort factual reasoning. We show that when personalized LLMs face factual queries, there exists a phenomenon where the model generates answers aligned with a user's prior history rather than the objective truth, resulting in personalization-induced hallucinations that degrade factual reliability and may propagate incorrect beliefs, due to representational entanglement between personalization and factual representations. To address this issue, we propose Factuality-Preserving Personalized Steering (FPPS), a lightweight inference-time approach that mitigates personalization-induced factual distortions while preserving personalized behavior. We further introduce PFQABench, the first benchmark designed to jointly evaluate factual and personalized question answering under personalization. Experiments across multiple LLM backbones and personalization methods show that FPPS substantially improves factual accuracy while maintaining personalized performance.

1 Introduction

Personalization improves user alignment but can distort factual reasoning when user history biases answers away from objective truth. The paper identifies representational entanglement as the mechanism and introduces FPPS and PFQABench to mitigate and evaluate this problem.

  • Personalized LLMs can answer factual queries according to users’ prior statements rather than objective truth, producing personalization-induced hallucinations.
  • These hallucinations can reinforce misconceptions and reduce the accuracy of knowledge acquired by users learning from personalized models.
  • Personalization introduces non-orthogonal preference directions that entangle with factual representations and shift activations toward factually incorrect regions.
  • FPPS is a lightweight inference-time framework that detects personalization-sensitive factual distortions and selectively restores factuality while preserving personalization benefits.
  • PFQABench jointly evaluates fact-driven and personalized questions within aligned realistic user sessions, enabling systematic analysis of personalization’s effects on factual reasoning.
  • FPPS consistently restores factual accuracy across evaluation settings without harming performance on personalized queries.

2 Related Work

Prior personalization research examines alignment, bias, and filter bubbles, while hallucination research generally treats errors as input-driven failures to suppress uniformly. This paper distinguishes personalization-induced hallucinations as a separate failure mode caused by entanglement between personalized signals and factual knowledge.

  • Personalized LLMs use prompting, lightweight adaptation, and preference optimization, with major assistants automatically extracting user traits and histories.
  • Existing studies report personalization-related bias, filter-bubble effects, output disparities, and explicit preference problems, but focus less on factual reasoning distortions.
  • Hallucination research commonly defines hallucinations as fluent but logically incorrect or factually ungrounded outputs and develops detection and mitigation techniques.
  • The paper identifies personalization-induced hallucinations as a distinct category arising when user profiles or long-term memories systematically bias factual reasoning.

3 Problem Formulation and Analyses

The paper formalizes personalization-induced hallucination as factual errors caused by user-specific information and analyzes how personalization affects factual knowledge learning. It then evaluates personalized versus standard LLM teachers in a controlled simulation, finding lower learned factual accuracy with personalization.

  • 3.1 Problem Definition: Personalized LLMs model responses as pθ(y | x, u), which can differ from the non-personalized distribution pθ(y | x).Here x is the query, u is user-specific information, and y is the generated response.
  • 3.1 Problem Definition: Personalization-induced hallucination is defined as personalization causing the model to output a factually incorrect answer.
  • 3.1 Problem Definition: Personalization is modeled as an implicit hidden-state shift, with v_u representing the net effect of personalization on internal activations.The shift is abstract rather than a fixed or explicit direction.
  • 3.1 Problem Definition: Because the personalization representation is generally non-orthogonal to the factual representation, personalization perturbs the factual subspace.
  • 3.1 Problem Definition: The proposed mechanism must detect factual–personalization entanglement and steer hidden representations toward the factual subspace when needed, while preserving personalization performance.The objective is to minimize personalization-induced factual errors while retaining personalized behavior.
  • 3.2 Effects of Personalized LLMs on Factual Knowledge Learning: The controlled simulation uses small LLMs as users and larger LLMs as teachers to test factual knowledge acquisition.The user models are LLaMA-3.2-1B and 3B, while teacher models include LLaMA-3.1-8B and Qwen-2.5-7B/14B/32B.
  • 3.2 Effects of Personalized LLMs on Factual Knowledge Learning: The experiment samples 100 PFQABench factual questions that users cannot answer unaided, then compares multi-turn learning from personalized and standard teachers.The personalized teacher uses retrieval-augmented generation, and learning is evaluated against ground-truth answers.
  • 3.2 Effects of Personalized LLMs on Factual Knowledge Learning: 10.5% average factual-accuracy drop occurred for users taught by personalized rather than standard LLMs across teacher–student pairs (paired t-test: t = 3.176, p = 0.016).Applying FPPS to personalized LLMs is reported to improve resulting knowledge accuracy and mitigate this influence.

4 The Proposed Method: FPPS

FPPS detects where personalization distorts factual representations, probes the associated risk, and adaptively steers hidden states to preserve factuality without unnecessarily removing useful personalization. Its variants span hard removal, continuous correction, and mixed risk-threshold control.

  • Method Overview: FPPS locates a personalization-sensitive layer, probes factual–preference entanglement, and adaptively steers hidden representations to suppress harmful personalization while preserving useful personalized reasoning.The framework operates through layer selection, entanglement probing, and adaptive representation steering.
  • Layer Selection: Layer selection compares factual-token likelihood changes with and without user history across harmful and beneficial personalization cases, selecting the layer with the most consistent maximal deviation.Contrastive inputs share identical decoding trajectories, and rankings are aggregated using inverted-rank fusion.
  • Probing Detector: A logistic-regression prober outputs p̂ ∈[0, 1], estimating whether personalization relies on representations that may affect factual reasoning.Factual-degraded examples are positive examples, while personalized-beneficial examples are negative examples.
  • Adaptive Steering: FPPS-H removes personalization when estimated entanglement exceeds threshold τ, restoring the hidden representation to its non-personalized counterpart.This strict intervention suppresses personalization-induced representation shifts when factual risk is detected.
  • Adaptive Steering: FPPS-S continuously modulates personalization using the prober output, with positive coefficients attenuating personalization and negative coefficients enhancing it when beneficial.The steer vector points from personalized-query representations toward factual-query representations, strengthening factual reasoning with positive steering.
  • Adaptive Steering: FPPS-M uses soft steering under low entanglement and hard removal under high entanglement, balancing continuous personalization control with complete suppression when factual prediction is at risk.This design addresses the restrictiveness of hard removal when personalization contributes positively to correct reasoning.

5 Experiments

Experiments use PFQABench to evaluate factual and personalized QA across multiple personalization methods and LLM backbones. FPPS improves factual reliability, with FPPS-M offering the strongest overall balance and robustness across analyses.

  • Evaluation Setup: PFQABench contains 1,000 examples across 500 user sessions, combining long-term user histories with fact-centric multi-hop questions.
  • Evaluation Setup: The evaluation reports P-Score for personalized accuracy, F-Score for factual accuracy under personalization, and Overall as their average across three instruction-tuned backbones.
  • Main Results: Overall improves by 50%+ on average with FPPS across models and personalization settings, while F-Score substantially recovers when personalization distorts reasoning.
  • Main Results: FPPS-H generally achieves the highest F-Score but can reduce P-Score, whereas FPPS-S better preserves personalization while providing limited hallucination mitigation.
  • Main Results: FPPS-M consistently achieves the best Overall performance across backbones and personalization methods, balancing factual reliability with personalization utility.
  • Further Analysis: Ablations show that replacing either the probing detector or steering vector with a random alternative degrades performance, indicating both components are needed.
  • Further Analysis: As user history length increases, base personalized models show declining F-Score, while FPPS-M maintains stable factual performance across history ratios and model families.
  • Further Analysis: Adding FPPS-M to personalized teachers consistently improves users’ factual learning accuracy across all teacher–student model pairs.

6 Conclusion

The paper identifies personalization-induced hallucinations as a reliability risk caused by entanglement between user-specific signals and factual knowledge. It proposes FPPS as a way to constrain personalization by factual consistency.

  • Personalization-induced hallucinations arise when user-specific signals become representationally entangled with factual knowledge and distort factual reasoning.
  • FPPS conditionally regulates personalization in model representations, supporting a design principle of constraining personalization by factual consistency rather than injecting unconditional bias.

Limitations and Broader Implications

The study’s evidence is bounded by its model access and benchmark scope. FPPS requires intermediate representations, and its behavior on larger, more diverse model families remains to be validated.

  • Experiments cover a limited set of open-weight backbones because of compute and access constraints, while FPPS is not directly applicable to closed-source API models.
  • Validation on larger and more diverse model families remains an important direction because FPPS addresses entanglement at inference time.
  • PFQABench captures essential properties of personalization-induced factual distortion, but richer and more comprehensive benchmarks are still needed.

Ethic Statements

Personalization-induced hallucinations pose ethical risks in high-stakes applications by potentially misleading users and harming knowledge acquisition and trust. The paper frames factual reliability as a first-class objective while retaining beneficial personalization.

  • Incorrect personalized outputs can mislead users and negatively affect knowledge acquisition and trust in education, healthcare, and decision support.
  • FPPS is designed to regulate personalization when it interferes with factual correctness while preserving it when beneficial.
  • The empirical analysis links unchecked personalization with adverse effects on users’ factual learning, supporting factual correctness as a first-class objective.
  • The study uses public benchmarks or synthetic user histories and does not introduce datasets containing personal, private, or sensitive information.
  • FPPS is not a complete safeguard against all hallucinations or misuse, so personalized outputs should not be treated as authoritative truth and human oversight remains essential in high-risk applications.

A Sensitivity Analysis

FPPS performance is most stable with intermediate risk thresholds and moderate steering applied at later, high-level semantic layers. Extreme thresholds or steering intensities either suppress beneficial personalization or fail to prevent hallucinations.

  • FPPS-M Risk Threshold: Intermediate τ values produce a broad stable-performance plateau across tasks and model backbones.Low τ over-applies hard intervention, while τ near 1 allows personalization-induced hallucinations to persist.
  • FPPS-M Risk Threshold: A consistent near-optimal τ range across personalization paradigms and model scales reduces reliance on fine-grained threshold tuning.The prober output therefore supports balancing factual correctness and personalized utility with a single global threshold.
  • FPPS-S Hyperparameters: Moderate steering intensity γ, approximately 0.3–0.5, yields the best FPPS-S performance.Smaller values insufficiently correct distortion, whereas larger values degrade both factual and personalized accuracy.
  • FPPS-S Hyperparameters: FPPS-S effectiveness depends on applying steering at later transformer layers, with accuracy peaking in upper semantic layers.This matches the personalization-sensitive layers identified by the layer-selection criterion.
  • Overall Interpretation: The sensitivity results support moderate steering at high-level semantic representations as a robust design choice.The findings are consistent with personalization-induced hallucination reflecting a representation-level distortion.

B Analysis of Personalization-Sensitive Layers

Personalization most strongly alters factual likelihoods and encodes factual–personalization entanglement in higher transformer layers. Perplexity-based layer selection and probing converge on these layers, motivating targeted intervention there.

  • Layer-wise Perplexity Analysis: Personalization increases factual perplexity in mid-to-late layers for factual-degraded examples but reduces it later for personalized-beneficial examples.User history can therefore distort factual prediction or constructively guide it when user-specific information is relevant.
  • Layer-wise Perplexity Analysis: The largest personalized-versus-non-personalized perplexity divergence occurs in upper transformer layers across models and contrastive conditions.The consistency of the maximal deviation supports selecting a single personalization-sensitive layer.
  • Layer-wise Perplexity Analysis: The perplexity criterion directly identifies layers where personalization most strongly entangles with factual reasoning.It establishes that personalization alters factual likelihoods in a layer-dependent manner.
  • Factuality Prober: Probing accuracy stays near chance in early layers, rises rapidly in middle layers, and stabilizes at a high level later.Higher representations progressively encode whether personalization interferes with factual reasoning.
  • Factuality Prober: The highest probe-accuracy layers align with perplexity-selected personalization-sensitive layers across model backbones.This agreement provides complementary evidence that distortions are primarily manifested in high-level semantic representations.
  • Factuality Prober: The prober captures meaningful personalization-related signals rather than spurious correlations and is strongly layer-dependent.This supports probing and steering at a carefully selected internal layer.
  • Benchmark Context: PFQABench aligns factually relevant questions with topically related but factually irrelevant user histories to create tension between personalization and factual reasoning.The benchmark addresses the separation between existing personalization and factual QA evaluations.

E.4 Dataset Assembly and Splits

PFQABench combines balanced factual and personalized questions in aligned user sessions, enabling controlled evaluation of factual distortion alongside personalization utility. Its evaluation reports separate factuality, personalization, and overall scores.

  • Dataset Assembly: PFQABench contains 500 factual and 500 personalized questions, yielding 1,000 balanced examples.The factual questions are sampled from aligned candidate pools and paired with personalized questions from LongMemEval.
  • Dataset Splits: The dataset uses a stratified split with 250 personalized and 250 factual questions for training, and the same counts for testing.The test set is reserved exclusively for evaluation.
  • Dataset Assembly: Factual queries are paired with strong personalization signals to enable controlled measurement of personalization-induced factual distortion.This construction creates a setting where personalization and factual reasoning are systematically in tension.
  • Evaluation Setup: The study evaluates FPPS across four prompting-based personalization baselines, including retrieval-, summary-, clustering-, and recurrent-summary-based approaches.The baselines are RAG, PAG, DPL, and LLM-TRSR.
  • Evaluation Protocol: An automated LLM-as-a-Judge protocol compares each generated response with its ground-truth answer.QWEN2.5-32B-INSTRUCT determines correctness for each test instance.
  • Evaluation Metrics: P-Score measures personalized accuracy, F-Score measures factuality accuracy, and Overall Score averages the two.Together, these metrics quantify the balance between personalization utility and factual correctness.

G Implementation Details

The experiments use three instruction-tuned backbones, standardized retrieval and decoding settings, and layer-specific FPPS interventions selected by perplexity. Hyperparameters are searched over steering strength and decision threshold, while representation analyses compare personalized and non-personalized responses.

  • Experimental Configuration: FPPS is evaluated on LLAMA-3.1-8B-INSTRUCT, QWEN2.5-7B-INSTRUCT, and QWEN2.5-14B-INSTRUCT backbones.Retrieval-augmented settings use the BGE-M3 dense retriever, and experiments primarily use greedy decoding with a 500-token maximum.
  • Intervention Layer Selection: Intervention layers are selected using the perplexity criterion: layer 25 for LLAMA-3.1-8B, layer 24 for QWEN2.5-7B, and layer 43 for QWEN2.5-14B.All interventions occur during generation.
  • Hyperparameter Search: The hyperparameter search covers γ from 0 to 3 in steps of 0.2 and τ from 0.05 to 1 in steps of 0.01.The grid search is conducted across RAG, PAG, DPL, and LLM-TRSR personalization baselines.
  • Optimal Configurations: Optimal FPPS configurations vary across backbone and personalization baseline, including γ values from 0.1 to 3.0 and multiple τ settings.For LLAMA-3.1-8B, γ = 3.0 and FPPS-M uses τ = 0.5 across baselines.
  • Representation Analysis: Representation analysis averages final-layer hidden states over response tokens and measures cosine similarity between personalized and non-personalized responses.Instances are grouped by truthful versus hallucinated personalized responses, with significance tested using a two-sided Welch’s t-test.
Loading 2601.11000v1…