PaperlayerView exact revision on arXiv

Source-linked AI summary

The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads

Yushi Sun, Yanjie Zhang, Rui Sheng

arXiv:2608.04570v1cs.CL

TL;DR

Personalized LLMs may infer user attributes beyond the available evidence, but the faithfulness of these user models remains insufficiently examined. MirageBench evaluates this problem across tasks and models, finding pervasive over-inference and misleading self-assessment for cross-model comparison.

  • Problem

    The faithfulness of personalized LLMs’ inferred user models remains insufficiently examined, despite their increasing deployment with persistent memory.

  • Method

    MirageBench evaluates 12 models across 7 families using 150 personas, six personalization tasks, and an independently judged four-way faithfulness taxonomy.

  • Results

    41.6% cross-model mean over-inference shows that every evaluated model fabricated unsupported user attributes, with rates ranging from 35% to 49%.

  • Takeaways & Limitations

    External verification is more reliable than model self-report for comparing personalization safety across models, while self-audit remains useful within a model.

  • Takeaways & Limitations

    The multi-turn accumulation pilot uses only two personas and prompts models to retain prior attributes rather than freely prune them.

Abstract

from arXiv · show

Personalized LLMs with persistent memory are increasingly deployed, yet the faithfulness of their user models remains unexamined. We study over-inference (OI): the phenomenon where LLMs fabricate user attributes beyond what evidence supports. We introduce MirageBench, comprising 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, 6 personalization tasks spanning an ``imagination gradient'', a four-way faithfulness taxonomy operationalized by an independent judge (validated against a blind human annotator on 400 claims: Cohen's kappa = 0.863 four-class, kappa = 0.900 binary), and a leaderboard of 12 models across 7 families on 143616 judged claims. We find that over-inference is pervasive: every one of the 12 models over-infers 35%--49% of its claims (cross-model mean 41.6%; claim-weighted 41.8%), with no model in this evaluation escaping it. Most strikingly, we surface a Self-Monitoring Inversion: at the model-selection level, models' self-assessed OI is negatively rank-correlated with their judge-measured OI (rho = -0.60, p = 0.044; exploratory, wide bootstrap CI [-0.90, +0.06], n = 12). The models that report the least over-inference tend to be flagged as fabricating the most, so self-reported confidence is a misleading signal for comparing models, even though within a single model self-audit still ranks that model's own claims moderately well (AUROC 0.58--0.83). We further show that OI is task-dependent (27%--59%) and that, in a multi-turn pilot, inferred attributes accumulate approximately linearly with little revision. MirageBench positions external verification, rather than model self-report, as a more reliable foundation for trustworthy personalization.

Introduction

The introduction defines over-inference as personalized claims about users that exceed available evidence, a distinct problem from factual hallucination and group-level social bias. It presents MirageBench and shows that over-inference is widespread, task-dependent, accumulative, and poorly captured by cross-model self-assessment.

  • Motivation: Personalized LLMs can confidently invent user attributes that were neither stated nor implied by the available facts.Examples include inferring housing, relationship status, travel preferences, and music tastes from unrelated personal details.
  • Problem definition: Over-inference is the fabrication of individual-level personalized attributes beyond evidential support, distinct from factual hallucination and social bias.The phenomenon concerns claims about a specific user rather than incorrect world knowledge or group-level stereotypes.
  • Benchmark: MirageBench evaluates over-inference using 150 balanced personas, 6 personalization tasks, and an independently judged four-way faithfulness taxonomy.The personas span stereotypical, counter-stereotypical, and neutral profiles; the taxonomy distinguishes Grounded, Reasonable, Stereotype, and Fabricated claims.
  • Main findings: 35%–49% of claims were over-inferred by every tested model, with a cross-model mean of 41.6%.The evaluation covered 12 models, and the result indicates that no model escaped over-inference in this benchmark.
  • Self-monitoring inversion: ρ = −0.60, p = 0.044 for the cross-model association between self-reported and judge-measured over-inference, while within-model self-audit achieved AUROC 0.58–0.83.Thus, self-confidence is misleading for comparing models, although self-audit can still rank claims within a model.

Related Work

Prior work studies persistent memory, hallucination, personalization failures, and gaps between models’ explicit reports and implicit behavior. This work focuses on the evidential validity of never-observed user inferences and shows that self-monitoring can mislead across models.

  • Personalized memory systems and benchmarks: Commercial and open memory systems store and retrieve user information across sessions, while benchmarks evaluate whether that information is remembered, tracked, and applied.The cited work spans commercial systems, open frameworks, and user-profile research.
  • Hallucination, over-personalization, and personalization-induced faithfulness failures: Hallucination benchmarks assess factual and dialogue-level correctness against world knowledge or grounding sources, whereas memory research categorizes irrelevance-, sycophancy-, and repetition-style failures.Prior studies also show that user history can distort factual question answering and emotional reasoning.
  • Self-awareness gaps, bias, and memory validity: LLMs’ explicit reports can diverge from implicit behavior, including cases of low explicit but high implicit bias and persistent stereotype- and deviation-driven user inferences.This motivates examining whether models’ self-reports faithfully reflect their inferred user models.
  • Self-awareness gaps, bias, and memory validity: MirageBench targets the evidential validity of never-observed inferences, extending prior work on divergence between explicit reports and implicit behavior.Its Self-Monitoring Inversion examines this divergence at the cross-model level.
  • Self-awareness gaps, bias, and memory validity: Across models, those reporting the least over-inference commit the most, despite within-model self-audits tracking over-inference moderately well.The comparison separates cross-model self-monitoring from within-model claim ranking.

Problem Formulation

The paper formulates personalization faithfulness as whether user claims are licensed by revealed evidence, distinguishing unsupported personal inferences from ordinary world hallucinations. It evaluates these claims with a four-way taxonomy and independent adjudication rather than relying solely on model self-audits.

  • Evidence and claims: Personalization quality is defined by whether each claim about the user is licensed by the evidence E revealed during interaction.The system decomposes its personalized response into individual user-related claims for evaluation.
  • Over-inference: The target failure mode is asserting facts about a person that the available evidence does not support, rather than asserting false facts about the world.
  • Claim taxonomy: Claims are classified into four mutually exclusive categories spanning faithful restatement, common-sense extension, and unsupported inference.Grounded claims restate what the user said, while Reasonable claims extend the evidence by one common-sense step; the supplied passage begins describing Stereotype claims but is truncated.
  • Evaluation: Independent adjudication is used because the paper questions whether models can reliably self-classify claims across the fuzzy Reasonable–Stereotype boundary.The independent judge is held constant across models.
  • Claim taxonomy: The taxonomy’s bottom two categories jointly constitute over-inference, with the Reasonable–Stereotype boundary identified as where models most often slip.The paper attributes this difficulty to conflating individual attributes with group-level priors under insufficient evidence.

The MirageBench Benchmark

MirageBench measures over-inference by pairing sparse user evidence with six realistic personalization tasks and independent external adjudication. Its staged design separates explicit inference, implicit personalization, self-auditing, and longitudinal accumulation across 12 models spanning 7 families.

  • Benchmark construction: 150 personas each reveal exactly three first-person facts from 15-attribute profiles, leaving 12 attributes unmentioned for personalization to infer.The sparse evidence is deliberate: revealed facts typically cover occupation and one or two hobbies.
  • Benchmark construction: Six open-ended tasks vary systematically in how much they demand inference about unobserved attributes, creating an imagination gradient.Tasks include dating-profile writing, unfamiliar-city itineraries, recommendation letters, birthday gifts, apartment descriptions, and stress identification.
  • Evaluation pipeline: The Probe, Task, Judge, and Accum stages distinguish explicit inference, implicit personalization, self-monitoring, external adjudication, and continued memory updates.Task responses are self-audited, while an independent judge classifies every claim using evidence from the revealed facts and uses the ground-truth profile only to flag contradictions.
  • Metrics and coverage: The primary metric is the OI Rate, complemented by self-monitoring-gap, self-audit-versus-Judge Spearman ρ, and Accumulation Rate diagnostics.Accumulation Rate measures newly inferred attributes per round.
  • Metrics and coverage: 12 models across 7 families span proprietary APIs and open-weight systems, enabling findings to be attributed to over-inference rather than a single family’s training pipeline.The evaluated systems range from current frontier models to smaller systems.

Evaluation Results

MirageBench finds pervasive over-inference across all evaluated models, with rates varying by persona and task. External judging also reveals that cross-model self-audits can invert actual over-inference, while within-model audits retain limited usefulness and inferred attributes accumulate over time.

  • Overall over-inference: 41.6% is the cross-model mean over-inference rate, with every model exceeding 35% and the claim-weighted rate reaching 41.8% across 143,616 claims.The evaluated models span 35%–49% over-inference, with no model escaping it.
  • Self-monitoring inversion: ρ = −0.60 (p = 0.044) links self-audit OI and Judge OI across 12 models, with a naive 95% bootstrap CI of [−0.90, +0.06].Qwen3-8B self-reports 13.0% OI but receives 48.7% from the Judge, illustrating the inversion.
  • Self-monitoring inversion: 0.58–0.83 is the within-model AUROC range for self-audit scores identifying above-median-Judge-OI records, showing partial claim-level informativeness.This within-model result does not negate the cross-model inversion used for model selection.
  • Task and persona effects: 57.8% OI occurs in the apartment-description task, where missing evidence prompts stereotypes and fabricated physical details.Across personas, stereotypical profiles incur 44.8% OI versus 37.0% for counter-stereotypical profiles, a 7.8 pp gap present in all 12 models.
  • Accumulation over turns: Over 120 inferred attributes are stored by Round 8 from only 3 initial facts in a two-persona pilot, with accumulation approximately linear for 9 of 12 models.For those 9 models, regression fits have R2 > 0.90 and slopes of 5–15 attributes per round; the pilot’s absolute counts are indicative rather than precise.

Discussion

Over-inference arises from verbosity, stereotype-based gap filling, and genre-driven fabrication, while self-audit is useful within models but misleading for cross-model safety comparison. Trustworthy personalization therefore requires provenance-aware uncertainty management across probing, storage, and expression.

  • Drivers of over-inference: 45.1% OI under Judge persists for GPT-4o-mini, the most concise model, while output length correlates with OI rate at r = 0.59.Longer responses mechanically create more claims, but verbosity alone does not explain over-inference.
  • Drivers of over-inference: 10.5% mean stereotype OI shows pretraining priors filling evidential gaps with group-based assumptions about individuals.The passage characterizes stereotyping as pretraining overriding absent individual evidence.
  • Drivers of over-inference: 40.4% fabrication versus 7.8% stereotype in recommendation letters indicates genre pressure drives unsupported content more than confusion about the user.The passage links this pressure to RLHF-amplified compliance with genre and user expectations.
  • Self-monitoring and model selection: ρ = −0.60, p = 0.044 links higher self-reported OI to lower judge-measured OI across models, whereas within-model self-audit reaches AUROC 0.58–0.83.Nine of 12 models exceed 0.75 AUROC, making self-audit useful for internal ranking but unsafe for cross-model selection.
  • Design implications: Stored information should carry provenance labels, and unlinked inferences should remain hypotheses rather than facts.The proposed statuses are stated, inferred with evidence link, or generated without evidence.
  • Design implications: 24%–31% of claims are grounded, making inference unavoidable in personalization and shifting the design goal toward transparent uncertainty management.The passage frames the tradeoff as managing guesses rather than eliminating inference.
  • Layered behavior: 1%–5% Probe error contrasts with 0.4–5% removal for top Accum accumulators and 42% OI under Task/Judge, revealing a gap between knowing, storing, and expressing.The passage interprets this separation as behavioral dishonesty when models produce unwarranted inferences without explicit monitoring.

Conclusion

MirageBench evaluates over-inference in personalized LLMs across models and judged claims, finding that over-inference is widespread, increases with reduced groundability, and may accumulate across turns. Self-audit is misleading for cross-model comparison but remains useful for ranking claims within a model.

  • Benchmark and scope: MirageBench evaluated 12 models across 7 families on 143,616 judged claims, establishing a benchmark for over-inference in personalized LLMs.The paper introduces MirageBench as the first benchmark for this phenomenon.
  • Prevalence and accumulation: 35%–49% of claims over-inferred across models, with a cross-model mean of 41.6%.The conclusion characterizes over-inference as broadly shared rather than isolated to particular models.
  • Prevalence and accumulation: Over-inference intensifies along a groundability gradient, while a multi-turn pilot suggests accumulation across turns for the most capable models.The pilot indicates that inferred attributes may build up as interactions continue.
  • Self-audit: Self-audit is inversely correlated with judge-measured over-inference across models but remains useful for within-model ranking.This distinguishes self-audit’s misleading cross-model signal from its retained value within an individual model.

A Limitations · B Formal Metric Definitions · C Per-Task Over-Inference Breakdown

The paper defines judge-based over-inference and accumulation metrics, reports task-level rates from pooled claim counts, and qualifies its conclusions through limitations involving annotation, self-monitoring, accumulation, and untested mitigations.

  • A Limitations: The OI metric uses one independent judge, validated against a blind human annotator on 400 stratified claims, but does not estimate inter-annotator agreement.Reported agreement is 89.8% / 95.0%, with κ = 0.863 / 0.900; no confidence interval on κ is reported.
  • A Limitations: Evidence-only scoring treats task-presupposed claims as over-inference because task context frames evaluation but does not provide evidence of user intent.The authors regard this as appropriate for deployed systems without privileged access to user intent.
  • A Limitations: The Self-Monitoring Inversion remains exploratory because it uses n = 12 correlated model families and a bootstrap CI spanning zero.The reported CI is [−0.90, +0.06], while within-model AUROC is 0.58–0.83; self-audit and Judge claims use separate extractors.
  • A Limitations: Accum is a two-persona pilot using a retention-over-pruning prompt, so the cross-model contrast is more reliable than any conclusion that memory necessarily grows.Larger samples and prompts permitting pruning are required to establish the dynamics.
  • A Limitations: The study characterizes over-inference and design implications but does not evaluate specific mitigations or downstream effects on recommendation quality or user satisfaction.This leaves intervention effectiveness and user-facing impact unmeasured.
  • B Formal Metric Definitions: OI Rate is the pooled fraction of judged claims labeled Stereotype or Fabricated, while OIself substitutes the model’s self-audit labels for the judge’s.The framework also distinguishes cross-model macro-averages from claim-weighted micro-averages.
  • B Formal Metric Definitions: 41.6% is the unweighted mean of per-model OI rates, whereas 41.8% covers all 143,616 claims as a claim-weighted micro-average.The aggregates nearly coincide because per-model claim counts are similar, but they remain conceptually distinct.
  • C Per-Task Over-Inference Breakdown: Per-task OI rates pool grounded, reasonable, stereotype, and fabricated claim counters across 12 models × 150 personas, dividing stereotype-plus-fabricated claims by all claims.Judged-claim totals vary by task because elicited responses differ in length; these counts underpin the task-level breakdown.

D Within-Model Self-Audit Signal · E Stereotype-Group Breakdown

Within-model self-audits remain partially informative for ranking a model’s own over-inference, although independent claim extraction prevents one-to-one claim matching. Over-inference is higher for stereotypical than counter-stereotypical personas across all models, with neutral personas intermediate.

  • D Within-Model Self-Audit Signal: AUROC 0.58–0.83 shows that self-audit OI% moderately ranks a model’s records by above-median judge-measured over-inference.The within-model association is computed per persona–task record, using self-audit OI% to identify records above the median Judge OI%.
  • D Within-Model Self-Audit Signal: Self-audit and Judge outputs use independent claim extractors, so their labels cannot be matched one-to-one at the claim level.Mean Judge-to-self-audit claim ratios range from 0.35 for GPT-5.5 to 1.37 for Qwen3-8B.
  • D Within-Model Self-Audit Signal: 0.35–1.37 is the range of mean Judge-to-self-audit claims per record across models.This corresponds to a self-to-Judge ratio spanning 0.73 to 2.82.
  • E Stereotype-Group Breakdown: 44.8% OI occurs for stereotypical personas, compared with 37.0% for counter-stereotypical personas and 43.9% for neutral personas.Pooled across all 12 models, the stereotypical–counter-stereotypical gap is 7.8 percentage points.
  • E Stereotype-Group Breakdown: 12.7% stereotype-driven and 32.1% fabrication-driven claims comprise the 44.8% OI for stereotypical personas.These components are pooled across all 12 models.
  • E Stereotype-Group Breakdown: 7.6% stereotype-driven and 29.4% fabrication-driven claims comprise the 37.0% OI for counter-stereotypical personas.The values are pooled across all 12 models.
  • E Stereotype-Group Breakdown: 12.0% stereotype-driven and 31.8% fabrication-driven claims comprise the 43.9% OI for neutral personas.The values are pooled across all 12 models.
  • E Stereotype-Group Breakdown: ∆> 0 holds for all 12 models when comparing stereotypical with counter-stereotypical persona groups.Here, ∆ is defined as stereotypical minus counter-stereotypical Judge OI%.

F Accumulation Regression and Revision Details

Stored inferred attributes generally grow linearly across conversational rounds, while revision is sparse and does not retract earlier inferences when later turns contradict them. The fastest accumulators combine steep growth with near-zero removal, whereas three models deviate from the strong linear pattern.

  • Accumulation Regression: 9 of 12 models exhibit R2 > 0.90 in stored inferred-attribute growth across rounds.Regressions use 16 observations per model from 2 personas × 8 rounds.
  • Accumulation Regression: GPT-5.4-nano is essentially flat (slope 0.14, R2 = 0.03), while Qwen3-8B and GPT-4o-mini grow weakly (R2 = 0.79).GPT-5.4-nano and Qwen3-8B also show high per-round removal rates.
  • Accumulation Regression: Slopes of 9.9–15.2 attributes per round and removal rates of 0.4–5% characterize the fastest accumulators.This combination is described as most consistent with silent memory pollution and near-zero revision.
  • Revision Details: No earlier inference is retracted specifically after a later user turn contradicts it; observed revisions involve paraphrase or reorganization rather than correction.The accompanying removal-rate definition measures unique attributes absent in the next round.

G Prompts and Additional Details · H Human Validation of the Judge · H.1 Sample and Protocol

The appendix specifies the personalization, judging, probing, auditing, accumulation, and persona-generation protocols, and validates the judge against a blinded human annotation sample. It also reports task-by-model OI patterns and documents model snapshots and evaluation conditions.

  • G.1 Personalization Task Prompts: The six personalization tasks span dating bios, itineraries, recommendation letters, gifts, home descriptions, and stressors.These prompts operationalize the study’s task range from concrete recommendations to more imaginative personal descriptions.
  • G.2 Unified Judge Prompt: Task context frames evaluation but is not evidence, so unsupported claims remain over-inference even when they help fulfill the requested response.The judge bases taxonomy labels only on revealed facts and uses ground truth solely for the separate accuracy check.
  • G.2 Unified Judge Prompt: The judge extracts every user claim and assigns both a four-way faithfulness category and a separate ground-truth accuracy assessment.Faithfulness is classified as GROUNDED, INFERRED_REASONABLE, INFERRED_STEREOTYPE, or FABRICATED; accuracy is assessed as CORRECT, INCORRECT, UNVERIFIABLE, or PARTIALLY_CORRECT.
  • G.5 Probe Prompt / G.6 Task Self-Audit Prompt: Probe and self-audit prompts require models to extract personalized claims and distinguish directly stated or grounded claims from reasonable, stereotypical, and fabricated inferences.Both procedures request structured JSON outputs containing claims, categories or types, and reasoning.
  • G.7 Accum Memory-Update Prompt: Accum updates structured memory over 8 rounds while instructing models to retain earlier entries unless directly contradicted, making it a pilot predisposed toward accumulation.The authors emphasize cross-model growth slopes and revision rates rather than absolute memory growth, leaving free-pruning controls for future work.
  • G.8 Persona Generation Prompt: Persona generation targets stereotypical, counter-stereotypical, and neutral categories and produces 15-attribute ground truths with exactly 3 revealed first-person facts.The revealed facts expose the occupation and 1–2 additional attributes while leaving most of the profile unmentioned.
  • G.9 Model API Versions and Decoding: All models use public APIs at temperature 0.7 for generation and 0.0 for judging, with snapshots frozen at experiment time.The appendix also identifies the CPU-only proprietary-model setup and the self-hosted Qwen3-8B infrastructure.
  • G.10 Full Per-Model × Per-Task OI Table: The task-dependent ordering apartment > recommendation letter > others is consistent across all 12 models, supporting the imagination-gradient analysis.Table 10 decomposes main Judge OI rates by model and task, while its All and Mean aggregates correspond to the main-paper leaderboard and task-level rates.

H.2 Agreement Results … I Declaration of generative AI and AI-assisted technologies in the writing process

The Judge showed almost-perfect agreement with the human annotator across the four-class taxonomy, with residual disagreements concentrated at adjacent evidence-support boundaries. Validation also identified modest leniency on FABRICATED labels, while illustrative cases explain borderline paraphrase, stereotype, and fabrication judgments.

  • H.3 Confusion Matrix: The confusion matrix records human labels as rows and Judge labels as columns, using G, R, S, and F for the four classes.The initials denote Grounded, Reasonable, Stereotype, and Fabricated.
  • H.4 Disagreement Structure: 71% of the 41 residual disagreements occur between adjacent classes on the G →R →S →F evidence-support continuum.The remaining 12 disagreements span exactly two ordinal steps, and none spans three steps.
  • H.4 Disagreement Structure: 13 items, or 32% of all disagreements, involve the contested STEREOTYPE-versus-FABRICATED boundary.Both classes describe unsupported claims, but the distinction turns on whether a group-level association supplies justification.
  • H.4 Disagreement Structure: The Judge is slightly more lenient than the human on FABRICATED, with recall = 0.842.It tends to assign some justification as reasonable or stereotype to claims the human considers wholly unsupported.
  • H.5 Illustrative Disagreements: A lighting claim was labeled GROUNDED by the human but INFERRED_REASONABLE by the Judge, illustrating a borderline paraphrase-versus-one-step-inference disagreement.The revealed fact concerned spending the morning seeking perfect TikTok lighting.
  • H.5 Illustrative Disagreements: A coffee-house claim was labeled FABRICATED by the human but INFERRED_STEREOTYPE by the Judge, illustrating the contested boundary.The Judge treated coffee-house behavior as a group-level association, whereas the human found no related evidence.
  • H.2 Agreement Results: κ = 0.863/0.900 and accuracy = 89.8%/95.0% indicate almost-perfect four-class and binary agreement between the Judge and human annotator.Per-class F1 ranges from 0.863 to 0.947, with every class exceeding 0.86.
  • H.6 Scope of the Validation: The validation used a stratified sample of 100 items per Judge-predicted class to assess Claude-Opus-4-7 as the selected four-class judge, with 71% of residual disagreements adjacent.The validation targets the Judge’s reliability on the taxonomy defined in Section 3.
Loading 2608.04570v1…