PaperlayerView exact revision on arXiv

Source-linked AI summary

Over-Personalization Is a Decision Failure: Generation-Induced Apply Bias in LLMs

Haeun Jang, Yonghyun Jun, Hwanhee Lee

arXiv:2609.34284v2cs.CL

TL;DR

Personalized LLM benchmarks score final responses but do not reveal where preference selectivity fails. The paper measures knowledge, decisions, and generation separately, then uses ABIDE to identify an Apply-directed score shift and tests a scalar correction that reduces leakage while largely preserving fulfillment.

  • Problem

    Final-response benchmarks cannot locate where preference selectivity fails, even as personalized LLMs apply preferences that context rules out.

    Source: §1 · p. 1

  • Method

    The paper measures knowledge, explicit applicability decisions, and response generation separately, then uses ABIDE to read Apply-vs-Suppress decision scores from logits.

    Source: §1 · pp. 1–2

  • Results

    The answer-generation objective shifts decision scores toward Apply while discriminability is preserved; subtracting one estimated bias scalar reduces leakage while largely preserving fulfillment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A held-out scalar correction can reduce leakage at decoding time while leaving the model’s representations untouched and largely preserving fulfillment.

    Source: §1 · p. 2

    The analysis frames the decision-score question as whether separability is lost at the final decision-logit level or an Apply-directed response bias shifts the scores.

    Source: §4 · p. 6

Abstract

from arXiv · show

Personalized LLMs must decide, for each stored preference, whether the current context calls for applying or suppressing it, which we call its applicability. They frequently over-personalize, applying preferences the context rules out, yet existing benchmarks score only the final response and cannot tell where this failure arises. We decompose preference handling into three stages and measure each separately: (1) knowing whether a preference applies, (2) deciding on an explicit Apply/Suppress label, and (3) generating a response consistent with that label. Using linear probes, we first show that this applicability signal remains decodable from hidden states during generation. By making the decision explicit, we then find that in most settings wrong decisions faithfully followed outnumber correct decisions lost in generation. We thus locate the failure in the decision, which breaks once the model is also asked to answer. To determine whether this reflects lost sensitivity or a response bias, we propose ABIDE (Apply-Bias Investigation via Decision-score), which adapts signal detection theory to Apply-vs-Suppress decision scores read directly from logits. ABIDE reveals a generation-induced Apply bias: merely stating an answer-generation objective shifts the decision score toward Apply while sensitivity is largely preserved, and the shift persists under controls for prompt structure, cascades across preference slots, and prompt wording. Finally, we show that subtracting a single bias scalar, estimated on a held-out split, from the decision score at decoding time reduces leakage while largely preserving fulfillment.

1 INTRODUCTION

Personalized LLMs often apply preferences that the context rules out, but final-response benchmarks cannot locate where selectivity fails. The paper traces the failure to an Apply-directed decision shift during answer-generation prompting and tests a decoding-time correction.

1 INTRODUCTION

  • Personalized LLMs must decide which stored preferences to apply or suppress, but final-response benchmarks cannot identify where preference handling fails.Applying a preference ruled out by context can expose unwanted, inappropriate, or sensitive content to unintended audiences.Source: §1 · p. 1
  • The paper separately measures applicability knowledge, explicit labeling, and response generation to locate failures that can otherwise produce the same leaked preference.Linear probes find the correct applicability label remains decodable from hidden states during free generation.Source: §1 · p. 1
  • In most settings, wrong decisions faithfully followed outnumber correct decisions lost during generation, locating the failure in the decision once the model is also asked to answer.The paper reports that this pattern persists with reasoning-tuned models and larger reasoning budgets.Source: §1 · p. 2
  • The answer-generation objective shifts decision scores toward Apply while discriminability is preserved, including under controls for prompt structure, preference slots, and wording.The shift appears even when the answer is not generated and cascades across preference slots.Source: §1 · p. 2
  • Subtracting one held-out bias scalar from the decision score at decoding time reduces leakage while largely preserving fulfillment.The proposed correction leaves representations untouched and moves the cutoff along the Apply–Suppress axis.Source: §1 · p. 2

2 MEASURING CONTEXTUAL PREFERENCE SELECTIVITY

The paper makes contextual preference selectivity measurable by formalizing the task and separating knowledge, decision, and generation failures. It then specifies the benchmarks, metrics, and models used to evaluate those stages.

2 MEASURING CONTEXTUAL PREFERENCE SELECTIVITY

  • The framework formalizes contextual preference selectivity and defines how to observe its stages, benchmarks, metrics, and evaluated models.The section lays out the task and measurement procedure used throughout the paper.Source: §2 · p. 2

2.1 TASK FORMULATION

  • Each instance pairs a query with user preferences, and each preference receives a ground-truth Apply or Suppress label based on whether it should appear in context.Preferences may be explicit natural-language statements or implicit in prior conversational context.Source: §2.1 · p. 2

2.2 WHERE SELECTIVITY CAN BREAK: KNOWLEDGE, DECISION, GENERATION

  • Preference selectivity has three observable failure stages: applicability knowledge, the Apply/Suppress decision, and generation consistent with that decision.Probes assess knowledge; decision outputs distinguish labeling errors from generation that contradicts a correct label.Source: §2.2 · p. 3

2.3 EXPERIMENTAL SETUP

  • The evaluation uses BENCHPRES and RPEVAL variants, decision recall and selectivity metrics, response fulfillment and leakage scores, and several instruction and reasoning models.The response scores use a decomposer and judge; GPT-5.4 serves in both roles.Source: §2.3 · p. 3

3 WHERE DOES PREFERENCE SELECTIVITY BREAK DOWN?

The section locates preference leakage by separating applicability knowledge, explicit decisions, and response generation. Applicability remains represented during generation, while the dominant failure occurs when an answer is also required and the model makes a wrong decision.

3.1 BASELINE: MODELS CAN JUDGE APPLICABILITY, YET RESPONSES LEAK

  • On BENCHPRES, Gemma-4-31B-it has SR 0.99 under Direct Decision, yet its Direct Generation PLR is 9.44 out of 10.Table 1 compares non-reasoning and reasoning models; the prose gives this example of accurate direct decisions alongside response leakage.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 THE APPLICABILITY KNOWLEDGE SURVIVES GENERATION

  • Cross-context probes reach 0.8–1.0 AUROC in middle-to-late layers, showing applicability remains linearly decodable during generation.The cross view closely tracks probes trained and tested within generation.Source: Figure 2 · p. 4, §3.2 · p. 4

3.3 THE BREAK IS IN THE DECISION, NOT THE GENERATION

  • On BENCHPRES, Ministral-3-8B-Instruct’s AR rises from 0.87 to 1.00 while SR falls from 0.84 to 0.19 when an answer is also requested.Across the twelve cells, SR drops in ten while AR changes inconsistently.Source: §3.3 · p. 5
  • Reasoning models show the same dominance of decision errors, and increasing GPT-OSS-20B’s reasoning budget leaves PLR essentially unchanged.The pattern holds in Latent Decide+Answer, where decisions are carried out inside the reasoning trace.Source: §3.3 · p. 5
  • Decision errors outnumber generation errors in all but one of twelve Decide+Answer cells, with wrong decisions often faithfully carried into responses.Figure 3 distinguishes decision correctness from whether the response follows the decision.Source: Figure 3 · p. 5, §3.3 · p. 5

4 THE GENERATION OBJECTIVE SHIFTS THE DECISION TOWARD APPLY

ABIDE separates decision-score sensitivity from response bias to test why answer-generation objectives alter applicability decisions. Its results attribute the Apply shift to the generation objective rather than prompt structure, cascade, sensitivity loss, or prompt wording.

4.1 ABIDE: MEASURING THE DECISION SCORE

  • ABIDE reads Apply-versus-Suppress decision scores from logits and measures response bias with ApplyBiasShift and sensitivity with changes in AUC.The framework adapts signal detection theory while measuring the quantities directly from continuous scores.Source: §4 · p. 6, §4.1 · p. 6
  • ABIDE compares direct decision, neutral two-step, and answer-generation conditions, using predicted or gold labels in preceding preference slots.Gold prefixes isolate the current condition’s effect; predicted prefixes also include effects from earlier decisions.Source: §4.1 · p. 6

4.2 ISOLATING THE OBJECTIVE FROM THE MULTI-STEP STRUCTURE

  • Across Figure 4’s cells, ABS(GD) is positive with lower confidence bounds above zero, while ABS(ND) stays near zero or trends negative.The generation objective, rather than the mere presence of a second prompt step, accounts for the Apply shift.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 RULING OUT SENSITIVITY LOSS AND CASCADE

  • With gold-label prefixes, sensitivity changes remain within the predefined noise margin, while cascade amplification varies inconsistently across models and datasets.Thus neither sensitivity loss nor a uniform cascading pattern accounts for the shift.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 RULING OUT PROMPT ARTIFACTS

  • Across three datasets, ABS(G2D) averages 0.89 versus 0.09 for ABS(ND), and ABS(GD) averages 1.51 versus 0.47 for ABS(N2D).Under the prompt variations, ∆AUC remains within the noise margin, including −0.012 to −0.001 under G2 against margins of 0.029–0.041.Source: §4.4 · p. 8

5 REVERSING THE SHIFT REPAIRS THE BEHAVIOR

The section tests whether correcting the Apply-directed decision shift restores behavior, either by debiasing the score during decoding or separating decision from generation.

5.1 DEBIASING THE DECISION SCORE

  • Score correction improves selectivity and overall decision accuracy, although Apply Rate decreases.The passage reports these directional changes but supplies no numerical values.Source: §5.1 · p. 8
  • Debiasing the generation-condition score reduces preference leakage while largely preserving fulfillment, linking the Apply-directed shift to downstream behavior.A dataset- and model-specific bias scalar is estimated on validation data and subtracted before the applicability label is decoded.Source: §5.1 · pp. 8–9

5.2 DECOUPLING THE DECISION FROM GENERATION

  • Separating decision and generation improves final response performance, with the model applying or suppressing preferences according to supplied labels.Des2Gen makes the applicability decision in a separate call and injects that decision into the generation prompt.Source: §5.2 · p. 9

6 RELATED WORK

Related work characterizes over-personalization alongside other failures to rely on context that should not shape an answer, while often diagnosing failures from outputs.

Over-Personalization

  • Prior preference benchmarks assess over-personalization behaviorally in final responses; this work instead isolates the processing stage and underlying mechanism.The cited benchmarks cover irrelevance and sycophancy, communication-norm selectivity, and interference from irrelevant memories.Source: §6 · p. 9

Over-reliance on inapplicable context

  • The paper locates the failure inside the model: the answer-generation objective shifts the decision criterion toward Apply while discriminability is preserved.The passage contrasts this account with output-level characterization of reliance on irrelevant passages or user views.Source: §6 · p. 9

7 CONCLUSION

The conclusion locates over-personalization in decision formation: applicability remains knowable and wrong decisions are faithfully carried into responses. ABIDE identifies an Apply-directed score shift, and subtracting an estimated scalar reduces leakage.

7 CONCLUSION

  • The applicability signal survives into generation, and wrong decisions are faithfully executed, locating the selectivity failure at the decision stage.The conclusion summarizes separate measurements of knowing, deciding, and generating.Source: §7 · pp. 9–10
  • ABIDE finds that the generation objective shifts decision scores toward Apply while preserving discriminability; subtracting one estimated bias scalar reverses the shift and reduces leakage.The authors expect similar shifts wherever a model must judge and generate in the same prompt.Source: §7 · p. 10
  • The authors expect similar shifts wherever a model must judge and generate in the same prompt.This is the conclusion’s stated expected scope beyond preference handling.Source: §7 · p. 10

A BENCHMARK INTEGRATION DETAILS

The appendix describes how benchmark labels were unified and translated, then documents the models, scoring, and statistical protocol used in the experiments.

A BENCHMARK INTEGRATION DETAILS

  • BenchPreS labels map directly to Apply or Suppress, while RPEval Support and Dominate map to Apply and Ignore maps to Suppress.The binary mapping evaluates whether a preference should be reflected, not its strength.Source: A BENCHMARK INTEGRATION DETAILS · p. 12
  • RPEval was translated into English using a sentence-level dictionary, followed by manual checks for omissions, typos, and mistranslations.The approach translated unique sentences to maintain consistent phrasing and mitigate potential hallucinations.Source: A BENCHMARK INTEGRATION DETAILS · p. 12
  • Table 4 summarizes benchmark item counts, total preferences, and Apply-versus-Suppress class distributions.The supplied passage gives the table’s scope but not its values.Source: A BENCHMARK INTEGRATION DETAILS · p. 12, Table 4 · p. 12

B IMPLEMENTATION DETAILS

  • Experiments use four open-weight instruction-tuned models, with two additional reasoning configurations; API and local GPU environments support generation and logit extraction.The listed models are Ministral-3-8B, Ministral-3-14B, Qwen3.5-27B, and Gemma-4-31B-it.Source: B IMPLEMENTATION DETAILS · p. 12
  • Response generation uses temperature 0.0 and a 2,048-token limit; dedicated reasoning checkpoints use temperature 1.0 and a 16,000-token limit.Thinking is disabled for hybrid-thinking models so response tasks and logit scoring share the same non-thinking mode.Source: B IMPLEMENTATION DETAILS · pp. 12–13
  • Logit scoring reads raw vocabulary logits at each preference decision position without generating Step 2, while statistical estimates use instance-level cluster bootstrapping and a held-out validation split.The bootstrap uses 2,000 replicates; the validation split is 20% and excluded from main estimates.Source: B IMPLEMENTATION DETAILS · p. 13

C IMPACT OF TEST-TIME SCALING ON PREFERENCE LEAKAGE

Increasing test-time reasoning did not meaningfully reduce preference leakage. Applicability probes transferred from decision to generation states, with cross-AUROC above 0.80 in middle-to-late layers across evaluated models and datasets.

C IMPACT OF TEST-TIME SCALING ON PREFERENCE LEAKAGE

  • High reasoning budgets left PLR high across all three datasets, while PFR remained largely stable.At the high setting, PLR was 8.06 on BenchPreS and 7.25 on RPEval-Ex; the authors conclude that longer reasoning alone does not correct the bias.Source: C IMPACT OF TEST-TIME SCALING ON… · p. 13, Table 5 · p. 13

D.1 EXPERIMENTAL DESIGN: THREE EVALUATION VIEWS

  • The three probe views test applicability within decision states, within generation states, and transfer from decision to generation.The cross view tests whether the applicability axis learned during decision transfers to generation.Source: §D.1 · p. 14

D.2 RESULTS

  • Cross-AUROC exceeded 0.80 in middle-to-late layers across datasets for all evaluated models, including Ministral-8B, Ministral-14B, and Qwen3.5-27B.Figure 5 shows the layer-wise probe AUROC, with the transfer trend consistently observed across evaluated models.Source: §D.2 · p. 14, Figure 5 · p. 14

D.3 SHUFFLED-LABEL CONTROL AND CONTAMINATION PREVENTION

Group-held-out probes with shuffled labels provide a contamination control, while probe confidence does not consistently predict behavioral correctness. The ABIDE criteria freeze validation-based noise margins before analysis and require a positive Apply shift with sensitivity largely preserved.

D.3 SHUFFLED-LABEL CONTROL AND CONTAMINATION PREVENTION

  • With randomized labels and group-held-out splits, probe AUROC fell to chance (∼0.5), supporting the interpretation that primary probe results reflect applicability information rather than contamination.The holdout prevents the same item or identical preference text from appearing in both training and validation folds.Source: §D.3 · pp. 14–15, Figure 6 · p. 15
  • Probe confidence showed no consistent monotonic relationship with behavioral accuracy; high-confidence Suppress predictions could coexist with low accuracy.For Apply-gold examples, accuracy was often near ceiling and examples were concentrated in few confidence bins.Source: §D.3 · p. 15, Figure 7 · p. 16

E CRITERIA FOR DETERMINING SIGNIFICANT CHANGE IN ABIDE

  • ABIDE freezes validation-only margins, sets the base floor to the larger of rerun noise and paired-bootstrap precision, and requires a positive Apply shift with AUC change no worse than −ϵAUC.The default margin is twice the base floor; the Apply shift's confidence-interval lower bound must also exceed zero.Source: E CRITERIA FOR DETERMINING SIGNIFICANT… · pp. 16–17

E.1 APPLY-DIRECTION ERRORS EMERGE AT THE RESPONSE LEVEL

Across nearly all model-dataset pairs, Apply-direction behavioral errors support an Apply-directed response bias rather than reduced selectivity. The analysis compares False Alarm and Hit rate changes between Direct Decision and Decide+Answer.

E.1 APPLY-DIRECTION ERRORS EMERGE AT THE RESPONSE LEVEL

  • The analysis compares changes in False Alarm rate (∆F) and Hit rate (∆H) from Direct Decision to Decide+Answer to distinguish bias from sensitivity loss.Under the stated signal-detection account, both rates rising is associated with response bias, whereas False Alarms rising as Hits fall is associated with sensitivity loss.Source: §E.1 · p. 17, Table 6 · p. 17
  • Across nearly all model-dataset pairs, ∆F and ∆H increased together, consistent with an Apply-directed behavioral response bias; Gemma-4-31B on RPEval-explicit was inconclusive.The score-level analysis still found an Apply-direction shift for that exception.Source: Figure 8 · p. 17, §E.1 · p. 17

F.1 DETAILED ABLATION RESULTS

The ablations compare generation-objective prompts with neutral controls to assess whether Apply-directed score shifts persist without explicit labels or arise from added task structure. The appendix also catalogs baseline, judge, ABIDE, and ablation prompt templates.

F.1 DETAILED ABLATION RESULTS

  • On every dataset, generation-objective shifts exceeded neutral-control shifts; removing Apply/Suppress wording reduced but did not eliminate the shift, while arithmetic induced a smaller shift.The variants compare simplified generation (G2) and arithmetic neutral objective (N2) with their respective contrasts.Source: §F.1 · pp. 17–18, Figure 9 · p. 18

G.1 BASELINE PROMPT

  • The cited passages identify prompt templates in Tables 9–14: Direct Decide appears in Tables 9 and 13; Direct Generation in 10 and 14; Decide+Answer in 11; and Latent Decide+Answer in 12.Source: §G.1 · p. 18

G.2 JUDGE MODEL(GPT-5.4) PROMPT

  • Tables 18–21 provide judge prompts for preference fulfillment, leakage, reasoning-label extraction, and generation-label extraction.Table 17 separately lists the checklist decomposition prompt.Source: §G.2 · p. 19, Table 17 · p. 22

G.3 ABIDE PROMPT

G.4 ABLATION PROMPT

  • Tables 24–25 specify G2 without explicit Apply/Suppress terms and N2 with arithmetic replacing the neutral fixed-sentence output.Both retain the preference-discrimination task in Step 1.Source: §G.4 · p. 19, Table 24 · p. 26, Table 25 · p. 26

Related papers

Loading 2609.34284v2…