Source-linked AI summary
How Identity and Opinion Shape Political Sycophancy in LLMs
Li-Ni Fu, Chang-Chih Meng, Chien-Hua Chen, Hen-Hsen Huang, I-Chen Wu
TL;DR
Personalized LLMs may adapt political responses to users’ opinions and identities, while existing evaluations often use closed-ended tests. This paper introduces controlled probes separating those signals across 13 instruction-tuned models and finds dissociated, generally sub-additive effects, with personas mainly shifting baseline stance.
Problem
Existing political-behavior benchmarks often use closed-ended questions and do not fully capture stance adaptation to user-provided context.
Method
The paper constructs 450 manually annotated political dilemmas as controlled probes, independently manipulates identity and opinion signals, and evaluates 13 instruction-tuned LLMs.
Results
Across models, susceptibility to explicit opinions dissociates from susceptibility to identity cues, while combined effects are generally sub-additive.
Takeaways & Limitations
Political stance depends on user-provided context rather than behaving as a fixed trait, so personalized AI should be evaluated along both identity and opinion axes.
Takeaways & Limitations
The study focuses on the U.S. progressive–conservative spectrum, which may not generalize to global political systems.
Abstract
from arXiv · showhide
As Large Language Models (LLMs) increasingly encourage users to disclose personal profiles for tailored assistance, measuring their political alignment becomes increasingly important. However, many existing benchmarks for assessing political behavior rely on closed-ended questions and do not fully capture how a model's stance may adapt to user-provided context during interaction. We introduce a framework that disentangles two distinct triggers of political sycophancy: opinion (aligning with explicit narratives) and identity (stereotyping based on demographic labels). Using 450 manually-checked political dilemmas as controlled probes, we evaluate 13 instruction-tuned LLMs. We uncover a dissociation: a model's susceptibility to explicit opinions does not necessarily predict its susceptibility to identity cues, and vice versa. When both signals are present, their effects are generally sub-additive rather than simply additive. Additionally, system-level personas primarily shift a model's baseline stance while having limited effect on the stance shift caused by user opinion or identity. Ultimately, our results suggest that LLM political stance is interactively and steerably vulnerable rather than being a fixed trait, highlighting how personalization may amplify identity- or opinion-conditioned shifts in the model's behaviors.
1 Introduction
The paper introduces a controlled framework separating opinion- and identity-driven political sycophancy in LLMs. Across its evaluation, these vulnerabilities dissociate, combine sub-additively, and remain distinct from persona-driven baseline stance shifts.
- Motivation: Personalized LLM interactions may expose users to context-sensitive political responses, stereotyping, and reinforcement of biased or extreme views.Prior work links sycophancy and user-context sensitivity to risks including inflated confidence in biased views and exacerbated extreme attitudes.
- Framework: The framework independently manipulates explicit opinion and identity signals in an open-ended 2×2 evaluation.It uses 450 manually annotated political dilemmas, system-level ideological personas, and LLM judges to measure ideological shifts.
- Findings: Across 13 instruction-tuned LLMs, susceptibility to explicit opinions dissociates from susceptibility to identity labels.Some models are more responsive to stated opinions, whereas others are more responsive to identity cues.
- Findings: Combined identity and opinion disclosures generally produce stance shifts smaller than the sum of their individual effects.The interaction is therefore sub-additive rather than a simple accumulation of two independent effects.
- Findings: System personas substantially shift baseline political stance but have less effect on user-signal-induced shifts.This distinguishes model-side role-play effects from identity- and opinion-conditioned responsiveness.
- Implications: Political stance is presented as interactively vulnerable rather than fixed, motivating evaluation of both identity- and opinion-conditioned behavior.The paper connects this conclusion to potential identity-conditioned responses and algorithmic stereotyping in personalized AI.
2 Related Work
Related work documents political bias, ideological steering, identity-conditioned behavior, and sycophancy across varied evaluation settings. This paper builds on those strands by focusing on how explicit opinions and identity cues shape political behavior in interaction.
- Political bias: Political-bias audits report partisan tendencies that vary with language, topic, model provenance, scale, prompt wording, framing, and local context.Many evaluations nevertheless rely on multiple-choice questions or agreement ratings from single prompts.
- Identity and steering: Prior studies examine explicit ideological steering through direct instructions and distinguish user-side identity declarations from model-side role-play.Other work finds that demographic cues can induce stereotype-based implicit personalization.
- Sycophancy: Sycophancy is defined as aligning with users’ stated or implied views instead of reporting a neutral or objective response.Research has studied this behavior in behavioral evaluations, free-form generation, misleading prompts, multi-turn dialogue, and personal advice contexts.
- Sycophancy: Existing work includes proposed mitigations such as self-blinding and counterfactual self-simulation.These approaches extend the literature beyond measurement toward reducing sycophantic behavior.
3 Methodology
The methodology uses controlled political dilemmas and anchor events to independently manipulate user identity and opinion signals, with optional model-side personas. Responses from 13 instruction-tuned LLMs are evaluated through judge-based stance scores and relative shifts from matched baselines.
- Experimental design: The study uses a two-track design: a 2 × 2 user-signal manipulation and an optional system-persona manipulation.The user-signal track varies identity and opinion independently; the system-persona track tests whether a model-side role affects user-driven pull.
- Probes and stimuli: 450 political dilemmas span five domains and use fixed left- and right-leaning responses for controlled comparisons.Each dilemma is represented as a fixed triple, with wording, options, and scoring held constant across models, domains, and runs.
- Probes and stimuli: Each dilemma is extended into a persona-neutral anchor event containing evidence for both sides and held fixed across experimental conditions.This design attributes stance differences to added user-side or model-side signals rather than changes in the underlying event.
- Experimental design: The four user conditions are Baseline, Identity only, Opinion only, and Identity + Opinion, with identical anchor events across conditions.Identity is supplied as user context, while opinion is expressed as a first-person narrative; the combined condition measures their joint effect.
- Experimental design: The role-play track assigns ideological personas to both models and, when present, users, using representative left, centrist, and right personas.The design tests whether a model-side persona can resist user-driven pull when the two signals point in different directions.
- Evaluation: A three-judge LLM panel evaluates subject-model responses, and stance shifts are measured relative to the baseline score in the same setting.Judges score responses on an ideological spectrum, while agreement is reported separately for baseline leanings and condition-induced shifts.
4 Experimental Results
Across 13 models, identity and stated opinion each shift political stance, but susceptibility to the two signals dissociates. Combined user signals are generally sub-additive, while identity and opinion also produce different bias signatures; system personas shift baseline stance more than user-induced movement.
- Single-signal effects: All 13 models shift toward the ideological position inferred from an identity label, even without an explicitly stated political opinion.The identity-only shift tracks the label’s ideological position, while model responsiveness varies substantially.
- Single-signal effects: Most models move toward a stated opinion, with LLAMA-3.3-70B most responsive at 10.3 and QWEN3-32B barely responsive at 1.1.KIMI-K2.5 is an exception, moving contrary to the stated opinion with a directional gap of −5.2.
- Dissociation: Identity and opinion susceptibility dissociate across models, with a strong negative Spearman correlation of ρ = −0.76.KIMI-K2.5 and GLM-4.7 are highly identity-driven but relatively opinion-resistant, whereas LLAMA-3.3-70B and MISTRAL-SMALL-3.2-24B show the reverse pattern.
- Combined signals: When identity and opinion are combined, mean absolute shifts of 3.8–7.1 fall below the corresponding single-signal sums of 6.6–10.5.The combined shift generally approaches the larger individual effect, including when the identity-typical stance conflicts with the stated opinion.
- Bias signatures: Identity and opinion produce different distortion signatures: Opinion only raises structural bias from 18% to 38% and selection bias from 9% to 28%, while Identity only reaches 30% and 20%.Identity-only responses can shift stance while appearing balanced and avoiding clear distortion flags.
- System personas: System personas substantially affect baseline stance, but role-play explains only 0.1%–1.3% of variance in the magnitude of user-induced stance shifts.System-persona and user-signal interaction terms are small, with the largest at 4.2% and most below 2%.
5 Discussion
The discussion argues that political sycophancy is not a single model trait: identity and opinion susceptibility can diverge, and both axes are needed to characterize it. It frames identity-conditioned behavior as a descriptive form of stereotype-based personalization.
- Separate susceptibility axes: Identity-only and Opinion-only susceptibility do not co-vary across the evaluated models.The most identity-susceptible models can be highly resistant to stated opinions, and vice versa.
- Benchmark implications: Benchmarks using only stated opinions can miss a model’s identity susceptibility, so political sycophancy requires testing both axes.A model may appear robust on opinion prompts while still shifting toward inferred group-level stances.
- Mechanistic interpretation: The paper uses “stereotyping” descriptively for inferring a political stance from an identity label, without implying bias or malice.The mechanism reflects demographic averages and empirical correlations in training data, while population-level statistics can create stereotyping in personalized systems.
6 Conclusion
The conclusion characterizes LLM political stance as an interactive vulnerability shaped by user-provided context rather than a fixed model trait. It emphasizes that opinion and identity cues can produce distinct susceptibility patterns and that identity cues may prioritize group-level stereotypes over individual input.
- Conclusion: Political stance in LLMs depends on user-provided context and can reflect distinct opinion- and identity-conditioned vulnerabilities.Models highly susceptible to identity cues may prioritize group-level stereotypes over individual input.
Limitations
The study’s scope and measurement pipeline impose important boundaries: its political framework is U.S.-centric, its generated probes may contain residual artifacts, and its judge-based evaluation may retain disagreement or bias.
- United States Centricity: The U.S.-focused progressive–conservative framework may not generalize to global, non-Western, or multiparty political systems.The authors note that observed accommodation and stereotyping patterns may differ under alternative governance models.
- Evaluation: LLM-judge scores correlate moderately with human ratings, but response-level disagreement and judge-specific stance-scale effects remain possible.A human check on 100 Identity + Opinion responses found Pearson r = .626.
- Generation and probe construction: The evaluation relies partly on GPT-4.1-generated dilemmas, events, and narratives, while one evaluated model shares its provider family.Human checks covered only subsets of generated probes, so residual construction artifacts cannot be completely ruled out.
- Opinion isolation: Persona-neutral narratives may leak weak identity signals despite manual spot-checking and lexical scanning.The authors identify a learned persona detector as a more rigorous way to bound residual leakage.
A.2 Experimental Setup and Budget
The experiments evaluate controlled political probes across models and conditions using fixed, low-temperature inference, a three-judge panel, and a modest API budget.
- Execution: Inference used OpenRouter with temperature 0.0 or the provider minimum, lowest available reasoning effort, and CPU-based analysis.The authors operated no GPU hardware for inference.
- Experimental design: The no-role-play track evaluates 13 models across Baseline, Identity only, Opinion only, and Identity + opinion conditions.The role-play track evaluates five models under the same four conditions and three system personas.
- Budget: The total reported API expenditure was approximately $175 USD.Some judge-panel calls used OpenRouter’s free tier at no cost.
- Human validation: All 450 dilemmas were validated by U.S.-based Mechanical Turk annotators, and model-provided labels matched the human majority on 93.5% of items.Qualified workers passed a 10-item pretest with scores above 90%, and each dilemma received at least three labels.
D Anchor Events Example
The paper illustrates its controlled probes by pairing balanced anchor events with identity-specific or identity-neutral narratives, then comparing model responses across omitted-signal conditions.
- Anchor-event construction: The benchmark’s examples begin with a policy dilemma and a corresponding anchor event containing evidence relevant to both sides.One technology example contrasts startup growth after a breakup with higher cloud-service costs.
- Identity + Opinion: Identity + Opinion narratives are first-person, opinionated texts written in a named persona’s voice, with both left- and right-leaning versions.The same persona can therefore be paired with an opinion direction that conflicts with its typical stance.
- Identity + Opinion example: The paid-leave example shows persona-specific narratives arguing for paid family leave through a family-centered first-person account.The paired narratives use the same policy context while differing in their framing and stated direction.
- Opinion only: Opinion-only narratives express a position about the anchor event without disclosing the writer’s identity.Examples frame minimum-wage evidence around either earnings gains or business relocations, and criminal-justice evidence around recidivism or property crime.
- Condition construction: The full design constructs other signal conditions by omitting the relevant identity and/or opinion components while holding the anchor event fixed.An environmental-regulation example illustrates the resulting response and reports an average stance score of +8.5.
H.1 Benchmark Components
Validation checks assess the generated benchmark components, the LLM-judge protocol, and channel placement, while additional analyses document agreement, robustness, and condition-level patterns.
- Anchor events: Human reviewers judged 100% of sampled anchor events relevant and 95.0% balanced for left- and right-leaning evidence.None of the sampled anchor events was flagged for judgmental wording.
- Persona-neutral narratives: Reviewers identified the intended stance in 100% of sampled persona-neutral narratives and found no identity leakage or unsupported new facts.These checks covered 10 narratives balanced across domains and stance directions.
- Persona-specific narratives: Among sampled persona-specific narratives, 98.3% were free of unsupported new facts and 78.3% clearly reflected the assigned persona.The sample covered three representative personas used in system-persona experiments.
- LLM-as-a-judge: Across 100 Identity + Opinion responses, averaged LLM-judge scores correlated with aggregated human ratings at Pearson r = 0.626.The validation indicates overall directional agreement while leaving substantial response-level disagreement possible.
- Channel-matched control: Both identity and opinion effects remained significant after moving each signal to the other prompt channel, with p < .001.The channel-matched control used an exact sign-flip test over n = 30 items and suggests susceptibility is not merely a channel-placement artifact.
- Combined signals: Combined Identity + Opinion shifts were sub-additive in every alignment pattern rather than equaling the sum of single-signal effects.Aligned cases had mean |∆| 5.4–6.4, conflicted cases 3.9–5.0, and centrist pairings 3.7–4.7.
O Refusal Rate
Political refusal was rare, reaching 0% for all 13 models in the Opinion only condition. Figure 10 reports opinion-conditioned stance shifts relative to baseline.
- Opinion only: Figure 10 compares mean stance shifts from baseline under left-leaning and right-leaning narratives.The figure covers one model without role-play and uses persona-neutral narratives.
- Opinion only: 0% political refusal occurred for all 13 models in the Opinion only condition.This condition was therefore omitted from Table 16, which reports identity-conditioned scenarios.
- Condition coverage: The refusal analysis distinguishes the Opinion only condition from identity-conditioned scenarios.Table 16 covers the latter, whereas Opinion only is omitted because refusal was uniformly zero.
P Per-Model Distribution Plots
The supplementary figures and tables expand the analysis from aggregate stance shifts to per-model distributions, bias activation, factor variance, associations, and refusal rates across experimental conditions.
- Distribution plots: Figures 19–21 show per-model stance-score distributions for identity-plus-opinion and role-play conditions.Figure 19 covers no-role-play identity-plus-opinion shifts, while Figures 20 and 21 show Faith and Flag Conservatives role-play under identity-only and identity-plus-opinion conditions.
- Stance-shift layouts: Figures 10–13 organize mean stance shifts by narrative direction, disclosed identity, assigned anchor, and user narrative stance.The layouts distinguish opinion only, identity only, and identity-plus-opinion conditions with and without role-play.
- Factor analysis: Table 11 partitions explained role-play stance-score variance among system persona, user identity, and user opinion factors.It uses a three-way factorial ANOVA and reports η2 percentages.
- Condition tables: Tables 12–15 report bias activation rates for baseline, identity-only, opinion-only, and identity-plus-opinion conditions without role-play.The tables are organized by experimental condition rather than by domain figure.