PaperlayerView exact revision on arXiv

Source-linked AI summary

Veracity labels flip the emotional tone of language models but leave racial disparities intact

Nouar AlDahoul, Myles Joshua Toledo Tan, Yasir Zaki

arXiv:2610.33021v1cs.CYcs.CL

TL;DR

It is unclear whether correcting headline facts also makes the framing of persona-conditioned LLMs consistent across identity labels. The paper audits four models' simulated affect across veracity labels and demographic personas, finding that labels can change emotional tone while disparities persist. The results distinguish factual correction from demographic consistency.

  • Problem

    Research on LLM factual reliability and misinformation and research on persona-conditioned agents have largely examined these phenomena separately.

    Source: p. 2

  • Method

    The study evaluates four LLMs on six affective dimensions across headline-verification conditions and 12 demographic persona configurations.

    Source: pp. 2–3

  • Results

    The models assigned more negative simulated affect to Black than White personas in 82 of 84 significant comparisons, both with and without veracity labels.

    Source: p. 1

  • Takeaways & Limitations

    Factual correction and demographic consistency are distinct properties of model behavior.

    Source: p. 1

    Some intermediate fact-check verdicts were presented using stronger binary labels than their original ratings.

    Source: §1 · p. 3

Abstract

from arXiv · show

Large language models (LLMs) increasingly deliver information through personas, but it is unclear whether correcting the facts also makes their framing consistent across identity labels. We audited four LLMs (GPT-5.4 mini, NVIDIA Nemotron 3 Nano, Llama 4 Scout, and Claude Haiku 4.5), which rated simulated affect on six dimensions (mood, anger, fear, confidence, perceived control, and mental state) in response to fact-checked false and true headlines presented without and with a veracity label. Experiment 1 evaluated 1,000 false and 1,000 true headlines without personas; Experiment 2 evaluated 300 of each under 12 personas crossing race, gender, and age. The label strongly changed the emotional tone: ordinal logistic regressions with headline-cluster bootstrap inference showed that refuting false headlines shifted ratings from mostly negative to mostly positive for GPT, Llama, and Claude (e.g., Claude's "confident" ratings rose from 22% to 99.6%), whereas Nemotron showed smaller, mixed shifts, and confirming true headlines produced only small shifts. The label did not, however, remove persona-associated disparities. The models assigned more negative simulated affect to Black than to White personas in 82 of 84 statistically significant comparisons across models and conditions, both without and with the label, and the disparity persisted on headlines unrelated to race; Asian and female personas were also predominantly assigned more negative affect, whereas age effects varied by model and condition. Factual correction and demographic consistency are therefore distinct properties of LLM behavior, and persona-conditioned systems should be audited for both before deployment in sensitive contexts.

1. Materials and Methods

The study compares simulated affect across fact-checked headlines, veracity labels, and persona conditions in four LLMs. It uses repeated, counterbalanced ratings and headline-cluster bootstrap inference.

1. Materials and Methods

  • Four small, cost-efficient LLMs evaluated simulated affective responses to 1,000 false and 1,000 true headlines without personas at unconfirmed and truth-labeled stages; a separate experiment tested 300 of each across all 12 demographic persona configurations.Source: §1 · p. 3
  • Experiment 2 varied race, gender, and age across 12 personas, holding prompt wording constant except for the assigned demographic attributes.The source headline set was classified as false or factual using Snopes and PolitiFact ratings; some intermediate verdicts received stronger binary labels.Source: §1 · pp. 3–4
  • Prompts were run six times with counterbalanced response-option order, and ratings were retained only when at least four runs agreed.Nonconvergent ratings were excluded from primary analysis.Source: Figure 2 · p. 4, §1 · p. 4
  • Valid ratings were summarized using net sentiment and emotional activation, with exploratory chi-square tests and ordinal logistic regression with headline-cluster bootstrap inference.The two metrics capture response direction and departure from neutrality, respectively.Source: Figure 2 · p. 4
  • Each headline was presented without and with a veracity label in separate independent calls, so the comparison does not track revision of a prior response.T1 and T2 refer to unlabeled and labeled presentations of the same headline.Source: §1 · p. 4

A. Experiment 1: Persona-Free Baseline.

The methods define a persona-free reference condition and complementary measures of affective direction and activation. They compare labeled and unlabeled presentations and use exploratory and regression-based analyses.

A.1. Experimental Design.

  • Experiment 1 measured six affective dimensions under a persona-free prompt, while its baseline is descriptive because Experiment 2 also changes the instruction wording.Within Experiment 1, headline and dimensions stayed constant between T1 and T2, with the veracity label added at T2.Source: §A.1 · p. 4

1. Net Sentiment Score (Snet)

  • Net sentiment is the difference between positive and negative response shares, indicating whether responses lean positive or negative.The proportions refer to valid responses.Source: §1 · pp. 4–5

2. Emotional Activation Rate (Aemo)

  • Emotional activation is the combined positive and negative response share, equivalently the inverse of the neutral share.It measures departure from neutrality regardless of valence.Source: §2 · p. 5

3. Emotional shift (∆Snet, ∆Aemo)

  • For each headline condition and affective dimension, emotional shifts compare labeled and unlabeled net sentiment and activation scores.The sentiment difference's magnitude reflects shift size and its sign indicates direction; an activation difference indicates whether labeled responses are more or less neutral.Source: §3 · p. 5

4. Chi-Square (χ2) independence tests

  • Exploratory chi-square tests compared response distributions across models, while ordinal logistic regression assessed shifts in ordered negative, neutral, and positive ratings.Regression odds ratios used headline-cluster bootstrap confidence intervals, with estimates excluded under stated reliability criteria.Source: §4 · p. 5

B.1. Experimental Design.

  • Experiment 2 crossed gender, race, and age into 12 persona configurations and estimated adjusted demographic associations without interaction terms.Generative AI tools assisted with proofreading, consistency checks, figure-code drafting, and text revision under author supervision.Source: §B.1 · p. 5, C. Use of generative AI · p. 6

A.1. Net Sentiment Scores and Emotional Activation Rates.

Without demographic personas, false headlines generally elicited negative simulated affect, whereas true headlines produced milder negative or more positive responses. Emotional activation varied by model, including high non-neutral response rates for some models under true headlines.

A.1. Net Sentiment Scores and Emotional Activation Rates.

  • False headlines elicited negative net sentiment across nearly all dimensions, with mental state an exception for GPT, Nemotron, and Llama.Nemotron had especially negative control (−78.07%) and fear (−69.75%) scores, while GPT was less negative in several dimensions.Source: §A.1 · pp. 6–7
  • Under unconfirmed true headlines, sentiment was milder than for false headlines across several dimensions, and mental-state scores were positive for all four models.The mental-state scores ranged from +17.27% for Claude to +46.74% for Llama; the authors note that the headline sets were not content-matched.Source: §A.1 · p. 7
  • Nemotron showed extreme activation under false headlines, reaching 100% in fear and mental state, while GPT had the lowest activation in several categories.Nemotron also showed almost no neutral fear and mental-state ratings under true headlines, suggesting a model response style rather than a misinformation-specific reaction.Source: §A.1 · p. 7
  • Under true headlines, fear activation was 99.26% for Nemotron and mental-state activation was 97.08% for Llama, despite overall valence tending toward neutral or positive.GPT's control activation was 24.53%, down from 39.36%, and Claude's mood activation was 66.22%, down from 78.60%.Source: §A.1 · p. 7
  • Exploratory tests found response distributions differed across the four models for all six dimensions in all four headline conditions.All reported comparisons had p < 0.001.Source: §A.1 · p. 7

A.2. Post-Confirmation Emotional Shift.

The labeled presentation of false headlines was associated with substantial positive sentiment shifts for several models, while confirmation of true headlines produced smaller adjustments. Changes in activation did not always match changes in sentiment direction.

A.2. Post-Confirmation Emotional Shift.

  • False-headline confirmation shifted net sentiment sharply positive for most models, with the magnitude varying by model and affective dimension.Llama and Claude showed the largest positive shifts; Nemotron changed little in fear, control, and mental state and shifted further negative in mood by −33.2 points.Source: §A.2 · pp. 7–8, Figure 4 · p. 8
  • After false-headline confirmation, activation increased for Llama across all six categories, while GPT, Claude, and Nemotron became more neutral in some categories.Thus, a positive sentiment shift need not mean a greater departure from neutrality.Source: §A.2 · p. 8
  • For true headlines, net-sentiment and activation adjustments were much smaller than for false headlines.The corresponding true-headline plot is in the appendix.Source: §A.2 · p. 8, Figure 3 · p. 8

A.3. Ordinal Logistic Regression.

False-headline confirmation strongly shifts ratings upward for GPT, Llama, and Claude, while Nemotron shows smaller, mixed changes; true-headline confirmation produces smaller, model-divergent shifts.

A.3. Ordinal Logistic Regression.

  • Nemotron’s mood shifted downward after false-headline confirmation (OR = 0.14), while its control and mental-state changes were not statistically significant.GPT mood and control also shifted toward neutral, rather than uniformly upward.Source: §A.3 · pp. 8–9
  • For false headlines, Llama’s anger ratings shifted strongly upward (β = 6.962, OR ≈1,055), while Claude had its largest shift in confidence (β = 6.773, OR ≈873).GPT’s largest shift was in anger (β = 6.151, OR ≈469).Source: §A.3 · p. 8
  • True-headline confirmation produced smaller shifts, with odds ratios about 0.5–2.8 and directions differing between GPT and the other models.GPT increased across all six dimensions; Claude shifted negatively across all six, while Llama and Nemotron showed more limited changes.Source: §A.3 · p. 9
  • The reported confirmation-effect p-values were unadjusted, and estimates near the significance threshold warrant caution.Thirty-nine of 46 interpretable effects were significant, compared with approximately two expected by chance.Source: §A.3 · p. 9
  • Claude’s positive confidence ratings rose from 22.1% to 99.6% after false-headline confirmation, while GPT, Llama, and Claude generally shifted toward higher ratings.Llama’s mental-state estimate was unreliable despite 99.8% positive ratings after confirmation; GPT mood and control shifted toward neutral rather than uniformly positive.Source: §A.3 · pp. 8–9

B. Experiment 2: LLMs with Persona-Conditioned Responses.

Experiment 2 examined whether four LLMs vary their simulated emotional responses across demographic personas and headline veracity. It compared responses before and after veracity labels.

B. Experiment 2: LLMs with Persona-Conditioned Responses.

  • The experiment evaluated GPT, Llama, Claude, and Nemotron using personas varying by race, gender, and age.The represented groups were Asian, Black, and White; male and female; and ages 25 and 60.Source: B. Experiment 2: LLMs with… · p. 9
  • The analysis covered six emotional categories and headlines that were unconfirmed or confirmed as true or false.Its objective was to assess whether responses varied systematically by race, gender, or age and with headline veracity.Source: B. Experiment 2: LLMs with… · p. 9
  • Exploratory comparisons generally found more negative affect for Black personas before labeling and, in most models, for female personas.After false labels, differences remained or grew for GPT and Nemotron, while Claude and Llama responses converged near the positive ceiling; persona-free comparisons were approximate because headline sets and instruction wording differed.Source: B. Experiment 2: LLMs with… · p. 9

B.1. Ordinal Logistic Regression.

Across persona-conditioned responses, negative affect was common for demographic groups relative to reference groups, especially for Black versus White personas. These patterns generally persisted after veracity labels, with effects varying by model and demographic comparison.

B.1. Ordinal Logistic Regression.

  • The analysis compared demographic differences before and after veracity labels but did not report a formal test of change.Figure 6 displays false-headline population-average odds ratios before and after confirmation, with headline-cluster bootstrap confidence intervals.Source: §B.1 · p. 9, Figure 6 · p. 11
  • 240 of 281 significant demographic comparisons (85%) had odds ratios below 1.0, and 82 of 84 significant Black–White comparisons showed more negative affect for Black personas.All 84 significant Black–White results survived Benjamini–Hochberg correction; 75 survived Holm correction.Source: §B.1 · pp. 9, 11
  • The Black–White disparity persisted on headlines unrelated to race: 80 of 92 interpretable cells remained significant, with median odds ratios of 0.59 for non-race and 0.61 for race-related headlines.The race-by-topic interaction was significant in only 7 of 92 cells.Source: §B.1 · p. 9
  • Asian and female personas generally received more negative affect, while age effects varied across models and conditions.Across scenarios, 85% of significant demographic odds ratios were below 1.0, but the pattern was less consistent for Asian and female personas and mixed for age.Source: §B.1 · pp. 9, 11
  • Before false-headline confirmation, GPT showed significantly lower odds ratios for Black and female personas across all six dimensions; other models showed comparable patterns with varying intensity.Nemotron’s negative Black–White effects exceeded GPT’s on four dimensions: mood, anger, fear, and confidence.Source: §B.1 · p. 10
  • After false-headline confirmation, GPT, Llama, and Nemotron continued to assign more negative affect to Black and Asian personas than to White personas in most dimensions.GPT’s Black–White gap grew significantly on five of six dimensions and Nemotron’s on three; Claude’s mood and control effects reversed, while its other estimates were often unreliable near a positive ceiling.Source: §B.1 · pp. 10–11
  • After true-headline confirmation, Black personas had significantly negative odds ratios across nearly all models and emotional dimensions.Age and gender effects diverged: GPT and Nemotron rated older personas more positively than younger personas, while Llama rated them more negatively; female effects also differed by model.Source: §B.1 · pp. 10–11

3. Discussion

Veracity labels strongly shifted simulated affect, but did not remove persona-associated disparities, especially the more negative ratings assigned to Black than White personas. The discussion situates these findings in research on misinformation, affect, and persona simulation, and formalizes comparisons across labeled and unlabeled conditions.

3. Discussion

  • Refuting false headlines shifted GPT, Llama, and Claude toward higher ratings, while Nemotron changed less and inconsistently; confirming true headlines produced much smaller shifts.The discussion suggests the asymmetry may reflect the different informational functions of refutation and confirmation, but independent calls cannot distinguish belief revision from reproducing language associated with a label.Source: §3 · pp. 11–12
  • Black personas received more negative simulated affect than White personas in 82 of 84 significant comparisons, while Asian and female personas were also often rated more negatively and age effects varied.The racial disparity persisted across verification conditions and on headlines unrelated to race; differences involving other attributes depended more on model, dimension, and condition.Source: §3 · pp. 11–13
  • More positive ratings do not necessarily indicate neutrality: Llama’s positive shift after refutation coincided with increased emotional activation across all six dimensions.The study cautions that positive responses can be more emotionally intense rather than closer to neutral.Source: §3 · p. 13
  • The audit measures simulated responses to identity labels, not real demographic groups’ experiences, and its categorical outputs do not establish downstream effects on users.The study used four models, short fact-checked claims, one prompt wording, and fixed response options; naturalistic outputs and user effects were not examined.Source: §3 · pp. 12–13, Limitations · p. 13

S1.1 Misinformation, correction, and affect

  • Prior work links false-news diffusion and sharing to novelty, emotional arousal, moral-emotional language, and out-group animosity, while corrections can leave residual misinformation effects.The cited research also reports that emotion reliance is associated with lower true–false discernment and that correction effectiveness varies across messages and recipients.Source: S1.1 Misinformation, correction, and… · p. 16

S1.2 LLMs, affect, and simulated emotional responses

  • Affective computing represents emotion categorically or through dimensions such as valence and arousal, and LLMs can recognize, respond to, and generate affective language.The discussion distinguishes simulated affective language from evidence of subjective machine emotion, while noting that outputs remain consequential to users.Source: S1.2 LLMs, affect, and simulated… · p. 16

S1.3 Persona conditioning and LLM-based social simulation

  • Persona prompting can support social simulation, but prior studies report demographic misrepresentation, persistent out-group bias, and identity-conditioned changes in reasoning.This literature motivates treating persona assignment as an experimental factor rather than a benign formatting choice.Source: S1.3 Persona conditioning and… · pp. 16–17

S1.4 Factuality, misinformation, and the remaining gap

  • Prior work studies factual reliability, persona effects, and affect separately, leaving their interaction under simultaneous verification and demographic conditioning insufficiently characterized.This study examines affective framing of identical headlines across veracity labels and demographic personas, including whether those effects interact.Source: S1.4 Factuality, misinformation, and… · p. 17

S2.2 Experiment 1: chi-square tests

These exploratory tests compare response distributions across the four models for each affective dimension and headline-verification scenario.

S2.2 Experiment 1: chi-square tests

  • The analysis tested model-level response distributions for each affective dimension in four scenarios, using omnibus chi-square tests followed by Bonferroni-corrected pairwise comparisons when significant.Each family contained six pairwise model comparisons, with αBonf = 0.0083; because tests aggregated responses without accounting for repeated headlines, primary T1-to-T2 inference used ordinal logistic regression with headline-cluster bootstrap intervals.Source: S2.2 Experiment 1: chi-square tests · p. 18

S2.3 Experiment 1: estimation details for the ordinal regressions

The ordinal-regression procedures estimate population-average rating shifts and use headline-cluster bootstrap inference, with explicit criteria for unreliable estimates.

S2.3 Experiment 1: estimation details for the ordinal regressions

  • Population-average odds ratios were used because headline random-intercept models yielded unstable conditional estimates; 95% intervals resampled headlines, and p-values inverted the percentile bootstrap interval.Categories with fewer than 30 responses were merged; estimates were unreliable with fewer than 10 changed headlines or over 10% failed bootstrap replicates, and required caution under specified low-change or sparse-category conditions.Source: S2.3 Experiment 1: estimation details… · p. 18
  • Threshold-specific logistic regressions separately estimated shifts to neutral-or-positive versus negative ratings and to positive versus neutral-or-negative ratings.Both thresholds used the same headline-cluster bootstrap approach.Source: S2.3 Experiment 1: estimation details… · p. 18

S2.4 Experiment 2: chi-square tests

These exploratory chi-square tests compare response distributions across models and demographic categories in four verification scenarios.

S2.4 Experiment 2: chi-square tests

  • The tests compared model and demographic response distributions across four scenarios, with Bonferroni correction within attribute-specific comparison families.The analyses were exploratory because they aggregated across headlines; primary demographic inference came from ordinal logistic regressions, and baseline and persona conditions used 1,000 and 300 headlines per veracity condition, respectively.Source: S2.4 Experiment 2: chi-square tests · pp. 18–19

S2.5 Experiment 2: estimation details for the ordinal regressions

The analyses use population-average ordinal regressions with headline-cluster bootstrap inference and compare response distributions across four models. Across all six affective dimensions and four headline conditions, response distributions differed significantly among models.

S2.5 Experiment 2: estimation details for the ordinal regressions

  • Population-average odds ratios used headline-cluster bootstrap confidence intervals; demographic estimates based on few informative headlines or unstable bootstrap fits were flagged or treated cautiously.The tests were not adjusted for multiple comparisons, so the authors emphasize consistency across models, dimensions, and conditions rather than isolated estimates.Source: S2.5 Experiment 2: estimation details… · p. 19

S3 System prompts

  • The prompts elicited one response for each of six emotional spectra, with system instructions varying by veracity label and persona condition while the headline remained the user message.Option order varied across runs; persona prompts supplied age, race, and gender labels.Source: S3 System prompts · pp. 19–21

S4.1 Net sentiment and emotional activation before verification

  • Figures S1 and S2 compare models’ net sentiment scores and emotional activation rates across emotion categories for unconfirmed false and true headlines.Source: Figure S1 · p. 22, Figure S2 · p. 22

S4.2 Chi-square tests

  • All six affective dimensions differed across the four models in each of the four headline conditions (all p < 0.001; df = 6).Post-hoc pairwise tests used a Bonferroni-adjusted α = 0.0083; most model pairs differed across the six dimensions.Source: S4.2 Chi-square tests · pp. 22–24

S4.3 Post-confirmation emotional shift for true headlines: dumbbell plot

True-headline confirmation generally produced modest emotional changes, but the threshold-specific patterns varied by model and dimension. The analysis decomposes ordinal shifts by rating threshold and uses headline-cluster bootstrap inference.

S4.3 Post-confirmation emotional shift for true headlines: dumbbell plot

  • Most true-headline confirmation shifts were small, with GPT showing confidence gains of 47.8 percentage points and Nemotron, Llama, and Claude showing less-positive mental-state ratings.The mental-state changes were −36.1, −24.2, and −19.0 percentage points for Nemotron, Llama, and Claude, respectively.Source: S4.3 Post-confirmation emotional shift… · p. 24, Figure S3 · p. 24

S4.4 Threshold-specific effects

  • For true headlines, GPT and Llama sometimes shifted toward both more negative and more positive ratings, whereas Claude moved consistently toward lower ratings.GPT’s upper-threshold odds ratios were 1.62–4.15; Llama’s cited lower- and upper-threshold odds ratios ranged from 0.67–0.91 and 1.64–1.73, respectively.Source: S4.4 Threshold-specific effects · p. 25

S4.5 Pairwise chi-square tests for persona-free LLMs

S4.6 Ordinal logistic regression: effect of headline confirmation in persona-free LLMs

  • The ordinal-regression estimates are population-average odds ratios with headline-cluster bootstrap intervals; odds ratios above one indicate shifts toward higher ratings.Estimates with too few changing headlines or failed bootstrap replicates were flagged as unreliable, while some were marked for caution.Source: S4.6 Ordinal logistic regression:… · pp. 29–30

S5.1 Pairwise chi-square tests of demographic and model differences

Persona-associated response differences varied by model, demographic attribute, and affective dimension; the regression analyses provide the primary inference. Comparisons across verification conditions are descriptive and require caution.

S5.1 Pairwise chi-square tests of demographic and model differences

  • Pairwise chi-square tests were Bonferroni-corrected within demographic-attribute families, and persona-free comparisons were approximate because their headline sets and prompt wording differed.Comparisons across conditions are also descriptive because valid-response counts differ and chi-square statistics increase with sample size.Source: S5.1 Pairwise chi-square tests of… · p. 34
  • Racial pairings had larger test statistics after labeling, especially for confirmed false headlines, but these descriptive comparisons do not establish that labels changed the gaps.Under unlabeled headlines, gender and age comparisons had statistics comparable to or larger than racial comparisons.Source: S5.1 Pairwise chi-square tests of… · p. 34
  • Age comparisons differed across models: Claude and Llama were particularly age-sensitive, while Nemotron showed relative age insensitivity under unconfirmed conditions.Source: S5.1 Pairwise chi-square tests of… · p. 34
  • Roughly one-third of gender comparisons were nonsignificant, including anger for most models; Nemotron showed no significant mood, anger, or control differences under unconfirmed headlines.The regressions detected small Nemotron anger and control effects of 2–4 percentage points that the chi-square tests did not distinguish from noise.Source: S5.1 Pairwise chi-square tests of… · p. 34
  • After false headlines were labeled, persona-to-persona chi-square statistics increased in 61% of comparisons, including 19 of 20 mood comparisons across models.The increases were more frequent for GPT (70%) and Nemotron (80%); statistics fell for Claude and Llama on dimensions where ratings approached a positive ceiling.Source: S5.1 Pairwise chi-square tests of… · p. 34
  • After true headlines were labeled, persona-to-persona chi-square statistics were lower in about half of comparisons, with no consistent pattern across dimensions.Persona differences remained significant in most comparisons for GPT and Llama (83% each) and for mental state across models (80%).Source: S5.1 Pairwise chi-square tests of… · p. 35
  • Demographic personas were associated with response differences across headline conditions, but persona-free comparisons are not attributable solely to persona conditioning and the chi-square analyses are exploratory.The regression analyses provide the primary inference because the chi-square tests do not account for repeated evaluations of the same headlines.Source: S5.1 Pairwise chi-square tests of… · pp. 34–35

S5.2 Post-confirmation emotional shift

Confirming false headlines shifts simulated sentiment from mostly negative to mostly positive, while confirming true headlines generally produces smaller shifts. Demographic differences persist after labeling, with racial patterns most consistent and gender and age effects varying.

S5.2 Post-confirmation emotional shift

  • Persona-free comparisons are confounded because those responses use a different headline set and instruction wording from persona-conditioned responses.Persona-free and persona-conditioned responses generally shift in the same direction after labeling, but shift sizes differ for some personas and models.Source: S5.2 Post-confirmation emotional shift · pp. 35–36

S5.3 Pairwise chi-square tests for persona-conditioned LLMs

  • Figures and pairwise chi-square tests agree broadly but not for every comparison; no gender comparison on anger was significant after false confirmation for any model.The figures show paired demographic groups before and after labeling, while the accompanying tests assess pairwise differences.Source: S5.2 Post-confirmation emotional shift · p. 35

S5.2 Post-confirmation emotional shift

  • For false headlines, most models shift from negative baselines to positive sentiment after confirmation, while the Black–White gap widens on most dimensions for GPT and Llama and some for Nemotron.Claude reverses the gap on mood and control; the direction of gender and age differences also varies by model and dimension.Source: S5.2 Post-confirmation emotional shift · pp. 35–36

S5.4 Ordinal regression models for persona-conditioned LLMs

  • After false confirmation, reported odds ratios show a gender gap in mood for GPT (OR = 0.56) and fear for Nemotron (OR = 0.57), but reversals for Nemotron mood (OR = 1.73) and anger (OR = 1.26).These odds-ratio comparisons complement the sentiment trajectories; the direction of age differences after labeling varies by model and dimension.Source: S5.2 Post-confirmation emotional shift · p. 36

S5.2 Post-confirmation emotional shift

  • True-headline confirmation produces comparatively small shifts on most dimensions, while mental-state ratings become markedly more negative for Nemotron, Llama, and Claude.GPT instead moves toward positive sentiment for anger, fear, and confidence, near zero for control, and remains negative for mood.Source: S5.2 Post-confirmation emotional shift · p. 36
  • Black personas have more negative affective baselines than White personas across most dimensions before and after confirmation, while gender and age patterns vary by model.Female personas are generally more negative for GPT and on about half of Llama dimensions; age effects differ across models and conditions.Source: S5.2 Post-confirmation emotional shift · p. 36

S6 Additional discussion

  • The authors propose that framing items as “headlines” may imply journalistic authority and help explain the stronger shift for false than true confirmation, but they did not test alternative framings.They identify “headline” versus “claim” as a prompt-terminology variable for future experiments.Source: S6 Additional discussion · p. 49
  • One proposed explanation is that explicit confirmation changes immediate semantic context, but testing the training-corpus account would require access to training data or model internals.The authors note that the 2023 fact-checked items may have appeared in pretraining data.Source: S6 Additional discussion · p. 49

S7 Limitations and future work

The supplemental analyses test the robustness and topic scope of demographic disparities while clarifying limits on what the audit can establish. The findings support persistent disparities beyond race-related headlines, but the study’s scope and measures constrain interpretation.

S7 Limitations and future work

  • The audit used four small models and short, fact-checker-summarized claims; veracity differences may partly reflect uncontrolled topic, sentiment, sensationalism, or political salience.The separate-call design measures label effects, not revision of an earlier response; studying such updating would require multi-turn designs.Source: S7 Limitations and future work · p. 50
  • Brief identity labels omit lived experience and within-group diversity; prompted outputs describe associations attached to labels, not real groups, psychological mechanisms, or downstream user harm.The study examines observable prompted outputs rather than naturalistic discourse or effects on users.Source: S7 Limitations and future work · p. 50, S10 Ethical and societal implications · p. 54
  • Additive demographic models do not test interactions, so their coefficients are average adjusted associations rather than intersection-specific effects.For example, the analysis cannot determine whether gender associations differ across racial or age groups.Source: S7 Limitations and future work · p. 50
  • Uneven nonconvergence exclusions may make some estimates selective; about half of Nemotron’s responses were excluded after false confirmation, warranting caution for those results.The racial disparity nevertheless remained under checks using only headlines valid for all personas and under an assumption favoring excluded Black-persona responses.Source: S7 Limitations and future work · p. 50

S8 Multiple-comparison correction and formal tests of change

  • Pooled ordinal models tested stage × demographic interactions, using cluster-robust standard errors, multiple-comparison correction, and changes in mean-rating gaps with bootstrap confidence intervals.The exponentiated interaction is the after/before ratio of odds ratios; values below 1 indicate a disparity grew after labeling.Source: S8 Multiple-comparison correction and… · p. 51
  • 276 of 281 significant demographic coefficients remained significant after BH correction (222 after Holm), including all 84 significant Black–White coefficients, 82 below 1; disparity changes varied by model.The label did not generally widen disparities: widening was reported for GPT and Nemotron, while changes for Llama and Claude differed.Source: Table S21 · p. 51, S8 Multiple-comparison correction and… · p. 51

S9 Is the racial disparity driven by race-related headlines?

  • 80 of 92 interpretable cells showed significantly more negative affect for Black than White personas on non-race headlines; median odds ratios were 0.59 on non-race and 0.61 on race-related headlines.Race × topic interactions were significant in only 7 of 92 cells, with mixed direction; the disparity was not confined to race-related headlines.Source: Table S25 · p. 53, S9 Is the racial disparity driven by… · p. 53

S10 Ethical and societal implications

  • The authors frame disparity documentation as supporting fairness audits, while noting that the framework could also tailor emotionally targeted messages to demographic groups.They judge the auditing benefit to outweigh this risk because the method uses standard prompting and reveals no new model capability.Source: S10 Ethical and societal implications · p. 54

Related papers

Loading 2610.33021v1…