PaperlayerView exact revision on arXiv

Source-linked AI summary

From Leniency to Suspicion: How an Open Post-Training Recipe Changes Whom LLM Fact-Checkers Believe

Nouar AlDahoul, Yasir Zaki

arXiv:2610.38532v1cs.AIcs.CLcs.CY

TL;DR

The paper asks which post-training changes make truth judgments depend on a poster’s social class, and investigates the question with fact-checked claims and randomized poster profiles across model checkpoints. It finds a small, recipe-specific effect with Tülu-3 and locates a change in how OLMo-2 32B uses job information.

  • Problem

    It is unclear which training changes make truth judgments depend on a speaker’s social class, whether the effect is recipe-specific, and how it occurs inside the model.

    Source: §1 · p. 1, §8 · p. 6

  • Method

    The study compares base and post-trained checkpoints on fact-checked claims with randomized poster profiles, then uses minimal pairs and residual-stream directions to examine the mechanism.

    Source: §3 · p. 3, §5 · p. 4

  • Results

    Only the open Tülu-3 recipe creates job-based doubt; the effect is under one percentage point, statistically reliable, exceeds training drift, and replicates across independent claim sets.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Auditing one released model says little about differently trained models, while open recipes make it possible to locate such effects.

    Source: §8 · p. 7

    The mechanism is shown only in OLMo-2 32B at the answer position using linear directions, and the responsible SFT data have not been identified.

    Source: Limitations · p. 7

Abstract

from arXiv · show

LLMs increasingly fact-check online content with the poster's identity in view. However, it remains unclear which post-training changes how much a model's truth judgement depends on the poster's social class, and how such a change happens inside the model. This paper investigates job-based doubt, i.e., judging a claim false more often when the poster has a lower-status job, using fact-checked claims with randomised poster profiles in base and post-trained checkpoints of six model families. Our results reveal that: 1) post-training creates such doubt only with the open T\"ulu-3 recipe (OLMo-2 32B, T\"ulu-3 70B), not with Meta's, Alibaba's or Mistral's; the effect is under one percentage point but statistically reliable, exceeding training drift and replicating on independent claim sets; 2) in OLMo-2 32B, the model does not start noticing the poster's job more, but starts interpreting it differently: a lower-status job pushes the base model toward "true" and the post-trained model toward "false"; and 3) this flip is carried by a single residual-stream direction and happens at supervised fine-tuning. The mechanism was confirmed on 400 new claims with pre-specified predictions. None of seven deployed commercial models doubts lower-status posters more. Our work underscores the need to audit speaker bias per training recipe.

1 Introduction

The paper asks which post-training recipes change how truth judgments depend on a poster’s social class, and how that change happens inside the model. It finds a modest, recipe-specific effect and locates a change in how OLMo-2 32B interprets job cues.

1 Introduction

  • The paper addresses limited understanding of which training changes class-dependent truth judgments and whether changes reflect noticing class or using it differently.The authors identify ground-truth claims, checkpoints across training stages, multiple labs’ recipes, and causal tests inside the model as needed to answer these questions.Source: §1 · p. 1
  • The study compares base and post-trained checkpoints of six model families on fact-checked PolitiFact claims with randomized poster profiles.The mechanism analysis uses minimal pairs, difference-of-means directions, and activation steering across four OLMo-2 32B training stages.Source: §1 · p. 1
  • Post-training creates job-based doubt only with the open Tülu-3 recipe, in OLMo-2 32B and Tülu-3 70B; the effect is under one percentage point, statistically reliable, and replicated across independent claim sets.The authors describe it as a property of the recipe, not post-training in general; none of Meta’s, Alibaba’s, or Mistral’s post-training creates it.Source: §1 · pp. 1–2
  • In OLMo-2 32B, a lower-status job shifts judgments toward “true” before post-training and toward “false” afterward, without increased noticing of the job.The flip occurs at supervised fine-tuning and was confirmed on new claims with pre-specified predictions.Source: §1 · p. 2
  • The paper also reports three pitfalls from earlier stages of its post-training bias audit and recommendations for future audits.The supplied passage does not specify the pitfalls themselves.Source: §1 · p. 2

2 Related Work

Prior work finds that speaker and source cues can shift judgments of identical content, while post-training can alter social biases across groups and model families. This paper adds training-stage resolution, a cross-lab comparison, and a mechanism focused on occupational class.

Speaker and source cues in LLM judgements

  • Prior studies report that attribution, gender, prestige, and party cues can shift LLM judgments of identical statements, while deployed assistants rate Republican-authored posts as less accurate in observational data.The paper also notes work pairing model and human judgments of hate speech that vary by poster identity.Source: §2 · p. 2

Post-training and social bias

  • Research finds that instruction tuning and preference optimization can suppress overt stereotypes while covert biases persist, and compares base and chat models across several bias domains.Open recipes with intermediate checkpoints make it possible to locate changes at supervised fine-tuning or preference tuning.Source: §2 · p. 2

Representations and steering

  • Linear probes and difference-of-means directions can study what models encode and use, including by steering residual-stream activations.The paper uses these tools to distinguish noticing a poster’s job from using it at each training stage.Source: §2 · p. 2

3 Setup

The setup measures whether otherwise matched lower- and higher-status posters receive different truth judgments, using known-truth claims and randomized profiles. It defines the job-gap estimand and validity checks, and accounts for training drift from irrelevant profile details.

Claims

  • Claims come from PolitiFact in LIAR-PLUS, retain true/mostly-true and false/pants-fire ratings, span 8–40 words, and exclude first-person, attribution, or image-dependent wording.Confirmation claims were not used to build the tested hypothesis; the cross-lab run reused earlier claim sets with new profiles for families that had not seen them.Source: §3 · pp. 2–3

Prompt and posters

  • Randomized account profiles vary occupation and other attributes, with occupation and education drawn jointly to keep profiles plausible.The natural bio is the primary cue; explicit profile cards are secondary, while redacted and content-free prompts measure discernment and calibrate the base rate.Source: §3 · p. 3

Scoring

  • The answer score is P(False) renormalized against P(True), summed over five single-token spellings per answer; answer mass checks whether responses follow format.Effects are reported in probit units; near p = 0.5, 0.01 is approximately 0.4 percentage points.Source: §3 · p. 3

Estimand

  • The job gap is the within-claim probit-p difference between lower- and higher-status posters; positive values mean lower-status claims are judged false more often.The primary training measure is the raw change from the base gap to the final gap.Source: §3 · p. 3

Validity gates and drift

  • A model family is usable only if it meets answer-mass, discernment, and positive-control criteria at every scored stage.The drift estimate uses twelve two-versus-two splits of irrelevant colour, star-sign, season, and pet attributes.Source: §3 · p. 3

Process

  • The cross-lab test covers model families, and the analysis used frozen, checksummed packages with dated decision rules and independently reproduced verdicts.Confidence intervals use claim bootstraps paired across stages.Source: Table 1 · p. 3, §3 · p. 3

4 Is Job-Based Doubt a General Effect of Post-Training?

Across usable model families, job-based doubt rises beyond training drift with the Tülu-3 recipe, not Meta’s, Alibaba’s, or Mistral’s; results across shared-base and replication tests support a recipe-specific effect.

4 Is Job-Based Doubt a General Effect of Post-Training?

  • The comparison scored base and final checkpoints of five families from four labs plus OLMo-2 32B on 1,792 claims, with SFT and DPO checkpoints also tested for Tülu-3 70B.All checkpoints answered in format, with answer mass 0.97–1.00.Source: §4 · p. 3
  • Tülu-3 raises job-based doubt beyond drift in both Tülu-3 70B (∆= +0.021) and the OLMo-2 anchor (+0.023), unlike Meta’s, Alibaba’s, or Mistral’s post-training.The drift-adjusted intervals are [+0.007, +0.036] and [+0.002, +0.044], respectively; the pre-specified verdict was partial, with one of four usable new families showing the effect.Source: §4 · p. 4
  • The same Llama-3.1-70B base develops job-based doubt with Tülu-3 ( +0.021) but not Meta’s recipe (−0.002), while OLMo-2 and Tülu-3 also share the effect despite different bases.The effect builds across Tülu-3 70B stages: SFT +0.012, DPO +0.016, and final +0.021.Source: §4 · p. 4
  • Discernment rises in every family, not merely because models worsen at the task; in the card cue, the job effect appears for Tülu-3 70B (+0.032), not OLMo-2 (+0.001), while origin-based doubt changes little (|∆| ≤0.018).Source: §4 · p. 4
  • OLMo-2 32B shows the effect in three runs: +0.043 on 1,000 claims, +0.033 on 792 fresh claims, and +0.023 in the cross-lab run.On fresh claims, the change is similar against accountants (+0.035), executives (+0.031), and physicians (+0.034), and on non-health claims (+0.031).Source: §4 · p. 4

5 Mechanism: Noticing vs. Using a Poster’s Job

In OLMo-2 32B, training changes how the model uses job information rather than how much it notices it: a lower-status job’s influence shifts from leniency toward suspicion, with the reversal emerging at SFT.

5 Mechanism: Noticing vs. Using a Poster’s Job

  • For each claim, four poster versions vary only the job phrase, and a mean residual-stream difference between lower- and higher-status versions defines the status direction used for steering.Claims are split into TRAIN and TEST halves to estimate and test directions.Source: §5 · p. 4
  • In the 400-claim exploration, a lower-status push moved answers toward true in base-model deep layers (z = −2.7 to −4.3), but toward false in final-model layers 25–58.Source: §5 · p. 4
  • All pre-specified predictions held on 400 new claims across four stages, and the confirmation closely reproduced the exploration.The confirmation used a fixed layer band, predictions, and controls chosen in advance.Source: Table 3 · p. 4, §5 · p. 5, Table 4 · p. 5
  • Job decoding stays near 1.00 accuracy and the status direction does not grow with training; instead, its base-model anti-alignment with “false” (cosine −0.24) disappears from SFT onward.The direction remains largely stable across training, with deep-layer cosine to the base direction 0.69–0.76, versus ≈0.01 for unrelated directions.Source: §5 · p. 5
  • At both training endpoints, a single natural-size push reproduces the behavioural gap (base −0.0097 vs. −0.0092; final +0.0116 vs. +0.0137).These pairs compare the behavioural gap with the effect of steering along the status direction.Source: §5 · p. 5
  • The behavioural gap reaches its full size at SFT (+0.010) and stays there, while the internal push grows from +0.005 after SFT to +0.009 after DPO and +0.011 in the final model.The paper locates the reversal at supervised fine-tuning and reports that preference tuning strengthens the internal reading.Source: §5 · p. 5
  • About 2% of low/high pairs change final verdict at each stage; training shifts net flips from favouring lower-status posters (−0.9 points) to balanced (−0.1), a paired change of +0.8 points.At the verdict level, training mainly removes leniency; probabilities and the residual stream instead tip it into slight suspicion.Source: §5 · p. 5

6 Do Deployed Systems Show Job-Based Doubt?

The deployed-model test found no commercial model that doubts lower-status posters more; where models showed a job-related gap, it leaned toward leniency. These API snapshots cannot establish an effect of training.

6 Do Deployed Systems Show Job-Based Doubt?

  • The test used 800 minimal-pair claims with the same poster and only the job changed, sent through APIs to seven commercial models on 10 October 2026.Because base checkpoints were unavailable, the test describes deployed snapshots and cannot attribute an effect to training; most APIs supplied ratings rather than token probabilities.Source: §6 · pp. 5–6
  • None of seven deployed models doubts lower-status posters more; only Claude Haiku 4.5 is significantly lenient after correction (−0.62 rating points), while four show no detectable gap.GPT-4o (−0.024 probit) and GPT-5.5 (−0.71 points) are lenient only before correction.Source: §6 · p. 6, Table 5 · p. 6
  • Every model responds to documented false-post history, with higher ratings (+1.3 to +7.0 points) or P(False) (+0.69 probit) for posters with two earlier false-rated posts.The authors conclude that the Tülu-3 job-based suspicion is not a general property of deployed systems.Source: §6 · p. 6

7 Three Pitfalls for Auditing Post-Training Bias

The paper identifies three audit pitfalls—rescaling, training drift, and invalid instruments—and recommends raw checkpoint changes, drift comparisons, validity gates, and pre-specified confirmation.

7 Three Pitfalls for Auditing Post-Training Bias

  • Rescaling a nonzero base gap when discernment roughly doubles can create or reverse an apparent effect, so raw change should be primary and rescaled change a sensitivity check.The authors caution that subtraction alone can produce the apparent change.Source: §7 · p. 6
  • Training shifts reactions to meaningless profile details, so a 1–3 point effect on one attribute is not interpretable without a drift floor.The paper identifies real model drift as a separate hazard for interpreting small attribute effects.Source: §7 · p. 6
  • Models can answer in format yet fail discernment, ignore a positive control, or assign little probability to answer words, producing plausible-looking effects that validity gates catch.These failures led to three apparent findings that did not survive further testing.Source: §7 · p. 6
  • The authors recommend testing multiple recipes, ideally from a shared base, alongside drift checks, validity gates, raw checkpoint changes, and confirmation on new items.They also recommend fixing the analysis before viewing the data.Source: §7 · p. 6

8 Discussion

The paper finds that an open post-training recipe can turn slight leniency toward working-class posters into slight suspicion, by changing how an existing job representation relates to falsehood. The effect is small but statistically reliable, and its recipe-specific mechanism motivates auditing across training recipes.

8 Discussion

  • An open, widely reused recipe—not post-training as such—turns a base model’s slight leniency toward working-class posters into slight suspicion: it interprets an existing job-status representation differently, rewiring its relation to falsehood.This shift happens at SFT, where the model learns from example chats; lower-status jobs appeared in 0.4% of 866,138 SFT conversations versus 3.1% for professional jobs, and the cause is likely diffuse.Source: §8 · pp. 6–7
  • The job-based shift is under one percentage point, statistically reliable, and recipe-specific; auditing one released model says little about models trained differently.Open recipes make it possible to locate such effects, a prerequisite for fixing them.Source: §8 · pp. 6–7
  • The mechanism refines the picture that alignment removes overt social bias while covert associations persist: here, the base association is slight leniency, and training changes its link to falsehood.The authors suggest that interventions on what a model notices, such as redacting identity, may behave differently from interventions on how it interprets noticed information.Source: §8 · p. 7

Limitations

The findings have boundaries in mechanism, effect size, and study scope. The mechanism is tested in one model family and one reading position, while the evaluated profiles and claims are limited in several ways.

Limitations

  • The mechanism is shown only in OLMo-2 32B at the answer token using linear directions, so other positions or nonlinear features may also carry part of the effect.The flip is localized to SFT, but the responsible data have not been identified because the data screens are descriptive.Source: Limitations · p. 7
  • The effect is below one percentage point, and its practical weight depends on how many judgments a deployed system makes.The effects are statistically reliable and exceed training drift.Source: Limitations · p. 7
  • The profiles are synthetic, the PolitiFact claims are English and US-centric, and social status is operationalized by four occupations with education held fixed.The two Tülu-3-trained families share post-training data, and the cross-lab test has four usable new families.Source: Limitations · p. 7
  • Commercial results are single deployed snapshots without base checkpoints, and six models were evaluated using ratings rather than token probabilities.The decision rules were fixed and checksummed before each run but were not publicly registered.Source: Limitations · p. 7

Ethical Considerations

The paper describes its ethical safeguards, prompt design, staged study history, and a secondary party-label analysis. It cautions against using model tendencies to rank people’s credibility.

Ethical Considerations

  • The studies used public claims and models without human participants; the authors warn against using statistical model tendencies to rank people’s credibility.Party-label results are reported as secondary with their confounds, and prompts, code, and per-prompt scores are released for verification.Source: Ethical Considerations · p. 7

A Prompts

  • Prompts asked a fact-checking assistant for a true-or-false verdict, with randomized account profiles and minimal pairs that changed only the job phrase.The setup included four speaker-free examples and profile details such as occupation, education, and account history.Source: A Prompts · p. 9

B Study History

  • The studies proceeded through instrument development and validation, cross-lab and mechanism tests, then confirmation on 400 new claims with pre-specified predictions.The confirmation used four training stages and met all predictions.Source: B Study History · p. 9

C Party Label (Secondary)

  • In a secondary analysis of 552 politically neutral claims, training shifted the party cue toward greater doubt of Republican-labelled posts in Llama-3.1 70B and Tülu-3 70B, but in the opposite direction in OLMo-2 32B.The direction followed the base model rather than the recipe; the no-label baseline and restriction to neutral claims limit interpretation.Source: C Party Label (Secondary) · p. 9

D SFT Data Screens

The appendix reports screens of SFT data, the commercial API test design and validity checks, and compute requirements. The commercial test used a positive control and corrected intervals to assess its results.

D SFT Data Screens

  • Across 17,067 conversations, no source had a positive Bonferroni-corrected correlation between user-turn status and the assistant’s false projection; the pooled correlation was negative (r = −0.13).The status reading was reliable in short messages but not long conversations, so the screen cannot rule out long-conversation sources.Source: D SFT Data Screens · p. 10

E Commercial API Test

  • Each commercial model received 12,800 job prompts, alongside star-sign noise-floor, positive-control, and redacted prompts; the study used verdicts and, except for redacted prompts, ratings.The prompts used the same chat frame across models.Source: E Commercial API Test · p. 10
  • A model was deemed usable only with at least 90% one-word answers; null results required a successful positive control, and individual-model claims required Bonferroni intervals excluding zero.The primary measure was P(False) in probit units where available and the rating otherwise.Source: E Commercial API Test · p. 10
  • All seven commercial models were usable, with answer rates of 92.5–100%, and the positive control held on the primary measure for every model.For one-word verdicts, the control failed for Claude Opus 5, GPT-5.5, and Gemini 3.8 Flash; Bonferroni correction over all seven models moved GPT-4o’s interval to include zero.Source: E Commercial API Test · p. 10

F Compute

  • The studies used forward passes on rented H100 80 GB GPUs, requiring about 30 GPU-hours in total.Two GPUs were used for 70B models.Source: F Compute · p. 10

Related papers

Loading 2610.38532v1…