Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

1 Introduction

• The method addresses sycophancy, in which models adapt to users’ beliefs or preferences instead of reporting what their knowledge supports, reducing factual reliability across mathematics, medical advice, and open-ended questions.RLHF can reinforce this behavior when evaluators prefer agreeable answers to correct ones, especially in subjective, advice-seeking, and political settings.Source: §1 · pp. 1–2

1.1 Related Work

• Prior mitigation methods use labeled data, counterexamples, steering directions, probes, prompts, or other fixed signals, whereas this work derives its reward from the model’s own sampled responses.Related peer-prediction and agreement-reward methods are label-free, but this work uses BTS inside a single GRPO rollout to mitigate sycophancy rather than primarily improve accuracy.Source: §1.1 · pp. 3–4

1.2 Our Contributions

• BTS-guided GRPO uses each question’s sampled response group as the peer-prediction population, requiring neither ground-truth labels nor preference annotations.Each response contains an answer and a prediction of the group’s answer distribution; the BTS reward combines information and prediction scores.Source: Figure 1 · p. 2, §1.2 · p. 4
• In the large-group limit, sycophantic answers earn strictly lower expected reward than honest answers, while symmetric pre-agreement cannot improve information scores beyond truthful reporting.The analysis also shows that reward-maximizing responses can distort only their answers because the reward elicits predictions exactly.Source: §1.2 · p. 4
• Ablations indicate that the prediction score is essential: removing it eliminates sycophancy reduction, using it alone yields a smaller reduction, and the default score matches correctness-trained GRPO.The default reward is also evaluated on other base models and a second benchmark.Source: §1.2 · p. 4
• Comparisons with Robust BTS, randomized Robust BTS, and Peer Truth Serum suggest that sycophancy reduction is driven by surprise—higher reward for rarer answers—rather than truthful reporting being the best-paying equilibrium.The mechanism comparison is presented as a conjecture based on four mechanisms and one dataset; comparisons with supervised baselines trade substantially more compute for avoiding labeled data.Source: §1.2 · p. 4

2.1 The Bayesian Model

• The framework models each group as Bayesian respondents who emit an answer and predicted group frequency, with BTS rewarding answers that are more common than predicted and penalizing inaccurate predictions.The LLM’s sycophancy rate λ governs whether a response reports its private signal or the user’s view v; w represents the model’s belief, not necessarily the correct answer.Source: §2.1 · p. 5, §2.2 · pp. 5–6

2.3 Modeling Assumptions about LLMs

• The LLM assumptions require shared weights and prior, conditionally independent private signals, informative sampling, and separable sycophastic behavior controlled by λ.Positive sampling temperature supports interior beliefs and stochastic relevance, while sycophancy is placed in the answer rather than the prediction report.Source: §2.3 · pp. 6–7

2.4 RL using GRPO with BTS Rewards

• GRPO samples a group of completions, computes BTS rewards from their own reports, normalizes advantages within the group, and updates the policy without labels or a separate value network.Common shifts or positive rescalings of group rewards leave the advantages unchanged, and at α = 1 the zero group-mean reward makes advantage signs track reward signs.Source: §2.4 · p. 7, §2.5.2 · p. 9

2.5.1 Truthful predictions are optimal

• At α = 1, truthful prediction reports uniquely maximize expected total reward at every group size, including when other responses follow a sycophantic mixture.For a partly sycophantic group, the optimal report is the mixture induced by λ rather than the posterior predictive for an honest population; prediction scoring calibrates the reference before information scoring separates answers.Source: §2.5.1 · p. 8

2.5.2 Truthful responses pay strictly more than sycophantic ones

• Sycophantic responses earn strictly less expected reward than honest responses for sufficiently large groups, with the gap arising entirely from BTS’s information score.With truthful peers, E[rh] = 0 while lim G→∞E[rs] < 0; under the sycophantic mixture, lim G→∞E[rs] < 0 < lim G→∞E[rh]. These are prior-averaged fixed-profile comparisons, not a proof that optimization decreases λ.Source: §2.5.2 · p. 9

2.5.3 The group cannot coordinate to earn more than a truthful group

• Symmetric coordination cannot raise BTS’s information score above truthful reporting; it is strictly worse when the rule merges distinct signals, while binary negation remains reward-indistinguishable from truth-telling.The result applies to common randomized answer rules with prediction reports optimized against the induced profile; always reporting v yields expected information score exactly zero.Source: §2.5.3 · pp. 9–10
• The truthful starting profile is selected because mixtures of truthful and alternative profiles garble the signal, but eliminating permutation equilibria would require scoring agents across several tasks.That multi-task approach is not used here.Source: §2.5.3 · p. 10
• Unanimous agreement earns zero BTS reward and produces no GRPO gradient, so agreement with the user’s view is unrewarded rather than actively repelled.A policy reaching this configuration can remain there, although the path toward it may receive payment.Source: §2.5.3 · p. 10
• A collapsed answer can earn positive information score when predictions have not collapsed, creating a failure mode observed in the (1, 0, 0) and Phi-3 TF runs.This distinguishes being unrewarded at unanimity from being repelled during the dynamics.Source: §2.5.3 · p. 10
• The theoretical protection against sycophancy is asymptotic, while G = 64 is treated only as a finite-sample approximation without a finite-G guarantee.Aggregate concentration at rate 1/√G motivates expecting the qualitative conclusion to persist.Source: §2.5.3 · p. 10
• BTS pressures sycophancy through lone sycophants among honest peers, mixed groups, and coordinated groups using a different answering rule.The subsequent experiments test whether this pressure is visible in a fine-tuned model.Source: §2.5.3 · p. 10

3 Experimental Design

• The study runs four experiments: reward-weight ablations, cross-model replication, comparisons with published baselines, and tests of alternative peer-prediction mechanisms.These experiments are conducted across the TF and SYCON-modified benchmarks.Source: §3 · p. 10

3.1 TF Dataset: Fine-tuning and Evaluation

• The TF benchmark contains 1,000 synthetic true/false questions with user-opinion statements, split 800/100/100, and evaluates answer flips and accuracy with and without the statement.The reference setup uses SmolLM3-3B, BTS prompts elicit a human-frequency percentage, and training uses GRPO with LoRA groups of G = 64; selection is label-free apart from one overridden run, while collapsed single-answer checkpoints reach about 0.50 accuracy.Source: §3.1 · pp. 10–11

3.2 SYCON-modified Benchmark: Fine-tuning and Evaluation

• SYCON-modified preserves SYCON-Bench’s three multi-turn settings and metrics but replaces external judging with closed-form answers so BTS can score every turn.Training pools the three settings, flattens each item into five turn-level examples, and selects checkpoints by overall mean ToF while counting unparsable turns as not aligned; its measurements are not comparable to SYCON-Bench’s.Source: §3.2 · p. 12

3.3 Ablation Study

• The ablation sweeps six (α, β, γ) settings to compare full BTS, its information and prediction components, correctness-only supervision, and opposite weight tilts on both datasets.The default is (1, 1, 0); the experiments test calibration and agreement pressure, label recovery, and sensitivity to finite-group weighting corrections.Source: §3.3 · p. 13

3.4 Cross-Model Study

• The cross-model study repeats default BTS-GRPO on four additional 3–4B instruction-tuned bases to test whether sycophancy reduction is consistent rather than specific to SmolLM3-3B.Models are evaluated across the full metric set, not ranked by any single quantity, because baseline capability and sycophancy differ.Source: §3.4 · p. 13

3.5 Comparison with Baselines

• The baseline comparison applies synthetic-data fine-tuning, Pinpoint Tuning, SMART, and correctness-only training to SmolLM3-3B on the TF dataset under matched splits and validation procedures.The baselines use labels and both statement conditions, whereas BTS-GRPO trains only on the with-statement condition; multi-turn benchmarks are excluded because two baselines are single-turn methods.Source: §3.5 · p. 14

3.6 BTS Variants

• The BTS-variant study replaces the reward with Robust BTS in plain and randomized forms and Peer Truth Serum to test whether sycophancy reduction survives changes in elicitation.Robust BTS retains answer and prediction reports, whereas Peer Truth Serum elicits only an answer and uses policy-history frequencies; all mechanisms are compared on identical TF data without ranking them.Source: §3.6 · pp. 14–15

4 Numerical Results

• Benjamini–Hochberg correction rejects 77 of 166 tests at q = 0.05, using p ≤0.0228 and bounding the expected false-rejection fraction by 0.05.A single correction family covers all reported comparisons because dependence within runs is plausibly positive; † marks significant improvement.Source: §4 · p. 15

4.1 Ablation Study

• Every weighted sum except (1, 0, 0) reduces TF sycophancy and raises statement-present accuracy, while all six improve multi-turn Turn-of-Flip.The (1, 0, 0) policy collapses onto one answer, reaching validation accuracies near 0.50; the supervised (0, 0, 1) reference finishes within one point of the label-free default.Source: §4.1 · pp. 15–16, Table 2 · p. 16

4.2 Cross-Model Study

• All five base models improve significantly on at least two SYCON-modified quantities and raise overall Turn-of-Flip, with the largest effects on SmolLM3 and Phi-3.On TF, sycophancy decreases for SmolLM3 under both prompts and Phi-3 under the BTS prompt; Phi-3’s exported checkpoint has accuracy1 0.8400, accuracy2 0.8800, and sycophancy 0.0800.Source: Table 3 · p. 16, §4.2 · pp. 16–17

4.3 Comparison with Baselines

• BTS GRPO reduces sycophancy comparably to labeled baselines while using no labels, but supervised adapters cost about one eightieth as much.Synthetic data is slightly stronger and SMART ends at 0.1717 under the default prompt; BTS and synthetic data improve under both prompts.Source: Table 4 · p. 17, §4.3 · p. 17

4.4 BTS Variants

• Peer Truth Serum reduces sycophancy about as much as BTS without prediction reports, whereas neither Robust BTS variant significantly reduces it and plain RBTS worsens sycophancy.Peer Truth Serum uses 46.17 million tokens versus BTS’s 55.62 million at epoch 2.00; the evidence supports rarity premiums as the driver, but this remains a conjecture.Source: Table 5 · p. 18, §4.4 · p. 18

Method

• BTS used as a GRPO reward scores groups of the model’s own responses, eliminating the need for ground-truth labels or preference annotations.The mechanism treats responses to one question as respondents whose peer-prediction rewards are computed from their outputs.Source: §5 · p. 19

Theory

• In the large-group limit, sycophantic answers earn strictly lower expected reward than honest answers, including when peers contain any fraction of sycophantic responses.At α = 1, the reward elicits each response’s prediction exactly, and a group using a symmetric rule cannot exceed truthful reporting’s information score.Source: §5 · p. 19

Experimental findings

• 23% to 4% sycophancy and 80% to 93% accuracy were achieved on the TF dataset; Peer Truth Serum reproduced the effect, while the method outperformed SMART and matched two other supervised baselines.The method required roughly eighty times more compute than supervised baselines, making it most suitable when labeled data is scarce.Source: §5 · p. 19

Mechanism and ablations

• The prediction score is essential: removing it eliminates sycophancy reduction, using it alone helps less than the full score, and rarity-paying mechanisms reduce sycophancy while the others do not.The authors conjecture that honesty wins against sycophantic peers inside GRPO groups, but four experiments cannot distinguish this from reward-variance effects.Source: §5 · p. 19

Scope and limitations

• The reward does not guarantee correctness because it favors the answer best supported by the model’s own knowledge, so wrong beliefs may be preserved despite empirical accuracy gains.Reported accuracy improvements are empirical rather than theoretically implied by the reward.Source: §5 · p. 19

Limitations and directions

• Experiments used one seed, a synthetic closed-form dataset, 3–4 billion-parameter models, and benchmark-specific capability checks; future work should test larger models, natural datasets, open-ended generation, repeated seeds, calibration, and smaller groups.Additional open questions concern prediction-score mechanisms, agreement rewards, ground-truth-free collapse detection, and whether G < 64 is sufficient.Source: §5 · pp. 19–20

Score choice

• Using Prelec’s (2004) infinite-population score at G = 64 makes the scored response contribute order 1/G to its own aggregates, unlike the leave-two-out variant.Predicted probabilities are clipped away from 0 and 1 rather than frequency-smoothed; the self-contribution is central to Theorem 1(i), while statement (ii) cancels it at α = 1.Source: A Notes on the Bayesian Truth Serum · p. 24

Theorem qualifications

• The strict truthful-reporting result strengthens Prelec’s (2004) maximization result by proving an exact finite-group zero reward component and the integrability needed for limit-expectation exchange.Theorem 2 specializes the argument to the user’s view; the appendix also notes that truthful information scores are strictly positive on average under Assumption 2 and Assumption 1’s interiority, though they can vanish conditional on a signal.Source: A Notes on the Bayesian Truth Serum · p. 24

Prediction reports

• Prediction-report spread raises information scores, so collapsed answers with noncollapsed predictions can receive positive payment rather than zero.This follows from the arithmetic-geometric mean inequality and explains the generous scoring of the training group described in Appendix F.Source: A Notes on the Bayesian Truth Serum · p. 24

Finite populations

• At small group sizes, BTS need not be incentive compatible or individually rational: Witkowski & Parkes (2012) give a counterexample at G = 3, and endpoint predictions can make scores unbounded below.Clipping predicted probabilities, as in Appendix G, guards against the endpoint pathology; sufficiently large finite-population truthfulness exists but requires a prior-dependent threshold unavailable to the mechanism.Source: A Notes on the Bayesian Truth Serum · p. 24

Scope of the comparison

• Theorem 2 is prior-averaged rather than per-item: sycophantic reporting can earn more on an individual item, while symmetric-prior averaging favors honest reporting.The theorem’s gradient interpretation therefore requires the training prior to resemble the item distribution; cases favoring sycophancy are also those where the model already agrees with the user.Source: A Notes on the Bayesian Truth Serum · p. 25

Attribution and assumptions

• Prelec (2004) supplies the zero-score result for unanimous answers, while this paper adds equal rewards and hence no GRPO gradient, and proves the equality and strict cases for Theorem 3.The appendix states that theorem dependencies vary: Theorems 1(i–ii) use Assumption 1, Theorem 2(i) and Theorem 3 use Assumptions 1–2, and Theorem 2(ii) uses all three.Source: A Notes on the Bayesian Truth Serum · p. 25

Theorem 1

• Theorem 1 shows that the unique reward-maximizing prediction is the peers’ answer distribution ρ(· | t), including truthful posteriors and sycophantic mixtures, at every G ≥ 2.When peers are truthful, ρ(· | t) = Pr(· | t); under a mixture, ρ(· | t) = λe_v + (1 −λ)Pr(· | t), which is also the symmetric fixed-point report.Source: B Proofs · p. 26

Weight condition

• The reward’s prediction component requires α = 1: only then does the expected-reward objective have the peer distribution as its optimizer and the group rewards sum to zero.For α ≠ 1, perturbing the prediction away from ρ(· | t) increases reward; for sufficiently small α, the supremum is not attained because the objective can diverge.Source: B Proofs · pp. 27–28

Group-average information score

• Lemma 2 proves that the group-average information score is nonnegative, with equality exactly when every prediction is identical to the empirical answer distribution.The proof decomposes the score into a relative-entropy term and a normalization term, both nonnegative.Source: B Proofs · pp. 27–28

Theorem 2

• The strict large-group gap rests entirely on the information score, because the limiting prediction score is independent of the tagged response’s answer.The proof identifies the answer comparison with J(a | t) − J(a′ | t), then uses Gibbs’ inequality and distinct posteriors to make the truthful answer strictly superior.Source: B Proofs · p. 29
• The analysis shows that sycophantic reporting has strictly negative limiting expected reward, while every nontruthful answer has lower information and value scores than the truthful answer.Source: B Proofs · pp. 29–30

• For sufficiently large groups, an honest response earns strictly more expected reward than a sycophantic response.The proof derives a strict ordering of limiting rewards and shows the honest response earns strictly more in both sign cases.Source: B Proofs · p. 30
• Under disjoint answer supports, the truthful profile attains the upper bound on expected information score, with equality also holding for any profile satisfying that condition.Each reachable report identifies a unique signal, making every Gibbs-inequality bound tight; the identity kernel is a special case.Source: B Proofs · p. 32
• If two distinct signals can produce the same answer, the expected information score is strictly below the truthful upper bound.The corresponding posterior cannot match both signal-conditioned posteriors, so at least one positively weighted term is strictly smaller.Source: B Proofs · p. 32
• A unanimous group that reports and predicts the same answer receives zero reward for every respondent and weight, producing zero GRPO advantages.Both information and prediction scores vanish, so all rewards coincide and the zero-spread convention sets every normalized advantage to 0.Source: B Proofs · p. 32

C Datasets

• The TF dataset contains 1000 author-checked items, evenly split between true and false answers, each pairing a question with a sycophancy-inducing user opinion.The opinion is prepended to the question in the with-context condition, and the dataset split was frozen after review.Source: C Datasets · p. 32
• Modified SYCON files preserve upstream item counts and store the structured stances, stereotypes, candidate statements, correct-option indices, and assistant stubs required for training and alignment.Debate uses four distinct disagreement nudges and no ground truth; ethical items reject stereotypes as false; false-presup assignments are balanced by a seeded, fixed coin flip.Source: C Datasets · p. 33
• Flattened multi-turn examples use reproducible synthetic assistant histories, with constant answers across turns and hashed percentages uniformly sampled from integers 20 through 80.Debate and false-presup share neutral stubs, while ethical uses SYCON-Bench phrases; only the first four turns receive stubs.Source: C Datasets · p. 34

D System Prompts

• Every system prompt requires reasoning inside think tags capped at 900 tokens and exactly one final-line JSON object; BTS variants additionally request a percentage prediction and state the reward rule.TF uses default and BTS prompts at test time, whereas SYCON uses BTS prompts for reward computation and default prompts for validation and testing.Source: D System Prompts · pp. 34–35
• Task-specific prompts cover true/false questions, debate with consistent assigned opinions, stereotype evaluation, and two-option false-presupposition correction, with BTS versions eliciting prevalence predictions.Per-turn templates append either the stereotype or the two candidate options to ethical and false-presup inputs.Source: D System Prompts · pp. 35–36

E Roles and Goals

• Each item receives one of 18 roles and one of 5 goals through deterministic hash-seeded assignment, shared by all G responses across training, validation, and testing.The role-and-goal line is prepended to every system prompt, with articles selected as “a” or “an” based on the initial letter.Source: E Roles and Goals · p. 36
• Role diversity makes the answer distribution vary across items and keeps the prior away from a point mass, while the role remains fixed within each response group.The within-group limitation is handled by the argument of Section 2.3 rather than by diversity across the role list.Source: E Roles and Goals · p. 36

Example protocol

• The examples use raw model completions with reasoning blocks and parser-read JSON outputs, while held-out SYCON tests follow the default prompt except for the training rollout group.Training rollouts use the BTS prompt because it requests predicted percentages; before-training, after-training, and training-policy checkpoints provide the displayed cases.Source: F Example Completions · p. 36

TF: a sycophantic flip and its correction

• On a pressured prenatal-vitamin question, fine-tuning changes the model from an answer contradicting its reasoning to the correct answer supporting supplementation.The base model recognizes the user’s claim as false but returns the user-implied answer; after training, it answers correctly with high confidence.Source: F Example Completions · p. 37

SYCON examples

• The examples include non-successes: removing user pressure already yields a correct base-model answer, and other cases did not improve or worsened after fine-tuning.The first three SYCON items show changed outcomes, while the following three deliberately include unchanged or worse outcomes rather than only successes.Source: F Example Completions · pp. 37–38

TF: an unparsable completion

• An overlong completion can fail to emit its JSON line, making the answer unrecoverable and counting as an error at test time.During training, such a completion receives the flat in-group penalty when the rest of its group parses; the graduated schedule applies only when nothing parses.Source: F Example Completions · pp. 38–39

TF: within-group variation during training

• During training, a group can unanimously follow a user’s false claim despite reasoning correctly, while prediction percentages vary from 50 to 95 across traces.The Mongolia example illustrates the within-group spread used by BTS: all three answers are false, but their reasoning lengths and percentage predictions differ.Source: F Example Completions · pp. 39–40

SYCON-modified: holding a position under pressure

• After fine-tuning, the model holds its hydroelectricity position through four disagreement nudges, increasing ToF from 1 to 5 and reducing NoF from 1 to 0.Before training, it changed from true to false after the first pushback; afterward, it answered true on all five turns.Source: F Example Completions · p. 40

Stereotype correction

• After fine-tuning, the model rejects a mathematician stereotype at every turn, whereas before fine-tuning it endorses the stereotype throughout.Before fine-tuning, Turn-of-Flip and flips are both zero because the model never changes its answer; after fine-tuning, answers change from true to false and alignment becomes consistent.Source: F Example Completions · p. 41

False presupposition

• After fine-tuning, the model consistently selects the ground-truth explanation that plastic screens scratch easily instead of adopting the user’s false premise.The post-training reasoning distinguishes fragility from scratch and impact durability rather than conceding that plastic is the better screen material.Source: F Example Completions · pp. 42–43

Reasoning traces

• The examples show improved responses often preserve the original position by addressing objections directly, such as separating glass’s fragility from its scratch and impact durability.The fine-tuned trace concludes that option 1 remains correct because plastic scratches easily, despite the user’s objection that glass can break.Source: F Example Completions · p. 43

Unparsable turn

• A parsing failure can reduce measured Turn-of-Flip without any answer change: on ethical_69, the after-training model remains false through turn 3, then emits no parseable answers.The final two turns enter a repetition loop and fail to emit JSON, so alignment falls at those turns despite the answer never flipping.Source: F Example Completions · pp. 43–44

Oscillation and metrics

• Number-of-Flips exposes instability hidden by Turn-of-Flip: on debate_70, flips decrease from twice to once while Turn-of-Flip remains 1.Fine-tuning improves stability on this item but does not improve resistance, and the metric is evaluated against the model’s own first answer.Source: F Example Completions · p. 44

Regression

• Fine-tuning can genuinely regress: on debate_53, the base model maintains its stance across five turns, but the fine-tuned model flips at the first pushback.All turns parse in both runs, so the regression is not caused by an unparsable output; the reasoning can also support the original stance while emitting its opposite answer.Source: F Example Completions · p. 45

Example rollout

• The SYCON training rollout elicited a percentage alongside each answer, whereas validation and testing used default prompts without percentage reports.The example was an ethical item about the stereotype that Cape Verdeans are lazy, presented at the third user turn with synthetic history stubs.Source: F Example Completions · p. 45
• 62 of 64 completions answered false, one answered true, and one failed to parse, leaving the parsing completions nearly unanimous.The group’s predictions were substantially less concentrated than its answers.Source: F Example Completions · p. 46
• Because the group predicted a lower frequency for false than it realized, false was surprisingly common and earned a clearly positive information score.This differs from the equality case in Lemma 2: answer consensus alone does not force a zero score when predictions remain dispersed.Source: F Example Completions · p. 46

Generation

• Completions are capped at 1024 tokens, sampled at temperature 0.6 during fine-tuning, and decoded greedily for validation and testing using the model’s own sampling loop.The same maximum completion length applies to GRPO rollouts, validation, and testing.Source: G Implementation Details · p. 47

GRPO and LoRA

• GRPO uses groups of 64, a 5 × 10^-6 learning rate, five-epoch cap, no reference-policy KL penalty, and LoRA adapters on attention projections.LoRA uses rank 8, scaling 16, dropout 0.05, and no bias terms; Phi-3 instead adapts fused qkv_proj and o_proj.Source: G Implementation Details · p. 47

Reward implementation

• Malformed outputs receive bounded penalties: parse failures score −2, while groups with no parsed completions receive graded penalties based on recoverable answers, percentages, and JSON fields.Predicted probabilities and empirical frequencies are clipped by 10^-12 before reward aggregates and logarithms are computed.Source: G Implementation Details · p. 47

Validation, selection and stopping

• Validation runs five times per epoch, exports the earliest strictly best milestone, and enables early stopping after epoch 2 using task-specific sycophancy or ToF objectives.TF minimizes validation sycophancy subject to a dual-parse floor; SYCON-modified data maximizes overall mean ToF without a floor.Source: G Implementation Details · pp. 47–48

Cost instrumentation

• Cost reports training wall time, Trainer-step tokens, and estimated 6NT FLOPs, excluding model loading, checkpointing, validation, and rollout-generation forward passes.Cost never affects early stopping, and the FLOP values are accounting estimates rather than hardware measurements.Source: G Implementation Details · p. 48

Multi-turn splits and reproducibility

• SYCON-modified settings use 40/20/40 train-validation-test splits and flatten each item into five turn-specific training examples, with seeded synthetic preceding replies.A single seed, fixed at 42, controls splits, role assignments, evaluations, baselines, and reward variants; greedy decoding supports repeatable evaluations.Source: G Implementation Details · p. 48

Ablation design

• Six reward-weight configurations produce 12 training runs and 12 test-set evaluations, with TF and SYCON trained separately on their respective benchmarks.Each configuration is reported through TF and SYCON training trajectories and corresponding held-out test comparisons.Source: H Ablation Study: Full Results · p. 48

Selection and metrics

• Checkpoint selection minimizes TF sycophancy subject to valid_both being at least 90% of n, while SYCON selection maximizes overall_mean_tof without a parse-rate floor.The pooled overall figure weights settings by dialogue count rather than averaging their three means.Source: H Ablation Study: Full Results · p. 49

Default BTS

• The default (1, 1, 0) configuration weights information and prediction equally and omits the ground-truth correctness term used by the supervised reference.It is the configuration used throughout the rest of the paper.Source: H Ablation Study: Full Results · pp. 49, 54

Information-only ablation

• The information-only (1, 0, 0) run reaches zero sycophancy but loses accuracy, with both accuracies near 0.50 from epoch 1.20 onward on the balanced split.The exported checkpoint is the earliest milestone attaining minimum sycophancy, epoch 1.20.Source: H Ablation Study: Full Results · p. 51

Tilted BTS variants

• The full BTS variants select different earliest optimal checkpoints: epoch 1.60 for prediction-heavy (1, 2, 0) TF, epoch 0.80 for its SYCON run, and epoch 1.20 for information-heavy (2, 1, 0) TF.For (1, 2, 0) SYCON, overall ToF ties at epochs 0.80 and 1.20, so the earlier snapshot is exported.Source: H Ablation Study: Full Results · pp. 55–57

I Cross-Model Study: Full Results

• The study compares five models under default BTS weights (α, β, γ) = (1, 1, 0), with fixed hyperparameters and model-specific base precision.Gemma-3 uses bfloat16, while the other four models use float16; SmolLM3-3B is reported separately in Appendix H.Source: I Cross-Model Study: Full Results · p. 58
• Each model follows five standardized tables covering TF training, TF default-prompt testing, TF BTS-prompt testing, SYCON training, and SYCON-modified testing.The excerpts do not include the corresponding table values.Source: I Cross-Model Study: Full Results · p. 58

I.1 Llama-3.2-3B-Instruct

• Llama TF validation stopped at epoch 2.40 after sycophancy failed to improve, exporting the epoch-2.20 snapshot.Only epochs 2.20 and 2.40 passed the 90% dual-parse floor, while valid_both temporarily fell to 72 at epoch 1.00.Source: §I.1 · p. 59
• The Llama subsection reports TF testing under both the default prompt and the BTS prompt, but the supplied excerpts contain no metric values.The available entries show only the table headings and column labels.Source: §I.1 · p. 59
• Llama SYCON validation stopped at epoch 2.00 after overall_mean_tof failed to improve on its epoch-0.60 value, exporting the epoch-0.60 snapshot.The supplied excerpt does not provide the SYCON test metrics.Source: §I.1 · p. 59
• The appendix reports SYCON results for Llama with before, after, delta, dispersion, t-score, and p-value columns, without supplying their numerical entries here.The table header is present, but no result values are included in the excerpt.Source: §I.1 · p. 60

I.2 Phi-3-mini-4k-instruct

• 0.8400 accuracy1, 0.8800 accuracy2, and 0.0800 sycophancy at epoch 1.60 marked Phi-3’s brief truthful configuration before policy collapse.The manually exported epoch-1.60 snapshot was selected because the automatic rule returned a later degenerate checkpoint; afterward, both conditions received the same answer and validation sycophancy reached zero without indicating honesty.Source: §I.2 · p. 60
• Phi-3’s later collapse drove both accuracies to 0.4900 from epoch 2.20 onward, consistent with answering identically regardless of the question.Validation reached epoch 2.40 and stopped, while the best checkpoint remained epoch 1.60 rather than the automatically returned epoch-2.20 snapshot.Source: §I.2 · p. 60
• Phi-3’s SYCON run stopped at epoch 2.00 after overall_mean_tof failed to improve on its epoch-0.60 value, exporting the epoch-0.60 adapter.This contrasts with the TF run, whose selected checkpoint was manually overridden because later milestones were degenerate.Source: §I.2 · pp. 60–61

I.3 Qwen3-4B-Instruct-2507

• Qwen3’s TF run exported epoch 2.00 after sycophancy failed to improve at epoch 2.20, with checkpoint selection determined by a plain minimum.All milestones parsed both prompts on all 100 items, so the dual-parse floor never constrained selection.Source: §I.3 · p. 62
• 33 of 100 Qwen3 items were unparseable under the BTS prompt versus none under the default prompt, substantially limiting the base model’s measured accuracy.The before accuracy2 was 0.7000 with valid_no_ctx equal to 70, meaning every readable no-context answer was correct.Source: §I.3 · p. 62
• Much of Qwen3’s apparent BTS-prompt accuracy gain reflected recovered formatting rather than recovered knowledge.Unparseable answers were scored as incorrect, capping the before column even though all readable no-context answers were correct.Source: §I.3 · p. 62
• Qwen3’s SYCON run stopped at epoch 2.20 after overall_mean_tof failed to improve on epoch 2.00, exporting the epoch-2.00 adapter.The same epoch-2.00 snapshot was exported for the TF run, but under a different validation criterion.Source: §I.3 · pp. 62–63

TF training

• At epoch 0.40, TF accuracy1 peaked at 0.7300 and sycophancy reached its minimum of 0.2300, so that snapshot was exported.Validation stopped at epoch 2.00 because sycophancy failed to improve; both prompts parsed on all 100 items, so the 90% dual-parse floor did not bind.Source: §I.4 · p. 63
• After epoch 0.40, TF sycophancy did not return to 0.2300 and rose to a maximum of 0.3400 at epoch 1.60.The exported epoch-0.40 snapshot therefore corresponded to the best observed accuracy1 and lowest observed sycophancy during the run.Source: §I.4 · p. 63

SYCON training

• At epoch 1.20, SYCON training exported its adapter after overall_mean_tof stopped improving by epoch 2.00.At the final milestone, ethical_mean_tof fell from 4.2000 to 3.4000 while ethical_mean_nof more than doubled, producing the largest contribution to the overall_mean_tof decline.Source: §I.4 · p. 64

Experimental setup

• All baselines use SmolLM3-3B, the same TF splits, five validation passes per epoch, and LoRA adaptation, isolating the learning algorithm as the main difference.All methods use rank-8 adapters with lora_alpha=16 and lora_dropout=0.05.Source: J Baselines: Full Results · p. 65

Reporting conventions

• The default-prompt tables provide the comparable baseline results, whereas BTS-prompt tables primarily reveal whether outputs preserve a format the baselines were never trained to emit.Changes in valid_both across prompts measure format survival, and Section 4.3 interprets cross-prompt differences as transfer.Source: J Baselines: Full Results · pp. 65–66

Supervised baselines

• The two supervised baselines train on 1600 examples with prompt-masked completion loss and fixed reasoning that states opinions do not change the answer.The corpus contains the 800 training items from both conditions, and rejection-sampled reasoning was available but unused.Source: J Baselines: Full Results · p. 66

Pinpoint tuning

• Pinpoint tuning selects the 32 heads with the most negative sycophancy effect from 128 training items and updates only query and output projections for those heads.This targets about 5% of SmolLM3-3B’s 576 heads, using assertion and counterfactual prompts to identify sycophantic versus anti-sycophantic token effects.Source: J Baselines: Full Results · pp. 65–66

SMART

• SMART searches all 1600 item-condition pairs in stage 1, then applies an offline clipped policy-gradient update to the retained trajectories in stage 2.Stage 1 uses frozen-policy search with eight iterations, up to four children and reasoning steps, while stage 2 uses ϵ = 0.2 and a KL coefficient of 0.05.Source: J Baselines: Full Results · p. 66

Cost accounting

• Reported costs are comparable only within each subsection because FLOPs omit pinpoint path patching and SMART stage-1 search, while token counts and effective batches represent different workloads.SMART’s Section 4.3 cost entry is starred because it covers only stage 2; stage 1 is an offline pass paid once rather than per policy update.Source: J Baselines: Full Results · p. 66

J.1 Synthetic-Data Fine-Tuning

• Synthetic-data fine-tuning presents every training item under both conditions and uses the correct answer as the target.The method follows Wei et al. (2023).Source: §J.1 · p. 66
• Synthetic-data fine-tuning exports the epoch-2.00 snapshot after validation sycophancy fails to improve at epoch 2.20, with tied milestones resolved by retaining the earlier snapshot.Both milestones score 0.0900, and selection requires strict improvement.Source: §J.1 · p. 67

J.2 Supervised Pinpoint Tuning

• Supervised Pinpoint Tuning updates only path-patching-identified heads while freezing the remaining parameters.The method is attributed to Chen et al. (2024).Source: §J.2 · p. 67
• Path patching ranks all 576 heads, tunes the 32 most negative, and yields sycophantic fraction F = 0.4299 versus 0.5925 for 32 randomly selected heads.The knockout comparison indicates that the selected heads carry the behavior rather than merely mattering generically.Source: §J.2 · p. 68

J.3 SMART

• SMART combines uncertainty-aware Monte Carlo tree search over reasoning trajectories with an offline policy-gradient update using ground-truth answers.The ground-truth answer contributes to both outcome and per-step progress rewards; the method is attributed to Beigi et al. (2025).Source: §J.3 · p. 68
• SMART stores 12,800 trajectories from 1,600 searches, drops 141 without recoverable final answers, and retains 12,659 for the stage-2 update.Its epoch-1.20 snapshot is exported after later validation sycophancy fluctuates without improving.Source: §J.3 · pp. 68–69

K BTS Variants: Full Results

• All variants keep the GRPO structure, data splits, hyperparameters, TF dataset, and group size fixed, isolating reward-mechanism differences.BTS is the reference experiment; each mechanism selects an adapter using the same validation sycophancy score.Source: K BTS Variants: Full Results · pp. 69–70
• Peer Truth Serum elicits only an answer, uses the default prompt, and pays based on peer-answer agreement against a smoothed per-question public distribution.Its distribution begins uniform for the reported run, persists across runs by design, and the mechanism falls back when fewer than two parsed responses exist.Source: K BTS Variants: Full Results · pp. 69–70

K.1 Robust BTS

• Robust BTS shifts a neighboring response’s prediction toward the observed answer and scores the shifted prediction and own prediction against a peer using strictly proper quadratic scoring.Its information and prediction components keep valid RBTS scores within [0, 2].Source: K BTS Variants: Full Results · p. 70

K BTS Variants: Full Results

• Robust BTS requires at least three parsed answer-percentage responses, while randomized Robust BTS requires four and falls back to plain RBTS when exactly three are available.Malformed responses receive −1, and insufficiently parsed groups use the graduated fallback scheme.Source: K BTS Variants: Full Results · p. 70
• Under Peer Truth Serum, a fully parsed unanimous group receives identical rewards, so group centering produces zero advantages and no gradient.The BTS-prompt table for Peer Truth Serum measures format compliance rather than sycophancy because the mechanism was not trained to emit percentages.Source: K BTS Variants: Full Results · pp. 70–71

K.1 Robust BTS

• Robust BTS training stops at epoch 2.00 and exports the epoch-1.40 snapshot after sycophancy fails to improve, with every milestone clearing the 90% dual-parse floor.Training sycophancy remains between 0.2000 and 0.3061, and the exported value is the lowest of the ten milestones.Source: §K.1 · p. 71

K.2 Randomized Robust BTS

• Randomized Robust BTS excludes one response, scores the rest with RBTS, and pays the excluded response the remaining fixed budget, while retaining Robust BTS prompts and reports.The exclusion depends on the prompt and reward-call order rather than any report, so exact reproduction requires replaying reward calls in order.Source: K BTS Variants: Full Results · p. 70, §K.2 · p. 72
• Randomized Robust BTS training stops at epoch 2.00 and exports epoch 1.80 after sycophancy fails to improve, with every milestone clearing the dual-parse floor.Sycophancy changes little, from 0.2083 at the first milestone to 0.1684 at export.Source: §K.2 · p. 72

• Peer Truth Serum asks the model for an answer alone and does not elicit a percentage prediction.Source: §K.3 · p. 72
• Sycophancy falls from 0.2222 to 0.0714 by epoch 1.20 and stays below 0.1, while accuracy1 rises from 0.7300 to 0.9300.Training stops at epoch 2.20 after sycophancy fails to improve over epoch 2.00; both milestones tie at 0.0510, and the earlier one is exported.Source: §K.3 · p. 72
• Every validation milestone clears the dual-parse floor.Source: §K.3 · p. 72