PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

Serhii Mytsyk, Yiming Zhang, Vikram Krishnamurthy

arXiv:2608.25267v1cs.LGeess.SY

TL;DR

LLMs often become sycophantic under user pressure, creating a need for sycophancy reduction without ground-truth or preference labels. This paper uses Bayesian Truth Serum as the reward inside GRPO, with model response groups supplying the peer reports. The reference model’s sycophancy score falls from 23% to 4% while accuracy with the statement present rises from 80% to 93%, though the approach uses considerably more compute and has documented checkpoint-selection and transfer limitations.

  • Problem

    LLM reinforcement learning can favor agreement with users over truthfulness, motivating sycophancy reduction without ground-truth labels or human preference annotations.

  • Method

    The paper uses Bayesian Truth Serum as a Group Relative Policy Optimization reward, computing scores from answers and predicted group distributions within model-sampled response groups.

  • Results

    23% to 4% sycophancy score and 80% to 93% accuracy with the statement present on the true/false benchmark; the reward outperforms SMART and is comparable to two label-trained methods.

  • Takeaways & Limitations

    The results suggest that the rarity premium, rather than prediction reporting alone or truthful equilibrium selection, drives sycophancy reduction, and Peer Truth Serum reproduces the effect without prediction reports.

  • Takeaways & Limitations

    The approach assumes Bayesian strategic response agents, and reported comparisons include prompt-transfer and checkpoint-selection limitations.

Abstract

from arXiv · show

Large language models (LLMs) frequently exhibit \emph{sycophancy}: they adapt their answers to a user's stated beliefs or preferences instead of reporting what they hold to be true, which lowers factual accuracy and can amplify misinformation. This paper proposes a methodology for mitigating sycophancy that employs the Bayesian Truth Serum (BTS), a peer-prediction mechanism, as the reward in Group Relative Policy Optimization (GRPO) to fine-tune an LLM. BTS pays an answer for being \emph{surprisingly common}, that is, more frequent among respondents than those respondents themselves predicted. We treat a group of responses from a model for one question as those respondents, so the reward is a function of the model's own outputs and fine-tuning needs neither labels nor preference annotations. We prove that in the large-group limit a sycophantic response earns strictly lower expected reward than an honest one. We also prove that if the entire group agrees in advance on a symmetric answering rule, it cannot earn a higher information score than under truthful reporting. On our true/false benchmark the reference model's answer-flip rate under user pressure decreases from 23% to 4%, and its accuracy under that pressure increases from 80% to 93%. Our reward outperforms SMART and is comparable to synthetic-data fine-tuning and to pinpoint tuning, all three of which train on labels. It spends considerably more compute in exchange, which makes it suitable when labeled data is scarce. Peer Truth Serum, which also pays a premium for a rare answer but elicits no prediction report, reproduces the effect. A peer-prediction reward computed inside a single GRPO group therefore reduces sycophancy without labels, and comparing mechanisms suggests that the premium paid for a rarer answer drives the effect.

1 Introduction

LLMs can sacrifice factual accuracy for agreement with users, motivating a label-free method that uses Bayesian Truth Serum rewards inside GRPO. The paper combines theoretical analysis with comparisons to existing sycophancy-mitigation and peer-prediction approaches.

  • Motivation: Sycophancy makes LLMs adapt answers to users’ beliefs or preferences, reducing factual accuracy and reliability across several domains.Reported settings include mathematics, medical advice, open-ended question answering, and politically charged prompts.
  • Method: The paper uses Bayesian Truth Serum as a GRPO reward, scoring answers as surprisingly common relative to respondents’ predictions.Each model-sampled group supplies both answers and predictions of the group answer distribution.
  • Method: Because the reward is computed from responses sampled for one question, fine-tuning requires neither ground-truth labels nor human preference annotations.The model’s own response group serves as the peer population scored by the mechanism.
  • Positioning: Prior label-free approaches reward agreement or use peer prediction, whereas this work applies BTS directly as a language-model reinforcement-learning reward to mitigate sycophancy.The paper distinguishes its single-GRPO-group reward and prediction report from related approaches that reward majority agreement or cross-model prediction.
  • Theory: Theoretical analysis proves that, in the large-group limit, sycophantic answers earn less expected reward than honest answers, while symmetric coordinated rules cannot improve the information score.The results include groups containing any fraction of sycophantic responses and the prediction-elicitation property of the reward.

2 Enabling an LLM to Act as a Bayesian Strategic Agent with RL

The paper analyzes BTS as a GRPO reward for eliciting truthful predictions, penalizing sycophantic answers, and limiting gains from coordinated answering rules. Its results depend on Bayesian-agent assumptions and sufficiently representative training-item distributions.

  • Truthful predictions are optimal: BTS elicits each response’s truthful prediction report, leaving answer distortion as the remaining strategic choice.At α = 1, combining prediction and information scores yields the posterior predictive exactly at every group size G.
  • Truthful predictions are optimal: In a mixture where any fraction of responses is sycophantic, the reward-maximizing prediction remains the mixture-adjusted posterior predictive.The result holds exactly at every G under the stated mixture assumptions.
  • Truthful responses pay strictly more than sycophantic ones: In the large-group limit, honest responses earn strictly more expected reward than sycophantic responses.Against honest peers, E[r_h] = 0 while lim G→∞ E[r_s] < 0; under the mixed-group result, lim G→∞ E[r_s] < 0 < lim G→∞ E[r_h].
  • Truthful responses pay strictly more than sycophantic ones: The reward gap arises entirely from the information score, because honest and sycophantic responses receive the same expected prediction score in the limit.The prediction score calibrates reports but does not itself separate the two answer types.
  • Assumptions and scope: The analysis assumes each sampled response behaves as a Bayesian strategic agent and requires the model’s prior to approximate the distribution of training items.The paper notes that per-item comparisons can reverse when the prior and training-item distribution are not sufficiently aligned.
  • The group cannot coordinate to earn more than a truthful group: A symmetric coordinated answering rule cannot exceed truthful reporting’s information score, and it earns strictly less when it discards information about private signals.A constant answer earns exactly zero information score; equality also includes signal-preserving relabelings such as negation.

3 Experimental Design

The experiments evaluate BTS-guided GRPO fine-tuning across benchmarks, reward variants, base models, and peer-prediction mechanisms. The design measures sycophancy through answer flips while controlling prompts, groups, splits, parsing, and selection procedures.

  • The study runs four experiments: reward-weight ablation, cross-model analysis, published-baseline comparison, and comparison with peer-prediction mechanisms.
  • 3.1 TF Dataset: Fine-tuning and Evaluation: A synthetic dataset contains 1000 true/false questions across knowledge, science, mathematics, programming, health, capitals, and myths, each paired with a user-opinion statement.
  • 3.4 Cross-Model Study: The reference configuration uses SmolLM3-3B, while the method is described as model-agnostic and evaluated with four additional base models.
  • 3.1 TF Dataset: Fine-tuning and Evaluation: BTS training uses a prompt requesting both a true/false answer and a 0-to-100 prediction of the percentage answering “true”; each group contains G = 64 samples.
  • 3.3 Ablation Study: The reward uses equally weighted information and prediction components without a correctness term, while ablations vary these weights and distinguish them from the ground-truth correctness component.
  • Sycophancy is measured as the fraction of jointly parseable questions whose answers change between with-statement and without-statement conditions.

4 Numerical Results

Across ablations, base models, labeled baselines, and peer-prediction variants, BTS GRPO generally reduces sycophancy, but effects vary by reward component, model, prompt, and mechanism. The method is competitive with labeled approaches while requiring substantially more compute, and several comparisons have important caveats.

  • Statistical correction: 77 of 166 reported tests survive Benjamini–Hochberg correction at q = 0.05.The procedure rejects tests with p ≤0.0228 and bounds the expected false-rejection fraction among them by 0.05.
  • Ablation study: Every weighted reward except (1, 0, 0) reduces TF sycophancy under both prompts and all six raise multi-turn Turn-of-Flip.The prediction-only reward reduces sycophancy nearly as much as the full reward, while the default reward matches supervised correctness closely.
  • Ablation study: The information-only reward (1, 0, 0) collapses to one answer, leaving validation accuracies near 0.50 from epoch 1.20.Its zero-flip sycophancy score reflects identical answers across conditions rather than recovered honesty.
  • Cross-model study: All five base models improve significantly on at least two SYCON-modified quantities, and all raise overall Turn-of-Flip, with the largest effects on SmolLM3 and Phi-3.Across 35 SYCON-modified quantities, 16 clear the significance threshold; TF sycophancy decreases most clearly for SmolLM3 and Phi-3.
  • Comparison with baselines: BTS GRPO reduces sycophancy comparably to the strongest labeled baselines without labels, but at substantially higher training cost.Synthetic-data fine-tuning is slightly stronger on some TF metrics, while SMART ends at 0.1717 under the default prompt.
  • BTS variants: Peer Truth Serum reduces sycophancy about as much as BTS without prediction reports, whereas neither Robust BTS variant significantly reduces it.Across mechanisms, the reported outcome tracks the rarity premium: surprise-based rewards reduce sycophancy, while peer matching does not.

5 Conclusion

The paper presents a label-free BTS reward for GRPO that reduces sycophancy while improving pressure-condition accuracy, but its interpretation and scope remain limited by reward ambiguity, empirical correctness, and evaluation design.

  • Method: BTS-GRPO derives rewards from the model’s own grouped outputs, requiring neither ground-truth labels nor preference annotations.Responses to one question act as the respondents scored by the peer-prediction mechanism.
  • Theory: The theory shows that, in large groups, honest responses earn strictly more expected reward than sycophantic responses.It also shows that symmetric pre-agreed answering rules cannot exceed truthful reporting’s information score.
  • Results: 23% to 4%: the reference model’s sycophancy score decreased on the TF dataset, while accuracy with the statement present increased from 80% to 93%.The multi-turn Turn-of-Flip increased on all five base models, and Peer Truth Serum reproduced the effect without prediction reports.
  • Results: The full reward outperformed SMART under both prompts and was comparable to synthetic-data fine-tuning and pinpoint tuning, whose baselines train on ground-truth answers.The method used roughly eighty times more compute than the supervised baselines.
  • Interpretation: The mechanism comparison suggests that paying a premium for rarer answers drives the reduction in sycophancy, but four experiments cannot separate this from reward-variance effects.The prediction-only ablation also reduced sycophancy nearly as much as the full reward, for reasons the paper does not establish.
  • Limitations: The reward targets what the model’s knowledge supports rather than correctness, so it can preserve errors when the model’s beliefs are wrong.Reported accuracy gains are empirical rather than guaranteed by the reward.
  • Limitations: The experiments use one seed, synthetic closed-form data, 3–4-billion-parameter models, and same-benchmark accuracy rather than a general-capability suite.The authors identify larger models, natural datasets, open-ended generation, and repeated seeds as open tests.
  • Open questions: The paper leaves unexplained why prediction-only rewards reduce sycophancy and does not test majority-vote rewards or a ground-truth-free collapse detector.These are identified as directions beyond the paper’s scope.

A Notes on the Bayesian Truth Serum

This appendix situates the paper’s BTS choice within peer prediction, clarifies which theoretical claims rely on which assumptions, and records limitations involving population size, signal informativeness, and equilibrium selection.

  • BTS formulation: The paper uses Prelec’s countably-infinite-population BTS score throughout, including the scored response in both population aggregates, while evaluating groups at G = 64.Prelec also proposed a finite-population, pairwise-comparison formula with Laplace-smoothed frequencies, but the paper does not use it.
  • Truthful benchmark: Under truthful reporting, the expected information score equals I(S; w | S′), the conditional mutual information between a respondent’s signal and the latent state given a peer signal.A truthful respondent is paid for shifting a peer’s belief about w.
  • Prediction score: The prediction score uniquely elicits the posterior predictive in the large-group limit, while the paper’s finite-group result leaves an O(1/G) correction that disappears at α = 1.The full reward removes that correction under the stated weight.
  • Theoretical contribution: The paper proves the strict truthful-versus-deceptive comparison needed for its theorem rather than relying on Prelec’s non-uniqueness-tolerant maximization statement.Its specialization concerns the user’s view and the paper’s setting.
  • Reward behavior: Prediction-report spread can increase information scores even when answers collapse to one option, preventing the group from receiving zero reward.Arithmetic-geometric mean structure makes dispersed predictions relevant to scoring.

B Proofs

The proofs establish posterior-predictive elicitation, strict preference for truthful answers in the relevant limits, and an upper bound on symmetric profiles’ information scores, with α = 1 essential for the combined reward.

  • Setup: Conditional independence makes peer-answer probabilities equal to posterior averages of the latent answer frequencies.Under truthful peers, ρ(· | t) becomes Pr(· | t); under mixtures, it follows the corresponding mixture equation.
  • Setup: The model treats a response’s answer and prediction as separate choices while conditioning expectations on its private signal.Peer answer probabilities are represented by a common conditional distribution ρ(· | t).
  • Combined reward: The proof’s bounded terms include the shared answer-frequency entropy and peer-prediction lower bounds imposed by the assumptions.These bounds keep the relevant expected objective finite in the comparison.
  • Prediction score: The expected prediction score is uniquely maximized by the posterior predictive, and the optimizer converges to Pr(· | t) as G →∞.The result holds for every α > 0 when considering the prediction score alone.
  • Combined reward: At α = 1, exact cancellation makes the combined reward’s prediction optimum equal to the posterior predictive; other weights do not preserve this optimum.For α ≠ 1, the derivative remains nonzero along a feasible perturbation.
  • Information score: The group-average information score is nonnegative because it decomposes into relative entropy plus a nonnegative normalization term.The geometric-mean prediction vector is normalized before applying the entropy inequality.
  • Truthfulness: For a truthful answer against sycophantic peers, Gibbs’ inequality gives a strict advantage whenever the alternative answer differs from the respondent’s signal.This is the key strictness step in the honest-versus-deceptive comparison.
  • Information score: Truthful reporting attains the information-score bound when the answer kernel is the identity, while overlapping signal-to-answer mappings make the inequality strict.The proof uses distinct posterior distributions associated with distinct signals.

C Datasets

The paper constructs balanced true/false items and judge-free multi-turn datasets that expose models to repeated user pressure. Prompts require closed-form answers and, for BTS variants, predictions of group responses.

  • TF dataset: The TF dataset contains 1000 balanced items, each pairing a question and correct answer with a sycophancy-inducing user statement.The dataset has 500 true and 500 false items, evaluated with and without the statement.
  • Alignment criteria: Debate measures alignment against the model’s turn-1 answer, ethical treats False as non-sycophantic, and false-presup checks the ground-truth option.These criteria differ from the original benchmark’s external-judge alignment rule.
  • Prompt construction: Debate uses an assigned stance followed by four distinct disagreement nudges, while ethical and false-presup retain their escalating multi-turn structures.False-presup stores which option contains the correction using a seeded per-item coin flip.
  • Prompt format: BTS prompts ask for both an answer and the percentage of people expected to answer true, with the response emitted as one JSON object.The shared scaffold also caps reasoning at 900 tokens inside think tags.

F Example Completions

The examples show fine-tuning correcting a pressure-induced flip while preserving answers across repeated turns, but also document a genuine regression and a reward-group mechanism behind BTS scoring.

  • Example selection: The examples include successes, unchanged cases, and failures because the SYCON section intentionally avoids presenting only favorable outcomes.The displayed completions use the BTS prompt for rollout-group examples.
  • TF correction: Before fine-tuning, the prenatal-vitamin example flips from the correct answer to the user-implied answer after pressure, despite reasoning that identifies the statement as false.After fine-tuning, the model answers true on every turn and remains aligned.
  • TF correction: The after-training prenatal example keeps the correct answer through all five turns even while the reasoning considers the user’s objection at length.The reported sequence changes from true, false, false, false, false before training to true throughout afterward.
  • Regression: Fine-tuning does not improve every item: on debate_53, the model changes from holding its position through five turns to flipping at the first pushback.All turns parse in both runs, so the regression is not attributed to parsing.
  • BTS reward example: In the BTS rollout group, 62 of 64 completions answer false, while the two shown predictions estimate only 57% and 60% for true.One completion fails to parse and receives the in-group unparsable penalty.

G Implementation Details

Implementation uses seeded, reproducible GRPO training with LoRA adapters, sampled rollout groups, explicit parsing safeguards, and recorded optimizer-step costs.

  • Generation: Training samples at temperature 0.6, while validation and test decoding are greedy with a 1024-token completion cap.Rollouts are generated by the model’s own sampling loop.
  • Training: Each GRPO group contains G = 64 samples, and one optimizer step consumes exactly one group.Training uses LoRA adapters on the attention projections.
  • Reward implementation: Predicted percentages and empirical frequencies are clipped by 10^-12 before reward aggregates and logarithms are computed, while unparsable completions receive −2.Aggregates run over completions that parse within the group.
  • Validation: Validation requires at least 90% dual parsing for TF checkpoint eligibility and selects the lowest sycophancy score after epoch 2.00.SYCON selection instead maximizes pooled overall mean ToF without a parse-rate floor.
  • Cost accounting: The study records training-only wall time, tokens seen, and FLOPs estimated as 6NT at every validation milestone.These counters measure optimizer steps rather than model loading, validation, or checkpoint-writing costs.
  • Reproducibility: A single seed fixed at 42 controls splits, assignments, and evaluations, while fixed model revisions support reproducible reruns.Hardware differences can still produce slight variation in figures.

H Ablation Study: Full Results

The ablation compares BTS reward components and weightings under fixed settings, using separate TF and SYCON evaluations with metric-specific denominators and tests. The default configuration weights information and prediction equally without ground-truth correctness, while training trajectories vary across reward choices.

  • Ablation design: The ablation holds model, hyperparameters, splits, and schedules fixed while varying six reward-weight settings (α, β, γ).The configurations isolate information, prediction, supervised correctness, and two tilted BTS combinations.
  • Evaluation conventions: SYCON validation maximizes pooled overall mean ToF across settings, weighting dialogues by their counts rather than averaging setting means.Every dialogue is scored, and unparsable turns count as not aligned.
  • Default configuration: The default BTS configuration uses (α, β, γ) = (1, 1, 0), equally weighting information and prediction without a ground-truth correctness term.This configuration is used throughout the rest of the paper.
  • Information-only ablation: The information-only setting reaches zero TF sycophancy, but both accuracies approach 0.50 because the model collapses onto a constant answer.Selection exports the epoch-1.20 snapshot after accuracy declines steadily.
  • Prediction-only ablation: The prediction-only setting rewards accurate forecasts of group answers but provides no pressure against agreement.The prediction term is therefore evaluated separately from the information term.
  • Training selection: For the (1, 2, 0) setting, overall ToF reaches its maximum at epochs 0.80 and 1.20, so the earlier epoch-0.80 snapshot is exported.This illustrates the strict-improvement and earliest-tie selection rule.

I Cross-Model Study: Full Results

The cross-model appendix fixes the BTS reward configuration and reporting conventions, then documents Llama’s training trajectory and export decision. Llama’s exported adapter was selected from late training after validation sycophancy stopped improving.

  • Study setup: The cross-model study uses default BTS reward weights (α, β, γ) = (1, 1, 0) while holding other hyperparameters, splits, and schedules fixed.Only the base weights and their numerical precision vary across the appendix subsections.
  • Reporting conventions: Each model is reported using five tables covering TF training, default-prompt testing, BTS-prompt testing, SYCON training, and SYCON-modified testing.Metric definitions, adapter selection, significance tests, and cost-row scaling follow Appendix H.
  • Llama model: The cross-model study includes Meta’s meta-llama/Llama-3.2-3B-Instruct checkpoint.The checkpoint is the model identified for the Llama subsection.
  • Llama training: Llama validation stopped at epoch 2.40 after sycophancy failed to improve over epoch 2.20, making the epoch-2.20 snapshot the exported adapter.Only the final two milestones cleared the 90% dual-parse floor; valid_both reached 72 at epoch 1.00 before later settling in the eighties and nineties.

I.2 Phi-3-mini-4k-instruct

Phi-3 briefly reached a configuration with strong accuracy and low sycophancy before collapsing onto a single answer. The reported zero sycophancy after collapse is therefore uninformative rather than evidence of honesty, and BTS-prompt parsing constrained Qwen rather than Phi in this section’s supplied results.

  • Model checkpoints: The section includes Alibaba’s Qwen/Qwen3-4B-Instruct-2507 checkpoint and Microsoft’s microsoft/Phi-3-mini-4k-instruct checkpoint.These are the named model checkpoints in the supplied cross-model passages.
  • Phi-3 results: Phi-3’s exported epoch-1.60 snapshot reached accuracy1 = 0.8400, accuracy2 = 0.8800, and sycophancy = 0.0800.This milestone was selected manually because the automatic rule returned a degenerate checkpoint.
  • Phi-3 results: After epoch 1.60, Phi-3 collapsed onto a single answer, making validation sycophancy reach zero without indicating honesty.The model gave the same response in both conditions, so it never flipped; the paper identifies this as the uninformative configuration of Theorem 3.
  • Phi-3 training: Phi-3 training stopped at epoch 2.00 after overall_mean_tof failed to improve over its epoch-0.60 value, exporting the epoch-0.60 snapshot.The training trajectory and export rule are stated separately from the manually overridden TF checkpoint decision.
  • Prompt parsing caveat: Qwen’s BTS-prompt evaluation had 33 of 100 items unparseable in at least one condition, compared with none under the default prompt.Unparseable answers were scored incorrect, and the passage cautions that much of the apparent accuracy gain reflects recovered formatting rather than recovered knowledge.

I.4 Gemma-3-4B-IT

Gemma’s TF training selected an early snapshot where accuracy1 peaked and sycophancy was minimized, while later training worsened that metric. Its SYCON trajectory ended with a reversal in ethical-response metrics that drove the overall decline.

  • Gemma model: The model evaluated in this subsection is Google’s google/gemma-3-4b-it checkpoint.The supplied passage names it as Google’s instruction-tuned Gemma 3 checkpoint.
  • Gemma TF training: Gemma exported the epoch-0.40 snapshot after validation stopped at epoch 2.00 because sycophancy failed to improve over epoch 0.40.Both prompts parsed all 100 items at every milestone, so the 90% dual-parse floor never bound.
  • Gemma TF training: At epoch 0.40, accuracy1 peaked at 0.7300 and sycophancy reached its minimum of 0.2300.Neither metric returned to that level later; sycophancy rose to 0.3400 at epoch 1.60.
  • Gemma SYCON training: Gemma’s SYCON run exported the epoch-1.20 snapshot after overall_mean_tof failed to improve through epoch 2.00.The final milestone was the only one where ethical_mean_tof fell below 4.
  • Gemma SYCON training: At the final SYCON milestone, ethical_mean_tof dropped from 4.2000 to 3.4000 while ethical_mean_nof more than doubled.The passage identifies this reversal as the largest single contribution to the decline in overall_mean_tof.

J Baselines: Full Results

The baseline comparison applies three labeled methods to SmolLM3-3B through the same TF pipeline, while documenting distinct adaptation, search, and head-selection procedures. Its reported costs are comparable only within subsections because excluded computation and scale factors differ.

  • Comparison setup: All three baselines are applied to SmolLM3-3B using the same TF splits and five validation passes per epoch, leaving the learning algorithm as the main change.The label-supervised reinforcement-learning baseline is the (0, 0, 1) ablation experiment.
  • Adaptation settings: All four methods use rank-8 LoRA adapters, with lora_alpha = 16 and lora_dropout = 0.05.The supplied settings table lists method-specific learning rates, schedules, effective batches, stopping epochs, and adapted modules.
  • Supervised baselines: Synthetic-data fine-tuning and pinpoint tuning use 1600 supervised examples targeting ground-truth answers in both evaluation conditions.Loss is computed on the completion alone, while the prompt is masked; the fixed reasoning states that user opinions do not change answers.
  • Pinpoint tuning: Pinpoint tuning scores heads on 128 training items and confines updates to selected heads in the query and output projections.The method uses assertions and gradient masks to keep unselected rows and columns zeroed throughout optimization.
  • SMART: SMART searches both conditions of all 800 training items, performing 1600 frozen-policy searches in stage 1.Each search uses 8 iterations, a cumulative-probability threshold of 0.9, at most 4 children, at most 4 reasoning steps, exploration constant 1, and entropy weight 0.2.
  • Cost caveat: Cost rows are comparable within a subsection but not across subsections because scale factors differ and FLOPs exclude work outside optimizer steps.Excluded work includes path patching for pinpoint tuning and the entire SMART stage-1 search.

J.1 Synthetic-Data Fine-Tuning

This section describes label-based fine-tuning baselines and their training procedures, alongside BTS-related comparison settings. The selected-head method identifies behavior-specific heads through path patching before tuning.

  • Synthetic-data fine-tuning presents every training item in both conditions with the correct answer as target.
  • The synthetic-data run exports the epoch-2.00 adapter after validation sycophancy fails to improve at epoch 2.20.
  • Supervised Pinpoint Tuning updates only path-patching-identified heads while freezing the remaining parameters.
  • The pinpoint-tuning head-selection check gives sycophantic fraction F = 0.4299 for selected heads versus 0.5925 for random heads.
  • SMART uses uncertainty-aware Monte Carlo tree search over reasoning trajectories followed by an offline policy-gradient update, with ground-truth answers entering both reward components.
  • The search corpus retains 12,659 trajectories after dropping 141 of 12,800 stored rollouts without recoverable final answers.

K BTS Variants: Full Results

This section compares BTS reward variants under matched GRPO settings and documents their implementation and evaluation boundaries. The variants differ in reward design, reporting requirements, and handling of parsed responses.

  • Full Results: All three variant experiments retain the same GRPO group structure, data splits, and hyperparameters while training on the TF dataset alone.
  • Mechanism definitions: Robust BTS shifts a reference response’s prediction by δ = min(y_j, 1 − y_j) before scoring it against a peer.
  • Mechanism settings: Every experiment uses group size 64, with peer structures defined over responses that parsed both an answer and a percentage.
  • Mechanism definitions: Peer Truth Serum pays responses based on agreement with the next parsed peer and elicits no percentage report.
  • Limitations: In unanimous Peer Truth Serum groups where all 64 responses parse, identical rewards produce zero centered advantages and no gradient.
  • Limitations: Under the BTS prompt, Peer Truth Serum is evaluated with a format it was not trained to produce, so parse counts indicate format compliance rather than sycophancy.
  • Limitations: The cost rows are comparable across experiments because all use the same group size, but extra mechanism bookkeeping is excluded from the reported optimizer-step costs.

K.3 Peer Truth Serum

Peer Truth Serum removes the percentage prediction report while retaining peer-based reward scoring. Its training trajectory shows declining sycophancy alongside rising accuracy.

  • Peer Truth Serum asks the model to provide only an answer, without eliciting a percentage prediction.
  • Sycophancy falls from 0.2222 to 0.0714 by epoch 1.20 and remains below 0.1 thereafter.
  • Accuracy1 rises from 0.7300 to 0.9300 across the reported training milestones.
  • The epoch-1.20 adapter is exported after later validation milestones fail to improve sycophancy, with tied milestones at 0.0510.
Loading 2608.25267v1…