PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment

Motivation

• Long-horizon search agents face sparse credit assignment: successful trajectories can include misleading actions, while failed ones can contain useful reasoning or evidence gathering.Tasks may require hundreds of tool-interaction turns and contexts spanning tens to hundreds of thousands of tokens.Source: §1 · pp. 1–2

Related work

• Prior dense-supervision approaches rely on costly or brittle evaluators, expensive trajectory expansion, or teacher targets misaligned with student behavior; direct privileged-policy imitation risks information leakage [6] [7] [8] [9] [10] [12] [13] [14].These limitations motivate using privileged information as evidence for credit assignment rather than as a policy to copy.Source: §1 · p. 2

Method

• PBSD scores intermediate actions by reformulating an intractable posterior-to-prior ratio for the verified answer as a privileged answer-conditioned versus student action-likelihood ratio.This Bayesian evidence score yields turn-level credit without enumerating future continuations, using external evaluators, or imitating privileged actions.Source: §1 · p. 2
• PBSD uses detached likelihood-ratio weights to calibrate trajectory-level advantages across turns, retaining the terminal verifiable reward as the global signal instead of adding a separate loss.Because the privileged answer informs only the weights, the student avoids directly imitating answer-conditioned behavior.Source: §1 · p. 2

Results

• PBSD shows superior performance over representative RLVR algorithms, needs substantially fewer training steps than outcome-only RLVR, and generalizes across datasets.The evaluation uses a 30B MoE model in long-horizon search-agent settings.Source: §1 · p. 2

2.1 Long-Horizon Credit Assignment in Agentic RL

• Prior credit-assignment methods use rubric/judge rewards [6] [15] [16] [17], heuristic process rewards [8] [18] [19], or tree search [9] [10], but face design, evidence, or cost limitations.PBSD instead derives a lightweight Bayesian calibration signal from sampled trajectories without handcrafted process rewards, external evaluators, or costly search-based estimation.Source: §2.1 · p. 3

2.2 On-Policy Distillation

• On-policy distillation uses student-sampled trajectories with teacher guidance [20] [21], while privileged self-distillation [13] [22] [23] [24] [25] [26] avoids a stronger teacher but risks information leakage and unstable training.Teacher reasoning may also mismatch the student’s exploration distribution, making otherwise correct token targets poorly aligned with on-policy behavior.Source: §2.2 · p. 3

2.3 Search Agents

• Search-R1 [29] and DeepResearcher [30] demonstrate outcome-based RL for multi-turn and real-world search, but final-answer or task-level rewards provide limited guidance on useful intermediate searches.Search-agent work also includes external information access and iterative querying, evidence inspection, and answer synthesis [5] [6] [27] [28].Source: §2.3 · p. 3

3.1 Preliminaries

• Standard GRPO gives every token in a trajectory the same advantage, a coarse signal that cannot reliably distinguish helpful from harmful turns in long-horizon search.Source: §3.1 · p. 4

3.2 PBSD: Privileged Bayesian Self-Distillation

• PBSD estimates trajectory support for the verified answer through a likelihood ratio between the standard student and an answer-conditioned teacher, avoiding direct posterior and prior estimation.Autoregressive decomposition yields turn-level scores that measure each turn’s incremental contribution to evidence for the verified answer.Source: §3.2 · pp. 4–5
• PBSD uses turn-level evidence to modulate inherited outcome advantages, strengthening evidence-supporting turns and down-weighting evidence-opposing turns without changing the outcome signal’s direction.A symmetric clipping threshold limits the weight range and the influence of outliers.Source: §3.2 · pp. 5–6
• Because per-turn evidence scores cluster near zero and can be noise-dominated, PBSD applies modulation only when evidence passes direction-specific positive or negative thresholds.Source: §3.2 · p. 6
• With R3 routing replay for MoE models, PBSD recomputes teacher and student evidence likelihoods using fresh routing, while retaining replayed log-probabilities for policy updates.This separates evidence scoring from routing-replay artifacts.Source: §3.2 · p. 6
• PBSD converts outcome-level supervision into turn-level credit calibration without adding an optimization objective, preserving standard policy-optimization stability and compatibility with existing RL pipelines.Source: §3.2 · p. 6

4.2 Experimental Results

• 40.87 validation accuracy and 35.83 BC(300) make PBSD the best-performing method across evaluation settings, beating GRPO by 2.62 and 3.50 points, respectively.Its gains are more pronounced on the easy and medium BrowseComp subsets, and Figure 2 shows favorable optimization behavior over the first 112 training steps.Source: Figure 2 · p. 7, §4.2 · pp. 8–9

4.1 Experimental Setup

• Evaluation uses mean@4 across four search and reasoning benchmarks, with BC(300) formed by randomly sampling 100 examples from each of three difficulty tiers.The tiers are defined by MiroThinker-mini’s solve rates over five runs per BrowseComp example.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 Experimental Results

• At 256K evaluation context after 64K-context training, SFT+PBSD leads trained agents on BrowseComp and improves over SFT across all benchmarks, while SFT+GRPO drops on BrowseComp.The benchmarks are BrowseComp, BrowseComp-ZH, GAIA (Text-Only), and xBench-DS-2505.Source: Table 2 · p. 8, §4.2 · p. 9
• During training, PBSD increases interaction turns while reducing generated tokens, and the share of turns with higher privileged-teacher likelihood rises as the teacher–student gap shrinks.The gap reduction occurs without directly optimizing KL divergence or distilling the privileged policy, mitigating privileged-information leakage.Source: Figure 3 · p. 9, §4.2 · p. 10
• Across all tested k values, SFT+PBSD exceeds SFT on BrowseComp Pass@k, whereas SFT+GRPO remains below SFT.The passage interprets the larger-k gains as more valid reasoning paths and the GRPO result as possible suppression of alternative solution modes.Source: Figure 4 · p. 9, §4.2 · p. 9

4.3 Ablation Study

• Removing fresh-routing evidence recomputation drops validation accuracy from 40.87 to 27.75 and BC(300) by 6.75 points; the full method’s three design components each contribute measurable gains.δ = 0.1 performs best, while both modulation extremes are suboptimal; asymmetric thresholds ϵ+ = 0.001 and ϵ− = 0.003 filter roughly 30% of low-SNR turns for the best overall trade-off. No filtering trails the full method by more than 6 validation points, filtering 10% is insufficient, and filtering 40% degrades all metrics.Source: §4.3 · p. 10

5 Conclusion

• PBSD converts posterior-to-prior trajectory evidence into an estimable action-likelihood ratio used as a detached calibration weight for trajectory-level advantages.It derives turn-level credit from privileged answer-conditioned likelihoods using Bayes’ rule.Source: §5 · p. 11
• PBSD consistently outperforms outcome-only RLVR and strong baselines, with robust cross-dataset and long-context generalization.Source: §5 · p. 11
• The results highlight Bayesian evidence calibration as an effective mechanism for improving credit assignment in long-horizon agent training.Source: §5 · p. 11

Limitations and Future Work

• PBSD requires verified final answers, which may be ambiguous, incomplete, or non-unique in open-ended agentic tasks.Its verified-answer assumption fits settings such as search QA and verifiable reasoning, where outcomes can be checked automatically.Source: §6 · p. 11
• If the answer-conditioned model is insufficiently capable or calibrated, its evidence ratios may reflect likelihood noise or miscalibration rather than reliable turn-level credit.Source: §6 · p. 11
• Future work could broaden PBSD to tasks without unique verified answers by using learned verifiers, stronger teacher models, or other evaluative signals.Source: §6 · p. 11

• Figure 5 presents the system prompt used by the search agent.Source: Figure 5 · p. 14

B Pseudocode

• PBSD scores generated tokens under original and answer-augmented contexts, using their likelihood difference to form a turn-level self-distillation signal.Turns more likely under the privileged context are amplified for positive trajectory advantages and suppressed for negative ones; the reverse adjustment applies when their likelihood is lower.Source: B Pseudocode · pp. 14–15
• Privileged answers are used only to reweight training advantages, while inference receives only the original prompt.The policy-gradient update remains otherwise unchanged.Source: B Pseudocode · p. 15
• For each prompt, PBSD samples a group of trajectories and computes group-normalized advantages from verifier rewards.These advantages initialize the token-level training signal.Source: B Pseudocode · p. 15
• The algorithm initializes token advantages from trajectory advantages, then applies a turn multiplier after a clipped soft correction; turns without mapped scoring positions use multiplier 1.The resulting reweighted token advantages feed the policy-gradient update.Source: B Pseudocode · p. 15

C Case Study: Turn-Level Credit Assignment

• The case studies visualize privileged likelihood signals per assistant turn; High-value aligns with the answer-conditioned teacher, Low-value marks misleading reasoning, and Filtered denotes a weak likelihood gap.Rollout logs do not store token-level likelihoods.Source: C Case Study: Turn-Level Credit… · pp. 15–16
• In the failed rollout, PBSD preserves useful Rubiaceae and Rodrigues-clue reasoning while penalizing the wrong Malvaceae/Dombeya branch and incorrect final answer.The ground truth is Octotropideae, but the model outputs Coffea.Source: C Case Study: Turn-Level Credit… · p. 16, Figure 7 · p. 18
• In the successful Amar Sangee rollout, PBSD credits turns linking the forest-film clue to Satyajit Ray and surfacing the title, while marking redundant or off-path verification low-value.Weak but plausible signals are filtered rather than confidently credited.Source: C Case Study: Turn-Level Credit… · p. 16, Figure 6 · p. 17