Unifying ICL, SFT, KL-Regularized RL Through a Bayesian Lens

SFT projection

• SFT learns the teacher or Bayes predictive distribution as its forward-KL projection onto the parametric model family.The projection is equivalent to maximum-likelihood training and can use direct samples from the target predictive distribution or an empirical approximation.Source: §1.1 · p. 2, §1.4 · p. 5

Bayesian target

• The Bayes posterior predictive is the normative target for few-shot in-context prediction.Source: §1.2 · p. 3, §1.4 · p. 5

Amortized ICL

• A sufficiently expressive pretrained Transformer can amortize this target, approximating it directly from the context during its forward pass.This formulation does not require test-time gradient descent, although prior work also interprets ICL through first-order or meta-learning updates.Source: §1.3 · p. 4, §1.4 · p. 5

ICL–SFT connection

• Thus, ICL and SFT are different approximations to the same underlying Bayesian predictor rather than identical procedures.Source: §1.4 · p. 5

Reward-based extension

• KL-regularized RL produces a Gibbs posterior from a reference policy and reward, while reward-weighted SFT is its forward-KL projection onto the policy family.The paper states this equivalence at the target-posterior and projection level, not as a claim that every practical RLHF implementation reaches the closed-form optimum.Source: §2.1 · pp. 6–7, §2.2 · p. 8

• RW-ICL defines a Gibbs posterior using the reference model as prior and exp(R/β) as a reward-based pseudo-likelihood.The reward aggregates information from human preferences, verifiers, or reward models conditioned on the context and query.Source: §3.1 · p. 9
• The RW-ICL population objective trains a meta-predictor Mθ to approximate this generalized posterior across tasks and contexts.The expectation follows the same hierarchical meta-distribution used for ordinary ICL, together with the process generating rewards.Source: §3.2 · p. 9
• RW-ICL is a forward-KL projection of the generalized Gibbs posterior onto the in-context model family.Stochastic gradient ascent implements the projection using reference-model samples reweighted by exp(R/β).Source: §3.2 · p. 9
• At the first-order update level, RW-ICL, reward-weighted SFT, and KL-regularized RL share reward-weighted score-gradient structure.The correspondence concerns update forms, with the model and signal changing across ICL, SFT, and RL.Source: §3.2 · p. 10
• At the distribution level, reward-weighted ICL approximately matches the KL-regularized RLHF/RLVR optimum, up to approximation error in the KL projection.Source: §3.2 · p. 10

Process-level KL-RLVR

• The KL-regularized policy-improvement optimum is a Gibbs posterior over actions, with Q(s, a)/β determining action preference relative to a reference policy.Source: §4.1 · pp. 10–11
• Advantage baselines shift Q by state-dependent constants that are absorbed into normalization, leaving the normalized policy unchanged.Source: §4.1 · p. 11

Advantage-weighted SFT

• AWSFT is the forward-KL projection of the statewise KL-RLVR Gibbs optimum onto the parametric policy family.The empirical objective uses estimated advantages as exponentiated weights over state-action pairs.Source: §4.2 · p. 11

Trajectory-level RW-ICL

• Trajectory-level RW-ICL uses per-step advantage weights and produces updates matching AWSFT, subject to state-distribution and context-window choices.Source: §4.3 · p. 12

Learning signals and credit assignment

• Across SFT, RL, and ICL, the equivalence concerns posterior objectives and first-order updates, not the source or granularity of supervision.SFT has dense token-level labels, RL begins with sparse trajectory rewards and explicit credit assignment, and ICL adapts in activations after dense pretraining supervision.Source: §4.4 · p. 14

Posterior design and projection

• RLHF/RLVR can be viewed as designing a Gibbs posterior from a reference model and reward, then projecting it into a deployable policy.Projection may use reward- or advantage-weighted SFT or an optimizer such as PPO.Source: §5.1 · p. 15

Design knobs

• Reward shaping, temperature β, and sampling regime control posterior concentration, reference-policy movement, and the stability–exploration trade-off.Small β concentrates the posterior, while offline and on-policy methods trade data reuse and stability against exploration.Source: §5.1 · p. 15

Cold start and on-policy distillation

• When the sampling distribution has little overlap with the high-reward Gibbs region, importance weights collapse and gradients become unstable.Source: §5.2 · p. 16
• A small supervised or behavior-cloning warm-up moves the policy toward the high-reward region, making Gibbs-posterior projection statistically feasible.Source: §5.2 · p. 16

Compute allocation and reasoning models

• Reasoning models combine test-time search over solution paths with training-time RL, OPD, or SFT that amortizes high-reward behavior into model weights.DeepSeek-R1 and o1-style systems are described as combining these inference-time and training-time regimes.Source: §5.3 · pp. 16–17

Few-shot prompting and posterior mismatch

• Few-shot prompting can degrade RL-tuned reasoning models when its richer contextual posterior differs from the posterior trained under zero-shot or lightly structured prompts.RW-ICL suggests defining rewards at the contextual level and training the model directly on that Gibbs posterior.Source: §5.3 · p. 17

• Few-shot ICL approximates the Bayes posterior predictive through an amortized forward-KL/MLE projection.Source: Final Conclusions · pp. 17–18
• KL-regularized RL defines a Gibbs posterior, while reward-weighted SFT, AWSFT, and RW-ICL project onto corresponding reward- or advantage-induced posteriors.Source: Final Conclusions · pp. 17–18
• The equivalence holds at posterior-target and first-order-update levels, not as an identity between practical algorithms.Source: Final Conclusions · p. 18
• Learning processes remain different: SFT uses dense token supervision, RL uses sparse rewards with credit assignment, and ICL adapts through forward activations without test-time gradients.Source: Final Conclusions · p. 18
• The common mechanism is a Bayesian or generalized Bayesian update followed by forward-KL projection onto a parameterized policy family.Source: Final Conclusions · p. 18

• The appendix reviews Wakayama and Suzuki and Reuter et al. as the main prior foundations for the paper’s Bayesian ICL arguments.Source: Appendix A · p. 20
• Full technical details and proofs are deferred to the original papers.Source: Appendix A · p. 21

• The ICL risk decomposition separates Bayes Gap, measuring distance from the Bayes predictor, from Posterior Variance, determined by task uncertainty and context length.Source: §A.1 · p. 21
• The Bayes-optimal predictor uniquely minimizes risk, making its posterior predictive the normative target for in-context prediction.Source: §A.1 · p. 21
• The non-asymptotic bound contains approximation error controlled by feature dimension and pretraining generalization error depending jointly on context length and prompt count.Source: §A.1 · p. 21
• In mixed-task settings, posterior task identification concentrates quickly as context grows, enabling rapid specialization to the true task family.Source: §A.1 · p. 21
• Under input-distribution shift, the out-of-distribution penalty increases the Bayes Gap, while Posterior Variance remains intrinsic to the target domain.Source: §A.1 · p. 21
• The paper uses these results to assume that sufficiently expressive, many-task-trained Transformers can approximate the Bayes predictor with a small Bayes Gap.Source: §A.1 · p. 21

• Reuter et al. study whether in-context Transformers can approximate full Bayesian posterior inference over latent variables.Source: §A.2 · p. 22
• They train learners that map datasets to predictive distributions or posterior samples over latent variables and responses.Source: §A.2 · p. 22
• They compare Transformer samples with HMC, SGLD, and variational approximations using C2ST, MMD, Wasserstein distance, and RMSE.Source: §A.2 · p. 22
• The experiments investigate robustness under training–test distribution shift and report that the in-context learner often remains close to Bayesian baselines.Source: §A.2 · p. 22
• Suitably trained Transformers often match or approach fully Bayesian baselines and sometimes outperform classical approximations on distributional metrics.Source: §A.2 · p. 22
• These findings support approximating the Bayes posterior predictive through a single forward pass.Source: §A.2 · p. 22

• Part I treats the Bayes posterior predictive as the canonical target for in-context prediction and decomposes ICL risk into Bayes Gap and posterior variance.Source: §A.3 · p. 22
• The amortized-inference view holds that a Transformer can approximate a dataset-level Bayesian posterior in a single forward pass.Source: §A.3 · p. 22
• Prior analyses show that, in linear and linearized regimes, Transformers can implement one-step gradient updates and attention can represent accumulated gradient corrections.Source: §A.3 · p. 22
• The appendix interprets these gradient-based results as algorithmic realizations of the Bayesian predictor rather than reproducing their derivations.Source: §A.3 · p. 22

• Within-task exchangeability is an assumption about the data-generating process, not about the neural architecture implementing in-context learning.Source: §A.4 · p. 23
• The appendix uses de Finetti–Hewitt–Savage representation to derive the hierarchical task–prior model from exchangeability.Source: §A.4 · p. 23
• The observations are modeled as sample pairs in X × Y generated within a single task episode.Source: §A.4.1 · p. 23

• A finite sequence is exchangeable when its joint distribution is invariant under every permutation of its indices.Source: §A.4.1 · p. 23
• An infinite sequence is exchangeable when every finite prefix is exchangeable.Source: §A.4.1 · p. 23
• Exchangeability means that only the multiset of observations matters, and it is strictly weaker than the i.i.d. assumption.Source: §A.4.1 · p. 23
• Within-task exchangeability assumes that the conceptual infinite sequence of observation pairs (X_i, Y_i) is exchangeable.Source: §A.4.1 · p. 23
• The framework treats the observed dataset D_k as a finite prefix of an exchangeable sequence, with the infinite extension serving as a probability-theoretic idealization.Source: §A.4.1 · p. 23
• The standard Borel structure of typical supervised-learning spaces permits application of the general de Finetti–Hewitt–Savage theorem.Source: §A.4.1 · p. 23

A de Finetti–Hewitt–Savage representation

• The representation theorem states that an infinite exchangeable sequence is conditionally i.i.d. given a random probability measure, whose mixing law is unique.Source: §A.4.2 · p. 24
• Under the exchangeability assumption, each episode therefore admits a mixture-of-i.i.d. representation.Source: §A.4.2 · p. 24

From a random measure to a task prior

• The random measure is identified with a task f, and its mixing distribution Π is renamed the task prior P(f).Source: §A.4.3 · p. 24
• This task–prior representation is a canonical reparameterization of the exchangeable joint law, not an additional ad hoc modeling assumption.Source: §A.4.3 · p. 25, §A.4.4 · p. 25
• The construction identifies each task with a probability measure on X × Y and permits the conditional task-function view f: X → Δ(Y).Source: §A.4.3 · p. 25

Remarks and scope

• For finite datasets, the infinite-sequence formulation requires an exchangeable extension, while finite de Finetti results provide a conceptual approximate alternative.Source: §A.4.4 · p. 25
• The task–prior model is presented as an effective theory whose empirical plausibility for real pretraining corpora remains an open question, and practical transformers may exhibit order effects.Source: §A.4.4 · pp. 25–26