PaperlayerView exact revision on arXiv

Source-linked AI summary

Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Weiqin Chen, Xinlong Du, Gesualdo Scutari, Harsha Honnappa

arXiv:2610.28977v1stat.MLcs.LG

TL;DR

DPT’s theoretical characterization as Bayesian posterior sampling relies on idealized assumptions, leaving open whether DPT and its variants behave that way in practice. The paper tests two necessary conditions across bandit settings and finds consistent violations of sufficient-statistic invariance, with additional martingale violations for DPT′ and GDPT.

  • Problem

    Whether DPT and its variants behave as Bayesian posterior samplers in practice remains open, since the theoretical characterization relies on assumptions such as model expressivity and infinite pretraining data.

    Source: §1 · p. 2

  • Method

    The paper derives sufficient-statistic invariance and the martingale property as necessary conditions and tests them statistically using action probabilities.

    Source: §1 · p. 2, §3.1.1 · p. 5, §3.1.2 · p. 5

  • Results

    Across bandit settings, DPT, DPT′, and GDPT violate sufficient-statistic invariance, while DPT′ and GDPT also violate the martingale property despite decision performance comparable to Thompson Sampling.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The tests provide a principled toolkit for auditing Bayesian claims about in-context reinforcement learning.

    Source: §1 · p. 2

    DPT’s theoretical Bayesian-sampling characterization assumes a sufficiently expressive model and infinite pretraining data, conditions that may not hold in practice.

    Source: §1 · p. 2

Abstract

from arXiv · show

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.

1 INTRODUCTION

The introduction asks whether DPT and its variants behave as Bayesian posterior samplers in practice, despite a theoretical characterization that depends on idealized assumptions. It proposes necessary-condition tests and reports consistent violations across bandit settings.

1 INTRODUCTION

  • ICRL aims to adapt pretrained transformers to unseen tasks from interaction context alone, while DPT predicts optimal actions from contexts using supervised pretraining.Unlike Algorithm Distillation, DPT does not imitate an evolving reinforcement-learning algorithm.Source: §1 · p. 1
  • DPT’s Bayesian-sampling characterization relies on a sufficiently expressive model and infinite pretraining data, assumptions that may not hold in practice.The introduction also reports an empirical gap between default DPT and Thompson sampling in Gaussian multi-armed bandits.Source: §1 · p. 2, Figure 1 · p. 2
  • DPT, DPT′, and GDPT violate sufficient-statistic invariance across the tested settings, while DPT′ and GDPT also violate the martingale property despite decision performance close to Thompson Sampling.Permutation-invariance ablations do not remove the violations; the tests are proposed as a toolkit for auditing Bayesian claims about ICRL.Source: §1 · p. 2
  • The paper derives sufficient-statistic invariance and the martingale property as necessary conditions, then turns them into statistical tests of action probabilities.Because DPT predicts actions only, DPT′ and GDPT add a reward head to make the martingale property testable.Source: §1 · p. 2

2 IN-CONTEXT REINFORCEMENT LEARNING UNDER DPT

DPT pretraining samples environments, gathers interaction contexts, and labels them with optimal actions before training a transformer. At deployment, the pretrained model acts without parameter updates, using either a provided context or one it collects online.

2 IN-CONTEXT REINFORCEMENT LEARNING UNDER DPT

  • DPT considers finite-horizon tasks defined by an action space, reward kernel, and horizon.The section frames the method as pretraining dataset collection, pretraining, and deployment.Source: §2 · p. 2

2.1 PRETRAINING DATASET COLLECTION

  • For each sampled environment, the dataset pairs context data with an optimal action label; the context-collection policy may be arbitrary.Contexts are ordered sequences of action–reward pairs, and collection may use a random policy, PPO, or posterior sampling.Source: §2.1 · pp. 2–3

2.2 PRETRAINING

  • A decoder-only transformer is pretrained on context–optimal-action examples using expected cross-entropy loss; its empty-context action distribution acts as the model’s prior.Causal attention makes each action distribution depend only on the context prefix available at that position.Source: §2.2 · p. 3

2.3 DEPLOYMENT

  • At test time, DPT receives a context and produces an action distribution without parameter updates; the context is either provided offline or collected through its own actions online.Online deployment appends each observed reward and action to the context before the next decision.Source: §2.3 · p. 3

3 ARE DPT AND ITS VARIANTS BAYESIAN POSTERIOR SAMPLERS?

The section characterizes Bayesian posterior sampling through a transformer’s action distribution, derives sufficient-statistic invariance and a martingale property, and develops tests of these conditions. It also introduces DPT variants intended to make the martingale condition testable.

3.1 TRANSFORMER AS A BAYESIAN POSTERIOR SAMPLER

  • A transformer is a Bayesian posterior sampler when its action distribution is a mixture of a context-independent decision kernel under a posterior induced by a prior and likelihood.The definition is behavioral: it does not require explicitly inferring a latent parameter or maintaining a posterior.Source: §3.1 · p. 4

3.1.1 SUFFICIENT-STATISTIC INVARIANCE

  • If the latent reward likelihood is context-independent and has the specified exponential-family form, action distributions depend on the context only through its sufficient statistic.Permutation invariance follows from context-independent likelihoods; sufficient-statistic invariance requires the additional exponential-family condition.Source: §3.1.1 · pp. 4–5

3.1.2 MARTINGALE PROPERTY

  • Under the Bayesian posterior-predictive continuation distribution, the expected future action distribution equals the current one, making action probabilities a martingale over contexts.The continuation is generated autoregressively from the transformer’s own future action and reward predictions.Source: §3.1.2 · p. 5

3.2 DPT VARIANTS

  • DPT′ jointly predicts the optimal action and its associated reward, while GDPT autoregressively predicts the next action-reward pair in context.DPT′ uses optimal action and reward supervision; GDPT does not require optimal labels but requires expert contexts, unlike DPT and DPT′.Source: §3.2 · p. 5
  • DPT′ and GDPT use Gaussian reward heads and can generate continuations from their posterior-predictive distributions, satisfying the stated conditions by construction.These variants enable testing the martingale property, which action-only DPT cannot be tested for directly.Source: §3.2 · p. 5, §3.1.2 · p. 5

3.3.1 SUFFICIENT STATISTIC

  • The framework tests whether practically trained ICRL schemes satisfy the two necessary conditions, using paired context transformations that preserve sufficient statistics.The relaxed null tests whether the expected paired action-probability discrepancy is zero; transformations must induce distinct context distributions to avoid symmetry making the expectation trivially zero.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.2 MARTINGALE PROPERTY

  • For the martingale test, the variants generate continuations from a fixed context; independent continuations yield conditionally i.i.d. increments tested with paired Hotelling tests.The tested increments are formed from action probabilities under generated continuations; the method uses the fact that their sum across actions is zero.Source: §3.3.2 · p. 6

4 EXPERIMENTS

Experiments in Gaussian, Bernoulli, and linear bandits test DPT and its reward-head variants against necessary Bayesian conditions. All three models violate sufficient-statistic invariance, while DPT′ and GDPT also violate the martingale property in nearly all tested settings.

4.1 EXPERIMENTAL SETUP

  • Experiments use Gaussian, Bernoulli, and linear bandits, with paired Hotelling tests at α = 0.05 on 5,000 sampled environments or continuations.Sufficient-statistic tests compare action probabilities under context transformations; martingale tests use model-generated continuations.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL RESULTS

  • All sufficient-statistic tests reject for DPT, DPT′, and GDPT; martingale tests reject for DPT′ and nearly all GDPT settings.GDPT has two exceptions among online Bernoulli tests: (n, m) = (50, 100) and (100, 400), with p-values 0.099 and 0.094.Source: §4.2 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 8
  • In Gaussian bandits, GDPT’s cumulative regret and offline suboptimality closely follow Thompson Sampling while its offline TV distance remains positive.Offline DPT′ likewise attains suboptimality close to Thompson Sampling while its TV distance remains positive.Source: §4.2 · p. 8
  • Making DPT and DPT′ permutation-invariant restores sorting invariance by construction, but reward-replacement and martingale tests still reject.The ablation therefore indicates that context-order sensitivity is not the main source of the observed violations.Source: §4.2 · p. 9

5 CONCLUSION

The paper finds that practical DPT-family models violate necessary Bayesian conditions, even when decision performance is comparable to Thompson Sampling. It situates these tests among theoretical Bayesian accounts and empirical martingale diagnostics, and identifies extensions to state-dependent problems as future work.

5 CONCLUSION

  • DPT, DPT′, and GDPT violate sufficient-statistic invariance, while DPT′ and GDPT also violate the martingale property despite decision performance comparable to Thompson Sampling.The tests rely on observable action predictions and provide a toolkit for examining Bayesian claims about ICRL.Source: §5 · p. 9
  • Enforcing permutation invariance restores context-order invariance but recovers neither Bayesian condition, indicating order sensitivity is not the main source of violations.The paper identifies extending the tests to state-dependent decision problems and determining suitable architectural and training choices as future work.Source: §5 · p. 9

A RELATED WORK

  • Traditional RL typically requires retraining a policy from scratch for each new task, whereas ICRL aims to generalize to unseen tasks without parameter updates.Source: A RELATED WORK · p. 13
  • Prior theory establishes Bayesian behavior under idealized generative assumptions, while Falck et al. (2024) use martingale diagnostics that reveal violations in pretrained LLMs.This paper applies that diagnostic perspective to ICRL.Source: A RELATED WORK · p. 13
  • Thompson sampling and episodic MDP methods provide exact posterior-sampling formulations; this paper investigates whether ICRL can be viewed that way in practice.The cited work describes posterior sampling as drawing an environment from the current posterior and acting optimally for that draw.Source: A RELATED WORK · p. 13

B.1 PROOFS OF PROPOSITIONS 1 AND 2

The proofs establish that Bayesian posterior sampling entails invariant action distributions for contexts with the same sufficient statistics and a martingale action-probability sequence. The appendices give context transformations that preserve sufficient statistics but change context distributions, identify bandit sufficient statistics, and describe a paired Hotelling test.

B.1 PROOFS OF PROPOSITIONS 1 AND 2

  • The tower property shows that a Bayesian posterior sampler’s action probabilities form a martingale, without requiring an exponential-family likelihood or sufficient statistic.The argument conditions on the observed history and uses the same joint Bayesian model for context continuations and posteriors.Source: §B.1 · p. 14

B.2 DISTINCT CONTEXT DISTRIBUTIONS AFTER SUFFICIENT STATISTICS-PRESERVING TRANSFORMATIONS

  • Sorting action–reward pairs preserves sufficient statistics in all three bandit settings but produces a context distribution distinct from the original.Sorting is by ascending reward, retaining original order within ties.Source: §B.2 · p. 15

B.3 SUFFICIENT STATISTICS FOR BANDITS

  • Across Gaussian, Bernoulli, and linear bandits, preserving every arm’s count and reward sum is equivalent to preserving the sufficient statistics.The Gaussian and Bernoulli corollaries state that action distributions depend on the context only through these per-arm quantities.Source: §B.2 · p. 15, §B.3 · p. 16

B.2 DISTINCT CONTEXT DISTRIBUTIONS AFTER SUFFICIENT STATISTICS-PRESERVING TRANSFORMATIONS

  • Reward replacement also preserves sufficient statistics while changing context distributions in Gaussian and linear bandits, assuming an arm is observed at least twice with positive probability.The replacement adds independent uniform noise while preserving each observed arm’s reward sum.Source: §B.2 · p. 15

C.1 PAIRED HOTELLING TEST

  • The paired Hotelling test tests whether the mean difference between paired probability vectors is zero, dropping one coordinate because the full difference covariance is singular.The setup uses paired i.i.d. vector observations and a central limit theorem for the sample mean.Source: §C.1 · p. 17

C.2 EXPERIMENTAL SETUP AND HYPERPARAMETERS

Table 4 summarizes environment, model, and statistical-test configurations for DPT, DPT′, and GDPT. The architecture ablations alter positional encoding, attention, and context pooling; GDPT is not ablated because its next-pair objective requires causal attention.

C.2 EXPERIMENTAL SETUP AND HYPERPARAMETERS

  • Table 4 summarizes environment, model, and statistical-test hyperparameters for DPT, DPT′, and GDPT.Unless a model-specific setting is indicated, the listed values are shared across the three models.Source: Table 4 · p. 18, §C.2 · p. 18
  • DPT and DPT′ ablations remove positional encoding, use bidirectional attention, and canonically order context tokens before mean-pooling hidden states; GDPT is not ablated because its objective requires causal attention.The ablated predictions are permutation invariant; exact equality under sorting is reported as “Equal” without a p-value.Source: §C.2 · p. 18

D.1.1 GAUSSIAN EVALUATION FIGURES

Figure 2 compares DPT and Thompson Sampling in Gaussian bandits under online and offline deployment.

D.1.1 GAUSSIAN EVALUATION FIGURES

  • Figure 2 compares DPT with Thompson Sampling in Gaussian bandits across online and offline deployment.Source: Figure 2 · p. 19

D.1.2 BERNOULLI EVALUATION FIGURES

Figure 3 compares DPT and Thompson Sampling in Bernoulli bandits under online and offline deployment.

D.1.2 BERNOULLI EVALUATION FIGURES

  • Figure 3 compares DPT with Thompson Sampling in Bernoulli bandits across online and offline deployment.Source: Figure 3 · p. 19

D.1.3 LINEAR EVALUATION FIGURES

These appendix materials compare DPT, DPT′, and GDPT with Thompson Sampling and show sufficient-statistic p-values and per-arm martingale increments across bandit settings and deployment modes.

D.1.3 LINEAR EVALUATION FIGURES

  • Figure 4 compares DPT with Thompson Sampling in linear bandits under online and offline deployment.Source: Figure 4 · p. 19

D.2.1 SUFFICIENT-STATISTICS TABLE

  • Table 5 reports sufficient-statistic invariance test p-values for DPT′ in Gaussian, Bernoulli, and linear bandits.The table specifies N = 5000, α = 0.05, and n as context length.Source: Table 5 · p. 20

D.2.2 GAUSSIAN EVALUATION FIGURES AND MARTINGALE PLOTS

  • Figures 5–7 compare DPT′ with Thompson Sampling and plot per-arm martingale increments in Gaussian bandits for offline and online starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 5 · p. 20, Figure 6 · p. 21, Figure 7 · p. 21

D.2.3 BERNOULLI EVALUATION FIGURES AND MARTINGALE PLOTS

  • Figures 8–10 compare DPT′ with Thompson Sampling and plot per-arm martingale increments in Bernoulli bandits for online and offline starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 8 · p. 22, Figure 9 · p. 22, Figure 10 · p. 23

D.2.4 LINEAR EVALUATION FIGURES AND MARTINGALE PLOTS

  • Figures 11–13 compare DPT′ with Thompson Sampling and plot per-arm martingale increments in linear bandits for online and offline starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 11 · p. 23, Figure 12 · p. 24, Figure 13 · p. 25

D.3.1 SUFFICIENT-STATISTICS TABLE

  • Table 6 reports sufficient-statistic invariance test p-values for GDPT in Gaussian, Bernoulli, and linear bandits.The table specifies N = 5000, α = 0.05, and n as context length.Source: Table 6 · p. 26

D.3.2 GAUSSIAN EVALUATION FIGURES AND MARTINGALE PLOTS; D.3.3 BERNOULLI EVALUATION FIGURES AND MARTINGALE PLOTS

  • Figures 14–19 compare GDPT with Thompson Sampling and plot per-arm martingale increments in Gaussian and Bernoulli bandits for online and offline starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 14 · p. 26, Figure 15 · p. 27, Figure 16 · p. 27

D.3.4 LINEAR EVALUATION FIGURES AND MARTINGALE PLOTS

  • Figures 20–22 compare GDPT with Thompson Sampling and plot per-arm martingale increments in linear bandits for online and offline starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 20 · p. 29, Figure 21 · p. 30, Figure 22 · p. 31

D.4.1 DPT SUFFICIENT-STATISTICS TABLE AND EVALUATION FIGURES

The appendix presents a Gaussian-bandit sufficient-statistics table and a DPT performance comparison under online and offline deployment.

D.4.1 DPT SUFFICIENT-STATISTICS TABLE AND EVALUATION FIGURES

  • Table 7 reports sufficient-statistic invariance test p-values for permutation-invariant DPT in Gaussian bandits.“Equal” denotes identical action probabilities, for which no p-value is reported.Source: Table 7 · p. 32
  • Figure 23 compares permutation-invariant DPT with Thompson Sampling in Gaussian bandits under online and offline deployment.Source: Figure 23 · p. 32

D.4.2 DPT′ SUFFICIENT-STATISTICS AND MARTINGALE TABLES AND EVALUATION FIGURES

The appendix presents sufficient-statistic and martingale test tables alongside permutation-invariant DPT′ comparisons and martingale plots in Gaussian bandits.

D.4.2 DPT′ SUFFICIENT-STATISTICS AND MARTINGALE TABLES AND EVALUATION FIGURES

  • Table 8 reports sufficient-statistic invariance test p-values for permutation-invariant DPT′ in Gaussian bandits.“Equal” denotes identical action probabilities, for which no p-value is reported.Source: Table 8 · p. 32
  • Table 9 reports martingale test p-values for permutation-invariant DPT′ in Gaussian bandits.The table identifies n and m as starting-context and continuation lengths.Source: Table 9 · p. 33
  • Figure 24 compares permutation-invariant DPT′ with Thompson Sampling in Gaussian bandits under online and offline deployment.Source: Figure 24 · p. 33
  • Figures 25–26 plot per-arm martingale increments for permutation-invariant DPT′ in Gaussian bandits with offline and online starting contexts.The plots identify m and t as continuation and starting-context lengths.Source: Figure 25 · p. 34, Figure 26 · p. 34

Related papers

Loading 2610.28977v1…