Source-linked AI summary
LLMs Are In-Context Bandit Reinforcement Learners
Giovanni Monea, Antoine Bosselut, Kianté Brantley, Yoav Artzi
TL;DR
The paper asks whether LLMs can extend supervised in-context learning to contextual bandit learning from online reward. It studies methods for eliciting this behavior and finds effective ICRL, while noting limits in reasoning about negative episodes and in the tested setting.
Problem
The paper asks whether LLMs' in-context learning extends from annotated demonstrations to contextual bandit reinforcement learning with stateful, single-step interactions.
Source: §1 · p. 1Method
Results
LLMs show effective bandit ICRL with exploration and positive-reward filtering; on Banking77, Qwen2.5-7B improves from 6.2% zero-shot accuracy to 72.2% with Stochastic ICRL.
Source: §1 · p. 2Takeaways & Limitations
Exploration combined with filtering negative-reward episodes supports strong ICRL trends, while effective reasoning about negative episodes remains an open challenge.
Source: §6 · p. 10The study uses contextual-bandit classification with binary rewards, leaving applicability to more complex rewards and multi-step interactions open.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context. We investigate a contextual bandit version of in-context reinforcement learning (ICRL), where models learn in-context, online, from external reward, instead of supervised data. We show that LLMs effectively demonstrate such learning, and provide a detailed study of the phenomena, experimenting with challenging classification tasks and models of sizes from 500M to 70B parameters. This includes identifying and addressing the instability of the process, demonstrating learning with both semantic and abstract labels, and showing scaling trends. Our findings highlight ICRL capabilities in LLMs, while also underscoring fundamental limitations in their implicit reasoning about errors.
1 Introduction
The paper asks whether LLMs can learn online from reward in contextual bandits, extending in-context learning beyond supervised demonstrations. It reports effective learning without parameter updates and identifies exploration, reward filtering, stability, and scale as key considerations.
1 Introduction
- The paper asks whether LLMs' in-context learning extends to contextual bandit reinforcement learning with stateful, single-step interaction episodes.Unlike supervised ICL, this setting learns online from external reward rather than annotated examples.Source: §1 · p. 1
- LLMs can learn new tasks interactively at deployment from human-, program-, or AI-generated reward, without demonstration data or parameter updates.The reward may be supplied by a human, a program that completes successfully, or another AI.Source: §1 · p. 2
- Effective ICRL requires relatively high stochasticity for exploration and performs best when retaining only positive-reward triplets.The model must generate positive triplets through its own exploration rather than receive them from an expert annotator.Source: §1 · p. 2
- Stochastic ICRL samples past episodes into prompts to encourage exploration, stabilize performance, shorten contexts, and allow learning with negative signals.The paper also observes sudden performance dips followed often by quick recovery.Source: §1 · p. 2
- Across Qwen2.5 sizes from 0.5B to 72B, performance correlates strongly with scale while method rankings for performance and stability remain consistent.Source: §1 · p. 2
- On Banking77, Qwen2.5-7B improves from 6.2% zero-shot accuracy to 72.2% with Stochastic ICRL, without gold labels or parameter updates.The paper reports that bandit ICRL consistently and significantly enhances LLM performance.Source: §1 · p. 2
2 In-context Reinforcement Learning
In contextual bandit ICRL, a language model receives an input, predicts an output, and learns in context from the reward. The section presents Naive and positive-reward-only variants, then introduces stochastic prompt construction to encourage exploration.
2 In-context Reinforcement Learning
- ICRL treats the LLM as a policy that conditions on prior episodes in its context for in-context policy improvement, without treating those episodes as part of the RL state.The paper's setting is contextual bandit RL, where each episode is one step.Source: §2 · pp. 2–3
2.1 Naive and Naive+ ICRL
- Naive stores each input, prediction, and reward as an episode, while Naive+ discards negative-reward episodes and both use a sliding window when context fills.Even with positive examples, ICRL relies on the model generating and iteratively refining its own outputs rather than using fixed annotated demonstrations.Source: §2.1 · pp. 3–4
- Naive ICRL does not learn effectively, whereas Naive+ is highly effective, especially at relatively high sampling temperature.The contrast indicates that negative-reward examples contribute to Naive's failure.Source: §2.1 · p. 3
2.2 Stochastic ICRL
- Stochastic ICRL samples past episodes independently for each prompt and retains only positive-reward episodes, using prompt-composition variation to encourage exploration.It is presented as an alternative to raising sampling temperature, with greater stability.Source: §2.2 · p. 4
- A smaller pkeep delays context-window saturation relative to Naive, but enough interactions eventually saturate even the largest model windows.Source: §2.2 · p. 4
3 Experimental Setup
The experiments evaluate off-the-shelf instruction-tuned LLMs on challenging many-label classification tasks framed as contextual bandits. They use binary reward feedback, compare against zero-shot and supervised ICL settings, and test semantic versus abstract labels.
Models
- The study uses instruction-tuned Llama 3.1 8B and Qwen2.5 models across sizes, plus Gemini 1.5 Flash 8B for the hardest tasks.Predictions use constrained decoding in multi-turn chat format.Source: §3 · p. 4
Tasks
- Five classification benchmarks span 6 to 150 labels; their labels determine rewards but are never shown to the model.The tasks are Banking77, CLINC150, NLU, TREC, and TREC-fine.Source: §3 · p. 4
- TREC and TREC-fine each use 5k training examples, while all test sets are subsampled to 500 examples to reduce computational cost.NLU has no standard test set, so its train and test splits are created by the authors.Source: §3 · p. 5
Semantic vs. Abstract Class Names
- Abstract-label experiments replace semantic class names with meaningless strings, while semantic labels are the default.Semantic class names can provide clues about which inputs map to a class.Source: §3 · p. 5
Rewards and Prompt Design
- Binary correctness feedback is presented in natural language, and running test accuracy is measured using held-out examples every 500 training steps.The zero-shot comparison uses no in-context examples; supervised ICL is an upper bound with the maximum possible number of gold demonstrations and is not directly comparable to ICRL.Source: §3 · p. 5
4 Results and Analysis
Naive+ and Stochastic ICRL improve held-out accuracy over zero-shot and Naive, with performance generally rising as examples accumulate. Reward-signal ablations, abstract-label tests, and scaling analyses reveal both the conditions for learning and its limits.
LLMs Learn In-Context From Their Own Predictions and Rewards
- Naive+ and Stochastic improve over zero-shot by 28.6–74.4% for Llama and 29.2–68.4% for Qwen, with gains often approaching supervised performance and continuing over time.The trend is especially evident on Banking77, CLINC150, and NLU, where high label counts demand more exploration.Source: §4 · pp. 5–6, Figure 2 · p. 6
Reward Signals Are Crucial, but Mistakes Remain Challenging
- Removing or inverting rewards yields negligible gains over zero-shot, indicating that learning depends on reward signals rather than merely including domain examples.Experiments without reward also show no improvement over zero-shot.Source: §4 · pp. 6, 8, Figure 3 · p. 6
- Stochastic partially retains learning with negative outcomes and remains robust when 10% of rewards are inverted, whereas Naive works effectively only with positive-reward episodes.The authors report that Stochastic’s performance is negatively impacted by negative outcomes.Source: §4 · p. 6, Figure 3 · p. 6
- The ablations identify an implicit-reasoning limit: models do not reliably learn from mistakes without explicit reasoning.The paper contrasts this limitation with explicit reasoning such as Chain-of-Thought.Source: §4 · p. 6
Label Semantics Contribute, but ICRL Occurs Without It
- With abstract labels, Qwen and Llama improve over time without exemplars; Gemini excels with an initial mapping but struggles in purely exploratory learning.With one exemplar per label, ICRL nearly reaches the supervised-ICL upper bound, but some cases still require online learning: Llama on CLINC150 with Stochastic reaches non-negligible accuracy only after 3k steps.Source: §4 · p. 7, Figure 4 · p. 7
- Abstract-label experiments without reward show no improvement over zero-shot, supporting the conclusion that reward—not domain exposure alone—drives learning.The authors omit those experiments from the figures because they showed no learning effect.Source: §4 · pp. 7–8
Bigger Models Are Better at ICRL
- Across Qwen models from 500M to 72B, both Naive+ and Stochastic improve substantially over zero-shot, while smaller models plateau at lower accuracies.The reported pattern indicates that ICRL performance correlates positively with model size.Source: §4 · p. 8, Figure 5 · p. 8
Stochastic is More Stable Than Naive+
- Stochastic learning is more stable than Naive+ on Banking77, CLINC150, and NLU across model scales; stability results for TREC and TREC-fine are inconclusive.Stability is quantified as Spearman’s rank correlation between accuracy and time step; larger models generally show higher stability.Source: §4 · pp. 8–9
5 Related Work
This work extends research on in-context learning to contextual-bandit reinforcement learning, while distinguishing innate learning from approaches that rely on explicit statistics or fine-tuning. It also connects the study to limits in learning from mistakes and to the context-window capacity that makes continual in-context learning possible.
Supervised ICL
- The authors present their work as the first evidence that LLMs can perform contextual-bandit reinforcement learning in context, beyond supervised learning (i.e., standard ICL).Prior work established that LLMs learn new tasks in context through supervised examples.Source: §5 · p. 9
ICRL
- Longer context windows support continual ICRL improvement until the context saturates, but the authors do not observe effective learning from negative-reward episodes.They suggest that learning from mistakes may require explicit reasoning, leaving this as a direction for further study.Source: §5 · p. 9
- Unlike approaches that track UCB statistics or fine-tune, Naive+, which filters negative outcomes, and Stochastic, which varies prompt construction, elicit strong contextual-bandit learning.Earlier multi-armed-bandit work used a simpler stateless setting, while a concurrent contextual-bandit study tracked learning statistics externally.Source: §5 · p. 9
- Prior multi-step RL toy studies found that models generally could not learn; Mirchandani et al. (2023) attributed failure to inability to explore, as also observed with Naive here.The cited works prompt models to improve trajectories or simulate policy iteration.Source: §5 · p. 9
Transformers and RL
- This study differs from work training Transformers from scratch for sequential decision-making because it examines ICRL emerging in pretrained LLMs without fine-tuning for that purpose.The discussion also notes that its study depended on recent increases in LLM context-window length.Source: §5 · p. 10
6 Discussion and Limitations
The discussion characterizes ICRL as an innate capability of off-the-shelf LLMs, with exploration and filtering negative-reward episodes central to the reported learning trends. It also identifies boundaries in the task setting, computational demands, and implicit reasoning about negative episodes.
Discussion
- The study outlines a straightforward ICRL algorithm and a more stable variant using stochastic prompt construction, and examines scaling, stability, reward signals, and abstract labels.Its focus is off-the-shelf LLMs in the contextual-bandit setting.Source: §6 · p. 10
- Exploration combined with filtering negative-reward episodes yields strong ICRL trends, while the dependence on that filtering leaves effective reasoning about negative episodes unresolved.Stochastic prompt construction can aid exploration.Source: §6 · p. 10
Limitations
- The experiments use classification benchmarks and binary rewards in a one-step contextual-bandit setting, leaving applicability to nuanced-reward or multi-step RL tasks open.The authors point to math and coding tasks as examples with multiple steps and complex evaluation challenges.Source: §6 · p. 10
- ICRL is relatively compute-intensive, especially after many episodes; Approximate ICRL reduces certain overheads but trades compute for robustness.Further reducing computational demands remains open.Source: §6 · p. 10
A Evaluation Measures in the Appendix
The appendix defines regret as accumulated incorrect predictions, complementing running train accuracy as an evaluation measure. It also reports sensitivity to temperature, context downsampling, and episode retention, alongside practical context-window and computational considerations.
A Evaluation Measures in the Appendix
- Regret measures accumulated incorrect examples over learning, capturing both final performance and how quickly it is reached; train accuracy is the running mean over the latest 256 episodes.Lower regret reflects reaching high performance with fewer mistakes.Source: A Evaluation Measures in the Appendix · p. 17
B.1 Naive and Naive+ ICRL Hyperparameters
- For Naive+, T = 2.0 balances performance and stability; Naive remains relatively poor across temperatures, while higher Naive+ temperatures can be stronger but less stable.For experiments combining Naive+ and Stochastic, the higher zero-shot accuracy at T = 1.0 or T = 2.0 is used as the baseline.Source: §B.1 · p. 17
B.2.1 Downsampling Strategies
- With context limits artificially set to 4k or 8k tokens, start-biased downsampling slightly outperforms unbiased, which slightly outperforms end-biased; the study uses unbiased as the straightforward default.For Banking77 and CLINC150, longer contexts generally help and strategy differences are described as negligible.Source: §B.2.1 · p. 17, Figure 8 · p. 19
B.2.2 Hyperparameter Tuning and Sensitivity
- Intermediate pkeep values learn better on both Llama and Phi; pkeep = 0.1 balances performance with shorter contexts, while too-low values retain too few examples.Without stochasticity at pkeep = 1.0, ICRL struggles, particularly on Phi; the authors fix pkeep = 0.1 for subsequent Stochastic experiments.Source: §B.2.2 · pp. 17–18, Figure 7 · p. 19
- The authors fix Stochastic sampling temperature at T = 1.0 without optimizing it, noting that a better selection could improve performance.They leave temperature optimization for future work.Source: §B.2.2 · p. 19
B.3.1 Stochastic ICRL Computational Costs
- Before context saturation, Naive approaches can reuse cached computations, whereas Stochastic requires a freshly constructed context for each input.The text attributes this to the exceptionally low probability of repeating a context or prefix.Source: §B.3.1 · pp. 19–20
B.3.2 Method
Approximate ICRL reduces the cost of Stochastic ICRL by maintaining a limited set of contexts that can be expanded rather than rebuilt for every input. Its approximation depends on available resources and has a boundary at contexts exceeding the model’s window length.
B.3.2 Method
- Approximate ICRL maintains a limited number of contexts and gradually expands them with new episodes, avoiding repeated construction of fresh contexts.It shares Stochastic’s focus on exploration through prompt stochasticity and, like Naive+, excludes negative-reward episodes.Source: §B.3.2 · p. 20
- At each step, the algorithm samples one stored context for prediction and, after positive reward, independently adds the episode to each context with probability pkeep.The algorithm uses uniform context sampling; exact probability-based sampling performed poorly because some contexts quickly became unlikely to be selected.Source: §B.3.2 · p. 20
- Approximate ICRL’s resource-dependent approximation becomes Stochastic with exact context sampling as K approaches infinity, but it is limited for contexts beyond the LLM window length.The authors identify overcoming this window-length limitation while retaining efficiency as future work.Source: §B.3.2 · pp. 20–21
- Stochastic was significantly faster in the authors’ setup, particularly in low-memory conditions where caching is unavailable and contexts contain only pkeep% of Naive+ episodes.This is an empirical result reported for their setup, not a general speed guarantee.Source: §B.3.2 · p. 20
B.3.3 Results
Approximate performs almost as well as Stochastic ICRL across tasks with Llama, but its effectiveness is sensitive to the model and number of contexts. It also processes substantially fewer tokens than Stochastic ICRL.
B.3.3 Results
- Approximate performs almost as well as Stochastic ICRL across tasks with Llama, but deteriorates quickly with Phi.The experiments test Approximate on Llama and Phi, using K=8 unless otherwise specified.Source: §B.3.3 · p. 21
- With K > 32, Phi learns effectively; at K=8, one label-biased context can spread its bias and collapse performance.The authors report that recovery is empirically possible but appears unlikely.Source: §B.3.3 · p. 21
- Stochastic processes two orders of magnitude more tokens than Approximate over full ICRL runs.Source: §B.3.3 · p. 21
C Experimental Setup
The appendix describes the hardware and prompts used in the experiments, estimates episode capacity under different context windows, and compares context-sampling choices.
C Experimental Setup
- Experiments used A100, H100, and A6000 GPUs, with multi-GPU configurations for models of 14B parameters and larger.Models of 7B parameters or smaller used one 80GB A100/H100 or two 48GB A6000 or 40GB A100 GPUs.Source: C Experimental Setup · p. 21
- Figure 10 compares context counts and sampling strategies: Phi needs K ≥ 32 to learn, while uniform sampling consistently beats exact sampling for Llama.Llama can learn across the tested context counts, though higher K performs better; Phi is more sensitive to the approximation.Source: Figure 10 · p. 22
C.1 Prompt Design
- The Llama prompt examples show two in-context examples, with ICRL feedback indicating the correct answer and instructing the model to use feedback.The prompt examples are provided for Llama.Source: §C.1 · p. 22, Figure 11 · p. 23, Figure 12 · p. 23
C.2 Context Windows and Episode Capacity
- Episode capacity is conservatively estimated by fitting observed examples in descending token length and assuming longest outputs and maximum-length reward messages.The estimate uses each model's maximum context window; Llama is also evaluated at 4,096 and 8,192 tokens for Banking77 and CLINC150.Source: §C.2 · p. 22
C.3 Datasets
The experiments cover five classification tasks and abstract-label variants, including intent, home-domain, and question classification. The tasks pose challenges because labels can be numerous or subtly different, and their full label lists are generally not given.
C.3 Datasets
- The experiments use five classification tasks and their abstract-label variants, with maximum episode capacity tabulated by model, task, and context window.Source: §C.3 · p. 23, Table 1 · p. 24
- Banking77 and CLINC150 are intent-classification datasets with 77 and 150 labels, respectively; CLINC150 out-of-scope queries are excluded.Source: §C.3 · pp. 23–24
- NLU covers home-domain human-robot interaction queries across 68 categories, while TREC and TREC-fine classify questions into 6 and 50 labels.Source: §C.3 · p. 24
- The label sets are numerous and sometimes subtle, and potential labels are generally not listed except in With Exemplars abstract-label experiments.Models must follow previously discovered labels or explore for other suitable ones.Source: §C.3 · p. 24
D Additional Results
Additional results detail Approximate's behavior across tasks and models, report accuracy and stability measures, and compare its token costs with Stochastic ICRL.
D Additional Results
- Figures 13–18 visualize test accuracy, a 256-step running average of training accuracy, per-context training accuracy, and per-context hit rate.They cover Banking77 and CLINC150 for Llama with exact or uniform sampling, and for Phi with uniform sampling.Source: Figure 13 · p. 25, Figure 14 · p. 25, Figure 15 · p. 26
- Tables 2–4 report initial and final accuracy and regret; Tables 5–6 give Qwen2.5 size-specific results and stability metric ρ.Source: Table 2 · p. 28, Table 3 · p. 28, Table 4 · p. 29
- Stochastic processes two orders of magnitude more tokens than Approximate over full ICRL runs.Source: Table 7 · p. 31