PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung

arXiv:2608.16707v1cs.CLcs.AI

TL;DR

LLM decision-making agents must balance exploration and exploitation, but natural-language task representations introduce semantic information absent from classical bandit formulations. The paper formalizes this with semantic bandits and finds that labels and reward polarity systematically bias exploration, improving or degrading performance depending on alignment.

  • Problem

    Existing classical reinforcement-learning evaluations omit semantic information in natural-language tasks, limiting evidence about how LLM agents balance exploration and exploitation.

  • Method

    The paper introduces a semantic multi-armed bandit that explicitly models textual action labels and scenario context alongside reward distributions.

  • Results

    Semantically informative labels reduce exploration under helpful alignment but increase regret under misalignment, while negative rewards trigger substantially more exploration than equivalent positive rewards.

  • Takeaways & Limitations

    Semantic context can aid adaptation or induce systematic errors, making its effects important to understand for reliable LLM-agent deployment.

  • Takeaways & Limitations

    Limited replicates and a simple bandit environment restrict conclusive statistical claims and the scope of the paper’s conclusions.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.

1 Introduction

The paper introduces the semantic multi-armed bandit to capture how textual action labels and reward polarity shape LLM exploration beyond classical reinforcement-learning frameworks. It shows that semantic priors can improve exploitation when aligned with rewards, misdirect decisions when misaligned, and reduce principled exploration.

  • Framework: The semantic multi-armed bandit extends the canonical bandit problem by parametrizing textual representations alongside standard reward distributions.This framework is designed to explain exploration biases arising from semantic priors, which have no formal counterpart in classical reinforcement learning.
  • Primary findings: Action nomenclatures significantly influence exploration even when textual labels are irrelevant to the task space.Aligned labels support low-regret exploitation with little exploration, whereas misaligned labels misdirect models toward low-reward actions and increase regret.
  • Primary findings: Negative reward values trigger substantially more exploration than positive values, indicating that models interpret rewards relative to an expected scale rather than as abstract numerical signals.Reward polarity acts as a strong contextual cue in in-context reinforcement learning.
  • Primary findings: Negative rewards make models much less likely to exploit biases from action nomenclature.The interaction between reward polarity and labels is illustrated in Figure 1.
  • Implications: Semantic priors can be strong enough to nullify principled exploration in a simple decision-making problem, motivating scrutiny of their effects in real-world deployments.The paper identifies this as a previously undercharacterized source of potentially misleading decision-making bias.

2 Related work

Prior work frames in-context reinforcement learning as sequential decision-making from reward feedback, while research on LLM reasoning questions whether benchmark success reflects generalized abstraction. These concerns motivate examining how language and learned priors influence LLM agent behavior.

  • In-context reinforcement learning (ICRL): In-context learning lets pretrained LLMs solve novel problems from task descriptions and examples, with reasoning logic learned during pretraining and adapted to demonstration format.Min et al. argue that models do not learn directly from in-context examples.
  • In-context reinforcement learning (ICRL): ICRL studies LLMs as decision-making agents that adapt entirely from in-context reward signals across sequential turns.The paper distinguishes ICRL as a special class of in-context learning and notes substantial application-focused decision-making research.
  • Reasoning in LLMs: LLM reasoning performance remains highly sensitive to structure-preserving linguistic perturbations, potentially indicating probabilistic pattern-matching rather than generalized abstract reasoning.This sensitivity appears across logical and mathematical reasoning tasks despite impressive benchmark performance.

3 Problem definition

The paper contrasts classical agents, which operate on symbolic renderings preserving only formal structure, with LLM agents, which act on textual renderings containing linguistic and prompting information. It therefore introduces the semantic bandit, an extension of the multi-armed bandit that explicitly represents this additional structure.

  • Classical reinforcement learning: A classical reinforcement-learning environment is defined by states, actions, reward distributions, and episode length, with policies invariant to information outside the formal structure.The environment is represented as {S, A, R, T}, while symbolic rendering preserves only formal structure.
  • Classical multi-armed bandit: A multi-armed bandit has one state, k arms with unknown reward distributions, and T turns in which the agent balances exploration against exploitation to maximize cumulative reward.The single-state formulation removes the transition function and discount factor.
  • LLMs as decision-making agents: An LLM agent receives a textual rendering that includes environmental language, scenarios, instructions, and prompting details, then maps a linguistic problem description to a linguistic action.The textual rendering maps S × A × T × C into token sequences, and the model’s output is passed to a parser.
  • The semantic bandit: The semantic bandit extends the MAB by making language-related structure explicit through an action nomenclature and a scenario alongside the standard MAB tuple.Because the LLM engages with a natural-language description, the problem is not fully specified by the reward distributions and episode length alone.

4 Experimental design

The experiments instantiate a three-arm semantic bandit and vary action nomenclature, reward scale, thematic scenario, and reward variance to study exploration–exploitation behaviour. They evaluate multiple LLM agents against UCB1 using regret, exploration counts, and scale-sweep exploration probability.

  • Experimental setup: The semantic bandit has one state, three actions representing high, medium, and low rewards, Gaussian reward distributions with shared variance, and three scenario configurations.The scenarios are bandit, farm, and clothing recommendation.
  • Experimental conditions: Four action nomenclatures vary semantic content: alphanumeric control, sentiment, ordinal ranking, and domain-specific world knowledge labels.Non-alphanumeric nomenclatures are tested in helpful and misleading configurations by assigning semantically favoured labels to the highest- or lowest-reward arms.
  • Reward conditions: The main reward conditions use High+ means (75, 50, 25) and High- means (−25, −50, −75), with shared variance σ ∈ {12.5, 6.25, 0}.Low+ and Low- conditions scale all reward values down by a factor of 100.
  • Reward scale sweep: A reward scale sweep varies one observed reward r ∈ [−2, 2] in 0.1 increments after forcing the first action to the semantically favoured arm, then measures subsequent switching.The scale-sweep protocol records turns until a different arm is selected and defines exploration probability as selecting a different arm at turn 1.
  • Evaluation protocol: Three instruction-tuned LLMs are evaluated in 10 replicates per main condition and compared with UCB1, with action labels resampled and arm order shuffled across replicates.The models are OLMo-3.1-32B-Instruct, Qwen3-32B, and Gemini 3.1 Flash Lite.
  • Evaluation metrics: Performance is measured using normalized cumulative regret, exploration count per turn across the three arms, and exploration probability in the scale-sweep experiments.Exploration count ranges from 1 for no exploration to 3 for full exploration, while lower normalized regret indicates more effective exploration–exploitation behaviour.

5 Results

Action nomenclatures bias LLM exploration toward semantically favoured actions, improving regret when semantic priors align with rewards but degrading performance when they do not. Exploration also increases under explicit instructions and negative observed rewards, revealing systematic effects of prompting and reward polarity.

  • Action nomenclature: Alphanumeric labels produce exploration comparable to the UCB symbolic baseline, whereas ordinal, world, and sentiment labels bias models toward exploiting semantically favoured actions.These results use positive rewards, helpful nomenclatures, and no variance.
  • Action nomenclature: Helpful semantic biases can yield much lower regret than symbolic methods, but misleading priors can cause dismal performance when labels conflict with reward structure.Table 2 reports normalized cumulative regret by model and nomenclature type under high reward scale and no variance.
  • Action nomenclature: Ordinal labels create the strongest bias, positive-sentiment words a weaker bias, and world-knowledge effects vary by task, strengthening on clothing recommendations relative to farming.The model is especially biased when arms are explicitly ordered.
  • Model differences: OLMo explored little, Qwen3 explored consistently with alphanumeric labels but purely exploited ordinal labels, and Gemini generally explored rapidly except with world knowledge on clothing.Each LLM displayed distinct exploration behaviour across nomenclatures and tasks.
  • Prompt debiasing: Warnings combined with explicit exploration instructions significantly raise exploration coverage, while warnings alone produce only a slight increase in exploration count.The intervention removes the need for the model to infer how to balance exploration and exploitation.
  • Reward polarity: Negative rewards trigger significantly more exploration than positive rewards, except when positive rewards already saturate the task at full exploration.The probability of immediate exploration is much higher for observed values below zero, with a larger and more abrupt effect for Qwen3 and OLMo than Gemini.

6 Discussion

The discussion argues that semantic context can completely determine LLM behavior even in a simple, noiseless three-arm bandit. This sensitivity complicates evaluation and raises concerns about exploration stability in more complex environments.

  • Core finding: Semantic influence completely determined behavior in a stationary, three-arm bandit with no reward variance.The authors identify this as the paper’s most important finding, beyond merely showing sensitivity to semantic context.
  • Implications: Failure on a simple synthetic task raises concerns about exploration stability in more complex environments.The authors note that realistic deployment contexts were not explored.
  • Evaluation: Evaluations using naturalistic, semantically coherent labels conflate reward-driven reasoning with semantic-prior exploitation.Practitioners cannot determine the proportion attributable to each process from such evaluations alone, creating concerns about generalizability.

7 Limitations and Future Work

The study is limited by few replicates and a simple semantic-bandit environment, which prevent conclusive statistical claims and constrain the scope of its conclusions. Future work should test more complex bandit environments.

  • Limitations: Limited replicates, caused by computational constraints, prevent conclusive statistical claims.The authors identify the number of replicates as a major limitation and call for further experimentation.
  • Limitations: The simple semantic-bandit environment demonstrates sensitivity to semantic priors but limits the scope of the claims.The environment is effective even in simple contexts, yet its simplicity restricts generalizability.
  • Future Work: Future work could evaluate more complex bandit environments.

8 Conclusion

The paper introduces the semantic bandit, an extension of the multi-armed bandit that makes action nomenclature and scenario context explicit for LLM decision tasks. It finds that semantic labels and reward polarity systematically bias exploration and exploitation.

  • Conclusion: The semantic bandit formalizes action nomenclature and scenario context in multi-armed decision tasks for LLM agents.It extends the classical multi-armed bandit setting.
  • Conclusion: Semantically informative labels reduce exploration in favour of exploitation, helping under alignment but severely degrading performance under misalignment.The effect depends on whether the labels align with the reward structure.
  • Conclusion: Negative reward values trigger substantially more exploration than formally equivalent positive values.Reward polarity therefore influences exploration behaviour even when rewards are formally equivalent.

Ethics Statement · A Experimental Design

The paper warns that semantic biases may manipulate LLM-agent behavior, while emphasizing that the findings come from a small set of controlled experiments. The appendix documents implementation choices, prompting alternatives, and granular analyses of reward, variance, model-scale, and arm-count conditions.

  • Ethics Statement: Semantic biases identified in the paper could be used to manipulate LLM-agent behavior.The authors present this as a caution to practitioners but acknowledge potential adversarial use.
  • Ethics Statement: The findings derive from a small set of experiments conducted in a controlled environment.Practitioners are advised to evaluate the effects in their own environments.
  • A Experimental Design: The appendix’s first section reports implementation details and results associated with experimental design choices.It includes implementation details, prompt templates, nomenclature sets, and alternative prompting schemes.
  • A Experimental Design: The first appendix section covers implementation details in A.1 and prompt templates and nomenclature sets in A.2.Alternative prompting-scheme results are listed under A.4 and A.3.
  • A Experimental Design: The appendix’s second section provides granular details of the main experimental results, including low reward scales and added reward-distribution variance.The passage states that results are stable across variance conditions.
  • A Experimental Design: The granular-results section is organized around cumulative regret, exploration count, Qwen-model scaling, and comparisons between three-arm and five-arm scenarios.These analyses appear in sections B.1 through B.4, respectively.

A.1 Implementation Details … B.2 Exploration Count

The paper details implementation and prompt configurations, then reports supplementary results on history formats, explicit debiasing, regret, and exploration. Across these analyses, summarized histories outperform full histories, exploration responds to prompting and reward sign, and results are organized across models, domains, scales, nomenclatures, and variance conditions.

  • A.1 Implementation Details: OLMo and Qwen use 150 thinking tokens, Gemini uses unrestricted thinking tokens, and other settings follow model-specific defaults.The setup contrasts open-source models with closed-source frontier reasoning models.
  • A.1 Implementation Details: Initial evaluations dropped Llama-3.1-8B-Instruct, Qwen3-8B, and Qwen3-14B because the 8B models explored insufficiently and Qwen3-14B matched Qwen3-32B.The larger models were retained because of known performance gains at scale.
  • A.2 Prompt Templates: The experiments use standardized prompt templates, with runtime-filled variables and a context block appearing only in the clothing contextual-bandit condition.Task-specific prompts vary by domain while other template fields remain constant.
  • A.3 Alternative History Passing Formats: Summarized history presents each arm’s observed mean reward and pull count, and it performs better than passing the full raw action-reward history at turn @9 for Qwen3-32B.The comparison is reported in the alternative history-passing analysis.
  • A.4 Additional Results from Prompting with Explicit Debiasing: Explicit warning-plus-exploration instructions significantly increase exploration coverage, while warnings alone tend to increase exploration slightly in helpful and misleading framing.The reported results use no variance and high scales.
  • A.4 Additional Results from Prompting with Explicit Debiasing: Debiasing instructions usually raise regret under helpful nomenclature and lower it under misleading nomenclature, with highly inconsistent effects across domains, nomenclatures, and models.The additional tables report exploration count and cumulative regret by model, nomenclature, and debiasing condition.
  • B Additional Experimental Results from Main Experiments: Supplementary main-experiment results cover low reward scales and added reward variance, with stability across variance conditions and separate analyses of cumulative regret, exploration, scaling, arm count, and reasoning traces.Cumulative-regret tables span all nomenclatures, domains, models, and variance levels.
  • B.2 Exploration Count: Exploration count is measured as unique arms tried at turns 3 and 10, and negative rewards produce more exploration than positive rewards in no-variance, low-scale results.The exploration tables vary reward scale, nomenclature type, model, and variance condition.

B.3 Qwen Scaling Results · B.4 3 Arm and 5 Arm Comparison

Qwen3-8B almost never explored, suggesting it was too small for the task, whereas Qwen3-14B and Qwen3-32B behaved similarly. Across three-arm and five-arm settings, Qwen explored slightly less, OLMo slightly more, and Gemini no longer saturated exploration counts.

  • B.3 Qwen Scaling Results: Qwen3-8B almost never explored across the tested nomenclature conditions.The table reports normalized cumulative regret and exploration count at turn 9 by Qwen model and nomenclature.
  • B.3 Qwen Scaling Results: Qwen3-14B and Qwen3-32B behaved fairly similarly in the scaling results.This similarity was reported across the evaluated domains and nomenclature conditions.
  • B.3 Qwen Scaling Results: The authors interpreted Qwen3-8B’s limited exploration as evidence that the model was too small to adequately perform the task.This interpretation follows the reported near-absence of exploration by Qwen3-8B.
  • B.4 3 Arm and 5 Arm Comparison: The comparison reports normalized cumulative regret and exploration count at turn 9 for three-arm and five-arm outputs.Results are organized by model, nomenclature, and domain, with H denoting helpful and M misleading framing.
  • B.4 3 Arm and 5 Arm Comparison: Exploration counts are not directly comparable between three-arm and five-arm settings because five arms increase the maximum count to 5.The table explicitly cautions that the exploration-count scale changes with the number of arms.
  • B.4 3 Arm and 5 Arm Comparison: Qwen showed slightly less exploration and OLMo slightly more exploration in the three-arm and five-arm comparison.These directional differences are reported across the compared model outputs.
  • B.4 3 Arm and 5 Arm Comparison: Gemini no longer saturated the exploration-count metric in the expanded arm comparison.The change was particularly notable for the abstract bandit.

B.5 Sample Reasoning Trace · Turn 8: The model returns to the midweight t-shirt

The trace shows Gemini prioritizing semantically favored clothing options despite contradictory rewards and climate evidence. By Turn 8, it returns to the consistently high-reward midweight t-shirt and treats exploitation as optimal.

  • B.5 Sample Reasoning Trace: Gemini rejects exploring the untried tank top because its negative semantic label conflicts with the cold-climate context.The model had previously observed rewards of 25 for the flannel long-sleeve shirt and 50 for the midweight t-shirt.
  • B.5 Sample Reasoning Trace: Gemini re-explores the flannel long-sleeve shirt because of its positive semantic label despite observing a lower reward.The flannel had an observed reward of 25, compared with 50 for the midweight t-shirt.
  • B.5 Sample Reasoning Trace: On Turn 6, Gemini selects the midweight t-shirt because it has the highest observed reward.The reasoning compares the midweight t-shirt’s observed mean of 50.0 after four trials with the flannel’s 25.0.
  • B.5 Sample Reasoning Trace: Gemini nevertheless selects the flannel on Turn 7, reasoning that its cold-climate suitability could imply a higher true mean despite its lower observed mean.The flannel’s observed mean is based on only two trials, while its semantic fit to the climate motivates further exploration.
  • Turn 8: The model returns to the midweight t-shirt: On Turn 8, Gemini considers exploring the untested tank top but dismisses it as unlikely to yield high reward in the cold climate.The trace frames the decision as balancing exploration of untested items against exploiting higher satisfaction scores.
  • Turn 8: The model returns to the midweight t-shirt: Gemini returns to the midweight t-shirt, citing five tests with a consistent mean of 50.0 and concluding that continued exploitation is optimal.The t-shirt’s mean is described as significantly higher than the flannel long-sleeve shirt’s.
Loading 2608.16707v1…