PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning

Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim

arXiv:2610.11152v1cs.LGcs.CL

TL;DR

The paper asks whether direct in-context reinforcement learning uses reward as an RL-like learning signal or instead reflects in-context learning. It probes this with controlled reward and trajectory experiments, finding that reward changes have little effect and that trajectory imitation better explains improvement.

  • Problem

    It is unclear whether LLM improvement from in-context experience–reward pairs reflects reinforcement learning or in-context learning, which is known to be sensitive to input distributions and surface forms but less sensitive to label content.

    Source: §1 · p. 1

  • Method

    The paper studies direct ICRL by exposing models to raw trajectory–reward pairs and conducting controlled experiments that alter reward fields and trajectory properties.

    Source: §1 · p. 1

  • Results

    Direct ICRL improves more through trajectory imitation than reward semantics; reward impact on regeneration probability is about 14% of the trajectory’s, and shuffled or corrupted trajectories perform comparably to real ones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that direct ICRL is better understood as a special case of ICL, making input distribution and demonstrations potential priorities for agent memory design.

    Source: §1 · p. 2, §7 · p. 10

    The findings cover direct ICRL with raw trajectory–reward pairs and standard meta-prompts across six models and three benchmarks, and do not rule out stronger reward-aware behavior under other prompts or model families.

    Source: §8 · p. 10

Abstract

from arXiv · show

LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.

1 Introduction

The paper asks whether experience–reward pairs make inference-time improvement genuinely RL-like or whether it reflects in-context learning instead. Controlled tests of direct ICRL find that reward semantics have little effect, motivating an ICL-based reframing of agent memory.

1 Introduction

  • Direct ICRL is studied in its simplest controllable form, where the model receives raw trajectory–reward pairs without added memory storage, selection, or summarization.The authors define this setting to isolate how direct ICRL works.Source: §1 · p. 1
  • The study tests whether reward text shifts policy probabilities toward high-reward trajectories and away from low-reward ones, and which trajectory properties drive improvement.The trajectory question distinguishes semantic content, surface form, and task relevance.Source: §1 · pp. 1–2
  • The controlled experiments span three benchmarks and six open-source and proprietary models, probing reward changes, trajectory regeneration, and trajectory perturbations.Reward conditions include flipped, randomized, or removed values; trajectory tests vary memory sources and corrupt content while preserving surface form.Source: §1 · p. 2
  • Direct ICRL improves mainly through trajectory imitation rather than reward semantics, and even shuffled or corrupted trajectories work comparably to real ones.Reward impact on regeneration probability is about 14% of the trajectory’s; task relevance is the factor whose loss degrades performance.Source: §1 · p. 2
  • The authors argue that direct ICRL is better framed as a special case of ICL, shifting attention toward input distribution and demonstrations in memory design.They note that accumulating failures in memory may induce replay rather than exploration.Source: §1 · p. 2

2 Related Work

Prior work studies ICRL through training, abstraction, and direct trajectory–reward accumulation, while ICL research highlights input distribution and surface form. This paper examines direct ICRL through the ICL framework.

2 Related Work

  • ICRL methods include extra-training approaches, abstraction-based methods that turn trajectory–reward pairs into instructions, and direct methods that stack pairs.Direct ICRL is the category most closely resembling RL; the paper examines its underlying mechanisms.Source: §2 · p. 2
  • ICL solves problems from context without additional training, and prior work identifies input distribution and surface form—not labels—as key factors.The paper positions its analysis as a bridge between ICL and ICRL research.Source: §2 · p. 2
  • Figure 1 frames direct ICRL alongside controlled interventions on reward and trajectory perturbations.Source: Figure 1 · p. 3

3 Preliminaries

The preliminaries distinguish parameter-updating RL from context-based adaptation with frozen language models. They define direct ICRL and describe coding and agentic task settings.

3.1 Reinforcement Learning

  • RL seeks policy parameters that maximize expected discounted return over trajectories generated by the policy and environment.The transition kernel governs state changes, while the reward function supplies step rewards and the discount factor weights future rewards.Source: §3.1 · p. 3

3.2 In-Context Reinforcement Learning

  • ICRL adapts behavior through accumulated contextual experience with a frozen model; direct ICRL serializes raw trajectory–reward pairs into context without abstraction.Different ICRL variants differ in how the rendering function represents retained experience.Source: §3.2 · p. 3

3.3 Tasks

  • Coding tasks use single-action trajectories with unit-test pass rate as reward, while ScienceWorld uses multi-step text interactions and rewards.The task regimes allow precise reward interventions in coding and test sequential decisions and action-level credit assignment in the agentic setting.Source: §3.3 · pp. 3–4

4 Reward Semantics Do Not Drive Improvement

Across reward interventions, direct ICRL’s improvement is largely insensitive to reward content. A fixed-memory control likewise finds few reward-sensitive cases, separating reward effects from changing trajectory composition.

4 Reward Semantics Do Not Drive Improvement

  • The section intervenes on reward fields while holding trajectories fixed, tracking episode-over-episode improvement and using a fixed-memory control to remove trajectory-composition confounds.Source: §4 · p. 4

4.1.1 Setup

  • In the accumulating-memory setup, trajectories and rewards are appended after each episode, with true, flipped, random, and omitted-reward conditions compared against no-context random sampling.The experiments use 10 episodes on ScienceWorld and 50 on HumanEval+ and MBPP+, with three seeds.Source: §4.1.1 · pp. 4–5

4.2.2 Results

  • Of 360 task-model pairs, 14 (3.9%) were reward-sensitive, 133 (36.9%) reward-agnostic, and 213 (59.2%) showed no improvement over random sampling.Reward-sensitive cases per model ranged from 0 to 2 out of 30 across six models.Source: Table 1 · p. 5, §4.2.2 · p. 6

4.1.2 Results

  • Flipped rewards remain well above random sampling and closer to true-reward performance, while random or omitted rewards track true rewards across benchmarks.True-reward direct ICRL also improves over random sampling on every benchmark; the full sign flip does not reverse that improvement.Source: §4.1.2 · p. 5

4.2 Effect of Reward in the Fixed Memory Setting

  • The fixed-memory control holds the same trajectories across reward conditions, isolating reward-field differences from changes in accumulated memory.Five randomly sampled trajectories form each fixed prompt, with evaluation averaged across samples and three seeds.Source: §4.2 · p. 5, §4.2.1 · p. 6

4.2.1 Setup

  • The fixed-memory analysis classifies outcomes as no improvement, reward-agnostic improvement, or reward-sensitive improvement, across six models on HumanEval and MBPP.Source: §4.2.1 · p. 6

5 Trajectory Exposure, Not Reward, Drives Replay in Direct ICRL

Trajectory exposure induces replay; reward shifts it in the RL-aligned direction but weakly, while position and same-task raw trajectories have larger effects.

5.1 Setup

  • Using N=10 sampled HumanEval and MBPP pairs with Qwen-3.5-9B, the study tracks highest- and lowest-reward targets through leave-one-out probabilities and AST-match replay rates.Source: §5.1 · p. 6

5.2 Showing a Trajectory Makes the Model Replay It

  • 34.3% vs 16.3% Memory Replay Rate when trajectories are shown; τ−/τ+ replay rises from 0.7% to 3.1% and from 1.1% to 1.8%.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Reward Shifts Replay in the RL Direction, but Only Slightly

  • Reward shifts replay in the RL-predicted direction, but changes ΔLOO by at most 0.058, about 14% of the 0.416 trajectory-presence effect; τ− replay remains 2.5%.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Position and Raw Trajectories Change Replay; Reward Labels and Instructions Do Not

  • Position and same-task raw trajectories matter more: placing τ+ last raises replay to 13.2%, while zero rewards and anti-replay instructions barely reduce replay and Cross-task/Reflexion lower it to about 13%/12%.Source: Table 4 · p. 8, §5.4 · p. 8

5 Trajectory Exposure, Not Reward, Drives Replay in Direct ICRL

  • Replay depends more on trajectory exposure, position, and same-task raw trajectories than on reward labels, matching the ICL pattern for demonstrations.Source: §5.4 · p. 8

6 What Drives Improvement: Surface Form and Task Relevance

Surface-form perturbations leave performance comparable, but trajectories must remain relevant to and actual attempts at the target task.

6.1 Setup

  • The memory-composition analysis varies memory contents while holding imitation fixed, using Qwen-3.5-9B, N=10 trajectories, and five random samples per configuration.Source: §6 · p. 8, §6.1 · p. 9

6.2 Results

  • ICRL History performs similarly to or worse than Random Sampling Memory on ScienceWorld and MBPP, so self-bootstrapping does not significantly affect performance.Source: Table 5 · p. 9, §6.2 · p. 9

6.1 Setup

  • The eight memory configurations span source variations and trajectory perturbations, including same-task, cross-task, random-valid-action, shuffle, crossover, and token-insertion conditions.Source: §6.1 · p. 9

6.2 Results

  • Perturbed trajectories perform comparably to ICRL History, while cross-task and random-valid-action memories degrade performance, indicating task relevance and actual attempts matter.Source: §6.2 · pp. 9–10
  • Shuffle and Crossover preserve performance comparable to ICRL History, and Insert Token does so despite introducing a syntax error in MBPP.Source: §6.2 · p. 9, §6.1 · p. 9
  • Cross-task attempts and random valid actions both degrade performance: trajectories must be target-task attempts, not merely task-matched or surface-similar material.Source: §6.2 · pp. 9–10

7 Discussion

Direct ICRL’s improvements are better explained by trajectory imitation than reward semantics, and its patterns resemble in-context learning. The discussion argues that memory design may need to prioritize ICL factors while recognizing limits imposed by pretrained preferences.

7 Discussion

  • Direct ICRL is better understood as ICL with a reward field than as inference-time RL, with input distribution and trajectory surface form central to memory design.The authors argue that RL-style filtering, self-bootstrapping, and reward-based design may need reconsideration.Source: §7 · p. 10
  • Failure trajectories provide only weak avoidance signals, so accumulating failures may not reliably induce exploration in direct ICRL.The authors identify abstraction-based approaches such as Reflexion or Expel as alternatives, while noting that stabilizing text-gradient methods remains challenging.Source: §7 · p. 10
  • Reward semantics do not effectively override imitation preferences, bounding direct ICRL’s applicability when target behavior conflicts with pretrained preferences.The experiments do not distinguish whether models cannot read reward content or read it but favor other trajectories.Source: §7 · p. 10

8 Limitation

The experiments isolate direct ICRL mechanisms, leaving abstraction-based variants, alternative prompts and model families, and conflicting-reward regimes outside the claims’ scope.

8 Limitation

  • The analysis exposes raw trajectory–reward pairs and does not cover abstraction-based variants such as reflections, skill libraries, or extracted insights.Results under other prompt formats, model families, or reward regimes conflicting with model preferences are not ruled out.Source: §8 · p. 10
  • The study reports standard meta-prompts across six models and three benchmarks, so its findings do not establish behavior under all prompts or model families.Models trained specifically for ICRL, such as the Decision Transformer family, are outside the reported scope.Source: §8 · p. 10
  • The reward manipulations do not fully characterize counterfactual regimes where reward semantics conflict with a model’s prior preferences.Source: §8 · p. 10

A Extended Discussion of Limitations

The limitations discussion confines the findings to direct ICRL under tested prompts, model families, and reward regimes, leaving abstraction-based methods and several boundary conditions unresolved.

Scope: direct ICRL only

  • The analysis is limited to direct ICRL with raw trajectory–reward pairs and does not establish whether findings extend to abstraction-based methods.Reflections, skill libraries, and extracted insights are excluded because they add degrees of freedom that are difficult to control in the same framework.Source: A Extended Discussion of Limitations · p. 13

Prompt formats and model families

  • The study uses standard meta-prompts and does not rule out stronger reward-aware behavior under other prompts or model families.Decision Transformer models are out of scope; models above 100B parameters are also untested.Source: A Extended Discussion of Limitations · p. 13

Counterfactual reward regimes

  • The experiments do not fully characterize counterfactual reward regimes in which reward semantics conflict with pretrained preferences.The authors say the override boundary likely depends on the gap between target behavior and the model’s existing preferences.Source: A Extended Discussion of Limitations · p. 13

B Experimental Details

The experimental details specify model and prompt settings, memory handling, and statistical analyses of reward sensitivity. Across benchmarks, task-level tests find no significant reward-condition differences, while per-task analysis finds substantial reward-agnostic improvement.

B.1 Model Setting

  • Model settings use temperature 0.8 except for GPT and Gemini, with maximum new-token limits of 32 for ScienceWorld and 1024 for MBPP+ and HumanEval+.GPT-5-nano and Mini use medium thinking intensity; Gemini uses low thinking effort.Source: §B.1 · p. 13

B.2.1 Meta-prompt for each experiment

  • Coding prompts request a code completion, ScienceWorld prompts direct imitation of higher-return memories, and the anti-prompt and Reflexion prompts respectively encourage different attempts and summarize failures.Source: §B.2.1 · p. 13

B.2.2 Experiments Details

  • Coding memories randomly retrieve 10 items from a pool, Top-k and Bottom-k retrieve ranked samples, and ScienceWorld uses all stored memories.Figure 2 runs took about five hours for 10 ScienceWorld episodes and about three hours for HumanEval and MBPP; ScienceWorld rewards are score changes on a 0–100 scale, with invalid or catastrophic behavior able to incur penalties such as −100.Source: §B.2.2 · p. 14

C.1 Statistical Analysis of Section 4.1

  • Across three benchmarks, all six task-level comparisons have confidence intervals containing zero and p-values above 0.4, with mean differences below 4% of each score range.Table 6 reports paired t-tests on final scores for 40 tasks per benchmark, averaging three seeds per task; intervals still allow small reward effects, and Wilcoxon tests give the same qualitative conclusion (all p > 0.4).Source: §C.1 · pp. 14–15, Table 6 · p. 14
  • The per-task tests use three seeds and have very low power, so the authors do not treat them as primary evidence.One HumanEval task had uncorrected p < 0.05 in the True vs. No Reward comparison, which they treat as expected sampling noise across 240 tests.Source: §C.1 · p. 14

C.2 Per-Task Analysis of Section 4.1

  • Figure 3 classifies task-level improvement using the average of three seeds; reward-agnostic improvement is substantial, including 50% of MBPP+ improvement and 42% for ScienceWorld.Even MBPP+, the most reward-sensitive benchmark, has only half its improvement categorized as reward-sensitive.Source: §C.2 · p. 15, Figure 3 · p. 15

C.3 Ablation of Section 4.1

The ablations find that verbal rewards and Reflexion can produce performance comparable to numerical-reward ICRL, while memory selection also affects outcomes. The setup varies memory contents and measures leave-one-out effects and replay behavior.

C.3 Ablation of Section 4.1

  • Repeating the ScienceWorld experiment with Llama-3.1-8B-Instruct yields similar findings, with seed variance in True, Flipped, and Random conditions.The authors conclude the finding is not specific to Qwen3.5-9B.Source: §C.3 · p. 15, Figure 4 · p. 16
  • Reflexion achieves performance comparable to ICRL-True, but text-gradient methods including Reflexion are noted as high-variance.The authors leave methods to address this variance and advance ICRL to future work.Source: §C.3 · pp. 15–16, Figure 6 · p. 17
  • Verbal rewards perform similarly to numerical rewards, though flipping verbal rewards lowers performance and does not produce a monotonic decline.The authors suggest sentence-level enhancements improve reward reading, while trajectory attraction masks its effect.Source: §C.3 · p. 16, Figure 5 · p. 17
  • Top-k memory selection outperforms Bottom-k, while random selection performs worse than Top-k under seed variance.Figure 7 compares these selection policies on MBPP+ and HumanEval+.Source: §C.3 · p. 16, Figure 7 · p. 18

C.4.1 Setup

  • The setup defaults to ten problem-solution-reward trajectories and changes one memory component at a time, measuring ΔLOO and behavior separately for failure and success targets.ΔLOO measures the score drop after removing the target trajectory; Target rate and Memory rate measure exact target matches and matches to any memory trajectory.Source: §C.4.1 · pp. 16, 18
  • The thirteen memory constructions include true, failed, flipped, randomized, and verbal rewards, plus an instruction to produce something different from memory.Specific failure and specific flipped conditions alter only the target trajectory’s reward.Source: §C.4.1 · pp. 16–17
  • Other constructions test chain-of-thought, target position, duplication, and cross-task memories, with cross-task results listed but not analyzed because the trajectories share no structure with the target.The target may be placed first or last, or duplicated five times to increase memory size to fifteen.Source: §C.4.1 · p. 18
  • Trajectory Only removes rewards, while Reflexion Only uses reflection text without raw trajectories and Reflexion + Trajectory includes both.Problem Only supplies the problem without memory; its 16.3% Memory Replay Rate is a chance-level baseline for matching a held-out memory pool.Source: §C.4.1 · p. 18

C.4.2 Analysis

Across memory analyses, reward changes have small, inconsistent effects, whereas trajectory presence, position, and frequency have larger effects. Anti-prompting does not shift behavior, and Reflexion’s LOO effect depends on including the trajectory.

C.4.2 Analysis

  • Changing only the target trajectory’s reward yields ΔLOO and behavior values close to changing rewards across all trajectories.The model does not react more strongly when the reward change is focused on the target.Source: §C.4.2 · p. 18
  • Position and frequency matter more than reward: Order Last gives a 14.9% τ − Target Replay Rate, while Duplicate reaches 22.8%.Both conditions also push τ − Memory Replay Rate above 40%; these effects are much weaker for τ +.Source: §C.4.2 · p. 18
  • Anti-prompting leaves ΔLOO at 0.33/0.16 and Memory Replay Rate at 30.0/30.2%, close to True Reward.The instruction to produce something different from memory is not enough to pull the model away from its content.Source: §C.4.2 · p. 19
  • Reflexion Only has ΔLOO near zero (−0.006/0.005) and about 11.7% Memory Replay Rate, while adding the trajectory yields ΔLOO 0.44/0.14.The authors attribute most of the LOO effect to the trajectory rather than the reflection.Source: §C.4.2 · p. 19
  • Reward variants produce small, inconsistent ΔLOO changes and memory replay rates near 30%, while trajectory-only memory remains close to true-reward memory.True Reward gives ΔLOO 0.36/0.17 (τ −/τ +); Trajectory Only gives 0.42/0.16 and about 33% Memory Replay Rate.Source: §C.4.2 · pp. 18–19, Table 7 · p. 19

C.5 Full Results of Section 6

  • Table 9 presents the complete results, including standard deviations, for the Table 5 experiment.The supplied passage does not state the table’s values.Source: §C.5 · p. 19

D Qualitative Results

The section presents qualitative examples of perturbed trajectories and reports checklist disclosures concerning assets, crowdsourcing or human-subject research, and LLM use.

D.2 Visualization of Shuffle

  • Shuffle keeps the first and last actions fixed while reordering middle steps into an incoherent plan, yet such trajectories yield performance comparable to originals as in-context memory.The example repeats a focus action and moves an object without refocusing; this supports the claim that performance benefits from trajectory surface form rather than causal or semantic content.Source: §D.2 · p. 20, Figure 9 · p. 22

D.1 visualization of Crossover

  • Crossover mixes randomly selected lines from two valid parent solutions into structurally broken child code that does not compile, yet performs comparably to real ICRL trajectories as in-context memory.Between one and three lines are selected at random, with a 50% chance of determining which parent each line inherits from; most resulting code has syntax errors.Source: §D.1 · p. 19, Figure 8 · p. 21

NeurIPS Paper Checklist

Related papers

Loading 2610.11152v1…