Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning

1 INTRODUCTION

• Critic-free RL methods such as GRPO avoid learned value networks while supporting scalable and stable LLM reinforcement learning.The introduction presents critic-free algorithms as background for TAMPO's inner-loop policy optimization.Source: §1 · p. 1
• Existing LLM RL methods treat temperature as fixed or manually tuned, despite its role in balancing diverse sampling against focused generation.Higher temperatures encourage diversity but can add noise; lower temperatures concentrate probability but risk missing promising alternatives.Source: §1 · p. 1
• The paper argues temperature should be a decision variable because it directly and transparently controls the text-sampling distribution.It contrasts temperature with entropy regularization coefficients and KL penalties as exploration influences.Source: §1 · p. 1
• TAMPO jointly optimizes the LLM policy with critic-free RL and a meta-policy that reinforces temperatures associated with high-advantage trajectories.The outer update reuses inner-loop rollouts, while the inner policy is optimized at a temperature selected by the meta-policy.Source: §1 · p. 2
• The approach reframes temperature selection as reward-driven meta-policy optimization for adaptive exploration in LLM RL.The paper positions this as a feedback-driven way to balance exploration and exploitation without manual temperature sweeps.Source: §1 · p. 2
• TAMPO outperforms fixed- or heuristic-temperature baselines on five mathematical reasoning benchmarks.The introduction reports the overall benchmark comparison without supplying numeric gains.Source: §1 · p. 2

2 RELATED WORK

• Critic-free LLM RL methods avoid learned value networks but still mainly use fixed sampling temperatures.The passage notes the risks of under-exploration at low temperatures and noisy samples or wasted computation at high temperatures.Source: §2 · p. 3
• Prior exploration methods include fixed or heuristic schedules, nucleus sampling, and entropy regularization, while optimal entropy during training remains unclear.Examples include temperature annealing, ϵ-greedy, and UCB; nucleus sampling typically uses a fixed p.Source: §2 · p. 3
• Adaptive inference-time temperature selection uses multiple sampled generations, whereas cited LLM RL approaches fix or manually tune temperature without trajectory feedback.TAMPO instead adapts temperature online using trajectory advantages and likelihood.Source: §2 · p. 3
• Conventional RL meta-policies and meta-gradient methods address high-level skills or differentiable hyperparameters, unlike TAMPO's non-differentiable sampling temperature.The paper says TAMPO reuses existing rollouts and does not require gradient-based optimization.Source: §2 · p. 3

3.1 BACKGROUND

• High temperatures encourage diverse but potentially noisy generations, while low temperatures favor precision but risk missing promising alternatives.The paper notes that temperature is typically fixed, limiting policy optimization.Source: §3.1 · p. 4

3.2 PROBLEM FORMULATION

• The bilevel formulation samples temperatures from the meta-policy for inner policy optimization, then optimizes that meta-policy for LLM performance.The evaluation metric may be expected reward.Source: §3.2 · p. 4

3.3 TAMPO: MECHANISM AND IMPLEMENTATION

• Decoupled rollout generation prevents gradients through sampling, while testing every candidate temperature with new rollouts is computationally prohibitive.TAMPO is designed to work with the existing decoupled pipeline without extra rollouts.Source: §3.3 · p. 5

3.3.1 TEMPERATURE-DEPENDENT TRAJECTORY LIKELIHOOD

• Figure 2 illustrates unimodal trajectory likelihood across temperatures; positive- and negative-advantage trajectories cluster around distinct likelihood-optimal temperatures.This supports selecting temperatures that preferentially increase the likelihood of high-advantage rollouts.Source: Figure 2 · p. 5, §3.3.1 · p. 5

3.3.2 META-POLICY OPTIMIZATION WITH TEMPERATURE REWARDS

• TAMPO weights each trajectory’s likelihood across candidate temperatures by its advantage, rewarding likelihood for positive-advantage trajectories and penalizing it for negative-advantage ones.An exponential moving average of temperature-specific advantages reduces batch variance while retaining responsiveness to new feedback.Source: §3.3.2 · pp. 5–6

3.4 OVERALL ALGORITHM

• The outer loop reuses inner-loop trajectories to update the temperature meta-policy, and TAMPO is compatible with critic-free RL at negligible additional cost.The inner loop updates the LLM policy using trajectories sampled at the meta-policy-selected temperature.Source: §3.4 · p. 6

4.1 SETUP OF MAIN EXPERIMENTS

• Training uses open-s1 and DS-Qwen-1.5B with GRPO; evaluation covers five math benchmarks using Pass@1 and Pass@8 at decoding temperature 1.0.During the first 10% of training, the LLM uses temperature 1.0 while the meta-policy updates; afterward, it selects temperatures dynamically.Source: §4.1 · p. 7, §4.2 · p. 7

4.2 MAIN RESULTS

• 1.9% and 1.7% improvements in Pass@1 and Pass@8, respectively, over the best fixed-temperature baseline; TAMPO is best or second-best on every dataset.Table 1 compares 1.5B models on five math benchmarks.Source: Table 1 · p. 8, §4.2 · p. 8
• Training time is approximately the same for TAMPO and GRPO baselines: 9 hours 54 minutes for 200 steps on an 8×V100 machine.The lightweight meta-policy is discarded at inference, and TAMPO reuses inner-loop rollouts rather than generating extra trajectories.Source: §4.2 · p. 8

4.3 ABLATION STUDY

• Increasing EMA α from 0.01 to 0.05 raises the average score from 41.6 to 44.5, while α=0.10 has mixed effects.The authors describe α=0.05 as balancing variance reduction with responsiveness to new feedback.Source: §4.3 · p. 8, Table 2 · p. 8
• For meta-policy temperature sampling, p=0.7 is the default balance; p=0.5 or greedy p=0 leads to under-exploration.The meta-policy samples from its distribution over candidate temperatures using nucleus sampling.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 ANALYSIS

• After 20 warm-up steps, the mean sampled temperature is around 1.3, then gradually decreases while retaining diversity.Figure 3 reports the mean and standard deviation over a sliding window of 25 training steps; variance arises from nucleus sampling.Source: §4.4 · p. 9, Figure 3 · p. 9

4.5 MORE EXPERIMENTS RESULTS

• On Qwen2.5-3B-Instruct evaluated with commonsense ECQA, TAMPO shows consistent improvements, extending the experiments beyond mathematical reasoning.Table 4 reports Pass@1 and Pass@8.Source: Table 4 · p. 9, §4.5 · p. 9

• TAMPO treats temperature as a learnable meta-policy in LLM reinforcement learning.Source: §5 · p. 10
• TAMPO reuses trajectories to update the temperature meta-policy without extra rollouts.Source: §5 · p. 10
• The paper reports consistent improvements on mathematical reasoning benchmarks and concludes that principled temperature adaptation is a practical and effective tool for advancing LLM reinforcement learning.Source: §5 · p. 10

• The temperature-adaptation method and algorithm are described in § 3.3.Source: REPRODUCIBILITY STATEMENT · p. 10
• Training details, including base models, hyperparameters, and optimization settings, are described in § 4.1.Source: REPRODUCIBILITY STATEMENT · p. 10
• The training datasets and evaluation benchmarks are publicly available, and the authors state that they will release source code upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

• The GRPO objective maximizes expected advantage while including KL regularization relative to a reference policy.Source: A GRPO OPTIMIZATION OBJECTIVE · p. 12
• The probability ratio ρ compares the current temperature-conditioned policy probability with that of the old policy for an output token.Source: A GRPO OPTIMIZATION OBJECTIVE · p. 12
• ϵ and β are hyperparameters in the GRPO objective.Source: A GRPO OPTIMIZATION OBJECTIVE · p. 12

B PROOF OF THE UNIMODAL OF TRAJECTORY LIKELIHOOD W.R.T. T

• Strict concavity in inverse temperature β implies unimodality in temperature T, with corresponding maximizers β*=1/T*.The trajectory is a sequence of tokens generated from states defined by the prompt and preceding tokens.Source: B PROOF OF THE UNIMODAL OF TRAJECTORY… · p. 12
• For trajectories outside edge cases, log-likelihood has a unique maximizing temperature and typically rises then falls as temperature increases.The proof combines strict concavity in inverse temperature with derivative signs at the low- and high-temperature limits.Source: B PROOF OF THE UNIMODAL OF TRAJECTORY… · pp. 12–13
• Greedy-token trajectories have likelihood that decreases monotonically with T, while trajectories with many rare tokens can gain likelihood as T increases.These are the stated edge cases to the typical rise-then-fall pattern.Source: B PROOF OF THE UNIMODAL OF TRAJECTORY… · p. 13
• The typical unimodal pattern and edge-case monotonicity support estimating trajectory likelihoods with discrete virtual temperatures.Source: B PROOF OF THE UNIMODAL OF TRAJECTORY… · p. 13

C CONNECTION BETWEEN TEMPERATURE AND ADVANTAGE

• The analysis used GRPO at fixed sampling temperatures 0.9, 1.2, and 1.5, computing each trajectory’s likelihood-optimal temperature.Means and variances were computed for rollouts every five training steps.Source: C CONNECTION BETWEEN TEMPERATURE AND… · p. 13
• Positive-advantage and negative-advantage trajectories cluster around distinct likelihood-optimal temperatures, while fixed sampling temperatures generally do not align with those optima.At T=1.5, most optimal temperatures are lower; at T=0.9, most are higher.Source: C CONNECTION BETWEEN TEMPERATURE AND… · p. 13
• At extreme sampling temperatures, trajectories tend toward moderate likelihood-optimal temperatures, consistent with reduced likelihood of advantageous rollouts at excessive randomness or determinism.Source: C CONNECTION BETWEEN TEMPERATURE AND… · p. 13

D SYSTEM PROMPT

• Figure 5 shows the system prompt used during training.Source: D SYSTEM PROMPT · p. 13, Figure 5 · p. 14
• Figure 4 presents likelihood-optimal temperature distributions under three fixed training temperatures.Source: Figure 4 · p. 14
• In Figure 4, green and red curves represent positive- and negative-advantage trajectories, respectively, and blue marks the sampled fixed temperature.Source: Figure 4 · p. 14