PaperlayerView exact revision on arXiv

Source-linked AI summary

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT

Joery Ariën de Vries, Neil David Lawrence, Zhenwen Dai

arXiv:2610.03361v1cs.LGcs.AI

TL;DR

Agentic RFT often depends on repeated group rollouts that are difficult to obtain in stateful environments with long, sparsely verified trajectories. FTW adapts cross-entropy optimization to replay-buffer ordinal filtering, and matches GRPO and PPO on Sokoban and published Search-R1 baselines. Its risk offset is bounded and adjustable, with memory overhead and limits under aleatoric noise and non-transitive rankings.

  • Problem

    Repeated group rollouts are impractical in some stateful agent environments, while long, sparsely verified trajectories make outcomes difficult to reproduce and high in variance.

    Source: §1 · pp. 1–2

  • Method

    FTW adapts the cross-entropy method to critic-free RFT by filtering replay-buffer trajectory minibatches according to ordinal comparisons and fitting the policy to selected samples.

    Source: §1 · p. 2, §3 · p. 5

  • Results

    FTW matches GRPO and PPO on Sokoban and published Search-R1 baselines, and trains more stably with lower seed variance than REINFORCE++.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FTW offers a viable trade-off that uses CPU memory instead of a value model or the environment duplication required for group rollouts.

    Source: §8 · p. 10

    FTW is less suited to environments with large irreducible noise, uses additional replay-buffer memory, and its elite-filtering interpretation assumes a total ordering.

    Source: §8 · p. 10

Abstract

from arXiv · show

Critic-free reinforcement fine-tuning (RFT) for agentic large language models is often done through GRPO-style methods, which compute a group baseline over repeated rollouts to reduce target variance. However, this setup is ill-suited to agents acting in stateful environments such as live services or security sandboxes, where repeated rollouts are impractical to obtain and aggressive updates entrench the noise of long, sparsely verified trajectories. We propose \textit{Follow the Winners} (FTW), a critic-free policy-learning algorithm that adapts the cross-entropy method to RFT, replacing group rollouts with an ordinal filter on replay-buffer samples that yields polynomial concentration in the order statistic of returns. We derive FTW through a control-as-inference lens, which also recovers GRPO and DPO as specific modelling choices, identifying GRPO as risk-neutral while DPO and FTW share a bounded risk-seeking offset that FTW controls. We identify this offset as an inherent trade-off of variance reduction through ordinal filters on samples, whereas a critic model induces a different trade-off between bias and variance. Scaled to agentic LLM post-training, FTW matches GRPO and PPO on Sokoban and Search-R1 baselines, showing a viable trade-off from a value model or group rollouts to CPU memory.

1 Introduction

Agentic RFT faces high-variance trajectories and costly or impractical repeated rollouts in stateful settings. FTW replaces group rollouts with replay-buffer ordinal filtering, and matches GRPO and PPO on Sokoban and published Search-R1 baselines.

1 Introduction

  • Long tool-call trajectories have high return variance from sparse verification, partial observability, and state aliasing, while repeated group rollouts require identical initial states.Critics can reduce variance but require GPU memory and may introduce bias through bootstrapping.Source: §1 · pp. 1–2
  • FTW adapts the cross-entropy method to replay-buffer samples, using ordinal filtering to concentrate polynomially on higher-return order statistics.The replay proposal amortizes data generation and supplies an alternative source of variance reduction.Source: §1 · p. 2
  • The Control-as-Inference framework recovers DPO as a pairwise Plackett–Luce case and GRPO as an upper bound for a multinomial potential, placing the methods in a shared design space.FTW and DPO share a bounded risk-seeking offset, while GRPO is risk-neutral; FTW controls its offset through selection pressure.Source: §1 · p. 2
  • FTW matches GRPO and PPO on Sokoban and the published Search-R1 baselines, while training more stably and with lower seed variance than REINFORCE++.The method uses no value model, group rollouts, or environment duplication.Source: §1 · p. 2

2 Background

The background formulates agentic LLMs as policies interacting with partially observable environments, then presents Control-as-Inference and cross-entropy optimization as foundations for policy improvement.

Large Language Models

  • LLMs define autoregressive distributions over token sequences, and their pretrained parameters are fitted by maximizing sequence likelihood on example data.Training practice may add positional encodings, layer normalization, gradient clipping, and tokenization.Source: §2 · p. 2

Reinforcement Learning for Agentic LLMs

  • Agentic LLM reinforcement learning is formulated as a partially observable Markov decision process, with textual histories conditioning the policy’s next action.Observations are context tokens masked from the policy’s output likelihood.Source: §2 · pp. 2–3

Control-as-Inference

  • Control-as-Inference tilts a prior trajectory distribution toward higher returns, yielding an optimality-conditioned posterior policy and a KL-regularized policy-improvement objective.The objective balances expected optimality likelihood against divergence from the base policy.Source: §2 · p. 3

The Cross-Entropy Method

  • The cross-entropy method samples a batch, scores its members, retains the top-quantile elite set, and refits the proposal distribution to those elites.Its indicator potential retains samples above the estimated quantile threshold.Source: §2 · p. 3

3 Follow the Winners: Model Derivation & Elite Filtering

FTW recasts policy improvement as inference over trajectory batches with ordinal feedback, then adapts cross-entropy filtering to replay-buffer samples. Its minibatch filter uses selection pressure to balance concentration and reuse of past data.

3 Follow the Winners: Model Derivation & Elite Filtering

  • The derivation conditions a batch of trajectories on ordinal preferences rather than conditioning individual trajectories on binary optimality, producing an evidence lower bound for filtering and projection.The potential tilts the proposal toward preferred items while the KL term constrains updates to the reference policy.Source: §3 · pp. 3–4
  • The construction assumes sparse final rewards and an IIA-consistent ranking, which ensures minibatch rankings agree with an underlying total ordering of conditional returns.The total-order requirement constrains use in non-transitive settings.Source: §3 · p. 4
  • The batch-level ELBO motivates alternating between filtering better trajectories and projecting the tilted distribution onto the policy class, the structure underlying cross-entropy optimization.The proposal and ordinal potential are chosen to support conservative updates, sample-only variance reduction, and reuse of past trajectories.Source: §3 · pp. 4–5
  • FTW samples independent minibatches from a replay buffer and keeps each minibatch’s top K of C trajectories, then uses their union for a cross-entropy gradient update.The buffer approximates a mixture of recent policies; repeated local wins increase a trajectory’s weight in later proposals.Source: §3 · p. 5
  • The induced rank-selection profile is polynomial: increasing C at fixed K sharpens concentration on high-ranked trajectories, while increasing K softens selection pressure.C=K gives uniform sampling, and C=N with one minibatch recovers standard CEM but discards the buffer’s non-elites each update.Source: Figure 1 · p. 4, §3 · p. 5

4 Theoretical Analysis

The analysis places GRPO and DPO within a shared inference framework and compares their risk profiles with FTW’s. It characterizes GRPO as risk-neutral and FTW’s bounded risk offset as controllable through its elite ratio.

4.1 Recovering GRPO and DPO

  • GRPO and DPO arise from different batch potentials: the multinomial potential yields GRPO’s group-mean advantage through a Jensen upper bound, while the pairwise Plackett–Luce case recovers DPO.Their remaining distinction in this framework is the baseline normalizer: uniform group-mean versus rank-dependent baselines.Source: §4.1 · pp. 5–6

4.2 Implicit Risk Profiles of RFT Algorithms

  • GRPO is risk-neutral, whereas DPO and FTW are boundedly risk-seeking toward a selection threshold; FTW’s elite ratio K/C determines the threshold’s position relative to the buffer mean.Mild filters place the attractor at or below the buffer mean, while aggressive filters place it above.Source: §4.2 · p. 7
  • Ordinal filtering’s bounded offset depends on how often noise swaps preference orderings rather than on return magnitudes, but under aleatoric noise it can bias the fixed point away from the mean-optimal policy.Under epistemic uncertainty, the offset can instead act as an exploration term.Source: §4.2 · p. 7
  • Rank-based potentials depend only on within-batch order statistics, which remain bounded for continuous distributions, unlike cardinal-return methods that can be destabilized by heavy tails.The multinomial exponential potential requires a finite return moment-generating function.Source: §4.2 · p. 7

5 Validation on Bandit & Deep Reinforcement Learning

Diagnostic bandits test FTW’s risk and stability properties; HalfCheetah tests whether its ordinal updates extend to deep reinforcement learning. Results align with the predicted stability trade-offs and show comparable end-budget returns to SAC.

Bandit settings

  • The three diagnostics isolate variance-sensitive arm selection, stability under truncated Pareto returns, and policy stochasticity under state aliasing.The aliasing task uses deterministic rewards and requires policy stochasticity for low regret.Source: §5 · p. 7
  • In the σ-trap, GRPO reaches the optimal arm only when its learning rate decreases as σ^-1/2; DPO is distracted earlier than FTW, but both eventually approach a uniform policy.FTW shares DPO’s risk offset at (C, K) = (2, 1), while FTW-exp collapses onto the distractor fastest.Source: §5 · pp. 7–8
  • Under heavy-tailed returns, GRPO destabilizes while FTW and DPO remain stable; on state aliasing, FTW has lower action-distribution L1 error than GRPO and DPO.These are the center and right diagnostics in Figure 2.Source: Figure 2 · p. 8, §5 · p. 8

Deep RL

  • By the end of the 10^9-step budget, FTW-K4-C64 reaches mean return comparable to SAC, with visibly tighter seed-to-seed variation, despite lower early sample efficiency.Figure 3 shows the HalfCheetah learning curves.Source: §5 · p. 8

6 LLM Post-Training Experiments

FTW is evaluated on multi-turn Sokoban and retrieval-based Search-R1. It matches established baselines in the reported comparisons, while Sokoban results expose sensitivity to the elite-filter setting.

Experimental setup

  • The evaluation uses sparse terminal rewards and a −0.1 formatting-error penalty; GRPO uses eight rollouts per prompt and is the only method requiring duplicated group rollouts.Sokoban uses a 2,000-item replay buffer; Search-R1 uses 10,000 items.Source: §6 · pp. 8–9

Benchmark results

  • FTW with K ∈ {1, 2} clusters with GRPO and PPO in Sokoban learning speed and performance, and matches published Search-R1 PPO and GRPO results within noise.Figure 4 shows Sokoban success-rate curves and Search-R1 final exact-match accuracy.Source: Figure 4 · p. 9, §6 · p. 9

Sokoban results

  • For K = 3, Sokoban performance collapses after peaking around 0.4; minibatches with too few successes fill elite slots with zero-return trajectories that the update imitates.The authors describe this as a finite-sample effect that larger batches and replay buffers can dampen at the cost of slower learning.Source: §6 · p. 9
  • On Sokoban, REINFORCE++ shows large seed variance and slower learning than the competitive FTW settings.Source: §6 · p. 9

7 Related Work

The related-work discussion connects FTW to control-as-inference, critic-free RFT, preference optimization, and policy-improvement methods.

Control as Inference

  • The framework extends control-as-inference by conditioning on permutations or winner indices over sampled batches rather than binary optimality variables for individual trajectories.Source: §7 · p. 9

Related methods

  • FTW shares a bootstrap-and-project structure with RAFT and related methods, while the framework places GRPO- and DPO-family methods on a common design axis.RAFT corresponds to FTW with C = N, K = 1, and a buffer refreshed every prompt.Source: §7 · p. 9
  • The derivation reaches the N = 2 Bradley–Terry limit top-down through the first-stage marginal of the Plackett–Luce model under control-as-inference.Source: §7 · p. 10

8 Conclusion

The conclusion presents FTW as a replay-buffer-based, critic-free alternative for agentic LLM post-training and frames the paper’s methods within broader control, preference, and policy-improvement work.

8 Conclusion

  • FTW adapts the cross-entropy method to agentic LLM post-training, using polynomial concentration and replay to match GRPO and PPO on Sokoban and Search-R1 while training more stably and faster than REINFORCE++ on Sokoban.The proposed trade-off uses CPU memory in place of a value model or environment duplication for group rollouts.Source: §8 · p. 10
  • FTW and DPO’s bounded risk-seeking offset can bias the fixed point away from the mean-optimal policy under aleatoric noise; the elite-filtering interpretation also assumes a total ordering.The authors identify large irreducible noise and non-transitive objectives such as Nash-equilibrium optimization as boundaries needing caution or adaptation.Source: §8 · p. 10

A Disclosure Statements

  • The disclosure describes LLM use for writing assistance, literature search, and experiment infrastructure; it also notes compute-related environmental costs and that code, models, and data are not yet released.The non-release is conditioned on manuscript acceptance and completion of internal legal procedures.Source: A Disclosure Statements · p. 19

B Extended Related Work

  • The related-work survey spans control-as-inference, RLHF and preference optimization, critic-free policy improvement, and cross-entropy methods.Source: B Extended Related Work · p. 19

B.1 Control as Inference.

  • Control-as-inference connects sequential decision-making with probabilistic inference; this paper extends the formulation to partial observability and ordinal likelihoods over sampled trajectory sets.Its proposed tilting mechanism is polynomial and order-statistic-based rather than exponential and return-based.Source: §B.1 · pp. 19–20

B.2 Reinforcement Learning from Human Feedback (RLHF).

  • The framework seeks to unify critic-free GRPO-family and DPO-family methods as consequences of modelling choices, rather than treating their common heuristics as fundamental differences.Source: §B.2 · p. 20

B.3 Learning to Rank and Listwise Preference Optimization.

  • The paper derives the Plackett–Luce likelihood top-down from control-as-inference, yielding an ELBO and policy gradient rather than imposing a ranking loss directly.Its comparison distinguishes this approach from classical learning-to-rank and direct listwise preference objectives.Source: §B.3 · pp. 20–21

B.4 Policy Improvement Operators for Critic-Free RL.

  • In critic-free RFT, noisy batch statistics can amplify into entrenched policy updates; FTW instead uses a polynomial potential over within-batch return order statistics.This differs from Tsallis regularization, which produces polynomial tails through the regularizer rather than the potential.Source: §B.4 · pp. 21–22

B.5 Cross-Entropy Method for Deep RL.

FTW uses CEM elite-filtering on replay-buffer trajectories as a policy-improvement step, rather than filtering policy parameters or candidate action sequences. The section situates this approach alongside GRPO and DPO and details the selection properties and assumptions behind the filtering procedure.

B.5 Cross-Entropy Method for Deep RL.

  • CEM’s other deep-RL uses filter policy parameters or candidate action sequences, rather than replay-buffer trajectories used for policy improvement here.The section identifies black-box policy search and short-horizon planning over learned dynamics as the two prior lines.Source: §B.5 · p. 22
  • FTW applies CEM elite-filtering to replay-buffer trajectories to optimize a Control-as-Inference ELBO, then projects filtered trajectories back onto policy parameters through SGD.Unlike prior CEM-RL approaches, it uses neither a learned dynamics model nor a population of policies.Source: §B.5 · p. 22

C GRPO and DPO Primer

  • GRPO samples a group of trajectories per prompt and uses standardized returns as advantages, reinforcing above-mean trajectories in a risk-neutral update.Its PPO-style objective includes a KL penalty toward a reference policy; the primer notes sensitivity to return scale and the high- versus low-variance group effects of standard-deviation normalization.Source: C GRPO and DPO Primer · pp. 22–23
  • DPO learns from pairwise preferences under a Bradley–Terry model, and its loss increases the preferred trajectory’s likelihood relative to the dispreferred one.In the experiments, preferences are induced by comparing sampled returns.Source: C GRPO and DPO Primer · p. 23

D.1 Rao-Blackwell Argument and GRPO upper-bound

  • The Rao–Blackwell argument identifies GRPO’s group-averaged baseline, while the conditional-batch coefficient αn(BN) is an unbiased Monte Carlo estimator.The passage also states that a Jensen bound follows by upper-bounding the coefficient αn.Source: §D.1 · p. 23

D.2 Probability of Selection under Elite-Filtering

  • For a buffer ranked by return, elite-filtering samples C trajectories without replacement and retains the top K, inducing a normalized occupancy for each rank.When K = C, the occupancy is uniform across ranks, recovering uniform sampling.Source: §D.2 · pp. 23–24
  • Lemma 1 assumes distinct buffer trajectories, uniform sampling without replacement within each draw, and independent repeated draws; its marginal probabilities can change when these conditions vary.The passage says the lemma illustrates the induced tilting distribution and leaves a full treatment of variants out of scope.Source: §D.2 · p. 24

D.3 Risk Offsets across Return Distributions

This section examines how return distributions and utility choices shape risk offsets, distinguishing risk-neutral GRPO from risk-seeking operators. It also qualifies FTW’s analysis by discussing distributional limitations and the CAI basis for ordinal feedback, including consistency of full-ranking and winner-only feedback across sampled batches.

D.3 Risk Offsets across Return Distributions

  • Ordinal grade utilities are invariant to strictly increasing return transformations, and sparse Bernoulli returns are ranked by their mean even when the certainty equivalent is not uniquely expressible.The sparse Bernoulli grade-based cells use 0† for mean-consistent rankings, not a computed zero offset.Source: §D.3 · pp. 25–26
  • MNL-exp uses return magnitudes rather than ranks: its risk offset is σ²/(2τ) at small noise across the considered families, while its sparse Bernoulli offset additionally depends on e^(m/τ).Its certainty equivalent is a scaled log moment-generating function; the exponential-family expression is defined only when θτ > 1.Source: §D.3 · p. 26
  • GRPO is exactly risk-neutral, grade-based DPO and FTW are boundedly risk-seeking, and MNL-exp is unboundedly risk-seeking and fragile.This ordering is summarized in Table 1; the offsets vanish when return noise vanishes, and GRPO’s affine utility gives an exact zero offset.Source: §D.3 · pp. 25, 26, 28, Table 1 · p. 26
  • FTW’s risk-seeking pull saturates at its elite threshold: mild filters place it at or below the buffer mean, half filters at the mean, and aggressive filters above it.The pull grows with an entry’s own return standard deviation but cannot score it above the threshold.Source: §D.3 · p. 28
  • With a noisy Gaussian distractor, ordinal comparisons approach coin flips as variance grows; aggressive FTW filters can instead raise the threshold above the optimal mean and converge to the trap arm.This failure can occur at moderate noise as the buffer spread increases, before comparisons reach the coin-flip regime.Source: §D.3 · p. 28
  • The FTW softmax-on-ranks Arrow–Pratt expansion applies in the small-noise regime σ ≪ σ0, which the section notes is not typically realistic.For counts, the corresponding normal-limit approximation also requires a large buffer rate λ0.Source: §D.3 · p. 28
  • FTW’s closed-form top-K/C cells are exact only for pairwise Gaussian returns, and its grade operators can fail on sparse or stepwise rewards.The threshold T is approximated by a normal although its exact law is skewed; on discrete rewards the operators lack a unique certainty equivalent and are jackpot-blind, unlike MNL-exp. The single-family return assumption also idealizes potentially heterogeneous mixture buffers.Source: §D.3 · p. 29

E.1 Preliminaries

  • CAI treats trajectory optimality as a formal soft indicator and motivates FTW’s cross-entropy fit by projecting samples reweighted by exp G onto the policy class.The optimality values are not directly used; the policy is optimized by maximizing the marginal likelihood of optimality.Source: §E.1 · pp. 29–30

E.2 Batch-Level CAI and Ordinal Potentials

  • Under Luce’s choice axiom, PL subset rankings align with the full-domain objective, while MNL preserves a globally best trajectory as winner in any batch containing it.These batch-consistency guarantees justify minibatch filtering: the elite set is unbiased in expectation.Source: §E.2 · p. 31

E.3 The Plackett–Luce Model

  • PL assigns positive weights to trajectories and satisfies batch-consistent pairwise preferences regardless of the weight function; an already optimal policy yields a uniform ranking.The weight function determines the likelihood’s inductive bias.Source: §E.3 · p. 32

E.4 The Multinomial (Winner) Model

  • The winner-only MNL model is exactly the first-stage marginal of PL and inherits its global consistency guarantees.Because batch indices are exchangeable, the derivation averages over all possible winner identities.Source: §E.4 · p. 32

E.5 Policy Gradients and Recovery of GRPO and DPO

This section derives GRPO and DPO as modeling choices within a KL-regularized, Boltzmann-weighted ELBO framework and extends batch control-as-inference to Gaussian-belief mirror descent for contextual bandits. It also describes batch-CAI policy updates and experimental resource requirements, including FTW’s use of replay-buffer samples to trade CPU memory for reduced rollout duplication without a separate critic.

E.5 Policy Gradients and Recovery of GRPO and DPO

  • Under the full Plackett–Luce model, middle-ranked trajectories receive the largest advantages, while top and bottom ranks have diluted gradient signals.The bottom trajectory’s advantage is zero; the top trajectory’s is nonzero but small.Source: §E.5 · p. 33
  • Under the MNL model, a shared baseline makes every advantage non-positive, so learning repels trajectories while the KL term anchors the policy.Unlike the full Plackett–Luce gradient, relative advantage magnitudes depend on return spread rather than rank position.Source: §E.5 · p. 33
  • A Jensen bound on the log-sum-exp baseline recovers GRPO’s group-mean advantage, with the slack corresponding to a risk bonus and the N-scaled KL factor absorbed into β.The derivation treats trajectories as independent draws, unlike GRPO’s shared-prompt rollout assumption; the ln N offset does not change optimization critical points.Source: §E.5 · p. 33
  • At N = 2, the pairwise Bradley–Terry likelihood under DPO’s reparameterization recovers DPO’s loss.The factor of 2 is absorbed into the KL coefficient, with the reference-policy identification arising from the EM coordinate-ascent argument.Source: §E.5 · pp. 33–34
  • GRPO and DPO both optimize a KL-regularized ELBO under a Boltzmann likelihood, differing in batch size and baseline choice.The framework presents these recoveries as substantive results without claiming a complete reduction of all preference-optimization methods.Source: §E.5 · p. 34

F Belief-Policy Mirror Descent under Batch CAI

  • For contextual bandits, the extension uses a Gaussian belief over linear-softmax policies to perform policy-space mirror descent and closed-form policy iteration from past data.The belief represents uncertainty about action hypotheses, while the history-conditioned lower bound includes potential and policy-KL terms.Source: F Belief-Policy Mirror Descent under… · p. 34
  • With Gaussian beliefs, the KL penalty becomes a Bregman divergence, yielding a mirror-descent update; the unrestricted belief update is proportional to the previous belief times the partition function.The belief optimization is framed as variational Bayesian filtering, and the implementation evaluates the gradient at the previous belief.Source: F Belief-Policy Mirror Descent under… · pp. 35–36
  • FTW applies filtered cross-entropy to replay-buffer samples, while DPO uses a closed-form Bradley–Terry loss and GRPO’s mixed-sign centered advantages require holding precision fixed.FTW’s Gauss–Newton step is its only approximation; DPO’s nonlinear loss has an indefinite exact Hessian, and GRPO’s centered-advantage Hessian is indefinite even under Gauss–Newton.Source: F Belief-Policy Mirror Descent under… · p. 37
  • The batch-CAI mean-only update is plain GRPO; controls replace its gradient step with a natural-gradient preconditioner or accumulate Fisher information in FTW’s belief-learning rule.GRPO-NPG uses the current batch’s per-arm softmax Fisher with Λ0; GRPO-ONG heuristically inserts the GRPO loss into FTW’s belief-learning rule.Source: F Belief-Policy Mirror Descent under… · p. 37

G.1 Hardware

  • Experiments used GCP A100 and H100 instances for Bandits, Brax, and Search-R1, and a Lambda 2× H100 instance for Sokoban.Reproduction was estimated at roughly 6,000 GPU-hours and $25,000–$35,000 USD at on-demand rates, excluding setup, development, and debugging.Source: §G.1 · p. 38

G.2 Compute and Memory Footprint

  • FTW stores 2,000 Sokoban and 10,000 Search-R1 trajectories in CPU memory; PPO used 4 GPUs, versus 2 for GRPO and REINFORCE++ and 1–2 for FTW.All four methods used the same number of rollouts per step in Sokoban, but GRPO required n rollouts from the same prompt and identical initial states. PPO additionally trained a critic; FTW performed a cross-entropy loss on filtered samples plus a KL penalty. REINFORCE++ was cheapest on every axis and lagged in Figure 4; adding a replay buffer to GRPO would match FTW’s memory footprint, leaving group duplication as the remaining difference.Source: §G.2 · p. 38

G.3 Software Stack

  • The four environments required separate software stacks because of package conflicts, hardware variation, and model choices; Table 2 lists key versions and Search-R1 requires a separate wiki-retrieval server.All cloud runs used Ubuntu 24.04.4 LTS, Python 3.12.3, driver 570.211.01, and NCCL_NET=Socket, which slowed runs but was required by GCP-related issues.Source: §G.3 · pp. 38–39, Table 2 · p. 39

G.4 Main Environments

The evaluation spans seven environments, from diagnostic bandits and Brax continuous control to Search-R1 and Sokoban. The appendices describe shared and method-specific configurations and supplementary analyses.

G.4 Main Environments

  • Seven environments cover four diagnostic bandits, HalfCheetah, Search-R1, and Sokoban.The bandits isolate features of learning; the other tasks cover continuous control, multi-turn text QA, and a vision-language grid puzzle.Source: §G.4 · p. 39
  • The four bandits test noise sensitivity, heavy-tailed returns, state aliasing, and contextual regret.The σ-trap sweeps distractor variance; the Pareto-α task varies tail heaviness; the corridor aliases states; the contextual task uses deterministic rewards.Source: §G.4 · p. 39
  • HalfCheetah uses continuous observations and bounded vector actions, with evaluation based on episode return.The setup uses a 1,000-step episode and standard Brax forward-velocity and control-cost rewards.Source: §G.4 · p. 40
  • Search-R1 is multi-turn open-domain QA with retrieval, while Sokoban is a vision-language box-pushing task.Search-R1 ends with a final answer or continues with a search query; Sokoban supplies rendered puzzle images and accepts directional actions.Source: §G.4 · p. 40

G.5 Hyperparameters

  • Brax methods share the optimizer, replay buffer, observation normalization, and network architecture; actor objective and update-to-data ratio vary by algorithm.The LLM-agent configurations share settings across tested algorithms, with FTW varying minibatch size C and elite count K.Source: §G.5 · p. 41
  • The hyperparameter tables document shared Brax training settings and shared LLM-agent training-target counts across FTW configurations.For FTW, J = 128/K, keeping J · K = 128 targets per update.Source: Table 3 · p. 41, Table 4 · p. 42

H Supplementary Results

  • Supplementary results cover LLM-agent learning curves, FTW selection variants, return-distribution sweeps, learning-rate robustness, and elite-filtering bias under ties.The tie-dominated setting concerns saturated returns and complements the reported robustness results.Source: H Supplementary Results · p. 43

H.1 Training-time Success Rates for Sokoban and Search-R1

Training-time results compare FTW with GRPO, PPO, and REINFORCE++ on Sokoban and Search-R1, and examine how FTW’s selection operator changes behavior in a noisy bandit trap.

H.1 Training-time Success Rates for Sokoban and Search-R1

  • On Sokoban, FTW-K1-C5 and FTW-K2-C5 cluster with GRPO and PPO, while REINFORCE++ trains slower with greater seed variance; FTW-K3-C5 peaks near 0.4 then collapses.With binary returns, groups with fewer than K successes fill remaining elite slots with zeros, a more frequent issue at K=3 than K=1.Source: §H.1 · p. 43, Figure 5 · p. 43
  • On Search-R1, FTW-K1-C4 dips during initial training before improving steadily; the training split covers only NQ and HotpotQA, unlike the main-text test distribution.Source: §H.1 · p. 43, Figure 5 · p. 43

H.2 Varying FTW Selection Operators for the noisy Bandit-trap

  • In the noisy bandit trap, mild FTW-polynomial filtering (C=4, K=3) keeps risk minimal at larger noise, whereas aggressive filtering (C=4, K=1) collapses onto the trap arm.For exponential-ordinal filtering, the greedy limit collapses even faster; the intermediate temperature 1.0 finds the optimal arm under moderate or low noise.Source: §H.2 · p. 44

H.3 Bandit Distribution Sweep

The bandit sweep tests FTW and DPO against GRPO across return distributions, while a learning-rate sweep tests FTW’s state-aliasing result. Rank-based methods remain stable in heavy-tailed regimes, and FTW’s lower L1 distance persists across learning rates.

H.3 Bandit Distribution Sweep

  • With bounded-support Beta returns, all three methods solve the bandit without destabilization across shapes, and FTW reduces scale-regret slightly faster early on.Source: §H.3 · p. 44
  • At low success probability p, GRPO converges slower than rank-based methods; scaling Binomial rewards by 1000 widens its gap, while FTW and DPO retain their unscaled curves.The scaled sweep isolates reward magnitude from sparsity; the passage attributes GRPO’s destabilization to magnitude rather than sparsity alone.Source: §H.3 · p. 44
  • Across five return distributions, rank-based potentials never destabilize, while GRPO destabilizes where the moment-generating function diverges.Under Pareto-α ≤1 and Student-t ν ≤2, GRPO fails to identify the optimal arm or its scale-regret drifts, while FTW and DPO remain stable.Source: §H.3 · pp. 44, 46

H.4 State-Aliasing under Learning-Rate Variation

  • Across the learning-rate sweep, FTW has lower average L1 distance than DPO and GRPO in the state-aliasing task.The comparison averages across learning rates at horizon H=16, rather than relying on a single step-size choice.Source: §H.4 · p. 46, Figure 8 · p. 46

H.5 FTW under Tie-Saturated Rewards

This case study examines FTW when Bernoulli rewards saturate near one, making sampled arms hard to distinguish. Finite replay buffers allow policy improvement to compound, while an infinite buffer can preserve early history.

H.5 FTW under Tie-Saturated Rewards

  • When both arms succeed with probability near one, their paired returns are usually tied, so selection favors the better arm by at most a factor of 1+δ.Under saturation, elite filtering is therefore nearly indistinguishable from the buffer occupancy.Source: §H.5 · pp. 46–47
  • Under saturation, FTW imitates the buffer with an O(δ) nudge toward the better arm, so improvement can proceed only as fast as the buffer moves.Finite sample age lets old data expire and the nudge compound; with h=∞, each new batch becomes a vanishing fraction of the buffer.Source: §H.5 · p. 47
  • For Bernoulli returns, FTW with a never-delete buffer stays near chance at small δ, while finite buffers avoid that anchor but introduce more tie-breaking and update noise when smaller.At δ=0.099, the never-delete buffer produces only a marginal shift toward the optimal arm; GRPO’s tied advantages vanish, making its learning more stable.Source: §H.5 · p. 47
  • With Gaussian returns, every finite FTW buffer finds the optimal arm, whereas GRPO’s learning signal appears stale for small δ.The experiment sweeps δ over {0.099, 0.009, 0.004, 0.001, 10^-4} and compares against on-policy GRPO.Source: §H.5 · pp. 46–47

H.6 Epistemic Contextual Bandit

The epistemic contextual-bandit case study compares FTW and DPO with GRPO variants using regret and belief diagnostics. FTW and DPO are statistically indistinguishable at their best settings, while FTW’s buffer-depth performance is insensitive across three orders of magnitude.

H.6 Epistemic Contextual Bandit

  • At their best parameter settings, FTW and DPO are statistically indistinguishable, while too little or too much selection pressure yields higher regret than intermediate pressure.Their final simple regret spans roughly fourfold across concentration settings.Source: §H.6 · p. 48
  • At similar regret, FTW’s posterior width decreases more sharply and predictive entropy ends lower than DPO’s.GRPO’s belief metrics are stale or overcontract under the model mismatch that prevents a proper update rule in this framework.Source: §H.6 · p. 49
  • FTW is insensitive to buffer depth across three orders of magnitude, whereas DPO degrades sharply at h=1024 and GRPO remains at its higher regret level.Source: §H.6 · p. 49, Figure 12 · p. 50
  • Plain GRPO converges prematurely at every fixed step size, while GRPO-NPG removes most of this gap in simple and cumulative regret; GRPO-ONG achieves the worst tested regret aside from uniform.The case study uses deterministic bounded rewards with equal marginal expected reward across arms, so uncertainty is epistemic.Source: §H.6 · pp. 48–49
Loading 2610.03361v1…