PaperlayerView exact revision on arXiv

Source-linked AI summary

Backpropagating Verifier Rewards Through Continuous Flow Language Models

Xu Han, Jiajing Hu, Li-Ping Liu

arXiv:2610.19080v1cs.LG

TL;DR

RL post-training for continuous flow language models remains understudied, despite the models’ continuous denoising trajectories. CFRL uses critics to estimate verifier outcomes at intermediate states and propagate reward through those trajectories. The paper reports reliable reasoning improvements across four tasks and identifies limits in model, task, reward, sampler, and task-comparison scope.

  • Problem

    RL post-training has improved autoregressive and discrete-diffusion language models, but remains understudied for continuous language models that generate token embeddings.

    Source: §1 · p. 1

  • Method

    CFRL trains a critic to predict verifier outcomes from intermediate denoising states and token representations, enabling reward feedback through the continuous trajectory.

    Source: §1 · pp. 1–2

  • Results

    CFRL reliably improves continuous-flow language-model reasoning through pathwise reinforcement learning, and its critic architecture consistently improves RL performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate that pathwise reinforcement learning can improve continuous language-model reasoning, with the proposed critic architecture adaptable to other denoising-based generative models.

    Source: §1 · p. 2

    Evaluation covers one 105M flow model and four synthetic tasks with exact verifiers; gains are sampler-dependent, and the autoregressive reference leads on two tasks.

    Source: §6 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose \method{}, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, \method{} reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.

1 INTRODUCTION

Continuous flow language models can generate quickly through parallel denoising, but RL post-training for these models remains understudied. CFRL uses a critic to pass verifier feedback through the differentiable denoising trajectory, and the paper reports reliable reasoning improvements across four program-verifiable tasks.

1 INTRODUCTION

  • Diffusion-based language models jointly refine token positions over fixed denoising steps, potentially reducing generation latency relative to autoregressive models’ sequential token evaluations.Continuous flow models denoise real-valued token embeddings and decode them into discrete tokens at the end.Source: §1 · p. 1
  • CFRL introduces a critic that predicts verifier outcomes from intermediate denoising states, allowing reward estimates to backpropagate through the continuous trajectory despite non-differentiable decoding.Its token-wise design predicts values for each token representation and trains them against the sequence-level verifier outcome.Source: §1 · pp. 1–2
  • Across four program-verifiable reasoning tasks, CFRL is reported to perform comparably to or better than baselines of similar or larger size.The tasks are Countdown, Sudoku, Knights-and-Knaves, and Game24.Source: §1 · p. 2
  • The paper reports the first reliable improvement in CLM reasoning through pathwise reinforcement learning, to the authors’ knowledge.It also presents a critic architecture that consistently improves RL performance and can be adapted to other denoising-based generative models.Source: §1 · p. 2

2 RELATED WORK

Related work spans discrete and continuous denoising language models, RL for denoising generators, and RL post-training for autoregressive language models. The approaches differ in their generation trajectories and policy-gradient interfaces.

Diffusion and flow language models

  • Continuous-state language models diffuse token embeddings, and ELF uses flow matching to close the generation gap with discrete counterparts while decoding only at the trajectory endpoint.Discrete diffusion language models instead generate through discrete or masked-token trajectories.Source: §2 · p. 2

RL for denoising generative models

  • RL for image and flow-based denoising models commonly treats denoising as a sequential decision process and uses score-function updates, while FPO uses a flow-matching likelihood surrogate within PPO.Discrete-diffusion language-model methods use score-function updates with likelihood approximations, variational objectives, or intermediate denoising actions.Source: §2 · p. 2

RL for autoregressive language models

  • Autoregressive RL methods optimize categorical token actions with score-function methods such as PPO and GRPO.Recent variants modify clipping and sampling.Source: §2 · p. 2

3 PRELIMINARIES: CONTINUOUS FLOW LANGUAGE MODELING AND RL

ELF models conditional text generation by transforming noisy token embeddings into clean embeddings with flow matching, then decoding the final embeddings into tokens. Its stochastic sampler supplies trajectories for the reinforcement-learning formulation.

3 PRELIMINARIES: CONTINUOUS FLOW LANGUAGE MODELING AND RL

  • ELF represents target tokens with embeddings from a frozen T5 encoder and models their conditional distribution given context.The context may include prompt-token embeddings.Source: §3 · p. 2
  • Flow matching interpolates Gaussian noise and clean target embeddings, while an x-prediction denoiser estimates the clean embeddings at each discretized flow time.The denoising update uses a sequence of flow-time steps.Source: §3 · pp. 2–3
  • The denoiser uses self-conditioning by taking its previous clean-embedding estimate as an additional input and is trained with mean squared error against the target embedding.It is implemented as a diffusion Transformer.Source: §3 · p. 3
  • At inference, stochastic SDE sampling injects noise at each generation step, yielding a distribution over trajectories rather than one deterministic path.The sampling procedure initializes from Gaussian noise and produces the clean embedding after K denoising steps.Source: §3 · p. 3
  • A learned decoder maps the final denoised embeddings to discrete tokens, and the flow model plus decoder define the conditional language distribution.A linear output layer projects each embedding to vocabulary logits.Source: §3 · p. 3

4 METHOD

CFRL formulates continuous-flow generation as a finite-horizon decision process with verifier reward at the decoded response. It compares score-function learning with pathwise critics, including a token-wise critic, and trains the policy and critic jointly with an optional self-training loop.

4.1 CFRL-GRR: LEARNING WITH GROUP-RELATIVE REINFORCE

  • CFRL-GRR applies group-relative REINFORCE to sampled response trajectories, using decoded verifier rewards and a scaled decoding term.The authors report that neither ratio clipping nor a KL penalty was necessary in their setting, while leaving other optimization strategies for continuous-state policies to future work.Source: §4.1 · pp. 3–4

4.2 CFRL-SVG-1D: LEARN WITH PATHWISE RL WITH A CRITIC

  • CFRL-SVG-1D uses a critic to estimate expected terminal verifier reward from intermediate states, then backpropagates that estimate through a denoising transition.The critic uses a copied denoiser Transformer with mean-pooled token representations and a value head, trained with binary cross-entropy against the terminal reward; the decoder is frozen so embeddings retain their semantics.Source: §4.2 · p. 4

4.3 CFRL-SVG-2D: TOKEN-WISE REWARD FEEDBACK

  • CFRL-SVG-2D predicts a reward value for each token representation and aggregates the predictions, so each token vector receives the reward signal directly.The design addresses the information bottleneck created when pooled transformer features produce a single value.Source: §4.3 · p. 5

4.4 ARCHITECTURE AND MODEL TRAINING

  • Training first task-fine-tunes the policy and pre-trains a critic on its rollout states paired with decoded verifier outcomes, then alternates policy updates with critic-head updates on fresh rollouts.A policy update differentiates through one middle denoising step; the critic backbone remains frozen during joint training.Source: §4.4 · p. 5
  • An optional self-training loop samples the RL policy on unlabelled problems, retains verifier-accepted answers, and fine-tunes with ELF’s supervised loss.Because this loss trains the whole denoiser, the reported gain becomes a model property rather than remaining tied to the RL sampler.Source: §4.4 · p. 5

5 EXPERIMENTS

Across four program-verifiable reasoning tasks, CFRL-SVG-2D is the strongest flow-model variant, while the experiments examine gradient estimation, critic design, training choices, and amortization across samplers.

Setup

  • The evaluation uses four program-verifiable reasoning tasks, with held-out test prompts for reported accuracy and separate validation prompts for checkpoint selection.The tasks are Countdown, Sudoku, Knights-and-Knaves, and Game24; the flow runs use task-SFT ELF-B (105M) as their base.Source: §5 · p. 6

Main results

  • CFRL-SVG-2D improves Game24 by 14.2 points and reaches 68.7 on Sudoku, making the 105M flow model strongest on both tasks.Every CFRL variant improves the base model across four tasks, and CFRL-SVG-2D is the best flow row on every task.Source: §5 · pp. 6–7

Effect of the gradient estimator

  • The pathwise gradient agrees with itself from 256 rollouts, while the flow model’s score-function gradient does not separate from a zero-mean control through 2,048 rollouts.A single pathwise update at a quarter or half of Λ raises held-out accuracy by about 4 points; score-function and random updates stay within about 1 point of the base.Source: §5 · p. 7

Effect of the critic design

  • Removing x_tk−1 or backbone fine-tuning reduces offline prediction and leads to brief improvement before collapse; random initialization never surpasses the base.Updating the backbone during RL provides no benefit, so joint training refits only the critic head.Source: §5 · pp. 7–8

1D versus 2D critic

  • After critics are exchanged at iteration 250, each Game24 run ends at the level associated with the critic then in charge, regardless of its inherited policy.The 2D critic’s advantage is made during joint training through the gradient it provides to the actor.Source: §5 · p. 8, Figure 3 · p. 8

Effect of the objective, the pooling and the sampler

  • On Game24, changing the objective, pooling, or sampler reduces accuracy relative to the full recipe; sampler mismatch costs the most, while minimum pooling costs less.The comparison reports changes at both the best and final checkpoints; the authors attribute the sampler cost to training states being defined by the sampler that produces them.Source: §5 · p. 8, Table 3 · p. 8

Effect of the amortization loop

  • After amortization, the cheap sampler gains about 20 points on Countdown and 25 on K&K, while deployment accuracy holds on Countdown and reaches 99.1 on K&K.The loop samples verifier-accepted answers from unlabelled problems and fine-tunes with ELF’s supervised loss.Source: §5 · pp. 8–9, Table 4 · p. 9

Effect of the loop across sampler cost

  • After the loop, accuracy varies by only 5 to 6 points across four samplers, and the 16-step sampler delivers nearly deployment accuracy at a quarter of the denoiser calls.The cheap-to-most-expensive sampler gap shrinks from about 20 to about 6 points on Countdown and from about 10 to about 5 on K&K.Source: §5 · p. 9, Table 4 · p. 9

6 CONCLUSION AND LIMITATIONS

CFRL combines score-function and pathwise RL post-training variants for continuous flow language models, with position-specific critic values improving policy optimization. Its evidence remains limited to one model and four synthetic reasoning tasks, and the RL advantage over an autoregressive reference varies by task.

6 CONCLUSION AND LIMITATIONS

  • CFRL offers three estimators and an amortization loop; per-position values raise policy performance faster and further than a scalar value, and the loop carries the gain to a cheap chain.Exchanging critics mid-training shows that the critic in charge determines the policy’s subsequent course.Source: §6 · p. 9
  • Results cover one 105M flow model and four synthetic tasks with exact verifiers; larger models, natural-language tasks, and learned rewards remain untested.The RL gain is tied to the training sampler; transferring it to cheaper samplers requires an amortization loop and unlabelled problems. The autoregressive reference leads on two of four tasks.Source: §6 · p. 9
  • The authors report that code, configurations, job scripts, exact splits, seeds, and verifier implementations will be released, alongside hyperparameters, evaluation details, and supporting results.They also report using generative AI for prose editing and experiment code, not for research ideas or uncomputed results.Source: AI USE STATEMENT · p. 10

A DATASETS AND VERIFIERS

The evaluation uses four program-verifiable reasoning tasks with generated or public puzzle data and task-specific training and held-out sets. Deployment and cheap sampler configurations are specified for each task.

A DATASETS AND VERIFIERS

  • All tasks use programmatic generation and verifiers that assess correctness without the gold answer, producing exact, label-free rewards.Deployment uses γ56@64, except γ28@32 on Sudoku; the cheap sampler is γ4@16, except γ14@16 on Sudoku.Source: A DATASETS AND VERIFIERS · p. 13
  • Countdown uses 2,000 held-out problems; its verifier checks arithmetic, number reuse, and whether the derivation reaches the target.Training has 500k examples, and the loop uses a disjoint pool of 50k unlabelled problems.Source: A DATASETS AND VERIFIERS · p. 13
  • Sudoku uses 2,000 held-out public puzzles; its verifier checks that each output is a valid grid consistent with the givens.The supervised set has 400k puzzles, and the loop’s 50k fresh puzzles do not overlap training, validation, or test data.Source: A DATASETS AND VERIFIERS · p. 13
  • Knights-and-Knaves puzzles have unique solutions, and the verifier checks that each output assignment satisfies every statement.Training uses 300k puzzles and evaluation uses 2,000; the RL base is the epoch-4 checkpoint, while the epoch-8 model is the supervised oracle.Source: A DATASETS AND VERIFIERS · p. 13
  • Game24 evaluation holds out all orderings of 269 number sets, while training uses 1,076 sets and the loop draws additional orderings from those training sets.The verifier checks derivations against 24; the RL base is the epoch-20 supervised checkpoint, where held-out accuracy peaks.Source: A DATASETS AND VERIFIERS · p. 13

B.1 THE STOCHASTIC SAMPLER AND ITS TRANSITION DENSITY

The stochastic sampler defines intermediate states from re-noised embeddings and self-conditioning estimates, enabling transition-level score-function and pathwise gradients. The analysis explains why intermediate critics are needed, how critic heads distribute gradients, and when token decisions commit.

B.1 THE STOCHASTIC SAMPLER AND ITS TRANSITION DENSITY

  • The sampler re-noises the current state, queries the denoiser at the lowered time, then moves toward its estimate; the state carries both re-noised embeddings and the prior estimate.The final step is deterministic, and the transition’s parameter-dependent mean comes through the denoiser estimate.Source: §B.1 · pp. 13–15

B.2 SCORE-FUNCTION AND PATHWISE GRADIENTS OF ONE TRANSITION

  • Flow score-function gradients retain a noise factor across coordinates, while pathwise gradients weight coordinates by the critic’s derivative and omit that factor.The score-function signal depends on reward covariance with each noise coordinate, while noise adds variance even when a coordinate does not affect reward.Source: §B.2 · p. 15

B.3 WHY DIFFERENTIATING ONE TRANSITION SUFFICES

  • Because the final decoded reward is piecewise constant with zero gradient almost everywhere, pathwise optimization needs a value estimate on intermediate states.A learned critic is refit on current-policy rollouts after each policy update; drawing one denoising step at random gives an unbiased estimate of its share of the gradient sum when the critic is exact.Source: §B.3 · p. 16

B.4 HEAD BEFORE OR AFTER THE MEAN

  • A head on pooled position features gives every position the same gradient direction, whereas a head per position uses each position’s own reading.The transformer features attend across the whole state, so the head order changes parametrization and joint-training behavior rather than the critic’s representable state functions.Source: §B.4 · p. 16

B.5 TOKEN COMMITMENT ALONG THE TRAJECTORY

  • On Countdown, 86% of positions commit at the first denoiser call, with the median at the first call and the 90th percentile at the third.This measures decoded-token commitment, not how much each coordinate influences reward.Source: §B.5 · p. 16

C METHOD DETAILS

The method uses a critic to provide differentiable reward estimates for single transitions in the denoising trajectory, then alternates policy and critic updates. It also amortizes policy improvement through verifier-filtered fine-tuning, optionally using the critic to guide sampling.

C.1 THE PATHWISE ESTIMATOR AND THE CRITIC

  • A single uniformly sampled middle-trajectory transition is differentiated while its preceding state and noise are held fixed, keeping backward-pass memory independent of trajectory length.The window excludes early states with flat values and late states with piecewise-constant decoded-response rewards; its endpoints are hyperparameters.Source: §C.1 · p. 17
  • The critic learns success probability from rollout states paired with their verifier outcomes, without requiring intermediate labels.It is pre-trained with a denoiser copy and then uses a frozen copy with a trainable head during RL.Source: §C.1 · p. 17
  • The per-position critic produces nonlinear readings for individual token representations, allowing verifier feedback to yield position-specific gradients.A head on mean-pooled features is restricted to linear weighting across positions.Source: §C.1 · p. 17

C.2 DESIGN CHOICES OF THE JOINT TRAINING

  • Joint training refits the critic head using recent rollout states, but its buffer mixes values from past policies and its check tests separation only at the window’s top step.The AUC check on newest states measures whether the critic separates outcomes under the current policy, not whether its slope matches across the window.Source: §C.2 · pp. 17–18

C.3 THE AMORTIZATION LOOP

  • Each amortization round keeps distinct verifier-accepted responses and fine-tunes on them with the model’s supervised objective, which trains the denoiser across all times.The round teaches directly only on prompts with at least one accepted response.Source: §C.3 · pp. 18–19
  • Optional critic-guided sampling ranks partial responses by their least-confident position, prunes to the strongest candidate, and nudges embeddings along the score gradient.The cheaper sampler’s trajectories differ from the training sampler’s, so using critic readings to rank them is an assumption.Source: §C.2 · p. 18, §C.3 · p. 19

D EXPERIMENTAL DETAILS

The experimental details specify the models, training procedures, and held-out evaluation used for comparisons. The reported autoregressive learning curves vary by task: token-level RL improves on Countdown and Game24, holds on Knights-and-Knaves, and does not move the sparse-reward Sudoku model.

D EXPERIMENTAL DETAILS

  • The flow experiments use an ELF-B 105M-parameter denoiser with frozen T5-small embeddings and task-specific supervised starting checkpoints.Rollouts and evaluation use classifier-free guidance weight 2; the critic copy is evaluated without guidance.Source: D EXPERIMENTAL DETAILS · p. 20
  • The critic reuses a copied denoiser as a feature extractor and compares mean-pooled 1D and per-position 2D heads, with pretraining selected by held-out field score.The copy is frozen during RL, while the head is trained; Countdown pretraining uses cheap-sampler states because deployment-sampler early and middle states carry little outcome information.Source: D EXPERIMENTAL DETAILS · p. 20
  • Pathwise variants use deployment-sampler rollouts with a replay buffer and critic refitting, while CFRL-GRR uses grouped rollouts under a 16-step sampler without a critic.CFRL-GRR uses 16 prompts and eight samples per iteration; pathwise training uses 32 actor and 32 buffer rollouts per iteration.Source: D EXPERIMENTAL DETAILS · pp. 20–21
  • The amortization loop samples task-specific pools, retains verifier-accepted responses, and fine-tunes from the prior round’s checkpoint; some controls instead train on gold labels.The loop runs four rounds on Countdown and five on Knights-and-Knaves, with an additional Sudoku round reported separately.Source: D EXPERIMENTAL DETAILS · p. 21
  • Held-out accuracy is measured on test prompts after checkpoints are selected by validation accuracy at the deployment sampler.The held-out prompts are split by index into validation and test halves.Source: D EXPERIMENTAL DETAILS · p. 21

E.1.1 AUTOREGRESSIVE REFERENCE: LEARNING CURVES

  • Token-level GRPO/PPO improves GPT-2 on Countdown and Game24, holds on Knights-and-Knaves, and does not move the sparse-reward Sudoku model.The comparison uses the same problems and verifier as the flow policies, with at least as many rollouts per iteration; the action space differs.Source: §E.1.1 · p. 22, Table 6 · p. 22

E.1.2 CFRL-GRR UNDER ITS OWN SAMPLER AND UNDER THE DEPLOYMENT SAMPLER

CFRL-GRR gains little under either sampler. On Countdown, runs read under their own sampler gain about one point over the base, while deployment-sampler training gains +0.1 on Countdown and +3.5 on Game24.

E.1.2 CFRL-GRR UNDER ITS OWN SAMPLER AND UNDER THE DEPLOYMENT SAMPLER

  • +0.1 on Countdown and +3.5 on Game24 are the gains when CFRL-GRR is trained under the deployment sampler; under its own sampler, Countdown gains about one point over the base.The paper reports that the estimator gains little under either sampler.Source: §E.1.2 · p. 22

E.1.3 THE MASKED-DIFFUSION REFERENCE: D1 ON OUR TASKS

Pathwise gradients are more reproducible and can improve held-out accuracy when applied with AdamW normalization, while critic usefulness depends on trajectory timing and refreshing it as the policy changes. The 2D critic’s advantage during training is not apparent in fixed-policy comparisons.

E.1.3 THE MASKED-DIFFUSION REFERENCE: D1 ON OUR TASKS

  • diffu-GRPO moves the supervised model by under three points on every task, consistent with d1’s results on easier three-number Countdown and 4×4 Sudoku.The comparison uses different architectures and training histories, so it does not establish the ceiling of masked-diffusion models.Source: §E.1.3 · pp. 22–23

E.2.1 GRPO WITH RATIO, CLIPPING AND ANCHOR

  • The 2D critic stays ahead on every task: a second step and clipping add a few points on Game24 and K&K, but the usual β=0.04 KL anchor returns performance to about the CFRL-GRR level.The ordering in Table 1 remains unchanged across these variants.Source: §E.2.1 · p. 23, Table 7 · p. 23

E.2.2 THE TWO ESTIMATORS ON THE SAME ROLLOUTS: SNR AND HALF-SPLIT COSINE

  • At 256 rollouts, pathwise gradients agree with themselves and have SNR nearly three times the floor, while the flow score-function estimator remains indistinguishable from zero through 2,048 rollouts.GPT-2’s token-level gradient separates from the zero-signal control as rollout count grows.Source: §E.2.2 · p. 23, Table 8 · p. 24

E.2.3 ONE STEP ALONG EACH GRADIENT

  • AdamW pathwise steps gain about four points on Game24 at 0.25–0.5Λ and 3–4 on Countdown from 0.5Λ onward; score-function and random steps stay within about one point.The raw pathwise step destroys the policy, and on Game24 gains turn over at η=Λ when the critic is not refit.Source: Table 9 · p. 24, §E.2.3 · p. 24

E.3.1 CRITIC QUALITY ALONG THE TRAJECTORY

  • The pooled critic value is near chance at the trajectory start, informative from roughly one-quarter in, and essentially an oracle in the last third.Per-position AUC is reported only for Countdown and Game24 because the K&K and Sudoku verifiers do not provide per-position labels.Source: §E.3.1 · p. 25, Table 10 · p. 25

E.3.2 PER-POSITION CRITIC VALUES AS A CREDIT MAP

  • The per-position critic readings localize some credit but are dominated by sequence-level signal; Sudoku’s within-failed-sample wrong-cell AUC is only 0.56–0.63 at every denoising step.Figure 4 shows the per-position readings for one correct and one incorrect held-out sample per task; Game24 maps organize by equation rather than acting as a step verifier.Source: §E.3.2 · p. 25, Figure 4 · p. 26

E.3.3 1D AGAINST 2D CRITIC OVER TRAINING, ALL TASKS

  • Across training, the 2D critic’s policy climbs further on Game24 and K&K, while policies turn back on Countdown and Sudoku; at a fixed policy, the critics remain indistinguishable.At a fixed policy they discriminate reward equally, prescribe updates at cosine 0.83–0.90, react equally and weakly to one wrong token, and refit with equal stability.Source: §E.3.3 · p. 26, §E.3.4 · p. 26

E.3.5 FROZEN VERSUS JOINT CRITIC

  • On Game24, a frozen critic’s value rises from about 3 logits at iteration 250 to over 80 at 1,500 as batch success falls from 78% to 0%, and held-out accuracy collapses.Refitting the head after every actor step keeps its value calibrated; a stronger frozen critic does not prevent exploitation.Source: §E.3.5 · p. 26

E.3.6 TRAINING DYNAMICS ON COUNTDOWN AND K&K

The 2D-critic runs show different training dynamics on Countdown and K&K: Countdown’s held-out accuracy plateaus while replay success continues rising, whereas K&K’s held-out accuracy rises through the end of training.

E.3.6 TRAINING DYNAMICS ON COUNTDOWN AND K&K

  • On Countdown, held-out accuracy rises for about 750 iterations and then holds while replay-buffer success continues climbing; on K&K, held-out accuracy rises monotonically to its peak at the schedule’s end.K&K’s buffer success rate and critic value rise together throughout the run.Source: §E.3.6 · pp. 26–27, Figure 6 · p. 27
  • Figure 5 tracks held-out accuracy and critic AUC across training, with curves showing both individual runs and their mean.The figure’s AUC is measured against outcomes on recent replay-buffer states, averaged over 250-iteration windows.Source: Figure 5 · p. 27

E.4 THE LOOP

  • Appendix E.4 supplies the loop rounds and controls behind Table 4, including the critic-guided sampler used for later Countdown rounds.Figure 6 plots held-out accuracy, replay-buffer success, and critic value for Countdown and K&K.Source: §E.4 · p. 27, Figure 6 · p. 27

E.4.1 THE LOOP IN FULL: ROUNDS, CONTROLS AND THE GUIDED SAMPLER

The amortization loop uses repeated sampled rounds, with a critic-guided sampler for later Countdown rounds, and controls separate the contributions of the starting policy and the rounds. The outcomes vary by task: guided rounds recover Countdown performance, while rounds alone close K&K and barely shift Sudoku accuracy.

E.4.1 THE LOOP IN FULL: ROUNDS, CONTROLS AND THE GUIDED SAMPLER

  • On Countdown and K&K, rounds lift the cheap sampler by 17–35 points from every start; on K&K, fresh puzzles give near-full coverage and rounds close the task, while RL adds nothing beyond them.On Sudoku, one round covers three quarters of the pool from every start and changes accuracy by at most 1.4 points at either sampler, consistent with the gold-solution control. In Countdown, critic-guided rounds recover 4.5–5.9 points versus plain rounds continued from the round-2 model, attributing the recovery to the guided pools rather than extra rounds.Source: §E.4.1 · p. 28, Table 12 · p. 29

F RELATED WORK

  • CFRL’s SVG variants apply stochastic value gradients to a generative sampler, fitting a success-probability value to verifier rewards and, in CFRL-SVG-2D, predicting a reading per position.The related-work passage frames this as the stochastic-value-gradient pattern with the sampler itself as transition model.Source: F RELATED WORK · p. 29
  • The amortization loop relates to self-training with verified generations, using an RL policy as sampler and, in guided form, the RL critic as search heuristic.The cited controls distinguish the starting policy’s contribution from the rounds’ contribution.Source: F RELATED WORK · p. 29

Related papers

Loading 2610.19080v1…