PaperlayerView exact revision on arXiv

Source-linked AI summary

TIPS: A Tiny Inference-time Policy for Scaling in Diffusion Language Models

Ya-Shi Zhang, Jian Tang, Pierre-André Noël, Oleksiy Ostapenko

arXiv:2610.22099v1cs.LGcs.AI

TL;DR

Masked diffusion decoding must decide which predictions become visible context and whether earlier commitments should be reconsidered. TIPS learns those visibility choices with a small controller around a frozen dLLM, and improves accuracy across GSM8K and MATH-500 while remaining competitive at higher budgets without retraining.

  • Problem

    Masked diffusion decoding must choose which predicted positions become visible, while many existing methods use hand-designed rules or require visibility to grow monotonically.

    Source: §1 · p. 1

  • Method

    TIPS is a small auxiliary policy that learns unmasking and remasking around a frozen dLLM, with a fixed schedule regulating expected progress.

    Source: §1 · p. 2

  • Results

    TIPS improves low- and moderate-budget accuracy on GSM8K and MATH-500 and remains competitive at higher budgets without retraining.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results suggest visibility control can improve the accuracy–compute trade-off while leaving the decoder backbone unchanged.

    Source: §6 · p. 9

    Evaluation centers on 256-token mathematical reasoning, transfer experiments use a single seed, and decoder-call budgets do not directly measure wall-clock cost.

    Source: §6 · p. 10

Abstract

from arXiv · show

Masked diffusion language models expose an inference-time choice: which positions become visible after each denoising pass. Many existing decoding methods use hand-designed rules or require visibility to grow monotonically. We present a $\textbf{T}$iny $\textbf{I}$nference-time $\textbf{P}$olicy for $\textbf{S}$caling ($\textbf{TIPS}$), an approximately $600$K-parameter controller that learns both unmasking and remasking while keeping the underlying language model frozen. TIPS uses visibility-aware confidence features and a shallow transformer to select which tokens should be visible, while a fixed schedule controls the expected visible fraction. We train the controller through supervised warmup followed by reinforcement learning from terminal correctness rewards. On GSM8K and MATH-500 with LLaDA-8B-Instruct, the full-canvas and two-block TIPS variants achieve $58.46\%$ and $57.97\%$ micro-averaged accuracy at a budget of $32$ decoder evaluations, exceeding the strongest baseline in our main comparison by $12.1$ and $11.6$ percentage points, respectively. Both variants are trained at a budget of $32$ evaluations and remain competitive at higher budgets without retraining. These results demonstrate that learning when to reveal and reconsider tokens can improve the accuracy-compute trade-off without updating the decoder backbone.

1 Introduction

TIPS learns which positions to reveal and whether to revise earlier commitments, addressing the visibility choices that shape generation with a frozen dLLM. Its controller combines visibility actions with a fixed expected-progress schedule and is trained with supervised warmup followed by correctness-based reinforcement learning.

1 Introduction

  • Existing approaches typically use hand-designed selection rules, preserve monotonic visibility, or learn remasking by updating the language model, motivating a small policy that learns both actions without modifying the dLLM.The visibility trajectory determines which predictions become context, when they appear, and whether they can be revised.Source: §1 · p. 1
  • A fixed power-law schedule controls the expected visible fraction, while supervised visibility-decision warmup precedes reinforcement learning on final-answer correctness.Only the auxiliary controller is updated; training requires neither gradients through the dLLM nor its sequence likelihood.Source: §1 · p. 2
  • TIPS is an approximately 600K-parameter policy that learns unmasking and remasking while keeping the dLLM backbone frozen.It assigns next-step visibility actions using confidence features, mask indicators, and normalized decoding time.Source: §1 · pp. 1–2
  • At most decoder NFE budgets, TIPS surpasses previous state-of-the-art methods in accuracy on GSM8K and MATH-500 by up to 13.25 percentage points, otherwise remaining competitive.The introduction reports full-canvas and blockwise decoding on LLaDA-8B-Instruct.Source: §1 · p. 2
  • Ablations and supplementary experiments examine commitment and revision, with analyses suggesting remasking can help by temporarily withdrawing harmful context.The paper also uses transfer experiments to characterize TIPS's scope and limits.Source: §1 · p. 2

2 Background

Finite-step decoding separates predicting candidate tokens from choosing which positions become visible as context. The section frames those visibility choices as actions in a finite-horizon Markov decision process.

2.1 Finite-step decoding

  • A dLLM predicts candidate tokens in parallel at masked positions, while a decoding algorithm chooses which positions will be visible in the next state.A visible token may be kept or remasked, and a masked position may be revealed or left masked.Source: §2.1 · p. 2
  • Visibility trajectories can change final responses because visible tokens condition later passes, and monotonic decoding prevents already visible tokens from being modified.Remasking and editing remove that constraint, allowing earlier predictions to return to the masked state or transition to another token.Source: §2.1 · p. 2

2.2 The decoding path as a Markov decision process

  • A decoding path is a finite-horizon Markov decision process whose state includes the prompt, partial response, and denoising time, with actions specifying next-step visibility.A dLLM evaluation supplies candidate distributions; transitions apply the selected visibility actions.Source: §2.2 · p. 3

3 Method

TIPS uses a lightweight policy to make reversible, position-wise visibility decisions over a frozen denoiser. A fixed schedule controls expected visibility, while training combines reference-derived supervision with policy optimization.

3 Method

  • TIPS extends position-wise decisions to next-step visibility, allowing masked tokens to be revealed or deferred and visible tokens to be retained or remasked.The policy uses a two-layer RoPE-DiT network to produce per-position visibility logits.Source: §3 · pp. 3–4
  • The policy’s two-dimensional features encode active-token log-odds and its support relative to the strongest alternative.The active token is the decoder’s greedy proposal at masked positions and the stored token at visible positions.Source: §3 · pp. 3–4
  • A fixed power-law schedule sets the expected visible fraction, while a shared logit shift preserves the policy’s preference ordering.TIPS samples position-wise Bernoulli visibility actions, so realized visibility remains stochastic around the scheduled expectation.Source: §3 · p. 4
  • Policy optimization samples groups of visibility trajectories and uses verifier-based terminal rewards with group-relative policy optimization.The likelihood is factorized across conditionally independent Bernoulli actions.Source: §3 · pp. 4–5

Supervised warmup.

  • Optional supervised warmup uses reference responses to create dense visibility targets and trains policy probabilities with binary cross-entropy.At visible positions, targets favor retaining reference-matching tokens and remasking mismatches; at masked positions, utility reflects decoder support for the reference token against its strongest competitor.Source: Supervised warmup · p. 5
  • For two-block decoding, TIPS operates within each block before a deterministic final pass unmasks remaining masked tokens.The total evaluation budget is divided across blocks; the paper evaluates full-canvas and two-block variants.Source: Supervised warmup · p. 5

4 Experiments

Experiments evaluate TIPS on mathematical reasoning across decoder-call budgets, then test transfer across backbones, block sizes, and sampler handoffs. Ablations examine design choices on a validation set.

4.1 Setup

  • Experiments use frozen LLaDA-8B-Instruct with 256-token responses, train on GSM8K and MATH, and report micro-averaged test accuracy over 1,819 problems.TIPS training uses a fixed budget of 32 calls across three seeds, with supervised warmup followed by 935 RLVR updates.Source: §4.1 · p. 5

Results.

  • TIPSfull achieves the highest accuracy among methods at 8–32 call caps, while TIPSblock is often highest at 64–256 calls and remains competitive at 256 calls.At 256 calls, TIPSblock reaches 68.18%, versus 67.73% for UNCODE and 67.01% for Fast-dLLM.Source: Results · p. 6, Figure 1 · p. 7

4.3 Supplementary Experiments

  • Supplementary experiments probe transfer across decoder backbones and decoding geometries, and continuation from states produced by another sampler.These experiments use one seed and, when applicable, budgets T ∈ {16, 32, 64}.Source: §4.3 · p. 7

Zero-shot transfer to Dream-7B.

Zero-shot transfer across block sizes.

4.4 Ablations

  • Ablations use a held-out validation set of 512 GSM8K and 512 MATH examples, so their accuracies are not directly comparable to test results.They cover warmup, remasking, visibility-aware features, schedule parameter γ, architecture, runtime, and training budgets.Source: §4.4 · p. 8

5 Related Work

Related work addresses reveal order, learned visibility control, remasking, and policy optimization for diffusion language models. TIPS combines learned visibility decisions with reversible actions while keeping the decoder fixed.

Training-free decoding order and adaptive parallelism

  • Training-free samplers vary reveal order or adapt parallelism using confidence, entropy, uncertainty calibration, temporal stability, or token-dependency signals.Examples include Fast-dLLM, EB-Sampler, UNCODE/PC-Sampler, KLASS, and DAPD.Source: §5 · pp. 8–9

Learned unmasking schedules

  • Learned unmasking controllers optimize position selection around a pretrained denoiser, but Jazbec et al. (2026) and UPO preserve monotonic visibility, unlike TIPS’s reversible decisions.Other approaches include supervised ranking, policy-weighted losses, and dense trajectory rewards.Source: §5 · p. 9

Re-masking and revision

  • Remasking methods allow visible tokens to be withdrawn, but most prescribe revision rules; RemeDi modifies the language model, whereas TIPS learns a separate visibility controller.TIPS, like RemeDi, uses supervised warmup followed by reinforcement learning.Source: §5 · p. 9

Reinforcement learning for diffusion language models

  • Unlike methods that apply policy gradients to the diffusion language model, TIPS updates only its auxiliary Bernoulli visibility policy and requires neither sequence likelihood nor decoder-parameter updates.Other dLLM policy-optimization approaches use intermediate rewards, mask-aware gradients, or sequence-likelihood surrogates.Source: §5 · p. 9

6 Discussion

TIPS learns reversible visibility control around a frozen diffusion language model, while a fixed schedule regulates expected progress. The reported results cover mathematical reasoning at 256 tokens, with transfer and stopping behavior still bounded by stated scope limitations.

6 Discussion

  • TIPS separates visibility control from token prediction and uses a fixed schedule to regulate expected progress.The discussion reports improved low- and moderate-budget accuracy, competitiveness at higher budgets without retraining, and complementary strengths of full-canvas and blockwise variants.Source: §6 · p. 9
  • Evaluation centers on 256-token mathematical reasoning, transfer experiments use one seed, and the fixed horizon forgoes early-termination savings; decoder-call budgets also do not directly measure wall-clock cost.Longer generations and broader task coverage remain important tests of generality.Source: §6 · p. 10

A Expanded methodology

The visibility schedule is fitted from an uncalibrated pilot policy’s validation trajectories, then fixed for calibrated training and evaluation. Fast-dLLM thresholds are adjusted to match average decoder-call budgets where feasible.

A.1 Fitting the visibility schedule

  • A least-squares fit projects the pilot’s mean expected next-state visibility onto an endpoint-anchored monotone schedule, without using validation answers or correctness labels.The fit excludes deterministic finalization; blockwise decoding fits one shared γ from visibility fractions averaged across validation trajectories and blocks.Source: §A.1 · p. 12

A.2 Fast-dLLM budget matching

  • Fast-dLLM thresholds are tuned on test rollouts to place mean NFE within [T, T + 0.5], allowing up to 0.5 extra calls on average.Threshold adjustment covers budgets 8–128; at 256 calls, non-adjusted results are reused, and the unavailable 4-call result is imputed from the 8-call result.Source: §A.2 · pp. 12–13
  • The main Fast-dLLM results use the mean-adjusted 8 × 32 setting, while Dream transfer uses 1 × 256 decoding.Unless otherwise specified, “Fast-dLLM” denotes the mean-adjusted version; the non-adjusted baseline uses the standard 0.9 confidence threshold.Source: §A.2 · p. 13

B.1 Main results with all baselines

Across 256-token configurations, TIPSfull leads at budgets 8–32, while TIPSblock leads at 128 and 256; the ablations examine training stages, reversible visibility, confidence features, occupancy, and schedule calibration.

B.1 Main results with all baselines

  • Across 256-token configurations, TIPSfull has the highest mean accuracy at budgets 8–32, while TIPSblock leads at 128 and 256.Fast-dLLM leads at 64 calls with 65.59% versus 64.49% for TIPSblock; the 128-token UPO configuration is reported separately from the matched comparisons.Source: §B.1 · p. 13, Table 4 · p. 14

B.2 Ablations

  • Ablation accuracies use a 1,024-example validation mixture, so their absolute values should not be compared directly with the main test results.The mixture contains 512 GSM8K and 512 MATH problems, unlike the 1,319:500 test mixture.Source: §B.2 · p. 13

B.2.1 Supervised warmup and reinforcement learning

  • Warmup followed by RLVR improves over warmup alone by 14.55, 17.87, and 14.45 points at T = 16, 32, and 64, and over RLVR alone by 0.49, 8.50, and 7.81 points.The RLVR continuations use the same 935-update length; warmup alone receives no RLVR updates.Source: §B.2.1 · p. 13, Table 5 · p. 15

B.2.2 The contribution of remasking

  • Disabling remasking reduces accuracy by 7.03–12.99 points, while retrained monotone policies remain 0.78, 8.40, and 6.93 points below reversible TIPS at T = 16, 32, and 64.The figure plots validation accuracy against decoder-call budget on a logarithmic axis, with one inference seed per trained policy.Source: §B.2.2 · p. 15, Figure 3 · p. 15

B.2.3 Visibility-consistent confidence features

  • The global argmax differed from the current visible token in 73,930 of 3,407,617 visible-position events (2.17%), motivating confidence features anchored to the active token.At visible positions, the policy decides whether to retain the current token, not whether to insert the decoder’s newly preferred token.Source: §B.2.3 · p. 15

B.2.4 Confidence-feature dimensionality study

  • Adding the strongest alternative probability makes K = 2 outperform K = 1 at every budget above one call, including gains of 3.56, 2.14, and 2.92 points at T = 16, 32, and 64.Expanding to ten or fifty probabilities does not improve over K = 2.Source: §B.2.4 · pp. 15–16, Figure 4 · p. 16, Table 6 · p. 16

B.2.5 Occupancy control and finalization deferral

  • At T = 32, fixed calibration leaves 6.89% of positions masked before finalization, versus 57.11% without occupancy control and 48.01% with the soft loss; accuracy rises from 45.90% to 50.83%.The post-transition jump to full visibility is imposed by deterministic completion, and the non-constraining soft loss is insufficient to alleviate finalization deferral.Source: §B.2.5 · p. 16, Figure 5 · p. 17, Table 7 · p. 17

B.2.6 Schedule shape and logit calibration

  • Changing schedule shape has budget-dependent effects: γ = 0.2 raises accuracy from 43.16% to 45.70% at T = 16 but lowers it from 56.15% to 53.22% at T = 32.Removing occupancy calibration instead lowers accuracy to 26.66%, 25.20%, and 25.59% at T = 16, 32, and 64.Source: §B.2.6 · p. 18, Figure 6 · p. 18, Table 8 · p. 18

B.2.7 Policy architecture and runtime

The two-layer DiT controller outperforms simpler alternatives at most reported budgets, while the MLP retains substantial accuracy with lower controller time.

B.2.7 Policy architecture and runtime

  • At T = 64, the per-token MLP reaches 54.88% accuracy versus 58.20% for the two-layer DiT; the two-layer DiT beats the MLP at all three budgets and the one-layer DiT at T = 32 and 64.The per-token MLP cannot exchange information between response positions, yet retains substantial policy effectiveness.Source: §B.2.7 · p. 18
  • At T = 32, cumulative policy time per generated example is 8.06 ms for the two-layer DiT, 6.88 ms for the one-layer DiT, and 6.22 ms for the MLP.The timings sum controller work across the rollout and exclude decoder and feature-extraction costs.Source: §B.2.7 · p. 19, Table 9 · p. 19

B.2.8 Training with a longer decoding budget

The supplementary analyses examine training-budget and controller-depth comparisons, visibility selection and revision behavior, and transfer beyond the training configuration. TIPS shows useful position selection and recovery behavior in focused tests, transfers to Dream-7B, and improves heuristic continuations, while results vary across settings.

B.2.8 Training with a longer decoding budget

  • The Ttrain = 32 policy exceeds Ttrain = 128 by 4.00, 6.84, and 4.98 percentage points at T = 16, 32, and 64, respectively.The comparison is not fully matched, and the longer-training configuration uses γ of approximately 0.8.Source: §B.2.8 · p. 19, Table 10 · p. 19, Figure 7 · p. 19

B.2.9 Depth-matched comparison with the learned unmasking policy

  • At T = 32, two-layer full-canvas TIPS reaches 56.15% validation accuracy versus 40.33% for the matched-depth unmasking policy, and matching controller depth does not remove TIPS’s advantage.The comparison uses one training/evaluation seed pair; the unmasking policy may terminate before its decoder-call cap.Source: §B.2.9 · p. 20, Table 11 · p. 20, Figure 8 · p. 21

B.3.1 Does TIPS select useful reveal and remask positions?

  • At T = 16, randomizing both reveal and remask positions reduces accuracy at all six tested sites, including 44.42% to 40.52% at the first-block 0.25 phase.The count-preserving intervention is a focused TIPSblock case study; separate reveal-only or remask-only interventions slightly improve accuracy at one site.Source: §B.3.1 · p. 22, Table 13 · p. 22, Figure 10 · p. 23

B.3.2 Recovery from controlled token corruption

  • TIPS restores substituted tokens without reference access, but corrupted conditions have lower answer accuracy than matched clean continuations and increase disagreement at untouched positions.TIPSblock restores 76.53% of first-block substitutions at k = 1 and 66.55% at k = 8; excess untouched-token disagreement is positive in all twelve conditions and generally grows with k.Source: §B.3.2 · p. 23, Table 15 · p. 24, Figure 11 · p. 24

B.3.3 Remasking is more frequent than token editing

  • Remasking is common but usually does not change token identity: 91.52–95.17% of full-canvas episodes and 92.91–98.16% of blockwise episodes return the original token.The observed episodes all resolve by decoding’s end; the proposed benefit of temporarily hiding unreliable context is a hypothesis, not an isolated mechanism.Source: §B.3.3 · p. 26, Table 18 · p. 26, Table 19 · p. 27

B.3.4 Continuing heuristic trajectories with TIPS

  • Sixteen of eighteen heuristic-to-TIPS hybrid conditions improve over the matched heuristic, but none exceeds pure TIPS at the same budget.At T = 64, full-canvas handoff after q = 1/4 raises accuracy from 23.91% to 40.85%; starting from an all-mask canvas can outperform retaining the heuristic state.Source: §B.3.4 · p. 27, Table 20 · p. 27, Figure 13 · p. 28

B.3.5 Repairing completed non-adjusted Fast-dLLM responses

  • Four additional TIPSfull calls raise completed Fast-dLLM responses from 67.01% to 67.29% micro-accuracy, correcting five initially wrong answers while preserving all 1,219 initially correct answers.This is a single-seed result on the GSM8K and MATH-500 test sets.Source: §B.3.5 · p. 29, Table 22 · p. 29, Figure 14 · p. 29

B.4.1 Zero-shot transfer to Dream-7B; B.4.2 Zero-shot transfer from mathematics to coding

  • Without retraining, TIPSfull beats matched Fast-dLLM on Dream-7B at T = 16, 32, and 64, while math-trained policies transfer to coding nontrivially but not consistently outperform baselines.On Dream-7B, scores are 35.18%, 46.56%, and 52.39% versus 22.87%, 27.43%, and 29.96%; on coding, math-trained TIPSfull leads at HumanEval T = 16 but trails the 8 × 32 Fast-dLLM baseline at T = 64.Source: §B.4.1 · p. 30, Table 23 · p. 30, Figure 15 · p. 30

B.4.3 Training TIPS for coding

Coding-specific training improves the full-canvas policy, although the blockwise policy responds less consistently. A frozen full-canvas policy also transfers to smaller decoding blocks, with stronger results when each block retains more compute, and small proposal noise is competitive with greedy proposals.

B.4.3 Training TIPS for coding

  • Coding training raises full-canvas HumanEval pass@1 to 19.51%, 25.61%, and 27.44%, and MBPP pass@1 to 24.00%, 31.80%, and 37.80%, but Fast-dLLM remains stronger at some larger budgets.The coding-trained full policy exceeds every retained baseline on HumanEval at T = 16 and MBPP at T = 16 and 32; the block policy improves at T = 16 on both tasks but falls below its zero-shot counterpart at T = 64.Source: Table 25 · p. 33, §B.4.3 · p. 33, Figure 17 · p. 34

B.4.4 Zero-shot transfer across block sizes

  • With the full-canvas-trained policy, T = 64 accuracy is 62.67% on the full canvas, 62.89% with two blocks, 59.81% with four, and 55.25% with eight.At T = 16, accuracy falls from 47.00% on the full canvas to 22.43% with eight blocks, which receive only two calls per block.Source: §B.4.4 · p. 35, Table 27 · p. 35, Figure 18 · p. 36

B.4.5 Sensitivity to decoder proposal temperature

  • Proposal temperature 0.1 reaches 48.32% at T = 16 and 63.33% at T = 64, slightly above greedy proposals, while temperature 1.0 reduces accuracy at all three budgets.Greedy proposals perform better at T = 32; finalization remains greedy in every condition.Source: §B.4.5 · p. 36, Table 28 · p. 36, Figure 19 · p. 37

B.5 Training diagnostics

This section presents TIPS training diagnostics and decoding trajectories, covering supervised-warmup traces, RLVR reward trajectories, and the full-canvas and two-block policies’ shared supervised initialization. It cautions that visibility changes and accepted final answers do not establish valid intermediate reasoning or show that edits caused success.

B.5 Training diagnostics

  • Mathematics supervised warmup has 3,230 initial updates and 4,074 final-phase updates after the state dataset is refreshed, with recorded loss traces shown for both phases.Figure 20 plots training soft binary cross-entropy; the two phases use separate optimizer-step axes.Source: Figure 20 · p. 37, §B.5 · p. 37
  • Post-hoc mathematics validation observations use a fixed bank of 36,363 states from 1,173 reserved training examples, distinct from the 1,024-example validation ablations, while coding traces show warmup training and validation losses.These figures report supervised warmup of the small policy network.Source: §B.5 · p. 37, Figure 21 · p. 38, Figure 22 · p. 38
  • Mathematics mean terminal-correctness reward rises from approximately 38% to 50% for both geometries; coding rises from 16.28% to 37.55% for the full policy and 13.66% to 27.41% for the block policy.Rewards are raw means over the first and last 50 updates; each nominal update contains 128 outcomes, and each run ends at update 935.Source: §B.5 · p. 38, Table 29 · p. 39, Figure 23 · p. 39

B.6 Illustrative decoding trajectories

  • At T = 32, the selected decoding examples include verifier-accepted final answers and an unsuccessful full-canvas trajectory, but acceptance does not certify intermediate reasoning or show that an edit caused success.Displayed canvases annotate reveals, remasks, and changed re-reveals; snapshots label consumed decoder calls, and finalized earlier blocks remain frozen.Source: §B.6 · p. 39
  • Repeated visibility changes did not prevent an incorrect GSM8K calculation, illustrating that remasking is an opportunity to revise, not a guarantee of correctness.The trajectory ends with a generated answer of $40,000.Source: Figure 28 · p. 44

C Experimental and implementation details

  • The implementation details distinguish TIPSfull from TIPSblock, identify their shared supervised initialization, and defer evaluation populations and compute accounting to appendix D.The section specifies architecture, numerical implementation, and training procedures for the mathematical-reasoning checkpoints used in the main comparison.Source: C Experimental and implementation… · p. 45

C.1 Frozen decoder and policy architecture

The implementation keeps LLaDA-8B-Instruct frozen while a compact controller selects reversible visibility actions. Its training combines reference-derived supervised warmup with terminal-correctness reinforcement learning.

C.1 Frozen decoder and policy architecture

  • A two-layer RoPE-DiT controller produces one visibility logit per response position, while the LLaDA-8B-Instruct decoder remains frozen.The controller has 628,739 trainable parameters; the two TIPSblock response blocks share weights.Source: §C.1 · p. 45

C.2 Confidence features and visibility updates

  • At each step, positions are updated synchronously: masked positions can reveal their proposal, while visible positions can retain or remask their current token.Block-local execution allocates H −1 learned transitions and one deterministic finalization per block, with occupancy targets set within the active block.Source: §C.2 · pp. 45–46

C.3 Numerical calibration of the visibility logits

  • A shared logit shift calibrates visibility probabilities to the target occupancy, using Newton iterations and a safeguarded fallback for residuals at least 10^-6.Both calibration paths remain in the automatic-differentiation graph during training.Source: §C.3 · p. 46

C.4 Supervised warmup

  • Supervised warmup uses dataset reference responses to construct visibility targets, then trains on decoder-derived confidence features, mask states, and time inputs.The two full-canvas stages use a combined bank of 495,070 states and comprise 7,304 optimizer updates.Source: §C.4 · pp. 46–47

C.5 RLVR and optimization settings

  • Separate full-canvas and two-block RLVR runs train from supervised policies for 32-call horizons using binary terminal correctness rewards.Each run uses groups of eight trajectories per prompt; finalization contributes neither an action nor a likelihood term.Source: §C.5 · p. 47

C.6 Baseline configurations

  • The baseline comparison retains distinct saved configurations, including variants that differ in response geometry or training or decoding variant.The 4 × 32 UPO configuration uses a 128-token response and is recorded only at T = 128.Source: §C.6 · p. 47, Table 33 · p. 48

D Evaluation protocols

The evaluation uses disjoint training, validation, and test populations, then reports problem-level micro-averaged accuracy across GSM8K and MATH-500. It evaluates exact decoder-call horizons while distinguishing capped comparisons from actual calls consumed.

D.1 Datasets, validation split, and prompts

  • The main test set combines 1,319 GSM8K problems and 500 MATH-500 problems, with a disjoint 1,024-problem validation set drawn from training sources.Validation uses an equal task mixture, unlike the test mixture, so its absolute accuracies are not directly comparable to main-test accuracies.Source: §D.1 · p. 48, Table 34 · p. 48

D.2 Decoder-call budgets and actual consumption

  • TIPS uses exact horizons from 4 to 256 decoder calls, but a baseline’s plotted call cap may exceed its mean actual consumption.Equal decoder-call counts do not imply equal wall-clock time or floating-point operation counts across response geometries.Source: §D.2 · p. 49

D.3 Output processing and answer verification

  • Generated responses are checked with task-specific extraction and correctness functions; timeouts, exceptions, mismatches, and missing or nonfinite predictions count as incorrect.The evaluation decodes the full 256-token response canvas without additional stop-string truncation.Source: §D.3 · p. 49

D.4 Randomness, aggregation, and uncertainty

  • Main accuracy is a problem-level micro-average over GSM8K and MATH-500, averaging three inference draws per policy and then three trained policies.Visibility actions are independent Bernoulli draws conditional on policy probabilities, while token proposals are greedy.Source: §D.4 · p. 49

Related papers

Loading 2610.22099v1…