PaperlayerView exact revision on arXiv

Source-linked AI summary

miniTE: Diagnosing and Bridging the Execution Gap in Native NVFP4 Reinforcement Learning

Zhen Li, Mingfa Feng, Yiming Zhang, Yang Yu, Changhai Zhou, Pengxiang Li, Dongkuan Wu, Congkai Xie, Hongxia Yang

arXiv:2610.23928v1cs.LGcs.AIstat.ML

TL;DR

Learner and sampler numerical execution can differ even with shared weights, and NVFP4’s format name does not specify the full execution path that affects policy evaluation. miniTE combines cross-engine diagnostics with native W4A4 expert computation and a connected quantization-to-GEMM implementation. On Qwen3-30B-A3B, it sustains long-horizon learning, outperforms standard TE NVFP4 on all four mathematics benchmarks, and achieves higher complete learner-update throughput than both TE NVFP4 and BF16.

  • Problem

    The numerical dependencies that must be controlled across separate learner and sampler engines remain unclear, although their probabilities enter importance ratios and clipping decisions.

    Source: §1 · p. 1

  • Method

    miniTE combines cross-engine checks of quantized operands, expert outputs, and policy evaluation with native W4A4 expert computation and a connected quantization-to-grouped-GEMM dataflow.

    Source: §1 · pp. 1–2

  • Results

    1.47× TE NVFP4 and 1.39× BF16: optimized learner-update throughput; miniTE also outperformed standard TE NVFP4 on all four mathematics benchmarks, with a five-task average close to BF16.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The framework provides configurable recipes, replaceable expert operators, reusable diagnostics, and reproducible launch artifacts for developing custom low-bit RL settings.

    Source: §1 · p. 2

    Learning evidence covers one MoE model with one run per configuration, and implementation gains were evaluated on Blackwell hardware.

    Source: §8 · p. 10

Abstract

from arXiv · show

Native NVFP4 promises to accelerate reinforcement learning (RL) for large language models (LLMs), but achieving stable learning across separate training and sampling engines remains challenging. We show that matching model weights and numerical formats is insufficient to ensure consistent policy evaluation, highlighting the importance of cross-engine numerical execution. We introduce miniTE, a lightweight framework that combines execution diagnostics, cross-engine numerical alignment, and efficient native W4A4 (4-bit weights and activations) GEMM computation for LLM RL. It improves numerical execution consistency without modifying the RL objective, while reducing implementation overhead to realize the benefits of native low-bit computation. Long-horizon RL experiments on Qwen3-30B-A3B demonstrate stable training and benchmark performance close to BF16, outperforming naive Transformer Engine (TE) NVFP4 in evaluation quality. On single NVIDIA B300 GPU node, the optimized learner reaches 33,540 tokens/s, yielding 1.39x and 1.47x the throughput of BF16 and TE NVFP4, respectively, for complete learner updates. We released the framework, kernels, diagnostics, and training configurations to make native low-bit RL easier.

1 Introduction

The introduction frames cross-engine numerical behavior as consequential because learner and sampler probabilities enter RL optimization, even when model weights match. miniTE combines execution diagnosis, native W4A4 expert computation, and practical tooling; its long-horizon runs improve over naive NVFP4 without changing the RL objective.

1 Introduction

  • Learner and sampler probabilities enter importance ratios and clipping decisions, making numerical execution differences consequential even at a shared parameter version.NVFP4 adds choices in scale domains, rounding, accumulation, and operator composition that a format name alone does not specify.Source: §1 · p. 1
  • miniTE checks quantized operands, expert outputs, and policy evaluation across sampler and learner, pairing a strict execution reference with per-token quantization and native grouped GEMM.Its default path uses BF16 backward.Source: §1 · p. 1
  • Frozen-policy, routing-conditioned, and scaling controls identify numerical differences beyond nominal NVFP4 recipe agreement, with strict execution serving as an actionable alignment reference.The diagnosis finds enlarged cross-engine log-probability tails and FC1/FC2 discrepancies despite matched target-layer assignments.Source: §1 · pp. 1–2
  • miniTE’s integrated W4A4 expert-forward path with BF16 backward sustains long-horizon learning and improves all four mathematics benchmarks over naive NVFP4 without modifying the RL objective.The introduction reports complete learner-update throughput of 1.39× BF16 in the optimized configuration and 1.33× in the repeated core study; same-recipe expert-forward speedup over TE U3045 is 2.46×.Source: §1 · p. 2
  • The framework exposes configurable numerical recipes, replaceable expert operators, reusable diagnostics, and reproducible launch artifacts for custom low-bit training settings.Released materials include Dockerfiles, reference configurations, and single-command launch scripts.Source: §1 · p. 2

2 Background and Problem Setup

The background distinguishes quantization error, cross-stack execution difference, and implementation difference, because each requires a different comparison. It also specifies the quantized format and the RL likelihood correction used in non-strict runs.

2 Background and Problem Setup

  • Non-strict RL runs use truncated importance sampling to correct learner–sampler likelihood differences, with its bounded correction computed from pre-update likelihood ratios.That correction is separate from clipping the learner’s update ratio.Source: §2 · p. 2
  • Teacher-forced scoring of identical sequences holds parameter version, token IDs, and scoring context fixed to separate execution differences from weight lag and sampling variation.Source: §2 · p. 2
  • NVFP4 represents values with E2M1 codes, one E4M3 scale per 16-value block, and an outer FP32 scale; row-wise outer scaling reduces dependence on maxima in other rows.The format’s scale and rounding details do not by themselves specify execution choices such as reduction order, padding, or placement of router weights and casts.Source: §2 · pp. 2–3
  • The paper separates quantization error within a stack, learner–sampler execution difference, and optimized-versus-reference implementation difference, assigning each a distinct measurement.Frozen-policy evaluation measures cross-stack differences; reference-implementation tests measure optimization effects; training and benchmark evaluation measure learning quality.Source: §2 · p. 3

3 Diagnosing the Execution Gap

The diagnosis finds cross-engine numerical differences before optimization and shows that matching recipes and target-layer routing is not sufficient to eliminate them. These controls motivate specifying the complete operator path rather than relying on finer scales alone.

3.1 Does the Gap Exist Before Optimization?

  • Before any optimizer update, the frozen-policy absolute log-probability tail rises from 0.1271 nats under BF16 to 0.2911 nats under NVFP4, exceeding the BF16 cross-engine floor.The paired increase is 0.1641, with a 95% confidence interval of [0.1311, 0.2005]; Figure 1’s left panel reports pooled P99.9 over all matched tokens.Source: §3.1 · p. 3, Figure 1 · p. 3
  • 1.203% versus 0.627%: numerical-only ratios exceed analysis clipping bounds for NVFP4 tokens versus BF16 tokens, a clipping-potential measure rather than a training outcome.Training also depends on advantages, updates, and TIS correction.Source: §3.1 · p. 3

3.2 Are Recipe and Routing Agreement Sufficient?

  • Matched target-layer assignments leave FC1 relative-L2 discrepancies of 0.0381–0.6124 and FC2 discrepancies of 0.0810–0.7624, while inputs retain upstream differences.Figure 1’s right panel covers all 11 depths; natural routing also shows changing expert selections and nonmonotonic depth-dependent divergence.Source: §3.2 · p. 3, Figure 1 · p. 3
  • Changing only the sampler activation outer scale from per-tensor to per-token leaves FC1/FC2 discrepancies unchanged, and the tail change is inconclusive on the four-prompt cohort.The early per-token candidate also retains a larger tail than standard NVFP4.Source: §3.2 · p. 3
  • The controls motivate specifying operands and composition for each execution, then evaluating a native training path for learning behavior and cost.Finer scales alone do not establish cross-engine agreement.Source: §3.2 · p. 4

4 MINITE: A Practical Native NVFP4 RL Framework

miniTE makes execution specifications and diagnostic references explicit, then connects a native NVFP4 learning path to a jointly optimized quantization-to-GEMM dataflow. The framework retains higher-precision backward and non-expert computation while reducing repeated preparation around routed expert GEMMs.

4.1 Execution Specification and Diagnostic References

  • An execution specification records scale domains, rounding, logical layouts, and cast and router-coefficient placement, while provenance and expert IDs distinguish naturally accumulated inputs from common-input operator comparisons.Source: §4.1 · p. 4
  • Strict execution matches learner kernels, formulas, reduction orders, and surrounding paths to the sampler; an operator reference instead tests whether optimizations preserve operands and outputs under a fixed recipe.The practical learner is evaluated as a complete learning configuration, with implementation checks using the operator reference.Source: §4.1 · p. 4

4.2 The Native NVFP4 Learning Path

  • Each dispatched activation row receives its own outer scale, reducing dependence on other tokens’ maxima; the native path combines packed FP4 operands with FP32 epilogue rescaling and BF16 backward.The path disables Hadamard transforms and stochastic rounding; a context-swap test preserves target-row FP4 codes and GEMM outputs bitwise for the tested miniTE path.Source: §4.2 · p. 4, Figure 2 · p. 5
  • The default backward uses a straight-through approximation with saved original BF16 activations and weights, computing gradients with BF16 GEMMs rather than quantizing the backward operands.Source: §4.2 · p. 5

4 MINITE: A Practical Native NVFP4 RL Framework

  • Routed expert computation includes operand preparation and scheduling, with group sizes, empty groups, and padding affecting the interface between them; miniTE optimizes this interface as a connected dataflow.Source: §4.3 · p. 5

4.3 Joint Design of Quantization and Grouped GEMM

  • miniTE writes packed FP4 codes, block scales, and outer-scale information into contiguous buffers consumed by GEMM, with device-side offsets and reusable metadata supporting expert-dependent groups and padding.This removes repeated splitting and argument preparation between grouped interfaces, including support for empty groups.Source: §4.3 · p. 5
  • The FC1 reference scans padded rows for maxima, whereas miniTE collects maxima during permutation and pads activations and statistics together.Source: §4.3 · p. 5

5 Experiments

The experiments evaluate miniTE’s long-horizon learning, complete learner efficiency, and numerical execution choices against BF16 and standard NVFP4. They report stronger late-run reward and mathematics scores than standard NVFP4, faster complete learner updates, and execution checks that validate tested numerical transformations.

5.2 Long-Horizon Training Behavior and Evaluation

  • Across 1,000 steps, miniTE maintains reward growth and ends above standard NVFP4, while BF16 reaches higher reward; the two NVFP4 paths have similar mean log-probability discrepancies.Figure 3 shows one run per configuration, with faint raw curves and solid centered 21-step means.Source: §5.2 · p. 6, Figure 3 · p. 7
  • At step 999, miniTE’s five-task mean is 56.35%, up from 54.57% for NVFP4 but 2.49 points below BF16; IFEval ranks NVFP4 highest.MiniTE gains 10.00 and 3.88 percentage points over NVFP4 on AIME 2024 and 2025, respectively; its IFEval score is 38.40%, versus 38.36% for BF16 and 43.99% for NVFP4.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Operator and Learner Efficiency

  • 33,540 tokens/s is the optimized miniTE complete-update throughput, 1.39× BF16 and 1.47× TE NVFP4 on fixed-text updates.These eight-B300-GPU measurements process 524,288 tokens per update and exclude rollout and weight synchronization.Source: §5.3 · p. 7, Table 2 · p. 8
  • 2.46× is miniTE’s geometric-mean expert-forward speedup over TE U3045 across FC1/FC2 and four workload sizes under the same forward recipe.The module comparison excludes dispatch and uses pre-permuted inputs; its timing boundary differs from complete learner updates.Source: §5.3 · p. 7
  • Balanced module timings do not predict complete learner ranking: miniTE with BF16 backward is slower than BF16 in seven of eight module forward-plus-backward cells.On GB200, random weights with mock data activate 50.8 experts per layer on average versus 126.2 for pretrained weights and real DAPO text, out of 128; the latter workload remains imbalanced.Source: §5.3 · p. 8
  • In a separate colocated RL window, miniTE reduces learner time to 42.56 s from 47.04 s, but the two measured stages total 97.82 s versus BF16’s 56.98 s.Weight conversion and synchronization rise to 55.26 s from 9.94 s; the sums omit rollout and other iteration stages.Source: §5.3 · p. 8

5.4 Execution References and Design Validation

  • Strict execution records about 10−8 log-probability difference and maintains reward where the unaligned run deteriorates, while retaining native NVFP4 expert forward and BF16 gradients.The strict reference jointly matches kernels, formulas, and reduction orders to the sampler and omits TIS.Source: §5.4 · p. 8, Figure 4 · p. 9
  • All 28 unfused quantization/grouped-GEMM checks pass bitwise against upstream, and tested fused-versus-unfused layouts show zero deviation.With expert forward held fixed, BF16 backward is approximately 1.21× faster than MXFP8 backward across eight module cells; maximum gradient relative-L2 differences are 0.001663 and 0.037653, respectively.Source: §5.4 · p. 8

6 Related Work

Related work spans native low-precision learning, quantized RL, and execution mismatch. miniTE is positioned around residual cross-engine execution differences and native W4A4 expert-forward computation.

Native low-precision learning

  • Native low-precision learning ranges from four-bit storage with BF16 computation in QLoRA [Dettmers et al., 2023] to native MXFP4 forward and gradient GEMMs in Quartet [Castro et al., 2025].NVFP4 pretraining, Composer 2, and Miles are also described as native low-bit learning paths; miniTE studies residual cross-engine execution differences and native expert-path implementation cost.Source: §6 · p. 9

Quantized RL

  • Quantized RL approaches combine quantized rollout with low-rank adaptation, unified FP8 training and rollout, learner-side W4A16 QAT, or sequence-level trust-band optimization.miniTE instead evaluates native W4A4 expert-forward computation with BF16 backward while retaining the existing TIS-corrected RL pipeline.Source: §6 · p. 9

Execution mismatch and routing

  • Prior work addresses sampler–learner consistency, rounding precision, expert selection, and execution-context dependence; miniTE diagnoses residual differences under matched target-layer assignments.Its native implementation is evaluated through operator equivalence, learning traces, and complete learner timing.Source: §6 · p. 9

7 Conclusion

The conclusion presents miniTE as a native low-bit RL framework that joins cross-engine diagnosis with numerical and systems design. On Qwen3-30B-A3B, it reports sustained learning, stronger mathematics results than standard TE NVFP4, and faster optimized learner updates.

Conclusion

  • On Qwen3-30B-A3B, miniTE sustains long-horizon learning, exceeds standard TE NVFP4 on all four mathematics benchmarks, and achieves a five-task average close to BF16.The optimized learner reaches 1.47× TE NVFP4 throughput and 1.39× BF16 throughput; separate expert-forward measurements show 2.46× speedup over Transformer Engine.Source: §7 · p. 9
  • miniTE combines frozen-policy and routing-conditioned diagnostics with native W4A4 forward computation and independently selected backward precision.Its jointly optimized quantization and grouped GEMM are part of the framework’s numerical and systems co-design.Source: §7 · p. 9
  • Configurable recipes, replaceable operators, and reproducible launch environments position miniTE as an extensible foundation for numerical and systems co-design in native low-bit RL.Source: §7 · p. 9

8 Limitations and Future Work

miniTE’s remaining quality gap does not isolate quantization error from residual execution differences, and selected-stage speedups do not establish end-to-end RL gains. The evidence is bounded to one MoE model and one run per configuration on Blackwell hardware, with learner benchmarks using fixed text rather than complete RL iterations.

8 Limitations and Future Work

  • miniTE still falls short of BF16 in average evaluation quality and strict execution in cross-engine agreement; the remaining gap does not isolate quantization error from execution differences.Future work targets tighter execution alignment and format-aware error compensation while retaining native W4A4 computation.Source: §8 · p. 10
  • Selected-stage measurements do not establish end-to-end RL gains; weight conversion and synchronization remain integration costs, while fusion requires joint numerical and performance validation.Efficient fusion must coordinate scale updates, accumulator rescaling, and data movement within SM resource constraints.Source: §8 · p. 10
  • Learning evidence covers one MoE model with one run per configuration, and implementation gains are evaluated on Blackwell hardware.Broader validation across models, tasks, independent seeds, and hardware platforms remains future work.Source: §8 · p. 10

B Experimental protocols and implementation identities

  • Non-strict RL runs use TIS, including BF16, naive NVFP4, and miniTE, whereas strict execution runs without TIS.Frozen-policy diagnostics, isolated modules, and fixed-text learner benchmarks do not apply an RL importance correction.Source: B Experimental protocols and… · p. 13
  • Complete learner-update benchmarks use fixed text and include forward, backward, optimizer, and data-loading costs, but are not complete RL iterations.The input corpus contains 657 sequences of 4,097 tokens, constructed from DAPO-Math-17k text.Source: B Experimental protocols and… · p. 13
  • Repeated and optimized throughput results use different fusion settings and estimators, and the optimized experiment has only one process trial per configuration.The repeated experiment reports a paired speedup estimator distinct from the ratio of aggregated throughputs.Source: B Experimental protocols and… · p. 14

C Additional execution-gap diagnostics

Routing and execution controls show that natural cross-engine differences persist when target-layer assignments are matched, and common-input replay still finds module-output discrepancies. Changing rollout scaling alone does not remove the measured differences.

Additional execution-gap diagnostics

  • The frozen-policy tail diagnostic scores identical chosen-token sequences under teacher forcing from a step-zero Qwen3-30B-A3B checkpoint.Its tail cohort contains 32 prompts and 69,088 matched tokens per condition; layer captures are not independent observations.Source: C Additional execution-gap diagnostics · p. 14
  • 0.0268 nats higher P99 absolute log-probability discrepancy and 0.080 percentage points higher clipping potential occur with the early per-token candidate versus standard NVFP4.The candidate pairs a nvfp4_cursor2 learner with per-token NVFP4 rollout and is distinct from the miniTE learner benchmark.Source: C Additional execution-gap diagnostics · p. 14
  • Natural routing disagreement ranges from 0.0459–0.7559 and MoE-output relative L2 discrepancy from 0.0072–0.2831 across BF16 and NVFP4 conditions.The figure describes a frozen-policy sensitivity diagnostic, not the clipping fraction recorded during an RL optimization step.Source: C Additional execution-gap diagnostics · p. 14, Figure 5 · p. 15
  • With target-layer expert assignments fixed, FC1 discrepancy remains 0.0381–0.6124 and FC2 discrepancy 0.0810–0.7624.Upstream routing remains free to differ, so this is a conditional target-layer comparison, not an all-layer fixed-path intervention.Source: C Additional execution-gap diagnostics · p. 15
  • Changing rollout global scaling alone leaves FC1 and FC2 discrepancy unchanged; P99 and clipping-potential difference intervals include zero.The control uses a separate four-prompt cohort with the learner, checkpoint, inputs, and observed layer fixed.Source: C Additional execution-gap diagnostics · p. 15, Table 5 · p. 15
  • 0.0575 at FC1 and 0.0931 at FC2 remain as miniTE discrepancies under a common MoE-layer input at layer 0.FC2 includes preceding FC1, activation, and composition differences; the replay does not partition natural discrepancy into causal percentages.Source: C Additional execution-gap diagnostics · p. 16, Table 6 · p. 16

D Numerical specification and correctness tests

The section specifies miniTE’s numerical execution path and tests its operator behavior, then reports training, evaluation, and throughput evidence across distinct configurations. Its results distinguish numerical correctness checks, benchmark outcomes, and complete learner-update performance.

D Numerical specification and correctness tests

  • Across six context swaps, miniTE preserves target FP4 payload hashes and bitwise GEMM outputs, while five swaps change the per-tensor NVFP4 shared amax and outputs.The high-amplitude 1,024-token context is a negative control; these tests do not establish quantization accuracy against BF16 or whole-network batch invariance.Source: D Numerical specification and… · p. 17, Table 7 · p. 17
  • All 28 tested forward API comparisons pass bitwise, and seven fused-scale-layout cases match payloads, decoded scales, and BF16 GEMM outputs.The API checks cover 12 activation, 12 weight, and four GEMM comparisons; the fused-layout checks include six synthetic cases and one real-activation case.Source: D Numerical specification and… · pp. 17–18
  • Extreme-input checks show saturation or underflow rather than NaN/Inf propagation, while backward tests report high directional agreement with FP32 references.Subnormal handling differs by path, and the backward comparisons do not measure final policy quality.Source: D Numerical specification and… · p. 18, Table 8 · p. 18

E.1 Main RL protocol and training traces

  • The main training figure compares one exported run each for BF16, naive NVFP4, and miniTE with TIS over steps 0–999, retaining raw traces and centered 21-point means.The checkpoint evaluation uses step-999 checkpoints, with BF16 inference for BF16 and NVFP4 inference for both NVFP4-trained configurations.Source: §E.1 · p. 18, §E.2 · p. 18

E.2 Checkpoint evaluation and instruction following

  • 56.35% is miniTE’s five-task mean, 1.78 percentage points above NVFP4+TIS but 2.49 points below BF16.The task-level gain row reports +10.00 points on AIME 2024, +3.88 on AIME 2025, +0.55 on MATH-500, +0.08 on GSM8K, and −5.59 on IFEval; repeated checkpoint evaluations are not independent training seeds, and scores are point estimates without confidence intervals.Source: §E.2 · pp. 18–19

E.3 Strict-alignment diagnostic

  • Strict alignment is a joint execution intervention matching learner paths to rollout formulas, kernels, and reduction order, rather than an isolated deterministic-mode switch.Its trajectory spans steps 0–1199; Figure 6 retains complementary entropy and PG-loss panels with 15-step moving averages.Source: §E.3 · p. 19, Figure 6 · p. 20

F.1 Complete learner updates and backward precision

  • 33,540 tokens/s is reached by the optimized miniTE configuration, equal to 1.390× BF16 and 1.471× NVFP4 for complete learner updates.The reported configuration combines dense-path execution, GEMM-kind selection, permutation fusion, and amax fusion; the comparison measures them together. A repeated study separately reports 33,140 tokens/s for CUTLASS NVFP4 and 32,517 tokens/s for miniTE.Source: §F.1 · p. 20, Table 11 · p. 21

F.2 Module timings and upstream implementation comparison

The section compares grouped-linear module timings, colocated RL and serving measurements, and workload sensitivity on GB200. It reports strong module and serving throughput results while showing that these measurements cover different stages and workload conditions.

F.2 Module timings and upstream implementation comparison

  • miniTE forward execution is 2.46× faster than TE U3045 by geometric mean across eight cells, while its forward-plus-backward geometric mean is 1.10×.Forward speedups range from 1.587× to 3.065×; results vary by shape, and the full-model TE U3045 runs are an implementation reference rather than a quality-matched replacement.Source: §F.2 · pp. 21–22, Table 12 · p. 21

F.3 Colocated RL and serving measurements

  • In colocated RL, the miniTE learner is 1.11× faster, but weight synchronization takes 5.56× as long; the two measured stages together cost 40.84 seconds more.Their sum excludes rollout and other iteration stages, so it does not establish a change in total RL iteration time.Source: §F.3 · p. 22, Table 13 · p. 22
  • Standalone serving shows a 1.642× aggregate output-throughput gain and a 1.69× time-per-output-token ratio under fixed-length generation.These fixed-generation serving measurements omit the weight-conversion and synchronization costs observed in colocated RL.Source: §F.3 · p. 22, Table 14 · p. 22

G GB200 workload sensitivity

  • Real text increases empty expert-layer cells under random initialization but decreases them with pretrained weights in the matched routing snapshots.The comparison uses one step-2 routing snapshot per condition; these occupancy counts do not partition latency across execution components.Source: G GB200 workload sensitivity · pp. 22–23
  • The independent GB200 learner timing uses a pretrained checkpoint and a 59.7-million-token corpus, but its short window provides timing evidence rather than policy-quality evidence.Each value summarizes 37 updates in one process after three warm-up updates.Source: G GB200 workload sensitivity · p. 23, Table 16 · p. 23

Related papers

Loading 2610.23928v1…