PaperlayerView exact revision on arXiv

Source-linked AI summary

ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Tasks

Zeyu Michael Li, William Xingxu Chen, Bingshuo Qian, Jiayin Liu, Xiang Cheng

arXiv:2609.29102v1cs.CLcs.LG

TL;DR

Continuous dLMs have shown progress on reasoning and code, but their performance on challenging tasks remains less established than that of autoregressive LLMs and masked dLMs. ELF-REG trains continuous ELF with teacher-supervised intermediate feature alignment and a global teacher representation jointly denoised with the response, and applies early-stop decoding without few-step training. ELF-REG-B reaches 38.11% on GSM8K at NFE 64, exceeding S-FLM (NFE 512), MDLM and Duo (NFE 1024); ELF-REG-L raises MATH-500 pass@1 from 10.55% to 13.39% over ELF-L.

  • Problem

    Continuous dLMs have shown progress on reasoning and code, but their performance on challenging tasks remains less established than that of autoregressive LLMs and masked dLMs.

    Source: §1 · p. 2

  • Method

    ELF-REG trains continuous ELF with teacher-supervised intermediate feature alignment and a global teacher representation jointly denoised with the response, and applies early-stop decoding without few-step training.

    Source: §1 · p. 2

  • Results

    ELF-REG-B reaches 38.11% on GSM8K at NFE 64, exceeding S-FLM (NFE 512), MDLM and Duo (NFE 1024); ELF-REG-L raises MATH-500 pass@1 from 10.55% to 13.39% over ELF-L.

    Source: §4 · p. 6

  • Takeaways & Limitations

    Early-stop lets the same task-specific checkpoints generate from intermediate clean predictions at low NFE without dedicated few-step training.

    Source: §6 · p. 9

    The AR LLM and large-dLM references use much larger training-data budgets and are not comparable with this method; other baselines vary in prompting, response budgets, and code postprocessing.

    Source: §4 · p. 6

Abstract

from arXiv · show

Fully continuous diffusion language models (dLMs) denoise continuous representations without intermediate discretization, then decode all response tokens in parallel at the final step. Their performance on challenging reasoning tasks remains less established than that of autoregressive (AR) LLMs and masked dLMs. We scale Embedded Language Flows (ELF) to mathematical reasoning and code generation on GSM8K, MATH-500, HumanEval, and MBPP. We introduce ELF-REG, which improves learning with representation alignment and entanglement (REPA+REG), where a frozen AR teacher supervises intermediate denoiser features and supplies a global representation that is jointly denoised with the response. ELF-REG-L achieves 55.96% pass@1 on GSM8K at 64 network function evaluations (NFE), and 13.39% on MATH-500 and 22.56% on HumanEval at 128 NFE. It outperforms the evaluated comparable-scale dLMs in pass@1 on GSM8K and code, and improves MATH-500 pass@1 from 10.55% for the ELF-L baseline to 13.39% with ELF-REG-L. Without few-step training, the same task-specific checkpoints support strong low-NFE performance through early-stop, which decodes an intermediate clean prediction without completing the denoising trajectory. At 16 NFE, ELF-REG-L reaches 41.21% HumanEval pass@10, outperforming recent continuous dLMs of comparable scale.

1 INTRODUCTION

The introduction motivates improving continuous diffusion language models for reasoning and code, then presents ELF-REG’s teacher-guided training and early-stop decoding. The method outperforms comparable-scale dLMs on GSM8K and code in the reported comparisons, while improving MATH-500 over ELF-L.

1 INTRODUCTION

  • Continuous dLMs had shown promise on reasoning, but the introduction identifies improving accuracy at low compute scales as a motivation.Prior work had established continuous denoising for language modeling and conditional text generation, while reasoning and code progress motivated further development.Source: §1 · p. 2
  • 55.96% GSM8K pass@1 at 64 NFE for ELF-REG-L, with higher pass@1 than comparable-scale dLMs on the reported GSM8K and code comparisons.ELF-REG-L also improves MATH-500 pass@1 over ELF-L; early-stop supports low-NFE generation without dedicated few-step training.Source: §1 · p. 2
  • REPA+REG uses a frozen AR teacher to supervise intermediate denoiser features and provide a global representation jointly denoised with the response.The paper extends ELF to GSM8K and MATH-500 reasoning and HumanEval and MBPP code generation.Source: §1 · p. 2
  • 46.39% pass@10 on MBPP-378 at 8 NFE with early-stop, exceeding PlaidQ-D16’s 40.49% at 17 NFE without dedicated few-step training.Early-stop decodes an intermediate clean prediction; on GSM8K, accuracy approaches its final level while responses continue changing.Source: §1 · p. 2

2 METHODOLOGY

ELF generates continuous response representations from noise and decodes all response tokens in parallel at the final step. ELF-REG adds teacher-based representation alignment and a jointly denoised global representation, while prefix early-stop enables generation from an intermediate clean prediction without few-step training.

2.1 ELF PRELIMINARIES

  • ELF starts response representations from Gaussian noise, predicts clean states with a bidirectional transformer, and decodes all response tokens in parallel in one final evaluation.Training combines velocity regression and token cross-entropy, with clean predictions used for self-conditioning.Source: §2.1 · pp. 2–3

2.2 REPRESENTATION ALIGNMENT AND ENTANGLEMENT

  • REPA aligns student states with clean teacher states, while REG jointly denoises a global teacher representation alongside the text response.A frozen AR teacher supplies the targets; at inference the REG state remains available to the decoder, which emits only text, while the teacher and projector are training-only.Source: §2.2 · p. 3

2.3 PREFIX EARLY-STOP GENERATION

  • Prefix early-stop runs the first N−1 updates of a longer denoising grid, then decodes predicted-clean text and REG states without few-step training.The method uses a master budget M = ρN and sets ρ = 8; NFE includes the final decoder evaluation.Source: §2.3 · p. 3

3 ANALYSIS

The analysis relates early-stop decoding to the remaining flow and evaluates endpoint error, response stability, and task answers along sampling trajectories. On GSM8K, ELF-REG-B has lower endpoint error and higher agreement than the baseline at NFE 64, while many early-stop numerical answers match the final answers.

3.1 EARLY-STOP DECODING AND THE REMAINING FLOW MAP

  • Early-stop replaces the remaining flow map with the denoiser’s clean prediction before applying the same deterministic parallel token decoder.The analysis compares that decoded prediction with the decoded endpoint; its flow-map framing assumes a smooth flow with fixed prompt and guidance scale.Source: §3.1 · p. 3, §3.2 · p. 4

3.2 ENDPOINT ERROR AND DECODER STABILITY ALONG THE SAMPLING TRAJECTORY

  • At NFE 64, ELF-REG-B has 16.1% lower mean text endpoint error and 8.61% endpoint response agreement, versus 1.46% for ELF-B baseline.Figure 2 tracks acceleration, endpoint error, stable tokens, response agreement, and pass@1; ELF-REG-B also reaches token stability earlier.Source: Figure 2 · p. 5, §3.2 · p. 5
  • At t = 5/64, ELF-REG-B’s median maximum remaining joint acceleration is 84.8% lower and mean joint endpoint error is 7.5% lower than ELF-B baseline.These are finite-difference measurements along the discrete sampler, whereas Proposition 1 concerns a smooth continuous flow.Source: §3.2 · p. 5
  • At NFE 64, numerical answers match NFE 512 on 87.11% of ELF-REG-B question–seed pairs and 80.27% of ELF-B baseline pairs.Further denoising yields only a 0.64 percentage-point correctness gain for ELF-REG-B; numerical-answer agreement is more permissive than complete-response agreement.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate task-specific ELF and ELF-REG models on mathematical reasoning and code generation, using zero-shot early-stop generation for headline results. They compare benchmark performance and analyze training components, sampling choices, and representation settings.

4 EXPERIMENTS

  • At NFE 128, ELF-REG-L exceeds PlaidQ+CFG at NFE 257 on all four code benchmarks, including MBPP-378 pass@1 28.92% vs 24.58% and HumanEval pass@1 22.56% vs 22.04%.ELF-REG-L also outperforms Open-dCoder and oDLM at matched NFE 128 on all four benchmarks.Source: Table 1 · p. 6, §4 · p. 6
  • 55.96% GSM8K pass@1 at NFE 64 and 13.39% MATH-500 pass@1 show ELF-REG-L’s mathematical-reasoning results; MATH-500 rises from 10.55% for ELF-L baseline.ELF-REG-B reaches 38.11% GSM8K pass@1 at NFE 64, exceeding several evaluated dLMs at higher NFE.Source: §4 · p. 6, Table 2 · p. 7
  • At NFE 128, ELF-REG-L reaches 55.05% MBPP-378 pass@10 and 48.87% HumanEval pass@10, versus 44.79% and 39.87% for PlaidQ+CFG at NFE 257.At NFE 16, ELF-REG-L already exceeds PlaidQ+CFG at NFE 257 on both code benchmarks’ pass@10; MATH-500 pass@16 rises from 44.0% to 52.2%.Source: §4 · p. 7, Table 3 · p. 7

4.1 ABLATIONS AND ADDITIONAL ANALYSES

  • At epoch 6 on GSM8K, REPA improves over Baseline, REPA+REG improves further, and stripped REPA+REG retains most of the added gain without REG at inference.REPA+OLT, which adds one learnable token to REPA, does not reproduce the REG benefit.Source: §4.1 · p. 7, Table 4 · p. 8
  • At matched NFE, prefix early-stop improves ELF-REG-B accuracy over full-span generation by decoding predicted-clean states rather than the updated latent.The comparison uses ELF-REG-B at epoch 12 and averages over sixteen generation seeds.Source: §4.1 · p. 8, Table 5 · p. 8
  • Errors induced by coarser, fewer-NFE trajectories are more damaging than random errors of matched magnitude.Source: §4.1 · p. 8
  • The Qwen3-0.6B-Base encoder supports more effective learning and gold-answer recovery under corruption than t5-small; target-position effects depend on corruption time.Position-shifted AR teacher targets strengthen forward perturbation influence at the supervised block, while final-block ordering depends on corruption time.Source: §4.1 · p. 8

5 RELATED WORKS

Prior work develops discrete and continuous diffusion language models for generation, reasoning, and code, alongside methods for faster sampling and representation supervision. ELF-REG combines teacher-based representation alignment with a jointly denoised global representation and inference-time early stopping.

5.1 DIFFUSION LANGUAGE MODELS

  • Discrete dLMs reverse token corruption or masking, while continuous dLMs denoise representations; approaches span language modeling, conditional generation, reasoning, code, and hybrid AR generation.The related methods include masked dLMs, flow-based continuous models, latent diffusion, and continuous diffusion within AR models.Source: §5.1 · p. 8
  • Few-step methods learn flow maps, refine confident predictions, or train separate few-step generators, whereas early stopping offers a sampling-time alternative.Continuous dLMs can decode an entire sequence early based on decoder confidence; masked dLMs may instead finalize individual tokens or commit remaining tokens.Source: §5.1 · pp. 8–9

5.2 REPRESENTATION ALIGNMENT

  • ELF-REG combines REPA alignment of noisy denoiser states to clean AR teacher states with REG’s jointly denoised global representation.Prior representation-alignment approaches differ in their targets and where alignment is applied.Source: §5.2 · p. 9
  • The paper pairs ELF-REG with inference-time early-stop to study low-NFE reasoning and code generation without dedicated few-step training, and analyzes early-stop properties.Source: §5.2 · p. 9, §5.1 · p. 9

6 CONCLUSION AND DISCUSSION

The paper scales fully continuous ELF to mathematical reasoning and code generation, reporting strong comparisons with similar-scale dLMs. It also studies early-stop generation without dedicated few-step training.

6 CONCLUSION AND DISCUSSION

  • ELF [24] scales its fully continuous dLM to mathematical reasoning and code generation using representation alignment and entanglement, outperforming similar-scale dLMs on GSM8K and code-generation benchmarks HumanEval and MBPP.Source: §6 · p. 9
  • ELF-REG-L exceeds PlaidQ-D16 on MBPP-378 pass@10 with fewer NFE, despite PlaidQ-D16’s dedicated few-step training; early-stop enables strong low-NFE generation without dedicated few-step training.Early-stop decodes an intermediate clean prediction, allowing the same trained models to generate responses with a limited denoising budget.Source: §6 · p. 9
  • Early-stop decodes an intermediate clean prediction, enabling the trained task-specific models to generate responses with a limited denoising budget.The authors identify improving and scaling continuous dLMs into generalist models, with the aim of matching AR LLM performance at lower cost, as future work.Source: §6 · p. 9

A METHOD DETAILS

The method combines teacher-derived representation targets and training losses with guidance-aware velocity targets, then samples using an Euler ODE procedure that supports early-stop. The appendix also documents default settings for the headline configurations.

A.1 CLEAN ENCODER AND TEACHER REPRESENTATIONS

  • The clean text states and global targets are normalized across features, while text REPA targets retain raw values and the REG position uses teacher-derived pooled and clean targets.The REPA target at the REG position uses the pooled representation from the REPA teacher layer; the clean REG denoising target uses the REG teacher layer.Source: §A.1 · p. 17

A.2 ALIGNMENT AND LOSS NORMALIZATION

  • The causal teacher sees clean prompt and response, while the bidirectional student sees a clean prompt and noisy response; REPA alignment excludes special tokens and padding.REPA uses a position-wise MLP projector, and its loss takes separate masked means for prompt and response positions before weighting them.Source: §A.2 · p. 17

A.3 TRAINING-TIME GUIDANCE AND VELOCITY TARGETS

  • Training samples self-conditioning and guidance settings, then uses auxiliary predictions at the same noisy state, time, and scale to construct guidance-aware targets.The trainable prediction uses detached clean states when self-conditioning is active; the guided targets train the network to incorporate guidance in a single inference denoiser evaluation.Source: §A.3 · p. 17
  • The training objective combines response-position ELF losses, teacher-alignment REPA losses, and REG reconstruction losses, with coefficients weighting the auxiliary losses.REPA aligns valid response content positions and the REG position, including decoder examples; REG velocity loss applies only to denoising examples.Source: §A.3 · p. 18

A.4 SAMPLING BEHAVIOR

  • Sampling uses an Euler ODE sampler whose time distribution and noise scale match denoising training; early-stop is available through the sampling algorithm.NFE counts denoiser evaluations plus one final decoder evaluation; FLOPs and latency differ across full-span denoising, vocabulary projections, and cached AR decoding.Source: §A.4 · p. 18

B.1 DEFAULT HYPERPARAMETERS

  • Table 6 documents default hyperparameters for headline configurations, with GSM8K settings distinguishing model sizes and shared entries spanning task columns.The table caption identifies REPA+REG with ELF-REG-B or ELF-REG-L according to model size and specifies that training epochs identify evaluated checkpoints.Source: Table 6 · pp. 18–19, §B.1 · p. 18

B.2 TRAINING DATA AND INITIALIZATION

The appendix describes task-data preparation, evaluation protocols, and accounting conventions for training, NFE, and model parameters. It also reports reduced training GPU-hours for ELF-REG at GSM8K accuracy thresholds.

B.2 TRAINING DATA AND INITIALIZATION

  • The evaluated checkpoints initialize ELF from scratch for GSM8K and code, while MATH models receive eight fine-tuning epochs from GSM8K weights at epoch 7.5.The MATH initialization uses the EMA 0.999 checkpoint; Table 7 reports epoch-based estimates of training tokens and GPU-hours.Source: §B.2 · p. 19, Table 7 · p. 20
  • At GSM8K accuracy thresholds, ELF-REG reduces training GPU-hours relative to the same-architecture baseline by 14.1% for ELF-B and 31.4% for ELF-L.Table 8 reports H200 GPU-hours to reach the Figure 3 thresholds and defines reductions relative to the baseline of the same architecture.Source: §B.2 · p. 19, Table 8 · p. 20
  • Training data comes from OpenMathInstruct-2 and MetaMathQA for GSM8K, OpenMathInstruct-2 for MATH, and OpenCodeInstruct for code, with filtering and benchmark-overlap removal.The processed datasets contain 2,404,028 GSM8K examples, 3,572,583 MATH examples, and 2,740,479 code examples.Source: §B.2 · p. 20

B.3 BENCHMARKS, METRICS, AND EVALUATION PROTOCOLS

  • Code evaluation uses EvalPlus extraction and execution; failures and timeouts count as incorrect, and benchmark-specific prompts and response budgets are specified.The protocol distinguishes HumanEval and HumanEval+ from MBPP-378 and MBPP-500, including their test suites and runtime limits.Source: §B.3 · p. 21
  • Evaluation reports mean pass@1 and standard deviation across seeds, with sixteen samples per question; headline math and code results use ρ = 8 early-stop.GSM8K, MATH-500, and code use specified answer-scoring and execution protocols; HumanEval+ requires passing both base and extended tests.Source: §B.3 · p. 21

B.4 NFE AND PARAMETER ACCOUNTING

  • NFE includes generative denoiser and decoder evaluations but excludes the fixed prompt-encoder pass, so it does not equate FLOPs or latency across architectures.PlaidQ CFG adds a model evaluation at each denoising step, while other works’ sampling-step budgets are sometimes retained without conversion to NFE.Source: §B.4 · p. 22
  • ELF-REG parameter counts include its REPA projector and REG projections; the frozen Qwen3 encoder runs at inference, while the teacher supplies training targets only.The additional frozen encoder and teacher parameter counts are separate from the ELF parameter counts.Source: §B.4 · p. 22

B.5 EXTERNAL COMPARISON SOURCES AND TRAINING BUDGETS

This appendix documents how external comparisons and analysis protocols are assembled, and describes methods for examining early-stop endpoint error, decoded-output agreement, and self-conditioned sampling. It also states the scope and assumptions of the training-budget comparisons and theoretical bounds.

B.5 EXTERNAL COMPARISON SOURCES AND TRAINING BUDGETS

  • External comparison curves and scores draw on prior papers, but training corpora, tokenizers, evaluation protocols, prompting, response budgets, and code postprocessing differ.Some curves combine configurations or use approximate values extracted from figures; NFE is unreported for some code scores.Source: §B.5 · pp. 22–23
  • Task-training token estimates exclude foundation pretraining, do not measure FLOPs, and are approximate for some baselines; TESS 2 is not comparable because of its scale.The PlaidQ bar excludes unreported distillation tokens, and the DBTM batch size may underestimate its budget.Source: §B.5 · p. 23

B.6 ANALYSIS PROTOCOLS

  • Analysis protocols set checkpoints, seeds, test sets, guidance, and sampling grids separately for learning curves, trajectory diagnostics, encoder comparisons, and training ablations.The protocols specify distinct evaluation counts and settings for each analysis.Source: §B.6 · p. 24

C.1 ENDPOINT ERROR

  • The endpoint-error derivation compares an integrated flow with the clean predictor’s one-step update.The supplied derivation integrates velocity and subtracts the update Dt(y) = y+(1−t)v(y,t).Source: §C.1 · p. 24

C.2 DECODING AND DISTRIBUTIONAL AGREEMENT

  • Coupling predictions from the same initial noise bounds latent Wasserstein distance, while decoded-output agreement additionally requires the decoder’s margin and sensitivity conditions.The expected-accuracy difference is bounded by total variation; the margin and local Lipschitz conditions must hold almost surely and may vary by trajectory.Source: §C.2 · p. 25
  • A small decoded-output total-variation bound follows when both the small-margin probability and the large endpoint-error probability are small.The stated bound concerns decoded outputs; task accuracy still depends on whether changed responses are correct.Source: §C.2 · p. 25

C.3 SELF-CONDITIONING, REG, AND THE DENOMINATOR CLAMP

  • The discrete trajectory state includes self-conditioning, and the prefix discrepancy separates denominator-clamp correction from velocity changes along that trajectory.A flow map on the latent state alone would discard self-conditioning information.Source: §C.3 · pp. 26–27
  • Prefix early-stop runs the first N−1 denoiser updates and decodes the latest clean prediction, while full-span generation decodes the updated endpoint.Both approaches use one decoder evaluation; early-stop executes only a prefix of a longer sampling grid.Source: §C.3 · p. 26

C.4 ENDPOINT-ERROR AND DISCRETE-ACCELERATION MEASUREMENTS

These measurements compare early-stop predictions with their own full-trajectory endpoints on GSM8K and characterize the sampled prediction times and interpolation SNR. The section also distinguishes discrete acceleration measurements from the continuous assumption used for the endpoint-error bound.

C.4 ENDPOINT-ERROR AND DISCRETE-ACCELERATION MEASUREMENTS

  • Endpoint-error measurements compare saved clean predictions with same-noise endpoint latents across the full generated canvas, and decoded agreement includes EOS status and length.The analysis averages question-level measurements within seeds and then across four seeds; reference-pass and diagnostic decoder costs are separate from reported exit NFE.Source: §C.4 · p. 27
  • At the NFE-64 exit, nominal training-interpolant power SNR is computed from t^2/[σ^2(1−t)^2] with σ = 2 and approximately unit-variance clean states.This describes the training corruption level at the exit, not a signal/noise decomposition of the generated ODE state.Source: §C.4 · p. 27, Table 11 · p. 28
  • Most numerical answers agree between NFE 64 and NFE 512 despite low complete-response agreement, while correctness gains and losses nearly cancel for ELF-B baseline.The comparison pairs the same question–seed samples and covers 1,319 test questions with four generation seeds.Source: Table 10 · p. 28
  • Discrete acceleration is a finite-difference measure along uniform-grid trajectories, but substituting its measured maximum for the continuous acceleration assumption does not certify the endpoint-error bound.The joint norm includes response positions after EOS and REG when present, but excludes the prompt.Source: §C.4 · p. 28, Table 12 · p. 28

D.1 ADDITIONAL HEADLINE RESULTS

The additional results report benchmark comparisons and analyses of encoder choice, token influence and settling, training settings, and early stopping. Across these analyses, the authors examine both performance and the behavior of intermediate predictions.

D.1 ADDITIONAL HEADLINE RESULTS

  • On MATH-500, ELF-REG-L records mean pass@1 of 13.31% at 64 NFE and 13.39% at 128 NFE; at 64 NFE, its pass@10 is 44.04%, versus 36.73% for the baseline.Source: §D.1 · p. 29

D.2 NUMERICAL VALUES UNDERLYING FIGURE 1

  • Tables 21–23 provide numerical results underlying Figure 1's MBPP-378 pass@10, GSM8K mean pass@1, and task-training-token panels.The supplied passages identify the panels and tables but do not give their cell values.Source: §D.2 · p. 31, Table 21 · p. 32, Table 22 · p. 32

D.3.1 CLEAN ENCODER COMPARISONS

  • Qwen3-0.6B-Base supports stronger GSM8K and MMLU learning and corrupted-input gold-answer recovery than t5-small, though the encoder configurations also differ in preprocessing.Both configurations reliably recover gold answers without corruption; Qwen3 retains higher recovery under corruption, especially on MMLU.Source: §D.3.1 · p. 31, Figure 4 · p. 32

D.3.2 SINGLE-LATENT PERTURBATION ACROSS CORRUPTION TIMES

  • At the REPA-supervised block, shift-1 produces stronger forward influence than shift-0 across three corruption times, but the final-block ordering varies with time.The comparison measures later-versus-earlier hidden-state changes after replacing one digit's latent in a GSM8K rationale.Source: §D.3.2 · p. 33, Table 24 · p. 34

D.3.3 POSITION-DEPENDENT TOKEN SETTLING

  • REPA+REG shift-0 settles earlier than the other tested conditions in every response-position bin, and settling does not progress monotonically across positions.Settling is the last observed update at which a token changes.Source: §D.3.3 · p. 33, Figure 7 · p. 35

D.4 TRAINING ABLATIONS

  • Training ablations compare configurations within each model size, and validation pass@1 is used to select settings for the headline results.The reported ablation tables use single-seed pass@1 at 65 NFE.Source: §D.4 · p. 35, Table 25 · p. 36, Table 26 · p. 36

D.5.1 EFFECT OF THE EARLY-STOP RATIO

  • The authors choose early-stop ratio ρ = 8 to balance low- and high-NFE performance; coarse-sampling errors are more damaging than matched-size random errors, with retention ordering depending on error source.ELF-REG-B retains more shared solutions under its own coarse error and at reduced full-span budgets, but recipient ordering changes with error source.Source: §D.5.1 · pp. 35, 37, Figure 8 · p. 37, Table 28 · p. 38

E COMPARISON WITH RELATED WORK

  • Table 29 organizes selected generation methods by generated state, generation structure, and training or sampling approach, and representation supervision by feature alignment versus jointly denoised representations.The table also identifies AR as autoregressive.Source: Table 29 · pp. 39–40

E.1 EARLY-STOP GENERATION

Early stopping returns an intermediate clean prediction rather than completing denoising. The paper relates this choice to prior stopping methods and analyzes endpoint error and decoded-response agreement.

E.1 EARLY-STOP GENERATION

  • Prior continuous-generation work considers fixed or adaptive exits, while masked dLMs use confidence to commit tokens or terminate remaining generation.The paper situates its approach among these stopping choices without treating their evidence or returned predictions as identical.Source: §E.1 · p. 40
  • The paper's prefix early-stop sampler returns a clean prediction, while its endpoint discrepancy compares that prediction with the endpoint of the same generated flow.The paper distinguishes this discrepancy from clean-sample mean-squared error and uses a decoder-margin condition to bound complete-response agreement.Source: §E.1 · pp. 40–41
  • The analysis bounds intermediate-prediction endpoint discrepancy through remaining acceleration and applies decoder margins to complete-response agreement.It separately derives the discrepancy for the implemented self-conditioned sampler.Source: §E.1 · pp. 40–41

F LIMITATIONS

The study uses separate task-specific models, focuses code experiments on Python, and leaves generalist training and variable response-canvas lengths for future work.

F LIMITATIONS

  • Models are trained separately for GSM8K, MATH, code, and MMLU; the MATH models start from corresponding GSM8K weights.Joint training of one generalist model across these tasks remains future work.Source: F LIMITATIONS · p. 41
  • The code study focuses on Python and does not cover other programming languages.The authors identify extending training data and benchmarks to additional languages as future work.Source: F LIMITATIONS · p. 41
  • ELF uses a fixed-length response canvas, and adapting it to different canvas lengths without performance reductions requires finetuning.The authors leave this adaptation for future work.Source: F LIMITATIONS · p. 41
  • Compute constraints prevented repeating training with multiple random seeds.Source: F LIMITATIONS · p. 41

G QUALITATIVE EXAMPLES

The qualitative examples reproduce correct and incorrect ELF-REG-L responses from the reported benchmark settings, including mathematical reasoning and code-generation tasks.

G QUALITATIVE EXAMPLES

  • Examples use 64 NFE for GSM8K and 128 NFE for MATH-500, HumanEval, and MBPP-378, with ρ = 8 early-stop.The section gives one correct and one incorrect response from each benchmark and includes reference answers for incorrect responses.Source: G QUALITATIVE EXAMPLES · p. 42
  • A GSM8K example calculates a brother's age as 21 after starting from a 9-year-old sibling's age and advancing three years.Source: G QUALITATIVE EXAMPLES · p. 42
  • A code example implements triangle area as a × h / 2, matching the supplied test case for side 5 and height 3.Source: G QUALITATIVE EXAMPLES · p. 44
Loading 2609.29102v1…