Self-Regularized Flow Language Models

• Autoregressive decoding remains sequential, whereas diffusion models reverse sequence-wide corruption and can generate multiple tokens in parallel.Continuous trajectories also support ODE- and SDE-based sampling and can be approximated with few integration steps.Source: §1 · pp. 1–2
• Prior image-diffusion work suggests denoising trajectories contain self-supervision beyond clean-data reconstruction, motivating the question of whether such objectives help continuous language models.Some prior approaches rely on external pretrained encoders, while others derive supervision within the generative model.Source: §1 · p. 2
• SERF trains on two corruption levels along the same flow path: an EMA teacher processes the cleaner input and supplies targets for the noisier student's denoising.This augments the standard objective without external pretrained models or changes to architecture or inference.Source: §1 · pp. 2–3
• SERF-L aligns shallow student features with deeper teacher features, while SERF-P aligns token distributions, including through an early student layer.The early-layer distribution variant uses an adapter and the teacher's output head.Source: §1 · p. 2
• On OpenWebText, SERF reaches FLM's MAUVE with up to 7.7× less wall-clock training and raises MAUVE by 24.3%; on GSM8K, S-FLM accuracy improves 28.8% with 64× fewer sampling steps.The GSM8K comparison matches S-FLM's 1024-step accuracy using 64× fewer sampling steps.Source: §1 · p. 3

Discrete Diffusion for Language

• Discrete language diffusion reverses masking or token-substitution corruptions, but tractable inference uses factorized reverse-process approximations and discrete states limit continuous optimization tools.SERF instead operates entirely in continuous space, avoiding discrete approximations during training and inference.Source: §2 · p. 3

Continuous Diffusion for Language

• Continuous language models use learned, pretrained, or one-hot representations, while FLM, S-FLM, and ELF use Euclidean, hyperspherical, and T5-embedding spaces, respectively.SERF applies to all three paradigms.Source: §2 · p. 3
• Flow-map distillation targets fewer sampling steps, whereas SERF targets the training objective and leaves inference unchanged.The passage describes these goals as orthogonal.Source: §2 · p. 3

Self-Supervised Learning in Image Diffusion

• Image-diffusion self-supervision includes external-encoder feature alignment and within-model objectives such as contrastive learning and teacher-student alignment along denoising trajectories.Prior work is cited as motivating whether auxiliary self-supervision can improve continuous flow-matching language models.Source: §2 · pp. 3–4

3.1 PRELIMINARY: FLOW LANGUAGE MODELS

• Flow language models represent token sequences as continuous one-hot vectors, train a denoiser to predict clean tokens from noisy inputs, and sample by integrating a probability-flow ODE.The final continuous state is decoded to tokens with argmax.Source: §3.1 · p. 4

3.2 SELF-REGULARIZED FLOW LANGUAGE MODELS

• SERF adds a weighted self-distillation loss to standard FLM training, using an EMA teacher on a cleaner input along the same interpolation path as the noisier student input.The two views share noise; only the student receives gradients, while the teacher is updated through EMA.Source: §3.2 · pp. 4–5

3.2.1 LATENT DISTILLATION

• On OWT samples generated with 1,024 sampling steps, SERF-L and SERF-Pk produce more fluent text, while FLM exhibits grammatical errors and repetitions.The compared models were trained for 1M or 300K steps and sampled from the same initial noise.Source: Figure 3 · p. 6
• SERF-L projects shallow student features to align them with deeper teacher features using cosine distance, encouraging early contextual representations.The projection head is discarded at inference.Source: §3.2.1 · pp. 5–6

3.2.2 PROBABILITY DISTRIBUTION DISTILLATION

• SERF-P aligns output token distributions, while SERF-Pk applies distribution supervision to an intermediate student layer through an adapter and the frozen teacher head.The shared teacher head maps both distributions to token probabilities and avoids interference with the student’s final output head.Source: §3.2.2 · pp. 6–7

3.3 STUDENT AND TEACHER NOISE LEVEL STRATEGIES.

• The noise-level strategy either uses a fixed teacher offset, clipped at t = 1, or randomizes student and teacher times while ensuring the teacher input is cleaner.Random offsets expose distillation to a range of student–teacher noise gaps.Source: §3.3 · p. 7

4.2 UNCONDITIONAL TEXT GENERATION RESULTS

• On OWT, SERF-L and SERF-Pk raise 1024-step MAUVE from 0.313 to 0.389 and 0.388, respectively, while using 300K rather than 1M training steps.Both variants also achieve lower Gen. PPL than FLM at every sampling budget; SERF-L’s 64-step Gen. PPL is 82.3 versus FLM’s 86.2.Source: §4.2 · pp. 8–9
• On OWT, SERF-L improves over ELF-B at every tested guidance scale, with the largest gains at w = 1 and w = 1.5.SERF-L and ELF-B were trained for five epochs and compared using 64 sampling steps across a sweep of guidance scales.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 REASONING ON GSM8K

• At τ = 1 with 1024 sampling steps, SERF-Pk improves GSM8K accuracy over S-FLM’s 17.7% baseline to 22.8%, while SERF-L reaches 22.3% and SERF-P 20.2%.With 16 steps, SERF-Pk and SERF-L exceed the S-FLM baseline at 19.9% and 18.7%, a 64× reduction; SERF-P exceeds it with 32 steps at 18.7%.Source: §4.3 · p. 9

4.4 ANALYSIS

• Aligning student layer 2 to teacher layer 10 gives SERF-L the highest MAUVE, while SERF-Pk performs best with student block k = 2.For SERF-L, aligning layer 10 to layer 2 performs worst; for SERF-Pk, k = 6 falls below FLM across the entropy range.Source: §4.4 · p. 10, Figure 7 · p. 10
• At 100K OWT training steps, SERF-Pk’s MAUVE front lies above all other compared models across the entropy range and extends to higher entropies than SERF-P.SERF-Pk applies the teacher’s distribution to an early student block, retaining a distribution-level cross-entropy loss.Source: §4.4 · p. 10, Figure 6 · p. 10
• For SERF-P, the random offset outperforms the baseline, whereas fixed δ = 0.8 brings no improvement; SERF-L performs similarly under both strategies.With δ = 0.8, clipping gives the teacher a fully clean input when s > 0.2, making its predictions nearly one-hot and adding little beyond the ground-truth label for SERF-P.Source: §4.4 · pp. 10–11, Figure 6 · p. 10

5 CONCLUSION

• On OWT, SERF-L and SERF-Pk match or exceed FLM trained for 1M steps with roughly a tenth of the training budget, while gains persist in few-step generation; on GSM8K, SERF improves S-FLM accuracy and outperforms prior continuous diffusion language models.The authors conclude that the denoising trajectory provides a rich source of self-supervision for language.Source: §5 · p. 11

AI USE STATEMENT

• The authors state that all scientific content was conceived and produced by them, while LLMs assisted with language polishing and LaTeX formatting.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

• The authors identify disinformation as a potential societal risk, noting that more realistic generated content may make it more convincing.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The authors report that experiments use publicly available datasets and that code and checkpoints will be released upon publication.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 FLOW LANGUAGE MODELS

• FLM experiments use a 179M-parameter, 12-block DiT and train on OWT with batch size 512, Adam, and a fixed learning rate after warmup.Sampling uses an Euler solver and temperature sweeps to trace a quality–diversity front.Source: §A.1 · p. 16

A.2 EMBEDDED LANGUAGE FLOWS

• ELF experiments use the 105M-parameter ELF-B backbone on T5-small sentence embeddings and train on OWT with batch size 512 using Muon.At inference, sweeping the self-conditioning guidance scale traces a quality–diversity front.Source: §A.2 · p. 16

A.3 HYPERSPHERICAL FLOW LANGUAGE MODELS

• S-FLM uses a 163M-parameter hyperspherical transformer with 12 blocks, and its embeddings and hidden states lie on the unit hypersphere.Source: §A.3 · p. 16
• S-FLM is trained on TinyGSM with batch size 512 and Adam; its normalized architecture provides stability without softcapping.Source: §A.3 · p. 16
• At each sampling step, the S-FLM distribution is truncated to its highest-probability token, and the final state is decoded with argmax.The problem statement remains fixed as clean embeddings, so the flow acts only on the answer.Source: §A.3 · pp. 16–17
• Figure 8 compares Mauve and Gen.PPL fronts for SERF-L and SERF-Pk at 300K steps against baseline FLM at 1M steps, with sampling temperature swept.Source: Figure 8 · p. 17

A.4 ADDITIONAL DETAILS ON SERF

• The EMA teacher uses α = 0.9999 and evaluates the student's same noised sequence at a cleaner time; SERF-L aligns student block 2 with teacher block 10 using a projection head.SERF-P and SERF-Pk distill output distributions, and their loss begins after 10K warm-up steps.Source: §A.4 · p. 17
• Despite making each training step about 30% slower, SERF-L reaches baseline quality 7.7× faster for MAUVE and Gen. PPL and 1.4× faster for GSM8K accuracy.The teacher and SERF-L projection head are used only during training, so sampling adds no cost.Source: §A.4 · p. 17
• Figure 9 presents a regularization-weight ablation and training convergence for SERF variants against S-FLM, evaluated on GSM8K accuracy with 32 sampling steps.Source: Figure 9 · p. 19
• GSM8K evaluation covers all test problems, holds the clean problem statement fixed, and reports the mean over five noise seeds.Source: §A.4 · p. 19
• The GSM8K comparison includes discrete and continuous diffusion models, while baseline and autoregressive results are taken from prior papers.Source: §A.4 · p. 19
• Training and sampling use eight NVIDIA GH200 96GB GPUs.Source: §A.4 · p. 19

A.5 ADDITIONAL EVALUATION DETAILS

• Gen. PPL is interpolated to target entropy H = 5.44, matching the entropy of OpenWebText validation text, using samples from a sweep of three temperatures.For each temperature, the procedure records Gen. PPL and mean per-sequence unigram entropy from 5,000 samples.Source: §A.5 · p. 19

B.1 TRAINING ACCELERATION ON TINYGSM

• 1.8× fewer training steps let SERF-L reach the released S-FLM checkpoint's 16.3% GSM8K accuracy at 140K steps, with 16.2% accuracy.SERF-P reaches the checkpoint's 16.3% accuracy after 180K steps, or 1.4× fewer training steps.Source: §B.1 · p. 19
• At 250K steps, SERF-L reaches 20.8% GSM8K accuracy and SERF-P reaches 18.7%, respectively 4.5 and 2.4 points above S-FLM.Accuracy is measured with 32 sampling steps and averaged over 4 evaluation seeds.Source: §B.1 · p. 19
• After roughly 30% per-step overhead, SERF-L still matches the baseline 1.4× faster in training time.Both SERF variants learn faster than the S-FLM baseline.Source: §B.1 · p. 19

B.2 EFFECT OF THE SELF-REGULARIZATION WEIGHT λ.

• SERF-L MAUVE rises from 0.321 at λ = 1.0 to 0.336 at λ = 2.0, while SERF-P falls from 0.320 at λ = 0.1 to 0.259 at λ = 0.5.The variants respond in opposite directions to λ.Source: §B.2 · pp. 19–20
• The different weight responses reflect loss scales: SERF-L's bounded, smaller cosine distance needs larger λ, whereas SERF-P's cross-entropy matches LFLM's scale.Increasing λ for SERF-P shifts the effective target away from ground truth toward the teacher prediction.Source: §B.2 · p. 20

C QUALITATIVE COMPARISONS

• With the same initial noise and 1024 sampling steps, SERF samples are more coherent and grammatical, while FLM more often produces degenerate fragments, repeated symbols, and locally incoherent phrases.The comparisons vary only the trained denoiser, and Euler sampling is deterministic given the initial noise.Source: C QUALITATIVE COMPARISONS · p. 20

D LIMITATIONS AND FUTURE WORK

• SERF is currently developed and evaluated for continuous flow-based language models.The paper identifies extending the framework to discrete diffusion language models as future work.Source: D LIMITATIONS AND FUTURE WORK · p. 20
• Discrete diffusion has different corruption and denoising dynamics than continuous flows, so adapting teacher supervision may require additional design considerations.The authors present extending SERF to discrete diffusion as a promising direction for broadening its applicability.Source: D LIMITATIONS AND FUTURE WORK · p. 20