Source-linked AI summary
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Ilya Koziev, Leonid Sinev, Ivan Oseledets
TL;DR
Orthrus claims lossless speculative decoding by combining a frozen autoregressive backbone with a parallel diffusion view, but the practical status of this claim under finite precision is unclear. This study independently reproduces Orthrus and compares trajectories and downstream performance across precisions, finding BF16 divergence but exact FP32 matching on all evaluated prompts. The results motivate evaluating exact trajectory equivalence separately from benchmark performance and specifying the precision used for losslessness claims.
Problem
Autoregressive decoding is inherently sequential, while Orthrus’s losslessness claim requires determining whether parallel decoding reproduces the autoregressive model’s exact output sequence in practice.
Method
The study independently implements Orthrus and evaluates its generated trajectories against corresponding frozen autoregressive trajectories across numerical precisions and downstream benchmarks.
Results
Under BF16, exact trajectories matched in 45% of released-checkpoint cases and 43% of independently trained-model cases, whereas FP32 matched all 1,190 evaluated prompts without systematic downstream benchmark degradation.
Takeaways & Limitations
Practical losslessness depends on numerical precision, so exact trajectory equivalence should be evaluated separately from downstream task performance.
Takeaways & Limitations
The specific computational stages responsible for precision-dependent deviations remain unidentified.
Abstract
from arXiv · showhide
Orthrus is a hybrid autoregressive-diffusion architecture that accelerates autoregressive language-model inference by generating multiple tokens in parallel while using a frozen autoregressive backbone. Its central claim is that an intra-model consensus mechanism enables lossless speculative decoding, producing the same output sequence as the autoregressive model. We independently reproduce Orthrus and examine this claim under different numerical precisions. Under BF16 inference, exact trajectory matching occurs in only 45% of cases for the authors' checkpoint and 43% for our independently trained model across 1,190 prompts from 12 domains. The probability of exact matching is also strongly associated with the response-conditional perplexity of the reference model. Despite this trajectory divergence, Orthrus does not show systematic degradation on downstream lm-eval-harness benchmarks. In contrast, repeating the trajectory evaluation with FP32 yields exact trajectory matching on all evaluated prompts. These results show that the practical losslessness of Orthrus depends on numerical precision and that exact trajectory equivalence should be evaluated separately from downstream task performance.
1. Introduction
Orthrus accelerates autoregressive decoding with a diffusion view over a frozen backbone, but this paper tests whether its claimed losslessness survives practical finite-precision inference. The study finds trajectory equivalence depends on numerical precision even when downstream benchmark performance does not systematically decline.
- Approach: Orthrus augments a frozen autoregressive backbone with a lightweight diffusion view to generate multiple future tokens in parallel.The AR component constructs the context representation while the diffusion component supports parallel decoding.
- Numerical precision: Finite-precision arithmetic can cause small numerical differences to change a discrete token choice and eventually produce divergent trajectories.The consensus mechanism can provide exact equivalence under idealized arithmetic, but need not reproduce the reference computation bit-for-bit in implementation.
- Numerical precision: BF16 produced non-zero output divergence from the corresponding frozen AR model, despite Orthrus being characterized as strictly lossless.The paper argues that losslessness requires a more precise operational definition for finite-precision systems with stateful KV caches.
- Downstream evaluation: Numerical differences did not necessarily degrade standard task-level evaluation, and Orthrus sometimes achieved slightly higher benchmark scores than the corresponding autoregressive model.Benchmark equality therefore cannot establish exact inference equivalence because systems may score similarly while generating different token sequences.
- Study design: The study independently implements Orthrus training and inference and evaluates the reproduction alongside the released checkpoint.The authors also investigate training objectives, data distributions, and hyperparameters using a configurable framework.
2. Training Orthrus
The independent Orthrus model is trained with teacher-generated, on-policy distillation data and evaluated across 12 text domains. It achieves slightly higher Tokens Per Forward than the released checkpoint in 10 of 12 domains, while showing qualitatively similar behavior.
- 2.1 Training setup: The independently trained model differs substantially from the released checkpoint in both training-data composition and hyperparameters.These differences allow the authors to assess whether observed effects depend on the released checkpoint’s specific training setup.
- 2.1 Training data: The independently trained model uses greedy responses generated by Qwen/Qwen3-1.7B from prompts drawn from publicly available HuggingFace datasets.This constructs a teacher-generated distillation corpus aligned with the frozen autoregressive model’s own prediction trajectories.
- 2.3 Evaluation: Evaluations use 12 text domains with 100 prompts per domain except gec-en, which contains 90 prompts.The domain breakdown is intended to assess variation in generation trajectories and their statistical properties.
- 2.3 Evaluation: 10 of 12 evaluation domains favored the independently trained model on Tokens Per Forward, while the released checkpoint performed better in the remaining two.The differences may reflect the models’ substantially different training data and configurations.
- 2.3 Evaluation: The independent model exhibits qualitatively similar behavior to the released checkpoint while achieving competitive or higher Tokens Per Forward in most domains.The study next evaluates the stronger claim of exact trajectory equivalence.
3. When Lossless Decoding Is Not Lossless
Under BF16 greedy decoding, Orthrus trajectories do not consistently match the autoregressive reference exactly, and divergence is associated with higher reference-model response-conditional perplexity.
- Higher response-conditional perplexity was strongly associated with a lower probability of exact trajectory matching after controlling for response length and domain.The estimated log-PPL coefficients were β1 = −8.10 for chiennv/Orthrus-Qwen3-1.7B and β1 = −10.92 for Orthrus-1.7B-final.
4. When Losses Become Gains
Trajectory divergence did not produce systematic degradation on the evaluated lm-eval-harness tasks, and the independently trained Orthrus model had higher point estimates than Qwen3 on all three benchmarks.
- The independently trained Orthrus model had higher point estimates than Qwen3 on GSM8K, HumanEval, and IFEval.The reported differences were not interpreted as statistically significant improvements because of uncertainty.
- Trajectory deviations did not consistently have a negative effect and did not imply systematic degradation in downstream task performance.Some experiments showed benchmark scores slightly higher than those of the corresponding autoregressive model.
5. The Effect of Numerical Precision
Repeating the trajectory evaluation in FP32 removed the BF16 mismatch observed for Orthrus, showing that practical trajectory equivalence depends on numerical precision.
- 100% of the 1,190 evaluated prompts produced exactly matching Orthrus and autoregressive trajectories under FP32.The model parameters, decoding procedure, and evaluation prompts were held unchanged from the BF16 comparison.
- The authors attribute BF16 trajectory divergence to finite-precision numerical effects rather than to a specified layer or computational operation.Identifying the responsible computational stages remains future work.
- The precision question concerns whether accelerated Orthrus inference reproduces the exact autoregressive reference trajectory, not merely whether inference is reproducible across configurations.This distinguishes lossless speculative decoding from broader inference reproducibility studies.
6. Conclusion
Direct trajectory comparisons show that Orthrus’s practical losslessness is precision-dependent: BF16 often diverges, whereas FP32 matched the reference on all evaluated prompts without systematic downstream degradation.
- Under BF16, exact trajectory matching occurred for 45% of the released-checkpoint cases and 43% of the independently trained-model cases.Matching was strongly associated with reference-model response-conditional perplexity.
- FP32 produced exact trajectory matching on all 1,190 evaluated prompts, while BF16 divergences did not cause systematic degradation on evaluated downstream benchmarks.The conclusion recommends specifying numerical precision and the operational criterion used to assess equivalence.