Source-linked AI summary
Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion
Chien Van Nguyen, Chaitra Hegde, Van Cuong Pham, Ryan A. Rossi, Franck Dernoncourt, Thien Huu Nguyen
TL;DR
Autoregressive decoding preserves exact causal fidelity but is sequential, while diffusion models offer parallel generation with distributional and training-cost limitations. Orthrus combines a frozen AR backbone with a lightweight diffusion view over a shared KV cache and uses consensus to retain exactness. It reports lossless inference, up to 7.8× speedup, and lightweight adaptation.
Problem
Autoregressive decoding is sequential, while diffusion models trade exact causal conditioning for parallelism and can require costly training while drifting from the AR distribution.
Method
Orthrus augments a frozen pre-trained AR Transformer with a lightweight trainable diffusion module that shares the AR-generated KV cache and uses intra-model consensus.
Results
Up to 7.8× speedup is reported with lossless inference parity, zero redundant historical KV-cache overhead, and 16% of model parameters trained using less than 1B tokens.
Takeaways & Limitations
Orthrus provides parallel token generation while preserving the exact predictive distribution of the base autoregressive model.
Takeaways & Limitations
Highly accelerated diffusion inference relies on a strong conditional-independence assumption, which can cause distributional drift from the autoregressive target.
Abstract
from arXiv · showhide
We introduce Orthrus, a simple and efficient dual-architecture framework that unifies the exact generation fidelity of autoregressive Large Language Models (LLMs) with the high-speed parallel token generation of diffusion models. The sequential nature of standard autoregressive decoding represents a fundamental bottleneck for high-throughput inference. While diffusion language models attempt to break this barrier via parallel generation, they suffer from significant performance degradation, high training costs, and a lack of rigorous convergence guarantees. Orthrus resolves this dichotomy natively. Designed to seamlessly integrate into existing Transformers, the framework augments a frozen LLM with a lightweight, trainable module to create a parallel diffusion view alongside the standard autoregressive view. In this unified system, both views attend to the exact same high-fidelity Key-Value (KV) cache; the autoregressive head executes context pre-filling to construct accurate KV representations, while the diffusion head executes parallel generation. By employing an exact consensus mechanism between the two views, Orthrus guarantees lossless inference, delivering up to a 7.8x speedup with only an O(1) memory cache overhead and minimal parameter additions.
1 Introduction
Orthrus unifies autoregressive fidelity with diffusion-style parallel generation in one Transformer. It uses a shared KV cache and consensus mechanism to preserve the base model’s predictive distribution while reducing decoding cost.
- Motivation: Autoregressive decoding generates tokens sequentially, creating a memory-bandwidth bottleneck, whereas diffusion models generate token blocks in parallel but often underperform comparable AR models.Diffusion adaptations can also require up to 500B tokens of continuous pre-training and fail to match the original AR predictive distribution.
- Framework: Orthrus freezes the pre-trained AR model for parallel context pre-filling and adds a lightweight trainable diffusion module alongside the AR attention heads.The AR view constructs a high-fidelity KV cache, while the diffusion view uses it for parallel generation.
- Framework: An intra-model consensus mechanism validates diffusion-generated token trajectories against the frozen AR view, preserving the base LLM’s exact predictive distribution.The shared context allows both views to operate over the same KV cache without redundant historical storage.
- Results: 7.8× speedup is achieved by using the diffusion head for parallel token generation, breaking the sequential decoding bottleneck.The reported result is presented as a contribution of Orthrus’s parallel generation design.
- Results: 16% of total model parameters are fine-tuned using less than 1B tokens, with training requiring under 24 hours on a single 8xH200 node.These figures characterize the framework’s parameter and training efficiency.
2 Preliminaries
The preliminaries contrast autoregressive exact causal conditioning with diffusion’s parallel denoising. Orthrus addresses the resulting fidelity–parallelism tension without permanently converting the base AR model.
- Autoregressive Paradigm: Autoregressive language modeling factorizes sequence probability by the chain rule, conditioning each token on the entire preceding trajectory.This exact causal conditioning requires sequential sampling and repeated KV-cache loading during inference.
- Autoregressive Paradigm: Autoregressive inference requires N distinct forward passes to generate N tokens, producing a memory-bandwidth-bound bottleneck.The bottleneck arises because the KV cache is repeatedly loaded during sequential generation.
- Diffusion Paradigm: Diffusion language models frame generation as parallel denoising, predicting masked future tokens simultaneously from historical context and a corrupted token block.The masked positions are represented by M, while the reverse process predicts the original tokens.
- Diffusion Paradigm: Highly accelerated diffusion inference relies on a strong conditional-independence assumption, and its predictions can drift from the true autoregressive distribution.The drift occurs because later predictions do not condition on the exact realized preceding token.
- Structural Unification: Adapting pre-trained AR models to diffusion changes the base model and introduces performance trade-offs, motivating Orthrus’s structural unification.Orthrus grounds parallel generation in frozen, high-fidelity AR representations instead of permanently converting the model.
3 Methodology: The Orthrus Architecture
Orthrus combines a frozen autoregressive view for high-fidelity context encoding with a trainable diffusion view for parallel generation, using one shared KV cache. Its training and inference procedures align parallel predictions with the autoregressive model while preventing information leakage and synchronizing accepted tokens.
- Unified Dual-View Attention Mechanism: Orthrus augments a standard autoregressive Transformer with a lightweight diffusion attention module, creating parallel AR and diffusion views.The diffusion projections are initialized from frozen AR counterparts, while the AR backbone remains frozen.
- Unified Dual-View Attention Mechanism: The frozen AR backbone prefills the context into causal KV representations, which the diffusion view reuses without additional historical cache storage.The shared cache supports both attention paths and eliminates redundant historical KV memory.
- Training: Dual-Pass Block Masking: During training, each block retains its first token as an anchor and replaces the remaining K−1 positions with <mask> tokens for simultaneous diffusion processing.Corrupted blocks are processed against the frozen AR KV cache using a specialized dual-pass attention mask.
- Training: Dual-Pass Block Masking: The diffusion mask gives each block causal access to preceding clean context while allowing bidirectional attention within the same masked block.This routing prevents future leakage while enabling parallel context aggregation across the block.
- Training: Dual-Pass Block Masking: Training minimizes forward KL divergence between diffusion predictions and the frozen AR model’s full predictive distributions over masked positions.Only the diffusion module receives gradients; the AR backbone stays strictly frozen.
- Inference: Exact Distribution Matching via Intra-Model Consensus: At inference, the diffusion view projects K candidate tokens in parallel, after which the frozen AR view computes exact target probabilities and synchronizes the trajectory left to right.Matching tokens are retained, while divergence triggers a causal correction and shared-cache truncation.
4 Experiments
Orthrus is evaluated as a dual-view architecture for efficient, lossless parallel generation across model scales, reasoning tasks, diffusion baselines, and speculative decoding methods. Experiments report substantial acceleration, exact parity with the frozen autoregressive baseline, shared-cache efficiency, and stable long-context performance.
- Experimental Setup: Orthrus evaluates frozen Qwen3 1.7B, 4B, and 8B backbones across mathematical reasoning, coding, and other zero-shot algorithmic tasks.Only the injected diffusion attention module is optimized; the original autoregressive backbone remains frozen.
- Efficiency and Fidelity: Orthrus reports an average TPF of 5.39 at the 8B scale while guaranteeing exact distributional parity with the autoregressive baseline.The reported guarantee is described as strictly lossless acceleration rather than a quality–speed trade-off.
- Comparison with Diffusion Models: Orthrus matches Qwen3-8B’s exact zero-shot accuracy, whereas diffusion baselines consistently lag in accuracy and SDAR degrades despite sharing the same foundation model.Orthrus fine-tunes 16% of model parameters on less than 1B tokens, requiring under 24 hours on one 8xH200 node.
- Comparison with Speculative Decoding: Orthrus delivers up to 6× speedup with strictly lossless MATH-500 performance, while Fast-dLLM-v2 incurs an 11.1-point accuracy degradation relative to its autoregressive baseline.The passage attributes Fast-dLLM-v2’s limited practical acceleration to iterative refinement steps.
- Memory Efficiency: Orthrus uses an intra-model diffusion head over the autoregressive head’s shared KV cache, eliminating redundant drafter caches and external speculative models.This unified design is presented as native parallel acceleration with zero redundant cache overhead.
- Scaling and Stability: Orthrus reaches an acceptance length of 11.7 on MATH-500 and maintains near-peak throughput at context lengths up to 40K.The reported acceptance length exceeds DFlash’s 7.9 and EAGLE-3’s 3.5; long-context throughput remains near-constant while DFlash deteriorates.
5 Ablation Study
The ablation study examines how the parallel block size and denoising strategy affect Orthrus throughput and latency. Larger blocks improve throughput without latency growth, whereas iterative denoising reduces throughput.
- Parallel Block Size: Scaling the parallel block size to K = 32 raises TPF to 6.35 and produces a 3.6× throughput multiplier over K = 4 with zero latency penalty.The diffusion view processes the extended block simultaneously against a pre-computed KV cache, maintaining constant forward-pass latency.
- Multi-Step Denoising: The 2-step prediction strategy reduces TPF by 1.8×, confirming single-step projection as the optimal denoising configuration for Orthrus.The additional sequential forward pass required for iterative refinement degrades throughput.
6 Conclusion
Orthrus reconciles autoregressive fidelity with diffusion-based parallelism through a unified architecture. The framework reports faster generation while preserving inference parity and avoiding redundant memory overhead.
- Orthrus unifies autoregressive generation fidelity with diffusion-based parallel token generation in one framework.
- The framework uses a frozen autoregressive backbone and a lightweight trainable diffusion module with a shared high-fidelity KV cache.
- Orthrus employs intra-model consensus to guarantee lossless inference parity with the base model.
- Orthrus delivers up to a 7.8× speedup across diverse mathematical and structural benchmarks with zero redundant memory overhead.
A Training Details
Orthrus training isolates the diffusion head while keeping the autoregressive backbone frozen. The pipeline uses balanced multi-domain distillation data and distributed optimization settings.
- Training isolates the diffusion head while keeping the autoregressive backbone strictly frozen.
- The training corpus samples Nemotron-Post-Training-Dataset-v2 and balances Mathematical Reasoning, Code Generation, and General Chat & Instruction Tuning.
- Training uses PyTorch FSDP-2 on 8 GPUs with an effective global batch size of 128 sequences per optimization step.
- The training setup uses bfloat16 diffusion parameters and runs for 2 epochs with a cosine learning rate scheduler.
B Analysis
The analysis examines distillation objectives and memory scaling. Soft distributional distillation supports the diffusion view, while shared caching keeps additional KV memory constant across sequence lengths.
- Orthrus trains the diffusion view with forward KL divergence to distill the autoregressive teacher’s full predictive distribution.
- Hard-label training causes structural misalignment that increases consensus rejections and reduces Effective Tokens Per Forward Pass from 6.35 to 5.86.
- Orthrus maintains a strictly constant O(1) KV cache overhead, with a fixed ∆≈4.5 MiB increase regardless of sequence length.
- Figure 6 reports a peak GPU memory overhead below 1% versus the Qwen3-8B baseline.
C Limitation
Orthrus’s generative capabilities are bounded by its frozen foundation model. It inherits the underlying model’s biases, knowledge gaps, and hallucination tendencies.
- Because the autoregressive backbone is frozen, Orthrus’s generative capabilities remain strictly upper-bounded by the foundation model.
- The framework inherently inherits biases, knowledge gaps, and hallucination tendencies from the underlying base model.