Source-linked AI summary
Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers
Yuhong Chou, Zehao Liu, Yuqi Pan, Qian Liu, Xianwei Chen, Zhongzhu Zhou, Jibin Wu
TL;DR
Long-context efficient attention can reduce computational complexity but degrade in-context recall, while hybrid approaches retain a quadratic bottleneck. Orthros assigns different attention computations to prefilling and decoding; across evaluations, it reports competitive performance and faster prefilling, including a 21.9× speedup at one million tokens.
Problem
Linear and sliding-window attention can degrade in-context recall, while hybrids that restore recall with full attention retain a quadratic complexity bottleneck.
Source: §1 · pp. 1–2Method
Orthros unifies sliding-window attention across the sequence with additional full attention at next-token prediction positions, using different computation patterns for prefilling and decoding.
Source: §1 · p. 2Results
Takeaways & Limitations
Orthros is presented as a compelling alternative to strong Transformer baselines, particularly for prefill-heavy applications.
Source: §5 · p. 12Orthros requires computing a doubled sequence during training, and its KV-cache memory efficiency remains unaddressed.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling linear complexity prefilling and full attention decoding. Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding. Beyond theoretical formulation and complexity analysis, extensive empirical evaluations demonstrate Orthros's superiority across multiple dimensions. First, we validate its fundamental language modeling capabilities, demonstrating scalability across varying model sizes and strong competitiveness in comprehensive architectural comparisons. Second, Orthros exhibits exceptional conversion feasibility via a data-efficient Transformer-to-Orthros conversion, yielding performance that even surpasses the original Transformer. Finally, Orthros achieves a $21.9\times$ prefilling speedup over Transformers with end-to-end inference gains, while preserving long-context capabilities, as demonstrated by near-perfect 128K needle-in-a-haystack results and Transformer-level performance on real-world in-context recall tasks. Overall, these advantages in high performance, flexibility, and efficiency position Orthros as a promising candidate to succeed the Transformer in industrial applications.
1. Introduction
Long-context attention must balance efficient computation with in-context recall: linear and sliding-window methods can degrade recall, while hybrids retain a quadratic bottleneck. Orthros decouples computation by sequence phase and reports competitive performance alongside faster prefilling.
1. Introduction
- Hybrid Transformers retain a quadratic bottleneck because full attention remains their most computationally intensive component.Linear and sliding-window attention reduce complexity but can degrade in-context recall, motivating hybrids that mix efficient and full attention.Source: §1 · pp. 1–2
- Orthros applies sliding-window attention across the sequence and additional full attention at next-token prediction positions, enabling linear-complexity prefilling and full-attention decoding.The heterogeneous computations are unified in one module, preserving training–inference consistency without broader architectural changes.Source: §1 · p. 2
- 21.9× speedup over Transformers in prefilling at one million tokens accompanies end-to-end inference gains and competitive or superior performance across the reported evaluations.The experiments also report near-perfect retrieval on the 128K needle-in-a-haystack test and Transformer-level performance on in-context recall tasks.Source: §1 · p. 2
2. Related Works
The related work contrasts efficient attention methods with architectures that combine them with full attention or share key-value caches across layers. It describes their complexity and flexibility trade-offs.
2.1. Efficient Attention
- Standard causal self-attention incurs O(T^2) computation, whereas linear and sliding-window attention bound per-step overhead and achieve O(T) full-sequence complexity.Linear attention uses a kernel feature map and recursive state computation; sliding-window attention restricts the receptive field to a local window.Source: §2.1 · p. 3
2.2. Cross-Layer Attention and Decoder-Decoder Architectures
- The decoder-decoder framework uses a shared KV cache for its cross-decoder, which can skip computation before generation while retaining a global decoding receptive field.Its extreme sharing may compromise performance, and adding state-maintaining attention to the cross-decoder breaks early exit before generation.Source: §2.2 · pp. 3–4
3. Method
Orthros separates local context assimilation from output prediction: sliding-window attention processes all tokens, while full cross-attention serves output positions. Training and inference share one formulation, and complexity improves when output length is small relative to context.
3.1. Orthros attention
- Orthros applies sliding-window attention across all tokens and full cross-attention to the materialized KV cache only at output-generation positions.Prefilling invokes full cross-attention only at the final prompt token; decoding pairs cache-updating sliding-window attention with output prediction by full cross-attention.Source: §3.1 · p. 4
3.2. Training & Inference Schemes
- Pre-training, SFT, prefilling, and decoding share one architecture, instantiated by changing Tin and Tout without structural modifications.Pre-training sets Tin=0 and Tout=T; SFT assigns Tout to response tokens; prefilling sets Tout=1; decoding sets Tin=0 and Tout=1.Source: §3.2 · pp. 6–7
3.3. Complexity Analysis
- O(ToutTdmodel) yields subquadratic complexity when Tout ≪ T, while Tout=O(1) makes the model strictly linear at O(Tdmodel^2).Only when Tout≈O(T), as in pre-training, does complexity return to vanilla Transformer's O(T^2dmodel).Source: §3.3 · p. 7, §3.2 · p. 7
- Although decoding processes a decoupled pair of tokens per step, its added arithmetic has negligible latency because weight loading dominates memory-I/O-bound decoding.In batched decoding, increasing active compute tokens can help the system move beyond the I/O bottleneck and inefficient GPU utilization.Source: §3.3 · p. 7
4. Experiments
Experiments assess Orthros across scaling, training, conversion, recall, and inference efficiency. Results report competitive model performance alongside reduced long-context prefilling latency.
4. Experiments
- Up to 21.9× prefilling speedup over Transformers accompanies end-to-end inference gains, while doubled decoding FLOPs are masked by decoding's I/O-bound nature.The experiments also report near-perfect 128K needle-in-a-haystack accuracy and competitive or superior benchmark performance.Source: §4 · p. 8
4.1. Scaling and Ablation Study
- Orthros scales competitively against Transformer baselines, while scale-offset provides a better parameter-efficiency/performance trade-off.The ablation compares the Transformer baseline with Orthros without scale-offset.Source: §4.1 · p. 8, Figure 2 · p. 8
4.2. Training from Scratch
- At 1.3B scale, Orthros achieves the best average Common Sense Reasoning performance among the compared architectures while remaining comparable across individual tasks.Models were trained from scratch on 100B tokens; Table 1 reports the 1.3B-scale evaluation.Source: §4.2 · p. 9, Table 1 · p. 9
4.3. Conversion Training
- For both Llama3.1-8B and Qwen3-8B, Orthros Conversion beats Base and same-architecture Base CPT on average.The comparison uses the same 80B-token continual-training budget for converted and Base CPT models.Source: §4.3 · pp. 9–10, Table 2 · p. 9
4.4. In-Context Recall Capacity
- On 1.3B in-context recall tasks, Orthros has the best average performance among architectures and is comparable with full attention on each task.The evaluated suite includes DROP, NarrativeQA, SWDE, TriviaQA, FDA, and SQuAD.Source: §4.4 · p. 10, Table 3 · p. 10
- After 20B-token long-context extension training, the converted 8B Orthros maintains near-perfect retrieval accuracy at 128K context length.The extension uses ProLong data and the Qwen3-converted Orthros model.Source: §4.4 · p. 10, Figure 3 · p. 10
4.5. Prefilling Efficiency
- Orthros has lower prefilling latency than Qwen3 across single-GPU 8K–128K and 8-GPU 64K–1M input lengths, with a larger advantage at longer sequences.The comparison uses high-performance FLA-style models and kernels.Source: §4.5 · p. 11, Figure 4 · p. 11
4.6. End-to-End Inference
- Orthros and Orthros-H reduce prefilling latency against their respective baselines while retaining comparable decode-only throughput; the end-to-end advantage shrinks as outputs lengthen.Faster prefilling yields lower end-to-end latency and higher output throughput, particularly for short outputs.Source: §4.6 · pp. 11–12, Table 4 · p. 11
5. Discussion, Limitation and Future Work
Orthros reports inference-efficiency gains without compromising downstream performance or long-context capabilities, but obtains these gains at the cost of doubled token computation during training. The discussion calls for evaluation that accounts for training cost, phase-specific inference cost, model quality, workload lengths, and deployment scale.
5. Discussion, Limitation and Future Work
- Orthros doubles token computation during training to maintain strict training–inference consistency, while the fairness of comparing this redistribution with uniform Transformer computation remains open.The authors argue that scaling analyses focused on training FLOPs and loss can undervalue architectures that trade extra training computation for sustained inference savings.Source: §5 · p. 12
- A broader evaluation framework should make comparison criteria explicit and assess training cost, phase-specific inference cost, model quality, input and output lengths, and deployment scale.The authors say this would support a more comprehensive assessment of computational efficiency.Source: §5 · p. 12
- Remaining work includes addressing KV-cache memory efficiency and adapting Orthros for Mixture-of-Experts routing, agentic reinforcement learning, and diverse inference settings.The paper also notes that Orthros currently requires computing a doubled sequence during training.Source: §5 · p. 12
A. Experiment Settings
The reported experiments use H100 GPUs with 80GB memory. The settings cover scaling and ablation, Transformer-to-Orthros conversion, and long-context and multi-GPU evaluations.
A. Experiment Settings
- Scaling and ablation models use sequence length 2048 for approximately 10B tokens, with Orthros using a sliding window of 128.All training was conducted on H100 GPUs with 80GB memory.Source: Table 5 · p. 19, A. Experiment Settings · p. 19
- In-context recall-intensive benchmarks used an 8B Orthros converted from Qwen3-8B; the needle-in-a-haystack task used training on 20B long context, and multi-GPU experiments enabled sequence parallelism.Source: A. Experiment Settings · p. 19
B. Experiment Metrics
The appendix reports ablation results in Table 7 and identifies the accuracy measures used across its tasks. It also describes the configurations compared in Transformer-to-Orthros conversion.
B. Experiment Metrics
- The conversion comparison includes the original pretrained checkpoint without additional training, Orthros Conversion, and Base CPT under the same training setting.Orthros Conversion replaces the original full-attention modules with Orthros Attention.Source: Table 6 · p. 20
- Table 7 reports normalized accuracy for ARC-Challenge, ARC-Easy, HellaSwag, OpenBookQA, and PIQA, plus accuracy for WinoGrande.The table covers ablations across different model scales.Source: B. Experiment Metrics · p. 19, Table 7 · p. 20