PaperlayerView exact revision on arXiv

Source-linked AI summary

SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

Junsong Chen, Jincheng Yu, Yitong Li, Shuchen Xue, Haozhe Liu, Jingyu Xin, Yuyang Zhao, Tian Ye, Zhangjie Wu, Zian Wang, Daquan Zhou, Ping Luo, Song Han, Enze Xie

arXiv:2607.21553v1cs.CV

TL;DR

Video generation faces costly quadratic attention and limited expressiveness in efficient linear-attention models. SANA-Video 2.0 combines mostly linear attention with periodic softmax anchors and attention residuals, achieving competitive quality with substantially lower latency, including a 3.2× faster DiT forward than full softmax at 720p/60s.

  • Problem

    Full-softmax video attention has punishing O(N^2) cost, while pure linear attention scales efficiently but cannot represent every token–token interaction.

  • Method

    SANA-Video 2.0 combines 75% gated linear attention, 25% periodic softmax anchors, and Block Attention Residuals in a unified video diffusion transformer.

  • Results

    VBench Total 84.30 is achieved in 13.2s at 480p on one H100, while the DiT forward is 3.2× faster than full softmax at 720p/60s.

  • Takeaways & Limitations

    The hybrid backbone provides a practical, efficient foundation for high-quality video generation with an efficiency advantage that scales better with video duration.

  • Takeaways & Limitations

    Training currently extends only to an 8s horizon, leaving minute-scale long-video generation and long-context global consistency for future work.

Abstract

from arXiv · show

We introduce SANA-Video 2.0, a hybrid video diffusion transformer instantiated at 5B and 14B scales under a unified architecture. Designed to generate high-quality video up to 720p on a single GPU, SANA-Video 2.0 matches full-softmax video DiTs in quality while retaining the favorable long-sequence scaling of linear attention. To avoid quadratic attention throughout, Hybrid Linear-Softmax Attention combines gated linear attention for O(N)-dominated mixing with periodic gated-softmax anchors at a 3:1 ratio, restoring the full-rank token interactions that pure linear attention lacks. To propagate these refreshed representations across depth, Block Attention Residuals (AttnRes) route completed block summaries into later linear layers, enabling anchor-feature reuse and boosting deep-layer effective rank by ~12%. Through from-scratch training, SANA-Video 2.0 learns the complete hybrid directly rather than linearizing pretrained models, with reduced-resolution proxy studies establishing 25% softmax as the optimal quality-efficiency trade-off. With 40-step sampling, SANA-Video 2.0 achieves a VBench score of 84.30 in 13.2s at 480p on a single H100, remaining competitive with far larger softmax video DiTs at a fraction of the latency. Its compiled DiT forward pass is 3.2x faster than a matched full-softmax baseline at 720p/60s, a gap that expands with video duration. Furthermore, full-stack Sol-Engine optimization (kernel fusion, caching, and sparse attention) accelerates this hardware-friendly backbone by a further 3.58x, bringing the 5B pipeline to 13.06s at 720p/5s and making it 120x faster than Wan 2.2-A14B on one H100. Overall, our hybrid design recovers softmax-level expressiveness at substantially reduced cost, unlocking scalable long, high resolution video generation.

1. Introduction

SANA-Video 2.0 addresses the quadratic cost and limited expressiveness of video attention with a hybrid linear-softmax architecture and attention residuals. The resulting models retain long-sequence efficiency while achieving competitive quality and substantial deployment speedups.

  • Motivation: O(N^2) full 3D softmax attention becomes prohibitively costly for long video sequences, motivating more scalable alternatives.Linear attention reduces complexity to O(N), but pure linear attention cannot represent every token–token interaction.
  • Results: 84.30 VBench Total is achieved with 40-step sampling while matching strong full-softmax Video DiTs in quality on a single GPU.SANA-Video 2.0 is instantiated at 5B and 14B scales, including a 14.25B-parameter larger backbone.
  • Hybrid Linear-Softmax Attention: 25% softmax anchors, interleaved with gated linear attention at a 3:1 ratio, restore richer token interactions while preserving O(N)-dominated scaling.Reduced-resolution proxy studies identify this anchor fraction as the practical quality–efficiency knee.
  • Block Attention Residuals: ∼12% higher deep-layer state effective rank results when Block Attention Residuals reuse anchor-refreshed summaries across depth.AttnRes avoids forcing later layers to re-derive information computed upstream and adapts routing for bidirectional video diffusion.
  • Deployment Results: 3.58× end-to-end speedup is measured for a separate 50-step Sol-Engine deployment on B200, alongside competitive VBench quality and better duration scaling.The backbone is O(N)-dominated, and quantization-aware training matches BF16 quality with MXFP4 weights and MXFP8 activations.

2. Preliminaries

The preliminaries define video DiTs trained with conditional optimal-transport flow matching and motivate efficient long-sequence processing. They then introduce gated linear attention, periodic softmax anchors, and Block Attention Residuals as the architectural foundations of the hybrid stack.

  • Video DiTs and flow matching: Video DiTs train with conditional optimal-transport flow matching, using timestep-modulated attention and feed-forward blocks over latent sequences spanning tens of thousands of sites at 1080p.The latent trajectory is z_t = (1−t)z + tε, with t sampled from a logit-normal distribution and velocity target ε−z.
  • Gated linear attention: Gated linear attention compresses token interactions into a fixed-size state with O(Nd^2) sequence complexity, trading efficiency for a rank bottleneck.The operator is bidirectional and gated rather than a causal delta-rule recurrence.
  • Gated linear attention: Periodic bidirectional softmax anchors provide less rank-constrained token mixing while leaving most sequence computation linear, using a regular 3:1 hybrid layout.The anchors use QK normalization, RoPE, and a sigmoid output gate.
  • Attention Residuals: Block Attention Residuals aggregate completed block summaries and running within-block residuals, exposing softmax-anchor updates to later mostly-linear layers.The video adaptation shares routing queries across depth instead of parameterizing them separately per sublayer.

3. SANA-Video 2.0

SANA-Video 2.0 combines hybrid sequence attention with block residual attention across depth in a video DiT operating on LTX-VAE 2.3 latents with Gemma-2-2B-IT text features. Its sparse softmax anchors refresh token interactions, while AttnRes routes completed block summaries to later layers through shared queries.

  • Hybrid attention: 75% linear and 25% softmax attention are interleaved at a 3:1 ratio, placing a softmax anchor every fourth layer.The linear majority provides O(N) token mixing, while anchors refresh interactions that compressed linear states cannot represent.
  • Block attention residuals: AttnRes exposes completed block summaries containing anchor updates to later self-attention, cross-attention, and FFN layers.The architecture routes representations through attention branches and SwiGLU FFNs before final aggregation.
  • Routing design: One shared routing query per branch is reused at every depth, matching per-layer routing in loss with substantially lower memory.The model uses a shared attention query and a shared FFN query, while depth-dependent source sets preserve nonuniform routing.
  • Block attention residuals: 8-layer blocks accumulate residual updates into partial sums, freeze each completed sum as a block summary, and begin a fresh partial sum.The router retains one summary per finished block while also reading the running partial sum.
  • Routing design: The final design sets timestep offsets to zero because the learned offset is nearly constant and changing or shuffling it has negligible validation effect.Timestep modulation remains in the DiT blocks.

4. Training Recipe

The training recipe combines a shared hybrid-attention flow-matching setup with staged data curation, curriculum scheduling, and preference-based post-training. It also uses noise-aware supervision and stratified validation to improve coverage and diagnose progress across noise regimes.

  • Training Recipe: 5B and 14B models share the hybrid design, while the 5B recipe combines data curation, resolution/duration curriculum, and preference-based post-training.The 14B run uses the same core flow-matching and hybrid-attention formulation with scale-specific pre-training settings.
  • Data Curation: The corpus pipeline segments shots, removes black bars and subtitles, filters defective clips, and scores quality across separate axes.Separate scoring preserves high-quality low-motion content without selecting only visually clean but nearly static clips.
  • Noise Scheduling: TQD routes each clip to the noise regime where its content contributes most while reducing the impact of its weaknesses.It addresses the motion–quality dilemma in scarce high-quality video data by distinguishing motion-rich artifacts from clean but nearly static clips.
  • Validation: Validation uses ten equal 100-step noise buckets instead of a single random-t mean, revealing regime-specific changes hidden by aggregate loss.Across four paired checkpoints, the bucket macro falls by 6.42%, including an 11.44% low-noise improvement.
  • Preference Alignment: Post-training applies offline Gemini-ranked Diffusion-DPO pairs, a preferred-sample flow-matching regularizer, and online ReFL reward optimization.ReFL uses frozen image reward models on the first, middle, and last decoded frames, with joint rewards from HPSv3++, DeQA-Score, and UniPercept.

5. Experiments

Experiments identify a 25% softmax hybrid as the quality–efficiency knee, validate AttnRes routing and rank recovery, and show increasing speed advantages with sequence length and hardware scaling. The resulting model reaches VBench Total 84.30 while improving training rewards across all three logged signals.

  • Generation quality: 84.30 VBench Total is achieved at 480×832×81, with 85.61 Quality, while the 5B model remains competitive with larger video generators.Bernini-R 14B is the only cited model scoring higher on Total and Semantic in the supplied comparison.
  • Attention-ratio ablation: 25% softmax anchors (3:1 ratio) are locked in as the measured proxy Pareto knee for subsequent experiments.Hybrid ratios outperform both all-linear and all-softmax endpoints, while 50% achieves the lowest loss at steep efficiency cost.
  • Sequence and model scaling: 3.17× full/25%-hybrid speedup is reached at 720p and 24fps for the 60s tensor shape under compiled best-kernels comparisons.Across six 480p–1080p shapes, speedup rises from 1.16× to 2.01×; the hybrid is faster at all eight tested model scales.
  • Hardware scaling: 1.98× H100-to-GB200 acceleration reduces one 14B hybrid forward from 789.7ms to 398.3ms at 736×1280×121.The architecture advantage persists on both devices, with full/hybrid speedup increasing from 1.40× to 2.73× as 720p duration grows.
  • AttnRes analysis: 56%/50% completed-block mass is reached by the router in the deepest attention/FFN blocks, demonstrating structured cross-depth reuse.The router favors the nearest completed block, assigning it 26.0%/25.3% attention/FFN mass.
  • Online ReFL: All three frozen reward signals—HPSv3++, DeQA-Score, and UniPercept—rise together across the reported 400-iteration online ReFL run.The trajectories indicate improvement in all three logged components of the joint objective rather than trading one against another.

6. Deployment and Applications

SANA-Video 2.0 is optimized for deployment through Sol-Engine acceleration and low-precision inference, while fine-tuning demonstrates potential for efficiency-sensitive Physical AI applications. QAT preserves BF16 VBench quality while reducing latency, storage, and memory.

  • Kernel-Friendly Deployment with Sol-Engine: Sol-Engine acceleration preserves standard attention and FFN primitives while avoiding the temporal-convolution FFN’s measured 20–29% overhead.The deployment design leaves the attention ratio as the main sequence-scaling variable and uses equally optimized compiled implementations for DiT-forward comparisons.
  • Kernel-Friendly Deployment with Sol-Engine: 3.58× pipeline acceleration is qualitatively compared against the baseline using matched prompts and matched progress through each clip.Figure 8 presents outputs before and after Sol-Engine acceleration.
  • Low-Precision Inference with QAT: 5.8% lower latency, 67.9% lower storage, and 56.9% lower peak memory result from MXFP4 weights and MXFP8 activations with QAT.On an NVIDIA GB200, one CFG-packed 81-frame backbone forward at 832×480 decreases from 203.08 to 191.30ms, while QAT matches the BF16 baseline on VBench Total.
  • Physical AI: Fine-tuning on roughly 5,000 hours of real-robot and egocentric videos produces realistic and physically plausible robot-manipulation videos for Physical AI scenarios.The fine-tuning uses 100k iterations at a learning rate of 1 × 10−4, and the presented comparison reports outperforming similar-sized Cosmos3-Edge (4B).

7. Related Work

Related video generators primarily use quadratic softmax token mixing, while other approaches reduce sequence costs through temporal chunking, latent compression, linear attention, or state-space models. These efficient alternatives trade computation for compressed representations that can restrict rank.

  • Most open video generators use diffusion transformers with quadratic softmax token mixing, including Wan 2.1/2.2, HunyuanVideo, CogVideoX, and Open-Sora.
  • MAGI-1 and LTX-Video reduce effective sequence cost through temporal chunking or stronger latent compression.
  • Linear attention and state-space models replace the N×N map with a fixed-size state, although compressed states can restrict representation rank.

8. Conclusion … B.4. Timestep-Stratified Checkpoint Monitoring

SANA-Video 2.0 combines mostly-linear attention, periodic softmax anchors, and cross-layer residual routing in 5B and 14B video diffusion transformers. The paper also details efficient-training components and timestep-stratified monitoring that exposes improvements and regressions across noise regimes.

  • 8. Conclusion; B.1. Model Configuration: SANA-Video 2.0 uses 75% gated linear attention, 25% periodic softmax anchors, and depth-shared Block AttnRes in 5B and 14B models.The design targets a practical quality–efficiency knee identified by video-specific proxy studies.
  • A.1. Video Diffusion and Efficient Sequence Modeling: Full-softmax video diffusion transformers provide expressive global mixing, but their quadratic cost increases with resolution and duration; linear attention instead uses a fixed-size state.The related work positions efficient sequence models as direct reductions in backbone cost.
  • A.2. Hybrid and Sparse Attention: Hybrid language models motivate regular 3:1 linear-to-softmax layouts, while SANA-Video 2.0 trains a bidirectional hybrid video diffusion transformer from scratch.Sparse methods reduce computation within softmax layers, whereas the hybrid design reduces how many layers invoke softmax.
  • A.3. Cross-Layer Residual Routing: Attention Residuals retrieve completed block features from earlier depths, and SANA-Video 2.0 routes completed block summaries to address bidirectional, timestep-modulated, large-token video diffusion.This extends adjacent-layer residual streams with cross-layer feature access.
  • B.2. Self-Flow Distillation and Dual-Timestep Tokens: Self-Flow adds feature distillation and assigns 𝑅M=0.1 of tokens to a second timestep during pre-training and continual training, while remaining disabled for SFT.All tokens remain in the flow-matching loss, so Self-Flow changes conditioning rather than reducing backbone compute.
  • B.3. Content-aware Flow-Shift: TQD applies mutually exclusive +1.1 and −1.1 logit biases to selected pre-training clips, producing median timesteps of approximately 0.75 and 0.25; later stages use tail-floored shifts of 3–6.The later shift schedule disables TQD and does not constitute a per-sample continuous handoff.
  • B.4. Timestep-Stratified Checkpoint Monitoring: A 1,000-step noise axis is divided into ten equal buckets, with four checkpoints evaluated on identical clips, features, noise tensors, and timestep assignments.The equal-bucket average is a monitoring statistic rather than the training objective.
  • B.4. Timestep-Stratified Checkpoint Monitoring: 6.4% macro MSE reduction coincides with up to 11.4% low-noise improvement and 1.2% highest-noise regression, while estimate variability falls 2.3× and VBench Total rises from 82.68 to 83.29.With four observations, the VBench change is presented as context rather than a correlation claim.

C. Data Curation and Progressive Selection · D. Evaluation and Measurement Protocols

The data pipeline progressively curates video into broader pre-training, cleaner continual-training, and trusted SFT pools using separate quality and motion criteria. Training schedules jointly increase resolution, frame rate, and duration while preserving rank-synchronized batching.

  • C. Data Curation and Progressive Selection: Six processing stages construct progressively supervised pre-training, continual-training, and SFT pools with increasingly strict selection thresholds.Pre-training prioritizes coverage, continual training raises quality and motion-consistency floors, and SFT uses the most trusted subset.
  • C. Data Curation and Progressive Selection: Clips are decoded, segmented into coherent shots, cleaned of overlays and artifacts, and rejected when undecodable, undersized, blurry, unstable, or poorly exposed.TransNetV2 splits long videos, while temporal crops remove letterboxing and subtitles when possible.
  • C. Data Curation and Progressive Selection: Separate quality, motion, color, and consistency signals evaluate clips, combining DOVER, UniMatch, VMAF-derived features, SigLIP, temporal aggregation, and frame-rate-normalized optical flow.This preserves informative low-motion clips instead of removing them solely for being slow.
  • C. Data Curation and Progressive Selection: Progressive per-axis thresholds produce broad pre-training, cleaner continual-training, and trusted SFT subsets with captions that increasingly structure subjects, actions, camera motion, scenes, lighting, interactions, and temporal evolution.Quantized motion descriptors are appended when available so motion remains explicit to the text conditioner.
  • C. Data Curation and Progressive Selection: The schedule advances resolution from 480p to 720p alongside frame rate and duration, increasing the 720p share in later stages.Frame counts are selected to yield an integer number of LTX-VAE latent frames.
  • C. Data Curation and Progressive Selection: Per-rank sampling assigns one aspect-ratio and frame-count tier per step under FSDP batch size one, preventing resolution and duration mixing from desynchronizing ranks.Image batches are interleaved with video batches.

D.1. Sampling and VBench Evaluation … E.2. Qualitative Effect of Online RL

The evaluation reports standardized 40-step VBench and latency measurements across SANA-Video 2.0 scales, resolutions, durations, and hardware protocols, alongside qualitative samples and matched SFT-versus-RL comparisons. These results characterize both quantitative efficiency and qualitative behavior under the paper’s stated evaluation settings.

  • D.1. Sampling and VBench Evaluation: 40 steps, motion bucket 30, seed 0, and Gemma-2-2B-IT text features define the reported SANA-Video 2.0 operating points.The 81-frame setting uses guidance 6.0 and flow-shift 6.0, while 121/193-frame settings use guidance 8.0 and flow-shift 12.0.
  • D.1. Sampling and VBench Evaluation: VBench evaluates all 16 dimensions using official prompts, with Quality averaging seven dimensions, Semantic averaging nine, and Total applying 4:1 quality–semantic weighting.Baseline scores come from official results or official pipelines with default inference settings, while unmarked scores use the common 480×832×81 protocol.
  • D.1. Sampling and VBench Evaluation: 40 steps anchor the headline single-H100 latency comparison, whose end-to-end timing includes text encoding, denoising, and VAE decoding at batch size one.Reported values average three steady-state videos after one full warmup, excluding checkpoint loading and video writing.
  • D.1. Sampling and VBench Evaluation: 13.2s and 29.1s are the 5B and 14B latencies at 480×832×81, versus 421s for Bernini-R.At 736×1280×81, the corresponding latencies are 30.9s and 69.3s; SANA-Video-2B officially reports 36s at 720p.
  • D.2. Latency and Profiling Protocols: 20 CUDA-event timed forwards with eight warmups profile compiled DiT forwards on one H100 80GB using bf16 and batch size one.The profiles use fused linear attention, FlashAttention for softmax, and torch.compile; they exclude text encoding and other pipeline stages.
  • D.2. Latency and Profiling Protocols: 30 fused-linear layers and 10 Flash-SDPA anchors form the 14B hybrid control against 40 Flash-SDPA full-softmax layers under fixed width, depth, inputs, and non-attention settings.AttnRes is disabled in both variants to isolate the attention backbone, and H100 and GB200 sweeps use identical software and compiled kernels.
  • E.1. Additional Qualitative Samples: Six additional 720p clips span people, wildlife, urban scenes, and stylized content across five frames sampled at 0/2/4/6/8s.The clips use the same 40-step, 193-frame recipe and cover 1280×736 resolution, 193 frames, and 8s at 24fps.
  • E.2. Qualitative Effect of Online RL: Five evenly spaced frames from four matched-prompt, matched-CFG eight-second clips compare SFT and RL checkpoints qualitatively.These examples complement training-time reward trajectories and provide prompt-level context rather than aggregate evaluation.

E.3. Qualitative Comparison with Other Methods · F. AttnRes Routing and Representation Analysis

SANA-Video 2.0 is qualitatively compared with several video-generation methods under matched native-resolution evaluation settings, including text- and image-conditioned scenarios. The AttnRes analysis examines router behavior across 5B checkpoints while distinguishing a mature 4.5B production ancestor for continued-training comparisons.

  • E.3. Qualitative Comparison with Other Methods: Figure 20 compares SANA-Video 2.0 with Wan 2.2-A14B, Bernini-R 14B, Cosmos-3 Nano, and LTX-2.3.The comparison uses four prompts targeting sustained action, color change, shape transformation, and repeated events.
  • E.3. Qualitative Comparison with Other Methods: Each method uses its official pipeline, native 720p-class shape, and step count on one H100 with seed 42.SANA-Video 2.0 uses 40 steps and 193 frames.
  • E.3. Qualitative Comparison with Other Methods: Because output durations differ, Figure 20 displays normalized start, midpoint, and end frames rather than absolute timestamps.This normalization supports temporal-behavior comparison across methods with different durations.
  • E.3. Qualitative Comparison with Other Methods: Figures 21 and 22 compare image-conditioned generation using the same first frame and caption from VBench-I2V for every method.Rows show normalized progress at 0/25/50/100%.
  • E.3. Qualitative Comparison with Other Methods: SANA-Video 2.0 generates 121 frames with its image-conditioned checkpoint, while LTX-2.3, Wan 2.2 TI2V-5B, and Cosmos-3 use official first-frame paths.All methods run at native 720p-class shapes with seed 42.
  • E.3. Qualitative Comparison with Other Methods: Bernini-R is omitted from the image-conditioned comparison because its public pipeline has no corresponding mode.The remaining methods use their official first-frame conditioning paths.
  • F. AttnRes Routing and Representation Analysis: The AttnRes section explains what the selected router learns and why its final form shares one query across depth without an explicit timestep input.This routing analysis uses 5B architecture-family checkpoints unless otherwise noted.
  • F. AttnRes Routing and Representation Analysis: The continued-training comparison uses the mature 4.5B production ancestor, unlike the default analyses based on 5B architecture-family checkpoints.The passage distinguishes these checkpoint families by analysis condition.

F.1. Late-Stage Continued-Training Probe … G.3. 14B Hybrid versus Full-Softmax Scaling

The analyses show that AttnRes preserves mature-model quality while selectively reusing completed representations, with explicit router timestep conditioning largely unnecessary. Efficiency studies find that hybrid attention scales favorably with model size, hardware generation, and sequence length.

  • F.1. Late-Stage Continued-Training Probe: 0.4851 versus 0.4855 mean MSE shows late-stage AttnRes continuation is effectively tied to the no-router trajectory, ruling out degradation.The comparison uses near-matched checkpoints, 100 held-out clips, and 20 noise levels; 17 of 20 buckets slightly favor AttnRes.
  • F.2. Routing Selectivity Metric: AttnRes aggregates initial embeddings, completed block summaries, and partial sums using query-key softmax weights, with normalized entropy measuring routing selectivity across source sets.The entropy convention makes uniform routing equal to 1 and selective routing approach 0, excluding the first single-source attention site.
  • F.3. Timestep Adaptivity: 2.15× more entropy variation for the shared router than the per-layer alternative demonstrates timestep-adaptive depth aggregation without a router-timestep offset.The result averages ten held-out batches across five timesteps with real text conditioning.
  • F.4. Timestep Conditioning Probe: 0.0002 maximum validation-loss difference across Normal, Mean, and Shuffle conditions, together with 0.979 mean cosine, shows the explicit router timestep offset is nearly constant.The varying component is only 13% of the constant component’s norm, motivating removal of explicit router-timestep projection while retaining per-block timestep modulation.
  • F.5. Depth-Routing Patterns: 56% of attention routing mass and 50% of FFN routing mass goes to completed summaries in layers 25–32, with operation order explaining their differing entry patterns.The detailed routing view favors the most recently completed block, supporting cross-depth reuse.
  • F.6. Completed-Block Routing Ablation: 82–91% rank reduction at block-entry layers 9, 17, and 25 after removing completed-block sources localizes their reuse to reset boundaries.The effect becomes small after the partial sum rebuilds within each block.
  • F.7. Same-Checkpoint Rank Probe: The same-checkpoint rank probe isolates AttnRes by toggling depth aggregation while pairing noise seeds 0–3, and reports per-layer mean and standard deviation with a deep-layer summary over layers ≥15.This protocol changes no weights or activations outside the AttnRes path.
  • G. Efficiency and Deployment Analysis; G.1. Model-Size Scaling at Fixed Sequence Length; G.2. 14B Cross-Hardware Scaling; G.3. 14B Hybrid versus Full-Softmax Scaling: 1.40× to 2.73× Full/Hybrid speedup on H100 and 1.58× to 3.07× on GB200 as duration grows from 5 to 45s confirms stronger long-sequence scaling at 14B.At fixed 720p/10s, absolute time saved by the 25% hybrid grows from 128 to 948ms while Full/Hybrid speedup narrows from 1.88× to 1.45×; GB200 separately reduces matched 14B forward latency by 1.69–2.02× versus H100, with median 1.91×.

G.4. Temporal-Convolution FFN Ablation · G.5. Module-Level Composition · G.6. Compile-Friendly AttnRes

The ablations show that removing the temporal-convolution FFN path simplifies the backbone while preserving attention-scaling benefits, and that softmax anchors increasingly dominate long-shape forward time. A compile-friendly AttnRes rewrite further improves compiled/eager speedups with negligible fp32 drift and limited bf16 drift.

  • G.4. Temporal-Convolution FFN Ablation: G.4. Temporal-Convolution FFN Ablation: Removing the temporal-convolution path simplifies the backbone and preserves the scaling benefit of the selected attention ratio.The comparison holds depth and width fixed while changing only the FFN path; the temporal-convolution overhead grows with duration and reaches 20–29% at the larger scale.
  • G.4. Temporal-Convolution FFN Ablation: G.4. Temporal-Convolution FFN Ablation: The large-scale 60s temporal-convolution timing is an isolated outlier excluded with its paired baseline.
  • G.5. Module-Level Composition: G.5. Module-Level Composition: 54.4% of forward time is consumed by eight softmax anchors at 111.3K tokens, versus 23.0% at 22.1K tokens.The shares are mutually exclusive and additive under the eager shape diagnostic.
  • G.5. Module-Level Composition: G.5. Module-Level Composition: Linear self-attention falls from 26.0% to 15.7% of forward time as tokens increase from 22.1K to 111.3K.The shift motivates anchor-specific kernels for long shapes.
  • G.6. Compile-Friendly AttnRes: G.6. Compile-Friendly AttnRes: Exposing at-most-five sources as a static list and accumulating their weighted sum in fp32 avoids stacking all sources and improves torch.compile compatibility.The original path mutates a source buffer and reduces a growing prefix, which is difficult for torch.compile to fuse.
  • G.6. Compile-Friendly AttnRes: G.6. Compile-Friendly AttnRes: Compiled/eager speedup rises from 1.47× to 2.11× at 8s and from 1.21× to 1.49× at 60s after the rewrite.Relative to the buffer path, full-forward drift is negligible in fp32 and 0.68% in bf16; these figures compare router implementations.
Loading 2607.21553v1…