PaperlayerView exact revision on arXiv

Source-linked AI summary

Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

Chongjian Ge, Hanwen Jiang, Tianyu Wang, Jiuxiang Gu, Yiran Xu, Ziwen Chen, Shaoteng Liu, Jing Shi, Yicong Hong, Zefan Cai, Hailin Jin, Hao Tan

arXiv:2607.28611v1cs.CV

TL;DR

Long-sequence visual generation makes full attention costly, while systematic scaling methods for heterogeneous diffusion backbones remain limited. Chimera combines efficient long-context attention, global interaction, local convolutions, and module-wise scaling laws, reaching the training loss of a matched full-attention Wan 2.1 2B baseline with 7.3× fewer FLOPs.

  • Problem

    Token-extensive visual generation is constrained by quadratic full attention and the lack of systematic scaling frameworks for such architectures.

  • Method

    Chimera unifies text, image, and video tokens with KDA, periodic MLA, modality-aware convolutions, sparse MoE, and HeteroP-based scaling laws.

  • Results

    7.3× fewer FLOPs are required to reach the training loss of a matched full-attention Wan 2.1 2B baseline.

  • Takeaways & Limitations

    The fitted laws prescribe nearly balanced image-pretraining growth in activated model size and tokens, while video pretraining modestly favors model capacity at higher budgets.

  • Takeaways & Limitations

    The scaling family holds residual-stream count, MoE routing configuration, and MLA KV-compression ratio fixed rather than scaling them with model size.

Abstract

from arXiv · show

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.

1 Introduction

Chimera is a hybrid visual diffusion backbone and scaling framework for token-extensive generation, unifying text, image, and video tokens while combining linear long-range tracking with periodic global attention. Its HeteroP parameterization enables module-wise hyperparameter transfer and compute-optimal scaling laws across activated capacity, training tokens, and modality composition.

  • Scaling framework: HeteroP derives tensor-specific scaling ratios from functional fan-in and model depth, translating them into module-specific rules for learning rates, initialization, and residual scaling.This allows a recipe tuned on a small proxy model to transfer across heterogeneous model scales.
  • Scaling framework: The scaling laws treat image–video mixture as an explicit variable, relating compute-optimal allocation to activated parameter count, visual training tokens, and modality composition.The controlled model family supports fitting Chinchilla-style laws over activated model size and visual training tokens.
  • Scaling results: Nopt ∝C0.48–0.52 for image pre-training, indicating nearly balanced growth of activated model size and training tokens.Across three independent estimators, the training-curve envelope, IsoFLOP profiles, and parametric loss-surface fitting agree on this allocation.
  • Long-sequence generation: 6.5% frame-wise FID increase accompanies zero-shot extension from 5-second training clips to 30-second videos without length-specific fine-tuning.This extends the generation horizon by 6×; prior arts show an increase of more than 50%.
  • Architecture: Chimera unifies text, image, and video into one token sequence, using KDA for linear-complexity long-range tracking and periodic MLA for direct global interaction.The architecture is designed for token-extensive generation and supports heterogeneous visual contexts.

2 Chimera Overview

Chimera is a single-stream multimodal diffusion backbone that combines efficient state tracking, periodic global attention, sparse capacity, and minimal modality-specific bias. Its controlled model family supports module-specific hyperparameter transfer and Chinchilla-style analysis of activated parameters, training tokens, and image-video data composition.

  • Model Architecture: Chimera concatenates text, image, and video tokens into one shared sequence, enabling direct token-level interaction across modalities without modality-specific branches.The shared stack also supports language-model practices for hyperparameter transfer and compute-optimal scaling.
  • Model Architecture: KDA provides O(N) long-sequence computation, while interleaved MLA layers restore explicit global token interactions through compressed key-value states.The backbone uses a single temporal-major raster scan rather than complex visual scanning patterns.
  • Model Architecture: Chimera avoids positional embeddings and uses KDA state tracking as a simpler alternative for visual modeling and length extrapolation.The passage motivates NoPE by noting that positional embeddings can hinder extrapolation beyond training length.
  • Model Architecture: Sparse MoE increases total parameters while controlling per-token computation, alongside identity hyper-connections and sandwich normalization for training stability.These mechanisms respectively target capacity and residual-stream or sub-layer variance regulation.
  • Pre-training and Chinchilla Scaling: HeteroP transfers hyperparameters across heterogeneous modules using module-specific scaling ratios instead of a single global multiplier.This reduces target-scale search costs and supports fair scaling comparisons across Chimera models.
  • Pre-training and Chinchilla Scaling: Chinchilla-style laws predict compute-optimal allocation between activated model parameters and training tokens, while IsoFLOP analysis estimates compute-dependent image-video mixtures.The analysis uses a controlled model family to study scaling under a fixed compute budget.

3 Chimera Architecture

Chimera is a single-stream latent diffusion Transformer that jointly processes text and visual tokens while combining specialized mechanisms for global interaction, long-context state tracking, and local structure. Its architecture uses scheduled KDA/MLA attention, modality-aware convolutions, sparse MoE feed-forwards, and explicit positional inductive biases without RoPE.

  • Hybrid backbone: Each Transformer block alternates KDA and MLA attention, inserts mShortConv before KDA updates, and uses dense boundary FFNs with sparse MoE FFNs internally.The first and last blocks use dense SwiGLU FFNs, whereas intermediate blocks use sparse MoE FFNs.
  • Single-stream input: Chimera concatenates text and visual tokens into one shared sequence, flattening visual tokens in temporal-major raster order for direct cross-modal interaction.Separate projections map text and visual tokens into the shared hidden space, while padding tokens are removed before concatenation.
  • Output and training: After the final block, residual streams are averaged and only visual-token positions enter the rectified-flow velocity head, while text tokens condition predictions through shared-backbone interactions.The diffusion loss is calculated only at visual-token positions.
  • Positional mechanisms: 98.5% of queries in the strongest previous-token head attend most strongly to the immediately preceding token, illustrating the positional capacity consumed by rotary attention.A fast-rotating pair fixes the target offset while slow pairs provide distance-robust matching or near-uniform background contributions.
  • Positional mechanisms: Chimera factors positional functions into short convolution for position selection, gated state decay for recency, and operator structure for layout encoding without RoPE.These mechanisms avoid consuming attention capacity for positional operations, manually partitioning channels, or coupling positional phases to training length.
  • Sparse expert layers: Token-choice MoE routing activates selected experts independently per token, using small SwiGLU experts whose hidden dimension is 1/K of the dense FFN width.A balancing bias reduces batch-level routing imbalance, with MaxVio dropping below 1 almost immediately and stabilizing around 0.5.

4 Scaling Recipe

Section 4 introduces HeteroP, a module-wise hyperparameter-transfer scheme for consistently tuning Chimera across width and depth, then fits Chinchilla-style compute-optimal laws for visual diffusion. The analysis extends these laws to the image–video data ratio, while defining diffusion loss as the scaling target.

  • HeteroP motivation: HeteroP transfers base hyperparameters tuned on a small proxy model to larger models across width and depth.This reduces retuning cost while making cross-scale loss comparisons more meaningful.
  • HeteroP parameterization: HeteroP uses module-specific fan-in ratios because Chimera’s heterogeneous modules expose distinct width dimensions.These include model width, MLA KV-compression rank, KDA head width, MoE expert width, router width, and timestep-conditioning state.
  • Width and depth transfer: HeteroP combines module-wise width scaling with depth corrections for attention, FFN, and MoE outputs before residual writes.The depth multiplier is determined by block count, while module-specific width multipliers are determined by functional fan-in.
  • Compute-optimal scaling: HeteroP yields mutually comparable Chimera models used to fit Chinchilla-style compute-optimal scaling laws for visual diffusion.The fitted loss is Chimera’s rectified-flow v-prediction diffusion loss rather than a downstream score such as FID.
  • Image–video data ratio: The scaling-law analysis adds image–video data ratio as a dimension because modalities differ in information content and token contributions.This mixture changes both the learning signal and how the compute budget is allocated.

5 Experiments

Experiments validate Chimera’s scaling methodology, compute-optimal training laws, and efficiency gains against dense baselines. The results show near-balanced parameter–token allocation, image-heavy data mixtures, and substantially improved compute efficiency.

  • Evaluation protocol: The evaluation uses sampler-independent Chimera diffusion training loss for scaling comparisons and downstream generation metrics for final models.The scaling analyses compare model sizes, training durations, and data mixtures under a fixed objective and noise parameterization.
  • HeteroP transfer: 10^-3 is the approximately scale-stable optimal base learning rate across 56× model-size and 8× depth ranges.The optimum drifts by at most one sweep-grid point, while the surrounding loss basin remains wide and flat.
  • Scaling laws: 0.481 and 0.511 are the image scaling exponents for activated parameters and visual tokens, indicating nearly even compute allocation.The estimates come from IsoFLOP minima and agree with the training-curve envelope.
  • Scaling laws: 0.564 and 0.431 are the video scaling exponents for activated parameters and training volume, showing a modest tilt toward model capacity.IsoFLOP estimates give corresponding exponents of 0.555 and 0.444 when high-compute slices are included.
  • Convergence efficiency: 1.7× is Chimera-dense’s compute-efficiency gain over Wan, while complete Chimera reaches 6.8× efficiency at the common training loss of 0.149.Wan requires 4.29 × 10^20 FLOPs, Chimera-dense 2.55 × 10^20 FLOPs, and complete Chimera 6.27 × 10^19 FLOPs.

6 Related Works

Related work spans diffusion architectures for images and videos, efficient attention mechanisms, and scaling laws. These lines of research progressively address visual-generation quality, long-horizon modeling, computational efficiency, and compute allocation.

  • Diffusion Models for Visual Generation: Diffusion models evolved from convolutional U-Net backbones and improved noise schedules toward hierarchical and latent-space designs for visual generation.Classifier-free guidance and strong language encoders were also combined with diffusion models to improve generation fidelity.
  • Diffusion Models for Visual Generation: Video diffusion initially added temporal modules to image diffusion models, then increasingly pursued long-horizon generation through autoregressive or causal rollouts.The early approach reused strong image priors while learning motion.
  • Efficient Attention Mechanisms: Efficient attention mechanisms reduce either the attended representation, the attention operator, or the set of token interactions.These categories include key-value compression, removal of softmax for linear-time or constant-state inference, and other interaction reductions.
  • Model Scaling: Scaling-law research characterizes how model quality changes with parameters, data, and compute, extending from language models to compute-optimal allocation studies.Early work identified predictable power-law trends, while later research studied optimal data and model-size allocation under fixed compute budgets.

7 Future Exploration and Conclusion

Chimera is presented as a heterogeneous, single-stream diffusion backbone co-designed with a principled scaling recipe for token-extensive visual generation. Future work should broaden scaling beyond width and depth to additional architectural axes currently held fixed.

  • Conclusion: Chimera combines KDA, periodic MLA, and modality-aware short convolutions in a unified text–image–video sequence without positional embeddings.KDA supports efficient long-context state tracking, MLA provides global interaction, and short convolutions capture local spatio-temporal structure.
  • Future Exploration: HeteroP currently isolates width and depth transfer while holding several structural scaling axes fixed.These include the iHC residual-stream count, MoE expert and top-K routing configuration, and MLA KV-compression ratio.

A Notation

Table 3 organizes the paper’s notation by component and explicitly identifies section-local or reused symbols. The notation covers MLA, MoE routing, RoPE analysis, and scaling-law quantities.

  • Notation overview: Table 3 groups the main notation by component and states whether symbols are section-local or reused.It serves as the paper’s notation reference.
  • MLA: MLA notation defines attention heads, compressed key–value latents, direct keys, latent keys and values, and head-specific keys.The direct-key branch is projected from the MLA input and shared across heads.
  • MoE: N = 56 experts and K = 8 activated experts per token specify the MoE notation, alongside routing weights, load-balancing bias, and load metrics.The notation also defines the top-K expert set selected for each token.
  • RoPE analysis: RoPE notation records rotation frequencies, token offsets, rotary-pair components and energies, plus local and far attention mass at Chebyshev distances ≤1 and ≥5.Per-pair energy is distinguished from expert Ei.
  • Scaling: Scaling notation distinguishes proxy and target models, block counts, fan-in and block-count ratios, transferred hyperparameters, activated parameters, visual positions, compute, and fitted diffusion-loss terms.The training-compute approximation is C ≈6ND.

B Representative Model Configurations

Table 4 specifies the representative Chimera configurations used in the experiments and visualized in Fig. 6. It distinguishes proxy, scaling-family, and final-model roles while holding key structural ratios fixed across scales.

  • Configuration roles: Table 4 reports the exact Chimera configurations visualized in Fig. 6.The proxy supports hyperparameter tuning, the scaling family supports compute-optimal analyses, and the final model is selected for full training.
  • Scale construction: All configurations instantiate the Sec. 3 architecture, with structural ratios held fixed across scales.Fixed ratios include MoE experts, the KDA-to-MLA ratio, residual streams, KV-compression ratio, and patchification.
  • Parameter accounting: Table 4 reports both per-token activated parameters and total denoiser parameters.Total parameters count all routed MoE experts, and the full sweep includes additional intermediate sizes.

C Visual RoPE Analysis: Protocol and Results

The analysis shows that visual RoPE frequencies divide labor across local position selection and far-field semantic matching. Fast-rotating pairs drive raster-neighbor attention, while slowly rotating channels support long-range attention.

  • Local Heads and Position Selection: 0.99 local attention mass and 98.1% exact-previous-token argmaxes identify Wan2.2’s strongest head as a raster-order previous-token head.Layer 33, head 30 occupies the same position in both denoising experts, whose top-eight local-head lists share seven entries.
  • Local Heads and Position Selection: 0.944 versus 0.967 and 0.934 versus 0.957 local attention mass show that Wan2.2’s fastest frequency thirds preserve locality, whereas the slowest thirds collapse it to 0.002.The values correspond respectively to the low-noise and high-noise experts; the ablation supports fast-frequency attribution.
  • Local Heads and Position Selection: Unrotated channels in FLUX.2 select attended tokens by content, while fast pairs place the attention peak at the target offset.This division emerges when attention logits are recomputed from the two channel groups together.
  • Far-Field Heads and Semantic Matching: 0.012 and 0.013 rad/token far-field frequency centroids in Wan2.2 and FLUX.2 are more than 20× slower than local-head medians of 0.26 and 0.25.Far-field heads are defined by far-field mass > 0.8, while local heads have local mass > 0.5; mean attended distances are approximately 20 and 30 tokens.
Loading 2607.28611v1…