Source-linked AI summary
Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generation
Yanzuo Lu, Xin Xia, Manlin Zhang, Huafeng Kuang, Jianbin Zheng, Yuxi Ren, Xuefeng Xiao
TL;DR
Interleaved multimodal tokens make autoregressive decoding and diffusion denoising computationally costly. Hyper-Bagel combines speculative decoding for understanding with multi-stage distillation for generation. It reports speedups across both task types, including a lossless 6-NFE model for generation and editing and a 1-NFE model for near-real-time interaction.
Problem
As multimodal contexts grow with interleaved tokens, iterative diffusion denoising and autoregressive decoding impose significant computational overhead.
Source: §1 · p. 1Method
Hyper-Bagel uses speculative decoding for next-token prediction and multi-stage diffusion distillation, separating control, structural integrity, and image fidelity objectives.
Source: §1 · pp. 4–5Results
The framework reports over 2x speedup in multimodal understanding, 16.67x in text-to-image generation, and 22x in image editing; the 6-NFE model maintains baseline-equivalent benchmark performance.
Source: §1 · p. 5, §4.2 · p. 11, §5 · p. 12Takeaways & Limitations
The 1-NFE model is presented as a near-real-time option for interactive editing and generation, using adversarial distillation and human feedback learning.
Source: §1 · p. 5
Abstract
from arXiv · showhide
Unified multimodal models have recently attracted considerable attention for their remarkable abilities in jointly understanding and generating diverse content. However, as contexts integrate increasingly numerous interleaved multimodal tokens, the iterative processes of diffusion denoising and autoregressive decoding impose significant computational overhead. To address this, we propose Hyper-Bagel, a unified acceleration framework designed to simultaneously speed up both multimodal understanding and generation tasks. Our approach uses a divide-and-conquer strategy, employing speculative decoding for next-token prediction and a multi-stage distillation process for diffusion denoising. The framework delivers substantial performance gains, achieving over a 2x speedup in multimodal understanding. For generative tasks, our resulting lossless 6-NFE model yields a 16.67x speedup in text-to-image generation and a 22x speedup in image editing, all while preserving the high-quality output of the original model. We further develop a highly efficient 1-NFE model that enables near real-time interactive editing and generation. By combining advanced adversarial distillation with human feedback learning, this model achieves ultimate cost-effectiveness and responsiveness, making complex multimodal interactions seamless and instantaneous.
1 Introduction
Interleaved multimodal tokens make autoregressive decoding and diffusion denoising computationally costly; Hyper-Bagel addresses both with speculative decoding and multi-stage distillation. The framework reports speedups across understanding and generation, alongside a 1-NFE model for interactive editing and generation.
1 Introduction
- A target-draft intermediate-layer architecture addresses multimodal token variation, raising TPS from 98.3 to approximately 212.4 and yielding a 2.16x speedup.The draft model predicts consecutive tokens that the target model batch-validates in parallel.Source: §1 · p. 4
- Diffusion distillation separates control, structural integrity, and image fidelity into three stages, including CFG control distillation, multi-head adversarial distillation, and DMDO score distillation.DMDO uses an ODE with an Euler discrete sampler and avoids additional ODE- or GAN-based regularizers to counterbalance KL divergence.Source: §1 · p. 4
- Over 2x speedup in multimodal understanding, 16.67x in text-to-image generation, and 22x in image editing are reported while preserving high-quality output.The 6-NFE model maintains equivalent performance on GenEval and GEdit-Bench metrics compared with the baseline.Source: §1 · pp. 4–5
- The 1-NFE model adds rectified-flow adversarial distillation and human preference alignment using Reward Feedback Learning to support near-real-time interactive editing and generation.The reward model uses VLM-based HPSv3, chosen for its visual-understanding capability.Source: §1 · pp. 4–5
2 Data
Hyper-Bagel training draws on open-source data for VLM tasks, text-to-image distillation, image editing, and interleaved sequences. The data includes millions of image-text pairs and editing examples, including multi-turn rounds.
2 Data
- All VLM image-text, text-to-image, image-editing, and interleaved data used for training come from open-source datasets.The sources include LLaVA-OneVision, JourneyDB, and SEED-Data-Edit.Source: §2 · p. 5
- Around 4 million LLaVA-OneVision image-text pairs train VLM tasks, with target-model answers generated for each question to align draft-model predictions.The target-generated answers replace the dataset answers for draft-model training.Source: §2 · p. 5
- JourneyDB supplies over 4 million synthetic Midjourney images with VLM-annotated captions for text-to-image diffusion distillation.The prompts were user-provided, and the passage says their distribution closely aligns with practical usage.Source: §2 · p. 5
- SEED-Data-Edit provides 52K real-world editing pairs and 21K human-annotated multi-turn rounds, totaling 95K editing pairs, while interleaved data is used to preserve BAGEL properties.The multi-turn rounds contain up to five turns each.Source: §2 · p. 5
3.1 Speculative Decoding
Hyper-Bagel strengthens speculative decoding for BAGEL by improving target-feature aggregation, initialization, and draft-model supervision. Its combined loss uses λ = 0.1 and outperforms vanilla forward KL in the reported experiments.
3.1 Speculative Decoding
- Hyper-Bagel follows the EAGLE-3 training paradigm, adds an intermediate layer, and uses two draft decoder layers to increase representation capacity.The intermediate-layer architecture is illustrated in Figure 3.Source: §3.1 · p. 5, Figure 3 · p. 6
3.1.1 Target Feature Aggregation with Meta Queries
- The draft model aggregates target-layer features through cross-attention using learnable meta queries, then processes the flattened outputs with a Transformer decoder layer.The intermediate features are not concatenated with token embeddings at this stage.Source: §3.1.1 · p. 6
3.1.2 Zero-Init Fully Connected Layers with Residuals
- Zero-initialized projections and skip connections let pretrained target-model weights guide training of the intermediate layer and draft-model fully connected layers.The design zero-initializes the final projections of cross-attention and the decoder, as well as fc1 and fc2.Source: §3.1.2 · pp. 6–7
3.1.3 Diminish Forward KL Divergence Supervision
- λ = 0.1 yields better performance than vanilla forward KL in the reported experiments.The loss combines forward KL with cross-entropy supervision from the target model’s one-hot hard labels.Source: §3.1.3 · p. 7
3.2 Diffusion Distillation
Hyper-Bagel distills diffusion models in three stages to produce a 6-NFE model, then adds two fine-tuning stages for a 1-NFE model. The process preserves control and image quality as objectives, with the 6-NFE model reported as lossless across benchmarks and its score-distillation stage improving image fidelity.
3.2 Diffusion Distillation
- The 6-NFE model follows CFG distillation, trajectory segmented consistency distillation, and DMDO; the 1-NFE model adds ADP and ReFL.The framework aims to preserve both control and quality capabilities.Source: §3.2 · p. 7
3.2.1 Stage-1: CFG Distillation
- Stage 1 distills CFG embeddings into one forward pass, with separate text- and image-scale encodings to retain control over instruction adherence and editing consistency.Image scale is used only for editing scenarios.Source: §3.2.1 · p. 7
3.2.2 Stage-2: Trajectory Segmented Consistency Distillation
- Stage 2 uses a three-segment consistency-distillation configuration in one stage, with adversarial loss and a multi-head discriminator to emphasize image composition.This replaces progressive segment reduction; the stated intent is to improve structural integrity while leaving fidelity to Stage 3.Source: §3.2.2 · pp. 7–8
3.2.3 Stage-3: Distribution Matching Distillation via ODE
- The alternating score-distillation updates produce a 6-NFE model reported as lossless across benchmark dimensions, with improved color vibrancy and detail richness.The authors describe the resulting fidelity as closely approximating the original model.Source: §3.2.3 · pp. 8–9
3.2.5 Stage-5: Reward Feedback Learning
- The 1-NFE model is fine-tuned from the 6-NFE model through ADP and ReFL, using human feedback to compensate for fidelity deficiencies.A single VLM-based reward model guides ReFL; its reward activation threshold αd is set to 6.0.Source: §3.2 · p. 7, §3.2.4 · p. 9, §3.2.5 · p. 9
3.2.4 Stage-4: Adversarial Diffusion Pre-training
- ADP aligns the 1-NFE generator structurally with the 6-NFE model by training against its ODE trajectory endpoints using latent- and pixel-space discriminators.The 6-NFE model supplies the ODE trajectories used as training targets.Source: §3.2.4 · p. 9
4 Experiments
The experiments report strong generation and editing performance at 6 NFE, with qualitative comparisons against high-NFE baselines. The setup separates denoising from prefilling so the original model handles prefilling.
4.2 Quantitative Results
- 0.8647 on GenEval for 6-NFE Hyper-BAGEL slightly exceeds 0.8640 for the 100-NFE BAGEL baseline, a 16.67x speedup without compromising generative quality.The 1-NFE model scores 0.7962 overall, on par with Janus-Pro-7B and MetaQuery-XL at 0.80.Source: Table 2 · p. 10, §4.2 · p. 11
- 6-NFE Hyper-BAGEL scores 6.612 in English and 6.671 in Chinese on GEdit-Bench, outperforming the 132-NFE baseline with an approximately 22x inference speedup.The baseline uses 132 NFE because text and image CFG are enabled together only within the [0.4, 1.0] interval.Source: Table 3 · p. 10, §4.2 · p. 11
4.1 Setup
- Setup evaluates generation and editing with GenEval and GEdit-Bench, reports draft acceptance metrics, and uses distilled weights for denoising while original weights handle prefilling.Freezing the understanding branch during distillation and reverting to the original model for prefilling are reported to preserve understanding performance.Source: §4.1 · pp. 9–10
4.3 Qualitative Results
- Across four Figure 5 cases, 6-NFE outputs are visually indistinguishable from the 100-NFE baseline; 1-NFE outputs remain prompt-relevant but can lose detail fidelity.The 1-NFE examples may omit prompt elements or deviate in details, while retaining core semantics and overall quality.Source: Figure 5 · p. 11, §4.3 · pp. 11–12
- Figure 6 compares editing against the 132-NFE baseline; 6-NFE edits are virtually indistinguishable, while 1-NFE preserves coherence but may compromise subtle details or photographic realism.The examples include removing peanuts, replacing text, and eliminating a person.Source: Figure 6 · p. 11, §4.3 · p. 12
5 Conclusion
Hyper-Bagel combines speculative decoding for understanding with multi-stage distillation for generation to reduce multimodal-model overhead. Its 6-NFE model speeds generation and editing while matching or exceeding high-NFE baselines, and its 1-NFE model supports near-real-time applications.
5 Conclusion
- Hyper-Bagel’s divide-and-conquer framework accelerates understanding and generation, with 6-NFE speedups above 16.67x for text-to-image and 22x for editing, plus doubled understanding speed.The conclusion describes 1-NFE as a practical option for near-real-time applications, especially interactive editing.Source: §5 · p. 12