Source-linked AI summary
Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models
Jiazheng Xing, Hangjie Yuan, Lingling Cai, Xinyu Liu, Yujie Wei, Fei Du, Tao Feng, Hai Ci, Jiasheng Tang, Weihua Chen, Fan Wang, Yong Liu
TL;DR
Large diffusion generators make connector-based video unified-model training expensive, constraining the joint pursuit of semantic alignment and visual fidelity. Lumos-Nexus trains only a lightweight generator, then uses Unified Progressive Frequency Bridging to hand off generation to a high-capacity model, improving VBench quality while preserving reasoning performance on VR-Bench.
Problem
Large high-fidelity generators make connector-based unified-model training computationally prohibitive, limiting practical visual quality.
Method
Lumos-Nexus aligns a lightweight generator during training and uses Unified Progressive Frequency Bridging in a shared latent space during inference.
Results
Lumos-Nexus improves visual realism and temporal coherence on VBench while maintaining strong reasoning-guided generation on VR-Bench.
Takeaways & Limitations
The framework separates semantic alignment from high-fidelity synthesis to reduce training cost while retaining reasoning-driven video generation.
Takeaways & Limitations
The framework assumes compatible homogeneous latent spaces; mismatched latent representations can severely degrade outputs.
Abstract
from arXiv · showhide
Connector-based video unified models have demonstrated strong capability in instruction-grounded video synthesis, but integrating a large high-fidelity generator into the unified training loop is computationally prohibitive, limiting achievable visual quality. We therefore propose Lumos-Nexus, a training-efficient unified video generation framework that facilitates the development of strong reasoning-driven generation capabilities while significantly enhancing visual fidelity. Lumos-Nexus adopts a two-stage design: 1) During training, only a lightweight generator is aligned with the understanding block to learn to take in reasoning-driven semantic control. 2) During inference, we introduce Unified Progressive Frequency Bridging (UPFB) to progressively hand off generation to a high-capacity pretrained generator in the shared latent space, enabling coarse-to-fine refinement and producing high-fidelity videos without compromising reasoning quality. To fill the gap in reasoning-driven video generation benchmarks, we introduce VR-Bench, which assesses a model's capability to translate inferred intent into coherent and semantically aligned video content. Extensive experiments demonstrate that Lumos-Nexus achieves substantial gains in visual realism and temporal coherence on VBench, while exhibiting strong reasoning-based generative performance on VR-Bench. Code and models are available at https://jiazheng-xing.github.io/nexus-lumos-home/.
1 Introduction
Lumos-Nexus addresses the cost of integrating large diffusion generators into connector-based video unified models by separating semantic alignment during training from high-fidelity synthesis during inference. It also introduces VR-Bench to evaluate reasoning-driven video generation, with experiments showing improved visual quality and preserved reasoning performance.
- Motivation: Connector-based video unified models avoid joint optimization but still face substantial fine-tuning overhead from large diffusion generators.This makes simultaneously achieving semantic alignment and high visual fidelity challenging in practice.
- Lumos-Nexus: Lumos-Nexus aligns only a lightweight diffusion generator with the understanding block during training to learn structured semantic and reasoning controls.The approach separates semantic alignment from high-fidelity video synthesis.
- Lumos-Nexus: Unified Progressive Frequency Bridging progressively hands generation from the lightweight model to a high-capacity generator for coarse-to-fine refinement.The framework uses a shared latent-space design to combine semantic construction with visual detail refinement.
- Evaluation: VR-Bench evaluates alignment between inferred intent and generated video content across physical-world reasoning, commonsense reasoning, and embodied interactions.The benchmark spans eight dimensions of reasoning-driven video generation.
- Results: Lumos-Nexus improves visual realism and temporal coherence on VBench while maintaining strong reasoning-guided generation on VR-Bench.The reported results support both high-fidelity synthesis and reasoning-oriented generation.
2 Related Works
Prior video-generation research centers on autoregressive and diffusion-based models, while video unified models combine multimodal understanding with visual generation in a single framework. Existing unified architectures include autoregressive-based, diffusion-based, and hybrid formulations.
- Video Generation Models: Autoregressive video models generate discrete token sequences step by step, enabling explicit temporal modeling but typically incurring high inference latency.They may also suffer from accumulated errors during generation.
- Video Generation Models: Diffusion-based models form a second major paradigm in recent video-generation research.The passage identifies diffusion modeling as a primary driver of recent progress alongside autoregressive approaches.
- Video Unified Models: Video unified models integrate multimodal understanding and visual generation so the two components can mutually reinforce one another.These models extend unified modeling from multimodal systems to the video domain.
- Video Unified Models: Existing video unified architectures are commonly categorized as autoregressive-based, diffusion-based, or hybrid formulations.The categories reflect different ways of combining understanding and visual generation.
3 Methods
Lumos-Nexus separates semantic alignment from high-fidelity synthesis by training only a lightweight generator and progressively bridging it with a large generator during inference. UPFB coordinates temporal and frequency-domain handoff to preserve semantic coherence while refining visual details, and VR-Bench evaluates reasoning-oriented video generation.
- Motivation: Connector-based video unified models require costly large-generator fine-tuning to combine semantic alignment with high visual fidelity.Temporal sequences further increase input length and computational complexity.
- Framework: Lumos-Nexus fine-tunes only a small generator with the understanding block, then uses a homogeneous latent space to bridge it with a large generator at inference.The small generator learns to incorporate understanding-derived semantic knowledge while the large generator remains outside the training loop.
- Unified Progressive Frequency Bridging: UPFB applies a coarse-to-fine handoff: the small generator constructs semantic layout and global structure, while the large generator refines late-stage textures.Temporal weights shift dominance from the small generator toward the large generator as sampling proceeds.
- Unified Progressive Frequency Bridging: Frequency decomposition separates low-frequency semantics and spatial layout from high-frequency edges and textures before asymmetric fusion.The large generator contributes high-frequency detail, while the small generator maintains semantic coherence.
- Unified Progressive Frequency Bridging: RMS alignment normalizes velocity magnitudes across timesteps to stabilize fusion and prevent over-exposure or unstable activations.The method applies pre-fusion adjustment followed by post-fusion energy re-balancing.
- Evaluation: VR-Bench addresses the omission of reasoning in existing video benchmarks by evaluating inferred-intent alignment across eight dimensions.The dimensions span physical-world reasoning, commonsense reasoning, and embodied interactions.
4 Experiments
Experiments use Omni-Video as the small-generator baseline, Wan2.1-T2V-14B as the homogeneous-latent-space large generator, and VBench-T2V plus VR-Bench for evaluation. Inference uses a specified UPFB configuration for coarse-to-fine generation.
- Model: Lumos-Nexus uses Omni-Video’s Wan2.1-T2V-1.3B as the small generator and Wan2.1-T2V-14B as the large generator in the same latent space.
- Evaluations: VBench measures overall perceptual quality, whereas VR-Bench measures reasoning alignment between inferred intent and generated video content.
- Evaluations: VR-Bench contains 216 evaluation cases spanning eight dimensions across high-level physical-world, commonsense, and embodied-physical reasoning.
- Implementation Details: UPFB uses γw = 0.3, σmin = 0.35, σmax = 0.70, and γhf = 0.7 to balance semantic consistency and detail refinement.Videos are generated at 480p with 81-frame clips, using 50 inference sampling steps.
5 Main Results
Lumos-Nexus combines semantic guidance from a unified model with high-capacity generation, achieving strong VBench-T2V and VR-Bench results. Qualitative comparisons further show improved physical and reasoning-aligned video content over the compared baselines.
- Quantitative Results: 84.12 overall VBench-T2V score makes Lumos-Nexus the highest-scoring method among conventional and video unified models.The design combines Omni-Video’s semantic understanding with Wan2.1-14B’s high-fidelity detail synthesis.
- Quantitative Results: 79.10 →80.52 semantic alignment accompanies superior overall video generation quality.
- Quantitative Results: 79.28 overall VR-Bench score is highest among Wan2.1-level generation models.Lumos-Nexus also achieves HL-Comm. 77.57 and Emb.-Phys. 81.54.
- Quantitative Results: DRF 97.92, ETV 70.43, CCR 84.87, and CAC 88.99 show superior performance across multiple VR-Bench metrics.
- Qualitative Comparison: Qualitatively, Lumos-Nexus produces more realistic and physically consistent results than Wan-T2V and Omni-Video across three VR-Bench reasoning dimensions.The comparison covers High-Level Physical World Reasoning, High-Level Commonsense Reasoning, and Embodied Physical Reasoning.
6 Ablation Study
The ablations show that UPFB performance depends on transition sharpness, bandwidth range, and RMS alignment. Moderate settings best balance semantic grounding, high-frequency refinement, and visual coherence.
- Temporal Transition Sharpness: γw = 0.3 delivers the best VR-Bench reasoning performance with an overall score of 79.28.It particularly improves HL-Comm. 77.57 and Emb.-Phys. 81.54.
- Temporal Transition Sharpness: Larger γw preserves more of the small generator’s coarse layout and camera motion, whereas smaller values accelerate high-frequency refinement.
- Bandwidth Schedule: σmin = 0.35 and σmax = 0.70 achieve the highest VBench total score of 84.12.Both overly narrow and overly wide bandwidth settings reduce semantic and perceptual performance.
- RMS Alignment: Removing RMS alignment lowers the VBench total score from 84.12 to 84.07 and the semantic score from 80.52 to 80.43.RMS alignment stabilizes fusion by preventing magnitude mismatch during denoising.
- RMS Alignment: RMS alignment raises the VBench quality score from 84.98 to 85.03.
7 Conclusion
Lumos-Nexus combines lightweight training-time semantic alignment with inference-time bridging to a high-capacity generator, and introduces VR-Bench for reasoning-driven video evaluation. The framework targets coherent, high-fidelity synthesis while assessing physical, commonsense, and embodied reasoning.
- Lumos-Nexus aligns only a lightweight generator with the understanding block during training and uses UPFB to transition generation to a high-capacity pretrained generator during inference.The design separates semantic alignment from high-fidelity synthesis through coarse-to-fine refinement.
- VR-Bench measures alignment between inferred intent and generated video content across eight reasoning dimensions.Its dimensions span physical-world, commonsense, and embodied physical reasoning.
- VR-Bench uses a unified eight-question structure whose layers progress from basic perception to relational reasoning and high-level causal or semantic reasoning.The layered design supports consistent and interpretable assessment across diverse reasoning aspects.
- The benchmark covers dynamic reference frames, energy transfer, material consistency, conceptual actions, cultural commonsense, and broader physical and embodied reasoning capabilities.These dimensions examine spatial relations, physical interactions, purposeful actions, and semantic coherence across time.
- VR-Bench provides a structured framework for diagnosing whether generated videos reflect physical, causal, biological, and cultural reasoning.The benchmark jointly evaluates these forms of reasoning to assess reasoning-driven video generation systems.
C Quantitative T2I Results
Lumos-Nexus shows strong quantitative performance on text-to-image evaluation and improves bridging between small and large generators. Its results support the combination of reasoning-guided semantics with UPFB-based refinement.
- 0.79 overall performance is achieved by Lumos-Nexus among unified models on GenEval.It outperforms prior approaches across most evaluation dimensions.
- 0.67 vs. 0.56 in Attribute Binding is reported for Lumos-Nexus versus Omni-Video on GenEval.The comparison is presented as evidence of stronger compositional understanding and instruction adherence.
- +0.84 on VBench and +1.95 on VR-Bench are achieved by Lumos-Nexus over Direct Add.The corresponding scores are 84.12 vs. 83.28 on VBench and 79.28 vs. 77.33 on VR-Bench.
- Frequency-aware, temporally scheduled fusion avoids the duplicated or structurally inconsistent subjects observed with Direct Add.The qualitative comparison contrasts uncontrolled averaging with Lumos-Nexus’s frequency-domain fusion and velocity decomposition.
D.2 VR-Bench Cross-Evaluator Robustness
VR-Bench evaluation remains broadly consistent when the evaluator is changed from Qwen3-VL-30B-A3B-Instruct to GPT-5.2. Lumos-Nexus remains the top-performing method under the alternate evaluator.
- The overall VR-Bench performance ranking is largely consistent between GPT-5.2 and Qwen3-VL evaluations.The study re-evaluates the main baseline methods using GPT-5.2 as a different judge model.
- Lumos-Nexus remains the top-performing method when GPT-5.2 is used as the VR-Bench evaluator.
D.3 Discussion of Model Efficiency
Lumos-Nexus keeps training aligned with the small generator while adding a modest inference overhead, and reports quality gains over the large model alone. Sensitivity results also favor moderate transition settings.
- 40.5 vs. 35.1 s/step gives Lumos-Nexus about 1.2× the latency of Wan2.1-14B alone.The corresponding FLOPs are 4.568P vs. 3.462P per step.
- +0.43 on VBench and +1.05 on VR-Bench are reported for Lumos-Nexus over Wan2.1-14B.The scores are 84.12 vs. 83.69 on VBench and 79.28 vs. 78.23 on VR-Bench.
- 2.25 s/it and 26.3GB are the training cost of Lumos-Nexus, matching the small generator and avoiding large-model training.Training the large generator requires 9.72 s/it and 94.7GB GPU memory.
- 81.90 is the best VR-Bench score for γw = 0.3 and (σmin, σmax) = (0.35, 0.70) in the reported sensitivity study.Both smaller and larger transition settings produce lower performance.
D.6 Human Evaluation on VR-Bench
Human evaluation on VR-Bench shows that Lumos-Nexus ranks first against Qwen3-VL*-based rankings and human preferences, while supplementary analyses examine latent-space compatibility and generality across generators.
- 86.19 and 69.33 are Lumos-Nexus’s scores under Qwen3-VL* and human evaluation, respectively, ranking it first in both evaluations.The evaluation used 20 randomly sampled VR-Bench videos and 13 annotators.
- Kendall’s τ = 0.73 indicates strong positive agreement between Qwen3-VL*-based rankings and human preferences.
- Lumos-Nexus requires a homogeneous latent space, and replacing its large generator with heterogeneous HunyuanVideo severely degrades performance.The framework assumes compatible VAE latent spaces whose noise or velocity predictions remain distributionally aligned.
- Training-stage alignment can adapt a lightweight generator to a heterogeneous large generator’s latent space more efficiently than retraining the large generator.Latent compatibility is quantified with Maximum Mean Discrepancy, where lower values indicate better alignment.
- 65.79 is the VR-Bench score of the inference-bridged CogVideoX* model, falling between CogVideoX-2B and CogVideoX-5B.The experiment directly bridged the models at inference time without retraining CogVideoX-2B through the full Lumos-Nexus pipeline.
E Q&A Evaluation Examples from VR-Bench
VR-Bench Q&A evaluations provide representative eight-question diagnostic assessments of Lumos-Nexus videos across physical, commonsense, and embodied reasoning dimensions.
- Each representative VR-Bench case receives a full eight-question diagnostic assessment aligned with examples in Figures 7 and 8.
- The expanded Q&A results support fine-grained inspection of reasoning consistency across physical, commonsense, and embodied reasoning.
- The evaluations examine whether generated videos translate inferred intent into coherent, semantically aligned content across the benchmark’s reasoning dimensions.
F Limitaion
The paper identifies benchmark coverage as a limitation: VR-Bench does not fully represent open-world reasoning, and broader or longer-horizon reasoning remains for future work.
- VR-Bench cannot fully cover the open-world diversity of real-world reasoning scenarios.
- Long-horizon causal chains and other broader reasoning categories remain unexplored.
- The paper leaves deeper exploration of video unified models and broader reasoning-dimension coverage in VR-Bench for future work.
- Weighted Average Score: 1.00 is reported for the river-flow temporal-consistency example.
- Weighted Average Score: 1.00 is reported for the lightning physical-reasoning example.
- Weighted Average Score: 1.00 is reported for the clay-deformation material-consistency example.
- Weighted Average Score: 0.8 is reported for the scene-viewpoint consistency example.
- The examples report successful prevention of splashing, penguin push-off motion, and voice–action synchronization in gaming.