Source-linked AI summary
CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition
Hongji Yang, Songlian Li, Yucheng Zhou, Xiaotong Zhao, Alan Zhao, Chengzhong Xu, Jianbing Shen
TL;DR
Existing video generators remain fragile when conditions are sparse, abstract, or complex, creating a gap between inputs and users' creative intent. CogOmniControl factorizes generation into CogVLM-based intent cognition and CogOmniDiT-based synthesis, adds evaluator-guided Best-of-N selection, and evaluates the system on professional-workflow benchmarks. Experiments report consistent gains over open-source controllable video generators while narrowing the gap with proprietary systems.
Problem
Video generation models struggle to align outputs with creative intent under sparse, abstract, or complex conditions, especially in professional workflows.
Method
CogOmniControl uses CogVLM for creative-intent reasoning and CogOmniDiT for intent-aligned generation, with evaluator-guided Best-of-N selection and reinforcement fine-tuning.
Results
CogOmniControl consistently outperforms state-of-the-art open-source controllable video generators and narrows the gap with strong proprietary systems.
Takeaways & Limitations
The framework provides a closed-loop approach for controllable video generation using professional reasoning, intent-aligned synthesis, and adaptive evaluation.
Abstract
from arXiv · showhide
Recent diffusion models achieve strong photorealism and fluency in video generation, yet remain fragile under abstract, sparse or complex conditions, leading to poor performance in professional production workflows such as storyboard sketches and clay render conditions. Existing video generation models, either inject conditions through adapters or couple a generic vision-language model (VLM) within a diffusion backbone, leaving a capability gap and failing to produce the videos that align with the user's creative intent. We present CogOmniControl, a reasoning-driven framework that factorizes controllable video generation into creative intent cognition and generation. Specifically, we train a specialized CogVLM using authentic anime production data. Compared to generic VLMs, it generates more professional and clear outputs, accurately cognizing user creative intent from sparse and abstract conditions and tuning these cues into dense reasoning output. Besides, CogOmniDiT unifies the controls from various conditions through in-context generation and is aligned to the CogVLM reasoning outputs via reinforcement learning. Furthermore, leveraging CogVLM's robust capability in guiding video generation, we release its potential in planning specific evaluators and enable a Best-of-N selection for the generated videos. This integration transforms the entire framework into a closed-loop "harness-like" architecture. We further introduce CogReasonBench and CogControlBench, built from professional workflows data that carry genuine creative intent rather than simulated ones. Experiments on two benchmarks show that CogOmniControl surpassed the existing open-source models. The project website: https://um-lab.github.io/CogOmniControl/
1 INTRODUCTION
CogOmniControl addresses the difficulty of translating sparse or abstract conditions into videos that match creative intent. It combines specialized reasoning, intent-aligned generation, evaluator-guided selection, and benchmarks drawn from professional workflows.
- CogOmniControl bridges abstract conditions and creative intent through a reasoning-driven controllable video generation framework.
- CogVLM infers creative intent from sparse, abstract, and multimodal conditions, producing dense reasoning outputs for generation.
- CogOmniDiT combines diverse control signals with CogVLM features to synthesize videos aligned with inferred intent.
- CogVLM emits evaluators alongside solutions, enabling Best-of-N candidate selection in a closed-loop reasoning-generation-verification system.
- CogReasonBench and CogControlBench evaluate reasoning and controllable generation using storyboards and clay renders from professional animation workflows.
2 RELATED WORK
Prior work advances conditional and omni-level video generation but remains limited in handling diverse, non-pixel-aligned conditions and robust unified control. Reinforcement learning methods have also been used to align visual generation with human preferences.
- Video Generation: Diffusion models support high-fidelity visual content, while adapter-based methods inject conditions without compromising original generative quality.
- Video Generation: Adapter-based paradigms have limited flexibility for diverse conditions, especially non-pixel-aligned inputs or visual references.
- Video Generation: OmniGen, OmniGen2, OmniControl, and UNO pursue unified or in-content multimodal generation, extending omni-level generation toward video.
- Video Generation: Open-source video models including VACE, UniVideo, and VINO attempt unified generation, but current systems still lack robust unified video generation.
- Reinforcement Learning for Visual Generation: DDPO, Diffusion-DPO, and DPOK apply preference-optimization methods to text-to-image diffusion, while later work explores denser reinforcement-learning rewards.
3 METHOD
CogOmniControl separates creative-intent reasoning from video generation to handle diverse, conflicting, and abstract conditions. CogVLM formulates production schemes and evaluator tools, while CogOmniDiT unifies conditions and generates videos aligned with the reasoning output.
- CogOmniControl Framework: CogOmniControl uses CogVLM for reasoning and CogOmniDiT for generation across pose, depth, lineart, storyboard sketch, and clay render conditions.The framework factorizes controllable generation into interpreting conditions and synthesizing the final video.
- CogOmniControl Framework: The condition set combines a control video, reference image, and textual description with temporal, spatial, appearance, and semantic guidance.These inputs are represented as C = {Vctrl, Iref, Tdesc}.
- CogVLM: CogVLM reasons over conflicting multimodal conditions, identifies cross-modal entities, and extrapolates implicit details into dense production schemes.The model is prompted to infer effects such as rippling water from rainy text and standing water in a reference image.
- CogVLM: CogVLM is trained with SFT and RFT using holistic and fact-verification rewards that assess intent, physical plausibility, information integrity, and motion description.The accuracy reward converts subjective narrative evaluation into a verifiable accuracy metric.
- CogOmniDiT: CogOmniDiT processes noisy latents and heterogeneous conditions in one sequence, using self-attention to inject controls into the latent representation.Its input sequence concatenates noisy, reference-image, control-video, and CogVLM embedding features.
- CogOmniDiT: CogOmniDiT receives reinforcement fine-tuning to adhere simultaneously to pixel-level conditions and high-level reasoning, optimizing condition following and video quality.Training occurs at lower resolution while inference uses high resolution through video diffusion transformer scaling.
- Benchmarks: CogControlBench and CogReasonBench incorporate professional workflow data and general video-generation data for evaluating condition understanding and controllable generation.The professional data comes from storyboard or clay-render videos and final videos collected from in-house anime production pipelines.
- Closed-Loop Verification: CogVLM emits reasoning and an adaptive evaluator harness, enabling repeated rollouts and Best-of-N selection based on condition-specific scores.The harness selects evaluators from a tools library rather than applying one fixed evaluator set to every generation.
4 BENCHMARK
CogControlBench combines professional anime-production clips with general controllable-generation data, using manually aligned control and final clips to preserve creative intent.
- Benchmark Construction: CogControlBench includes storyboard and clay-render videos paired with final videos from in-house professional anime production pipelines.The data reflects the gap between abstract user conditions and raw creative intent in professional production.
- Benchmark Construction: The benchmark also incorporates general controllable-generation data, including edited reference-to-video samples and frame-by-frame extracted conditions.
- Benchmark Construction: Professional workflow samples receive manual semantic alignment and annotation so control clips and final clips share the same semantics.
- Benchmark Scope: CogReasonBench evaluates CogVLM, while CogControlBench evaluates CogOmniControl on conditions understanding, abstract reasoning, and instruction following.
- Benchmark Scale: CogControlBench contains 200 high-resolution representative samples curated to match anime-production quality while enabling efficient validation.
5 EXPERIMENT
Experiments evaluate multimodal intent following and visual quality with VBench-based metrics and VLM judging. CogOmniControl leads open-source competitors, with further gains from adaptive Best-of-N evaluator selection.
- Evaluation: CogOmniControl is evaluated with VBench-based numeric metrics and a VLM-as-a-Judge paradigm across condition following and visual quality.Condition following measures holistic multimodal intent alignment, while visual quality covers aesthetics, imaging, temporal and motion properties, identity consistency, and dynamic plausibility.
- CogVLM Results: Generic VLMs fail to cognize creative intent from multimodal inputs and underperform CogVLM in information integrity and motion description.They also struggle to describe inferred physical effects accurately.
- Quantitative Results: 0.727 average score is achieved by CogOmniControl on CogControlBench, exceeding VINO (0.686) and VACE-Wan2.1 (0.665).The score is the highest among the reported open-source competitors and approaches Seedance2.0 (0.750).
- Quantitative Results: 0.742 is obtained with specific evaluators suggested by CogVLM during inference, compared with 0.733 using the full evaluator set at N=4.Adaptive evaluator selection yields better Best-of-N performance than using the full set of evaluators.
- Qualitative Results: Qualitatively, adapter-based methods exhibit artifacts and semantic misalignment on sparse clay-render controls, while VINO produces nearly static outputs.CogOmniControl remains strong on reference-to-video generation in the reported comparisons.
- Ablation Studies: 3.588 versus 3.142 is reported after CogVLM SFT compared with vanilla Qwen3-VL-8B-Thinking in the ablation study.The study attributes the improvement to better multimodal-intent capture and supports using a VLM specialized for controllable generation.
6 CONCLUSION
CogOmniControl factorizes controllable video generation into creative-intent cognition and video generation, extending the framework with evaluator-guided Best-of-N selection. It is evaluated against other models in clay-render and general video-generation settings, with experiments reporting strong comparative performance.
- CogOmniControl separates creative-intent cognition from generation through CogVLM and CogOmniDiT, respectively.CogVLM transcribes sparse multimodal cues into production reasoning, while CogOmniDiT combines pixel-level conditions with semantic features and aligns generation to the reasoning output.
- CogVLM adaptively plans evaluators for Best-of-N selection, extending the pipeline into a closed-loop reasoning-generation-verification system.The evaluator harness scores generated candidates and selects among them.
- The framework is further supported by two curated benchmarks for CogVLM and CogOmniDiT.
- In clay-render comparisons, adapter-based methods tend to produce pixel-level artifacts and semantic misalignment under sparse, abstract control conditions.The qualitative comparison concerns clay render, a common intermediate draft stage in animation production.
- Extensive experiments show that CogOmniControl consistently outperforms state-of-the-art open-source controllable video generators and narrows the gap with strong proprietary systems.
A TRAINING DETAILS
CogOmniDiT is trained through three SFT stages followed by low-resolution GRPO-based reinforcement fine-tuning, with higher-resolution inference.
- CogOmniDiT uses three SFT stages for in-context generation, CogVLM alignment, and joint training.
- RFT applies GRPO training at 256P resolution and performs inference at 720P resolution.The procedure follows Flow-Factoy and PaCo-RL.
B EVALUATOR HARNESS
The evaluator harness uses common and reasoning-selected evaluators to assess generated videos for artifacts, prompt or annotation adherence, temporal quality, and cross-modal causal consistency. Its evaluator specifications define inputs, checking procedures, and 0–5 scoring criteria for selected quality dimensions.
- Evaluator selection: Best-of-N selection always invokes common evaluators, while CogVLM conditionally selects specialized evaluators from the input reasoning.For storyboard inputs, the system can invoke a storyboard-annotation evaluator when handwritten annotations require verification.
- Artifact evaluation: The artifact evaluator targets multiple heads or limbs, deformation, floating objects, rendering failures, identity distortion, smearing, and abnormal noise.Its scoring criteria range from no artifacts at 5 to total collapse at 0.
- Storyboard Annotation Evaluators: The storyboard evaluator checks annotations independently from the prompt and reference image, verifying each instruction and requiring described dynamics to appear as clear motion.Its inputs include a reference image, control video, text prompt, and generated video.
- Storyboard Annotation Evaluators: Storyboard annotations are assessed across scene dynamics, character actions, and environmental atmosphere, including natural motion and faithful execution.
- Cross-modal causal reasoning: The cross-modal causal evaluator judges whether text, image, and control-video cues produce logically sound causal interactions, such as rain producing ripples in puddles.