Source-linked AI summary
LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows
Xiaofeng Mao, Peijia Lin, Shaohao Rui, Yibo Zhang, Haibin Wan, Weijie Ma
TL;DR
Video diffusion models are difficult to control reliably, while agentic visual creation supplies explicit conditions without ensuring object or character fidelity. LynnReal-Omni combines them in a unified multimodal framework, with competitive instruction and visual-world fidelity and lower-latency Flash deployment.
Problem
Video diffusion is stochastic and difficult to control, with repeated sampling often needed and long-horizon scenes prone to appearance, interaction, and temporal drift.
Method
LynnReal-Omni uses a shared multimodal diffusion transformer to unify multiple video-generation and editing tasks, accept heterogeneous visual conditions, and support efficient long-video and agent-guided generation.
Results
LynnReal-Omni reaches 77.76 in T2V and 79.20 in I2V, while retaining competitive instruction and visual-world fidelity as a real-time world model.
Takeaways & Limitations
The framework provides a unified, controllable basis for agentic visual creation and supports practical real-time video generation.
Takeaways & Limitations
Independent frame editing for temporal repair does not enforce temporal consistency, and reported latency measures warm execution rather than cold-start or request-to-display latency.
Abstract
from arXiv · showhide
Video diffusion models are stochastic and hard to control: precise content often requires repeated sampling without guaranteed success, and long-horizon scenes drift in appearance, interactions, and temporal coherence. Agentic visual creation provides explicit references, editable 3D scenes, or executable game states for stable control, but does not by itself guarantee high object or character fidelity. Combining the two can enable stable, high-quality generation. To realize this combination, we present LynnReal-Omni, a native multimodal video generation framework built on a 32B shared multimodal diffusion transformer that unifies text-to-video, image-conditioned generation, reference-guided generation, structural control, editing, degraded video restoration, and long-video generation. It accepts heterogeneous visual inputs, including appearance references, editable 3D renders, and game recordings, allowing agents to compose visual conditions within a unified model. We also train a dedicated 27B Flash shared multimodal diffusion transformer for real-time rendering. We build a systematic data pipeline for video cleaning, subject association, multimodal annotation, and aligned control construction, yielding a curated corpus of multi-shot audiovisual segments, and introduce MSAVP, a 100-prompt, 20-metric evaluation design that separates instruction following, generating plausibility, visual quality, temporal behavior, and audio coordination. LynnReal-Omni-Flash further reduces inference cost through model and decoding acceleration, including a lightweight VAE decoder; on one H100, warm generation and decoding of a 22-frame 540p video take 843 ms with LynnReal-Omni and 377 ms with Flash. These results provide a foundation for real-time streaming video generation, making LynnReal-Omni a unified, controllable, and efficient basis for agentic visual creation.
1 Introduction
Video diffusion offers strong visual fidelity but remains stochastic and difficult to control, while agentic visual creation supplies explicit controls without guaranteeing fidelity. LynnReal-Omni combines these strengths in one multimodal framework and targets practical, real-time generation.
- LynnReal-Omni unifies text-to-video, image-conditioned generation, reference-guided generation, structural control, editing, and long-video generation in one shared model.It replaces fragmented task-specific stacks with a shared denoiser and task-specific input layouts.
- LynnReal-Omni accepts heterogeneous references and controls while preserving their distinct roles through native multimodal representations.The representation encodes modality identity, temporal coordinates, noise levels, and output targets for appearance references, 3D renders, game recordings, and causal history.
- A systematic pipeline produces curated multi-shot audiovisual training units with subject association, multimodal annotations, and aligned conditioning assets.The pipeline covers video cleaning, subject association, annotation, and aligned control construction.
- MSAVP evaluates 100 prompts across 25 metrics while separating semantic alignment, visual quality, temporal consistency, physical plausibility, controllability, and audio quality.Its design keeps semantic compliance distinct from observed physics and maintains separate visual, temporal, and audio measures.
- 909 ms for LynnReal-Omni and 591 ms for Flash warm generation and decoding of a 22-frame 540p video on one H100.Flash reduces inference cost through model and decoding acceleration, including a lightweight VAE decoder.
2 Related Work
Prior work advances video generation, multimodal conditioning, distillation, and agentic workflows, but evaluation and deployment remain fragmented. LynnReal-Omni builds on these directions with shared conditioning and separately evaluated efficiency paths.
- Video foundation models combine spatiotemporal compression with scalable diffusion transformers, while newer systems extend generation to synchronized audio-video and shared reference-conditioned editing.LynnReal-Omni builds directly on MiniMax-H3’s joint video-audio transformer, modality-specific codecs, and native keyframe interfaces.
- Distillation research reduces denoising cost through distribution matching, trajectory matching, and autoregressive student rollouts, motivating few-step generation.Standard and Flash paths have different deployment topologies and require separate trained configurations and ablations.
- Existing video benchmarks separate some quality dimensions but do not jointly cover multi-shot continuity, action binding, physical plausibility, controllability, and audio quality.MSAVP adds structured accounting for multishot action, binding, sound semantics, and event timing.
- Agentic workflows have progressed from prompt-based image generation toward visual reasoning, video planning, executable games, and interactive applications.These systems increasingly combine reference inspection, editing, coding, and tool use.
3 Data Processing
The data pipeline converts public videos into clean, coherent, and richly annotated multishot audiovisual clips. It combines automated evidence extraction, multimodal organization, quality screening, asset construction, and consistency checks.
- 3.1 Overview: from public videos to high-quality multishot data: Approximately 0.6% of the original storage footprint remains as high-quality data after processing, with retained clips covering diverse story, action, audiovisual, and visual domains.The corpus includes clips no longer than one minute and balances varied content categories.
- 3.1 Overview: from public videos to high-quality multishot data: The five-stage pipeline segments and cleans videos, groups scenes and subjects, captions coherent clips, prepares multimodal assets, and performs consistency checks.It produces clip-level and per-shot descriptions for same-scene multishot clips of at most one minute.
- 3.2 Shot Segmentation: Shot processing distinguishes real cuts from flashes, camera shake, pans, and occlusions, while preserving continuous windows of at most 60 seconds without dropping frames or audio.Candidate boundaries are proposed with high recall and filtered using visual evidence.
- 3.3 Scene and Subject Organization: Scene grouping and subject linking enforce physical and narrative continuity and maintain stable identities across shots for multishot captions.Clips are grouped by shared place and continuous event context, not merely by person or color palette.
- 3.4 Content Selection: Quality screening retains clips with recognizable subjects, clear visuals, sustained motion, and observable event development while separating rejected from unscreened content.Duplicates are detected using source identifiers, media signatures, time ranges, and caption signatures.
- 3.5–3.7 Condition Assets and Annotation: The pipeline constructs subject, background, pose, speech, and environmental-audio references, then converts selected clips into training controls for conditional generation and editing.Specialized models provide measurable evidence, multimodal models organize context, and deterministic checks limit cross-stage errors.
- 3.7 Video Editing Dataset: A large-scale video-editing dataset substantially improves textual instruction following and text-conditioned control.
4 Method
LynnReal-Omni is a shared multimodal diffusion framework designed to generate coherent video from heterogeneous conditions while supporting efficient, long-horizon, and agent-guided workflows.
- 4.1 Overview: LynnReal-Omni combines a shared multimodal denoiser, multitask flow training, few-step distillation, long-video history interfaces, lightweight decoding, and agent-generated spatial guidance.These components jointly support heterogeneous multimodal video generation.
4.2 Native Multimodal Backbone and Task representation
LynnReal-Omni uses a shared multimodal denoiser with task-specific layouts that preserve modality roles, temporal alignment, and conditioning semantics across video-generation tasks.
- Native multimodal backbone: The backbone uses 50 transformer blocks with width 5,376, 56 attention heads of dimension 128, and 24-channel video latents patched at 1 × 2 × 2.The audio stream has 32 input channels and text conditioning has dimension 5,120.
- Native multimodal backbone: A shared denoiser predicts video and optional audio velocity from modality-tagged rows, while separate decoders reconstruct RGB frames and soundtrack.Each row carries modality identity, row indices, rotary positions, and noise times before the joint forward pass.
- Task representation: Task-specific layouts distinguish reference images, keyframes, semantic video references, frame-aligned controls, and target outputs within the unified model.Reference images use ordered picture-reference or keyframe partitions, while pose, depth, editing, and game-render controls describe the target timeline frame by frame.
- Task representation: Aligned controls are sampled on the 24-fps model clock, resized to the target canvas, and extended with their last valid frame when shorter than requested.Source frame rate, selected duration, and terminal padding must be recorded because padded frames are not new observations.
- Task representation: A constant positive rotary-coordinate offset keeps reference and generated tokens in separate domains, preserving the intended conditioning semantics.Equal absolute coordinates would change how the model interprets the conditioning relationship.
4.3 Multitask Training and Few-Step Distillation
LynnReal-Omni trains one multimodal model across aligned tasks and modalities, then distills accelerated variants while retaining separate video and audio schedules and valid-row supervision.
- Multitask training: Task sampling trains text generation, image conditioning, reference controls, editing, and long-video generation in one model with source-linked temporal alignment.Dataset windows, captions, modality masks, and target intervals are carried together to prevent controls from referring to a different segment.
- Multitask training: The objective averages squared velocity error over valid video and audio entries, excluding conditioning and padded target rows from their corresponding losses.Audio contributes only when valid target audio exists, with a fixed coefficient of 0.1 in the stated objective.
- Multitask training: Guidance-aware fitting trains only the conditional branch while retaining visual controls in the captionless prediction, allowing inference to use one conditional prediction per denoising step.The recorded training scale is w = 3, and inference does not add an unconditional forward at each step.
- Multitask training: Video and audio share denoising evaluations but use distinct modality-specific noise schedules derived from separate row-to-time mappings.In native 768P, video clean-time steps are approximately (0, 0.0270, 0.0769, 0.2000), while audio steps are (0, 0.1000, 0.2500, 0.5000).
- Flash variant and inference acceleration: Flash retains 42 of 50 transformer blocks, uses three denoising evaluations, compresses spatial video tokens to about one quarter in middle blocks, and reconstructs omitted-token updates by nearest retained neighbors.Text and audio tokens remain uncompressed, while residual connections preserve full-resolution features for later refinement.
- Flash variant and inference acceleration: Flash training uses a frozen full-depth teacher and trainable fake-score critic for trajectory distribution matching, while inference requires three student evaluations followed by codec decoding.Additional real-data flow regression and audio interval supervision are training-only computations.
4.4 Long Video Generation with Bounded History
Long-video generation proceeds in fixed-length chunks with a shared boundary frame and bounded temporal context, maintaining transitions while keeping conditioning cost independent of video length.
- Chunked generation: Each chunk conditions on preceding video context and generates 17 new frames by jointly decoding one fixed boundary latent with five future latents, then removing the duplicate boundary frame.The shared RGB boundary frame is reused across adjacent chunks to maintain temporal coherence.
- Bounded history: The temporal context keeps one earliest latent at full resolution, samples two recent latents with stride 2, and samples up to eight earlier frames with stride 4.The shared boundary frame is excluded from conditioning to avoid redundancy.
- Bounded history: The resulting context contains no more tokens than two full-resolution latent frames, keeping attention cost independent of video length.This fixed-budget representation preserves recent fine-grained dynamics alongside coarse long-range information.
- Bounded history: Once storage capacity is reached, the system retains the initial and most recent frames while discarding intermediate entries.Bounded storage and fixed-budget tokenization enable long-form generation without increasing temporal conditioning cost.
4.5 Lightweight Decoder Distillation
LynnReal-Omni distills a shallower video decoder that preserves the teacher’s latent interface while reducing decoding cost through multi-source supervision and reconstruction regularization.
- Distillation objectives: The frozen encoder constrains student and teacher re-encoded posteriors, with logarithmic KL handling and delayed cycle-loss activation improving latent consistency.The cycle loss starts after 800 training steps, is evaluated every 8 updates, and is disabled for source-reconstruction samples.
- Architecture: The student is initialized by reducing the 36-block teacher decoder to 26 transformer blocks while retaining the 2,048 hidden dimension and 4× MLP expansion ratio.Corresponding teacher and student blocks establish the feature-distillation alignment.
- Architecture: The distilled decoder preserves the teacher’s latent interface, allowing direct replacement without modifying the upstream generative model or re-encoding latent outputs.The encoder, latent dimensionality, spatial scaling, and temporal structure remain unchanged.
- Training distribution and supervision: Training mixes data-encoded and model-generated latents, adds perturbed and single-frame samples, and selects an image-specialized teacher when it improves image reconstruction fidelity.Latents come from encoded data with probability 0.6 and from the generative model with probability 0.4; image-teacher selection uses lower pixel-space L1 error.
- Distillation objectives: The objective combines teacher reconstruction, structural, feature, perceptual, latent-consistency, and paired pixel-space losses to preserve spatial detail and temporal behavior.The reconstruction term is Lrec = mean |yS − yT| + 5 mean (yS − yT)^2, while auxiliary terms regularize edges, local detail, seams, motion, and acceleration.
- Evaluation and deployment: Decoder training and inference retain tiled processing, while checkpoint selection jointly evaluates spatial fidelity and temporal consistency and separately ablates depth, precision, and tiling.The decoder uses rectangular training tiles and preserves native temporal padding and output trimming at inference.
4.6 Temporal repair of generated-video artifacts
LynnReal-Omni repairs generated-video artifacts by independently editing each source frame through its shared image-editing interface and reassembling the results in original temporal order.
- Repair procedure: The shared image-editing interface accepts a corrupted video and repair instruction without requiring a scene-specific restoration network.Instructions specify which artifacts to suppress and which scene content to retain; optional appearance references or aligned guide videos can also be used.
- Repair procedure: The repair procedure independently edits every source frame with the same seed, selects one generated image per timestamp, and preserves the original frame order and playback timing.Each image edit generates a short internal clip, from which a configurable frame is selected.
- Illustrative result: The candle example reduces background grids and speckles across 120 repaired frames, but retains brightness and shape fluctuations, residual points, and reduced flame-height variation.The example uses a standard BF16 DiT, seed 7, a 1344×768 canvas, and four denoiser evaluations per source frame.
- Limitations: The example provides qualitative evidence without clean ground truth and does not establish pixel-exact recovery or restoration quality across arbitrary scenes.Independent frame editing leaves temporal consistency as a limitation.
4.7 Agent-generated 3D scene and executable game controls
The agentic workflow constructs editable 3D scenes and executable game states from references and task descriptions, then renders controlled video through explicit simulation and camera behavior.
- Scene representation: Agents construct executable 3D scenes that encode geometry, materials, transforms, lighting, cameras, state updates, and control inputs in an editable representation.Code execution, rendered observations, and iterative refinement connect scene specification to visual output.
- Scene construction: Reference images and task descriptions guide object, environment, appearance, action, and interaction design through analysis of silhouettes, scales, occlusions, and support relationships.Shared coordinate conventions and interfaces support independent development and revision of scene components.
- Pose and camera refinement: Rendered reference-view comparisons refine camera parameters and object transforms from global composition toward part proportions, materials, and lighting.Correspondence fitting can assist parameter estimation during iterative render-and-compare refinement.
- Scene validation: Scene validation combines appearance checks with alternate-view geometry, support, intersection, measurement, and collision checks before further correction.Corrections adjust object height, assembly transforms, or local geometry and are followed by additional checks.
- Game logic and temporal control: Executable game logic produces visible motion through explicit state updates for position, velocity, orientation, action phase, acceleration, gravity, contact, collisions, and interaction events.Object poses, part animations, and cameras are updated from the simulated state.
- Game logic and temporal control: Recording scripts advance simulations at fixed output intervals, capture rendered frames, and store states and interaction events alongside them, separating recording time from playback speed.Camera trajectories or tracking rules jointly determine the output sequence with the simulation.
5 Experiments
The experiments evaluate LynnReal-Omni’s benchmark performance, reconstruction fidelity, inference efficiency, and editing behavior under controlled settings. Results show competitive MSAVP scores, near-official temporal reconstruction quality, substantial operator-fusion and decoder-speed gains, and editing demonstrations with explicitly limited claims.
- 5 Experiments: The evaluation protocol fixes source media, prompts, seeds, and relevant artifacts within paired comparisons, while reporting generation and decoding studies separately from reconstruction and editing demonstrations.The experiments distinguish reconstruction measurements from generated-video scores and avoid transferring timings across output geometries or canvas sizes.
- 5.2.1 Benchmark overview: MSAVP uses 100 fixed prompts and six equally weighted capability families, with separate instruction, physical, visual, temporal, and audio-related measurements.The benchmark emphasizes multishot consistency, physical plausibility, audiovisual coordination, and auditable checklist-based assessment.
- 5.2.2 Evaluation workflow: LynnReal-Omni scores 77.76 in T2V and 79.20 in I2V on MSAVP, while leading I2V Prompt Fidelity and showing strength in entity fidelity, world-state consistency, aesthetic quality, and imaging quality.Seedance 2.0 leads T2V at 79.39, and MiniMax-H3 leads overall I2V; LynnReal-Omni is positioned as a real-time world model rather than an offline generator.
- 5.3 Lightweight VAE Evaluation: Native-tile lightweight decoding keeps temporal-difference MAE close to the official decoder: 0.006012 versus 0.006015 on the motion clip, while the official decoder is lower on the clay clip, 0.005816 versus 0.005926.Framewise curves show broadly similar trends, including a late quality drop shared by all configurations.
- 5.4 Inference Latency: Operator fusion reduces generation wall time by 43.6% for Standard and 31.6% for Flash, while replacing the 36-block decoder with the 26-block student reduces decoding from 469 to 340 ms.These paired measurements isolate operator-fusion and decoder effects, but numerical changes do not establish equivalent visual quality.
- 5.5 Reference-guided weather and appearance editing: Reference-guided weather and appearance edits change illumination and synthesize detail hidden by fog, so they demonstrate generative editing rather than recovery of a paired clear target.The paper does not report restoration PSNR or infer improved video physics from these edits.
6 Conclusion
LynnReal-Omni unifies multimodal video diffusion with explicit controls over appearance, geometry, motion, and history. The conclusion emphasizes useful acceleration while distinguishing demonstrated capabilities from claims requiring further evidence.
- LynnReal-Omni couples native multimodal diffusion with explicit controls over appearance, geometry, motion, and history.Its data pipeline separates physical timing from semantic grouping, verifies conditioning assets, and supports multiple input layouts through a shared standard denoiser.
- Agent-produced scenes and executable games provide inspectable controls whose visual realization can be altered through reference-guided generation.
- Warm 540p measurements demonstrate useful acceleration but do not establish equivalent native 768p speed or live-interaction latency.The conclusion also distinguishes plausible image editing from repaired video physics and catalog counts from measured training exposure or completed evaluations.
7 Contributors
The report identifies its contributors and affiliations, with equal-contribution, project-lead, and corresponding-author roles marked separately. It also includes qualitative development figures alongside the contributor material.
- The listed affiliations are LynnReal Lab, Shanghai Innovation Institute, Shanghai Jiao Tong University, and Fudan University.
- Contribution metadata distinguishes equal contributors, the project lead, and the corresponding author.
- The report includes qualitative examples of recorded-game control and mesh-camera-tour conditioning at matched timestamps.The game example uses four denoiser evaluations and does not measure live input-to-display latency; the mesh example does not establish exact geometric reconstruction.