Source-linked AI summary
RealMaster: Lifting Rendered Scenes into Photorealistic Video
Dana Cohen-Bar, Ido Sobol, Raphael Bensadoun, Shelly Sheynin, Oran Gafni, Or Patashnik, Daniel Cohen-Or, Amit Zohar
TL;DR
Rendered 3D scenes provide precise control and consistency but lack photorealism, while video generation models provide photorealism without equivalent structural guarantees. RealMaster builds paired data through anchor-based propagation, distills the process into an IC-LoRA, and significantly outperforms video editing baselines on complex GTA-V sequences while preserving geometry, dynamics, and identity.
Problem
Video generation models lack precise control and guaranteed geometric consistency, whereas 3D-engine renderings often remain visually unphotorealistic, creating a sim-to-real gap.
Method
RealMaster enhances the first and last rendered frames, propagates their appearance across intermediate frames with edge-conditioned video generation, and trains an IC-LoRA on the resulting paired videos.
Results
RealMaster significantly outperforms state-of-the-art video editing baselines on diverse GTA-V sequences in preserving input structure and improving photorealism.
Takeaways & Limitations
The framework treats video diffusion as a learned second-stage renderer atop explicit 3D engines, separating structural control from visual realization.
Takeaways & Limitations
Output realism is bounded by current image editing models, and simulator-inherited locomotion, gestures, and pose dynamics may remain implausible.
Abstract
from arXiv · showhide
State-of-the-art video generation models produce remarkable photorealism, but they lack the precise control required to align generated content with specific scene requirements. Furthermore, without an underlying explicit geometry, these models cannot guarantee 3D consistency. Conversely, 3D engines offer granular control over every scene element and provide native 3D consistency by design, yet their output often remains trapped in the "uncanny valley". Bridging this sim-to-real gap requires both structural precision, where the output must exactly preserve the geometry and dynamics of the input, and global semantic transformation, where materials, lighting, and textures must be holistically transformed to achieve photorealism. We present RealMaster, a method that leverages video diffusion models to lift rendered video into photorealistic video while maintaining full alignment with the output of the 3D engine. To train this model, we generate a paired dataset via an anchor-based propagation strategy, where the first and last frames are enhanced for realism and propagated across the intermediate frames using geometric conditioning cues. We then train an IC-LoRA on these paired videos to distill the high-quality outputs of the pipeline into a model that generalizes beyond the pipeline's constraints, handling objects and characters that appear mid-sequence and enabling inference without requiring anchor frames. Evaluated on complex GTA-V sequences, RealMaster significantly outperforms existing video editing baselines, improving photorealism while preserving the geometry, dynamics, and identity specified by the original 3D control.
1 Introduction
RealMaster addresses sim-to-real video translation by combining the precise structural control of 3D engines with the photorealism of video diffusion models. It targets the joint preservation of scene structure and dynamics alongside global appearance transformation.
- Video generation models offer photorealism but lack precise scene control and guaranteed geometric consistency across frames.
- 3D engines provide granular control and native geometric consistency, yet rendered outputs often retain a sterile, uncanny-valley appearance.
- Sim-to-real translation must preserve the rendered scene’s structure and dynamics while transforming its appearance globally.
- RealMaster lifts rendered video into photorealistic video through sparse-to-dense appearance propagation and an IC-LoRA trained on paired rendered–photorealistic videos.
- On diverse GTA-V sequences, RealMaster outperforms state-of-the-art video editing baselines in both input preservation and photorealism.
- Existing video editing methods struggle because they either make minimal changes to rendered inputs or alter scene structure and character identity excessively.
3 Method
RealMaster constructs paired training videos by enhancing sparse keyframes and propagating their appearance with edge-based structural conditioning, then distills this process into an IC-LoRA model. The trained model performs direct sim-to-real translation while preserving structure, dynamics, and identity beyond the pipeline’s anchor-frame constraints.
- Method overview: The method first constructs paired training data, then trains an IC-LoRA adapter to translate rendered videos directly at inference time.
- Data generation: The data pipeline addresses the lack of aligned rendered and photorealistic videos by constructing photorealistic counterparts from rendered sequences.
- Data generation: The first and last rendered frames are edited into photorealistic appearance anchors that define the target look for the sequence.
- Data generation: VACE propagates anchor appearance to intermediate frames using edge maps extracted from the input video, preserving scene layout and dynamics.
- Model training: An IC-LoRA adapter is fine-tuned over a pretrained text-to-video diffusion backbone to distill the propagation pipeline into a lightweight sim-to-real model.
- Inference: The distilled model removes the pipeline’s need for first and last frames and improves handling of objects and characters emerging mid-sequence.
- Implementation: The implementation generates paired clips with edge-conditioned VACE propagation and trains Wan2.2 T2V-A14B using a rank-32 LoRA adapter.
- Results: Qualitative results show improved material and lighting realism while preserving scene structure, motion, character identity, and appearance.
4 Experiments
RealMaster is evaluated against video-editing baselines using automatic metrics, human preferences, qualitative comparisons, and ablations. Across these evaluations, it improves photorealism while preserving scene structure, identity, and temporal consistency, with edge-conditioned training pairs and model training providing the strongest results.
- Qualitative Results: RealMaster transforms rendered videos toward photorealism while preserving scene structure, motion, character identity, and appearance.The method remains effective in dynamic, cluttered scenes with interacting characters, camera motion, and frequent occlusions.
- Qualitative Results: RealMaster preserves structure and identity more reliably than Runway-Aleph, LucyEdit, and Editto while substantially improving visual realism.The baselines introduce color shifts, identity changes, game-like appearance, or significant scene-content deviations.
- Quantitative Comparison: RealMaster achieves the highest GPT-RSno-ref and GPT-RSref scores, the best ArcFace score, and the lowest DINO score among evaluated methods.These results indicate stronger photorealism, character-identity preservation, and structural fidelity on the benchmark.
- Quantitative Comparison: RealMaster matches the best Temporal Flickering score and achieves comparable Motion Smoothness, while LucyEdit’s slightly higher smoothness accompanies video blurring.Blurring can reduce high-frequency detail and degrade structural precision despite improving the smoothness metric.
- User Study: Participants preferred RealMaster over all baselines for realism, faithfulness, and overall visual quality across 675 pairwise comparisons from 45 participants.Each trial compared RealMaster with one baseline alongside the original rendered input.
- Ablation Studies: Edge conditioning preserves object boundaries and facial details more reliably than multiple anchors or depth, producing stronger pipeline scores and stable temporal metrics.Multiple anchors introduce flickering, while depth can miss identity- and expression-related high-frequency cues.
- Ablation Studies: Training RealMaster on edge-propagated video pairs further improves all metrics, with the largest gains in structure and temporal consistency.The trained model also generalizes beyond the data-generation pipeline when objects appear between boundary anchors.
5 Additional Applications
RealMaster extends sim-to-real translation beyond standard appearance transfer by supporting dynamic weather effects and cross-simulator generalization. The model introduces realistic environmental details and transfers from GTA-V-trained data to CARLA scenes.
- Dynamic Weather Effects: RealMaster can add dynamic rain or snow through text prompts at inference time.The resulting effects include wet-surface reflections, falling raindrops, and snow accumulation.
- Cross-Simulator Generalization: The model generalizes from GTA-V training data to rendered videos from the CARLA simulator without additional training.CARLA contains egocentric driving scenes with vehicles, unlike the third-person, character-centric SAIL-VOS training data.
- Cross-Simulator Generalization: This cross-simulator performance suggests that RealMaster learns a general rendered-to-real appearance mapping rather than overfitting to its training domain.
6 Discussion, Limitations and Future Work
The discussion frames RealMaster as a learned second-stage renderer that combines explicit structural control with photorealistic realization. It also identifies limits from anchor-generation models and simulator-inherited motion, while proposing real-time and motion-focused extensions.
- Discussion: RealMaster separates structural control from visual realization, combining 3D-engine determinism and editability with learned real-world appearance priors.
- Discussion: Maintaining identity, materials, and fine-scale details across frames is presented as essential to realism alongside improved texture and lighting.The paper attributes this consistency to anchoring generation to the underlying rendered structure and to structure-constrained paired supervision.
- Limitations: Output realism remains bounded by the capabilities of the image-editing models used to create photorealistic anchors and may fall short of full photorealism.
- Limitations: Motion and dynamics are preserved from the rendered input, but implausible locomotion, gestures, or pose dynamics are not explicitly modeled or refined.This may limit realism when the simulator's underlying animation is itself implausible.
- Future Work: Future work includes causal real-time streaming and learned priors over body dynamics and gestures to address motion realism.
A Failure Cases
RealMaster exhibits two main failure modes: conservative enhancement of small distant objects and temporal artifacts under fast motion. Both limitations are inherited from components of the data-generation or diffusion pipeline.
- Failure Modes: Scenes with many small, distant objects receive overly conservative enhancement, producing subtle changes that are difficult to notice at full-frame resolution.This behavior is inherited from Qwen-Image-Edit, which similarly struggles to enhance small objects.
- Failure Modes: Fast camera or character motion can produce temporal artifacts because the base video diffusion model was not designed for large inter-frame displacements.
B Additional Implementation Details
This appendix section covers implementation details for RealMaster's data-generation pipeline and LoRA training.
- Data Generation: The section provides additional implementation details for the data-generation pipeline.
- LoRA Training: The section also provides additional implementation details for LoRA training.
- Scope: Its scope is implementation detail rather than a new experimental result or application.
B.1 Data generation pipeline
The data-generation pipeline creates paired rendered–photorealistic clips by editing endpoint frames and propagating their appearance through each sequence, then filters pairs for identity consistency.
- Clip preparation: 81-frame clips are upsampled from 8 fps to 16 fps at 800 × 1200 resolution before photorealistic editing.The first and last frames are edited with Qwen-Image-Edit using a photorealism prompt.
- Anchor propagation: VACE propagates the appearance of edited endpoint anchors to intermediate frames using each input frame’s edge representation.This produces photorealistic sequences aligned with the rendered inputs.
- Identity filtering: ArcFace filtering retains clips whose mean face cosine similarity exceeds 0.4, preserving 1,216 of 3,050 initial clips.Approximately 60% of generated data is removed by this identity-consistency filter.
B.2 Model training
RealMaster is fine-tuned with IC-LoRA by conditioning denoising on rendered clips while learning photorealistic targets, using the reported training configuration and baseline settings.
- IC-LoRA training: Wan2.2 T2V-A14B is fine-tuned with an IC-LoRA setup using rendered clips as clean reference tokens and photorealistic clips as denoising targets.Rendered tokens use timestep t=0 and share positional encoding with noisy target tokens.
- Training configuration: The training hyperparameters for fine-tuning RealMaster are summarized in Table 3.
- Baseline settings: Runway-Aleph, LucyEdit, and Editto are evaluated with the prompt “make the video look photorealistic” and their authors’ default settings.
C Evaluation Metric Details C.1 GPT-RS
GPT-RS evaluates photorealism and, when a rendered reference is provided, faithfulness to the original scene. Scores use a 1–10 rubric focused on realism and structural consistency.
- Evaluation settings: GPT-RSwith-ref supplies both rendered and edited frames, whereas GPT-RSno-ref supplies only the edited frame.The two settings assess realism relative to the input versus realism in isolation.
- Faithfulness criteria: The faithfulness evaluation checks scene layout, camera viewpoint, object positions, colors, proportions, and major geometry.
- Photorealism rubric: The photorealism rubric scores edited frames from 1 to 10, ranging from indistinguishable real footage to obviously synthetic or broken output.
- Photorealism criteria: Photorealism judgments examine geometry stability, lighting and shadows, materials and textures, fine detail, text, signage, and neural artifacts.