PaperlayerView exact revision on arXiv

Source-linked AI summary

Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision

Gonzalo Gómez-Nogales, Daohan Lu, Peiye Zhuang, Tinghui Zhou, Yi Zhou

arXiv:2610.28954v1cs.CV

TL;DR

Paired coarse–realistic videos are expensive to create, limiting scalable coarse-to-real training. The paper uses image pairs to build pseudo-coarse controls for training C2R# without paired videos; it improves nine of ten reported metrics over the original C2R model, though that model retains the higher realism mean.

  • Problem

    Existing coarse-to-real systems require paired training data, and aligned synthetic videos are expensive to create, with costs multiplying across coarse modalities.

    Source: §1 · pp. 1–2

  • Method

    DINO-R2C transfers realistic images to coarse DINO features and creates pseudo-coarse videos frame-wise, which train C2R# to generate the corresponding real videos.

    Source: §1 · p. 2

  • Results

    C2R# improves nine of ten reported metrics over the original C2R model, but the original model retains the higher realism mean (75.50 versus 74.17).

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    After training DINO-R2C for a coarse modality, any real-video collection can provide C2R training data, while a new modality requires image pairs rather than paired videos.

    Source: §1 · p. 2

    Performance may degrade when pseudo-coarse videos differ substantially from actual coarse inputs, and actual-coarse evaluation lacks paired realistic ground truth.

    Source: §5 · p. 9

Abstract

from arXiv · show

Coarse-to-real (C2R) video generation converts coarse animation, simulation, or game-rendered videos into photorealistic videos while preserving scene layout, camera motion, and dynamics. Training such models requires aligned (coarse, realistic) video pairs, which are expensive to create and confine prior systems to narrow synthetic domains. We exploit a simple asymmetry: paired videos are hard to obtain, but paired real-to-coarse images are easy to construct with existing editing and 3D reconstruction tools. Using these image pairs, we train the DINO-R2C model to transfer realistic images to coarse images in DINO feature space, and apply it frame-wise to ordinary real videos to synthesize pseudo-coarse videos. The C2R# model then learns to generate each real video from its pseudo-coarse counterpart, free from needing (coarse, realistic) video pairs. At inference, the DINO-R2C model is discarded and actual coarse videos drive the C2R# model directly. Thus, our pair-free training scales with any real-video corpora rather than with paired synthetic videos. We will release the model and project webpage at https://gonzalognogales.github.io/coarse2real-sharp/.

1 INTRODUCTION

Coarse-to-real video generation uses coarse inputs to control scene layout, camera motion, and dynamics while a video model supplies photorealistic appearance. This work replaces costly paired-video supervision with an image-level real-to-coarse bridge, then trains C2R# on pseudo-paired videos.

1 INTRODUCTION

  • Paired synthetic coarse–realistic videos are expensive to create, and their cost multiplies with each coarse modality, making paired-video supervision difficult to scale.The paper identifies paired training data as a major limitation of existing coarse-to-real systems.Source: §1 · pp. 1–2
  • Paired real-to-coarse images are easier to construct, but editing every frame of a large real-video corpus would incur substantial generation costs.The paper points to image editing and 3D fitting as ways to create aligned image pairs.Source: §1 · p. 2
  • DINO-R2C predicts coarse-counterpart DINO features from realistic images and applies frame-wise to real videos to create pseudo-coarse controls for C2R# training.At inference, actual coarse video drives C2R# directly, so DINO-R2C is no longer required.Source: §1 · p. 2
  • Camera augmentation reduces temporal jitter in DINO-R2C outputs, while the temporally coherent real-video target helps C2R# tolerate remaining jitter.To prevent realistic-input cues unavailable at inference from leaking into control, predicted features are decoded to pixels with a frozen RAE decoder and re-encoded.Source: §1 · p. 2
  • Once DINO-R2C is trained for a coarse modality, any real-video collection can provide C2R training data; adding a modality requires new image pairs, not paired videos.This shifts the expensive supervision requirement from videos to images.Source: §1 · p. 2

2 RELATED WORK

Prior coarse-to-real work obtains supervision from rendering pipelines, per-scene synthesis, or authored video pairs, while controllable video models use explicit or implicit structural controls. This paper combines image-level pairing with DINO features to train without paired videos.

Generative rendering from coarse structure

  • Earlier rendering-to-real approaches use adversarial enhancement, paired label-to-video synthesis, structural conditioning, or propagated keyframes, with supervision from pipelines, per-scene synthesis, or authored video pairs.The paper assumes only a coarse RGB video at test time and no paired videos.Source: §2 · p. 2

The original C2R paper

  • The original C2R predecessor uses a Wan2.1 backbone and scarce paired synthetic videos to anchor controllability, whereas this framework removes paired videos from training.The predecessor's control adapter mainly sees reconstructed real-video features during training but coarse-video features at test time.Source: §2 · p. 2

Controllable video diffusion

  • Video generators use ControlNet-style branches, unified context blocks, or warped noise for control, while explicit structural controls can over-constrain generation or leak control appearance.C2R# instead conditions on implicit DINO features that carry layout and semantics while leaving appearance to the video model.Source: §2 · pp. 2–3

Self-supervised features as a cross-domain bridge

  • This work combines image models that provide coarse–realistic image pairs with a DINO-space real-to-coarse bridge decoded through an RAE, producing pseudo-paired videos to supervise the generator.Related work includes DINO-based domain transfer and image models that synthesize paired data for editors.Source: §2 · p. 3

3 METHOD

The method replaces paired coarse-real video supervision with aligned image pairs and ordinary real videos. It converts those videos into pseudo-coarse controls for training, then uses actual coarse videos to drive the trained generator at inference.

3.1 PROBLEM SETUP

  • Training assumes ordinary real videos and aligned coarse-real image pairs, but no paired coarse-real videos.Source: §3.1 · p. 3

3.2 IMAGE-LEVEL REAL-TO-COARSE BRIDGE: THE DINO-R2C

  • For each of four coarse modalities, aligned image pairs train DINO-R2C to map realistic images to DINO features of their coarse counterparts.The pairs preserve subjects, layout, pose, and camera; training also uses paired camera homographies to stabilize mappings across viewpoint changes.Source: §3.2 · p. 4

3.3 CONSTRUCTING PSEUDO-PAIRED VIDEOS

  • Predicted DINO features are decoded to coarse pixels and re-encoded with the inference encoder before serving as training controls.This removes realistic-input cues that cannot be expressed as a coarse image and would otherwise create a train-test mismatch.Source: §3.3 · p. 5

3.4 COARSE-TO-REAL VIDEO GENERATION: THE C2R#

  • C2R# adds zero-initialized control adapters to a frozen Wan2.2 video backbone, with one adapter for each timestep-routed denoiser expert.Source: §3.4 · p. 5
  • The trainable adapters learn a rectified-flow objective on real-video latents, and inference conditions them on encoded actual coarse videos and text prompts.Neither DINO-R2C nor the RAE decoder is used at inference.Source: §3.4 · p. 5

4 EXPERIMENTS

The experiments evaluate DINO-R2C video stability and compare C2R# with prior methods across three coarse-video domains. They also examine training efficiency, pseudo-paired-data transfer, and domain transfer.

4.1 TRAINING DATA.

  • DINO-R2C synthesizes pseudo-coarse controls for 389,854 real videos, while the adapted Editto and VACE baselines use the same videos and controls.The shared DINO-R2C training set contains 14,000 aligned image pairs per coarse modality.Source: §4.1 · p. 5

4.2 DINO-R2C

  • The selected DINO-R2C recipe achieves the best smoothed-frame flicker, motion error, boundary stability, and content consistency, and ranks second in perceptual flicker.With LPIPS enabled, increasing augmentation from 1× to 2× improves all six reported video metrics; the selected setting balances reconstruction fidelity and temporal behavior rather than optimizing every metric.Source: §4.2 · p. 6

4.3 C2R# ABLATIONS AND EFFICIENCY

  • C2R# uses 180.54M trainable control and fusion adapter parameters, 0.63% of the counted model parameters, with Wan2.2 frozen.The RGB bottleneck decodes predicted features to pixels and re-encodes them; bypassing it produces visible artifacts and is excluded from quantitative comparisons.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 COMPARISONS

  • C2R# quality gains do not imply the strongest preservation on every control metric; additional plausible detail and secondary motion may contribute to source-output disagreement.The adapted Editto model has the lowest DINO displacement, while default VACE 2.2 leads local-flow agreement and camera-pose accuracy.Source: §4.4 · p. 7
  • Pseudo-paired supervision improves all ten reported aggregate metrics for adapted Editto Sim2Real, but its effect on VACE 2.2 is mixed.In CSD-L style-feature space, DINO-R2C maps real videos into four coarse style clusters, while C2R# maps actual coarse videos toward the real-reference distribution.Source: §4.4 · p. 7, Figure 6 · p. 9
  • C2R# leads DOVER technical and aesthetic quality and all three VEFX dimensions, while ranking second in background consistency and VLM-judged realism.It improves nine of ten reported metrics over the original C2R model, whose realism mean remains higher (75.50 versus 74.17); the C2R# realism score exceeds every external baseline.Source: §4.4 · p. 7, Table 2 · p. 8

5 LIMITATIONS AND DISCUSSION

The method has input-distribution and control-alignment limitations. Its frozen video prior may trade exact control alignment for natural, temporally coherent output, especially under abrupt motion changes.

5 LIMITATIONS AND DISCUSSION

  • Performance may degrade when DINO-R2C-generated pseudo-coarse videos differ substantially from actual coarse inputs, and evaluation lacks paired realistic ground truth.The authors therefore use reference-free metrics and human judgment for evaluation on actual coarse videos.Source: §5 · p. 9
  • Abrupt pose or motion changes can make C2R# adjust motion pace or synthesize intermediate states, producing more natural, coherent video but reducing alignment with the input.Figure 7 also notes that abrupt control changes can lead to unwanted content and reduced alignment.Source: §5 · p. 9, Figure 7 · p. 9
  • The implementation is constrained by the quality of the available non-commercial backbone model and training data.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The implementation builds aligned image pairs across four coarse modalities using image editing or 3D reconstruction. Each modality uses a 14,000-pair, 720p training set.

A.1 IMAGE-PAIR CORPUS

  • Each of four modalities uses 14,000 aligned coarse–realistic image pairs at 1280 × 720, with realistic source images generated by GPT-Image 2.Larger per-modality sets gave only marginal improvements.Source: §A.1 · p. 14
  • GPT-Image 2 edits realistic images into coarse geometry, coarser anime, and coarse anime targets while preserving subjects, layout, poses, and viewpoint.Source: §A.1 · p. 14
  • For coarser geometry, the pipeline reconstructs and simplifies people, objects, and terrain, then renders the composition from the source camera to preserve layout, perspective, and occlusion.People and background are processed separately, with human poses retargeted to blocky Roblox R15 characters.Source: §A.1 · p. 14

A.2 DINO-R2C ARCHITECTURE AND TRAINING

DINO-R2C predicts coarse-image features with a shared backbone and modality-specific heads, using camera augmentation to address frame-wise processing. The resulting pseudo-coarse videos are paired with real videos for training.

A.2 DINO-R2C ARCHITECTURE AND TRAINING

  • DINO-R2C uses a shared DINOv3-L/16 backbone and four modality-specific residual token-MLP heads; its frozen encoder produces a 45 × 80 × 1024 feature grid at 1280 × 720.The RAE decoder is fine-tuned for reconstruction, then frozen to decode predicted features to coarse RGB pixels during training and synthesis.Source: §A.2 · p. 14

A.3 CAMERA AUGMENTATION

  • Camera augmentation applies the same aspect-corrected homography to paired images; the selected 2× recipe doubles perturbation bounds and maximum removable crop area.The 2× recipe retains crop area in [0.50, 1.00]; the multiplier changes augmentation strength, not training-set size.Source: §A.3 · p. 14

A.4 PSEUDO-PAIRED VIDEO SYNTHESIS

  • The corpus contains 389,854 real videos, processed as 81-frame, 16-fps clips; DINO-R2C converts frames independently and the frozen RAE decoder produces pseudo-coarse videos paired with their real sources.Pseudo-paired videos are synthesized on the fly with equal sampling budgets across four modalities.Source: §A.4 · pp. 14–15

A.5 C2R# ARCHITECTURE AND TRAINING

C2R# adds control and fusion adapters to a video backbone, mapping DINO control features onto its spatial token grid. Training updates only the adapters, and inference uses a fixed denoising and guidance setup.

A.5 C2R# ARCHITECTURE AND TRAINING

  • C2R# uses two control adapters on the Wan2.2 T2V-A14B backbone, followed by a zero-initialized fusion adapter; DINO spatial cells align directly with DiT spatial tokens.The control adapters include causal temporal compression from 81 input frames to 21 latent frames.Source: §A.5 · p. 15
  • Training updates only the control and fusion adapters, optimizing 180,540,416 trainable parameters for one epoch over 389,854 eligible records.The selected run uses 12,183 optimization steps with global batch size 32.Source: §A.5 · p. 15

A.6 INFERENCE SETTINGS

  • Inference generates 81 frames at 16 fps and 832 × 480 resolution using 50 denoising steps and adaptive projected guidance at scale 10.Each step uses two forward passes with the same coarse control, one with the scene prompt and one with the default negative prompt.Source: §A.6 · p. 15

B.1 EVALUATION SETS

The evaluation uses 300 coarse control videos across game renders, geometry blockouts, and stylized animations, with equal weight per video in aggregate metrics.

B.1 EVALUATION SETS

  • The three evaluation sets contain 100 videos each, covering game renders, geometry blockouts, and stylized animations.Together, they provide 300 control videos for all compared methods.Source: §B.1 · p. 15
  • The game-render set comprises recordings of Roblox game renderings, while all 100 stylized animations were generated with Wan 2.2 T2V.The geometry-blockout set contains Roblox Studio scenes prototyped with an agent; all 100 scenes were created in under two hours total.Source: §B.1 · p. 15
  • Table 2 aggregates metrics across all 300 videos with equal weight per video.Source: §B.1 · p. 15

B.2 COMPARED METHODS

The compared methods use the same pseudo-paired training data, coarse modalities, sampling budgets, and media format.

B.2 COMPARED METHODS

  • C2R# and the adapted VACE 2.2 and Editto Sim2Real baselines are fine-tuned on the same 389,854 real videos paired with DINO-R2C-generated pseudo-coarse controls.Source: §B.2 · p. 16
  • All three methods use the same four coarse modalities with equal sampling budgets.Source: §B.2 · p. 16
  • All three methods use 832 × 480, 81-frame, 16-fps media.Source: §B.2 · p. 16

B.3 METRIC DEFINITIONS

The appendix defines image-level reconstruction metrics, video-level temporal and control metrics, and the measures used for the domain-transfer visualization.

B.3 METRIC DEFINITIONS

  • Image metrics compare each prediction with its paired coarse target across 1,000 held-out pairs; latent MSE uses the native feature grid, while other metrics use resized decoded images.The DINO-R2C model operates at 1280×720; decoded images are clipped to [0, 1] and resized to 256 × 256 for the other metrics.Source: §B.3 · p. 16
  • Latent MSE measures feature-grid reconstruction; RGB MSE and RGB L1 measure pixel error, with L1 less sensitive to a few large differences.Latent MSE and pixel-space MSE use different scales and should not be compared numerically.Source: §B.3 · p. 16
  • LPIPS measures perceptual distance, DINO patch cosine measures local feature agreement, and depth RMSE measures aligned relative-depth agreement rather than measured distances.SSIM assesses luminance structure, while boundary F1 measures matched Sobel edges within a two-pixel tolerance.Source: §B.3 · p. 16
  • Video metrics compare motion-aligned neighboring output frames with source-video motion, using RAFT-Small correspondences and a forward-backward consistency mask.The mask retains in-bounds correspondences whose forward-then-backward flow returns near the starting pixel.Source: §B.3 · p. 17
  • Relative perceptual and smoothed-frame flicker divide motion-aligned output errors by source errors; values below one indicate less error under that normalization.Spatial smoothing reduces sensitivity to fine texture, so its flicker score should be read alongside motion and content measures.Source: §B.3 · p. 17
  • EPE measures output-versus-source flow-vector distance, motion magnitude ratio measures flow amount, and content consistency measures consecutive-frame DINO similarity.Boundary stability tracks outline persistence; content consistency can hide local flicker or reward overly static output.Source: §B.3 · p. 17
  • Boundary stability measures overlap of motion-aligned output edge maps across neighboring frames, not agreement with ground-truth coarse boundaries.The soft Dice score is higher when outlines persist more consistently.Source: §B.3 · p. 17
  • C2R# assessment separates technical and aesthetic quality, editing scores, temporal consistency, control alignment, and VLM-judged realism.The DINO, flow, and camera measures assess spatial correspondence, local flow agreement, and relative camera-pose error; VLM realism scores visible realism, not control similarity.Source: §B.3 · pp. 17–18

B.5 ADDITIONAL DINO-R2C RESULTS

The appendix reports DINO-R2C ablations and qualitative examples, parameter and efficiency information, a reference-conditioned extension, and a qualitative comparison with Seedance 2.5.

B.5 ADDITIONAL DINO-R2C RESULTS

  • The 2× augmentation recipe with LPIPS achieves the best latent MSE, RGB L1, LPIPS, DINO patch cosine, and depth RMSE, but other recipes lead on RGB MSE, SSIM, or boundary F1.Table 3 compares six recipes on 1,000 held-out Coarse geometry image pairs.Source: §B.5 · p. 19, Table 3 · p. 19
  • Figure 8 presents decoded coarse predictions for four modalities, illustrating modality-dependent appearance transfer while retaining scene structure.Pseudo-Coarse videos and additional image-pair results are available on the project page.Source: Figure 8 · p. 19, §B.5 · p. 19

B.6 EFFICIENCY AND COMPUTATIONAL COST

  • C2R# optimizes 180.54M adapter parameters, or 0.63% of counted model parameters, while keeping the Wan2.2 backbone frozen.Table 4 distinguishes total model capacity from parameters optimized during adaptation.Source: §B.6 · p. 20, Table 4 · p. 20
  • Table 5 reports model size, training hardware and cost, and native-720p inference performance for the shared four-head DINO-R2C model.Reported inference timings include the projector and frozen RAE decoder but exclude loading, transfers, codecs, and storage I/O.Source: §B.6 · p. 20, Table 5 · p. 20

B.7 REFERENCE-CONDITIONED GENERATION

  • A separately trained C2R# extension accepts a reference image for additional appearance cues while the coarse video controls layout and motion.The extension is evaluated separately from standard control-video-and-text comparisons.Source: §B.7 · p. 21, Figure 9 · p. 21

B.8 QUALITATIVE COMPARISON WITH SEEDANCE 2.5

  • In first-attempt examples using the same controls and prompts, C2R# appears more photorealistic, while Seedance 2.5 may retain coarse style or change scene layout.The comparison covers game renders, geometry blockouts, and stylized animations; the authors report that commercial editing retries are costly.Source: §B.8 · p. 22, Figure 10 · p. 22
Loading 2610.28954v1…