PaperlayerView exact revision on arXiv

Source-linked AI summary

SolarRenderer: Few-Step Generative Reference Rendering

Guian Fang, Shengju Qian, Dongyue Lu, Junchao Huang, Lutao Jiang, Xin Wang, Yiren Song, Li Jiang, Mike Zheng Shou

arXiv:2610.28630v1cs.CV

TL;DR

Existing renderers are tied to particular structural representations, can require many denoising steps, and often lack frame-aligned training pairs. SolarRenderer uses a single control LoRA for untyped structural references and post-trains a pretrained four-step LoRA; it reports state-of-the-art results across two benchmarks and 12.3 times faster rendering than 50-step sampling.

  • Problem

    Existing renderers are tied to one engine’s representation, can require tens of denoising steps or costly distillation, and often train on conditions not aligned frame by frame with target videos.

    Source: §1 · p. 2

  • Method

    SolarRenderer puts untyped structural videos through a frozen backbone’s native reference pathway and uses structure-guided distribution matching to post-train a pretrained four-step LoRA.

    Source: §1 · p. 2

  • Results

    SolarRenderer achieves state-of-the-art results on in-distribution and generalization benchmarks, with 12.3 times faster rendering than 50-step sampling and generalization to 11 of 16 representations unseen in training.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    Its block-causal variant extends structure-conditioned rendering to 768p video in real time.

    Source: §1 · p. 2, §6 · p. 14

Abstract

from arXiv · show

Game engines and coding agents can specify the layout, motion, and camera of a scene, yet turning such coarse structure into realistic video requires a renderer that follows the structure and completes the appearance it leaves open. Existing renderers are tied to the representation of one engine, need tens of denoising steps or a costly distillation of their own, and learn from conditions that are not aligned with their target videos. We present SolarRenderer, a structure-conditioned video renderer that adapts a frozen MiniMax-H3 33B backbone with two lightweight LoRAs. Structural videos enter the native reference pathway of the backbone at the positions of the frames they describe, without a type identifier and described only in plain text, so that a single control LoRA follows arbitrary structural representations. Starting from a pretrained four-step LoRA, structure-guided distribution matching renders in four steps, 12.3 times faster than 50-step sampling, while preserving the learned control. A block-causal variant extends the same design to streaming and renders 768p video in real time. To train the model, we build a frame-aligned corpus of 136K clips from game engines, simulators, and YouTube videos. Experiments on an in-distribution test set and a generalization benchmark of 16 structural representations show that SolarRenderer achieves state-of-the-art results across both, and generalizes to 11 unseen representations, including pose skeletons, drawings, and point clouds.

1 Introduction

SolarRenderer addresses representation-specific control, slow rendering, and frame-misaligned training data with a single structure-conditioned renderer. It combines untyped structural references, four-step post-training, and a frame-aligned corpus.

1 Introduction

  • A single control LoRA follows arbitrary structural representations through the backbone’s native reference pathway, without type identifiers or added branches.Structural videos are described in plain text and placed at the positions of the target frames they describe.Source: §1 · p. 2
  • Structure-guided distribution matching post-trains a pretrained four-step LoRA to render in four steps while preserving structural control.This is 12.3 times faster than 50-step sampling.Source: §1 · p. 2
  • The 136K-clip corpus pairs structural and target frames in alignment, drawing on game engines, simulators, and YouTube videos.The sources provide structures exported by engines and simulators or fitted to YouTube videos.Source: §1 · p. 2
  • SolarRenderer achieves state-of-the-art results on an in-distribution test set and a generalization benchmark, while its causal variant renders 768p video in real time.The generalization benchmark spans 16 structural representations, 11 unseen during training.Source: §1 · p. 2

2 Related Work

Prior work renders engine state through representation-specific systems, injects control through branches or condition tokens, or accelerates generation with distillation and adapters. SolarRenderer’s approach differs by using one LoRA for untyped structure and post-training an acceleration adapter on the control model.

Rendering Engine and Agent State into Video

  • Engine-driven renderers generally support their engine’s exported representation, so a new engine or proxy calls for a new renderer.Prior approaches use white-box frames, semantic maps, depth and normal maps, projected boxes, proxy videos, or full G-buffers.Source: §2 · p. 3

Structure-Conditioned Video Generation

  • Control methods commonly add branches or control blocks, place condition tokens beside target tokens, or use other steering mechanisms.Some methods label condition types or train on a fixed set of condition kinds; SolarRenderer instead uses untyped structural video with one LoRA.Source: §2 · p. 3

Few-Step Distillation and Acceleration Adapters

  • Few-step generation uses distillation or plug-in acceleration LoRAs, but unchanged generic adapters can lower a control model’s visual quality and first-frame fidelity.SolarRenderer post-trains a four-step LoRA on the frozen control model using a teacher extrapolated toward its structure-conditioned prediction.Source: §2 · p. 3

3 SolarRenderer: Native Reference Rendering

SolarRenderer adapts a frozen video backbone to render targets that follow structural videos while completing appearance the structure leaves unspecified. It combines frame-aligned, untyped reference conditioning with separate control and acceleration adapters, structure-guided four-step rendering, and a streaming extension.

3.1 Problem Formulation and Overview

  • Given structural video, an optional first frame, and text, SolarRenderer renders a target that follows the structure where informative and completes unspecified appearance.Structural video specifies scene state such as layout, geometry, character bodies, motion, and camera; materials, illumination, and small structures remain undetermined.Source: §3.1 · pp. 3–4

3.2 Frame-Aligned Reference Conditioning

  • The control adapter reads structural references through the backbone’s native reference pathway, with each reference token assigned the position of the target token it describes.References remain untyped; fixed text descriptions specify what each representation shows and which attributes to follow.Source: §3.2 · pp. 4–5

SolarRenderer

  • SolarRenderer separates control adaptation from acceleration adaptation: it trains a control LoRA, then updates a pretrained acceleration LoRA while keeping the control adapter frozen.This decouples few-step distillation from the learned structural control.Source: SolarRenderer · p. 5
  • The training corpus combines 136K clips from eight sources, pairing each source’s condition with its ground-truth frame.The corpus figure also depicts clips by input combination and caption-related scene and time-of-day information.Source: Figure 3 · p. 6

25.3K CARLA

  • Structure-guided distribution matching amplifies the teacher’s structural signal so the four-step student is pushed toward a sharper, more decisive target than the teacher alone.The structural-video tokens are removed for the comparison prediction while the first frame and prompt stay unchanged; w = 1 recovers plain distribution matching, and the method uses w = 1.5.Source: 25.3K CARLA · p. 6

3.4 Causal Streaming Extension

  • The block-causal extension generates blocks of at most five latent frames without attending to later blocks, then outputs each block after four-step sampling.With a streaming VAE decoder, it renders 768p video in real time at up to 28 FPS.Source: §3.4 · pp. 6–7

4 Engine-Aligned Paired Data

SolarRenderer is trained on paired clips whose structural conditions match their target frames, using diverse engine, simulator, and fitted-video sources. The data pipeline filters misaligned or insufficient conditions and fixes input combinations so prompts describe only supplied inputs.

4 Engine-Aligned Paired Data

  • 136K samples totaling about 243 hours form the training set, with every clip a 158-frame window at 24 fps and 768p.The sources include game engines, simulators, and YouTube videos.Source: §4.1 · p. 7, §4 · p. 7

4.1 Data Sources

  • Engine and simulator sources supply varied controls, while YouTube clips use fitted SMPL body videos to specify protagonist position, pose, and motion.WildWorld and OmniWorld contribute appearance data with only a first-frame condition because their structural annotations do not align closely to frames.Source: §4.1 · p. 7

4.2 Data Checks

  • Temporal checks reject nonzero motion-offset peaks, while spatial checks reject conditions whose target-edge recall does not exceed that of a mirrored condition.Additional filters remove near-duplicates, dark or low-contrast clips, and conditions covering too little of the frame; samples are manually inspected.Source: §4.2 · pp. 7–8
  • Conditions are retained as structure only when aligned and sufficiently informative: coarse boxes omit geometry, while depth shading conveys layout and object shape.Training mixes structure-plus-first-frame (67.1%), structure-only (16.7%), first-frame-only (13.2%), and text-only (2.9%) samples, with prompts fixed to each supplied input combination.Source: §4.2 · p. 8, §4.3 · p. 8

4.3 Prompts and Input Combinations

  • Captions combine target-video descriptions with fixed, code-generated reference descriptions that identify each condition and the attributes the output should retain.The target-video description is generated after watching only the target video; the fixed reference text remains identical across clips with the same input combination.Source: §4.3 · p. 8

5 Experiments

SolarRenderer leads the reported comparisons across both test sets and transfers to unseen structural representations, while its causal variant trades some overall rating for streaming generation. Ablations indicate that guided post-training, native reference conditioning, and frame-aligned structural data matter to the reported results.

5.2 Comparison with Existing Methods

  • 1441 overall on the generalization benchmark ties LynnReal-Omni, while SolarRenderer leads it by 100 points in-distribution and has about 95 points higher structure ratings on both sets.The four-step model rates 1441 versus 1389 and 1510 versus 1299 overall against its 50-step Stage 1 model on the generalization and in-distribution sets, respectively.Source: Comparison with Structure-Conditioned… · pp. 10, 12

Comparison with Structure-Conditioned Renderers. SolarRenderer

  • The causal variant rates 1191 and 1199 overall on the two test sets, below the bidirectional model’s 1441 and 1510 but above Streaming CWM and StreamRender-H3.It renders blocks from past frames only and outputs each block when sampled; its structure rating remains close to the bidirectional model, while later blocks drift from first-frame colours.Source: Comparison with Structure-Conditioned… · pp. 10–11
  • On every difficulty tag, SolarRenderer beats compared methods except its own 50-step model on fast camera motion (-0.10); its margins over engine-specific renderers are +0.67 to +1.00.On unseen representations, it scores +0.46 over LongLive-Plug and +0.70 over Streaming CWM, and has the best structure rating among methods on those cases.Source: Figure 5 · p. 9, Comparison with Structure-Conditioned… · p. 11
  • Across representation categories, SolarRenderer leads LongLive-Plug by +0.25 to +0.69 and beats its own 50-step model on body and pose (+0.45) and drawings (+0.41).The benchmark covers 16 representations, including 11 unseen in training.Source: §5.1 · p. 9, Figure 6 · p. 10, Comparison with Structure-Conditioned… · p. 11

5.3 Ablation Studies

  • Structure-guided post-training raises DMAD’s overall rating from 1024 to 1156, while unguided matching lowers it to 820; guidance also improves LongLive-Plug and self-distillation starts.Guidance only at re-noising levels σ ≥ 0.9 rates 1107 versus 1096 with guidance at every level, while two and eight sampling steps rate 961 and 1080 versus the four-step model.Source: §5.3 · pp. 12–14
  • Native reference conditioning scores 48.0 for structure versus 46.8 with a ControlNet branch, and full-resolution structure input scores 48.0 versus 46.6 at half resolution.Using misaligned WildWorld boxes lowers the structure score from 47.4 to 46.6; training only on structured samples lowers stability, and reducing data to one tenth changes scores by at most 0.2.Source: §5.3 · p. 14

6 Conclusion

SolarRenderer renders structure from game engines and coding agents into realistic video through a frozen backbone and lightweight adapters. Its reported results cover in-distribution and generalization benchmarks, including unseen representations, with four-step rendering and a real-time causal variant.

6 Conclusion

  • SolarRenderer uses a frozen MiniMax-H3 backbone, untyped structural references, a single control LoRA, and structure-guided matching to adapt a pretrained four-step LoRA.It is trained on 136K frame-aligned clips and generalizes to unseen structural representations.Source: §6 · p. 14
  • The method achieves state-of-the-art results on both test sets and renders 12.3 times faster than 50-step sampling, while its block-causal variant renders 768p video in real time.Source: §6 · p. 14

A.1 Implementation Details

Implementation uses a frozen MiniMax-H3 33B Ref2VA backbone with separate control, acceleration, and fake-score LoRAs. The causal control and acceleration adapters follow distinct training schedules.

A.1 Implementation Details

  • The frozen MiniMax-H3 33B Ref2VA backbone receives a rank-384 control LoRA trained for 38,400 steps and a rank-384 four-step LoRA post-trained for 800 iterations.The four-step LoRA starts with 128 ranks from pretrained DMAD and uses w = 1.5; the generator and fake-score model update at a 1:5 ratio.Source: §A.1 · p. 15
  • The causal control LoRA trains for 19,200 teacher-forced steps, then the causal acceleration LoRA trains for 1,600 iterations with causal control frozen.The reported schedules are for one 8-GPU node; released weights used as many samples at a larger batch.Source: §A.1 · p. 15

A.2 Evaluation Protocol and Metrics

The protocol combines appearance, temporal stability, structural adherence, and first-frame preservation measures, then aggregates metric scores and uses pairwise judgments to compare renderings.

1. Visual Quality

  • Three learned predictors rate the appearance of each rendering: DOVER, the LAION aesthetic predictor, and VideoScore2.Their reported scales are [0, 1], [1, 10], and five quality levels, respectively.Source: §1 · p. 15

2. Temporal Stability

  • Flicker measures frame-to-frame change after optical-flow warping and low-pass filtering, with lower values indicating less flicker.Only pixels with agreeing forward and backward flow are compared.Source: §2 · p. 16

3. Structural Adherence

  • Structural adherence is assessed through tracked-point motion error and edge F1 between rendered and structural videos.Motion error uses reliable CoTracker3 tracks; edge F1 matches Canny edges within 2 px.Source: §3 · p. 16

4. First-Frame Preservation

  • First-frame preservation evaluates whether the given frame’s appearance persists through the clip, using frames from the last second.The displayed equation defines the measure over that final-second frame set.Source: §4 · p. 16

5. Automatic Score

  • Each metric is mapped to a fixed 0–100 scale, and each evaluation group’s automatic score is the mean of its metrics.The metric-specific input ranges differ, and structure metrics for the official backbone and Wan2.2-VACE-Fun use the frames they render.Source: §5 · p. 17

6. Pairwise Judge and Elo

  • Gemini 3.8 Flash compares pairs on visual quality, structural and motion adherence, first-frame preservation, and overall preference; Elo ratings are fit separately for each question.Pairwise verdicts are made in both presentation orders, and ties count as half a win in the Bradley–Terry fit.Source: §6 · p. 17

A.3 Compared Methods and Generation Settings

The comparison includes structure-conditioned renderers, unadapted and Stage 1 baselines, and acceleration LoRAs attached to the control model, using each method’s released or specified generation settings.

A.3 Compared Methods and Generation Settings

  • The comparison groups methods into structure-conditioned renderers, baselines, and existing acceleration LoRAs, and scores outputs after resizing to 1344 × 768 at 24 fps.Each method otherwise uses its released resolution, frame count, sampler, and step count.Source: §A.3 · p. 18

A.3.1 Structure-Conditioned Renderers

  • Structure-conditioned baselines include four-step LynnReal-Omni, two-step streaming StreamRender-H3, four-step Streaming CWM, and 40-step PAI ControlNet-Union.Their inputs and frame handling differ: LynnReal-Omni uses the first frame as a style reference, StreamRender-H3 uses an 832 × 480 reference, Streaming CWM concatenates ten 17-frame chunks, and PAI uses the first frame for inpainting.Source: §A.3.1 · p. 18
  • Cosmos-Transfer2.5 uses 35-step generation on 93-frame chunks, while Wan2.2-VACE-Fun generates 81 frames at 16 fps with 50 steps.Both use the first frame as a style or start-image reference, respectively.Source: §A.3.1 · p. 18

A.3.2 Baselines

  • The official MiniMax-H3 Ref2VA backbone receives the structural video as an ordinary reference and is sampled for 124 frames at 1344 × 768 and 24 fps with 50 steps.Stage 1 is the paper’s control model before post-training, also sampled with 50 steps.Source: §A.3.2 · p. 18

A.3.3 Existing Acceleration LoRAs

  • LongLive-Plug, DMAD, and FastH3 are each attached to the Stage 1 control LoRA without further training and sampled in four steps.These are acceleration LoRAs trained on the backbone.Source: §A.3.3 · p. 18

B Data and Prompt Specifications

The appendix describes caption construction and validation, then reports alignment and selection audits for engine and YouTube sources.

B Data and Prompt Specifications

  • The appendix illustrates training captions for Magpie block-outs, UnrealZoo object masks and skeletons, YouTube SMPL bodies, WildWorld first frames, and CARLA semantic maps.The cited figure descriptions identify these source-condition examples.Source: Figure 12 · p. 19, Figure 13 · p. 19, Figure 14 · p. 20

B.1 Training Captions

  • The training captions use the backbone’s six-section prompt format, combining a target-video description with fixed reference definitions.The fixed definitions explain what each structural reference shows and which attributes to follow or omit.Source: §B.1 · p. 19

B.2 Caption Validation

  • Caption validation requires all six fields, defined reference tags, a 350–500-word body, and increasing shot times, with up to three regeneration attempts.When a first frame is supplied, the first shot must begin from it.Source: §B.2 · p. 19

B.3 Source Audit Details

  • The Magpie synchronization audit retained 32,027 of 44,266 clips; most rejections involved shot changes, static content, structural mismatch, or darkness.Near-duplicates, black frames, and unmatched jumps accounted for additional rejections.Source: §B.3 · p. 20
  • For engine recordings, 80 of 84 motion windows had their best alignment offset at zero, and depth-and-camera reprojection had a median error of 0.06%.Near-range depth conversion retained 13,700 of 13,702 clips; a finer proxy was not adopted because flicker was four to seven times higher.Source: §B.3 · p. 20
  • The YouTube subset kept 137 of 559 videos with a clear protagonist, and the target and SMPL-fitting streams differed by about 2 grey levels per pixel.The mesh videos matched the target in crop, resolution, length, and frame rate.Source: §B.3 · p. 20

C.1 Detailed Results

SolarRenderer performs strongly on unseen representations and across the reported pairwise and automatic evaluations, while results vary by representation and evaluation dimension.

Generalization to Unseen Representations

  • On 74 unseen-representation cases, SolarRenderer has the highest quality and structure ratings, including a 1360 structure rating against 1275 for LynnReal-Omni.Its overall rating is six points below that competitor, while its quality rating is higher and first-frame rating slightly lower; it also leads the total automatic score at 70.6, versus Stage 1 at 70.4.Source: §C.1 · pp. 20–21

Pairwise Comparison with Existing Methods

  • Pairwise margins are nonnegative on both test sets; against LynnReal-Omni, the generalization comparison is even overall but has a +0.27 structure margin.All four questions favor SolarRenderer over Stage 1 on the in-distribution test set, and quality, structure, and first-frame preservation improve against Stage 1 on both sets.Source: §C.1 · p. 21, Figure 15 · p. 21

Results by Representation and Source

  • Against Stage 1 + LongLive-Plug, SolarRenderer has positive margins in every broad representation group, but against Stage 1 it loses on colour-mapped depth, voxels, and pencil storyboards.It wins six of eight agent-built skeleton cases and six of seven silhouettes against Stage 1.Source: §C.1 · p. 22, Figure 16 · p. 22

Comparison on Automatic Metrics

  • On automatic metrics, SolarRenderer has the highest in-distribution total and matches Stage 1’s generalization total at reported precision, with improved control on both sets.Post-training also improves in-distribution quality but decreases stability and first-frame preservation; the total rises in-distribution and remains nearly unchanged on generalization.Source: Table 8 · p. 21, §C.1 · p. 22

Visual Comparison with Other Methods

  • Across 86 demonstration cases from 15 works, SolarRenderer wins 49 comparisons, ties 13, and loses 24, for a +0.29 net score.The comparison uses cases selected from other methods’ project pages and outputs.Source: §C.1 · p. 22

C.2 Additional Ablations

The additional ablations examine how structure guidance, sampling settings, conditioning design, and training data affect rendering quality, control, stability, and efficiency. They show trade-offs across automatic scores and Elo ratings, with four-step sampling favored among the tested settings.

C.2 Additional Ablations

  • From DMAD, guidance raises control from 59.2 to 61.0 and quality from 68.0 to 69.1, but lowers stability, first-frame preservation, and automatic total as pairwise ratings improve.Higher Elo ratings reflect the judge’s joint preference over appearance and structure.Source: Table 9 · p. 23, §C.2 · p. 23
  • Four steps achieve the highest overall, quality, and first-frame Elo ratings among the tested settings, balancing rendering quality with sampling cost.Eight steps improve quality slightly over four but leave the automatic total unchanged at reported precision; two steps have the highest control score but lower first-frame preservation.Source: Table 10 · p. 23, §C.2 · p. 23
  • Figure 17 compares SolarRenderer’s net score with outputs from 15 works on project-page demonstrations; positive scores favor SolarRenderer.Source: Figure 17 · p. 23
  • The native full-resolution pathway improves stability and control over the ControlNet branch with a small quality reduction; higher structural resolution raises control from 46.6 to 48.0 and total from 66.2 to 66.5.At half resolution, shared RoPE also improves quality and control over separate positions.Source: §C.2 · p. 23
  • Removing misaligned structural supervision improves quality, stability, and control over using all data, raising the total from 66.2 to 66.5.Structured-only data raises control but lowers stability and first-frame preservation; reducing data to one tenth has less effect than variation between full-data runs.Source: §C.2 · p. 23

Related papers

Loading 2610.28630v1…