PaperlayerView exact revision on arXiv

Source-linked AI summary

Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model

Hai Nguyen-Truong, Tuan-Anh Vu, Dang Huynh

arXiv:2608.29904v1cs.CV

TL;DR

Video generators often violate basic physical dynamics because standard objectives fit visual distributions without explicitly penalizing those errors. OMR injects a frozen V-JEPA 2.1 surprise gradient through an adapter during middle ODE steps in one sampling trajectory, raising Wan2.2’s VideoPhy-2 joint score from 47.0% to 52.0% at 1.71× base runtime.

  • Problem

    Standard video-generation objectives fit visual data distributions but do not explicitly penalize violations such as floating objects, broken trajectories, and disappearing contacts.

  • Method

    OMR guides one frozen-generator sampling trajectory with scheduled adapter-space V-JEPA 2.1 surprise-gradient corrections during middle ODE steps.

  • Results

    47.0% to 52.0%: OMR raises Wan2.2’s joint score on the fixed 400-prompt VideoPhy-2 detailed subset.

  • Takeaways & Limitations

    OMR’s frozen-world-model guidance is supported across generators and benchmarks, including CogVideoX-5B and Wan2.2-I2V pilots.

  • Takeaways & Limitations

    OMR inherits the world model’s blind spots and data biases, and predictive plausibility does not guarantee physical correctness.

Abstract

from arXiv · show

Modern video generators routinely fail at physical dynamics: objects float, trajectories violate gravity, contacts vanish. Standard denoising and flow-matching objectives fit visual data distributions but do not explicitly penalize such physical violations. Existing remedies can improve physical consistency, but typically add substantial inference or training cost. Candidate-selection methods generate and score multiple videos, while gradient-based world-model guidance repeatedly decodes and re-encodes intermediate estimates. Generator-internal refinement adds perturbation and re-denoising loops, whereas post-training requires curated data and additional optimization. We propose Off-Manifold Refinement (OMR), an inference-time method that instead injects world-model feedback directly into a single sampling trajectory. During scheduled middle ODE steps, we augment the generator velocity with the gradient of an adapter-space V-JEPA 2.1 surprise energy. This external correction can move the latent away from the uncorrected sampling trajectory and toward regions ranked as more physically plausible by the frozen predictor, after which the generator continues rendering from the corrected state. A small trained latent-to-embedding adapter keeps the gradient tractable at inference, and both the video generator and the world model remain frozen. On our fixed 400-prompt VideoPhy-2 detailed subset, OMR lifts the joint Semantic-Adherence-and-Physical-Commonsense metric from 47.0% to 52.0% (+5.0pp absolute, +10.6% relative) over the base Wan2.2-T2V-A14B sampler. On a separate fixed 50-prompt efficiency subset, it requires $1.71 \times$ the base runtime rather than the multiplicative cost of reward/search alternatives. Project page: https://itruonghai.github.io/omr.

1 Introduction

OMR targets physical violations that standard video-generation objectives do not explicitly penalize by injecting frozen-world-model feedback into a single sampling trajectory. Its adapter-space correction improves physical-consistency scores while avoiding candidate-generation and video decode–encode loops.

  • Motivation: Video generators can reproduce visual statistics without reliably modeling gravity, contacts, continuity, or material responses.Standard flow-matching and denoising objectives do not explicitly penalize these physical violations.
  • Method: OMR applies a V-JEPA 2.1 surprise-gradient correction during scheduled middle ODE steps, then resumes sampling from the corrected latent.The correction can move the latent away from an implausible uncorrected trajectory while subsequent generator steps continue rendering.
  • Efficiency: OMR uses one monotone trajectory without perturbation, re-noising, or mask blending, remaining complementary to candidate search.Unlike candidate selection, it spends the world-model signal before the sample is finished.
  • Method: A 33.3M latent-to-embedding adapter avoids in-loop VAE decoding and video-encoder passes during guidance.The adapter maps generator latents directly into world-model embedding space, while only the predictor runs at sampling time.
  • Results: 47.0% to 52.0%: OMR raises Wan2.2 joint score on the fixed 400-prompt VideoPhy-2 detailed subset.The reported joint score requires both Semantic Adherence and Physical Commonsense scores to be at least 4 on a 1–5 scale.
  • Results: OMR yields non-decreasing PC and joint scores across every reported category, with largest joint gains in contact dynamics and fluid/deformable motion.The largest reported gains are +11.0pp for contact dynamics and +9.8pp for fluid/deformable motion.

2 Related Work

Related work improves physical consistency by modifying generators, scoring completed candidates, or steering intermediate samples. OMR instead keeps the generator frozen and uses an encoder-free adapter path to apply a V-JEPA predictive signal within one trajectory.

  • Diffusion and flow-based video generation: Diffusion and flow-based video systems use denoising, score-modeling, latent-diffusion, transformer, consistency-distillation, and flow-matching techniques.Video systems additionally model temporal structure through video objectives, temporal modules, or image-to-video adaptation.
  • Physics-aware video generation: Physics-aware methods inject physical information by changing generators, distilling foundation-model relations, or optimizing physics-aware rewards and preferences.Examples include explicit physical controls, relation distillation, reward objectives, and DPO-style training.
  • Physics-aware video generation: V-JEPA-style world models have been used as verifiers, rewards, best-of-K selectors, rejection mechanisms, and trajectory-steering signals.These approaches demonstrate the usefulness of external physical priors but can require generator adaptation or decode-and-encode paths.
  • Inference-time guidance: Inference-time guidance extends classifier and classifier-free guidance to differentiable losses, inverse-problem likelihoods, restarts, and reward functions.OMR follows this broad guidance template with a self-supervised video world model rather than a classifier or learned scalar reward.
  • Video world models: JEPA-family video models learn predictive dynamics over masked space-time patches, while OMR uses V-JEPA 2.1 dense patch features and a frozen predictor.These components define the adapter-space surprise energy used for guidance.

3 Method

OMR guides flow-matching video generation with a frozen V-JEPA 2.1 world model through an adapter-space surprise gradient applied during scheduled middle ODE steps. The generator continues from the corrected latent, while the adapter avoids VAE decoding and world-model encoding at inference.

  • Flow-matching video generation: Flow matching transports latents from pure noise at t = 0 to a clean video latent at t = 1 using a learned velocity field.The same field provides an implicit clean prediction from intermediate states for OMR evaluation.
  • World-model signal: OMR evaluates the implicit clean prediction in V-JEPA 2.1 embedding space and uses its predictive surprise as a dense differentiable guidance signal.Low energy indicates that masked future content is predictable from visible context in representations trained on real videos.
  • OMR update: At scheduled steps with η(t) > 0, OMR subtracts the surprise gradient from the generator update, then uses the generator velocity alone on remaining steps.The gradient is computed through the predictor and adapter without backpropagating through the generator velocity network.
  • Latent-to-embedding adapter: A 33.3M convolutional adapter maps generator latents directly into V-JEPA 2.1 embeddings, replacing the encoder during inference.Only the frozen predictor runs, avoiding VAE decoding and V-JEPA encoding in the guidance path.
  • Off-manifold refinement: The correction can move sampling away from the generator’s uncorrected trajectory because the separately trained world-model gradient need not align with the generator flow field.Subsequent ODE steps continue rendering from the corrected state.
  • Schedule and constraints: Guidance is concentrated in a middle clean-time band because early clean predictions are noise-dominated and late corrections have diminishing returns.The adapter is trained offline on paired Wan2.2 latents and frozen V-JEPA 2.1 targets, while the generator, adapter, and world model remain frozen at inference.

4 Experiments

Experiments evaluate OMR with frozen generators and world-model components across fixed VideoPhy-2 and Physics-IQ subsets. OMR improves physical-consistency metrics while avoiding the much larger cost of reward-gradient search alternatives.

  • Setup: Experiments use Wan2.2 as the primary text-to-video generator, with CogVideoX-5B and Wan2.2-I2V testing transfer.Wan2.2 uses 480 × 832 resolution, 81 frames, 16 FPS, and 40 ODE steps; the world model and adapters remain frozen.
  • Setup: Evaluations use fixed prompt subsets: 400 detailed VideoPhy-2 prompts for the default comparison and separate 50-prompt subsets for efficiency, transfer, and Physics-IQ.The same 400 prompt IDs are used for every default method without output- or score-based filtering.
  • Main results: OMR improves VideoPhy-2 joint success from 47.00% to 52.00% (+5.00pp absolute; +10.6% relative) over Wan2.2 and exceeds P&P by 3.25pp.The gain is physical: PC mean rises from 3.660 to 3.700 and PC≥4 from 57.0% to 60.0%, while SA≥4 remains 68.0%.
  • Efficiency and transfer: OMR achieves 54.0% Joint on the 50-prompt Wan2.2 efficiency subset at 1.71× runtime and 74.1 GB peak VRAM.WMReward-∇ with BoN-4 reaches 53.6% but requires 19.88× runtime and 134 GB memory.
  • Efficiency and transfer: OMR raises CogVideoX Joint from 32.0% to 42.0% at 1.67× runtime and 30.4 GB peak VRAM.WMReward-∇ reaches 38.0% at 4.97× runtime and 83.2 GB, while its BoN-4/8 variants reach 40.0% at 19.88×/39.76×.
  • Efficiency and transfer: On Wan2.2-I2V, OMR improves Physics-IQ by +4.90 without in-loop VAE decoding or V-JEPA encoding, reaching +8.27 with BoN-4.The controlled comparison is treated as transfer evidence against mixed-modality published references.
  • Qualitative and human evaluation: Qualitative examples show OMR better aligning impact with material response, maintaining tool contact, and preserving support and momentum.These improvements target contact dynamics and fluid/deformable motion, where the largest joint gains occur.
  • Qualitative and human evaluation: In a blind study on 100 matched prompts, OMR wins physical plausibility over P&P, 49.69% to 20.23%, while ties dominate prompt-adherence outcomes.Thirteen annotators provide 1,300 judgments per question.

5 Conclusion

OMR guides a frozen video generator with feedback from a separately trained, frozen world model within one sampling trajectory. It improves the primary Wan2.2 result while retaining a bounded inference-time protocol, but its evaluation and guarantees remain limited.

  • OMR guides a separately trained generator within one sampling trajectory using a frozen video world model.A generator-specific adapter maps provisional clean latents into V-JEPA 2.1 feature space and supplies masked-future surprise as a dense correction.
  • 5.0 percentage points: OMR improves Wan2.2 from 47.0% to 52.0% on the fixed 400-prompt VideoPhy-2 subset.
  • 1.71× runtime: OMR’s separate 50-prompt study reports this cost relative to the base sampler.
  • OMR inherits the frozen world model’s blind spots and data biases, and predictive plausibility does not guarantee physical correctness.
  • Evaluation covers one world-model family, two generators, adapter-specific training resolutions, one generation per 400-prompt prompt, and pilot point estimates without paired confidence intervals.

Supplementary Material

The supplementary material identifies the paper’s authors.

  • Hai Nguyen-Truong is listed as an author of the paper.
  • Tuan-Anh Vu is listed as an author of the paper.
  • Dang Huynh is listed as an author of the paper.

A Adapter architecture details

The Wan2.2 latent-to-embedding adapter is a convolutional, per-token architecture that projects generator latents onto the frozen V-JEPA 2.1 representation grid. Its relational loss preserves token geometry so the downstream predictor receives structurally meaningful embeddings.

  • A.1 Block diagram and tensor shapes: 33.3M parameters: the Wan2.2 adapter is a convolutional network rather than a transformer.It uses local 3D convolutions and a shared per-token MLP head.
  • A.1 Block diagram and tensor shapes: The adapter trilinearly interpolates Wan2.2 VAE latents to the V-JEPA 2.1 token grid before predicting hierarchical embeddings.The target grid is defined by the V-JEPA 2.1 encoder, so the adapter does not require self-attention layers.
  • A.1 Block diagram and tensor shapes: 6656 dimensions: the main Wan2.2 adapter predicts four hierarchical V-JEPA 2.1 distillation layers of 1664 dimensions each.A single-layer 1664-dimensional target is supported by the code but is not used in the main experiments.
  • A.1 Block diagram and tensor shapes: The CogVideoX transfer experiment retrains the same adapter architecture on CogVideoX latents instead of reusing Wan2.2 adapter weights.This isolates transfer of the OMR procedure rather than assuming shared VAE latent coordinates.
  • A.2 Adapter loss: The relational loss matches pairwise token similarity geometry in frozen V-JEPA 2.1 representations, complementing direct embedding supervision.Its small coefficient stabilizes context–future and neighborhood structure without acting as an additional physics reward.

B Mid-trajectory adapter-fidelity protocol

The mid-trajectory protocol tests whether the adapter remains faithful when applied to predicted clean latents during OMR sampling. Results are close to those for corresponding finished latents, supporting use on the deployment distribution while preserving important caveats.

  • B. Mid-trajectory adapter-fidelity protocol: 200 state-level evaluations and 50 finished-latent evaluations measure deployment-time adapter fidelity across 50 sampling trajectories.Four guided mid-trajectory predicted clean latents were retained from each trajectory, alongside one finished latent.
  • B. Mid-trajectory adapter-fidelity protocol: 0.762/0.617/0.446: mid-trajectory predicted clean latents achieve cosine/ρ/τ scores, compared with 0.769/0.624/0.451 for finished latents.
  • B. Mid-trajectory adapter-fidelity protocol: The proximity between mid-trajectory and finished-latent rows supports adapter use on OMR’s operating distribution but does not show that distribution shift disappears or that encoder reproduction is exact.

Per-category VideoPhy-2 breakdown

OMR improves physical consistency across VideoPhy-2 categories, with its largest joint-score gains in contact dynamics and fluid/deformable motion. Improvements also appear in mixed physical phenomena, while the object-permanence slice is too small to interpret.

  • Contact dynamics improves from 44.5% to 55.5% joint score (+11.0pp), the largest reported category gain.
  • Fluid/deformable motion improves from 43.9% to 53.7% joint score (+9.8pp).
  • The mixed-phenomena “Other physics principles” row improves from 46.6% to 47.6% joint score and from 56.6% to 58.7% on PC≥4.
  • Gravity/free-fall improves on PC≥4 from 60.0% to 62.5%, while the two-prompt object-permanence slice is unchanged and too small to interpret.

D Cross-benchmark and cross-generator pilot protocols

The paper evaluates OMR through matched pilots and seed checks across benchmarks and generators, while distinguishing its single-trajectory guidance from candidate-selection and repeated decode-and-encode methods. These protocols constrain comparisons by holding prompts, checkpoints, evaluators, and other settings fixed where stated.

  • Cross-benchmark protocols: The Physics-IQ pilot uses a fixed 50-prompt image-to-video subset with shared Wan2.2-I2V-A14B checkpoint, conditioning images, prompts, and evaluator.
  • Cross-generator protocols: The CogVideoX-5B pilot holds prompts, checkpoint, resolution, evaluator, and hardware constant across OMR, base, BoN-4, and WMReward-∇ variants.
  • Metric interpretation: Physics-IQ measures spatiotemporal overlap with reference motion, whereas VideoPhy-2 selection targets a world-model proxy rather than the final joint metric.
  • Guidance path: OMR advances once from t = 0 to t = 1 under a modified velocity field, without inner stochastic loops, re-noising, or explicit re-projection.
  • Seed stability: The seed-sensitivity check uses a fixed 50-prompt subset because a full 400-prompt × 3-seed evaluation would require 1,200 high-resolution generations per method.
  • Human comparison: The blind P&P comparison matches prompt, seed, resolution, frame count, and sampler budget, with 13 annotators providing 1,300 judgments per question.
Loading 2608.29904v1…