PaperlayerView exact revision on arXiv

Source-linked AI summary

HiPhy: Hierarchical Alignment for Physically-Plausible Multi-Principle Video Generation

Tahira Kazimi, Shubhankar Borse, Munawar Hayat, Fatih Porikli, Pinar Yanardag

arXiv:2610.02197v1cs.CV

TL;DR

Video generators often violate physical laws, particularly when a scene combines concurrent physical principles. HiPhy uses reinforcement learning with local temporal rewards and global scene alignment; across multiple benchmarks, it achieves the strongest physical commonsense and semantic alignment, with its largest margins on multi-principle prompts.

  • Problem

    Video generators can violate physical laws, while existing learning-based and guidance methods typically supervise only one physical principle per video and lack mechanisms to ensure compound events are jointly satisfied.

    Source: §1 · p. 2

  • Method

    HiPhy uses reinforcement learning to score temporally ordered sub-stages for each physical principle and align them with global semantic and visual coherence.

    Source: §1 · p. 2, §4.2 · pp. 4–5

  • Results

    Across VideoPhy2, PhyWorldBench, and its multi-principle benchmark, HiPhy achieves the strongest physical commonsense and semantic alignment, with the largest margins on prompts containing concurrent principles.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The results suggest explicit temporal decomposition combined with a learned refinement signal is an effective approach to closing the gap between video generation and physical realism.

    Source: §8 · p. 10

    Because HiPhy uses a frozen backbone text-to-video model, its fidelity is bounded by the visual capacity of that generator.

    Source: §7 · p. 10

Abstract

from arXiv · show

Video generation models have achieved remarkable visual fidelity and have strong potential to become general-purpose world simulators. Despite this progress, they still fail to generate videos which adhere to laws of physics. The problem becomes even more apparent in realistic settings where multiple physical principles must work together within the same video; for example, "a balloon floating upward while steam rises from a pot" requires buoyancy and fluid dynamics to unfold coherently and simultaneously. Yet existing methods largely ignore multi-principle interactions, focusing on a single principle per video. We propose HiPhy (Hierarchical Physical Alignment), a reinforcement learning framework that grounds video generation in physical laws through a dual-level objective: locally enforcing the temporal dynamics of individual physical principles, and globally ensuring the physical and semantic coherence of the entire scene. To support multi-principle generation, we construct a 50K-prompt dataset and introduce a prompt benchmark MultiPhyBench, spanning a diverse range of co-occurring physical events. Our experiments show that HiPhy significantly outperforms prior methods and baselines, improving physical commonsense and semantic alignment significantly across various benchmarks, with the largest gains on scenes involving multiple concurrent physical principles where competing methods degrade most sharply.

1 Introduction

Although video generators have become visually capable, they often violate physical laws and struggle to represent multiple concurrent principles. HiPhy addresses this gap with stage-level supervision for each principle and global scene alignment.

1 Introduction

  • Current generators can produce plausible individual principles, but compound prompts expose omitted secondary physics and compressed event progressions.For example, an apple may fall realistically while leaving the water undisturbed, or a splash may appear as a static effect.Source: §1 · p. 2
  • Existing learning-based and guidance methods typically supervise one principle per video and can degrade sharply as concurrent processes increase.3D engines can simulate concurrent dynamics but are computationally heavy and limited to rigid domains.Source: §1 · p. 2
  • HiPhy treats concurrent physical events as independent learning streams, applying stage-level supervision while aligning stages with global semantic and visual coherence.The framework is designed to address omission of secondary principles and shortcutting of temporal dynamics.Source: §1 · p. 2
  • HiPhy is reported to improve physical commonsense by up to 44% and semantic alignment by up to 80% over prior physics-aware baselines.The paper also introduces a 50K-prompt training set and a 1K-prompt benchmark for concurrent physical principles.Source: §1 · p. 3

2 Related Work

Prior approaches inject physical knowledge through fine-tuning, simulation, auxiliary signals, or inference-time optimization, but face limits in coverage, cost, or temporal supervision. HiPhy is presented as a hierarchical objective for multi-principle scenes.

Physically Plausible Video Generation

  • Prior work generally supervises one physical principle per video, whereas HiPhy targets temporal progression in multi-principle scenes with a hierarchical stage-level objective.The passage describes this as HiPhy's distinction among physically plausible video-generation methods.Source: §2 · p. 3
  • Fine-tuning on physically annotated data remains tied to training-set coverage, while 3D simulation methods are costly and constrained to fixed physical categories.The passage notes particularly heavy costs for phenomena such as fluids or steam.Source: §2 · p. 3

RL-based Video Generation

  • Inference-time verifiers and test-time optimization can require large memory or long runtimes, and trajectory refinement may leave implausible initial plans unaddressed.Video preference-learning work also covers global state, temporal locality, and motion smoothness, but the passage distinguishes HiPhy by its multi-principle physical objective.Source: §2 · p. 3

3 Background

GRPO updates a policy using group-normalized advantages from sampled outputs, avoiding a separate critic. The standard objective can collapse distinct reward combinations into identical advantages in multi-reward optimization.

Background

  • GRPO computes advantages by normalizing sampled-output rewards within a group, eliminating the need for a separate critic network.For a query, it samples responses from the old policy and updates the current policy.Source: §3 · p. 3
  • The standard GRPO objective maximizes expected reward and upweights samples whose rewards exceed the group average.This is the stated effect of advantage normalization.Source: §3 · p. 3
  • In multi-reward optimization, standard GRPO can assign identical advantages to distinct reward combinations.Source: §3 · p. 3

4 Method

HiPhy trains a policy to generate videos depicting concurrent physical principles with correct temporal progression. Its hierarchical objective scores individual processes locally and scene-wide physical, semantic, and visual quality globally.

4 Method

  • HiPhy trains a reasoning policy with GRPO to generate videos depicting all prompt-described principles with correct sub-stage progression.The policy operates with a frozen text-to-video backbone.Source: §4 · pp. 3–4

4.1 Multi-Principle Dataset Curation

  • The 50K-prompt training set and 1,000-prompt MultiPhyBench combine one to three events and provide chronologically ordered hierarchical annotations.The prompts draw on processes from VideoPhy2 and WISA-80k, with an LLM filtering implausible event combinations.Source: §4.1 · p. 4

4.2 HiPhy: Hierarchical Physical Alignment

  • For each principle, HiPhy matches reference and observed stage trees level by level, blending sentence-encoder similarity with a VLM judge.The cosine term handles paraphrases, while the judge catches similar wording that describes a different event.Source: §4.2 · pp. 4–5
  • The local reward averages alignment, completeness, and chronological ordering, with failed parent stages disqualifying dependent leaves.A leaf counts as matched only if its similarity and all parent-node similarities exceed a threshold.Source: §4.2 · p. 5
  • A global reward combines visual quality, physical commonsense, and semantic alignment to assess whole-video quality.The physical-commonsense term checks whether the primary action follows real-world rules; semantic alignment checks whether described events appear.Source: §4.2 · p. 5
  • HiPhy independently normalizes reward streams and scales the local-reward sum by 1/√n to maintain consistent variance across prompts with different principle counts.The scaling counters the O(n) variance growth that would otherwise skew gradients for prompts with more principles.Source: §4.2 · p. 6

5 Experiments

Experiments compare HiPhy with video-generation and physics-aware baselines using qualitative examples, benchmark metrics, a user study, and inference-time measurements. HiPhy performs strongly across physics benchmarks, preserves video quality, and adds little inference overhead.

5 Experiments

  • The method is evaluated on Wan2.1 and compared with fine-tuning methods; experiments use two NVIDIA H200 GPUs and approximately 50K prompts with 1–3 concepts equally distributed.Source: §5 · p. 6, §5.1 · p. 7

5.3 Quantitative Experiments

  • HiPhy achieves the strongest performance on all physics benchmarks, with gains of +10% physical and +16% alignment on VideoPhy2 and ≈+30% physical and +45% semantic on PhyWorldBench.On the multi-principle benchmark, hierarchical optimization maintains the largest margin; VBench Dynamics and Appearance match or exceed the Wan2.1 baseline.Source: Table 1 · p. 7, §5.3 · p. 8

5.2 Qualitative Comparisons

  • On two- and three-principle prompts, HiPhy renders constituent processes coherently while baselines omit events, hallucinate objects, or produce incoherent motion.Examples include omitted sliding books, simmering sauce, rain, boiling water, and whisking actions.Source: §5.2 · p. 7, Figure 5 · p. 8

5.4 Human Evaluation.

  • In a 50-person study of 30 prompts, HiPhy outperforms all baselines across metrics, with its largest margin in text alignment.HiPhy also maintains steady, high ratings across individual events, while baselines often fail to depict some principles.Source: §5.4 · p. 9, Table 2 · p. 9

5.5 Computational Efficiency.

  • HiPhy adds only ∼2.4 seconds of overhead on a single H200 GPU, making total inference time effectively identical to the base model.The method runs once before generation.Source: §5.5 · p. 9

6 Ablation

Ablations show that local and global rewards complement one another, HiPhy transfers to multiple backbones, and its advantage widens as prompts combine more physical principles.

6.1 Reward Composition

  • Global-only training favors semantic alignment and visual quality but weakens physical commonsense, while local-only training improves multi-process physics but degrades semantic alignment and dynamic degree.Combining both reward families with per-stream normalization recovers the best results.Source: §6.1 · p. 9

6.2 Generalization across video backbones.

  • HiPhy restores a spiral coin trajectory on both Wan and VEO backbones, which otherwise produce near-static coins for that prompt.The method transfers across backbones with sufficient physical priors without retraining.Source: §6.2 · p. 9, Figure 6 · p. 10

6.3 Performance vs. principle count.

  • HiPhy maintains a joint PC & SA score of approximately 50 on three-process prompts, while PnP and PhyT2V fall below 20.PnP and PhyT2V lose approximately half their PC scores from one- to three-process prompts; the per-process reward stream provides distinct signals for each event.Source: Figure 7 · p. 10, §6.3 · p. 10

7 Limitations and Broader Impacts

HiPhy’s fidelity is bounded by the visual capacity of its frozen text-to-video backbone.

7 Limitations and Broader Impacts

  • Because HiPhy uses a frozen text-to-video backbone, its fidelity is upper-bounded by that generator’s visual capacity.Source: §7 · p. 10

8 Conclusion

HiPhy decomposes physical correctness into local, per-principle rewards over temporal sub-stages and a global reward for scene coherence. Across three benchmarks, it performs strongest on physical commonsense and semantic alignment, with its largest margins on multi-principle prompts.

8 Conclusion

  • Across VideoPhy2, PhyWorldBench, and the multi-principle benchmark, HiPhy achieves the strongest physical commonsense and semantic alignment, with its largest margins on concurrent-principle prompts.Prior methods degrade most sharply on those prompts.Source: §8 · p. 10
  • HiPhy uses per-principle local rewards over temporal sub-stages and a global reward for semantic and visual coherence.Source: §8 · p. 10
  • The results suggest explicit temporal decomposition combined with a learned refinement signal is an effective approach to closing the gap between video generation and physical realism.Source: §8 · p. 10

A Multi-Process Dataset Curation

The paper constructs prompts with concurrent physical processes and ordered stage annotations, then aligns generated and reference process trees hierarchically. Human evaluations support the extracted stage structure, while an ablation finds that flattening the reward sharply degrades MultiPhyBench.

A Multi-Process Dataset Curation

  • The dataset combines one to three processes per prompt, with the supported number scaling with video length, and includes both inter-domain and intra-domain combinations.An LLM filters out combinations judged physically incompatible.Source: A Multi-Process Dataset Curation · p. 15
  • Source prompts are converted into process-and-domain labels, then plausible combinations receive coherent scene descriptions and chronologically ordered stages for each process.The stage annotations serve as pseudo-label references for the local stage-aware reward.Source: A Multi-Process Dataset Curation · p. 15

B Hierarchical alignment algorithm

  • Hierarchical alignment pairs reference and observed process trees from their roots, solves local assignments, and recurses only into matches above a threshold.Its Hungarian assignment cost is O((K + M)^3) per node, but small branching factors make alignment negligible compared with video generation.Source: B Hierarchical alignment algorithm · pp. 15–16

C More Qualitative

  • The supplementary webpage provides additional qualitative examples and video samples mentioned in the paper.Source: C More Qualitative · p. 16

D Extracted Stage validation

  • For 50 videos rated by five annotators, extracted stages received mean quality and ordering ratings of 4.21/5 and 4.42/5, respectively.Optics events scored lowest on stage quality at 3.88.Source: D Extracted Stage validation · p. 16, Table 4 · p. 16
  • Physics graduate raters scored extracted stage trees comparably to one another and rated corrupted trees lower, with the largest gap for shuffled stage ordering.The comparison used 20 canonical events and included trees with shuffled stages or an inserted physically impossible stage.Source: D Extracted Stage validation · p. 16, Table 5 · p. 17

E Hierarchical vs. flat rewards.

  • Removing sub-stages to create a flat reward sharply degrades MultiPhyBench, though it barely affects VideoPhy2, where prompts contain one physical principle.Without sub-stages, a broad stage such as “splash” cannot distinguish a static spray from impact followed by crown formation.Source: E Hierarchical vs. flat rewards · pp. 16–17

F Ablation on individual alignment, completeness, and ordering

The reward ablation tests whether its terms supervise distinct diagnostics, while noting two cross-effects between those diagnostics.

F Ablation on individual alignment, completeness, and ordering

  • Removing each reward term degrades its corresponding omission, ordering, or stage-match diagnostic more than the others, supporting distinct roles for the terms.The evaluation used 30 videos per variant and automatic scores from Gemini Flash.Source: F Ablation on individual alignment,… · p. 17
  • Automatic-to-human label agreement was 0.84 for omission, 0.77 for ordering, and 0.82 for stage match.Source: Table 7 · p. 17
  • Out-of-order processes can lower stage match through less coherent extracted descriptions, while leaf-count normalization gives alignment a soft completeness signal.Source: F Ablation on individual alignment,… · p. 17

G Ablation on Normalization vs. scaling

The ablations distinguish per-stream normalization from rescaling the sum of local rewards, and the user study compares how methods depict concurrent physical events. Supplementary material describes the error bars, stage-generation prompt, and sample prompts.

G Ablation on Normalization vs. scaling

  • Per-stream normalization limits domination by high-variance reward streams, while 1/√n rescaling controls variance growth as local streams are summed and preserves balance with the global reward.The rescaling matters as principles co-occur; at N = 1 it is inactive and per-stream normalization has little to balance.Source: G Ablation on Normalization vs. scaling · pp. 17–18, Table 8 · p. 18

H User Study

  • In a study of 50 participants rating 30 prompts across five methods, HiPhy consistently received the highest scores across principles and maintained balanced adherence to concurrent events.Source: H User Study · p. 18

I Error Bars for the quantitative comparisons

J Stage Generation Prompt

  • The stage-generation prompt instructs an LLM to identify distinct physical processes and decompose each into chronological stages with observable physical interactions.The resulting JSON is used as the supervision target for hierarchical training.Source: J Stage Generation Prompt · p. 20

K Video Prompts

Related papers

Loading 2610.02197v1…