PaperlayerView exact revision on arXiv

Source-linked AI summary

World Embedding Benchmark

Yiqi Liu, Ruifeng Yuan, Yang Wang, Long Li, Fengyu Cai, Hou Pong Chan, Jialin Yu, Hao Zhang, Chenghua Lin, Chenghao Xiao

arXiv:2610.03632v1cs.CVcs.CL

TL;DR

How video embeddings preserve physical information and align it with language remains insufficiently understood. The paper introduces a controlled benchmark and finds weak alignment despite recoverable physical signals, alongside a trade-off from contrastive adaptation and gains in generated-video fidelity from retrieved references.

  • Problem

    How video representations preserve physical information and make it accessible through language alignment remains unclear, although both capabilities matter for physical retrieval, property estimation, and generation references.

    Source: §1 · p. 1

  • Method

    The World Embedding Benchmark pairs 8,000 controlled simulation cases with physical annotations and evaluates embeddings through retrieval, property regression, and video-description pair classification.

    Source: §1 · p. 2

  • Results

    Pre-trained embeddings retain probe-recoverable physical signals despite weak cross-modal alignment; contrastive adaptation improves retrieval and pair classification but degrades property regression, while retrieved references improve generated-video fidelity.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Physical alignment and quantitative information recoverability should be evaluated jointly, and physical embeddings can support retrieval-augmented video generation in the reported experiments.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Physical fidelity has received increasing attention in world models and video generation, yet how video representations encode physical information remains less understood. We introduce the World Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification. We use these tasks to distinguish cross-modal physical alignment from the recoverability of quantitative physical information. Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings. Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability. Finally, we use the embeddings to retrieve reference videos for retrieval-augmented generation with MiniMax-H3. Retrieved references improve the physical fidelity of generated videos, with stronger retrieval models yielding larger gains in our experiments. Together, these findings highlight the need to evaluate physical alignment and property recoverability jointly, and demonstrate the utility of physical representations for improving video generation.

1 INTRODUCTION

The paper introduces a benchmark to examine what physical information video embeddings preserve and how readily that information aligns with language. It finds a separation between recoverable physical signals and cross-modal alignment, with contrastive adaptation trading improved matching for weaker property regression.

1 INTRODUCTION

  • Physical representation evaluation must separate recoverability of physical quantities from their alignment with language, since success in one does not imply success in the other.A representation may recognize a broad scene type yet miss differences in properties such as viscosity or stiffness.Source: §1 · pp. 1–2
  • The World Embedding Benchmark pairs 8,000 controlled simulation cases across 80 physical families with tasks for evaluating video embeddings.The cases span fluid mechanics, solid mechanics, dynamics, and optics and electromagnetism.Source: §1 · p. 2
  • Pre-trained embeddings show weak retrieval and near-chance within-family classification despite probes recovering physical signals; adaptation improves alignment but degrades property regression, while retrieved references improve generated-video fidelity.Stronger physics retrieval models yield larger generation improvements in the reported experiments.Source: §1 · p. 2

2 WORLD EMBEDDING BENCHMARK

The benchmark combines videos from parameterized physics simulations with physical annotations and evaluates embeddings through retrieval, property regression, and pair classification. Its simulation set spans four branches, and its formulation tests both cross-modal matching and quantitative or fine-grained physical information.

2 WORLD EMBEDDING BENCHMARK

  • The benchmark contains 8,000 simulation cases from 80 families, with 100 parameter-varied instances per family and three tasks for assessing physical representations.Its tasks evaluate text-video retrieval, regression from frozen video embeddings, and video-description pair classification.Source: §2 · pp. 2–3

2.1 PHYSICS SIMULATION

  • Parameterized simulations use governing models, material and geometric conditions, and branch-specific engines to generate physical responses rendered as video frames.Simulation outputs provide access to generating parameters and computed physical states.Source: §2.1 · p. 3

2.1.1 FLUID MECHANICS

  • Fluid mechanics contributes 700 videos across seven families, covering varied flow configurations and parameters such as Reynolds and Péclet numbers, viscosity, and diffusivity.Outputs include velocity, temperature, and transported-scalar fields.Source: §2.1.1 · p. 4

2.1.2 SOLID MECHANICS

  • Solid mechanics contributes 2,700 videos across 27 families, spanning elasticity and nonlinear deformation under varied materials, geometries, and loading conditions.Videos primarily visualize displacement magnitude on the deformed geometry.Source: §2.1.2 · p. 4

2.1.3 DYNAMICS

  • Dynamics contributes 1,900 videos across 19 families, ranging from rigid-body motion and collisions to vibration, wave propagation, and structural response.Cases vary physical parameters including mass, gravity, friction, stiffness, and damping.Source: §2.1.3 · p. 5

2.1.4 OPTICS & ELECTROMAGNETISM

  • Optics and electromagnetism contribute 2,700 videos across 27 families, covering optical phenomena and electromagnetic systems such as waveguides, gratings, and scattering.Optics uses analytical relations or HCIPy, with Mitsuba 3 for selected configurations; eight electromagnetic families use Meep FDTD.Source: §2.1.4 · p. 5

2.2 BENCHMARK FORMULATION

  • The formulation evaluates branch-wise bidirectional retrieval, ten continuous-property regression tasks, and binary pair choices testing within- and cross-family discrimination.Retrieval queries instantiate family templates with simulation metadata; regression uses frozen embeddings and reports nRMSE and Spearman correlation.Source: §2.2 · p. 5

3 EXPERIMENTS

The experiments evaluate physical information in video embeddings across benchmark tasks, physics-specific contrastive adaptation, and retrieval-augmented video generation. Pre-trained embeddings show weak physical alignment despite recoverable physical information, while adaptation improves alignment but reduces regression performance; retrieved references improve generated-video fidelity.

3.1 MAIN RESULTS

  • Pre-trained embeddings show low physics-aware retrieval and near-chance within-family classification, while prompted MLLMs perform better on classification and V-JEPA2 achieves strong physical-property regression.The results distinguish physical information preserved in video representations from information exposed through similarity matching.Source: §3.1 · p. 6, Table 2 · p. 6

3.2 ENHANCEMENT OF PHYSICS CAPABILITIES

  • Physics adaptation trains on 8,000 simulation-derived video-text pairs, using same-family hard negatives and physics-only or general-plus-physics data settings.The experiments initialize from LCO-Embedding-Omni-3B and 7B models.Source: §3.2 · p. 7
  • Retrieval keeps improving throughout training without signs of convergence, and mixing general-domain video-text data slows physics-aware retrieval training.This contrasts with general-domain MLLM embedding training, for which two or fewer epochs are typically sufficient.Source: Figure 3 · p. 7, §3.2 · p. 8
  • After identical contrastive training, 7B backbones yield stronger physics retrieval than 3B backbones across the four physics branches.The authors interpret this as supporting a generation-representation scaling pattern in physics representation learning.Source: §3.2 · p. 8, Figure 4 · p. 8
  • Contrastive adaptation improves retrieval, but physical-property regression mostly underperforms the original LCO-Embedding-Omni, consistent with a trade-off between alignment and quantitative recovery.The authors suggest similarity-space calibration may reduce the linear accessibility of some properties and note that explicit representation-regression objectives may help.Source: §3.2 · p. 8

3.3 RETRIEVAL-AUGMENTED VIDEO GENERATION

  • On PhyGenBench, physics-adapted retrieval improves MiniMax-H3 generation, with general-plus-physics scoring 0.66 average and physics-only scoring 0.65.The original LCO-Embedding-Omni-3B also improves over no-RAG, while VideoCLIP-XL references slightly degrade average performance.Source: §3.3 · p. 8, Table 3 · p. 9

4 RELATED WORK

Prior work studies physical systems through numerical simulation benchmarks and visual tests of causal reasoning or event prediction. This paper positions its benchmark across video representations, physical quantities, generation fidelity, and multimodal embedding evaluation.

4.1 PHYSICS SIMULATION

  • Numerical benchmarks evaluate parameterized physical systems, while visual benchmarks focus on causal reasoning and event prediction; this benchmark pairs rendered simulations with physical quantities for video retrieval and regression.The cited numerical works include Takamoto et al. (2022), Bonnet et al. (2022), Gupta & Brandstetter (2023), Toshev et al. (2023), Luo et al. (2024), and Ohana et al. (2024b); visual works include Riochet et al. (2020), Yi et al. (2020), Bear et al. (2021), and Chen et al. (2025).Source: §4.1 · p. 9

4.2 PHYSICS-AWARE VIDEO GENERATION AND WORLD MODELING

  • Video-generation research addresses physical plausibility through prompt refinement, physics-oriented training, and retrieved video references; this work examines physical information in pretrained representations and its uses.The passage cites Xue et al. (2025), Wang et al. (2025), and Cheng et al. (2026) for those approaches, respectively.Source: §4.2 · p. 9

4.3 OMNIMODAL REPRESENTATION LEARNING

  • Omnimodal embedding evaluation has broadened across modalities, while this benchmark targets fine-grained physical information through physical-description alignment and quantitative property recovery.The passage describes prior embedding approaches based on text-only contrastive learning, instruction-conditioned training, and language-centric contrastive refinement.Source: §4.3 · p. 9

5 CONCLUSION

The World Embedding Benchmark reveals a gap between recoverable physical signals and cross-modal alignment in pretrained omnimodal embeddings. Contrastive adaptation improves alignment but degrades property regression, while physically relevant retrieval improves generated-video fidelity.

5 CONCLUSION

  • Across 8,000 controlled cases, pretrained embeddings preserve recoverable physical signals but show weak alignment; physics adaptation improves retrieval and pair classification while degrading regression.Physically relevant retrieval also improves generated-video fidelity, with stronger retrieval models yielding larger gains.Source: §5 · p. 9

B.1 DATASET OVERVIEW

This section outlines the simulation dataset’s organization into controlled families and describes the formulations, computation methods, and visualizations used across its physical branches. It also lists parameterized cases spanning mechanical systems, optics, and electromagnetism, including optical outputs represented as irradiance or normalized fields.

B.1 DATASET OVERVIEW

  • The benchmark contains 8,000 cases across 80 families and four branches, with 100 cases per family; records include configurations, physical responses and quantities, and rendered videos.Cases within a family share a physical scenario while varying selected material, geometric, boundary, loading, or excitation parameters.Source: §B.1 · p. 15

B.2 PHYSICAL FORMULATIONS AND SIMULATION

  • Simulation families use established backends, analytical evaluation, or numerical ODE integration; Table 4 maps their formulations, computation methods, case parameters, and field outputs.Source: §B.2 · p. 15, Table 4 · p. 19

B.2.1 FLUID MECHANICS

  • The fluid subset groups 15 simulation configurations into seven families by dominant mechanism, covers internal and external flows, transport, buoyancy, and non-Newtonian rheology, and uses OpenFOAM for field-rendered videos.Principal outputs include velocity magnitude and, for applicable families, temperature or transported-scalar fields.Source: §B.2.1 · pp. 15–16

B.2.2 SOLID MECHANICS

  • Solid-mechanics families include linear-elastic, finite-strain, and geometrically nonlinear buckling cases; DOLFINx solves them, and videos primarily show displacement magnitude on deformed geometry.Frame progression follows successive loading states.Source: §B.2.2 · p. 16

B.2.3 DYNAMICS

  • Dynamics spans rigid-body, lumped-parameter, mechanism, and structural responses, using Chrono and analytical or ODE integration to render family-dependent fields over physical time.Displayed quantities include velocity, displacement, angular speed, stress, plastic strain, and damage.Source: §B.2.3 · pp. 17–18

B.2.4 OPTICS AND ELECTROMAGNETISM

  • Optical families use analytical or ODE integration and HCIPy scalar-wave propagation, while electromagnetic families solve Maxwell’s equations with Meep FDTD; optical outputs are rendered as irradiance.Optical sequences may follow parameter scans, whereas Meep sequences follow simulation time.Source: §B.2.4 · p. 19
  • Mechanical cases vary geometry, material properties, loads, damping, and initial conditions across beam, plate, mass-spring, and articulated-system simulations.Listed inputs include beam dimensions and elastic properties, plate modes, spring and damping parameters, and link geometry and drive conditions.Source: §B.2.4 · pp. 20–21
  • Optical irradiance cases cover reflection, refraction, focusing, polarization, and interference, parameterized by incidence angles, refractive indices, geometry, and wavelength.The listed examples include mirrors, dielectric blocks, lenses, slits, polarizers, and waveplates.Source: §B.2.4 · pp. 21–22
  • Diffraction and propagation cases vary slit or aperture geometry, wavelength, and screen distance, while reporting irradiance distributions.Examples include double-slit interference, circular-aperture Airy patterns, diffraction gratings, and Gaussian-beam propagation.Source: §B.2.4 · p. 21
  • Electromagnetic cases use Meep finite-difference time-domain simulations to produce normalized Q(x, y, t) fields for structures including cylinders, waveguides, and photonic crystals.The listed inputs include refractive indices, feature dimensions, and source wavelength.Source: §B.2.4 · p. 22

C.1 RETRIEVAL

Retrieval uses one deterministic, family-templated query per simulation case, paired with its target video. Training and adaptation additionally use five deterministic hard negatives sampled from the same family.

C.1 RETRIEVAL

  • Each simulation case receives one deterministic primary query built from a family-specific template and paired with its designated target video.Templates combine the physical scenario, mechanism, visualized quantity, case-specific conditions, and quantitative diagnostics; duplicate query texts are disambiguated when necessary.Source: §C.1 · p. 22
  • Training and adaptation provide five distinct hard negatives per anchor, sampled from the other 99 cases in its family using a fixed random seed.These negatives are stored separately from the retrieval evaluation candidate set.Source: §C.1 · p. 22

C.2 PHYSICAL PROPERTY REGRESSION

Physical-property regression tests whether frozen video representations contain recoverable continuous quantities, using an independent linear probe for each selected family. Performance is assessed with normalized error and rank correlation.

C.2 PHYSICAL PROPERTY REGRESSION

  • Ten regression families each define an independent task with one scalar physical target and unit, predicted from the video representation without textual input.Table 5 summarizes the target quantities.Source: §C.2 · p. 22, Table 5 · p. 23
  • A separate linear regression probe predicts each family’s target from the encoder’s frozen video representation.Because the encoder remains fixed, the probe measures target-quantity accessibility in the learned representation.Source: §C.2 · p. 22
  • Normalized root mean squared error is the primary regression metric, with Spearman’s rank correlation measuring preservation of the physical quantity’s ordering.Source: §C.2 · p. 23

D REGRESSION AND PAIR CLASSIFICATION RESULTS AFTER TRAINING

Figures 6 and 7 display regression and pair-classification results across training steps; the supplied passage does not state the outcomes.

D REGRESSION AND PAIR CLASSIFICATION RESULTS AFTER TRAINING

E REGRESSION AND PAIR CLASSIFICATION RESULTS AFTER TRAINING

The appendix reports regression, pair-classification, and retrieval training-step analyses; 7B retrieval training dynamics are consistent with those of the 3B experiments.

E REGRESSION AND PAIR CLASSIFICATION RESULTS AFTER TRAINING

  • The 7B variant’s retrieval training dynamics are consistent with the 3B experiments.Source: E REGRESSION AND PAIR CLASSIFICATION… · p. 23
  • Figure 6 plots linear regression performance for contrastively trained checkpoints against the number of probe examples.Source: Figure 6 · p. 24
  • Figure 7 plots within-family and cross-family pair classification performance across training steps.Source: Figure 7 · p. 24
  • Figure 8 plots retrieval performance across training steps and training-dataset settings.Source: Figure 8 · p. 25
Loading 2610.03632v1…