Source-linked AI summary
TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Image
Xin Zhang, Yabo Chen, Zixuan Duan, Haibin Huang, Chi Zhang, Feng Xu, Xuelong Li
TL;DR
Interactive world models must separate observation from physical intervention because appearance-driven generation can lose persistent geometric and physical consistency. TourPhysics builds an online stateful model from one image and a declarative configuration, using fixed simulator trajectories, geometry-aware controls, and accepted appearance memory. On simulator-defined tours and manipulations, it improves prescribed-trajectory alignment, preserves the input scene, and reduces long-horizon appearance drift.
Problem
Interactive world models need to distinguish observation from intervention, while appearance-driven video generation may lose persistent geometric and physical consistency over long horizons.
Method
TourPhysics constructs an online world model from one image and a declarative configuration, separating simulator state, geometry, generator controls, and accepted appearance memory.
Results
TourPhysics improves alignment with prescribed camera and object trajectories, preserves the input scene, and reduces long-horizon appearance drift on simulator-defined tours and manipulations.
Takeaways & Limitations
The framework turns physics-grounded generation into a persistent exploration-and-manipulation loop with explicit state ownership and geometry-routed appearance updates.
Takeaways & Limitations
TourPhysics models a controllable physical hypothesis whose behavior depends on supplied proxies, materials, boundary conditions, and simulator parameters, so prescribed trajectories do not guarantee real-world physical accuracy.
Abstract
from arXiv · showhide
Interactive visual world models must distinguish observation from physical intervention. Camera motion reveals new surfaces, whereas intervention changes object motion, contact, and deformation. Current video world models are largely driven by appearance priors and often lose physical or spatial consistency over long horizons. We present TourPhysics, an online framework initialized from a single image and a declarative physical configuration. TourPhysics extends PhysOmni, our ACM Multimedia 2026 work, from finite physics-grounded video synthesis to persistent exploration and manipulation. TourPhysics combines deterministic simulation with video generation while assigning separate roles to simulator state, geometric evidence, generator controls, and appearance memory. For each action, the simulator computes a finite physical and camera trajectory before the corresponding observation is generated. Accepted observations publish the terminal state and update the appearance memory and subsequent generator controls, while the committed state and simulator geometry remain fixed throughout synthesis and retry. We further separate the simulator geometry used for projection and visibility from the relative depth used to condition the generator. A reference-anchored memory retrieves accepted static appearance through geometric cross-view correspondence and incorporates it through a bounded residual that reverts to the native path when no valid correspondence exists. On simulator-defined camera tours and object manipulations, TourPhysics follows prescribed camera and object trajectories more closely than the evaluated baselines, preserves the input scene, and reduces appearance drift during long-horizon revisits.
1 Introduction
TourPhysics treats interactive world modeling as a separation between physical intervention and observation, then maintains that separation across persistent camera exploration and manipulation. Its simulator fixes trajectories and state, while accepted observations update appearance evidence through geometry-aware memory.
- Camera motion reveals unseen surfaces, whereas intervention changes object positions, contacts, and deformations.
- Appearance-driven video models can produce plausible frames without preserving persistent geometry or physically faithful interactions over revisits.
- TourPhysics extends physics-grounded single-image generation into an online world-model loop with explicit state ownership across action–observation cycles.
- The system evaluates prescribed camera and object trajectories, event timing and deformation, input-scene similarity, and long-horizon appearance stability across multiple physical proxies.
- For each action, the simulator fixes the physical and camera trajectory before synthesis, while quality-gated commits publish accepted state and appearance evidence atomically.
- Simulator-consistent geometry governs projection and visibility, while relative depth conditions the generator and reference-anchored memory retrieves accepted appearance through cross-view correspondence.
2 Related Work
Related work combines strong video appearance and motion priors with camera, geometry, physics, and memory controls, but TourPhysics emphasizes their explicit coordination for persistent interaction.
- Video generators provide strong appearance and motion priors, while temporal prediction and interpolation studies target consistency across adjacent frames.
- Camera-controlled and persistent approaches use poses, depth, optical flow, reconstructed evidence, point clouds, or explicit 3D/4D representations for viewpoint consistency.
- Physics-aware generation incorporates learned dynamics, trajectories, forces, material controls, or explicit simulation into image and video synthesis.
- Long-video and reference-conditioned methods extend temporal support through windows, recurrent contexts, bounded caches, correspondence, dynamic memory, and cross-attention.
3 Problem Formulation
TourPhysics formulates a persistent world model from one image and a declarative configuration, advancing fixed physical–camera windows and committing only accepted observations. The formulation separates scene lifting, coupled transitions, and long-horizon output.
- 3 Problem Formulation: The model takes a single RGB image and declarative configuration as input and targets long-horizon observations following declared physical–camera trajectories.
- 3 Problem Formulation: The formulation has three stages: one-time scene lifting, coupled physical and epistemic transitions, and long-horizon tour-and-manipulate video output.
- 3 Problem Formulation: State ownership is explicit: the physical branch determines simulator state and geometry, the observation branch synthesizes a fixed segment, and the epistemic branch publishes accepted evidence.
- 3.1 Scene Lifting and Persistent Reference: Scene lifting constructs a persistent hypothesis, estimates reference camera and calibrated geometry, and preserves the input image and cached background as permanent references.
- 3.1 Scene Lifting and Persistent Reference: The declarative configuration specifies controllable objects, material and physical-model assignments, boundary conditions, interventions, and camera trajectory controls.
- 3.1 Scene Lifting and Persistent Reference: At each action boundary, the committed state is represented as S_n = (X_n, C_n, K_n, E_n).
- 3.1 Scene Lifting and Persistent Reference: The index counts accepted windows; rejected attempts retry the same action without changing the index or committed state.
1. Scene Lifting (One-time Preprocessing)
The scene-lifting inputs combine a single image instance with object masks, background treatment, per-object meshes, and user-specified physical controls.
- The scene-lifting input includes a single image instance, object masks, background inpainting, and per-object 3D meshes.
- A user declarative configuration supplies the scene-level control specification.
- Material and physical-model assignments define the physical behavior used during scene construction and interaction.
- Forces and velocity keyframes provide explicit intervention controls for the physical hypothesis.
2. Coupled Physical and Epistemic Transitions
TourPhysics fixes each physical and camera trajectory before observation synthesis, then commits or discards the resulting state and appearance evidence atomically. Its observation model combines simulator geometry, generator-facing depth, a real-image anchor, and committed appearance memory to produce consistent windows for long-horizon exploration and manipulation.
- Physical transition: Each action window uses a fixed physical and camera trajectory before RGB observation synthesis.The associated geometric record supplies simulator-consistent depth, cameras, masks, motion, visibility, correspondence, and validity information.
- Observation synthesis: The observation model conditions generation on the geometric record, generator-facing depth, real-image anchor, and committed appearance memory.Generator-facing depth is constructed from the fixed geometric record and accepted tail evidence, while the real-image anchor remains immutable.
- Quality-gated commit: Candidate appearance memory and tail evidence remain private until the quality gate evaluates the current window.Rejected candidates discard staged updates and leave committed memory and accepted evidence unchanged.
- Quality-gated commit: An accepted window atomically publishes terminal simulator and camera states with staged visual evidence, while a retry changes only the sampling seed.The action, trajectory, geometric record, generator controls, and committed memory remain fixed across retries.
- Long-horizon output: The final video concatenates committed observations while removing the first frame of each subsequent window to avoid duplicated boundaries.Rejected attempts contribute no output frames, and the resulting sequence is intended to support free-viewpoint exploration, physical manipulation, and long-horizon consistency.
4 Methods
TourPhysics builds a persistent physical hypothesis from a single image and declarative configuration, then generates each observation from a fixed simulator-defined trajectory. Its geometric record and generator-facing controls remain distinct, while accepted evidence updates appearance without altering committed physical state or simulator geometry.
- Scene Lifting: Scene lifting constructs one editable physical hypothesis from the input image and frozen declaration, including explicit object meshes, a cached background plate, shared gravity-aware alignment, and calibrated reference geometry.The initialization is performed once before interaction and is not repeated as online state correction.
- Physical and Camera Transition: Each action clones the committed physical and camera states, computes a complete finite trajectory and geometric record, and takes the terminal candidate at the declared sampled endpoint.The fixed segment is rendered once and reused during retry, preventing accidental double advancement.
- Physical and Camera Transition: The physical backend routes rigid, articulated, continuum, volumetric, thin, and constraint-dominated systems to RBD, MPM, or PBD according to the declaration.Gravity and scheduled force fields continue during camera-only actions, so an action without intervention can still advance the physical state.
- Generator Controls: Simulator geometry controls projection, visibility, correspondence, and trajectory evaluation, whereas normalized relative depth is the separate control supplied to the video generator.The split prevents generator-facing depth from being interpreted as a new measurement of simulator depth.
- Appearance Memory: Historical appearance enters generator attention only through geometry-eligible, confidence-filtered correspondence, and the native reference-conditioned path is recovered exactly when no valid page exists.A target reads at most one earlier committed page, selected by valid coverage and mean confidence.
5 Experiments
TourPhysics evaluates trajectory adherence, physical interaction timing, and long-horizon appearance consistency on simulator-defined camera tours and manipulations. Across these tests, it preserves prescribed motion and improves scene and appearance stability.
- Experimental Setup: The benchmark contains 108 simulator-defined interaction sequences initialized from 61 source-image identities, spanning camera tours and rigid, cloth, elastic, granular, and fluid manipulations.Each sequence uses a single image and declarative configuration, with simulator-exported trajectories and geometric records.
- Experimental Setup: TourPhysics evaluates agreement with fixed camera and object trajectories rather than unconstrained video continuation.Each observation is generated for a fixed simulator window, and the next action starts only after the window-level commit rule passes.
- Camera and Object Motion: TourPhysics achieves the lowest camera ATE (0.0233), highest translation-direction hits (98.26%), highest rotation-direction hits (93.00%), and lowest object ADE (0.0297).It also reports the lowest endpoint error (0.0524) and 81.18% of retained objects within the 5% diagonal threshold.
- Simulator-Defined Interactions: TourPhysics reaches the lowest contact-timing error (3.3304 s), lowest area drift (0.0866), and highest recovery accuracy (64.29%).Its shape error is second only to LingBot-Cam on the supported subset; perceptual metrics are auxiliary diagnostics.
- Appearance and Camera-Control Diagnostics: Long-horizon diagnostics score 0.9743 for input-video subject consistency, 0.9791 for background consistency, and 0.8921 for camera-motion classification accuracy.These measures connect the observation branch and memory loop to retaining the reference scene while following intended camera motion.
- Long-Horizon Cache and Appearance Memory: The persistent sink cache reduces end-to-start drift from 26.33 to 11.17 and tail-to-head drift from 20.38 to 7.74.Geometry-routed memory further reduces historical-region error by 8.25% on development and 2.39% on held-out identities.
6 Conclusion
TourPhysics is an online framework for exploring and manipulating a controllable scene hypothesis initialized from one image and a declarative physical configuration. It separates simulation, geometry, generator controls, and accepted appearance memory, improving trajectory alignment, scene preservation, and long-horizon stability on simulator-defined tasks.
- Contribution: TourPhysics initializes an online, controllable scene hypothesis from a single image and declarative physical configuration.Its architecture assigns separate roles to simulator state, simulator-consistent geometry, generator controls, and accepted appearance memory.
- Contribution: For each action, TourPhysics simulates the physical and camera trajectory before generating an observation, committing terminal state and appearance evidence only after acceptance.Committed state and geometry remain fixed during synthesis and retry, while accepted observations update later memory and generator-facing depth.
- Conclusion: On simulator-defined tours and manipulations, TourPhysics improves alignment with prescribed trajectories, preserves the input scene, and reduces long-horizon appearance drift.The stated future directions are real-world physical evaluation, more complete scene geometry, and fixed-capacity memory for longer tours.
A Implementation and Evaluation Details
The implementation builds explicit scene proxies and a frozen physical declaration, then uses multi-physics simulation to generate fixed trajectory segments for quality-gated online observations. Static appearance memory is restricted to reliable regions and published only after commitment.
- Scene Initialization: TourPhysics lifts image-mask pairs into explicit triangular meshes that provide simulator contact and collision interfaces.Foreground geometry is removed cumulatively to create an inpainted background plate, while the unmodified input image remains the permanent appearance reference.
- Scene Initialization: Gravity alignment fits a robust ground plane from low object anchors, rotates the scene to the world z-axis, and corrects residual overlaps within the declared initial hypothesis.The method uses the lowest 5% of vertices by default and relaxes to 10% when too few anchors are available.
- Reference-Camera and Depth Alignment: Camera refinement aligns rendered mesh silhouettes with the input image and fits affine depth alignment over pixels with valid monocular and simulator depth.The procedure uses coarse-to-fine camera search followed by bounded image-plane correlation refinement.
- Physical Configuration: A structured physical declaration specifies material families, continuous parameters, boundary conditions, optional initial velocity, keyframes, force fields, and camera actions.An optional vision-language proposal is whitelist-checked, completed with defaults, manually editable, and frozen before trajectory simulation.
- Multi-Physics Solver Routing: The system routes rigid objects to RBD, continuum materials to MPM, and thin shells or constraint-dominated particles to PBD.MPM supports elastic, plastic, granular, and fluid trajectories, while PBD enforces stretch, bending, volume, density, and attachment constraints.
- Trajectory Simulation and Sampling: For each action, the simulator advances a working copy, records state, camera pose, depth, visibility, labels, and confidence, and fixes the segment during video generation.The terminal state is committed only after the resulting observation passes the quality gate; simulation runs at 60 fps and generated observations at 16 fps.
- Reference-Anchored Appearance Memory: Only reliable static regions enter long-term memory; moving and deforming objects, unreliable boundaries, and pages before window commitment are excluded.The support uses declared static support, motion/deformation fields, and a fixed confidence threshold.
B.1 Benchmark and Comparison Setting
The benchmark tests whether generated observations agree with declared simulator trajectories across camera tours and object manipulations. It does not test whether the declarative physical parameters are the true parameters of the source scene.
- Benchmark: The benchmark contains 108 interaction sequences from 61 source-image identities, covering camera tours and rigid, cloth, elastic, granular, and fluid proxies.When sequences share an input image, the evaluation reports sequence-level and identity-balanced statistics where appropriate.
- Scope: The benchmark evaluates agreement with a declared simulation trajectory, not recovery of the source scene’s true physical parameters.Single-image initialization leaves scale, occluded geometry, density, friction, and deformation parameters underdetermined.
- Comparison Setting: Comparison methods expose different conditioning interfaces, and temporally normalized results are not treated as equivalent to native online generation.Inputs may include only an image, an explicit camera trajectory, an event description, or privileged control.
B.2 Camera and Object Motion
The section defines paired deterministic protocols for evaluating camera and object-motion adherence, deformation, contact, response, and auxiliary perceptual quality. Metrics use visibility-filtered simulator/generated comparisons with sequence-level aggregation.
- Camera motion: Camera adherence is evaluated with aligned ATE, PLR, translation-direction, and rotation-direction metrics over valid reference sequences.ATE and PLR use a single sequence-wide similarity transform; direction metrics test the correct half-space for eligible translation or rotation segments.
- Object motion: Object-motion evaluation tracks dynamic-object centers from paired simulator and generated videos using frozen masks and normalized image-diagonal distances.Records are filtered by joint visibility and aggregated within sequences before equal-weight averaging across sequences.
- Interaction and deformation: Contact, response, overlap, shape, area, and recovery metrics quantify whether generated objects interact, move, deform, and recover consistently with the simulator.Contact and response use simulator-defined targets and thresholds, while deformable-object metrics require sufficient jointly tracked frames.
- Interaction and deformation: Area Drift is a dimensionless projected-area proxy rather than a mass-conservation measure, and it requires at least eight valid frames.Shape Err. likewise uses normalized, Procrustes-aligned point configurations for deformable objects.
- Auxiliary diagnostics: VideoPhy-2 AutoEval reports Semantic Adherence, Physical Commonsense, and their arithmetic mean as auxiliary perceptual diagnostics, not trajectory verification.These scores use custom-data prompts on a 1–5 scale and are separate from the paired deterministic metrics.
- Auxiliary diagnostics: TourPhysics obtains 0.9743 input-video subject consistency, 0.9791 input-video background consistency, and 0.8921 camera-motion classification accuracy.These quantities assess preservation of source appearance and intended camera-motion class under the reported protocol.
C.1 Cache and Memory Ablations
The ablations show that persistent memory improves long-horizon appearance stability beyond a finite temporal cache, while geometry-routed memory reduces historical and known-region errors. When correspondence is unavailable, the system returns exactly to its native observation path.
- Memory ablation: Geometry-routed appearance memory reduces historical-region error by 8.25% on development data and 2.39% on the held-out split.The held-out improvement is smaller because camera coverage and geometric correspondence quality vary.
- Memory ablation: Disabling all valid historical correspondences makes the memory residual zero and recovers the native observation path exactly.This fallback prevents unavailable or unreliable memory from adding visual bias.
C.2 Limitations
TourPhysics is a controllable physical hypothesis rather than a unique reconstruction of the real scene, and its long-horizon appearance preservation remains bounded by incomplete geometry, object-memory scope, runtime, and memory growth.
- Physical scope: Physical behavior depends on supplied object proxies, material families, boundary conditions, and simulator parameters, so trajectory adherence does not guarantee real-world physical accuracy.The system represents a controllable physical hypothesis rather than a unique real-scene reconstruction.
- Geometry and appearance scope: Incomplete occluded or newly exposed geometry limits the available scene evidence, while memory excludes moving and deforming objects.Incorrect camera or depth estimates can further reduce valid cross-view correspondences.
- System scope: Finite-window diffusion is not real-time, and uncompressed historical pages grow with the number of accepted views.The paper identifies fixed-capacity eviction, object-local deforming-surface memory, real-world evaluation, and longer tours as future directions.