Source-linked AI summary
4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting
Yueen Ma, Zenglin Xu, Irwin King
TL;DR
Image-based world action models can produce visually faithful predictions but repeatedly process redundant backgrounds and leave object-level spatial structure implicit. 4DGS-WAM lifts observations into a persistent 4D Gaussian representation, predicts dynamic-object actions and splat transformations, and reuses accumulated static content; on KITTI-MOT, it reports strong reconstruction and short-horizon prediction results, with interaction-related limitations.
Problem
Image-based WAMs lack explicit object-level spatial structure and repeatedly regenerate redundant static background content.
Method
4DGS-WAM decomposes an explicit 4D Gaussian state into static background and dynamic objects, then uses policy actions and a world model to transport dynamic splats while reusing static content.
Results
On KITTI-MOT, accumulated 4DGS reconstruction achieves higher reported full-frame metrics than 3DGS, point-cloud, and other 4DGS baselines, while policy-fed prediction leads evaluated video models at a given camera.
Takeaways & Limitations
Persistent 4D scene state lets future prediction focus on dynamic-object evolution rather than regenerating previously observed static content.
Takeaways & Limitations
The per-object world model omits collisions and newly appearing objects, lacks shape completion for unobserved surfaces, and depends on vision-foundation-model perception.
Abstract
from arXiv · showhide
Current world action models (WAMs) typically operate on 2D visual data. These models can achieve exceptional visual quality, but they lack explicit spatial structure for individual objects and repeatedly process redundant background content. Although point clouds can represent the world in 3D space, they can be difficult to align and accumulate across viewpoints. In this paper, we leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene. For dynamic objects, we use a policy model to predict future actor actions and a world model to predict transformations of their observed Gaussian splats. The static background need not be regenerated for future states, as much of it has already been observed in past frames. This forms an object-centric world action model, which we name 4DGS-WAM. It lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction. Future-state extrapolation can then focus on modeling the evolution of dynamic objects. Experiments on KITTI-MOT evaluate short-horizon prediction and past reconstruction.
1 INTRODUCTION
4DGS-WAM addresses the redundancy and implicit spatial structure of image-based world action models by maintaining an explicit, object-centric 4D representation. It separately models persistent static content and action-driven dynamic objects for reconstruction and future prediction.
- Motivation: Image-based WAMs repeatedly regenerate redundant static backgrounds while leaving 3D structure and dynamic-object motion implicit.These limitations arise because consecutive frames contain substantial unchanged content and image-space models must repeatedly generate it.
- Method: 4DGS-WAM represents scenes with explicit 4D Gaussian Splatting and decomposes them into persistent static and evolving dynamic components.The static component captures accumulated background content, while the dynamic component represents objects whose states change over time.
- Method: A policy predicts future actions for dynamic objects, and a world model predicts transformations of their Gaussian splats instead of regenerating the entire future image.The extrapolated dynamic component is then composed with the accumulated static component to represent future scenes.
- Method: Vision foundation models provide segmentation, depth, camera pose, and optical-flow cues that lift observations into common world coordinates and track object motion.Optical flow additionally supplies pixel-level cues for modeling non-rigid object transformations.
- Evaluation: 4DGS-WAM is evaluated on KITTI-MOT for future prediction and reconstruction against video, 4DGS, 3D mapping, and point-cloud methods.The reported evaluation includes rendering accumulated 4DGS states from observed camera views and comparing future extrapolation with video- and 4DGS-based world models.
2 RELATED WORK
Prior world action models primarily predict images, videos, and actions in unified token or diffusion frameworks, while 4DGS-WAM maintains an explicit decomposed Gaussian state. Related 4D reconstruction and mapping methods provide spatial representations but generally do not model action-conditioned future states.
- 4D Gaussian Splatting: 4DGS-WAM maintains a decomposed, renderable 4D Gaussian state whose static bank persists while policy-conditioned dynamics transport only dynamic object splats.This design is contrasted with unified image-and-action token prediction.
- World action models: Representative autoregressive WAMs jointly predict future image and action tokens in unified models, while DreamZero jointly generates future video and actions with video diffusion.The comparison places these approaches in the image- or video-based WAM family.
- World action models: Driving-focused methods generate future video, with Epona additionally modeling future trajectories, whereas PointWorld predicts action-conditioned 3D point flows.These methods differ in whether they represent future content as video, trajectories, or point motion.
- 4D Gaussian Splatting: 4DGS-SLAM and Flow4DGS-SLAM extend Gaussian splatting to dynamic mapping, but they do not model action-conditioned future states.NeoVerse supports novel-trajectory video generation, while DriveDreamer4D is another related 4D reconstruction direction.
- Vision foundation models: Vision foundation models supply complementary cues for segmentation, object identity, depth, camera geometry, and optical flow in object-centric 4D reconstruction.The cited components include SAM 3, DA3, VGGT-family models, and WAFT.
3 METHOD
4DGS-WAM reconstructs observations as an object-centric 4D Gaussian state, then predicts future actor actions and dynamic Gaussian transformations while reusing the static background. Its policy operates on actor trajectories, and its world model applies action-conditioned geometric and appearance updates before rendering future views.
- Overview: The pipeline first reconstructs past observations into a scene-level 4DGS state, then extrapolates that state and renders future observations.The state contains active Gaussian snapshots, object associations, and trajectories.
- Past perception: Vision foundation models provide segmentation, optical flow, depth, and camera poses for object tracking and world-coordinate lifting.These estimates produce 3D object trajectories and flow correspondences in a common coordinate frame.
- Policy: The time-conditioned policy receives actor-center trajectories and predicts one six-vector SE(3) action for each actor at the queried horizon.For horizons h > 1, the action is produced in a single evaluation without autoregressive intermediate actions; the ego action determines the future camera pose.
- World model: The world model conditions on horizon and predicted actions, updates dynamic-object Gaussians with residual SE(3) twists, and composes those transformations with object actions.It updates Gaussian means, orientations, and appearance, while the accumulated static background is rendered at the target camera pose.
- Training: The method supervises geometry with 3D flow transport and appearance with masked photometry.The stated transport objective combines splat and center terms, while photometry is evaluated within target-time object masks.
- Evaluation: Table 1 evaluates future prediction on KITTI-MOT at horizons h=1 and h=3 using mean±std over three sequence-level means.All methods receive the same observed prefixes and target frames; given-camera and predicted-camera rows are treated separately.
4 EXPERIMENTS
Experiments on KITTI-MOT evaluate future-observation prediction and past reconstruction using qualitative comparisons and reported metrics. 4DGS-WAM leads video baselines at the given camera and achieves the highest reported reconstruction metrics, with evaluation-specific rendering conditions.
- Evaluation setup: 4DGS-WAM is evaluated on KITTI-MOT for future-observation prediction and past reconstruction of observed frames.The experiments compare future RGB predictions with video and 4DGS world models and compare reconstructions with point-cloud, 3DGS, and 4DGS mapping methods.
- Future prediction: Future-prediction comparisons use the same observed prefixes, while methods differ in whether they render at given or predicted future cameras.Epona and DriveDreamer-2 use given future cameras; Envision4D and 4DGS-WAM use predicted cameras in the corresponding setting.
- Future prediction: At the given camera, policy-fed 4DGS-WAM leads the video baselines on all three metrics.At the predicted ego camera, it trails Envision4D, which jointly models content and camera.
- Past reconstruction: 4DGS-WAM has the highest reported full-frame and dynamic-region metrics for past reconstruction on KITTI-MOT.The reconstruction merges static leftovers and dynamic object splats; full-frame scores use a frozen NeoVerse fusion underlay, while dynamic-region scores use no fusion.
- Qualitative results: Qualitative future-prediction panels compare ground truth, Epona, DriveDreamer-2, Envision4D, and policy-fed 4DGS-WAM across horizons h=1 and h=2.The 4DGS-WAM panels include the frozen fusion underlay.
- Qualitative results: Qualitative past-reconstruction panels compare 3DGS, MonoGS, EmbodiedSplat, VGGT-Ω, VGGT-SLAM, 4DGS-SLAM, Flow4DGS-SLAM, and 4DGS-WAM against ground truth.The 4DGS-WAM panel shows its merged static–dynamic reconstruction with the frozen fusion underlay.
5 CONCLUSION
The conclusion presents 4DGS-WAM as an object-centric world action model that maintains an explicit 4D Gaussian Splatting state and predicts future changes through dynamic objects.
- 5 CONCLUSION: 4DGS-WAM reconstructs observations into static-background and dynamic-object Gaussians, then predicts actor actions and transports only dynamic splats.The static background remains accumulated in place while the policy predicts SE(3) actor actions, including ego motion.
- 5 CONCLUSION: On KITTI-MOT, 4DGS-WAM obtains the highest reported full-frame and dynamic-region reconstruction metrics.Policy-fed short-horizon prediction leads the evaluated video world models at a given camera.
- 5 CONCLUSION: The persistent 4D representation reuses reconstructed static content while future extrapolation models the evolution of dynamic objects.This summarizes the model’s object-centric prediction strategy.
6 LIMITATIONS AND FUTURE WORK
The paper identifies scope and modeling limitations involving object interactions, unseen content, perception errors, evaluation, and rollout horizon.
- Modeling scope: The per-object world model assumes objects do not collide and cannot instantiate objects appearing after time t.These approximations fit the non-contact driving examples but are insufficient for manipulation involving contact and inter-object interactions.
- Representation limits: The current 4DGS state lacks shape completion for unobserved object sides, such as a vehicle’s back seen only from the ego camera.Handling deformable objects may require more sophisticated 4DGS representations.
- Perception dependence: Perception errors from the vision foundation model suite can propagate into the reconstructed state and predicted dynamics.The method depends on segmentation, depth, camera-pose, and related perceptual estimates.
- Rendering boundary: The static bank cannot synthesize newly revealed background regions, so quantitative renders use a frozen fusion underlay for uncovered pixels.Baselines are scored on their native outputs, creating different rendering conditions for full-frame comparison.
- Evaluation scope: Prediction is evaluated only at short horizons, leaving longer-horizon rollout and a unified end-to-end model for future work.The stated evaluation scope does not establish long-term rollout behavior.
A.1 NOTATION
The notation section records the symbols used in the main text and implementation details, including conventions for vectors and structured states.
- A.1 NOTATION: Table 3 collects the notation used in the main text and implementation details.The table follows the conventions established throughout the paper.
- A.1 NOTATION: Bold lower-case symbols denote vectors, while calligraphic symbols denote sets or structured states.These conventions distinguish common symbol categories in the paper’s equations and descriptions.
A.2 POLICY NETWORK
The policy predicts target-horizon actor actions as bounded residuals over a constant-velocity prior, using recent actor and ego observations with horizon conditioning and temporal attention.
- Action convention: The six-vector action convention reduces to planar yaw about the vertical axis when roll and pitch are zero.
- Action prediction: The action head predicts target-horizon actions as bounded residuals over a per-step constant-velocity prior.The horizon map converts the per-step parameterization into the cumulative transform passed to the world model.
- Input representation: Each actor token sequence uses the latest eight center-state observations, with an optional uncertainty feature and a matching ego-token layout.
- Temporal modeling: Tokens are projected to width 96 and processed by three causal temporal-attention blocks with four heads, RMS normalization, and SwiGLU layers.A learned horizon embedding conditions history tokens and a memory token that attends to valid observations.
A.3 WORLD MODEL
The world model represents object splats and sampled part nodes in a typed geometric graph, then decodes action-conditioned residual motion and appearance updates.
- Part-token construction: Farthest-point sampling creates K part seeds, with K=1 for most vehicle categories, K=3 for bicycles and motorcycles, and K=5 for people.The K=1 configuration provides a rigid fallback.
- Action conditioning: Splat and part features encode supplied actions through invariant scalar descriptors and equivariant vector channels.Action direction enters vector channels directly, while invariant action features condition scalar updates through FiLM.
- Typed geometric tensor layers: The packed graph uses splat–splat kNN edges, bidirectional splat–part edges, and a complete part–part graph with attention-pooled edge invariants.The splat–splat neighborhood uses k=16.
- Residual motion and composition: Only part nodes decode residual twists, and the final splat–part attention produces the weights used in motion composition.
- Residual motion and composition: The composed transformation applies rotation and translation to splat positions, while zero-initialized twist decoding starts from the supplied rigid action.
- Appearance transport: The appearance head transports local source color and predicts bounded color, opacity, and scale-related updates.Its final layers are zero-initialized, so the untrained appearance path is an identity update.
A.4 TRAINING OBJECTIVES
Training combines normalized policy supervision with world-model losses for geometry, center transport, photometric reconstruction, missing alpha, and multi-horizon prediction.
- Policy loss: The policy objective operates in normalized residual coordinates and uses teacher residuals, dimension masks, confidence values, and an epsilon stabilizer.
- Policy loss: Actor losses are confidence-weighted within each sample, then sample losses are averaged over active samples.This implements the schematic action-space loss in Equation 4.
- World-model geometry losses: The world-model objective includes dense flow supervision and a center-transport term over supervised correspondences.The center-transport term averages predicted displacements over the correspondence index set.
- World-model geometry losses: World-model supervision uses the supplied target-horizon teacher action for geometric correspondence targets.
- Photometric and total loss: Photometric loss measures mean absolute RGB error over ground-truth object masks, with an additional missing-alpha penalty.The missing-alpha term uses rendered alpha attribution for each object.
- Photometric and total loss: For multi-horizon examples sharing one observed prefix, the world objective aggregates supervision across the associated horizons.
- Photometric and total loss: The total world loss combines photometric, missing, splat, and center terms with weights 1, 0.2, 0.5, and 0.5, plus a 0.05 anchor term.The anchor term constrains geometry parameters toward their initialization.