PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation

Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu, JoungBin Lee, Jiyoung Kim, Siyoon Jin, Yunsung Lee, Jaeyoon Jung, Suhwan Choi, Seungryong Kim, Yang Zhou

arXiv:2603.16871v1cs.CV

TL;DR

Existing interactive gaming world models struggle with precise action control and 3D consistency, while available gameplay datasets are limited in geometry, motion diversity, or reproducibility. WorldCam uses camera pose as a geometric anchor, combining Lie-algebra-based action control with pose-based retrieval, and reports improvements in action accuracy, long-horizon visual quality, and 3D consistency.

  • Problem

    Interactive gaming world models still struggle with precise action control and 3D world consistency, while existing gameplay datasets often have simplified geometry or restricted accessibility.

  • Method

    WorldCam uses camera pose as a unifying geometric representation, deriving precise 6-DoF poses from Lie-algebra action inputs and using camera-pose retrieval for long-horizon generation.

  • Results

    WorldCam improves action accuracy, long-horizon visual quality, and 3D consistency over prior interactive world models and camera-controlled baselines.

  • Takeaways & Limitations

    WorldCam-50h provides 3,000 minutes of annotated human gameplay, including open-licensed games, supporting reproducible research on interactive gaming worlds.

  • Takeaways & Limitations

    The framework adopts camera-pose retrieval because accurate field-of-view computation would require precise generated-frame depth maps and add overhead.

Abstract

from arXiv · show

Recent advances in video diffusion transformers have enabled interactive gaming world models that allow users to explore generated environments over extended horizons. However, existing approaches struggle with precise action control and long-horizon 3D consistency. Most prior works treat user actions as abstract conditioning signals, overlooking the fundamental geometric coupling between actions and the 3D world, whereby actions induce relative camera motions that accumulate into a global camera pose within a 3D world. In this paper, we establish camera pose as a unifying geometric representation to jointly ground immediate action control and long-term 3D consistency. First, we define a physics-based continuous action space and represent user inputs in the Lie algebra to derive precise 6-DoF camera poses, which are injected into the generative model via a camera embedder to ensure accurate action alignment. Second, we use global camera poses as spatial indices to retrieve relevant past observations, enabling geometrically consistent revisiting of locations during long-horizon navigation. To support this research, we introduce a large-scale dataset comprising 3,000 minutes of authentic human gameplay annotated with camera trajectories and textual descriptions. Extensive experiments show that our approach substantially outperforms state-of-the-art interactive gaming world models in action controllability, long-horizon visual quality, and 3D spatial consistency.

1 Introduction

WorldCam addresses the coupled challenge of precise action control and long-horizon 3D consistency by using camera pose as a shared geometric representation. It introduces physics-based action-to-camera mapping, pose-based memory retrieval, and a gameplay dataset, with experiments showing improvements across the targeted capabilities.

  • Existing interactive gaming world models struggle with precise action control and 3D world consistency despite plausible visual outputs.
  • WorldCam uses camera pose as a unifying geometric representation for immediate action control and long-horizon 3D consistency.
  • Physics-based continuous actions are represented in Lie algebra and converted into geometrically accurate 6-DoF poses injected through a camera embedder.
  • Camera-pose similarity retrieves relevant past latents and pose embeddings to establish geometric correspondences during long-horizon generation.
  • WorldCam-50h contains 3,000 minutes of authentic gameplay from three games, annotated with textual descriptions and pseudo ground-truth camera poses.
  • Extensive experiments report simultaneous gains in precise action alignment, sustained long-horizon visual quality, and robust 3D spatial consistency over prior methods.

2 Related Work

Prior work uses raw actions, text, or simplified action-to-camera mappings, while gaming datasets often lack visual diversity, geometric complexity, or reproducible access. These limitations motivate explicit geometric modeling and more suitable gameplay data.

  • Interactive gaming models commonly inject raw action signals through cross-attention, AdaLN, or text, which can produce inaccurate camera motion and poor 3D consistency.
  • GameCraft linearly approximates user inputs into camera poses but does not capture the coupled dynamics of complex motion.
  • Prior gaming datasets often use Minecraft data with limited visual diversity and simplified geometry, or closed-licensed videos that hinder reproducibility.

3 WorldCam

WorldCam autoregressively generates interactive video conditioned on geometrically derived camera motion. It uses pose accumulation and pose-indexed memory, alongside progressive noise scheduling, to support action alignment, long-horizon consistency, and stable rollout.

  • WorldCam autoregressively generates frames that follow user actions, remain consistent with one 3D world, and preserve visual quality over long horizons.
  • Action-to-Camera Mapping: User actions are represented as se(3) twists and mapped through the matrix exponential to relative camera poses in SE(3).The twist contains linear and angular velocities, while the 4 × 4 twist matrix is exponentiated to obtain the relative pose.
  • Action-to-Camera Mapping: Linear and angular velocities are jointly integrated on the SE(3) manifold, preserving coupled translation–rotation dynamics in complex actions.
  • Camera Conditioning: Relative poses are accumulated into global camera poses, converted into Plücker embeddings, and injected into DiT features through a camera embedding module.Temporal compression is handled by concatenating consecutive Plücker embeddings for each latent frame.
  • Pose-Anchored Long-Term Memory: Global camera poses index long-term memory: WorldCam retrieves nearby positions, then selects entries with aligned viewing directions before conditioning generation.Retrieved latents and their realigned pose embeddings are concatenated with the current latent sequence to establish geometric correspondences.
  • Progressive Autoregressive Inference: Progressive per-frame noise uses monotonically increasing levels and stage-wise denoising, then shifts the sequence by decoding the earliest frame and appending pure noise.The schedule provides a low-noise early-frame anchor while keeping future frames correctable.
  • WorldCam-50h: Figure 3 summarizes gameplay samples, duration distributions, and linear and angular velocity distributions, illustrating diverse human gameplay behavior.
  • Progressive Autoregressive Inference: An attention sink retains initial frames as anchors to preserve frame fidelity, scene style, and UI consistency during inference.

4 WorldCam-50h

WorldCam-50h is a large-scale human-gameplay dataset designed to address limited visual diversity and restricted reproducibility in existing training data. It provides textual descriptions and camera-pose annotations from varied 3D-game environments.

  • WorldCam-50h addresses prior datasets’ limited visual diversity and closed licensing by providing large-scale human gameplay videos with detailed annotations.
  • Data Collection: The dataset records gameplay from Counter-Strike, Xonotic, and Unvanquished, selected for complex 3D environments and high interactivity.The collected data focuses on single-player exploration of static environments.
  • Captioning: Each training video chunk receives a detailed caption describing static world layout, visual theme, and ambient environmental conditions.Captions are generated with Qwen2.5-VL-7B using a prompt focused on global spatial and visual properties.
  • Camera Annotation: Global camera intrinsics and extrinsics are estimated for each one-minute segment, with filtering applied to remove unrealistically large translations.

5 Experiments

WorldCam is evaluated against interactive gaming and camera-controlled video models across action controllability, visual quality, and long-horizon 3D consistency. It consistently outperforms prior methods, while ablations support Lie algebra action mapping, pose-based memory retrieval, and long-term context for spatial consistency.

  • Evaluation setup: Experiments compare WorldCam with interactive gaming world models and camera-controlled video generation methods across action controllability, visual quality, and 3D consistency.The evaluation uses 200-frame sequences for interactive gaming models, while camera-controlled methods are evaluated with 16-frame sequences because they lack long-horizon inference.
  • Quantitative comparison: WorldCam consistently outperforms interactive gaming world-model baselines by a large margin on action controllability, visual quality, and 3D consistency in 200-frame sequences.The comparison covers the three principal evaluation aspects on long-horizon generations.
  • Action controllability: 16.3% RPEcamera improvement is reported over GameCraft, while the short-horizon comparison shows a 36.1% improvement over CameraCtrl.The reported values are 0.100 versus 0.086 for GameCraft and 0.030 versus 0.083 for CameraCtrl.
  • Visual quality: 0.844 VBench average is achieved for long-horizon visual quality, exceeding GameCraft’s 0.781 by a relative margin of 8.1%.The paper attributes this improvement to progressive noise scheduling, attention sinks, and longer short-term latents.
  • 3D consistency: WorldCam achieves the best performance on all reported 3D-consistency metrics, while Yume’s comparable PSNR is associated with substantially lower sharpness.Sharpness is reported as 95 for Yume versus 577 for real videos.
  • Ablation study: The ablations show that Lie algebra action mapping, camera-pose memory retrieval, and additional long-term memory improve the corresponding control or spatial-consistency measures.Pose-based retrieval outperforms random and temporal retrieval; more long-term memory improves PSNR and SSIM, while visual quality remains stable without it.

6 Conclusion

WorldCam uses camera pose as a unified geometric anchor for precise action control, long-horizon generation, and consistent 3D world modeling. Experiments show improvements over prior models and camera-controlled baselines, supported by qualitative long-horizon results.

  • WorldCam uses camera pose as a unified geometric anchor for precise action control, long-horizon generation, and consistent 3D world modeling.
  • The WorldCam-50h dataset contains human gameplay from open-licensed games to support research reproducibility.
  • WorldCam improves action accuracy, long-horizon visual quality, and 3D consistency over prior interactive world models and camera-controlled baselines.
  • Qualitative results show fine-grained control with simultaneous keyboard and mouse inputs and videos exceeding 10 seconds at 20 FPS without error drift.
  • Revisited views preserve consistent underlying 3D geometry beyond the denoising window.

Appendix Overview

The appendix covers data preprocessing, comparisons with existing methods, Lie algebra-based action-to-camera modeling, and limitations with runtime analysis.

  • Appendix Overview: Section A describes the data preprocessing pipeline.
  • Appendix Overview: Section B compares existing works and details evaluation for each comparison baseline.
  • Appendix Overview: Section C explains linear and Lie algebra-based action-to-camera approximations.
  • Appendix Overview: Section D discusses limitations and runtime comparisons.

A Data Preprocessing

The preprocessing pipeline segments gameplay into long clips, extracts camera parameters, and organizes short- and long-term memory for generation.

  • Videos averaging 8 minutes are segmented into 1-minute clips containing 1,800 frames at 30 FPS.
  • ViPE extracts global camera extrinsic and intrinsic parameters for each clip.
  • Each 1,800-frame clip is divided into 64-frame segments with 32 frames for short-term memory and 32 for generation.
  • For each segment, four 4-frame long-term memory clips are selected from the remaining 1,736 frames using camera translation and rotation similarity.
  • VAE latents are precomputed and cached to reduce training time.

B.1 Comparison with Interactive Gaming World Models

Prior interactive gaming world models generally fail to satisfy precise action control, 3D consistency, and long-horizon inference together. WorldCam unifies these requirements through camera poses.

  • Comparison with Interactive Gaming World Models: The comparison evaluates WorldCam against Matrix-Game 2.0, GameCraft, and Yume on interactive gaming world-model requirements.
  • Comparison with Interactive Gaming World Models: The three key requirements are precise action control, 3D consistency, and long-horizon inference.
  • Comparison with Interactive Gaming World Models: Raw-action methods cannot ensure 3D consistency because raw actions cannot serve directly as spatial indices for memory retrieval.
  • Comparison with Interactive Gaming World Models: GameCraft uses camera poses for action control but does not use them to maintain long-term 3D consistency.
  • Comparison with Interactive Gaming World Models: WorldCam adopts camera poses as a unified representation for precise action control and long-horizon 3D consistency, satisfying all three requirements.

B.2 Evaluation Details of Comparison Works

The evaluation translates WorldCam’s continuous camera-pose actions into each comparison model’s predefined action interface. These conversions account for differences in action granularity, speed control, and generation structure.

  • Matrix-Game 2.0: Matrix-Game 2.0 receives WorldCam actions by thresholding linear velocities into binary keyboard inputs and angular velocities into mouse movements.The mappings are vx, vz to W, S, A, D and ωx, ωy to dy, dx, respectively.
  • Matrix-Game 2.0: Matrix-Game 2.0 injects raw actions directly, while WorldCam maps actions to camera poses for precise control and long-horizon 3D consistency.Matrix-Game 2.0 uses cross-attention for keyboard actions and intermediate feature injection for mouse movements.
  • GameCraft: WorldCam maps its velocity-based actions into GameCraft’s predefined actions and scales velocities to match GameCraft’s speed settings.The evaluation selects a speed scaling factor for compatibility with the pretrained model.
  • GameCraft: WorldCam’s action-to-camera mapping uses Lie algebra, whereas GameCraft relies on linear approximations and chunk-wise generation.WorldCam instead uses autoregressive progressive noise scheduling for latent-by-latent generation and predicted camera poses for long-horizon consistency.
  • Yume: Yume encodes actions through textual prompts, so WorldCam actions are averaged across eight latents, thresholded, and converted into Yume’s predefined textual action space.Speed-related cues are added to the prompts to approximate motion speed, but the resulting control is relatively coarse.
  • Yume: Unlike Yume’s text-based action encoding and FramePack history compression, WorldCam trains directly on camera poses and retrieves only relevant historical latents.This comparison highlights the distinct action and memory interfaces used by the two models.

C Action-to-Camera Approximation

The paper compares independent linear updates with joint Lie algebra integration on SE(3) for converting 6-DoF actions into camera poses. Joint integration captures coupled translation and rotation, avoiding the trajectory errors produced by linear approximations.

  • Action Representation: The action at each timestep is represented as a 6-DoF velocity with linear velocity vi and angular velocity ωi.The vectors vi and ωi each lie in R3 and together describe the action’s translational and rotational components.
  • Linear Approximation: Linear approximations update translation and rotation independently, failing to capture coupled rigid-body motions.Under screw motion, this separate treatment leads to accumulated trajectory drift.
  • Lie Algebra Approximation: Equation 13 jointly integrates linear and angular velocities on SE(3), making translation part of a rigid transformation coupled with rotation.The resulting representation captures screw motions and other complex camera trajectories involving simultaneous translation and rotation.
  • Trajectory Comparison: In a forward-while-turning example, the ground-truth camera follows a smooth curved path that linear approximation misses, while the Lie algebra formulation closely matches it.The comparison is visualized in Figure 7.
  • Quantitative Comparison: Across 50 trajectories of length 200 frames, Lie algebra-based approximation maintains near-zero RPEtrans and ATE, unlike linear approximations with large relative and accumulated errors.ATEavg and ATEfinal measure drift over the full sequence and at the final frame, respectively.

D Limitation

Inference efficiency remains a limitation of the approach. Although latent-wise generation supports low-latency interaction, the method uses more sampling steps than distilled alternatives and does not use distillation.

  • Efficiency Limitation: Inference efficiency remains a limitation despite improvements in generation quality and controllability.The authors identify acceleration and diffusion-distillation techniques as possible future remedies.
  • Evaluation Setting: The inference-time comparison evaluates methods at 480 × 832 resolution on a single H100 GPU.This is the stated setting for Table 12’s complexity comparison.
  • Chunk-wise Generation: Chunk-wise models generate 32 or 33 frames per chunk but require all corresponding actions before generation, limiting responsiveness.This design improves throughput per chunk but is less suited to interactive world models requiring low-latency interaction.
  • Latent-wise Generation: WorldCam generates one latent token using eight sampling steps and does not use distillation, whereas Matrix-Game 2.0 generates three latent tokens every three steps with distillation.Both models use latent-wise autoregressive generation for low-latency interaction.
Loading 2603.16871v1…