PaperlayerView exact revision on arXiv

Source-linked AI summary

SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Models

Zizhao Tong, Yeying Jin, Hongfeng Lai, Zeqing Wang, Zhaohu Xing, Kexu Cheng, Haoran Xu, Zhao Pu, Shangwen Zhu, Ruili Feng, Jian Zhao, Yan Zhang, Hao Tang, Ling Shao

arXiv:2605.23345v2cs.CV

TL;DR

FPS world models struggle with dense, overlapping controls because global action conditioning perturbs unaffected regions. SCOPE uses per-pixel action conditioning and CrossFPS to separate localized action effects from stable scene generation, achieving robust controllability and zero-shot generalization to unseen scenes.

  • Problem

    Global action conditioning fails under dense FPS inputs by perturbing every pixel when commands should affect only localized regions.

  • Method

    SCOPE uses per-pixel conditioning with local visual content to separate in-scope discrete responses from out-of-scope continuous generation, trained on the CrossFPS multi-game dataset.

  • Results

    SCOPE demonstrates robust controllability, effective scope separation, and zero-shot generalization to unseen scenes across FPS games.

  • Takeaways & Limitations

    CrossFPS training supports general visual-to-action mappings that transfer across game titles without retraining.

  • Takeaways & Limitations

    Generalization remains limited for multi-step weapon mechanics, item use, fine-grained manipulation, and geometric transformations because interaction diversity is limited and the diffusion backbone is texture-biased.

Abstract

from arXiv · show

Interactive world models for first-person shooter (FPS) games must resolve high-frequency overlapping control signals at every frame without disrupting unaffected regions. Existing methods inject actions globally and train on single titles, failing under dense FPS inputs. We observe that FPS actions are spatially selective: discrete events such as firing or reloading affect only a localized region around the weapon (the scope), while continuous camera and movement signals govern stable surroundings. We propose SCOPE, which inserts a conditioning module into each transformer block of a pretrained video diffusion model. It reshapes features into per-pixel temporal sequences so that each position computes its action response from local visual content. This separates in-scope effects from out-of-scope generation without segmentation labels. We also introduce CrossFPS, the first multi-game FPS dataset with frame-aligned action telemetry. It comprises 69K clips from 7 titles with 10-DoF controller signals, curated to remove gameplay bias. The model learns general visual-to-action mappings rather than game-specific patterns, enabling zero-shot transfer to unseen scenes. Experiments confirm strong action responsiveness, precise scope separation, and effective cross-game generalization.

1 Introduction

SCOPE addresses dense FPS control by modeling action effects spatially: localized discrete events alter the weapon-centered scope, while continuous camera and movement controls preserve surrounding regions. CrossFPS provides multi-game, frame-aligned telemetry to support zero-shot generalization across unseen FPS scenes.

  • Motivation: FPS gameplay challenges global action conditioning with rapid camera sweeps, simultaneous firing and movement, and chained discrete events within one generation window.Existing methods broadcast a single action embedding uniformly across spatial positions, which is ill-suited to dense overlapping controls.
  • Motivation: FPS actions are spatially selective: firing and reloading affect a localized weapon-centered scope, whereas walls, sky, and distant environments should remain stable.This motivates separating in-scope discrete action-to-visual correspondences from out-of-scope generation under continuous controls.
  • Method: SCOPE inserts per-pixel temporal conditioning into every transformer block, using local visual content to compute action responses independently at each position.Visually queried cross-attention confines discrete effects to in-scope regions, while temporal self-attention models smooth ego-motion.
  • Dataset: 69,000 clips across seven FPS titles with 10-dimensional controller telemetry comprise CrossFPS, the first multi-game FPS dataset with frame-aligned action annotation.The dataset is curated to remove gameplay bias and enables learning shared action-visual dynamics across titles.
  • Contributions: SCOPE enables robust controllability on unseen scenes and effective zero-shot generalization without segmentation supervision, while benefiting from data scaling.The architecture decomposes discrete in-scope responses from continuous out-of-scope generation through end-to-end training.

2 Related Work

Related work spans world models for prediction, planning, and control, video diffusion models for temporally coherent generation, and game-specific interactive world models. Existing game systems have advanced interactive video generation but remain constrained by simplified actuation.

  • World Models: World models learn environment dynamics for prediction, planning, and control, including transition simulation in reinforcement learning and temporally coherent video continuation in computer vision.Recent work also pursues long-horizon consistency, long-horizon memory, physical plausibility, and real-time inference.
  • Video Diffusion Models: Diffusion-based generative models have driven rapid progress in visual synthesis, while video frameworks generate temporally coherent content across diverse domains.Latent diffusion and Transformer-based architectures further advanced image and video generation.
  • Game World Models: Games serve as natural test beds for interactive world models because they combine visual dynamics with rule-based logic.Early GAN-based methods showed limited generative capabilities, whereas later diffusion systems enabled interactive video generation for Atari, DOOM, and Minecraft, despite simplified actuation constraints.

3 Method

SCOPE conditions a pretrained video diffusion transformer on FPS controls by reshaping features into per-pixel temporal sequences and processing discrete and continuous actions through dedicated pathways. This design localizes discrete effects to visually relevant regions while modeling continuous out-of-scope ego-motion, without explicit region annotations.

  • SCOPE Architecture: SCOPE inserts a module into each of 30 transformer blocks, rerouting action conditioning through per-pixel temporal sequences based on local visual content.The module is placed between text cross-attention and the FFN.
  • Action Representation: The model represents FPS controls as 4-dimensional continuous analog signals and 6-dimensional discrete button-event signals.Continuous controls cover movement and camera axes; discrete events cover fire, ADS, reload, jump, melee, and weapon switch.
  • Dual-Pathway Processing: Visually queried cross-attention processes discrete events so in-scope pixels attend to action signals while out-of-scope pixels produce near-zero responses without region annotations.The discrete pathway models localized effects such as muzzle flashes, zoom, and interaction reactions.
  • Dual-Pathway Processing: Continuous controls are fused with per-pixel features and processed by temporal self-attention to model smooth ego-motion and stable out-of-scope generation.The method uses raw-frame action windows with temporal compression ratio r = 4 for each latent frame.
  • Dual-Pathway Processing: The discrete and continuous residuals are summed with the reshaped features, then restored to the standard token layout before the FFN.The combined update is ˆx + ∆xc + ∆xd.

4 Experiments

Experiments show that SCOPE provides strong action responsiveness and spatial stability, outperforming global-conditioning baselines on most metrics. Ablations and unseen-scene evaluations further support the importance of spatially selective conditioning and cross-game generalization.

  • Quantitative comparison: SCOPE achieves the best performance on 7 of 8 metrics, while Matrix-Game 3.0 leads Motion Smoothness through action suppression rather than faithful rendering.Under identical high-frequency camera rotations, SCOPE maintains smooth viewpoint changes while baselines suppress motion or produce artifacts.
  • Quantitative comparison: Dynamic Degree 0.910 and Flow Score 18.24 substantially outperform baselines, while Photometric Smoothness 0.198 confirms strong spatial stability.LingBot-World reaches Dynamic Degree 0.868, Matrix-Game 3.0 reaches 0.661, and HY-World 1.5 reaches 0.225.
  • Ablation studies: Removing spatial selectivity worsens Photometric Smoothness 3.8× from 0.198 to 0.745 and reduces Dynamic Degree to 0.521, reproducing global-conditioning failures.Removing temporal self-attention reduces Flow Score from 18.24 to 11.60, while removing discrete cross-attention causes out-of-scope leakage.
  • Ablation studies: End-to-end training improves FVD monotonically from 775.4 for Frozen through 732.1 for Two-stage to 690.3 for End-to-end.Performance also scales with data volume and diversity without saturation.
  • Zero-shot generalization: On synthesized unseen scenes, JEPA declines modestly to 0.777 from 0.806 and Photometric Smoothness to 0.231 from 0.198, while all categories remain at Photometric Smoothness ≤0.251.Sci-fi corridors achieve near-parity with in-distribution performance, supporting transfer of scope separation to novel visual domains.
  • Zero-shot generalization: SCOPE reaches 71.5% controllability on unseen scenes versus 38.3% for LingBot-World, with performance declining from 92% for single actions to 54% for environment interactions.Compositions score 75%, while Matrix-Game 3.0 and HY-World 1.5 achieve 0.5% and 8.0%, respectively.

5 Limitations and Future Work

SCOPE generalizes across scenes and basic actions, but struggles with complex interactions, geometric transformations, and severely blurred inputs. Future work will use richer data and extend the model to long-horizon, multi-stage game control.

  • Current limitations: SCOPE’s current generalization covers cross-scene visual transfer and basic action interactions, while multi-step weapon mechanics, item usage, and fine-grained manipulation remain challenging.The limitation is attributed to limited interaction diversity in the training data.
  • Current limitations: Appearance-level responses such as fire, smoke, and lighting are handled better than geometric transformations because the diffusion backbone exhibits texture bias.Structural deformation and physics-driven reactions are particularly difficult.
  • Current limitations: Extreme blur in initial frames degrades performance, causing regression toward the average training appearance.This failure mode reflects sensitivity to severely degraded visual inputs.
  • Future work: Performance scales monotonically with data volume and diversity without saturation, motivating richer interaction data to expand learnable behaviors.Future work targets long-horizon, multi-stage task execution with consistent state across extended gameplay and full game-level control.

6 Conclusion

SCOPE is an interactive FPS world model that separates in-scope and out-of-scope regions through per-pixel action conditioning without segmentation labels. End-to-end CrossFPS training enables backbone–module co-adaptation, with performance scaling monotonically with data volume and diversity.

  • 6 Conclusion: SCOPE separates in-scope and out-of-scope regions through per-pixel action conditioning.Each pixel is conditioned on local visual content rather than broadcast global embeddings.
  • 6 Conclusion: End-to-end training on CrossFPS enables co-adaptation between the pretrained backbone and SCOPE modules.
  • 6 Conclusion: Performance scales monotonically with CrossFPS data volume and diversity.

A CrossFPS Dataset Details

The appendix provides complete CrossFPS dataset details, covering dataset statistics, action telemetry, data processing, and text caption generation.

  • Dataset overview: Section A.1 presents the CrossFPS dataset overview and per-game statistics.
  • Action telemetry: Section A.2 specifies the action telemetry format.
  • Data processing: Section A.3 describes the data processing pipeline.
  • Text captions: Section A.4 details the text caption generation procedure.

A.1 Overview and Statistics · A.2 Action Telemetry Format

CrossFPS contains 69,000 frame-aligned five-second clips from seven FPS titles, with high-frequency action diversity and de-biased telemetry designed to support cross-game learning. Each clip provides 10-dimensional per-frame controller signals spanning movement, camera, combat, and utility.

  • A.1 Overview and Statistics: 69,000 five-second clips cover seven FPS titles at 20 fps and 480×832 resolution, split 95:3:2 into train, validation, and test sets.The clips come from NitroGen and WorldCam, with frame-aligned controller telemetry.
  • A.1 Overview and Statistics: 65,557 training clips provide the basis for kinematic and diversity statistics, with continuous signals normalized to [−1, 1].These statistics characterize the processed training subset and its signal distributions.
  • A.1 Overview and Statistics: 0.48 mean linear velocity reflects the ≥70% activity filter, while angular velocity of 0.26 ± 0.18 spans precision aiming and rapid flicks.The angular range corresponds to approximately 30◦–60◦/s at 20 fps.
  • A.1 Overview and Statistics: 0.78±0.14 peak angular acceleration indicates abundant high-frequency events, including flick shots and 180-degree snap turns.These events stress-test scene stability.
  • A.1 Overview and Statistics: 2.94 ± 0.31 bits action entropy approaches the theoretical maximum for the discretized 10-dimensional action space, rising from 1.85 to 2.94 bits after de-biasing.The resulting diversity prevents reliance on simple temporal priors and requires learning physical action-visual mappings.
  • A.1 Overview and Statistics: 0.38 : 1.0 strafe-to-forward ratio exceeds the typically < 0.1 ratio of navigation datasets, indicating greater lateral-movement diversity.This statistic is reported as part of the post-processing analysis of CrossFPS.
  • A.1 Overview and Statistics: 0.034 post-normalization gain variance and r = 0.91 ± 0.03 input-to-optical-flow correlation validate a unified action space across titles.Before calibration, cross-engine variance exceeds 0.8 because identical stick displacement produces 10◦ rotation in Halo but 30◦ in Call of Duty.
  • A.2 Action Telemetry Format: 10-dimensional per-frame telemetry is organized into Movement, Camera, Combat, and Utility groups containing 4, 2, 3, and 3 signals, respectively.Movement and camera use continuous analog axes, while combat and utility controls are binary indicators sampled each frame.

A.3 Data Processing Pipeline · A.4 Text Caption Generation

CrossFPS uses a multi-stage processing pipeline to remove strategic bias, balance action diversity, normalize cross-game kinetics, and verify training-data integrity. Text captions provide standardized two-sentence conditioning that preserves both environment and player-state details across games.

  • A.3 Data Processing Pipeline: The pipeline targets human-action bias so SCOPE learns physical action-visual mappings instead of memorizing game strategies.It enforces diversity, balance, and cross-game consistency through four processing phases.
  • A.3 Data Processing Pipeline: Videos are cropped to the active game area, split at scene transitions, segmented into non-overlapping 5-second windows, and normalized to 20 fps.Frame-rate normalization uses subsampling for 60 fps sources and interpolation for 30 fps sources.
  • A.3 Data Processing Pipeline: High-intensity clips in the top 15% are oversampled 3× after filtering idle clips with left-stick activity below the 70% threshold.The balancing step targets long-tail gameplay distributions dominated by low-intensity states and avoids smooth, low-motion generation.
  • A.3 Data Processing Pipeline: The de-biasing stage retains inefficient actions and forcefully includes clips in the bottom 20% of visual-action mutual information.These samples are selected to reduce strategic priors and expose raw physical mappings.
  • A.3 Data Processing Pipeline: Optical-flow gain calibration normalizes game-specific camera responses by fitting a linear relation between mean pixel displacement and camera signals.The method addresses gradient conflicts caused by different engine mappings from identical analog-stick displacements to camera rotation speeds.
  • A.3 Data Processing Pipeline: 100% of the 65,557 training clips passed integrity checks covering video readability, first-frame decodability, frame count, resolution, and action-file completeness.The check was completed before training.
  • A.4 Text Caption Generation: Gemini generates a standardized two-sentence caption for the first frame of every clip, describing the environment first and the player state and salient visual elements second.The format covers setting, lighting, architecture, atmosphere, weapon type, HUD indicators, nearby objects, and game-specific UI, preserving consistent conditioning across games.

B Implementation Details · B.1 Training and Inference Pseudocode

SCOPE uses a Wan2.2-TI2V-5B backbone with per-block conditioning modules that separately process continuous controls and discrete events. Training and inference use action dropout, noisy latent interpolation, patchification, per-pixel temporal reshaping, and distinct continuous and discrete action pathways.

  • B Implementation Details: The backbone is Wan2.2-TI2V-5B with 30 transformer layers, hidden dimension 3072, 24 attention heads, patch size, and FFN dimension 14336.UMT5-XXL produces 4096-dimensional text embeddings, while the VAE applies 8× spatial and 4× temporal compression.
  • B Implementation Details: Each SCOPE module combines a fusion MLP for continuous controls, cross-attention for discrete events, and temporal RoPE embeddings.All output projections are zero-initialized.
  • B.1 Training and Inference Pseudocode: The continuous pathway applies self-attention to fused features and controls, while the discrete pathway uses cross-attention with embedded event actions.The updates are Δxc ← SelfAttn(MLPfuse([x̂; ac])) and Δxd ← CrossAttn(Q=x̂, K=V=MLPembed(ad)).
  • B.1 Training and Inference Pseudocode: The complete training and inference procedures are specified in Algorithm 1 and Algorithm 2.These algorithms define the end-to-end procedures for SCOPE conditioning.
  • B.1 Training and Inference Pseudocode: Training requires an action-dropout probability pdrop and a learnable null embedding anull for classifier-free guidance.The pseudocode replaces action conditions with anull according to pdrop.
  • B.1 Training and Inference Pseudocode: Each training step samples t ∼ U(0, 1) and ϵ ∼ N(0, I), interpolates zt ← (1 − t)z0 + tϵ, and patchifies the resulting latent.The sequence follows action dropout before latent noising and patchification.
  • B.1 Training and Inference Pseudocode: Within every transformer layer, standard DiT processing precedes reshaping features into per-pixel temporal sequences for spatially selective conditioning.The standard block uses self-attention and text cross-attention.
  • B.1 Training and Inference Pseudocode: The layer recombines reshaped features with both action updates, applies the layer FFN, and repeats this process across transformer layers.The pseudocode performs x ← Reshape(x̂ + Δxc + Δxd); x ← FFNl(x) inside the loop l = 1, …, L.

C Evaluation Metrics … C.3 Visual Quality

The evaluation uses eight metrics on 1,378 CrossFPS test clips at 480×832 resolution, covering action responsiveness, spatial stability, and visual quality. These metrics assess motion response, geometric consistency, semantic fidelity, realism, perceptual distortion, and temporal coherence.

  • C Evaluation Metrics: Eight metrics evaluate generated videos on 1,378 CrossFPS test clips at 480×832 resolution across three primary axes.The evaluation is summarized in Table 8.
  • C.1 Action Responsiveness: Action responsiveness measures genuine dynamic changes under sequential actions, penalizing static or unresponsive outputs.Dynamic Degree captures overall spatiotemporal activity, while Flow Score measures average inter-frame optical-flow magnitude; near-zero scores typically indicate frozen frames.
  • C.2 Spatial Stability: Spatial stability measures whether videos preserve consistent 3D geometry over time without collapse, warping, or unauthorized deformation.Photometric Smoothness evaluates adjacent-frame color consistency, and Depth Accuracy evaluates reprojection consistency of predicted depth maps.
  • C.2 Spatial Stability: Lower photometric errors indicate stable appearance without flickering or texture artifacts, while higher depth accuracy confirms rigidly maintained scene geometry.Both metrics use temporal reprojection: photometric error compares backward-warped pixels, whereas depth accuracy compares reprojected and directly estimated depths.
  • C.3 Visual Quality: Visual quality evaluates image clarity, perceptual realism, semantic fidelity, and motion coherence.Its metrics compare generated content with reference or real videos and assess perceptual and temporal properties.
  • C.3 Visual Quality: Higher JEPA Similarity indicates that generated videos preserve the reference sequence’s semantic and physical structure.The metric computes cosine similarity between V-JEPA feature vectors extracted from generated and ground-truth videos.
  • C.3 Visual Quality: Lower FVD indicates that generated spatiotemporal feature distributions more closely match real video distributions.FVD models generated and real I3D feature collections as multivariate Gaussians.
  • C.3 Visual Quality: LPIPS measures perceptual frame distortion, while Motion Smoothness penalizes abrupt optical-flow acceleration associated with jitter or stuttering.LPIPS uses VGG feature distances; high motion smoothness indicates physically plausible, continuous inertial motion.

D Scalability Details

Scalability analysis compares SCOPE training regimes and data configurations. End-to-end training performs best, while progressive training improves stability at intermediate scale and broader multi-game data supports scaling.

  • Training strategy: JEPA 0.806 is strongest with full end-to-end training, which enables deep co-adaptation between SCOPE modules and the pretrained backbone.The compared regimes are Frozen, Two-stage, and End-to-end; Frozen trains only the 30 SCOPE modules, while End-to-end trains all parameters from the start.
  • Training strategy: Flow Score rises 15.57→17.13→18.24 across the compared training regimes, favoring full end-to-end optimization.The sequence reports the training-strategy comparison described in the passage, with end-to-end training producing the strongest overall results.
  • Training strategy: Photometric Smoothness 0.264 for Frozen remains far better than 0.745 without Spatial Selectivity, showing per-pixel conditioning preserves out-of-scope stability.This result indicates that the SCOPE module’s spatial design contributes independently of backbone adaptation.
  • Data scale and diversity: FVD 1033 vs. 845 for progressive training at 10K clips shows that moderate cross-domain diversity can destabilize single-stage training.At 1K–5K clips within a single series, single-stage training works well; at 10K clips spanning 2 series, progressive training is more stable.
Loading 2605.23345v2…