PaperlayerView exact revision on arXiv

Source-linked AI summary

Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

Wenxuan Shen, Dongna Jin, Dongping Chen

arXiv:2608.24680v1cs.CV

TL;DR

In-the-wild gameplay is abundant and diverse but entangles world content with screen-space interfaces that can impair training signals. Game2World and GameCleaner address this through taxonomy-guided data processing and mask-free temporal UI removal, with reported gains in both training utility and removal quality.

  • Problem

    Gameplay videos provide scalable world-model data, but raw footage entangles game-world observations with screen-space interfaces and irrelevant dynamics.

  • Method

    The paper introduces GameUI-Taxonomy, G2WEngine, Game2World, and GameCleaner for structured UI grounding, synthesis, and mask-free temporally coherent removal.

  • Results

    6.83% higher overall VideoReward is achieved with UI-free training data, while GameCleaner improves UI-removal performance across synthetic and in-the-wild evaluations.

  • Takeaways & Limitations

    Gameplay UI removal improves the utility of in-the-wild gameplay footage as world-model training data rather than serving only as visual editing.

  • Takeaways & Limitations

    Game2World does not cover the full diversity of games, interfaces, and recording conditions, and the downstream study uses text-conditioned video generation as a proxy.

Abstract

from arXiv · show

Video games provide a scalable source of training data for video world models, offering diverse environments, complex interactions, and abundant in-the-wild gameplay videos. However, raw gameplay footage entangles the game world with screen-space interfaces, introducing game-specific biases and irrelevant dynamics that hinder world-model training. To address this problem, we introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal. G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage. Using this engine, we construct Game2World, comprising 96K synthetic paired videos with precise reconstruction targets and 1,079 in-the-wild clips from 303 games for realistic evaluation. Its asset library contains 5,132 verified UI elements across 21 taxonomy categories, collected from 1,010 representative gameplay frames. Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities. Unlike mask-based methods, GameCleaner directly identifies and removes diverse HUD elements while preserving the underlying scene content and temporal dynamics. In a controlled pilot, world models trained on UI-free gameplay improve overall VideoReward by 6.83% over those trained on UI-overlaid data. On UI-removal evaluation, GameCleaner achieves an average AAR of 95.36 on synthetic videos, outperforming the strongest temporal mask baseline by 57.3%, and obtains the best in-the-wild AAR of 80.05 with 99.8 background preservation. These results demonstrate the scalable potential of transforming Internet gameplay videos into high-quality world-model training data. Code, dataset, and model will be available at https://github.com/Dongping-Chen/Game2World.

1. Introduction

Game2World addresses the challenge of using abundant gameplay videos for world-model training when screen-space interfaces introduce irrelevant, game-specific dynamics. It combines a UI taxonomy, scalable data engine, dataset, and mask-free removal model, with evaluations showing improved training utility and UI removal.

  • In-the-wild gameplay videos offer diverse environments, viewpoints, tasks, dynamics, and player behaviors for scalable world-model training.
  • Raw gameplay mixes game-world content with HUDs, menus, notifications, watermarks, and streaming overlays that introduce unrelated supervision.
  • 6.83% higher overall VideoReward is achieved when world models train on clean rather than UI-overlaid gameplay clips.
  • GameUI-Taxonomy and G2WEngine formalize interface–world disentanglement by grounding, removing, and temporally reconstructing gameplay overlays.
  • Game2World combines 96K synthetic paired videos with 1,079 in-the-wild clips, while GameCleaner removes diverse HUD elements without input masks.
  • GameCleaner reaches an average score of 0.5697 on synthetic videos and 0.4886 on in-the-wild videos, outperforming relevant baselines across the two subsets.

2. Related Work

Prior work develops gameplay world models and general video-inpainting methods, but these directions generally do not separate game content from structured gameplay overlays. The paper positions UI removal as distinct from ordinary physical-object removal.

  • Gameplay world-model work learns simulators from interactions or recorded videos, while related systems often treat rendered frames as direct world observations.
  • Video-inpainting methods target spatially plausible and temporally consistent removal using convolutions, flow completion, attention, feature matching, and propagation.

3. Pilot Study: UI-Removed Gameplay Videos Are Great Training Data for World Model

The pilot study tests whether removing gameplay interfaces improves video-based world-model training signals while controlling for source content. Clean-video training improves most reported quality measures despite a small clarity decrease.

  • Experiment Setups: The study compares clean and synthesized UI-overlaid versions of identical 5,442 gameplay clips using a text-to-video model proxy.
  • Empirical Results: 6.83% higher overall VideoReward is obtained from models trained on clean gameplay videos than from models trained on UI-overlaid videos.
  • Empirical Results: +18.8% motion quality and +2.7% video quality are achieved with clean-video training, while temporal alignment remains comparable.
  • Empirical Results: 8.59% higher aesthetic quality and 4.36% higher motion consistency accompany a small -1.43% change in low-level clarity.

4. G2WEngine: Constructing World Model Data from In-the-Wild Gameplay Videos

G2WEngine converts heterogeneous gameplay footage into structured supervision by modeling UI semantics, extracting reusable assets, curating clean clips, and synthesizing temporally coherent overlays. Its outputs support controlled reconstruction and in-the-wild evaluation.

  • G2WEngine performs interface–world disentanglement through taxonomy definition, UI-asset extraction, clean-video curation, and temporally coherent overlay synthesis.
  • GameUI-Taxonomy organizes HUD elements by semantic role and rendering behavior, enabling consistent annotation, category-aware sampling, and controllable synthesis.
  • Representative gameplay frames receive model-proposed UI boxes and taxonomy labels followed by human verification before conversion into transparent assets with metadata.
  • Game2World-W provides temporally annotated in-the-wild videos and evaluates UI removal with an MLLM judge using element-level bounding-box checklists.
  • Clean gameplay clips are filtered across genres and visual styles, segmented into 5-second clips, and deduplicated using CLIP embeddings and temporal filtering.
  • Synthetic samples include aligned clean and overlaid videos, frame-level masks, bounding boxes, taxonomy labels, and rendering metadata.

5. Game2World: The Dataset

Game2World combines synthetic paired videos with authentic in-the-wild gameplay clips to evaluate gameplay UI grounding and removal under controlled and realistic conditions. Its taxonomy and annotations support analysis of diverse, dynamic interfaces.

  • Game2World-S: Synthetic UI Gameplay Dataset: 5,132 verified UI assets across 21 taxonomy categories were collected from 1,010 representative keyframes spanning 303 games.Categories include player status, maps and radars, action information, communication panels, and external overlays.
  • Game2World-S: Synthetic UI Gameplay Dataset: 96k paired clean and UI-corrupted videos provide exact supervision for UI grounding, segmentation, temporal localization, and removal.All clips are rendered at 720p and 30fps with frame-aligned masks, instance boxes, category labels, and rendering metadata.
  • Game2World-W: In-the-Wild Gameplay Dataset: 1,079 authentic five-second clips spanning 303 games provide realistic evaluation despite lacking clean reference targets.The in-the-wild subset captures unknown layouts, transparency, dynamic transitions, and recording artifacts.
  • Analysis of UI on Gameplay Videos: UI-D averages 69.3, UI-R@16 has a median of 20 with a range of 2 to 40, and Context-UI-Lift is positive for 90.8% of games.These metrics characterize variation, functional UI-slot diversity, and context dependence in gameplay interfaces.

6. GameCleaner: UI Removal Model

GameCleaner performs mask-free gameplay UI removal by combining multimodal semantic understanding with video editing while preserving game content and temporal structure. Its architecture uses an MLLM-conditioned video diffusion transformer and paired clean targets for training.

  • Model Goal: GameCleaner targets diverse HUDs, menus, notifications, watermarks, and streaming overlays while preserving underlying game content and temporal dynamics.The model performs removal directly from source videos and is designed to generalize across games with different interface styles and temporal behaviors.
  • Model Architecture: The architecture couples an MLLM encoder with a video diffusion transformer to recognize interface elements from their appearance and context.Learnable latent queries extract task-relevant semantic information and project it into the DiT conditioning space.
  • Model Architecture: Source-video VAE latents are injected into the noisy target to preserve spatial and temporal structure during reconstruction.This design complements semantic UI recognition with structural information from the source video.
  • Training: The model is post-trained on paired UI-overlaid and clean gameplay videos at 720p using LoRA on the DiT and a standard flow-matching objective.The objective uses clean-target latents, Gaussian-noise latents, and MLLM-derived removal conditions.

7. Experiments

Experiments evaluate GameCleaner across synthetic and in-the-wild UI-removal benchmarks, compare reference conditions and baselines, and examine scaling, reliability, and judge sensitivity. GameCleaner generally balances UI removal with preservation, while larger and more diverse training data improves real-world generalization.

  • Overall performance: GameCleaner achieves the strongest overall removal–preservation trade-off, with reference-conditioned Overall 93.34 exceeding mask-assisted VACE-1.3B by 23.97 points and mask-free Kiwi-Edit by 32.76 points.
  • Synthetic benchmark: 95.36 AAR is achieved on the synthetic benchmark with BG 99.00, while EffectErase reaches AAR 56.17 and BG 84.45 after artifact adjustment.
  • Fine-grained removal quality: 68.12 Track-Clean on the wild benchmark indicates cleaner complete-track removal for GameCleaner than EffectErase’s 16.12 despite EffectErase’s higher raw removal scores.
  • Ablation study: 20% reference dropping balances reference-conditioned performance and reference-free robustness, raising wild AAR to 80.05 while more aggressive 50% and 80% dropping reduces it to 56.23 and 54.64.
  • Evaluation reliability: Evaluation scores remain sensitive to video resolution, motivating standardized reporting alongside uncertainty and multi-rating reliability measures.

8. Discussion and Limitations

The framework supports scalable gameplay-video processing, but the current study remains bounded by coverage, reconstruction difficulty, downstream modeling scope, and computational scale. Future work targets broader data, action-conditioned world modeling, and efficient inference.

  • Scaling behavior and task scalability: The current training study covers approximately 100K paired videos and shows favorable scaling without clear saturation, but does not establish a formal power-law relationship.
  • Scalable data collection and processing: G2WEngine converts heterogeneous gameplay footage into structured supervision and cleaned videos through automated, parallelizable UI asset extraction, curation, synthesis, and removal.
  • Limitations and future directions: Game2World does not yet cover the full diversity of games, interfaces, and recording conditions, and GameCleaner may reconstruct imperfectly for opaque, rapidly changing, or scene-entangled UI.
  • Limitations and future directions: The downstream study uses text-conditioned video generation as a proxy rather than a fully action-conditioned gameplay world model, and reliable inverse-dynamics models remain unavailable across heterogeneous games.
  • Future directions: Future work will expand benchmarks, recover actions for observation–action trajectories, and distill GameCleaner into a few-step model with additional compute and ablations.

9. Conclusion

The paper frames gameplay UI removal as interface–world disentanglement and presents a unified taxonomy, scalable data engine, benchmark, and mask-free removal model. Experiments indicate that removing UI preserves scene content and temporal dynamics while improving gameplay data for world-model training.

  • Game2World formulates gameplay UI removal as interface–world disentanglement.
  • The framework combines a unified taxonomy, scalable data engine, large-scale benchmark, and mask-free removal model.
  • Removing gameplay UI can preserve underlying scene content and temporal dynamics while improving gameplay videos for downstream world-model training.

B. Game2World Dataset Construction

Game2World construction standardizes diverse clean gameplay clips, organizes verified UI assets, and synthesizes persistent and transient overlays. The pipeline uses diversity-aware sampling, temporal deduplication, spatial priors, style variation, collision handling, and event modeling to produce temporally consistent paired data.

  • Dataset construction: The dataset pipeline curates clean clips, selects and spatially organizes verified UI assets, and synthesizes persistent and transient elements.Together, these stages produce diverse and temporally consistent paired data for UI removal and world-model training.
  • Clean gameplay corpus: Selected videos are quality-checked and standardized to five seconds, 720p, and 30 FPS while preserving source aspect ratio with padding when necessary.
  • Persistent UI synthesis: Persistent widgets use human-verified, taxonomy-filtered assets with category-specific spatial priors, seeded position variation, motion envelopes, and collision resolution.Recorded positions provide initial placement priors when available; otherwise, category-specific anchors support controlled variation.
  • Synthetic UI composition: Synthetic composition varies global styles, persistent-widget sampling, popup counts, event categories, and animation behavior while retaining synthesis records.The five-second profile expects 4.00 persistent widgets and exactly 1.5 popup events per video.

C. Additional Dataset Statistics and Details

Additional dataset details define benchmark cases and optional appearance matching, while the renderer selects diverse keyframes and synthesizes temporally varying interface assets. These procedures support controlled evaluation, reproducible asset selection, and realistic UI dynamics.

  • Benchmark cases: Synthetic benchmark cases pair UI-overlaid sources with clean references, whereas in-the-wild cases lack clean references and rely on timestamp-aligned annotations.
  • Appearance matching: Optional aesthetic matching weights taxonomy-compatible assets toward the clean video's visual appearance while retaining sampling diversity.
  • Appearance matching: Appearance descriptors combine normalized color, intensity, gradient, and statistical features, optionally complemented by normalized EfficientNet-B0 representations.
  • Appearance matching: The renderer records selection scores and alternatives, supporting reproducibility and inspection of appearance-consistent asset choices.
  • Dynamic UI assets: Status assets animate internal filled regions when available, while separated inventory assets support deterministic icon resampling and time-varying slot states.Procedural status panels are fallback options when suitable assets are unavailable.

F. Synthetic Supervision and Reproducibility Records

The paper builds reproducible synthetic supervision and evaluation records by pairing clean and UI-overlaid videos with structured metadata, deterministic annotations, and timestamp-aligned judging. Its GameCleaner implementation uses multimodal conditioning and flow matching to reconstruct clean target videos.

  • Synthetic supervision: Each synthesized example pairs a clean target video with a UI-overlaid input, binary UI mask, and structured metadata.The record can also retain optional visual summaries and appearance comparisons for quality control.
  • Reproducibility records: The synthesis record stores layout, style, realized elements, positions, opacity, provenance, timing, and rendering parameters for reproducible generation.These fields support deterministic reconstruction of frame-level annotations.
  • Annotation records: Frame-level annotations use clipped half-open coordinates and preserve instance identities, semantic categories, visibility, parent-child relations, and temporal trajectories.Persistent and transient elements receive boxes derived from metadata, while inventory icons can be represented as child instances.
  • GameCleaner implementation: The GameCleaner architecture combines MLLM-derived removal conditioning, latent queries, connector projections, source-latent injection, and standard flow matching.The clean target latent is paired with Gaussian noise, and the model learns temporally coherent reconstruction of occluded game content.
  • Evaluation protocol: Evaluation uses frozen prompts and explicit evidence ordering, compares timestamp-matched frames across frame rates, and records exact protocol fields.The evaluator receives overall frame pairs before element crops and uses canonical scaling without additional crop resizing.
  • Evaluation criteria: The judge independently evaluates each listed element and rates removal, fill artifacts, and preservation outside annotated UI regions using concrete visual evidence.The protocol distinguishes surviving original graphics from unrelated content and treats localized seams or inconsistent blur as artifacts.

J.6. Judge execution and reproducibility

Judge execution uses a frozen, schema-validated, one-call-per-video protocol with timestamp-aligned evidence. The reported Halo Infinite evaluation records complete HUD removal and preservation, while other in-the-wild examples expose residual UI elements.

  • Judge execution: Each video is evaluated in one read-only judge call using the complete evidence set, frozen model, frozen reasoning setting, and exact JSON schema.Parallel processing changes concurrency but not the one-call-per-video rule.
  • Reproducibility: The protocol uses one frozen stochastic judgment per video, and repeated-call audits must predeclare repetition or consensus rules.Selecting the best result from repeated calls is disallowed.
  • In-the-wild result: Halo Infinite records 100.0% HUD removal success across 82 element observations, with 100.0% clean, adjusted, and preservation scores.The reported removal/preservation ratio is 82/0.
  • In-the-wild result: The Halo Infinite visual judgments mark diverse elements absent, including top bars, kill-feed text, enemy markers, radar labels, reticles, and weapon/ammo glyphs.The listed observations cover both textual and graphical HUD components.
  • Additional visual judgments: Other visual judgments report absent HUD components such as cyan labels, radar circles, reticles, status bars, and small numeric text.The records span multiple gameplay examples and element categories.
  • Additional visual judgments: Some judged examples retain interface remnants, including inventory text, JAMMED indicators, colored HUD curves, dialogue text, and weapon icons.These observations indicate that the evaluation records both successful removals and surviving UI content.
Loading 2608.24680v1…