PaperlayerView exact revision on arXiv

Source-linked AI summary

GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

Kang He, Wenshuo Peng, Chuanhao Li, Zihui Gao, Xiaojie Xu, Zhengyuan Lin, Yuzhe Ding, Donghong Ji, Kaipeng Zhang, Yongtao Ge

arXiv:2610.37257v1cs.CV

TL;DR

Interactive world models struggle to maintain coherence beyond roughly one minute, while existing gameplay datasets typically offer short clips and coarse captions. GameVerse combines minute-scale gameplay footage with stable camera trajectories and event-aligned, multi-granular captions. Training on GameVerse improves generation quality and consistency, though experiments cover generation only up to 60 seconds.

  • Problem

    Existing gameplay datasets typically contain clips under ten seconds with whole-clip captions, while current world models struggle to maintain coherence beyond roughly one minute.

    Source: §1 · p. 2

  • Method

    GameVerse constructs a minute-scale gameplay dataset pairing scene-consistent clips with validated frame-level camera trajectories and four-level captions separating static context, dynamic events, and agent actions.

    Source: §1 · pp. 2–3

  • Results

    Training on GameVerse improves temporal coherence, geometric consistency, and visual quality in 60-second generation, with same-backbone comparisons validating benefits of fine-tuning.

    Source: §6 · p. 10

  • Takeaways & Limitations

    GameVerse provides 69.7K clips across 1,655 hours, including 1,358 hours of continuous two-minute gameplay, with jointly aligned trajectory and semantic supervision.

    Source: §1 · p. 2, §6 · p. 10

    GameVerse focuses on gameplay environments; cross-domain generalization remains unexplored, continuous clips stop at two minutes, and generation experiments cover up to 60 seconds.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent advances in interactive world models have made it possible to generate controllable, dynamic environments from action inputs. Yet current models struggle to maintain coherence beyond roughly one minute of rollout, a limitation rooted not in model capacity but in the scarcity of long-horizon training data. Existing gameplay video datasets typically consist of clips under ten seconds, with captions that describe whole clips in coarse and undifferentiated text, which is insufficient for controllable long-horizon generation. We address this with GameVerse, a minute-scale gameplay video dataset purpose-built for long-horizon interactive world modeling. GameVerse contains 69.7K clips spanning 1,655 hours and 349.6 million frames across diverse first- and third-person gameplay, including 1,358 hours of continuous two-minute sequences, an order of magnitude longer than prior datasets. Clips pass through a unified pipeline of scene-consistent segmentation, quality filtering, camera pose estimation, and event-based caption annotation. Each clip is paired with a temporally stable camera trajectory that remains coherent across minutes, together with captions that separate static scene context from dynamic foreground events and align to event boundaries rather than fixed time windows. Comprehensive analysis shows that GameVerse surpasses existing public datasets in scale, diversity, and annotation quality. Experiments further demonstrate that training on GameVerse substantially extends the temporal horizon of interactive world models.

1 Introduction

The introduction argues that minute-scale world-model coherence is constrained by training data: existing gameplay corpora are short and their captions conflate scene context with events. GameVerse combines long continuous footage, validated camera trajectories, and event-aligned structured captions, and training on it improves generation metrics over gameplay-domain baselines.

Motivation

  • Existing gameplay systems lose scene coherence after roughly one minute, while their training clips are mostly under ten seconds and use captions that conflate static context with dynamic events.The introduction attributes this ceiling primarily to data scarcity rather than model capacity.Source: §1 · p. 2

Gap in existing datasets

  • Prior datasets distribute minute-scale footage, reliable camera trajectories, diverse interactions, and event-aligned semantics across separate sources rather than providing all four together.Table 1 defines comparison dimensions including long-duration footage, released reliable poses, and multi-granular semantics.Source: §1 · p. 2, Table 1 · p. 3

Contributions

  • GameVerse pairs 69.7K clips and 1,655 hours of gameplay with stable frame-level camera trajectories, four-level captions, and diverse first- and third-person interactions.The corpus includes 1,358 hours of continuous two-minute sequences, roughly an order of magnitude longer than prior gameplay or exploration datasets.Source: §1 · p. 2

Construction and annotation

  • GameVerse uses scene-consistent segmentation, multi-stage quality filtering, ViPE-based pose estimation with trajectory validation, and event-aligned multi-granular captioning.Its four caption levels separate global and local static context, dynamic events, and agent actions.Source: §1 · pp. 2–3

Empirical validation

  • Training on GameVerse improves VBench temporal consistency and visual quality and most WorldScore controllability metrics over gameplay-domain baselines on a held-out long-duration benchmark.The introduction also reports improved WorldScore content alignment and 3D consistency on a 30-second benchmark.Source: §1 · pp. 2–3

2 Related Work

Prior work advances interactive generation, camera control, and captioning, but open gameplay data remains fragmented across those needs. GameVerse addresses the data-side gap with a public minute-scale corpus, validated trajectories, and structured captions.

2.1 Interactive World Models and Gameplay Datasets

  • Earlier interactive video systems established generation methods, but narrow or closed corpora limited reuse and broad cross-game study.WorldMem explored long-term Minecraft simulation, but its evaluation remained narrow and lacked a transferable large-scale gameplay dataset.Source: §2.1 · p. 3
  • Recent systems improve controllability, yet several rely on proprietary data; GameVerse complements memory-based Matrix-Game 3.0 with an openly released, multi-game, minute-scale corpus.The passage identifies the lack of a public, diverse, multi-game, minute-scale dataset as a principal bottleneck.Source: §2.1 · p. 3

2.2 Camera-Aware Video Supervision

  • Camera-conditioned generation depends on reliable trajectory supervision, while existing released trajectories may be noisy or absent; GameVerse releases frame-level trajectories filtered for jumps, jitter, and drift.The passage notes that Sekai ships estimated trajectories but does not validate those three error modes.Source: §2.2 · p. 4

2.3 Structured Caption Annotations for Video

  • Existing caption datasets often flatten events or mix scene context with behavior, whereas GameVerse separates static context, dynamic events, and agent actions in an event-aligned four-level schema.The schema is generated by a vision-language model and designed as a hierarchical conditioning signal.Source: §2.3 · p. 4

3 Dataset Construction Pipeline

The construction pipeline turns long-form gameplay recordings into aligned minute-scale clips with validated camera trajectories and multi-granular captions. Its stages address fragmented temporal structure, video quality, pose reliability, and flat semantic annotation.

3 Dataset Construction Pipeline

  • The pipeline transforms a curated pool of long-form gameplay into aligned video segments, validated camera trajectories, and multi-granular captions through four processing stages.The stages are scene-consistent segmentation, multi-stage video quality evaluation, camera pose estimation and validation, and structured captioning.Source: §3 · p. 4

3.1 Data Collection

  • The source pool comprises 11,002 videos totaling 4,692 hours and approximately 986.6 million frames, selected from open-world AAA and indie games spanning perspectives and behaviors.Videos are retained at the highest available quality, up to 1080p and 30–60 FPS.Source: §3.1 · pp. 4–5

3.2 Scene-consistent Segmentation

  • Hard-cut detection and CLIP-based semantic distance identify abrupt and gradual transitions to create scene-consistent segments up to 120 seconds.After segmentation, the corpus contains 272,128 clips totaling 4,661 hours, including approximately 2,773 hours of 120-second clips.Source: §3.2 · p. 5

3.3 Multi-stage Video Quality Evaluation

  • Five quality-filtering stages screen visual defects, overlays, instability, redundancy, and temporal consistency, while stage-specific constraints prevent severe defects from being offset by other scores.The pipeline also restores some stationary-camera clips when scene content continues to evolve.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 Camera Pose Estimation and Validation

  • ViPE estimates frame-wise intrinsic and extrinsic poses, then trajectory scoring and threshold filtering target jumps, high-frequency jitter, and long-term drift.The unified quality function combines local relative-pose increments with a global trajectory smoothness term.Source: §3.4 · p. 6

3.5 Multi-granular Structured Captioning

  • The captioning framework separates global static context, local static scene configurations, dynamic events, and agent actions to support hierarchical conditional generation.It addresses prior annotations that mix static scene context with dynamic behavior.Source: §3.5 · p. 6

4 Dataset Statistics

GameVerse’s statistics characterize its temporal scale, gameplay and scene diversity, camera motion, and semantic annotations. The reported distributions include extensive two-minute footage, varied perspectives and behaviors, and distinct caption levels.

4.1 Scale and Long-Horizon Distribution

  • Over 40.6K clips reach 120 seconds and contribute approximately 1,358 hours of continuous gameplay, an order of magnitude more continuous footage than prior gameplay corpora.Clips of 60 seconds or longer provide additional minute-scale supervision.Source: §4.1 · p. 7

4.2 Interaction and Environment Diversity

  • GameVerse spans at least 110 game titles, with approximately 24% first-person, 73% third-person follow-camera, and 3% specialized-perspective clips.The corpus covers eight behavior categories and diverse environments, weather, times of day, seasons, and crowd densities.Source: §4.2 · p. 7

4.3 Camera Trajectory Diversity

  • Forward motion accounts for 34.7% of camera patterns, while complex translation-plus-rotation motion contributes 22.0%.The corpus also includes static, rotational, lateral, orbit, and mixed rotational patterns.Source: §4.3 · p. 7

4.4 Caption Annotation Statistics

  • Local static descriptions and dynamic-event captions have median lengths of 895 and 775 characters, respectively, while global context and agent-action captions are more concise.Agent actions use a constrained vocabulary, and the four-level schema provides more total semantic information per clip than single-line or clip-level captions.Source: §4.4 · p. 8
  • The four caption levels use largely disjoint vocabularies for scene attributes, spatial layouts, environment transitions, and explicit motion.This pattern is reported as evidence that annotations are semantically separated rather than overlapping rephrasings.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Experiments

Experiments evaluate 60-second generation and annotation quality using shared-input comparisons, benchmark metrics, trajectory audits, and human caption ratings. GameVerse fine-tuning improves same-backbone results, while cross-system strengths vary by metric.

5.1 Experimental Setup

  • The 60-second image-to-video comparison uses 103 shared inputs and evaluates six VBench dimensions alongside seven normalized WorldScore components and WS-Static.Source: §5.1 · p. 8, §5.2 · p. 8

5.3 Minute-Scale Generation Performance

5.4 Long-Horizon Consistency Analysis

  • HY-World-FT lifts 30-second similarity from 0.609 to 0.642 and delays the below-0.6 crossing from 30.4s to 37.9s; its activity gains are 1.15, 1.85, 2.49, and 1.58 points.Source: §5.4 · p. 9, Table 4 · p. 10

5.5 Annotation Quality Validation

  • Retained audited trajectories have complete valid poses; against DROID-SLAM, rotation RPE is 0.75–1.47 degrees and translation-direction error 4.84–11.03 degrees, not absolute-accuracy measures.Source: §5.5 · p. 9
  • Across 400 clips, structured captions score 4.82/5 for scene/event accuracy and 4.64/5 for overall alignment, with 5.5% hallucination and 4.0% temporal-mismatch rates.Source: Table 5 · p. 10, §5.5 · p. 10
  • Qualitative examples show coherent scene structures and camera-aware motion across outdoor exploration, urban navigation, horseback riding, and nighttime driving.Source: §5.5 · p. 10

6 Conclusion and Limitations

GameVerse combines minute-scale gameplay video with camera trajectories and structured captions, and experiments report improvements in long-horizon generation. The paper identifies domain, duration, and evaluation limits alongside remaining annotation errors.

6 Conclusion

  • GameVerse contains 69.7K clips totaling 1,655 hours, including 1,358 hours of continuous two-minute gameplay, with quality-filtered camera trajectories and multi-granular structured captions.Source: §6 · p. 10
  • Experiments on 60-second generation show improvements in temporal coherence, geometric consistency, and visual quality, supported by same-backbone comparisons and annotation-quality assessments.Source: §6 · p. 10

Limitations

  • Real-world generalization remains unexplored; clips stop at two minutes, experiments at 60 seconds, and estimated poses and captions may retain errors.Source: §6 · p. 10

A Implementation Details

Implementation combines large-scale video processing with fine-tuning of LTX-Video 2.3 on a curated GameVerse subset. Training removes audio and uses mixed video-generation objectives with camera and action conditioning.

Training Configuration

  • LTX-Video 2.3 is fine-tuned on 24,380 segments at 540p/24 fps for 3,700 steps across 64 GPUs, using AdamW at a learning rate of 1 × 10^-4.Source: A Implementation Details · p. 14
  • Training removes the audio branch and mixes image-to-video, video-to-video, and text-to-video objectives at 60%, 20%, and 20%, with Plücker camera injection and 81-class action embeddings.Source: A Implementation Details · p. 14

Data Processing Efficiency

  • The segmentation pipeline processes about 400 FPS per worker, while distributed ViPE pose estimation takes about six minutes per gameplay video.Source: A Implementation Details · p. 14

B Detailed Multi-stage Video Quality Assessment Pipeline

GameVerse uses a multi-stage quality pipeline to remove degraded, contaminated, or unstable gameplay clips. It combines explicit rejection rules with learned quality scores before final selection.

B Detailed Multi-stage Video Quality Assessment Pipeline

  • The pipeline assesses visual quality, content cleanliness, and temporal stability, then retains only clips that meet hard constraints and required quality thresholds.Hard constraints remove low-quality samples; learned scores provide finer-grained control.Source: B Detailed Multi-stage Video Quality… · pp. 14–15
  • Visual filters enforce resolution, frame-rate, encoding, exposure, and border-artifact requirements before clips proceed.For example, clips must be at least 720p, run at 24–65 fps, and avoid prolonged extreme luminance or borders covering over 10% of a frame.Source: B Detailed Multi-stage Video Quality… · p. 14
  • OCR and temporal-spatial UI detection remove subtitles, watermarks, and interface overlays, while motion and freeze checks reject static or unstable footage.UI coverage above 4% triggers rejection; clips are also filtered for persistent freezing and frequent camera-jitter spikes.Source: B Detailed Multi-stage Video Quality… · pp. 14–15
  • Learned quality scoring retains the top 70% by combined aesthetic and technical scores, alongside VBench thresholds for consistency and flickering.Final selection combines these scores with hard constraints such as OCR, UI, motion, and freeze rejection.Source: B Detailed Multi-stage Video Quality… · p. 15
  • The authors report that the pipeline produces clips with high visual clarity, clean semantic content, and stable temporal dynamics.They describe it as more robust and comprehensive than simple filtering strategies.Source: B Detailed Multi-stage Video Quality… · p. 15

C.1 Evaluation Metrics

The evaluation section describes generation-quality metrics and camera-pose supervision, then reports trajectory reliability and an original-backbone diagnostic. It notes that estimated-pose agreement is not a measure of absolute accuracy.

C.1 Evaluation Metrics

  • VBench measures temporal coherence, motion smoothness, visual fidelity, and text alignment through subject, background, motion, aesthetic, imaging, and overall-consistency dimensions.WorldScore covers camera and object control, content alignment, 3D and photometric consistency, style consistency, and subjective quality.Source: §C.1 · p. 15

C.2 Camera-Pose Supervision Sources

  • WorldScore comparison accounts for estimated versus engine-provided poses: GameVerse uses filtered ViPE trajectories, whose supervision lacks the precision guarantees of engine-rendered ground truth.The authors note that camera-control differences may reflect both pose-supervision sources and conditioning architectures.Source: §C.2 · p. 16

C.3 Camera-Trajectory Reliability by Motion Type

  • Across six motion categories, all audited retained trajectories contain valid poses throughout their duration; against DROID-SLAM after Sim(3) alignment, rotation RPE is 0.75–1.47 degrees and translation-direction error is 4.84–11.03 degrees.These measurements show cross-estimator agreement, not absolute pose accuracy against ground truth.Source: §C.3 · p. 16, Table 6 · p. 16

C.4 Diagnostic Comparison of Original Backbones

  • Wan2.2 averages 1.87 points higher in aesthetic quality and 11.64 points higher in imaging quality than LTX-Video 2.3 across three durations without fine-tuning or camera conditioning.The diagnostic does not isolate the causes of differences in 60-second generation; the passage identifies the controlled HY-World 1.5 comparison as more direct evidence of GameVerse benefits.Source: §C.4 · p. 17, Table 7 · p. 17

D Visualization

The visual examples present continuous gameplay sequences and a structured annotation format. They span varied environments and illustrate temporal continuity and camera-motion stability.

D Visualization

  • Figure 7 depicts videos represented by global metadata and temporally aligned segments covering scene states, events, agent behavior, environmental changes, and scene evolution.The format combines global metadata with segment-level annotations.Source: Figure 7 · p. 18
  • Tables 8 and 9 provide example annotations from historical urban and third-person cinematic exploration sequences..Source: Table 8 · p. 19, Table 9 · p. 19
  • Figure 6 samples temporally continuous frames from individual clips across indoor, natural, urban, sunset-city, and nighttime gameplay scenes.The examples show diverse environments, lighting, perspectives, and interaction dynamics, with stable camera motion over long horizons.Source: D Visualization · pp. 17, 19

E Four-level Structured Semantic Annotations

GameVerse separates gameplay annotations into four semantic levels rather than one whole-clip caption. Qualitative examples show long-duration generation conditioned on an initial frame, camera trajectories, and structured captions.

E Four-level Structured Semantic Annotations

  • The annotation framework separates global static context, local static structure, dynamic events, and agent actions instead of describing each clip with one sentence.These levels provide complementary supervision for long-horizon understanding and generation.Source: E Four-level Structured Semantic… · pp. 19–21
  • Global-static annotations capture persistent environmental properties, including environment type, architecture, weather, lighting, season, camera perspective, and scene composition.The examples cover settings such as historical cities, deserts, coastal ruins, caves, and natural environments.Source: E Four-level Structured Semantic… · pp. 19–20
  • Local-static annotations describe fine-grained layouts within aligned segments, including changing streets, plazas, tunnels, shorelines, and grottoes.They are intended to preserve spatial continuity and geometric organization across long sequences.Source: E Four-level Structured Semantic… · p. 20
  • Dynamic-event annotations track evolving scenes and environmental transitions, such as entering caves, crossing districts, approaching landmarks, and moving between indoor and outdoor spaces.They also capture events such as lightning, motion changes, and camera-view transitions.Source: E Four-level Structured Semantic… · p. 20
  • Agent-action annotations describe player and surrounding-agent behaviors together with their interaction context, including walking, climbing, driving, and NPC activities.This is intended to support controllable interaction modeling and embodied world simulation.Source: E Four-level Structured Semantic… · p. 20

F.1 Qualitative Long-horizon Generation.

  • With one initial frame, camera trajectories, and structured captions, the fine-tuned model generates coherent long-duration gameplay videos across diverse environments and interactions.Examples show scene consistency and stable camera-aware motion over long horizons.Source: §F.1 · p. 21, Figure 8 · p. 21

F.2 Qualitative Camera-Controlled Generation.

Figure 9 qualitatively evaluates image-to-video generation conditioned on camera trajectories and structured semantic captions. The GameVerse-finetuned LTX-Video-2 examples exhibit camera-aware motion and coherent geometry over long-duration gameplay.

F.2 Qualitative Camera-Controlled Generation.

  • Blue trajectory overlays show the input camera-pose control signals; generation is conditioned on both these trajectories and structured semantic captions.Source: §F.2 · p. 21
  • Across road navigation, horseback riding, forest exploration, snowy environments, and urban scenes, generated videos follow complex camera motions while preserving long-term scene consistency.Source: §F.2 · p. 22
  • Large camera movements retain consistent scene geometry and spatial layouts, suggesting GameVerse camera-trajectory supervision improves geometry-aware, controllable long-duration generation.Source: §F.2 · p. 22

Related papers

Loading 2610.37257v1…