Source-linked AI summary
EgoCS-400K: An Egocentric Gameplay Dataset for World Models
Rongjin Guo, Dong Liang, Yuhao Liu, Fang Liu, Tianyu Huang, Gerhard P. Hancke, Rynson W. H. Lau
TL;DR
World-model data needs temporally aligned video, actions, states, and events, but existing datasets lack this combination at scale. EgoCS-400K addresses this gap with replay-grounded Counter-Strike trajectories aligned to rendered first-person videos and dense annotations, providing a large testbed for interactive visual modeling.
Problem
Existing web, egocentric, robotic, and simulation datasets do not provide large-scale, temporally aligned egocentric video-action-state-event trajectories for world modeling.
Method
EgoCS-400K builds replay-grounded Counter-Strike trajectories from public demos, aligning rendered first-person videos with human actions, camera motion, player states, and game events.
Results
EgoCS-400K provides over 400,000 first-person videos spanning more than 10,000 hours, 1,000 matches, 40,000 rounds, 13 maps, and 10 viewpoints per round for interactive visual modeling.
Takeaways & Limitations
EgoCS-400K offers a practical intermediate testbed for studying how first-person scenes evolve under human actions across interactive visual modeling tasks.
Takeaways & Limitations
Because its behavior and dynamics are centered on Counter-Strike, EgoCS-400K does not directly model continuous physical interaction, tactile feedback, or everyday real-world behavior.
Abstract
from arXiv · showhide
The shift from video generation to interactive world modeling places new demands on data: beyond captioned videos, world models require temporally aligned video-action-language trajectories grounded in the actions, camera motion, states, and events that drive future scene changes. However, such data is difficult to obtain at scale. Web video datasets offer broad visual coverage but lack executable actions and reliable states; robotic datasets provide action and state supervision but are costly and limited in scene diversity; and existing simulators often lack large-scale human-driven interaction trajectories. In this paper, we introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models, built from public professional CS and CS2 match demos that preserve human gameplay trajectories and enable parsing, replaying, rendering, and temporal alignment. We extract player states, view directions, movements, keyboard/button inputs, view-angle changes, weapon usage, game events, and round-level context, and render clean first-person videos from the same trajectories. EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds, covering 13 maps and 10 player viewpoints per round. It supports a range of interactive visual modeling tasks, including action-conditioned future prediction, state- and event-aware scene rollout, replay-grounded captioning, and agent egocentric action understanding. By connecting visual observations with human actions, camera motion, game states, and events at scale, EgoCS-400K serves as a practical bridge between passive web videos, controllable game simulation, and costly real-world embodied data.
W. H. Lau
The section identifies equal contribution and corresponding-author roles, and lists the paper’s keywords: world models, egocentric video, gaming agents, and video generations.
- Author notes: The paper notes equal contribution and identifies a corresponding author.The passage marks equal contribution with an asterisk and corresponding authorship with a dagger.
- Keywords: Keywords: World Models, Egocentric Video, Gaming Agent, Video Generations.
1. Introduction
EgoCS-400K addresses the lack of scalable, replay-grounded action and state supervision for world models by using professional Counter-Strike demos to align first-person video with gameplay trajectories. It provides multi-grained annotations and large-scale coverage for interactive visual modeling tasks.
- Motivation: World models require visual dynamics and action-conditioned dynamics, but existing web, egocentric, game, and simulation datasets lack sufficiently aligned actions, states, or world-model-specific design.Web videos are broadly diverse but mostly passive; egocentric videos typically lack precise control traces and reliable internal states.
- Data source: Counter-Strike demos preserve replay-grounded human trajectories containing player states, camera motion, actions, game events, and round-level context for replay, rendering, and temporal alignment.This makes demos suitable for constructing human-driven, first-person, visually rich, temporally precise training data without new manual collection.
- Dataset construction: EgoCS-400K aligns clean first-person videos with human actions, camera motion, player states, and game events through replay-derived parsing and rendering.The pipeline extracts states, view directions, movements, weapon usage, utility events, combat events, and round-level context.
- Capabilities: Its hierarchical annotations organize player-view sequences, segments, protected action chains, protected atomic actions, and per-tick state traces, with captions and prompts constrained by replay-derived facts.These annotations support action-conditioned future prediction, state- and event-aware rollout, controllable egocentric video simulation, and egocentric action understanding.
2. Related Work
Related work frames interactive world models around action-conditioned, real-time, long-horizon, memory-aware, and geometrically grounded generation. Existing egocentric, video-language, game, and embodied datasets provide broad supervision, while EgoCS-400K links first-person observations to replay-derived controls, camera motion, game state, and events.
- Interactive world models: Interactive world models increasingly generate controllable future frames from user inputs, latent actions, keyboard/mouse commands, or high-level instructions.Recent work also targets real-time or streaming interaction, long-horizon consistency, memory, and geometric grounding.
- Egocentric video datasets: Ego4D, EPIC-KITCHENS, and related egocentric datasets support large-scale first-person perception, action understanding, narrations, social interaction, hand-object reasoning, and episodic memory.Their action labels are not the low-level controls that generate future observations.
- Egocentric video datasets: EgoCS-400K complements real-world egocentric video by tying every segment to replay-derived controls, camera motion, internal game state, and discrete game events.It is less semantically open-ended than real-world egocentric video but provides executable, structured supervision.
- Video-language and action recognition datasets: Video understanding datasets standardize action classification, temporal localization, procedural description, and video-text learning through curated and web-supervised collections.Representative datasets include Kinetics, ActivityNet, UCF101, Something-Something, YouCook2, HowTo100M, and WebVid.
- Game and embodied-agent datasets: Game and embodied-agent datasets use controllable environments for imitation, reinforcement learning, generalist virtual-world agents, and multimodal interaction.MineRL, MineDojo, and PLAICraft provide Minecraft demonstrations, open-ended tasks, or time-aligned video, audio, speech, mouse, and keyboard modalities.
- Multimodal data engines and structured supervision: EgoCS-400K uses a replay data engine in which demo files provide ground truth, rendering produces first-person observations, and parsing yields dense annotations with an audit path to replay events.The construction pipeline renders synchronized videos, filters invalid outputs, parses per-tick player data, and derives keyboard/mouse annotations, action sequences, action chains, and video segments.
3. Method
EgoCS-400K constructs source-traceable video-language-action data by treating the replay timeline as canonical supervision and rendered first-person video as visual evidence. The method parses synchronized player traces into atomic actions, preserves action continuity during dynamic-programming segmentation, and builds local VLM prompt instances from aligned annotations.
- Overview: The replay timeline provides authoritative timing for player states, actions, camera motion, weapon states, and game events, while rendered first-person video supplies corresponding visual evidence.This design decouples canonical supervision from visual observation while retaining temporal alignment.
- Method pipeline: The pipeline collects demos and renders and filters first-person videos, parses tick traces into action spans, then constructs prior-guided VLM inputs.These steps are organized into three phases: data preparation, trace parsing and segmentation, and prompt construction.
- Player-level parsing: Per-tick parsing records positions, view angles, movement and input states, weapon information, and other player-level signals from playable rounds.Keyboard and mouse annotations are reconstructed from synchronized button bitmasks and view-angle changes, with the latter serving as continuous mouse-motion proxies.
- Atomic action extraction: Rule-based detectors convert synchronized raw signals and game events into temporally bounded atomic action spans with timing, identity, source, confidence, and structured details.Multi-stage actions can be represented as linked sub-events when their underlying signals support them.
- Action timeline and segmentation: Dynamic-programming segmentation divides long player-view sequences into short segments while preserving cut-protected action chains and incorporating pre-action visual context.The optimization uses valid segment and gap edges, with costs for uncovered gaps, pre-action context shortfall, segment count, duration preference, and boundary preference.
- Prompt construction: For each caption target, tick traces, action spans, movement and camera events, and state summaries are clipped and re-based to local time for VLM reasoning.Targets are either DP-selected training segments or dense protected action intervals.
4. Analysis and Limitations
EgoCS-400K is a large-scale, densely annotated egocentric gameplay dataset that aligns first-person video with actions, states, inputs, events, and captions. Its Counter-Strike-centered discrete mechanics make it a useful intermediate testbed for action-conditioned modeling, while limiting direct real-world embodiment.
- Dataset scale: More than 400K round-player videos span over 10K hours, more than 40K rounds, over 1K professional matches, 13 maps, and up to 10 synchronized viewpoints per round.Each basic unit is one player’s first-person view within one playable round.
- Annotations: Dense tick-aligned annotations combine player states, keyboard and mouse signals, weapon and movement states, action spans and timelines, protected chains, training segments, and multi-grained captions.Segment captions describe clip-level visual progression, while protected-chain captions focus on dense action intervals.
- Research utility: The dataset exposes action, camera, state, and event signals, enabling study of how first-person observations evolve during navigation, viewpoint control, visual search, tactical interaction, and rapid transitions.This design positions EgoCS-400K as a scalable testbed for action-conditioned modeling and agent-oriented learning.
- Qualitative alignment: A four-second example synchronizes first-person frames, keyboard and mouse traces, action timelines, and generated prompts to show temporally ordered gameplay changes.The illustrated sequence includes weapon switching, inspection, airborne movement, grenade preparation, projectile flight, and subsequent weapon state.
- Limitations: Counter-Strike-centered behaviors and mechanics limit coverage of broader embodiment, because controls are discrete and game-defined state machines, physics, and map design govern interactions.The dataset is therefore characterized as an intermediate testbed rather than a direct model of real-world embodiment.
5. Conclusion
EgoCS-400K offers scalable first-person video-language-action data by aligning rendered videos with dense annotations across a shared timeline. It also serves as a testbed and bridge for learning how visual observations and human actions are temporally coupled.
- Conclusion: EgoCS-400K pairs clean rendered videos with dense annotations, aligning visual observations, language descriptions, actions, states, and event structure within a shared timeline.This design enables fine-grained analysis of how first-person scenes evolve under human actions.
- Conclusion: EgoCS-400K provides a practical testbed for action-conditioned generation and agent-oriented egocentric modeling.
- Conclusion: The game-based setting introduces domain gaps to real-world embodiment but offers a scalable bridge toward models learning temporal coupling between visual observations and actions.