Source-linked AI summary
GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents
Yunzhe Wang, Runhui Xu, Kexin Zheng, Tianyi Zhang, Jayavibhav Niranjan Kogundi, Soham Hans, Volkan Ustun
TL;DR
Existing video benchmarks do not adequately evaluate dense state changes, entity attribution, and concurrent multi-agent reasoning needed for autonomous agents. GameplayQA builds a densely annotated, synchronized multi-POV gameplay benchmark with structured distractors and evaluates frontier MLLMs, finding systematic weaknesses in temporal and cross-video grounding, other-agent attribution, and decision-dense scenarios. The framework also supports fine-grained diagnosis and cross-domain evaluation, while excluding direct best-action decision reasoning and facing substantial annotation costs and ambiguity.
Problem
Existing benchmarks inadequately test dense state-action tracking, other-agent modeling, environment grounding, hallucination diagnosis, and multi-video understanding for agentic perception.
Method
GameplayQA densely annotates synchronized multiplayer gameplay videos using a Self–Other–World schema and generates 2.4K diagnostic QA pairs across three cognitive levels with structured distractors.
Results
Frontier MLLMs show a performance gap against humans and degrade from basic perception to temporal reasoning and cross-video understanding, especially for other-agent attribution and temporal grounding.
Takeaways & Limitations
GameplayQA provides a scalable, diagnosable framework for evaluating agentic perception in dynamic multi-agent worlds and generalizes across autonomous-driving and egocentric collaboration domains.
Takeaways & Limitations
The benchmark does not cover best-action decision reasoning, and dense annotation is labor-intensive, error-prone, and can propagate labeling mistakes across questions.
Abstract
from arXiv · showhide
Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective, capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World, a natural decomposition for multi-agent environments. From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.
1 Introduction
GameplayQA addresses gaps in evaluating agentic perception by benchmarking dense, multi-agent, temporally and cross-video grounded understanding in 3D gameplay. It provides an end-to-end pipeline and shows frontier MLLMs struggle with fast-paced decision density, other-agent modeling, temporal reasoning, and cross-video grounding.
- Motivation: Existing video benchmarks often lack embodiment, dense state transitions, hallucination diagnosis, and multi-video understanding.These limitations make it difficult to identify failures involving temporal misinterpretation, object fabrication, or role confusion.
- Framework: GameplayQA is an end-to-end framework combining structured annotation, automated question generation, quality assurance, and diagnosable error analysis.The framework is designed to evaluate cognitive foundations of agency in 3D virtual environments.
- Data and annotation: 1.22 labels/second are annotated across synchronized gameplay videos from 9 multiplayer games using a Self–Other–World decomposition.The schema separately captures the POV agent, external agents, and persistent or transient world elements.
- Benchmark construction: 2.4K QA pairs span basic perception, temporal reasoning, and cross-video understanding, with structured distractors for diagnosing hallucinations.The generation process uses combinatorial templates and distractors categorized around model failure modes.
- Findings: Frontier MLLMs show a performance gap against humans, especially on fast-paced decision-dense scenarios, other-agent questions, and long-horizon temporal or cross-video grounding.The benchmark exposes failures in attributing actions to the correct entities and reasoning across synchronized viewpoints.
2 Related Work
Related work has expanded video understanding toward temporal, spatial, and long-context reasoning, but MLLMs remain prone to hallucination. Existing benchmarks span general, egocentric, narrative, and game-video settings, motivating GameplayQA’s focus on agentic and cross-entity reasoning.
- Multimodal Large Language Models: Recent MLLMs can process extended visual streams for video understanding but remain prone to hallucinating objects and misinterpreting temporal dynamics.Reported hallucinations also include confusing causal relationships.
- Video understanding benchmarks: General video benchmarks evaluate fine-grained temporal perception, spatial grounding, and multistep inference across long-context videos.The cited benchmark landscape includes MVBench, LongVideoBench, Video-MME, and MVU-Eval.
- Video understanding benchmarks: Existing work also targets narrative understanding in movies and television and reasoning grounded in physical environments.These domains broaden video evaluation beyond generic action recognition.
- Game-video benchmarks: Game-video benchmarks include event-centric QA for 2D platformers and evaluations of graphics glitches and physical phenomena.Examples include MarioQA, GlitchBench, VideoGameQA-Bench, and PhysGame.
3 The GameplayQA Framework
GameplayQA builds a diagnostic benchmark from synchronized multi-POV gameplay, using structured entity and temporal annotations to test increasingly complex perception and reasoning.
- The benchmark contains 2.4K QA pairs generated from multiplayer gameplay footage across 9 commercial games.
- The Self–Other–World decomposition separates the POV agent, external entities, and the shared environment for multi-agent perception.
- Questions are organized across three cognitive levels, progressing from single-video perception to temporal reasoning and cross-video understanding.
- Structured lexical, scene, temporal, role, and cross-video distractors diagnose whether errors arise from semantic, temporal, attribution, or synchronization failures.
- Six independently annotated tracks support overlapping actions, states, objects, and events for concurrent timeline captioning.
- 2,709 true labels across 2,219.41 seconds yield a decision density of ρ ≈1.22 labels/second.
4 Experiments
Experiments show that frontier MLLMs remain below human performance, with accuracy declining as tasks require more temporal, entity-level, and cross-video reasoning.
- 4.1 Main Results: Gemini 2.5 Pro reaches 71.3% overall accuracy, compared with 80.5% for humans.
- 4.1 Main Results: Average accuracy falls from 61.2% at L1 to 56.0% at L2 and 49.4% at L3.
- 4.1 Main Results: Occurrence Count averages 36.5%, while Cross-Video Ordering averages 38.8%, making them the hardest L2 and L3 tasks.
- 4.2 Error Source Analysis: World-Object recognition reaches 62.0%, versus 54.0% for Other-Action and 55.4% for Other-State.
- 4.2 Error Source Analysis: Cross-video and temporal distractors produce the most errors, and difficulty increases with game pace, video length, and synchronized-video count.
- 4.3 Language Prior and Temporal Ablation: Shuffled frames retain near-baseline L1 accuracy but reduce L2 to 52.6% and L3 to 53.4%, showing the importance of temporal order.
- 4.4 Cross-Domain Generalization: Cross-domain evaluation produces 213 questions and preserves the main benchmark’s model and task difficulty ordering.
5 Conclusion
GameplayQA evaluates agentic perception in decision-dense 3D environments through synchronized multi-POV videos, structured entity decomposition, and diagnostic questions.
- The framework refines 2.4K diagnostic QA pairs with structured distractors that identify model hallucination patterns.
- Across 16 frontier MLLMs, performance declines from basic perception to temporal reasoning to cross-video understanding.
- Models particularly fail at other-agent attribution, temporal grounding, and fast-paced decision-dense scenarios.
- The framework transfers to real-world domains with minimal adaptation while preserving difficulty and model rankings.
Limitations
GameplayQA’s limitations concern decision-reasoning coverage, subjective intent labels, and costly annotation whose errors can propagate into multiple questions. The deterministic generation pipeline shifts benchmark noise primarily to annotation quality.
- Scope boundaries: GameplayQA does not cover decision-reasoning questions about the best action to take from video.Such questions require estimating rewards or action values while learning implicit reward structures and world dynamics.
- Scope boundaries: Intent identification is subjective, and approximately 8% of questions received ambiguous ground-truth labels during human evaluation.The authors retain intent questions because anticipating intent is considered diagnostically valuable for planning agents.
- Annotation burden: Annotating a 30-second clip requires 25–35 minutes because annotators track more than 100 labels and distractors across entities and temporal windows.The dense labeling process is described as extremely labor-intensive and susceptible to human error.
- Annotation burden: A single timestamp, entity-type, or description error can propagate across multiple questions because generated questions reuse labels.Quality assurance mitigates but does not eliminate the resulting annotation noise.
- Annotation burden: The deterministic, template-based generator introduces no algorithmic benchmark noise when annotations are perfectly accurate; residual noise comes from annotation errors.The authors state that some annotation noise inevitably remains despite quality assurance.
B Model Details and Inference Settings
The benchmark uses model-specific inference configurations, including direct full-video input for video-native models and frame sampling for others. Special provider limits, API instability, and rate limits constrain some evaluations.
- Video input: Video-native models receive entire videos, whereas non-video-native models receive up to 32 frames sampled at 1 FPS or uniformly over longer videos.Inputs are resized with the longer side at 720p while preserving aspect ratio.
- Special cases: Qwen 30B and 235B are capped at 30 frames, split evenly across videos for multi-video questions.For three synchronized videos, this yields 10 frames per video.
- Special cases: Seed 1.6 and Seed 1.6 Flash use 32-frame sampling because raw-video API calls were unstable.The sampling strategy was adopted to ensure consistent evaluation.
- Special cases: Gemini-3-Pro was not benchmarked because a 250-API-call daily rate limit made full evaluation infeasible.This constraint applied when the paper was written.
- Reasoning settings: Configurable reasoning modes use each provider’s default reasoning-effort setting.The benchmark reports these default modes by model family.
C.1 Games Selection and Data Source
GameplayQA combines single-POV and synchronized multi-POV footage from nine commercial games spanning diverse genres. Multi-POV recordings come from existing datasets or manually aligned streamer videos from the same match.
- Dataset composition: The dataset contains footage from nine commercially released games spanning diverse genres.The games are divided into single-POV and multi-POV synchronized groups.
- Single-POV games: Single-POV games include Minecraft, Apex Legends, No Man’s Sky, Elden Ring, Cyberpunk 2077, and Valheim.
- Multi-POV games: Multi-POV synchronized games include Counter-Strike 2, Battlefield 6, and Arc Raiders.
- Data sources: Synchronized multi-POV footage was sourced from existing datasets or manually aligned recordings of Twitch streamers playing together in one match.
C.2 Label Distribution
The label distribution reports 2,709 annotated true labels across 2,219.41 seconds of footage, corresponding to approximately 1.22 labels per second. Dataset figures and prompt templates describe question composition and single- versus multi-video evaluation formats.
- Label distribution: 2,709 true labels were annotated across 2,219.41 seconds of footage, yielding a decision density of ρ ≈1.22 labels/second.
- Question distribution: Figure 6 summarizes question opening phrases, question-code counts by task type and cognitive level, and word distributions by entity type.
- Evaluation formats: Single-video evaluation presents sampled video frames with a multiple-choice question and requires the model to return one answer letter.The evaluation prompt may contain up to 32 frames.
- Evaluation formats: Multi-video evaluation presents frame sequences from Video 1 and Video 2 before the question and answer options.The prompt template supports multiple synchronized video inputs.
- Answer extraction: An LLM judge extracts the selected option from model responses using explicit letters, implied choices, or content matching.Unclear, empty, erroneous, or non-definitive responses are assigned X.
E.1 Annotator Demographics and Expertise
The annotation team comprised five internally recruited graduate-student coauthors, with assigned labeling and evaluation roles and documented gaming experience. Their workflow used a custom tool for synchronized video labeling, verification, quality control, and question generation.
- Annotator Demographics and Expertise: Five graduate-student coauthors served as annotators, with four labeler roles and two evaluator roles, including one participant in both capacities.They were ages 21–31, comprised three male and two female participants, and were not financially compensated.
- Annotator Demographics and Expertise: 60% of annotators played video games regularly, while the remaining 40% played occasionally during the previous 100 days.Regular play was defined as 3–5 times per week and occasional play as 1–2 times per week.
- Annotator Demographics and Expertise: 60% of annotators reported 8+ years of video-game experience, while the others reported 3–8 years or 1–3 years.The experience distribution was 60%, 20%, and 20%, respectively.
- Annotation Interface: The custom annotation tool supported single-video and multi-video labeling with generated timeline captions and distractor candidates for six target entity types.Annotators verified each caption and distractor for type, content, and timeline; the tool also supported synchronized multi-video inspection.
- Annotation Workflow: The workflow comprised label generation, verification, and question preview, followed by systematic category review and removal of ambiguous labels.Verification checked event occurrence, temporal boundaries, entity type, and distractor non-occurrence conditions.