Multi-Label Perceptual Bug Detection in Video Games using Deep Learning on Gameplay Footage

I. INTRODUCTION

• Manual testing is time-consuming and expensive, and automated detection can require complex, resource-heavy game-engine instrumentation.The introduction identifies a need for detection methods that do not require deep integration into the game engine.Source: §I · p. 1
• Perceptual bugs can involve visual and temporal cues, and multiple bugs may occur together, making simultaneous identification harder than multiclass classification.Some bugs, such as Z-fighting, flicker across frames; cascading defects can produce multiple bugs in one clip.Source: §I · p. 1
• The paper frames perceptual bug detection as supervised multi-label classification and proposes an architecture combining spatial and temporal feature extraction with a multi-task objective.The model is intended to act as an automated test oracle for multi-label bug classification.Source: §I · p. 1
• The introduced benchmark contains 77,969 gameplay clips with combinations of up to three classes from five perceptual bug classes.The dataset comprises recorded gameplay sessions and includes clips containing multiple perceptual bugs.Source: §I · p. 1

A. Traditional Game Testing

• Manual game testing relies on human observation and can make evaluation costly and time-consuming.A cited exploratory-testing framework uncovered 80 bugs, but its authors noted the approach depended heavily on human testers.Source: A. Traditional Game Testing · p. 2

B. Automated Bug Detection

• Automated testing can require greater initial effort than manual testing, though comparative findings report advantages for repetitive testing and defect detection.The literature does not establish a definitive overall effectiveness advantage for automated over traditional testing.Source: B. Automated Bug Detection · p. 2
• Some automated approaches rely on game-state access and manually defined rules, while model-based and scripted methods report limits involving complex games, path coverage, or unpredictability,.The model-based approach also requires users with software-engineering experience to create models.Source: B. Automated Bug Detection · p. 2
• Other work includes gameplay-video anomaly analysis, navigation frameworks with limited generalization, and agent-based approaches that can struggle to identify primary bugs,,.GELID’s categorization and context-based clustering steps were reported to need refinement for reliable practical use.Source: B. Automated Bug Detection · p. 2

C. AI and Machine Learning in Game Testing

• Prior AI testing often centers on game-environment exploration, whereas this work targets detecting multiple bugs in human-played gameplay videos.The paper notes that earlier rule-based approaches did not detect high-level perceptual and behavioural bugs,.Source: C. AI and Machine Learning in Game… · p. 3
• World of Bugs uses rendered scenes to detect perceptual bugs, with reported accuracy varying across categories.The cited authors suggested rear-geometry rendering instances may have led the model to miss some bugs.Source: C. AI and Machine Learning in Game… · p. 3
• A prior hierarchical system detects visual anomalies using game-state information, while earlier work from the authors combined temporal anomaly detection with clustering to categorize bugs.The earlier pipeline used ConvLSTM over gameplay-frame sequences and DBSCAN to cluster predicted buggy frames.Source: C. AI and Machine Learning in Game… · p. 3
• Other video-based methods include static-frame classification for a single glitch per frame and object-detection approaches for visual bugs, including multi-frame defects,.The cited works use differing setups, including scarce-label training and real-time localization.Source: C. AI and Machine Learning in Game… · p. 3
• The paper identifies the absence of prior work and datasets for simultaneous perceptual-bug detection in gameplay footage and introduces a dataset and model architecture to address that gap.The benchmark consists of labelled, human-played footage with multiple perceptual bugs occurring simultaneously.Source: C. AI and Machine Learning in Game… · pp. 3–4

III. MATERIALS AND METHODS

• A figure presents the dataset preparation pipeline.The supplied passage gives the figure title but no pipeline stages.Source: Figure 1 · p. 4

A. Dataset Preparation

• Gameplay sessions were recorded from Open Nights, Unity3D Mario Kart Racing Game, and FPS Microgame, with study data collection approved by the institutional Research Ethics Board.Source: A. Dataset Preparation · p. 4

1) Open Nights:

• Open Nights is a first-person survival horror game with low-light environments and static camera angles, selected to test detection of texture corruption and Z-fighting in low light.Source: 1) Open Nights · p. 4
• Unity3D Mario Kart Racing Game provides high-speed movement on colourful tracks, testing bugs in complex 3D environments with fast-moving assets.Source: 3) FPS Microgame · p. 4

3) FPS Microgame:

• The FPS Microgame is a fast-paced first-person shooter with rapid camera movements, selected to test distinguishing motion blur from geometry corruption and Z-fighting.Source: 3) FPS Microgame · p. 4
• The dataset’s game titles were selected to provide diverse genres, graphical demands, and bugs for testing models.Source: 3) FPS Microgame · p. 4

B. Dataset Description

• Up to three perceptual bugs, drawn from five classes, were injected simultaneously when the character came in close range of a predetermined game object; every class appeared in each game's gameplay sessions.Footage was resized to 224 × 224 and divided into 2-second clips at 8 fps. The dataset was split into 70% training, 15% validation, and 15% test sets, maintaining the proportions of bug combinations in each split.Source: 3) FPS Microgame · p. 4
• Five perceptual bug classes—boundary hole, texture corruption, geometry corruption, Z-clipping, and Z-fighting—can occur in combinations of up to three concurrent bugs.The three-bug maximum was set to prevent overlapping bugs from occluding one another.Source: B. Dataset Description · pp. 4–5
• 77,969 two-second clips at 8 fps contain 1,247,504 frames and are split into 70% training, 15% validation, and 15% test data.Each split includes all five bug classes and approximately the same proportions of bug combinations.Source: B. Dataset Description · p. 5
• Table II reports bug-count combination distributions across splits, while Table III reports individual bug-class distributions across splits.Source: Table II · p. 5, Table III · p. 5

• ResNet-BiLSTM extracts frame-level visual features with pretrained ResNet-18, then uses a BiLSTM to extract temporal information across video frames.A temporal attention layer selects essential features before dropout regularization and classification.Source: C. Model Architecture · p. 5
• Three classification heads predict multiple bug classes with sigmoid activation, bug count from 0 to 3 with softmax, and bug presence with sigmoid.The model architecture and output shapes are shown in Fig. 3.Source: C. Model Architecture · p. 5

A. Hardware, Environment and Hyperparameters

• Experiments ran on an NVIDIA H100 server using Fedora Linux, with models and experiments built in PyTorch and tracked with Weights & Biases.Source: A. Hardware, Environment and… · p. 5

B. Loss function

• The multi-task objective combines three loss terms for multi-label bug classification, bug-count detection, and bug-presence detection.Source: B. Loss function · pp. 5–6
• Asymmetric Loss handles bug classification, Focal Cross-Entropy handles bug counts, and Binary Cross-Entropy handles bug presence; scalar weights control the auxiliary losses.Source: B. Loss function · p. 6

C. Evaluation metrics

• Micro-averaged F1 gives equal weight to each prediction rather than each class.The authors select it as their multi-label bug classification metric because it suits class imbalance.Source: C. Evaluation metrics · p. 6
• The authors identify class imbalance as a reason for using micro-averaged F1.Source: C. Evaluation metrics · p. 6
• Precision is TP/(TP+FP), while recall is TP/(TP+FN).TP, FP, and FN denote true positives, false positives, and false negatives, respectively.Source: C. Evaluation metrics · p. 6

V. RESULTS

• 85.78% F1 was achieved by ResNet-BiLSTM, which outperformed I3D and R3D-18 on the test dataset.The comparison models were fine-tuned with the same hyperparameters.Source: §V · pp. 6–7, Table IV · p. 7

VI. DISCUSSION

• ResNet-18 with GRU scored 19.34% F1; the BiLSTM model had a 76.44% F1 score gain over GRU.The authors suggest BiLSTM may provide more contextual information.Source: §V · p. 7, §VI · p. 7
• Performance was strongest for boundary-hole combinations and moderate for Z-clipping, while geometry corruption and Z-fighting were hardest to detect.The authors suggest subtler visual artifacts may explain the weaker performance for geometry corruption and Z-fighting.Source: §VI · p. 7
• Videos with three bug classes performed better than those with fewer classes, which the authors attribute to more pronounced visual features.Source: §VI · p. 7
• 65.55% of clips contained no perceptual bugs, reflecting the authors’ aim to simulate games with fewer bug instances.The authors say limited computational resources constrained their choice of distribution, although bug frequency can vary by game.Source: §VI · p. 7
• The dataset covers three games and five bug classes; synthetic injected bugs may lack real-world nuance, and combinations always begin and end together.The authors note these constraints may limit generalization and identify varied bug-event durations as a direction for further testing.Source: §VI · p. 7

VII. CONCLUSION AND FUTURE WORK

• The dataset contains gameplay from three games, with combinations of one to three bug types from five classes, and ResNet-BiLSTM achieved 85.78% F1.Source: §VII · p. 7
• The authors conclude that temporal context is necessary to capture features for video-based bug detection.Source: §VII · p. 7
• Future work should add videos from more game genres and examine ablation, real-time inference, and temporal localization on full-length gameplay.Source: §VII · p. 7