PaperlayerView exact revision on arXiv

Source-linked AI summary

Emoception: Selective Affective Layer Fine-Tuning of Video Vision Transformers for Player Arousal Change Recognition From Gameplay Footage

Yi Xia, Ibrahim Khan, Mury Fajar Dewantoro, Wenwen Ouyang, Ruck Thawonmas

arXiv:2610.07603v1cs.HCcs.AIcs.CV

TL;DR

Recognizing player arousal changes from gameplay footage is an emerging affective-computing focus. SALFT selects a ViViT layer using its parameter change after adaptation, and maintains statistically comparable performance across nine games while reducing trainable parameters by approximately 92%.

  • Problem

    Recognizing players’ emotional dynamics during gameplay is an emerging focus, and modeling arousal changes offers insight into engagement and cognitive states.

    Source: §I · p. 1

  • Method

    SALFT selects the most task-responsive ViViT layer using the L2-norm change in layer parameters.

    Source: §I · p. 1

  • Results

    Across all nine games, SALFT ViViT maintained statistically comparable performance to full fine-tuning and the best-performing baseline, reducing trainable parameters by approximately 92%.

    Source: §VI · p. 10

  • Takeaways & Limitations

    The authors position SALFT as a practical and robust solution for resource-constrained affective computing in gaming.

    Source: §VI · p. 10

    Generalizability of the layer selection strategy requires further validation across more diverse games and domains.

    Source: E. Limitations · p. 9

Abstract

from arXiv · show

This article proposes Selective Affective Layer Fine-Tuning (SALFT), an efficient adaptation framework for Video Vision Transformers in player arousal recognition from gameplay. To bypass computationally expensive full fine-tuning, SALFT introduces a selection criterion based on the L2-norm change in layer parameters after brief adaptation, directly measuring representational shifts and providing a more stable basis than gradient-based alternatives. Evaluated via five-fold cross-validation on the Arousal Video Game AnnotatIoN dataset, SALFT achieves performance comparable to full fine-tuning across all games without statistically significant degradation ($p>0.05$), while updating only $\approx$8% of parameters (over 92% reduction). Notably, in one game, SALFT consistently outperforms both full fine-tuning and the best baseline across all metrics and folds, reaching the theoretical minimum p-value (p=0.0625, exact two-sided Wilcoxon signed-rank test). In addition, we introduce an interpretability method to trace attention patterns, enhancing model transparency. These results establish SALFT as an effective and efficient approach for affective game computing.

I. INTRODUCTION

The paper targets subject-agnostic recognition of player arousal changes from gameplay footage, proposing SALFT to adapt ViViT by selecting a task-responsive layer. It also introduces spatiotemporal relevance maps and reports competitive results with a small trainable-parameter fraction.

A. Subject-Agnostic Affective Computing in Games

  • The paper frames arousal as an evolving affective cue in gameplay, building on subject-agnostic inference from in-game signals without physiological sensors.Prior work cited in the introduction used gameplay footage or footage combined with telemetry.Source: §I · p. 1, A. Subject-Agnostic Affective… · p. 2

I. INTRODUCTION

  • SALFT selects the most task-responsive ViViT layer using L2-norm parameter change, improving parameter efficiency for arousal recognition.The paper presents the criterion as a directly measurable alternative for identifying which layer to fine-tune.Source: §I · p. 1
  • ∼8% trainable parameters let SALFT consistently match or numerically exceed full fine-tuning across all nine games.Comparisons with baselines including pixel-based ResNet+LSTM confirm its competitiveness.Source: §I · p. 2
  • The proposed ViViT interpretability method generates efficient, fine-grained spatiotemporal relevance maps for examining model decisions.The introduction describes these maps as revealing how the model attends to gameplay dynamics when predicting arousal changes.Source: §I · pp. 1–2

B. Efficient Adaptation of Video Transformers

The paper motivates SALFT against the costs and limitations of existing efficient-adaptation strategies, and combines a ViViT attribution method with a gameplay-footage task formulation that retains stable-arousal segments.

B. Efficient Adaptation of Video Transformers

  • Earlier selective strategies require repeated experiments or synthetic data, while gradient heuristics may not reliably choose the best block; SALFT instead uses post-adaptation parameter displacement.Its L2-norm change criterion measures how much layer parameters adjust to the task.Source: B. Efficient Adaptation of Video… · p. 2

C. Explainability in Video Transformers

  • The ViViT attribution method combines Grad-SAM’s gradient weighting with multi-layer rollout to produce pixel-level spatiotemporal maps.The design addresses the computational cost of directly adapting LRP to ViViT.Source: C. Explainability in Video Transformers · p. 2

A. Task Reformulation and data preprocessing

  • Following, each sample uses consecutive 3-second windows, retains stable as well as changed-arousal segments, and feeds 32 frames to ViViT.The resulting segment is 6 seconds; threshold selection and tuning are beyond the study’s scope.Source: A. Task Reformulation and data… · pp. 2–3

B. Selective Affective Layer Fine-Tuning

SALFT first uses exploratory full fine-tuning to identify the layer with the greatest stable parameter displacement, then selectively fine-tunes that layer and the classifier while freezing the rest.

B. Selective Affective Layer Fine-Tuning

  • The two-stage procedure first explores full fine-tuning to identify the layer whose parameters deviate most, then selectively fine-tunes that layer.The framework uses cross-entropy loss throughout both stages.Source: B. Selective Affective Layer Fine-Tuning · p. 3
  • Layer parameters are flattened and concatenated, then compared with their pretrained vectors using ℓ2 distance to form displacement values.The classifier layer is excluded from displacement computations and stability analysis.Source: B. Selective Affective Layer Fine-Tuning · p. 3
  • The displacement ranking is checked across successive epochs using Spearman correlation, and the layer with the largest displacement at the selected epoch is chosen.The stated stability condition requires correlation above the threshold and statistical significance.Source: B. Selective Affective Layer Fine-Tuning · pp. 3–4
  • The second stage reinitializes the model and trains only the selected layer and classifier, freezing all other layers.The optimizer state is reset before this selective run.Source: B. Selective Affective Layer Fine-Tuning · p. 4

C. Model interpretability

The interpretability method combines class-specific gradients with raw ViViT attention to build token-level relevance, adapting the aggregation for spatiotemporal inputs. It turns these scores into per-frame heatmaps for visualizing model attribution.

C. Model interpretability

  • Raw attention replaces computationally expensive LRP relevance in a Chefer-style, non-averaging aggregation adapted to ViViT.The approach combines Chefer et al.’s aggregation framework with Grad-SAM’s gradient-weighting principle.Source: C. Model interpretability · p. 4
  • The method extracts raw multi-head self-attention maps from each Transformer encoder layer before aggregating them.The maps have dimensions H×N×N, where H is the number of heads and N is the number of tokens.Source: C. Model interpretability · p. 4
  • Class-specific gradients weight each layer’s attention maps, and the identity term accounts for residual connections in the intermediate relevance matrix.The matrix is row-normalized before backward recursive aggregation, initialized from the identity matrix at the output layer.Source: C. Model interpretability · p. 4
  • The final matrix C0 provides target-class token relevance, which is reshaped from the [CLS] row into a spatiotemporal grid.Temporal slices are replicated over their corresponding frames, then upsampled and overlaid on the input video.Source: C. Model interpretability · p. 4

A. Dataset

The study uses the clean AGAIN subset, focusing on gameplay video and continuous arousal traces across nine games. Its block-stratified split groups temporally adjacent samples to limit leakage while preserving class balance.

A. Dataset

  • The clean AGAIN subset contains 995 two-minute sessions from 122 participants across nine games and three genres, with video, telemetry, and continuous self-reported arousal annotations.This study uses video and associated arousal traces, with annotations normalized to [0,1].Source: A. Dataset · p. 4

B. Data Split

  • Distributing users across folds addresses varied gameplay strategies, while the authors report intermixed player embeddings and mean intra-user label entropy of 0.75 as evidence against identity-driven affect prediction.The reported within-subject variation supports learning content-based rather than trivial subject-specific cues.Source: B. Data Split · p. 5
  • The split uses non-overlapping six-second temporal blocks to keep overlapping one-second windows together, then stratifies blocks across five folds and reserves 20% for testing.The remaining blocks are divided into 72% training and 8% validation while maintaining block-level grouping.Source: B. Data Split · p. 5

C. Experimental Setup

The experiments independently fine-tune a pretrained ViViT model on each AGAIN game and compare SALFT with full fine-tuning and other baselines. The comparison strategy reflects computational constraints and includes a task-aware layer-selection procedure.

C. Experimental Setup

  • ViViT-B/16×2 pretrained on Kinetics-400 is evaluated independently on nine games, with AdamW, batch size 4, learning rate 5×10^-5, and a fixed seed.Results are from a single run due to computational constraints.Source: C. Experimental Setup · p. 5

D. Baseline and Comparison Strategy

  • All methods are fine-tuned for 14 epochs, and the reported test result comes from the checkpoint with the best validation performance.The epoch limit follows the 14-layer ViViT architecture; the gradual-unfreezing schedule was excluded as a baseline based on prior work.Source: D. Baseline and Comparison Strategy · p. 5
  • The comparison includes random forest, full ViViT fine-tuning, and a fully fine-tuned 3D-ResNet-50+LSTM baseline.The random-forest analysis extends prior single-game evaluation to all nine games; the ResNet+LSTM provides a different architectural comparison.Source: D. Baseline and Comparison Strategy · p. 6
  • SALFT is also applied to ResNet+LSTM using exploratory full tuning followed by selective layer tuning to assess its use beyond ViViT.The two-stage procedure tests the layer-selection criterion on a substantially different network structure.Source: D. Baseline and Comparison Strategy · p. 6
  • LoRA was tested on ViViT but failed to converge under both reported configurations, while SALFT’s one-pass selection is motivated by the cost of training a separate model for every candidate block.A preliminary study reports that the L2 criterion selected the same layer as Cross-Val.Source: D. Baseline and Comparison Strategy · pp. 5–6

E. Evaluation.

Evaluation prioritizes weighted F1 alongside macro F1 and accuracy, with performance differences assessed per game and metric. Efficiency comparisons measure trainable parameters and computational cost for ViViT SALFT against full fine-tuning.

E. Evaluation.

  • Weighted F1 is the primary metric because it accounts for class frequency, while macro F1 treats classes equally and accuracy supports comparison with prior work.Results are reported as fold means with 95% confidence intervals; exact two-sided Wilcoxon signed-rank tests compare SALFT with full ViViT fine-tuning and the best baseline.Source: E. Evaluation · p. 6
  • With five folds, p=0.0625 is the theoretical minimum for the exact two-sided Wilcoxon test, so p<0.05 is unattainable.The analysis uses this minimum to denote consistent empirical superiority across all folds.Source: E. Evaluation · p. 6

1) Parameter Efficiency:

2) Computational Efficiency Analysis:

G. Interpretability Methods Evaluation

The evaluation benchmarks SALFT explanations against three attribution baselines and tests fidelity by perturbing tubelets for predicted and ground-truth classes.

1) Baselines:

  • The three baselines are vanilla Rollout, ViViT-adapted Grad-CAM, and Grad-SAM; LRP-based methods were excluded because adaptation caused out-of-memory errors on available hardware.Source: 1) Baselines · p. 7

2) Perturbation Analysis Protocol:

  • Perturbation tests target the predicted and ground-truth classes on each game’s test set, masking tubelet embeddings in 10 steps from 10% to 90%.Source: 2) Perturbation Analysis Protocol · p. 7
  • Deletion masks the most relevant tubelets first, while preservation masks the least relevant first; faithful explanations should yield lower deletion AUC and higher preservation AUC.Source: 2) Perturbation Analysis Protocol · p. 7

A. Overall Performance Comparison

SALFT matches or exceeds full fine-tuning across nine games with about 8% of parameters, while game-specific exceptions and visual-style interactions qualify its generalization.

A. Overall Performance Comparison

B. Analysis of Generalization Across Games

C. Efficiency Comparison

SALFT reduces trainable parameters and per-step computation, and the paper evaluates its explanations through visualizations and perturbation-based AUC.

1) Parameter Efficiency:

  • Fewer than 8% of parameters are updated, versus 100% in full fine-tuning, while task performance remains comparable across five-fold cross-validation.Source: 1) Parameter Efficiency · p. 8

2) Computational Efficiency:

D. Interpretability Analysis

E. Limitations

SALFT maintained statistically comparable performance across nine games while reducing trainable parameters and training FLOPs. The authors note limits involving generalizability, sparse visual environments, threshold sensitivity, single-layer tuning, and five-fold statistical testing.

E. Limitations

  • The layer-selection strategy needs validation across more diverse games and domains, while vision-only performance is limited in visually sparse environments.The authors identify multimodal integration as a future opportunity.Source: E. Limitations · p. 9
  • The study fixed the arousal-change threshold, left its performance sensitivity unanalyzed, and notes that single-layer tuning may miss complex feature interactions.Only a subset of improvements reached the theoretical minimum p-value, a constraint attributed to five-fold cross-validation.Source: E. Limitations · p. 10

VI. CONCLUSIONS

  • Across all nine games, SALFT ViViT maintained statistically comparable performance to full fine-tuning ViViT and the best-performing baseline, while reducing trainable parameters approximately 92% (88.65M to 7.09M) and training FLOPs 30.5%.Source: §VI · p. 10
  • Interpretability analyses found that SALFT dynamically attends to semantically meaningful game elements across different genres.The conclusion positions SALFT as a practical, robust solution for resource-constrained affective computing in gaming.Source: §VI · p. 10
Loading 2610.07603v1…