Pixels to Keys: Exploring Spatial and Motion Cues in Gameplay Inverse Dynamics

1 Introduction

• The study targets automatic keyboard-and-mouse annotation because few gameplay videos have action labels and manual annotation at scale is impractical.Collecting new frame-and-action datasets is costly and time consuming.Source: §1 · p. 1
• Delayed effects, visually indistinguishable actions, weak visual cues, class imbalance, and camera-character motion entanglement make pixel-to-key reconstruction difficult.Inertia can affect camera motion without steady input, while turning against a wall may produce poorly visible evidence.Source: §1 · p. 2
• The paper narrows the question from broad action recovery to how well navigation inputs can be recovered from visual data in a data-constrained setting.Prior systems VPT and D2E establish large-scale feasibility, while this work examines constraints relevant to later applications.Source: §1 · p. 2
• The contributions include isolating model and motion factors, advocating per-key and unweighted evaluation, and applying the recipe to Cyberpunk 2077 with attention-map analysis.The listed factors include model architecture, optical flow, and contrastive learning; attention analysis examines camera-motion disentanglement.Source: §1 · p. 2

Large-Scale IDMs and Action-Level Evaluation:

• Latent-action methods learn an action space from action-free frames, whereas this paper studies recovery of true key presses in a fixed, discrete action space.LAPO and Genie learn actions later mapped using expert labels; D2E is a closer comparison for true key-press recovery.Source: Large-Scale IDMs and Action-Level… · p. 3
• Aggregate action metrics can hide large per-action differences, motivating evaluation and training with per-key metrics.Reported examples include per-action F1 values ranging from 0.14 to 0.86, while Trackmania's forward key occurs four times as often as left or right.Source: Large-Scale IDMs and Action-Level… · p. 3
• For continuous-action recovery, H2O and OKAMI estimate body pose and retarget it to humanoid joint angles, whereas key-press predictions can be scored against labels directly.The pose target in those robotics methods is itself an estimate, so validation uses downstream task success.Source: Large-Scale IDMs and Action-Level… · p. 3

Motion, Temporal and Semantic Space Design:

• The paper tests optical flow and video-oriented or hybrid architectures for motion handling, and compares pretrained embeddings and alternative objectives for game-video semantics and sparse targets.The compared objectives include adaptive asymmetric loss, soft-F1, and action-supervised contrastive initialization.Source: Motion, Temporal and Semantic Space… · pp. 3–4

3.1 Problem Formulation

• The IDM maps a five-frame RGB clip to independent probabilities for five navigation keys, retaining the five-frame input after longer clips and joint-action outputs did not improve performance.Multiple keys may be active simultaneously, and probabilities are thresholded at 0.5 for inference.Source: §3.1 · p. 4

3.2 Datasets

• Trackmania and Cyberpunk 2077 are modeled separately using active-gameplay sequences captured at 20 Hz and 512×512 resolution, with splits drawn from different recordings.Table 1 reports counts of overlapping five-frame sequences for each dataset and split.Source: §3.2 · p. 4, Table 1 · p. 5

Optimization

• Training uses AdamW, scheduled learning rates, dropout, and input masking, and selects the checkpoint with the highest validation macro-F1.Validation and testing use complete, unaugmented and unsampled splits, while training resamples sequences with inverse-frequency weighting to address class imbalance.Source: Optimization · p. 5, §3.2 · p. 5

Training Loss

• The authors adopt Soft-F1 because key-press probabilities are strongly imbalanced, computing soft true-positive, false-positive, and false-negative counts per key over each minibatch.Other tested loss variants are described as having minor effects on the final result.Source: Training Loss · p. 5, §3.4 · p. 6

3.4 Ablation study (primary factors)

• Architecture, optical-flow input, and contrastive training objective emerge as the three main factors affecting correct frame-level key-set prediction.Frame resolution, BCE and asymmetric loss, and pretrained embeddings have minor effects in the reported experiments.Source: §3.4 · pp. 5–6

Architectures

• The architecture comparisons include CNNs, spatiotemporal ViTs, and hybrid transformers, differing in internal structure and whether they predict all frames or only the center frame.CNN and ViT models are supervised over the full five-frame sequence, while the described hybrid model is supervised at the center frame.Source: Architectures · p. 6

Additional motion flow in input

• The hybrid RGB-flow model concatenates RAFT-Large displacement fields from adjacent frame pairs with RGB channels as additional motion input.For five input frames, the four flow fields contribute eight channels, increasing the hybrid model input from 15 to 23 channels.Source: Additional motion flow in input · p. 6

Contrastive embeddings

• Contrastive initialization trains embeddings from two augmentations of each sequence, grouping examples with matching center-action labels before the prediction head is reinitialized for supervised training.The contrastive stage runs for 20 epochs, followed by 25 epochs with the prescribed prediction loss.Source: Contrastive embeddings · p. 7

4.1 Evaluation metrics

• Accuracy can be inflated by majority and inactive labels; per-key F1 reduces imbalance bias, while F macro 1 averages per-key F1 and is the primary metric.F macro 1 gives rare keys equal weight regardless of press frequency; F micro 1 pools counts across keys and can remain affected by frequency imbalance.Source: §4.1 · pp. 7–8

4.2 Results

• Table 2 reports test metrics for Trackmania configurations and a final Cyberpunk 2077 application, with checkpoints selected by validation F macro 1.The Cyberpunk row reports only navigation keys shared with Trackmania.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Quantitative Analysis

• F macro 1 is 0.746 for the center-frame HT model, versus 0.637 for the all-frames ViT and 0.367 for the all-frames CNN; the authors report other metrics also favor HT.The authors note that this architecture comparison used one seed.Source: §4.3 · pp. 8–9
• Adding motion flow raises F macro 1 by approximately 6% relative, from 0.746 to 0.791; F1,Sp increases by 45%.The authors say more seeds may be needed to establish the flow advantage more precisely.Source: §4.3 · p. 9
• Contrastive initialization improves most core navigation metrics over HT-RGBF-E2E, while F1,Sp is the exception.The reported average improvement is not large, and real-world DINOv3 embeddings did not improve performance in these experiments.Source: §4.3 · p. 9

Trackmania

• In Trackmania, skid marks, wheel pose, and track-relative motion support correct predictions, but camera motion can make reverse-right resemble forward-left and airborne braking may have no immediate visual effect.The authors identify ego-versus-camera motion separation and longer temporal context as future needs for delayed effects.Source: Trackmania · p. 10, Figure 2 · p. 10, Figure 3 · p. 10

Cyberpunk

• In Cyberpunk 2077, sustained backward motion is sometimes recovered despite combat effects, but both illustrated failures invert backward input, and weapon or hit-response motion may contribute false movement predictions.The authors note optical flow cannot fully separate player and camera motion from combat-specific motion in first-person scenes.Source: Cyberpunk · pp. 10–11, Figure 4 · p. 11, Figure 5 · p. 11

5.1 Positive Results and Design Implications

• The spatiotemporal ViT outperforms traditional CNNs, and HT with optical flow and contrastive initialization is the strongest combination studied.Accuracy can hide minority-key errors because inactive or always-on keys may dominate it.Source: §5.1 · p. 11
• The appendix evidence for training with a balanced metric is insufficient, and the authors call for more tests.Their experiments use soft-F1 rather than BCE loss.Source: §5.1 · pp. 11–12
• Higher frame resolution brings small, consistent performance improvements at higher compute cost.The authors favor the highest possible resolution without exceeding a limit when compute matters, especially for detail-heavy games like Cyberpunk.Source: §5.1 · p. 12

5.2 Limitations and Future Directions

• IDMs achieve only slightly better performance than same-game behavioral-cloning models; the authors suggest they may partly learn the dataset’s average policy, an interpretation requiring further validation.Optical flow measures displacement but not its cause, so disentangling 3D motion and independently moving cameras may require explicit scene and ego-motion modeling.Source: §5.2 · p. 12
• Some key-presses have little or no immediate visual evidence, making pixel-based reconstruction ambiguous.When such actions alter latent state without appearing within the T-frame context, recovery may require longer context, explicit state memory, and high resolution.Source: §5.2 · p. 12

• The secondary ablations cover frame resolution, pretrained embeddings, and loss functions, factors with minor, no, or negative effects on performance.Source: A Ablation study (secondary factors) · p. 16

• Higher-resolution input frames provide a small, consistent performance advantage over lower-resolution frames, at increased compute cost.CNN-128 uses 128 × 128 frames and CNN-512 uses 512 × 512 frames; both have 16,685 parameters, isolating input resolution.Source: Frame Resolution · p. 16

• With the same training recipe, frozen DINOv3 embeddings show no clear advantage over end-to-end learned embeddings; small differences warrant more seeds and games.The authors caution against a firm conclusion about pretrained real-world embeddings transferring to video games.Source: Pretrained Embeddings · p. 16
• Loss-function performance differences are minor, though the metrics suggest a slight soft-F1 advantage, especially for rarer keys.The authors note that more seeds and games may be needed to draw a reliable conclusion.Source: Loss Functions · p. 16

Loss Functions:

• The experiment configurations include the loss used for training and the number of training epochs.Source: Table 3 · p. 17
• The HT-RGBF-PT-F1 model and training recipe are applied to Cyberpunk 2077 as a second-game application.Source: Loss Functions · p. 17
• Table 4 reports accuracy and F1 metrics on Trackmania’s test split for the secondary ablations.The caption defines 20+25 as 20 contrastive and 25 supervised epochs.Source: Table 4 · p. 17
• The authors identify the Sp key as their preferred choice.Source: Loss Functions · p. 17

B Training and Validation Curves

• Reported checkpoints maximize validation macro-F1 within the epoch budgets used in Table 2.Source: B Training and Validation Curves · p. 17
• Figure 6 tracks supervised contrastive loss and positive- and negative-pair cosine similarity over 20 pretraining epochs for Trackmania and Cyberpunk 2077.Source: Figure 6 · p. 17
• Figure 7 plots training loss and validation soft-F1 loss, accuracy, micro-F1, and macro-F1, with thresholded metrics at 0.5.Training-objective values are comparable only for runs using the same loss; asymmetric loss has a different scale.Source: Figure 7 · p. 18