PaperlayerView exact revision on arXiv

Source-linked AI summary

Kuration SDK: Addressing the Virtual2Real Gap via Data Curation

Nirmit Desai, Eric Song, Mayank Sengupta, Tejal Bedmutha, Siri Reddy, Sahiti Dharmavaram, Kunal Sawarkar

arXiv:2610.09305v1cs.LGcs.AIcs.CVcs.RO

TL;DR

General-purpose benchmarks provide limited signal for assessing action-conditioned world models, despite the need to evaluate their playability. The paper uses matched CounterStrike datasets and Kuration SDK diagnostics to investigate the Virtual2Real gap, finding sharply different playability despite comparable or better benchmark scores.

  • Problem

    Existing benchmarks provide limited general-purpose signal for evaluating task-agnostic action-conditioned world models, making data curation difficult to assess against reliable capability measures.

    Source: §1 · p. 2

  • Method

    The authors compare diffusion world models trained on matched CounterStrike corpora and use Kuration SDK diagnostics to test data-curation hypotheses, including action-state consistency.

    Source: §1 · p. 2, §1.1 · p. 2, §7 · p. 6

  • Results

    Matched CS:GO and CS2 models differ sharply in qualitative playability despite comparable or better FVD, LPIPS, and JEDi scores.

    Source: §1.1 · p. 2, §1 · p. 2

  • Takeaways & Limitations

    Kuration SDK provides an open-source way to test gameplay-data diagnostics and curation strategies against qualitative playability.

    Source: §8 · p. 7

    Playability judgments rely on the authors’ unblinded rollout inspection, without a labeled benchmark or independent second rater.

    Source: §7.2 · p. 7

Abstract

from arXiv · show

Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to action-semantic and physically-grounded metrics. However, for domain and task-agnostic action-conditioned world model training, existing benchmarks provide a limited signal. By training and evaluating diffusion world models on CounterStrike gameplay data, we confirm that qualitative playability does not correspond with metrics such as FVD, LPIPS, and JEDi. We term this the Virtual2Real gap. We posit that, in lieu of reliable benchmarks, curating raw gameplay data and measuring a variety of diagnostic properties provides a more robust signal to bridge the gap, before the training even begins. We present several curation strategies and a general-purpose kit for physical AI data curation called Kuration SDK, which is being open-sourced with this paper. The SDK was instrumental in uncovering the root cause of the virtual2real gap in a specific case: why two world models trained on identical gameplay map, action and state distribution, behaved very differently when played in spite of having very similar LPIPS and FVD scores. Thus, Kuration SDK has the potential to uncover the root causes of Virtual2Real gap in specific datasets and accelerate development of sample-efficient training datasets.

1 Introduction

The paper asks how to curate training data for task-agnostic action-conditioned world models when general-purpose benchmarks provide limited signal. It reports that matched CounterStrike datasets can produce models with very different playability despite comparable benchmark scores, and introduces Kuration SDK to investigate the gap.

1 Introduction

  • Playability failures include map teleportation, geometry clipping, physics-violating actions, and weak control responsiveness or recovery.The paper assesses these failures through interactive rollout inspection, not a formal protocol or external blinded raters.Source: §1 · p. 2

1.1 Contributions

  • Matched CS:GO and CS2 models diverge in playability despite comparable or better FVD, LPIPS, and JEDi scores.The corpora match on state and action distributions, map, scale, frame rate, and resolution; the measured visual-shift magnitude differs by 1.4–2.1×.Source: §1.1 · p. 2, §1 · p. 2

1 Introduction

  • Existing benchmarks provide limited general-purpose signal for task-agnostic action-conditioned world models, motivating data-level diagnostics.The paper contrasts these evolving metrics with meaningful language-model benchmarks that supported data-curation ablations.Source: §1.1 · p. 2, §1 · p. 2

1.1 Contributions

  • The SDK identifies action-state consistency, especially visual-shift magnitude, as a leading explanation for the gap; the corpora differ by roughly 2× on this axis.FVD and LPIPS do not detect that difference.Source: §1.1 · p. 2
  • Kuration SDK is released as an open-source toolkit alongside the paper’s curation strategies and evaluation results.The authors present it as a way to reproduce, extend, or apply diagnostics to new corpora.Source: §1.1 · p. 2

2 Related work

Prior work covers world models, video-generation metrics, action-oriented benchmarks, and data curation, but the cited approaches leave a gap for general-purpose gameplay evaluation and diagnostics.

World models for physical AI

  • World models predicting future observations conditioned on actions have been proposed as planning and policy-learning substrates without further real-world interaction [Ha and Schmidhuber, 2018, Hafner et al., 2023]; Genie pursues data scale via internet-scale video pretraining without action labels [Bruce et al., 2024].Source: §2 · p. 2
  • DIAMOND [Alonso et al., 2024] is the closest comparison and serves as the baseline; prior reported teleportation and clipping failures had not been tested against a format- and scale-matched independent corpus.The paper describes its controlled comparison as closing that gap.Source: §2 · p. 3

Sim2Real vs. Virtual2Real

  • Virtual2Real is framed as a data-distribution problem in which models inherit degenerate corners of uncurated virtual-environment samples, unlike Sim2Real’s dynamics-fidelity problem addressed with domain randomization [Zhao et al., 2020, Tobin et al., 2017].Source: §2 · p. 3

Video generation metrics and their limits

  • FVD and LPIPS do not target single-interaction physical violations, while newer action-semantic benchmarks require game-specific estimators or probes, and task-specific benchmarks do not generalize.Long-horizon round-trip drift metrics can return to the start along an arbitrarily wrong path because no individual step is verified.Source: §2 · p. 3

Data curation

  • Gameplay-specific curation is motivated because fixed text or image embeddings used in language and vision-language subset selection do not expose physical interaction, action-state consistency, or spatial trajectory.Source: §2 · p. 3

3 Establishing a training baseline: reproducing DIAMOND

The authors reproduce DIAMOND’s training and evaluation pipeline on its published CS:GO data to establish a baseline before testing a new corpus. The reproduction matches reported quantitative performance within evaluation variability and retains the reported qualitative failures.

Architecture and setup

  • The authors reproduce DIAMOND before swapping in a new corpus so later differences can be attributed to data rather than training or evaluation code.The setup uses a two-stage denoiser conditioned on the previous four frames and actions, plus an independently trained upsampler.Source: §3 · p. 3

Parity result

  • The reproduced model retains long-horizon map teleportation and geometry clipping despite matching DIAMOND’s published short-horizon qualitative standard.These multi-frame failures are not visible in a single frame, helping explain why frame-level and short-clip metrics struggle to detect them.Source: §3 · p. 3
  • FVD is 4.4% below DIAMOND’s reported value at epoch 300, while LPIPS is 4.2% above it at epoch 420.Released DIAMOND weights reproduce the paper’s numbers within 1%, and both differences fall within evaluation stochasticity.Source: §3 · p. 3, Table 1 · p. 4

4 Controlled replication: Training on partner CS2 data

On the matched CS2 test set, baseline_v1 scored better on LPIPS than DIAMOND but remained less playable, illustrating that metric gains did not track rollout quality.

4 Controlled replication: Training on partner CS2 data

  • 20.9% better LPIPS than DIAMOND on the CS2 test set by epoch 600 did not translate to better playability; baseline_v1 showed more severe clipping and spatial discontinuity.The models were evaluated on the same held-out CS2 episodes.Source: §4 · p. 4
  • The CS2 corpus matched DIAMOND’s CS:GO data on de_dust2, 16 fps, action encoding, and corpus scale, while recorded players, matches, and moments necessarily differed.baseline_v1 used 8,635 raw episodes and was evaluated on 1,090 held-out episodes.Source: §4 · p. 4
  • Across training, LPIPS improved with noise, FVD was competitive only early before oscillating, and JEDi improved at every checkpoint despite worse playability.No JEDi reference was available for DIAMOND on this test set.Source: Figure 1 · p. 4, §4 · p. 4

5 Action-State consistency: the signal behind the gap

Action-state consistency measures how reliably controls produce characteristic screen motion; the matched-format corpora differed on this property, which tracked playability where FVD and LPIPS did not.

5 Action-State consistency: the signal behind the gap

  • The corpora matched on map, resolution, frame rate, and action encoding, and the action_state_consistency hypothesis was validated as a difference between them.The authors caution that the games use different engine generations, which may contribute to the measured gap.Source: §5 · pp. 4–5
  • The operator measures optical-flow displacement during uninterrupted key presses along the expected camera-motion axis, then aggregates responses by control and press length.The paper notes that engine features such as input buffering and lag compensation can break the input-to-motion contract.Source: §5 · p. 5
  • CS:GO peak pixel shifts were 1.4–2.1× larger across four movement controls, and its multi-frame responses accelerated through release (112–126%) and took 2–3× longer to decay than CS2.The single-tap figure also shows consistently larger CS:GO responses and longer decay.Source: §5 · p. 5, Figure 2 · p. 5
  • CS2’s backward response was 80% of its forward response, versus 26% for CS:GO; CS2 release response was 76–78% of hold magnitude, while CS:GO reached 112–126%.The paper characterizes CS2’s backward-forward response as closer to physical near-symmetry.Source: §5 · p. 5
  • Action-state consistency tracked playability where FVD and LPIPS did not; magnitude is the leading proposed driver, but no controlled ablation isolates it from momentum and symmetry.The proposed mechanism is that a larger, more salient pixel shift may give an action-conditioned denoising model a stronger training signal.Source: §5 · p. 5

6 Kuration SDK

Kuration SDK is an open-source toolkit for turning gameplay and other sequential data into reproducible, training-ready shards through configurable curation pipelines.

6 Kuration SDK

  • The pipeline segments source data, measures per-segment features, selects or filters under a time or count budget, and writes shards with a config, manifest, and checksum.The train/test split is leakage-free, and the config alone reproduces the curated dataset.Source: §6 · p. 6
  • Gameplay operators include pixel_shift and action_state_consistency, collision detection for oversampling wall hits, and camera-trajectory reconstruction scored for gaps, impossible motion, and diversity.The operators target action-to-motion consistency, wall clipping, and spatial coverage, respectively.Source: §6 · p. 6
  • Three reusable selection recipes provide quantile-bin coverage, rare-event oversampling to a target rate, and cluster-then-sample diversity selection.The recipes compose with the operators without additional code.Source: §6 · p. 6

7 Curation strategies and results

Among four tested CS2 curation strategies, only action-state-consistency selection noticeably improved playability; the authors qualify the finding by incomplete evaluations and the narrow experimental scope.

7 Curation strategies and results

  • The four strategies covered quantile bins of aim velocity, input entropy, and movement speed; up-weighted rare high-speed trajectories; emphasized collision-flagged segments; or selected large per-action pixel shifts.Coverage training ran 120 of a planned 600 epochs; rare-trajectory and collision-aware runs were fine-tuned from baseline_v1’s epoch-300 checkpoint.Source: §7 · p. 6

7.1 One strategy improves playability

  • Only action-state-consistency selection noticeably improved playability over baseline_v1; coverage-based, rare-trajectory, and collision-aware selection did not.The authors do not know whether the improvement closes the gap to CS:GO DIAMOND, and have not ruled out effects from fine-tuning itself.Source: §7.1 · p. 6
  • Metric movement was not a reliable indicator of playability: coverage-based training reached near-baseline LPIPS and FVD in 105 of 120 epochs without improving rollouts.Collision-aware selection improved JEDi over the comparably fine-tuned rare-trajectory run, also without qualitative playability improvement.Source: §7.1 · p. 6
  • Table 2 combines each strategy’s best value per metric, not values from one checkpoint; blank cells denote evaluations that were not completed.The action-state-consistency training schedule was not finalized.Source: §7.1 · p. 7, Table 2 · p. 7

7.2 Limitations

  • Playability judgments came from author-inspected rollouts without a labeled benchmark or independent rater, and the raters were not blinded to strategy.The authors note that expectation bias cannot be ruled out.Source: §7.2 · p. 7
  • Results cover one map, one DIAMOND architecture, and one game family; task-agnostic generality remains un demonstrated beyond this setting.CS:GO and CS2 also differ in engine generation, physics, and netcode, which may partly explain the action-state-consistency gap.Source: §7.2 · p. 7

8 Conclusion and Future Work

Format- and scale-matched CounterStrike corpora show a Virtual2Real gap: benchmarks improve slightly while playability worsens significantly. The Kuration SDK helped test curation hypotheses against playability, while transfer of the tested curation axes beyond CS2 remains an open question.

8 Conclusion and Future Work

  • In matched CounterStrike corpora, benchmarks improve slightly while playability gets significantly worse, demonstrating the Virtual2Real gap.The SDK tested curation hypotheses against qualitative playability; only the hypothesis grounded in a measured corpus difference moved playability.Source: §8 · p. 7
  • The SDK’s value was testing and discarding curation hypotheses without one-off scripts, though this efficiency was not measured against another workflow.The authors release the SDK, strategy specifications, and evaluation results; they propose a physics-conformance metric and a shared benchmark for more conclusive comparisons.Source: §8 · p. 7
  • Whether interaction density, spatial coverage, and action-state consistency transfer to robot teleoperation or egocentric video remains an open question.The authors identify this transfer as necessary to make the case for domain-general curation infrastructure concrete rather than analogical.Source: §8 · p. 8

A Full Per-Checkpoint Results

The CS:GO reproduction’s FVD and LPIPS plateau after epoch 300 even as denoiser losses continue to fall. Table 3 reports noisier, non-monotonic JEDi for this run, while Table 4 covers baseline_v1 checkpoints on CS2 with one checkpoint unavailable.

A Full Per-Checkpoint Results

  • Denoiser train loss falls from 0.0682 at epoch 30 to 0.0339 at epoch 600, while test loss drops from 0.0469 at epoch 60 to 0.0240 at epoch 600.JEDi is noisier than on the CS2 run, does not improve monotonically, and reaches its best value late in training.Source: A Full Per-Checkpoint Results · p. 10
  • FVD crosses below DIAMOND’s reported value at epoch 180 and reaches its best value at epoch 300; LPIPS and FVD then plateau and oscillate despite continued loss decreases.The trend is shown across the 600-epoch CS:GO reproduction checkpoints.Source: Figure 3 · p. 10, A Full Per-Checkpoint Results · p. 10, Table 3 · p. 10
  • Table 4 covers baseline_v1 checkpoints on the 1,090-episode CS2 test set; epoch 510 is omitted because of a storage-retention error.The unavailable checkpoint is omitted rather than interpolated.Source: A Full Per-Checkpoint Results · p. 11, Table 4 · p. 11

B Action-State Consistency Pipeline Example

The example windows gameplay into segments, measures W-key pixel shift, scores action-state consistency against a supplied baseline, and selects a coverage-balanced set. The output retains a per-window anomaly count for later curation.

B Action-State Consistency Pipeline Example

Loading 2610.09305v1…