PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Video Generation for Reasoning: At What Cost?

Weihang Guo, Xiaoyu Wu, Yifei Wang, Niloofar Mireshghallah, Lydia E. Kavraki

arXiv:2609.36599v1cs.CVcs.AIcs.LG

TL;DR

The paper asks whether video generators can reason about hidden past states, rather than merely produce plausible videos, and how much compute reliable state prediction requires. It studies this in a controlled Rubik’s Cube video task, finding that lower validation MSE does not reliably signal better state accuracy and that symbolic state supervision improves accuracy at matched data exposure.

  • Problem

    Under partial observation, correct prediction may require using earlier frames and intervening actions to infer hidden-state changes, while existing loss-scaling trends alone do not establish whether state predictions become more accurate.

    Source: §1 · p. 2

  • Method

    The paper evaluates action-conditioned video models on a solved-cube task with nine prescribed moves, a fixed three-face view, and simulator ground truth, and also tests sticker-state guidance.

    Source: §1 · p. 2, §3.4 · p. 8

  • Results

    After 1.5M training videos, the 1B AR model has lower MSE but 3.67% average frame accuracy, versus 41.00% for the 70M model; symbolic guidance raises the 20M model’s average frame accuracy from 31.1% to 67.3%.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings support direct state evaluation and suggest that learning representations of state changes can complement scaling, though the handcrafted sticker representation may not generalize to other cases.

    Source: §3.4 · p. 9, §5 · p. 9

    The benchmark always starts from a solved configuration with a fixed color convention specifying the full initial state, including hidden faces.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

We study whether scaling video generation enables models to reason about hidden information from the past frames, and at what computational cost. Our controlled benchmark requires predicting nine prescribed moves of an initially solved 2x2x2 Rubik's Cube from a fixed view of three faces. Correct predictions require inferring how actions change hidden states, and the simulator provides exact ground truth for evaluation. Models learn plausible cube geometry early, while correct sticker configurations require substantially more training. Although validation MSE follows approximate power-law scaling, lower MSE loss does not reliably indicate downstream reasoning capabilities. Smaller autoregressive models achieve higher state accuracy with limited compute, while larger models reach higher accuracy after more training. At roughly 0.1 PF-days, the 70M-parameter model correctly predicts the visible sticker configuration in 44.6% of post-action frames, compared with 0.3% for the 1B model, which reaches 83.7% at 3.14 PF-days. Symbolic state supervision raises the 20M model's frame accuracy from 31.1% to 67.3% at the same training-data budget, suggesting that learning representations of state changes can complement scaling.

1 INTRODUCTION

The paper tests whether video generators can infer hidden cube states from action history, a capability that visual plausibility and validation flow MSE alone may not establish. In a controlled Rubik’s Cube task, it finds that state accuracy depends on model size, compute, and explicit state supervision.

Motivation

  • The study asks whether lower validation flow MSE indicates better hidden-state reasoning and how compute grows as state predictions become more reliable.A visually convincing video can still contradict preceding events when sticker colors do not reflect prescribed moves.Source: §1 · p. 2
  • The 270M AR model reproduces cube geometry after 3.4×10−2 PF-days but first predicts all 12 visible stickers correctly in the example after 5.7 × 10−1 PF-days.The sticker prediction requires about 17 times as much compute as reproducing geometry.Source: Figure 2 · p. 2

Approach

  • The benchmark follows nine prescribed moves from a solved 2 × 2 × 2 cube, with a fixed camera showing three faces and exact simulator ground truth.The study trains bidirectional and autoregressive video diffusion transformers from 20M to 1B parameters and measures flow MSE, action following, and state accuracy.Source: §1 · p. 2

Main findings

  • After 1.5M training videos, the 1B AR model’s average frame accuracy is 3.67%, versus 41.00% for the 70M model, despite lower MSE.Lower validation flow MSE does not reliably indicate higher action or state accuracy across model sizes.Source: §1 · p. 2
  • For the 120M AR model, increasing training data from 5M to 8M videos raises average frame accuracy only from 48.11% to 49.33%.Larger models improve accuracy, but extrapolations of frame and full-trajectory error point to substantial further compute as prediction becomes more reliable.Source: §1 · p. 3
  • At the same training-data budget, state guidance raises the 20M AR-k = 1 model’s average frame accuracy from 31.1% to 67.3%.The result supports learning representations for state reasoning as a complement to scaling.Source: §1 · p. 3

2 A CONTROLLED STUDY OF STATE PREDICTION

The study uses a fixed-view cube task in which prescribed turns move stickers between visible and hidden faces. It conditions video generation on the action sequence and evaluates generated states against simulator-defined targets.

2.1 STATE PREDICTION UNDER PARTIAL OBSERVATION

  • The task begins with a solved cube and predicts observations after prescribed turns; the fixed view exposes 12 of 24 stickers, while turns move stickers between visible and hidden faces.The initial state and ordered actions determine later states, providing an unambiguous visual target.Source: §2.1 · p. 3

2.2 ACTION-CONDITIONED VIDEO PREDICTION

  • The video model conditions future latent-frame prediction on the initial image and ordered action sequence, using either joint bidirectional generation or autoregressive chunks conditioned on preceding visual history.During autoregressive generation, the model uses its own previous predictions, so earlier errors can affect later rollout frames.Source: §2.2 · pp. 3–4

2.3 EVALUATING STATE PREDICTION

  • At each post-turn stationary frame, decoded sticker colors are compared with the expected configuration; unreadable stickers count as incorrect.Sticker and frame scores distinguish partial state recovery from a completely correct visible configuration, while action following and latent velocity prediction are measured separately.Source: §2.3 · p. 4, Table 1 · p. 4

3 THE SCALING EXPERIMENTS

The experiments compare video-generation architectures and scaling under partial observation, then test whether symbolic state supervision improves prediction. They find that reliable state prediction requires more compute as errors become rare, while validation MSE is not a dependable cross-model measure of reasoning accuracy.

3.1 EXPERIMENTAL SETUP

  • After 1M videos, wide–shallow AR-k = 1 leads the tested configurations on action, sticker, and frame accuracy and is selected for scaling experiments.The architecture comparison evaluates three depth–width configurations with AR-k = 1, AR-k = 4, and bidirectional generation.Source: §3.1 · pp. 4–5, Table 2 · p. 5

3.2 UNDERSTANDING THE DIFFICULTY OF HIDDEN STATE PREDICTION

  • Approximately 2.8M-parameter vector Transformers learn accurate sticker-state prediction at much lower cost than video models, despite tracking stickers that leave view.The vector-space task evaluates sequences of 20 actions, rather than the nine-action video setting.Source: §3.2 · p. 5, Table 3 · p. 6
  • Making all six faces visible substantially improves frame accuracy; AR stays highly accurate throughout, while Bidir benefits mainly in early actions.The comparison changes both state visibility and the visual prediction target, so differences are not attributed solely to memory or rendering capacity.Source: §3.2 · p. 6, Figure 5 · p. 6

3.3 SCALING VIDEO GENERATOR FOR REASONING

  • Additional training yields diminishing accuracy gains at fixed model size, with later checkpoints showing increasingly similar profiles.The study compares increasing training exposure at fixed size with larger models at a common data budget.Source: §3.1 · p. 5, §3.3 · p. 6, Figure 6 · p. 7
  • Both fitting windows project substantially more compute to reach 98% accuracy, especially for complete AR trajectories, and the projections are illustrative rather than reliable training budgets.The compute demand rises as errors become rare, particularly when every state in a sequence must be correct.Source: §3.3 · p. 7, Figure 7 · p. 7
  • At 1.5M videos, the 1B AR model has lower MSE than the 70M model but worse action, sticker, and frame accuracy.MSE tracks broad progress within a model but does not reliably compare reasoning across model sizes.Source: §3.3 · pp. 7–8, Figure 8 · p. 8

3.4 SYMBOLIC GUIDANCE IMPROVES VIDEO PREDICTION

  • After training on the same 3M videos, the 20M Front12 model reaches 67.3% average frame accuracy at 0.0445 PF-days, comparable to 67.1% for a 550M video-only model at 1.33 PF-days.Both variants predict and supervise all 24 stickers; Front12 feeds back only the 12 visible-sticker distributions, and hidden-sticker feedback is not required.Source: §3.4 · p. 8

4 RELATED WORK

Related work frames video generation as both visual synthesis and prediction, and connects hidden-state tracking to state inference and action-conditioned representations. This study complements existing evaluation suites with a controlled scaling experiment and tests symbolic state guidance for video generation.

5 CONCLUSION

The conclusion reports that hidden-state reasoning was studied through controlled Rubik’s Cube video prediction while scaling autoregressive and bidirectional models. It emphasizes that loss and state accuracy can diverge, and that symbolic supervision improves accuracy while generalizable representations remain future work.

  • The study scales autoregressive and bidirectional Rubik’s Cube video predictors from 20M to 1B parameters to examine hidden-state reasoning.Source: §5 · p. 9
  • Lower validation MSE does not reliably indicate more accurate state prediction; smaller autoregressive models perform better with limited compute, while larger models reach higher accuracy after more training.Source: §5 · p. 9
  • Symbolic state supervision and predicted-state feedback improve accuracy at matched data exposure, motivating future work on generalizable representations beyond handcrafted states for one environment.Source: §5 · p. 9

A.1 RUBIK’S CUBE

The benchmark makes hidden sticker-state changes measurable under fixed, deterministic cube moves, while testing video prediction under partial and full observation. State guidance and expanded visibility provide controlled comparisons of video state prediction.

A.1 RUBIK’S CUBE

  • The fixed camera exposes 12 of 24 stickers, and each settled post-move frame is scored by visible-sticker accuracy and exact-frame accuracy.Exact-frame accuracy requires all 12 visible stickers to be correct; the evaluator reads fixed simulator regions rather than using a learned detector or visual judge.Source: §A.1 · p. 13
  • The frozen Wan VAE preserves every evaluated sticker configuration when reconstructing cached ground-truth latents.The comparison applies the same color decoder to original videos and their VAE reconstructions.Source: §A.1 · p. 13, Table 4 · p. 13

A.2 SHARED LANGUAGE AND ACTION TOKENIZATION

  • The frozen language encoder supplies action features, while the generator receives no parallel action-ID tensor, structured transition, or simulator state.Action-token embeddings are initialized from ordinary-language glosses and then frozen with the ModernBERT backbone.Source: §A.2 · p. 13

A.3 VANILLA VIDEO GENERATOR

  • The historical vanilla generator compares joint bidirectional prediction with autoregressive prediction in five chronological chunks of four latent frames.At inference, the autoregressive model feeds back generated chunks; the bidirectional model predicts all 20 target latent frames jointly.Source: §A.3 · p. 14

A.4 JOINT STATE AND VIDEO PREDICTION

  • The state-guided models predict sticker colors at action boundaries and use those predictions to condition video generation, without simulator-state feedback.Their loss combines flow loss with 0.1 times state cross-entropy; both variants use the same 3M videos as the video-only baseline.Source: §A.4 · p. 14

A.5 OBSERVATION CONTROL: THREE VERSUS SIX FACES

  • Mean exact-frame accuracy rises from 34.67% to 93.44% for AR when moving from three to six faces, while action-nine accuracy rises from 1% to 88%.For Bidir, mean accuracy rises from 27.89% to 42.89%, while action-nine accuracy remains 0% in both settings; visibility and the visual prediction target both change.Source: §A.5 · pp. 14–15

A.6 ACTION-FOLLOWING SANITY CHECK

The independent motion probe separates following a prompted turn from maintaining the correct sticker state. A categorical vector version of the task isolates sequence modeling while still requiring history to track stickers that leave and re-enter view.

A.6 ACTION-FOLLOWING SANITY CHECK

  • At action nine, AR and Bidir correctly follow the requested motion in 80.6% and 67.7% of episodes, respectively, while their exact-frame accuracies are 0.3% and 0.0%.Motion is measured separately from sticker colors using a frozen optical-flow probe trained on grayscale-permuted cube videos.Source: §A.6 · p. 15, Table 5 · p. 15

A.7 VECTOR-SPACE FORMULATION

  • The vector diagnostic represents the 12 visible stickers as a 12 × 6 one-hot array, so stickers returning to view require the model to retain intervening history.The state contains one six-way color variable for each visible sticker; out-of-view stickers are not represented until they return.Source: §A.7 · pp. 15–16
  • Vector models receive the initial state and prompt, but no video, structured actions, simulator transition operator, or intermediate oracle states.Autoregressive evaluation conditions each step on the model’s own preceding prediction, although training is teacher-forced.Source: §A.7 · p. 16
  • Teacher-forced AR evaluation reaches 99.17% final-state success, while the reported test metrics also include strict-rollout final-state and full-trajectory exact accuracy.The strict test evaluates all 12 visible sticker colors after move 20.Source: §A.7 · p. 16

B VIDEO DIT ARCHITECTURE AND TRAINING PROTOCOL

The study calibrates a depth–width family before measuring model–data scaling, using training FLOPs rather than parameter count alone as its primary resource variable. Hyperparameters are chosen through matched-data pilot comparisons.

B VIDEO DIT ARCHITECTURE AND TRAINING PROTOCOL

  • The two-stage design first calibrates depth–width allocation, then varies model size, data, and compute within the selected architecture family.Training FLOPs are the primary resource variable because AR history length and target length also affect attention compute.Source: B VIDEO DIT ARCHITECTURE AND TRAINING… · p. 17
  • Stage A selects wide–shallow AR-k = 1 on matched-data evidence, while Stage B varies seven approximately geometric parameter targets and unique training episodes.The Stage-A pilot does not claim iso-FLOP optimality; Stage B holds the selected family fixed.Source: B VIDEO DIT ARCHITECTURE AND TRAINING… · pp. 17–18
  • The scaling analysis evaluates 159 AR and 265 Bidir EMA checkpoints on the same 100 paired episodes, with each training video consumed once.A separate ground-truth-history diagnostic compares AR models at a common exposure of 5M videos.Source: B VIDEO DIT ARCHITECTURE AND TRAINING… · p. 18

B.1 HYPERPARAMETER SELECTION

  • Batch size 16 and peak learning rate 4 × 10−4 give the lowest validation loss in their respective pilot comparisons.The selected settings are used in both Stage A and Stage B; Table 8 reports the learning-rate and effective-batch-size searches.Source: §B.1 · p. 18, Table 8 · p. 18

C STAGE A: ARCHITECTURE CALIBRATION

In the matched-exposure Stage-A pilot, wide–shallow AR-k = 1 leads the nine configurations on action-following, sticker, and exact-frame accuracy. These results support carrying that configuration into Stage B, but do not establish an asymptotic performance limit.

C STAGE A: ARCHITECTURE CALIBRATION

  • The pilot compares three approximately 95M-parameter shapes across AR-k = 1, AR-k = 4, and Bidir, using the same 1M episodes and training settings.Evaluation uses EMA checkpoints on 100 held-out episodes with matched sampling-noise identities.Source: C STAGE A: ARCHITECTURE CALIBRATION · p. 19
  • Wide–shallow AR-k = 1 has the highest point estimates across all three metrics, with 90.78% action-following and 37.22% exact-frame accuracy.Figure 11 separates action-following, visible-sticker, and exact-frame accuracy along the action sequence.Source: C STAGE A: ARCHITECTURE CALIBRATION · p. 19, Table 10 · p. 19, Figure 11 · p. 20
  • The EMA endpoints support selecting wide–shallow AR-k = 1 for Stage B, but do not establish an asymptotic convergence floor or reasoning ceiling.Figure 12 shows rollouts for all nine setups on the first fixed development episode alongside aggregate measurements.Source: C STAGE A: ARCHITECTURE CALIBRATION · pp. 19–20, Figure 12 · p. 21

D FULL SCALING RESULTS

The full scaling analysis evaluates checkpoint performance across paired episodes, while a separate action-aligned diagnostic compares free-running predictions with predictions conditioned on ground-truth history.

D FULL SCALING RESULTS

  • 159 AR and 265 Bidir EMA checkpoints are evaluated on the same 100 paired episodes, with accuracy averaged across nine post-action boundaries.Full-trajectory accuracy requires all nine post-action frames in an episode to be correct.Source: D FULL SCALING RESULTS · p. 22

D.1 ACTION-ALIGNED GROUND-TRUTH HISTORY

  • The ground-truth-history diagnostic uses a separate 140-checkpoint AR snapshot and 16 midpoint steps per AR-k = 1 chunk.Each separately tested action receives ground-truth history only through the last causal VAE latent before the action, then predicts the bridge frames and action without further refresh.Source: D FULL SCALING RESULTS · p. 22, §D.1 · p. 22
  • Table 11 compares matched free-running and action-aligned ground-truth-history evaluations at 5M videos, reporting percentages on the same 100 episodes.The paired evaluations share model weights, episode identities, sampling noise, and the 16-step midpoint sampler.Source: Table 11 · p. 22, §D.1 · p. 22

E GENERATOR COMPUTE AND TRAINING DETAILS

The section specifies how model size and per-token compute are defined, then describes training-compute estimates and windowed fits to observed accuracy frontiers.

E GENERATOR COMPUTE AND TRAINING DETAILS

  • Each Transformer block combines visual attention, language cross-attention, and a SwiGLU layer, while scaling varies block count and hidden width.The empirical scaling analysis uses the full trainable parameter count rather than the block-only count.Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 23
  • Compute counts target and context tokens from cached video latents, language tokens, and permitted visual history, with one multiply–accumulate counted as two FLOPs.Table 12 reports generator parameters and forward compute per target visual patch token, omitting biases, normalization, and nonlinearities.Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 23, Table 12 · p. 23
  • Training FLOPs are estimated as three times forward cost, excluding optimizer and EMA updates, validation, and sampling.The frozen video autoencoder and language encoder are also excluded; autoregressive causal masks prevent access to future latent frames.Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 23
  • Observed accuracy frontiers retain record improvements by training compute, then fit separate power laws to their middle and final thirds.The frontier segments contain 16 AR average-frame, 6 AR full-trajectory, and 14 Bidir average-frame points.Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 23
  • The fitted 98% compute intersections illustrate fit-window sensitivity, not validated predictions of required compute.Bidir has no successful full trajectories in these evaluations, so that metric is not extrapolated; Table 13 reports the fitted coefficients and intersections.Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 23, Table 13 · p. 24
  • GPU-hour estimates convert PF-days using an assumed 50% model FLOPs utilization and a dense BF16 peak of 989 TFLOP/s.The conversion is 24C/(0.5 × 0.989).Source: E GENERATOR COMPUTE AND TRAINING DETAILS · p. 24

E.1 TRAINING PROCEDURE

Training uses rectified-flow velocity prediction with an exponential moving average of generator parameters; the autoregressive procedure partitions targets into chunks.

E.1 TRAINING PROCEDURE

  • Rectified-flow training interpolates noise and clean targets at sampled times, then minimizes minibatch latent-element MSE against the target velocity.After each optimizer update, the procedure updates a float32 EMA of trainable generator parameters.Source: §E.1 · p. 24
  • The bidirectional and autoregressive procedures update generator parameters using the rectified-flow MSE objective and maintain an EMA.The EMA is initialized from generator parameters and restored with the optimizer when training resumes; evaluations use saved EMA checkpoints.Source: §E.1 · p. 24
  • The autoregressive procedure partitions future latents into chunks, computes chunk losses in parallel, averages them, and updates parameters once.The EMA is updated after that optimizer step.Source: §E.1 · p. 24

Related papers

Loading 2609.36599v1…