PaperlayerView exact revision on arXiv

Source-linked AI summary

SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation

Wei-Cheng Tseng, Gashon Hussein, Yuzhu Dong, Allen Z. Ren, Lucy X. Shi, XuDong Wang, Sergey Levine, Zhaoshuo Li, Jinwei Gu, Florian Shkurti, Ming-Yu Liu, Quan Vuong

arXiv:2606.18610v3cs.ROcs.CV

TL;DR

Real-world evaluation of generalist robot manipulation policies is costly, and video-model evaluators face drift, cross-view inconsistency, and policy distribution shift. SC3-Eval adapts a pre-trained video foundation model with three forms of consistency; across seven real-world VLA policies, it outperforms prior video-model-based baselines and generalizes to a held-out task semantic.

  • Problem

    Real-world evaluation of generalist robot manipulation policies is expensive, slow, and difficult to scale, while video-world-model evaluators must address rollout drift, inconsistent camera views, and policy distribution shift.

    Source: §1 · pp. 1–2

  • Method

    SC3-Eval adapts a pre-trained video foundation model through forward-inverse dynamics, cross-view, and test-time consistency, jointly training modes and using inverse dynamics to terminate drifting rollouts.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    Across seven real-world VLA policies, SC3-Eval achieves a closed-loop Pearson correlation of 0.929, outperforms three strong prior video-model-based baselines, generalizes to a held-out task semantic, and reproduces per-trajectory failure modes.

    Source: §6 · p. 9

  • Takeaways & Limitations

    SC3-Eval supports fine-grained diagnostic comparison by reproducing policy failure modes as well as aggregate success-rate performance.

    Source: §1 · p. 2

    The evaluator is trained and validated on short-horizon manipulation, with a table-bussing episode lasting roughly 20 seconds; substantially longer horizons may introduce accumulated drift and visual degradation.

    Source: §5 · p. 8

Abstract

from arXiv · show

Evaluating generalist robot manipulation policies in the real world is expensive, slow, and difficult to scale. Action-conditioned video world models offer a scalable alternative by simulating policy rollouts. Autoregressive rollouts accumulate compounding errors, observations across multiple camera views must remain mutually consistent, and the evaluator must generalize to policies whose behaviors lie outside the training distribution. We address these challenges with SC3-Eval, a self-consistent video generation recipe that adapts a pre-trained video foundation model into an accurate policy evaluator by enforcing three complementary forms of consistency. First, forward-inverse dynamics consistency jointly trains the model to predict frames from actions and to recover actions from frames, anchoring generated rollouts to a physically plausible action manifold and counteracting the drift a forward-only model cannot penalize. Second, cross-view consistency trains the model to inpaint each camera view from the other, keeping the multi-camera observation coherent over long rollouts without any explicit memory mechanism. Third, test-time consistency reuses the inverse dynamics mode at inference as a per-action-chunk uncertainty signal that terminates rollouts whose generated frames drift away from the requested actions. We also demonstrate SC3-Eval rollouts reproduce the failure modes that policies exhibit in real-world rollouts, supporting fine-grained diagnostic comparison rather than aggregate ranking alone. Across seven real-world vision-language-action policies, SC3-Eval attains a closed-loop Pearson correlation of $0.929$ and MMRV of $0.119$, outperforming three strong prior video-model-based baselines, and generalizes to new tasks.

1 Introduction

Real-world evaluation of generalist robot manipulation policies is costly, while video-world-model evaluators face rollout drift, cross-view inconsistency, and policy distribution shift. SC3-Eval addresses these challenges with three forms of consistency and reports stronger evaluation performance than prior video-model-based baselines.

1 Introduction

  • Real-world policy evaluation is expensive, slow, and difficult to scale because it requires robot time, environment resets, and human supervision.The costs compound when evaluating policies across diverse tasks and environments.Source: §1 · p. 1
  • Action-conditioned video rollouts face compounding autoregressive drift, inconsistent multi-camera observations, and generalization to behaviors outside the evaluator’s training distribution.These issues can make simulated policy evaluation unreliable or visually incoherent.Source: §1 · pp. 1–2
  • SC3-Eval jointly enforces forward-inverse dynamics, cross-view, and test-time consistency to adapt a pre-trained video foundation model into a closed-loop policy evaluator.Inverse dynamics anchors generated frames to actions, cross-view inpainting keeps camera observations coherent, and test-time disagreement terminates drifting rollouts.Source: §1 · p. 2
  • Across seven VLA policy checkpoints, SC3-Eval attains a closed-loop Pearson correlation of 0.929 and MMRV of 0.119, outperforming three strong video-model-based baselines.It generalizes to a held-out task variant and reproduces specific real-world policy failure modes.Source: §1 · p. 2

2 Related Work

Prior work offers physics simulation, real-to-sim reconstruction, and video-model-based policy evaluation, each with distinct requirements. SC3-Eval extends video-model evaluators with explicit multi-view supervision and a single-model forward-inverse uncertainty signal for terminating unreliable evaluation rollouts.

Real-world policy evaluation

  • Video-model evaluators avoid asset construction and physics tuning, while physics simulators require engineered assets and dynamics tuning and real-to-sim approaches require per-scene reconstruction.The paper extends the video-model-based family.Source: §2 · p. 3

Action-conditioned video world models

  • Unlike prior multi-view models that concatenate views and rely on spatial self-attention, SC3-Eval explicitly trains the model to inpaint missing views from the remaining view.The adopted unified forward-inverse architecture was originally proposed for action generation [7] [8].Source: §2 · p. 3

Distribution shift and uncertainty in model-based RL

  • SC3-Eval derives off-manifold uncertainty from forward-inverse consistency in one model and terminates unreliable evaluation rollouts before further drift contaminates their scores.Unlike ensemble-based signals, this approach removes extra training cost and is applied during policy evaluation rather than offline RL training.Source: §2 · p. 3

3 SC3-Eval

SC3-Eval adapts a pre-trained unified dynamics model for closed-loop policy evaluation through joint consistency training and test-time rollout checks. The section specifies the evaluation objective, shared-model training modes, and inference schedule that uses inverse dynamics to detect unreliable chunks.

3.1 Preliminaries

  • The evaluation goal is a world simulator whose policy-success estimates correlate with real-world success and preserve pairwise policy rankings.Pearson correlation captures linear agreement, while MMRV captures consistency of pairwise rankings.Source: §3.1 · pp. 3–4
  • A shared-token unified dynamics model can predict frames from actions, recover actions from frames, or fill unobserved frames according to which tokens are exposed.SC3-Eval adopts this architecture as its backbone.Source: §3.1 · p. 4

3.2 Self-Consistent Training

  • Forward-inverse training shares parameters so generated frames must support recovery of commanded actions, implicitly anchoring rollouts to a physically plausible action manifold and counteracting drift.Cross-view inpainting trains the model to predict held-out camera views from the remaining views, enforcing multi-view consistency without an explicit memory mechanism.Source: §3.2 · pp. 4–5
  • Three objectives are jointly trained by randomly assigning each instance a mode, applying its clean/noisy token pattern, and optimizing a flow-matching loss on shared transformer parameters.Mode coefficients are realized through relative sampling frequencies rather than an explicit weighted sum at every step.Source: §3.2 · p. 5

3.3 Inference with Test-Time Consistency

  • At inference, the policy proposes a chunk, the forward model renders it, and only the first l frames are appended before replanning; training uses a longer horizon l′ > l to preserve the backbone prior and provide richer motion supervision.The extra l′ − l frames are discarded at deployment.Source: §3.3 · p. 5
  • Inverse dynamics recovers actions from generated frames to compute a per-chunk consistency error, and rollout terminates when that error exceeds a held-out threshold τ.The signal is an empirical reliability indicator, not a calibrated error probability, because inverse dynamics sees ground-truth frames in training but generated frames at deployment.Source: §3.3 · p. 5

4 Experiments

Experiments compare SC3-Eval with three video-model-based evaluators on table bussing and reverse table bussing, assessing policy-performance agreement, failure-mode reproduction, and design ablations. The results show strong checkpoint-ranking performance and examine how closed-loop evaluation behaves across task splits.

4.1 Experimental Setup

  • The evaluation compares seven VLA checkpoints on trained table bussing and reverse table bussing, an out-of-distribution task with a changed object-to-destination mapping.The protocol includes matched real-world, offline open-loop, and online closed-loop trajectories, observed through three synchronized camera views and scored on language following, object lifting, and object placing.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Evaluating the Policy under SC3-Eval

  • SC3-Eval achieves the best MMRV in all four evaluation settings and the best Pearson r in three of four, while evaluators generally degrade on the out-of-distribution split, especially online.The settings combine in-distribution and out-of-distribution tasks with offline and online evaluation.Source: §4.2 · pp. 6–7, Figure 4 · p. 6, Table 1 · p. 7
  • 0.984 online Pearson r exceeds 0.959 offline on the in-distribution split, while the remaining degradation is concentrated out of distribution.The paper attributes online compensation to policy reactions to generated frames; closed-loop correction cannot fully recover when the world model is itself off-distribution.Source: §4.2 · p. 7

4.3 Reproducing Policy Failure Modes

  • Under closed-loop evaluation, SC3-Eval has the highest average outcome-reproduction rate and leads in all four failure categories.The measure is the fraction of matched real-world trajectories whose online rollouts fall in the same outcome category.Source: §4.3 · pp. 7–8, Figure 6 · p. 7

4.4 Ablation Study

  • Removing inverse dynamics, cross-view inpainting, early termination, or prediction-execution horizon decoupling degrades closed-loop metrics.The reported effects include weaker out-of-distribution performance without inverse dynamics or early termination, reduced wrist-view re-entry consistency without cross-view training, and poorer dynamics fitting at the shorter execution horizon.Source: §4.4 · p. 8, Table 2 · p. 8

5 Limitations

SC3-Eval is limited by slow online inference and evaluation on short-horizon manipulation. Longer rollouts are expected to face accumulated drift and visual degradation.

5 Limitations

  • At 2.3 seconds per chunk on one GB200 GPU, online evaluation is several orders slower than faster-than-real-time physics simulators, limiting routine checkpoint selection.Diffusion-sampler acceleration and caching intermediate representations are proposed as follow-ups.Source: §5 · p. 8
  • Evaluation covers short-horizon manipulation, with table-bussing episodes lasting roughly 20 seconds; longer horizons are expected to expose accumulated drift and visual degradation.The authors attribute these expected failure modes to a corrective signal that is only local and the backbone’s temporal-coherence prior.Source: §5 · p. 8

6 Conclusion

SC3-Eval repurposes a pre-trained unified dynamics model as a real-world policy evaluator and reports strong policy-level agreement, held-out task-semantic generalization, and per-trajectory failure-mode reproduction. The appendix provides complementary dataset, training, and ablation details.

6 Conclusion

  • Across seven real-world VLA policies, SC3-Eval achieves closed-loop Pearson correlation 0.929 and MMRV 0.119, outperforming three prior video-model-based baselines.It also generalizes to a held-out task semantic and reproduces per-trajectory failure modes rather than only aggregate success rates.Source: §6 · p. 9

A Dataset and Evaluation Details

  • The appendices provide dataset, training, and ablation details that complement the main paper’s Section 4.Full rollout videos and additional qualitative comparisons are hosted on the project website.Source: A Dataset and Evaluation Details · p. 12

A.1 Training Dataset

SC3-Eval is trained on a large real-world table-bussing dataset with synchronized multi-camera views, varied objects, and normalized end-effector actions.

A.1 Training Dataset

  • The training dataset contains 381 hours of robot trajectories from one physical scene, with 12 object categories and three synchronized camera views recorded at 20 Hz.The views comprise two third-person cameras and one wrist-mounted camera; streams are also subsampled to 10 Hz for a multi-FPS training mix.Source: §A.1 · p. 12
  • Trajectories cover table bussing, moving utensils to a utensil bin and trash items to a trash can, but exclude reverse table bussing.Reverse table bussing swaps the two destinations and forms the out-of-distribution evaluation split.Source: §A.1 · p. 12
  • Actions are 7-dimensional delta end-effector poses encoding translation, axis-angle rotation, and gripper width, with each dimension normalized to unit variance.Normalization statistics are computed over the full training corpus.Source: §A.1 · p. 12

A.2 Evaluation Setup

The evaluation compares matched physical-robot and world-model rollouts across in-distribution and reverse-bussing policies, using separately scored criteria and policy-level metrics. The appendix also describes the model initialization and shared action-conditioning pathway.

A.2 Evaluation Setup

  • Seven π0.5 checkpoints are evaluated across four table-bussing and three reverse-bussing policies, with 36–37 matched initial conditions per checkpoint run both physically and in the evaluator.Reverse bussing preserves the workspace, objects, and language template while swapping object-to-destination mappings; all rollout modes have a 20-second limit.Source: §A.2 · p. 12
  • Each checkpoint predicts 25 actions at 10 Hz, with the first 16 executed before replanning; Table 3 reports physical-robot success rates by criterion.The four in-distribution checkpoints perform table bussing and the three out-of-distribution checkpoints perform reverse table bussing.Source: §A.2 · p. 12, Table 3 · p. 12

A.3 Success Criteria and Annotation Protocol

  • Language following, object lifting, and object placing are scored separately per object, then averaged across objects and 36–37 trajectories to yield checkpoint-level success rates.Each checkpoint-criterion pair is one correlation data point, yielding 21 per evaluation mode.Source: §A.3 · p. 13
  • Pearson correlation measures linear agreement and absolute calibration, while MMRV measures pairwise rank consistency; online closed-loop rollouts have no meaningful frame-level reference metric.Offline open-loop evaluation additionally reports rollout PSNR because the model follows the real-world action sequence.Source: §A.3 · p. 13

B.1 Network Architecture

  • Annotators see uniformly rendered, randomized trajectories with source labels removed and assign failed trajectories to the first violated criterion: language, lifting, or placing.SC3-Eval initializes from pre-trained Cosmos3-Nano weights and injects shared action embeddings into each Diffusion Transformer block’s time-embedding pathway for both dynamics objectives.Source: §A.3 · p. 13, §B.1 · p. 13

B.2 Hardware and Compute

SC3-Eval combines multi-view forward dynamics, cross-view inpainting, and inverse dynamics during training, then uses chunked predictions and an uncertainty threshold at inference. The appendix reports training resources and explains how optimization and augmentation choices support this pipeline.

B.2 Hardware and Compute

  • Training used 32 NVIDIA GB200 GPUs for approximately 2.2 days, and closed-loop inference took 2.3 seconds per 24-frame chunk on one GPU.The appendix also reports 24,000 training steps with an effective batch size of 512 trajectories per step.Source: §B.2 · p. 13, §B.3 · p. 13

B.3 Optimization

  • Each training instance sampled one mode: multi-view forward dynamics with probability 0.8, cross-view inpainting with 0.1, or inverse dynamics with 0.1.Each instance contributed gradients to a single mode rather than to an explicit weighted sum across modes.Source: §B.3 · p. 14

B.4 Training Augmentations

  • With probability 0.5, training replaced a policy command with a pseudo-action computed from consecutive real-world end-effector poses, providing marginal robustness gains across slightly different action manifolds.The pseudo-action and policy-command distributions occupy slightly different action manifolds.Source: §B.4 · p. 14
  • Training jointly sampled 10 and 20 FPS, exposing 24-frame chunks to complementary temporal statistics without explicitly conditioning the model on frame rate.The respective action horizons were 2.4 and 1.2 seconds; 20 FPS emphasizes short-range physics and contact, while 10 FPS spans more global movement.Source: §B.4 · p. 14

B.5 Inference Procedure

  • At inference, the model predicts 24 frames but retains only the first 16, matching the policy’s receding-horizon execution while avoiding a shorter-than-pretraining prediction horizon.The policy proposes 25 actions, of which the first 24 are passed to the world model for each chunk.Source: §B.5 · p. 14
  • The early-termination threshold was tuned to 0.02 on 30 held-out trajectories, terminating approximately 4.88% of rollouts; performance was stable from 0.02 to 0.05 but degraded at 0.01.The appendix describes Tab. 2 as evaluating design choices with closed-loop policy-level correlations and Tab. 4 as reporting offline rollout PSNR by method and ablation.Source: §B.5 · p. 14, C Additional Ablation Study · p. 14

C.1 Training Modes

The appendix examines how inverse dynamics and cross-view inpainting affect offline rollout quality, and whether prediction-execution horizon decoupling also benefits open-loop generation. It reports qualitative and PSNR evidence for the training modes and a consistent PSNR drop without horizon decoupling.

C.1 Training Modes

  • Without inverse dynamics, offline PSNR drops on both InD and OOD splits, with a larger 1.05-PSNR OOD drop than the 0.47-PSNR InD drop.The accompanying example tracks the ground truth for one second before drifting to another scene, while the inverse-grounded variant follows the bottle pick-and-place motion to the end.Source: §C.1 · p. 15, Figure 7 · p. 15, Table 4 · p. 15
  • Without cross-view inpainting, the wrist view hallucinates the workspace during camera re-entry and fails to recover; with it, the correct workspace state returns under occlusion.The comparison samples four timesteps across a re-entry segment, with the wrist view above two third-person views.Source: §C.1 · p. 15, Figure 8 · p. 16

C.2 Prediction-Execution Horizon Decoupling

  • The early-termination example stops at chunk 10 of 14, the first chunk whose Uchunk exceeds the threshold τ = 0.02, before visible drift compounds.Table 5 covers InD/OOD results across termination-threshold settings; Table 4 covers offline PSNR by method and ablation.Source: Table 4 · p. 15, Figure 9 · p. 16, Table 5 · p. 17
  • Setting the prediction and execution horizons equal shortens each video chunk below the world model’s pretraining horizon and consistently lowers offline PSNR as well as closed-loop policy-level correlation.SC3-Eval introduces inverse dynamics and cross-view inpainting as grounding modes alongside multi-view forward dynamics.Source: §C.1 · p. 14, §C.2 · p. 16

C.3 Uncertainty-Driven Early Termination

The appendix finds that uncertainty-driven early termination benefits OOD evaluation consistently, while aggressive termination can hurt InD results. It also reports that Uchunk improves VLM-based policy scoring and that SC3-Eval retains the highest online outcome reproduction rate across categories.

C.3 Uncertainty-Driven Early Termination

  • Across every tested threshold, early termination improves OOD results: OOD r rises from 0.853 without termination to 0.902 at τ3 = 0.01, while OOD MMRV falls from 0.197 to 0.159.On InD, both metrics are best at τ2 = 0.02 and degrade with more aggressive termination.Source: §C.3 · p. 17
  • At τ3 = 0.01, the 10.22% termination rate coincides with InD r falling from 0.984 to 0.941 and InD MMRV rising from 0.022 to 0.101, so τ2 = 0.02 is the adopted operating point.The authors interpret the additional rollouts terminated at τ3 as predominantly rollouts the model would have continued correctly.Source: §C.3 · p. 17
  • Adding Uchunk to rendered frames raises VLM-derived policy-level success from 0.712 to 0.761, closer to the human-annotated result.The VLM tested was Gemini 3.1-pro.Source: §C.3 · p. 17

C.4 Per-Category Outcome Reproduction

  • In closed-loop outcome reproduction, SC3-Eval has the highest reproduction rate in every category, while baselines drop sharply compared with the easier offline setting.Table 6 presents offline and online fractions per category; Fig. 6 visualizes the per-category rates as a bar plot.Source: §C.4 · pp. 17–18

Related papers

Loading 2606.18610v3…