PaperlayerView exact revision on arXiv

Source-linked AI summary

World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry

Yuejiang Liu, Fan Feng, Lingjing Kong, Weifeng Lu, Jinzhou Tang, Kun Zhang, Kevin Murphy, Chelsea Finn, Yilun Du

arXiv:2604.01985v2cs.LGcs.AIcs.RO

TL;DR

World models need reliable predictions across broad action distributions, but action-labeled robot data are costly and existing verification is weak in under-explored regimes. WAV decomposes verification into state plausibility and action reachability using action-free videos, sparse inverse dynamics, and cycle-consistent forward rollouts. Across nine tasks, it reports 2× greater sample efficiency and more than 22% higher downstream policy performance.

  • Problem

    World models must predict suboptimal and exploratory actions, yet diverse action-labeled robot interactions are costly to collect and verification is unreliable in under-explored regimes.

  • Method

    WAV decomposes prediction verification into state plausibility and action reachability, exploiting abundant action-free data and lower-dimensional action-relevant features.

  • Results

    2× greater sample efficiency in exploration and 22% improved downstream policy performance were reported across MiniGrid, RoboMimic, and ManiSkill.

  • Takeaways & Limitations

    WAV uses asymmetric forward-inverse verification to gather informative interactions for self-improving world models.

  • Takeaways & Limitations

    The framework is studied with a small action-labeled robot dataset, a large action-free video dataset, and theory assuming a low-dimensional action-relevant state slice.

Abstract

from arXiv · show

General-purpose world models promise scalable policy evaluation, optimization, and planning, yet achieving the required level of robustness remains challenging. Unlike policy learning which primarily focuses on optimal actions, a world model needs to be reliable over a vast space of suboptimal actions, which are often underrepresented in action-labeled robot interactions. To address this challenge, we propose World Action Verifier (WAV), a framework that enables world models to identify their own prediction errors and self-improve. The key idea is to decompose action-conditioned state prediction into two independently verifiable factors: state plausibility and action reachability. We show that verifying these factors is significantly more tractable than direct forward prediction due to two underlying asymmetries: the broader availability of action-free data and the lower dimensionality of action-relevant features. Leveraging these asymmetries, we augment a world model with (i) a diverse subgoal generator obtained from video corpora and (ii) a sparse inverse model that infers actions from a subset of state features. By enforcing cycle consistency among proposed subgoals, inferred actions, and forward rollouts, WAV provides an effective verification mechanism in under-explored regimes, where existing methods often fail. Across nine tasks spanning MiniGrid, RoboMimic, and ManiSkill, our method achieves 2x higher sample efficiency while improving downstream policy performance by over 22%.

1 Introduction

General-purpose world models must predict the effects of broad action distributions, but diverse action-labeled robot data are costly to collect and existing verification is unreliable in under-explored regimes. WAV addresses this challenge by decomposing prediction verification into state plausibility and action reachability, using asymmetric forward-inverse cycles to self-improve.

  • Motivation: World models support policy evaluation, optimization, and planning but must remain reliable beyond optimal actions, including suboptimal and exploratory actions.These broader action distributions are often underrepresented in action-labeled robot interactions.
  • Motivation: Collecting robot interactions across diverse actions is slow, expensive, and sometimes unsafe, making interaction selection a central data-efficiency challenge.The paper frames deciding which interactions to collect as a pressing problem under limited robot-data budgets.
  • Verification gap: Existing verification is more reliable in well-explored regions than in under-explored regions, where accurate error estimates are most needed.This creates a tension because informative exploratory interactions are precisely those with the least prior verification information.
  • WAV: WAV decomposes action-conditioned predictions into state plausibility and action reachability, enabling each factor to be verified separately.State plausibility concerns visual realism, while action reachability concerns physical achievability under the given action.
  • WAV: WAV exploits abundant action-free video data for state verification and lower-dimensional action-relevant features for reachability verification.These asymmetries make verification more tractable than directly verifying the full forward prediction.
  • Results: 2× higher sample efficiency and over 22% better downstream policy performance were achieved across nine MiniGrid, RoboMimic, and ManiSkill tasks.The reported improvements accompany a cycle involving subgoal proposals, inferred actions, and forward-rollout consistency checks.

2 Method: World Action Verifier for Self-Improving World Models

WAV formulates world-model verification as ranking candidate interactions by likely prediction error, then decomposes verification into state plausibility and action reachability. Its goal-oriented cycle proposes plausible subgoals, infers reaching actions, compares forward rollouts with those subgoals, and collects the most discrepant transition for iterative updates.

  • Problem setup: The semi-supervised setting combines a small action-labeled robot dataset with a larger action-free video dataset spanning broader state transitions.The missing action labels in videos create an action-following challenge for world models.
  • Verification objective: WAV targets informative interactions by prioritizing transitions likely to produce large prediction errors rather than transitions the model already predicts accurately.The verifier need not be perfectly calibrated; preserving the relative ranking of candidate-action errors is sufficient for exploration.
  • Two verification factors: Forward prediction is decomposed into state plausibility and action reachability: predicted states must be plausible futures and reachable under the specified action.The decomposition is motivated by a factorization separating the transition prior from inverse action prediction.
  • State verification: A state verifier uses a transition prior trained with action-free videos to sample diverse plausible future subgoals, including longer-horizon states reached after action chunks.This targets visual-plausibility errors such as blurry or inconsistent rollouts after limited interaction-data post-training.
  • Action verification: An inverse dynamics model checks reachability by inferring actions from the current state and candidate future state, using a complementary verification role to the subgoal generator.The subgoal generator checks plausibility, while the inverse model checks whether the future can be reached through an inferred action.
  • WAV-guided exploration: By verifying multiple candidate rollouts before acting, WAV trades scalable computation for fewer unnecessary real-world interactions and improved data efficiency.This ordering places the least reliable forward model at the final verification step.
  • WAV-guided exploration: The goal-oriented cycle samples subgoals, infers actions, rolls them forward, and executes the action whose rollout has the largest discrepancy from its proposed subgoal.The resulting transition is added to the action-labeled dataset, and both forward and inverse models are iteratively updated.

3 Theory: When Does Inverse Verification Outperform Forward Prediction?

The theory analyzes when sparse inverse verification is easier than dense forward prediction in a linear-Gaussian setting. The advantage grows with lower action-relevant dimensionality, lower action-recovery ambiguity relative to forward stochasticity, and limited labeled sample sizes.

  • Setup: The analysis compares a dense forward model using full state-action inputs with a sparse inverse model using a low-dimensional action-relevant state slice.The inverse model is trained on inputs of dimension 2d_z, while the forward model uses d_s+d_a inputs.
  • Conditions: Inverse verification is expected to help most when the action-relevant feature dimension is much smaller than the full state dimension and labeled data are limited.The result is stated under the linear-Gaussian assumptions and sample-size conditions n > d_s+d_a+1 and n > 2d_z+1.
  • Comparison: The theoretical comparison evaluates both models in state space and scales inverse action errors by λ, the worst-case amplification from action error to state error.This places forward and inverse verification errors in common units.
  • Interpretation: The predicted advantage of inverse verification factorizes into dimensionality, stochasticity, and sample-size effects.Forward prediction estimates a higher-dimensional map and suffers environment noise, whereas inverse verification depends on action-recovery ambiguity scaled by λ.

4 Experiments

The experiments test WAV’s verification robustness, self-improvement, downstream policy benefits, and OOD adaptation across MiniGrid, RoboMimic, and ManiSkill. Results show stronger inverse verification, improved world-model learning, and better adaptation under difficult shifts.

  • Experimental Questions: WAV is evaluated through five research questions covering inverse verification, sparse-IDM generalization, self-improvement, downstream policy learning, and OOD adaptation.The evaluation spans MiniGrid, RoboMimic, and ManiSkill tasks.
  • Robustness of World Action Verification: Across controlled shifts, sparse inverse dynamics consistently outperform action-conditioned world models, especially with limited data, increasing state complexity, or observation noise.The performance gap widens in low-data regimes, while inverse models remain more stable as state dimensionality and stochasticity increase.
  • Robustness of World Action Verification: Sparse IDMs maintain stronger out-of-distribution performance than vanilla IDMs on toggle and swap tasks when data is limited.The result supports enforcing sparsity for more robust action inference.
  • Effectiveness of World Model Learning: WAV and Oracle achieve the best prediction error, while WAV also attains the highest Action Following Score among exploration methods.The method prioritizes transitions with disagreement between video-prior and world-model predictions, favoring sparse interaction actions.
  • World-Model Learning on Manipulation Tasks: WAV consistently outperforms baselines on 32-frame prediction error, with especially large gains in the low-data regime across RoboMimic and ManiSkill.Sparse inverse models also outperform dense variants on object-manipulation tasks, and improved prediction supports stronger imagination-based policy refinement.
  • OOD Adaptation and Downstream Policy Learning: WAV achieves stronger OOD adaptation under visual and object/interaction shifts, with approximately 22% improvement on novel environments with new objects and interactions.The gains include lower prediction error and stronger downstream reward using the same 200 target trajectories.

5 Related Work

Related work positions WAV at the intersection of world-model exploration and world action models. Existing approaches use uncertainty, progress, prediction error, goal discovery, or action-free video to improve modeling and planning.

  • Exploration for World Models: World-model exploration selects interactions using coverage, uncertainty, disagreement, learning progress, prediction error, curiosity, or goal discovery.These methods estimate which interactions may improve a learned dynamics model.
  • World Action Models: World action models combine action-free internet video with action-labeled robot data for unified prediction, visual planning, or foresight-based action prediction.This line of work improves representations or planning while preserving downstream policy interfaces.

6 Conclusion

WAV exploits an asymmetry between forward and inverse dynamics: inferring an action for a plausible transition can be easier than predicting the transition’s full outcome. Across three benchmarks, this supports more efficient exploration and stronger downstream policy performance.

  • Conclusion: 2× greater sample efficiency and 22% higher downstream policy performance are reported across MiniGrid, RoboMimic, and ManiSkill.The method uses cycle consistency among a diverse subgoal generator, sparse inverse model, and forward world model.

A.1 Robustness of World Action Verification

The robustness evaluation compares verification scores with oracle difficulty rankings on held-out samples. WAV’s scores more faithfully track the oracle ranking, supporting reliable data selection.

  • Robustness Evaluation: WAV verification scores more faithfully reflect the true oracle difficulty ranking of held-out samples.The evaluation uses Spearman’s rank correlation over 100 samples withheld from world-model training.

A.2 Full Results on World Model Prediction

With 200 training samples, Ours achieves the lowest world-model prediction error among non-Oracle methods across six tasks, outperforming non-IDM baselines and Vanilla IDM.

  • Ours achieves the lowest prediction error among all non-Oracle methods across six tasks under the 200-sample budget.The result indicates that inverse-verification-based data selection provides a stronger adaptation signal than uncertainty-, progress-, or novelty-based acquisition.
  • Ours significantly outperforms non-IDM baselines across all tasks under a Wilcoxon signed-rank test at the 5% level.
  • Ours further reduces prediction error compared with Vanilla IDM, showing that verification also selects more informative transitions for adaptation.

A.3 Full Results on Policy Learning

Policies refined with WAV-based world models achieve higher rewards across RoboMimic and ManiSkill than policies using baseline world models, approaching the oracle selection model.

  • Policies refined with WAV-based world models achieve higher rewards than those refined with baseline world models across RoboMimic and ManiSkill tasks.The same base diffusion policy is refined through imagination-based search with a fixed budget of 1,000 trajectories.
  • WAV-based policies are second only to the oracle model that uses privileged ground-truth actions for sample selection.
  • Figure 7 reports downstream policy performance on RoboMimic and ManiSkill, with error bars denoting standard error over 3 seeds.
  • The largest downstream gains occur on Can, Square, and PokeCube, which have ambiguous or contact-rich dynamics.
  • Simpler tasks such as Lift show smaller gaps across methods, indicating that policy gains vary with interaction complexity.

C Additional Theoretical Analysis

Theoretical analysis identifies conditions under which sparse inverse verification generalizes beyond labeled-transition support and makes forward–inverse mismatch reflect world-model error rather than action ambiguity.

  • The learned mask selects a sparse action-relevant latent block containing agent-centric variables such as proprioception or end-effector motion.
  • The generation–verification gap allows full state–action pairs to be out-of-support while restricted action-relevant pairs remain on-support.
  • Under the stated structural conditions, a seed-trained inverse model can recover actions from the restricted subset even when the full scene transition is novel.
  • The forward–inverse mismatch therefore localizes forward-model error rather than action-label ambiguity, making each exploration round add trustworthy coverage.
  • WAV requires three inference passes, making the current implementation more computationally expensive than prior exploration methods.

E.2.2 Dataset Composition.

The evaluation combines random-play datasets with varying object counts and stochasticity, compositional OOD tests, multiple prediction metrics, ranking measures, and sparse-versus-vanilla inverse-model comparisons.

  • Dataset Composition: Random-play datasets vary object counts from 6 to 14 to study state complexity, while noisy floor tiles range from 0 to 4 to study environmental stochasticity.
  • Dataset Composition: The exploration data comprise 28,000 unlabeled pre-training transitions, 28,273 acquisition transitions, and a 10,368-transition action-balanced test set.
  • Dataset Composition: Compositional OOD evaluation holds out action–object–color combinations, testing generalization beyond observed training compositions.
  • Evaluation Metrics: Dynamics Accuracy masks invariant background regions and evaluates only temporally changing visual and agent attributes.
  • Evaluation Metrics: Exploration uses held-out next-state prediction loss for sensitivity to training stability, whereas Dynamics Accuracy is used for final predictive performance.
  • Evaluation Metrics: Ranking quality is measured with Spearman’s ρ and Kendall’s τ, where higher values indicate stronger agreement with Oracle rankings.
  • Evaluation Metrics: The Action Following Score measures how well predicted future states preserve distinctions between different actions.
  • Inverse Dynamics Models: Sparse IDM applies a learnable feature mask to filter irrelevant information, whereas Vanilla IDM uses the entire observation frame and proprioceptive state.

E.2.5 Exploration Methods in MiniGrid

The section compares exploration and verification strategies, showing that WAV more faithfully identifies difficult transitions and improves long-horizon world-model predictions while relying on low-dimensional action-relevant structure. Its guarantees are strongest when the verifier remains on-support and action effects are distinguishable.

  • Exploration outcomes: Actively selected data better captures interaction-induced changes than Random, which over-samples frequent movement actions.The clearest differences occur for interaction-centric actions such as Toggle and Swap rather than simple motion-dominated transitions.
  • Verification quality: WAV’s verification score aligns monotonically with ground-truth error, unlike Uncertainty and Progress, which scatter and frequently misrank samples.WAV achieves the highest Spearman and Kendall correlations, indicating that its scores capture transition difficulty for data selection.
  • Inverse dynamics: Sparse inverse models mitigate long-horizon rollout errors, producing more stable dynamics and preserving fine-grained predicted details.They improve on vanilla IDM rollouts that retain discrepancies such as misaligned gripper orientations and inaccurate object occlusions.
  • Self-improvement: WAV’s verification mechanism can recover missing action labels for out-of-support transitions, expanding the action-labeled support under the stated identifiability assumptions.The guarantee depends on a verification subset that remains on-support and is sufficient to recover the action.
  • Generation–verification gap: WAV is most attractive when the verifying subset is much lower-dimensional than the full state and remains fixed as scene complexity grows.Its benefits degrade when different actions produce indistinguishable subset transitions, causing ambiguous recovered actions and degraded self-improvement.

F.2 Detailed Derivation for Sec. 3

The appendix analyzes WAV’s forward–inverse asymmetry in a stylized linear-Gaussian setting, comparing dense forward prediction with sparse inverse verification. The resulting error-ratio interpretation attributes the advantage to dimensionality, stochasticity, and sample-size factors, while explicitly limiting the analysis to an idealized regime.

  • Setup and motivation: WAV’s analysis isolates when sparse inverse verification can outperform dense forward prediction: action information must be preserved in a low-dimensional state slice.The appendix assumes a linear noisy dynamical system and an action-relevant slice from which actions can be recovered up to irreducible ambiguity.
  • Exact risk comparison: The exact linear-Gaussian derivation applies under n > ds + da + 1 and n > 2dz + 1, yielding a forward–inverse error ratio.The proof applies the OLS excess-risk result separately to the forward and inverse regressions, then maps inverse error back to state space.
  • Model comparison: The forward regressor uses a dense feature vector of dimension ds + da, whereas the inverse regressor uses a 2dz-dimensional action-relevant representation.Both models are compared in state space after fitting ordinary least squares on n i.i.d. labeled transitions.
  • Interpretation of the bound: The error ratio factorizes into dimensionality, stochasticity, and sample-size advantages favoring inverse verification in the analyzed regime.Forward prediction faces higher input dimensionality and environment noise, while inverse verification depends on action ambiguity in the selected slice and can be more stable with limited samples.
  • Scope and limitations: The analysis does not claim that real robotic dynamics are globally linear or Gaussian; it formalizes a statistical intuition rather than a universal guarantee.The paper states that deployment in safety-sensitive robotics requires task-specific validation, monitoring, and human oversight when verifier assumptions may fail under domain shift.
Loading 2604.01985v2…