PaperlayerView exact revision on arXiv

Source-linked AI summary

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

Gaytri Jena, Kapil Wanaskar, Vinija Jain, Aman Chadha, Vasu Sharma, Amitava Das

arXiv:2609.29669v1cs.ROcs.AI

TL;DR

Existing benchmarks do not establish whether world modelling gives direct policies a measurable closed-loop advantage, or for which capabilities. This survey maps 160 benchmarks and their evaluation modes, capabilities, and model families. It finds that explicit VLA–world-model contrasts are rare, and argues that benchmarks must be built to ask the question.

  • Problem

    Existing evaluations separate open-loop prediction or generation from closed-loop task success, leaving unanswered whether world modelling gives a direct policy a measurable closed-loop advantage.

    Source: §1 · p. 2

  • Method

    The survey treats the benchmark as its unit, maps the evaluation landscape, and proposes an evaluation loop and four advantage-aware metrics anchored on named testbeds.

    Source: §1 · pp. 2–3, §2.1 · p. 4

  • Results

    138 of 160 benchmarks are model-agnostic, and only 11 (7%) build an explicit VLA-versus-world-model contrast; counterfactual capability is almost entirely unmeasured, and four benchmarks turn prediction into executed action.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that answering whether prediction helps requires benchmarks built to compare it with direct policies in closed loop.

    Source: §1 · pp. 2–3

    The survey maps benchmarks rather than models, so it makes no claims about which world models or VLA policies are better.

    Source: §6 · p. 20

Abstract

from arXiv · show

Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions and are scored by closed-loop task success. On the other, predictive and generative world models forecast future observations and are scored by open-loop prediction or generation quality. A natural question sits between them: does world modelling earn a measurable, closed-loop advantage over a direct policy, and for which robotic capabilities? We argue that the field cannot yet answer this question, and that the reason is a gap in how it is measured, not in the models themselves. World-model benchmarks score prediction without ever executing it, while task-success suites host a single policy and never build a world-model versus VLA contrast. This survey maps the evaluation landscape around that gap. We catalogue 160 web-verified benchmarks spanning 2017 to 2026 and organise them by evaluation mode, robotic capability, and model family into four lanes: policy suites, embodied agents, world model evaluation, and prediction-to-action bridges. Across the corpus, 138 of 160 benchmarks are model-agnostic and only 11 (7%) build an explicit VLA-versus-world-model contrast; counterfactual capability is almost entirely unmeasured, and only four benchmarks turn prediction into executed action. We contribute an operational taxonomy, a coverage comparison against the eight closest surveys (ours is the only one to cross capability with model family), an evaluation loop that isolates the advantage of prediction, and an actionable protocol of four advantage-aware metrics anchored on named testbeds. The organising claim is not that world models help or do not help, but that answering the question requires benchmarks built to ask it.

1 Introduction

The survey asks whether world modelling provides a measurable closed-loop advantage over direct policies, and finds that existing benchmarks are not built to answer it. It maps the evaluation landscape and identifies how future benchmarks could make the comparison.

1 Introduction

  • The survey’s organising question is whether prediction earns a measurable closed-loop advantage over a direct policy, and for which robotic capabilities.The authors argue that current benchmarks have not instrumented this question; they do not claim that world models help or do not help.Source: §1 · pp. 2–3
  • World-model benchmarks score prediction without executing it, while task-success suites run a single policy and do not compare it with a world-model policy.A useful contrast must run both branches in one closed loop and report gains by capability.Source: §1 · p. 2
  • 138 of 160 benchmarks are model-agnostic, only 11 (7%) make an explicit VLA–world-model contrast, counterfactual capability is almost unmeasured, and four execute prediction as action.The corpus comprises 160 web-verified benchmarks spanning 2017 to 2026.Source: §1 · p. 2
  • The survey organises 160 benchmarks into four evaluation lanes and twelve capability areas, and compares coverage across the eight closest surveys.It also proposes an evaluation loop and four advantage-aware metrics anchored on named testbeds.Source: §1 · p. 3

1.1 Related surveys and the empty cell

  • Among the eight closest surveys, none crosses robotic capability with model family; only a position paper [153] partially reaches the advantage-of-prediction axis.The survey is the only one to organise evaluation mode, capability, and model family together.Source: §1.1 · p. 3

2 Survey methodology and corpus

The survey defines its unit as a benchmark, gathers and verifies candidates through web search, and places them using operational rules. It separates a representative core taxonomy from a complete landscape catalogue.

2.1 Scope, search, and verification

  • The survey’s unit is an individually citeable benchmark, suite, or dataset paired with a scoring protocol or interactive task distribution.Individual models and policies, raw datasets without scoring protocols, and pure simulators without task distributions are excluded.Source: §2.1 · p. 4
  • 160 web-verified benchmarks drawing on 164 references form the corpus, after candidates were checked against arXiv or venue pages and unverifiable entries were dropped.The search covered arXiv, publisher venues, and project pages; entries were de-duplicated by citation key and arXiv identifier.Source: §2.1 · p. 4

2.2 Placement, definitions, and the core-versus-appendix rule

  • Each benchmark is placed on four axes: evaluation lane, primary robotic capability, evaluation mode, and whether it builds an explicit, partial, or model-agnostic native VLA-versus-world-model contrast.A benchmark counts as building a contrast only if a direct-policy and a world-model policy are run under one protocol and compared.Source: §2.2 · pp. 4–5

Appendix A). Every count in the survey reconciles with that table.

The catalogue spans four evaluation lanes and twelve capability clusters, with closed-loop task success dominating its evaluation modes. Most benchmarks do not build a VLA-versus-world-model contrast.

2.3 The corpus at a glance

  • Across a single paper, 160 benchmarks divide by lane into 37 policy suites, 85 embodied-agent benchmarks, 34 world-model-evaluation benchmarks, and 4 prediction-to-action bridges.Source: §2.3 · p. 5
  • The passages report 43 benchmarks scoring open-loop prediction or generation, while only 4 bridge that evaluation with closed-loop task success, showing limited overlap between the settings.Publication years show a steady rise from 2 in 2017 to a peak of 36 in 2024.Source: §2.3 · p. 6
  • 138 of 160 benchmarks are model-agnostic, while 11 (7%) build an explicit VLA-versus-world-model contrast.The profile figure reports exact column tallies from the catalogue.Source: §2.3 · p. 6, Figure 5 · p. 7

3 A taxonomy of robot-evaluation benchmarks

This section organizes representative robot-evaluation benchmarks into four capability-grouped lanes according to how closely scoring approaches executed behavior, from executed policy and embodied-agent evaluation to open-loop world-model scoring and prediction-to-action bridges. It also considers the four bridges that carry predictions through to executed action, noting that none isolates a world model’s advantage over a direct VLA policy under a shared protocol.

3.1 Policy suites

  • The 37 policy suites execute policies and count task success, but their model-agnostic design cannot by itself establish whether a world-model policy beats a direct one.Native contrasts cluster in hidden-state, theory-of-mind, and counterfactual reasoning cuts; long-horizon and social task-success cuts are largely model-agnostic.Source: §3.1 · pp. 8, 9, 12, Table 5 · p. 12

3 A taxonomy of robot-evaluation benchmarks

  • Figure 7 organizes 86 representative benchmarks into four evaluation lanes ordered by proximity to executed behavior, with benchmarks grouped by the capabilities they stress.The lanes progress from closed-loop policy suites and embodied-agent benchmarks through open-loop world-model evaluation to prediction-to-action bridges.Source: §3 · p. 7, Figure 7 · p. 10

3.2 Embodied agents

  • The embodied-agent lane is largest, with 85 benchmarks, but is overwhelmingly model-agnostic; its thin counterfactual subfamily almost never runs a closed-loop VLA-versus-world-model comparison.The lane spans household and planning, social and multi-agent, navigation, driving, counterfactual reasoning, and locomotion tasks.Source: §3.2 · pp. 12–13

3.3 World-model evaluation

  • The 34 world-model evaluation benchmarks score prediction or generation without action; a high generation score measures visual quality, not task success.Generation-quality suites dominate and are generally physics-agnostic; physical-reasoning and physical-commonsense subtypes are the ones that test physical law.Source: §3.3 · p. 13, Table 6 · p. 13

3.4 Prediction-to-action bridges

  • The bridge lane is a tiny frontier of four benchmarks: RoboWM-Bench [73] tests prediction executability, while World-in-World [157] and WorldArena [124] place world models in a closed loop and report scoring.WorldSimBench [115] evaluates video-to-action consistency.Source: §3 · p. 8, §3.4 · p. 14
  • Only four benchmarks carry prediction through to executed action in controllable simulators.They form the shaded frontier of Figure 7.Source: §3.4 · p. 14
  • None compares direct VLA and world-model policies head-to-head per capability under one protocol, so the bridges close the loop without isolating prediction’s advantage.That gap is formalised in Section 4.Source: §3.4 · p. 14

4 Does prediction help acting? The contrast gap

The survey finds that existing benchmark lanes do not jointly provide closed-loop scoring, capability-level comparison, and a VLA-versus-world-model contrast. It specifies the comparison such a benchmark would need to make.

4.2 Why each lane leaves the contrast unbuilt

  • Policy and embodied suites score closed-loop success but hold model family fixed; world-model suites vary model family but score open-loop prediction; four bridges execute world-model policies without capability-sliced VLA head-to-heads.No lane combines closed-loop scoring, capability slicing, and a native VLA-versus-world-model contrast.Source: §4.2 · p. 16

4 Does prediction help acting? The contrast gap

  • The required comparison runs a direct policy and a predict-then-plan world-model policy in one closed loop, measures task-success gain against a matched baseline, and slices it by capability.The survey notes that prediction may matter more where hidden state, occlusion, or long horizons make foresight valuable.Source: §4 · p. 16, Figure 10 · p. 17

4.3 Measuring the advantage: assessed, not adopted

  • Only four benchmarks reach executed action, counterfactual capability is almost entirely unmeasured, and 11 of 160 build an explicit contrast.The survey assesses candidate advantage-of-prediction instruments rather than adopting one as ground truth.Source: §4.3 · pp. 16–17

4.1 “World model” is three things, none of them sufficient alone

  • “World model” spans representation and role: no single sense is simultaneously action-conditioned, closed-loop, and readily VLA-comparable under one protocol.Latent and action-conditioned models act but are rarely scored by standard generation metrics, while generative models carry metrics but are rarely closed into a loop.Source: §4.1 · p. 16, Table 7 · p. 18

5 An agenda for advantage-aware evaluation

The agenda proposes four measurable quantities for evaluating whether prediction helps acting, each anchored to named testbeds. The protocol specifies target measurements rather than reporting benchmark results.

5 An agenda for advantage-aware evaluation

  • The advantage-of-prediction curve measures a world-model policy’s closed-loop success gain over a matched direct-VLA baseline per capability.It is instantiated on LIBERO [91] and CALVIN [101] under one harness.Source: §5 · p. 18
  • Counterfactual accuracy measures task success on episodes requiring reasoning about unobserved or hypothetical dynamics.The proposed testbed combines WorldPrediction [25] with occlusion and memory splits in the style of LIBERO-Mem [36].Source: §5 · pp. 18–19
  • The prediction-to-action fidelity gap is the difference between a world model’s open-loop generation quality and its closed-loop task success.Bridge protocols include WorldSimBench [115] and World-in-World [157].Source: §5 · p. 19
  • Per-capability contrast coverage audits the fraction of capabilities with a VLA-versus-world-model head-to-head under one benchmark protocol.Each quantity is reported as a distribution or curve, not a single headline number.Source: §5 · p. 19, Table 9 · p. 19
  • Figure 11 sketches the target shapes of the four measurements; it presents a protocol, not measured results.The proposed testbeds include LIBERO / CALVIN, WorldPrediction, and bridge suites.Source: §5 · pp. 18, 20, Figure 11 · p. 20, (b) Counterfactual accuracy · p. 20

6 Limitations

The survey maps benchmarks rather than ranking models, and its proposed advantage-of-prediction instruments remain candidates rather than validated measures. Its coverage and contrast classifications also have explicit scope boundaries.

6 Limitations

  • The corpus maps benchmarks, not models, and does not rank world models or VLA policies using metrics the field has not validated.The advantage-of-prediction instruments are assessed as candidates rather than adopted.Source: §6 · p. 20
  • A contrast counts only when both policy branches run under one protocol; a more permissive threshold would reclassify some benchmarks as partial.The qualitative finding that explicit contrasts are rare and counterfactual coverage is thin is described as robust to the threshold.Source: §6 · p. 20
  • Corpus coverage is biased toward recent, arXiv-indexed, English-language work reachable by web search.The funnel records only logged stages, and gallery omissions do not establish that results are absent.Source: §6 · p. 20

7 Conclusion

The survey finds that benchmarks rarely compare world-model policies with direct VLA policies in closed-loop, per-capability evaluations. It makes this gap measurable through a taxonomy, comparison, evaluation loop, and proposed protocol, while withholding a verdict on whether world models help.

7 Conclusion

  • 138 of 160 benchmarks are model-agnostic, only 11 build an explicit contrast, counterfactual capability is nearly unmeasured, and only four carry prediction into executed action.These findings describe the benchmarks built to evaluate the question, not the relative strength of model families.Source: §7 · p. 21
  • World-model suites score prediction without executing it, while task-success suites execute a single policy without contrasting model families.The survey identifies this as a measurement gap rather than a conclusion about the models themselves.Source: §7 · p. 21
  • The survey contributes a four-lane taxonomy, a comparison with prior surveys, an evaluation loop, and a four-metric protocol anchored on named testbeds.It reports that no prior survey crosses robotic capability with model family.Source: §7 · p. 21
  • The paper’s conclusion is that answering whether prediction helps requires benchmarks that run both policy branches in one loop and measure their difference.The next benchmark should be built to ask that question, rather than judged by task count or render realism alone.Source: §7 · p. 21

A The full benchmark landscape

This section presents the benchmark landscape through a core taxonomy of 86 cleanly characterisable benchmarks and a full catalogue of 160 web-verified benchmarks, organized by evaluation lane and capability and compared across four axes. It catalogs video-generation, physical-reasoning, and counterfactual benchmarks, as well as four prediction-to-action bridges.

B Provenance of the galleries and collage

The galleries and collage use figures reproduced from benchmark papers, with provenance tables recording their sources and figure numbers where verified. Unlocated figures were dropped rather than guessed.

B Provenance of the galleries and collage

  • Figures 8 and 9 and the Figure 1 collage use real figures reproduced from benchmark papers and hyperlinked to their arXiv sources.Tables 11–13 document panel and tile provenance; figures whose source could not be located were dropped rather than guessed.Source: B Provenance of the galleries and… · p. 32
  • Table 11 records Figure 8 panel provenance, giving verified figure numbers or noting the stated basis when a number was unavailable.The gallery panels are cropped from benchmark papers; gap tiles were hand-picked from per-paper candidate contact sheets after classification.Source: Table 11 · p. 34
  • Table 12 records Figure 9 results-panel provenance; plots are contain-fitted so bars, axes, and radar spokes remain visible.A benchmark may appear more than once when it reports multiple evidence types.Source: Table 12
  • Table 13 records Figure 1 tile provenance, while the collage includes EWMScore ranking bars and evaluation radars.The radar examples include VBench’s 16-dimension radar and EWMBench’s 5-dimension EWMScore radar.Source: Table 13 · p. 34, Figure 1 · p. 34
Loading 2609.29669v1…