Source-linked AI summary
HarnessEval-W: Agentifying the Evaluation of Visual Worlds
Weiliang Chen, Haowen Sun, Jun Gao, Jiawei Chi, Hanyang Wang, Qiyu Dai, Yihao Li, Hao Li, Jingnan Gao, Yi-Hsin Hung, Xingzhuo Guo, Shangchen Miao, Zhiyuan Shi, Xiang Li, Fengrui Tian, Weihua Du, Ziqi Huang, Shenyuan Gao, Siqiao Huang, Mingyu Liu, Yifei Li, Shizun Wang, Xi Wang, Tianqi Zhang, Xue Luo, Xiyin Ren, Jinshan Ren, Xiaoyang Shen, Xiaobo Hu, Zhiyang Dou, Mingyu Ding, Yichao Yan, Xinchao Wang, Yizhou Wang, Shilong Liu, Wenzhao Zheng, Yueqi Duan, Yuan Gong, Ziwei Liu, Ming-Yu Liu, Jialong Wu, Jiangran Lyu, Fangfu Liu
TL;DR
Existing world-model benchmarks do not automate the reasoning humans use to assess physical causality, consistency, and realism, leaving scores difficult to explain or verify. HarnessEval-W uses a context-dependent hierarchical agentic workflow to decompose cases, gather specialized evidence, and validate reasoning; across 330 cases and 18 models, its judgments closely align with human preferences while producing transparent diagnoses.
Problem
Existing world-model benchmarks do not automate human-like assessment of physical causality, geometric consistency, and observation realism, leaving scores unexplained and unverifiable.
Method
HarnessEval-W routes each context-dependent case to relevant skills, decomposes evaluation into measurable subquestions, and aggregates validated specialized-agent evidence into a reasoning trace.
Results
Across 330 evaluation cases and 18 representative world models, HarnessEval-W judgments closely align with human preferences while providing verifiable, fine-grained rollout diagnoses.
Takeaways & Limitations
HarnessEval-W provides a living, open-source benchmark whose executable agentic workflow can grow new evaluation skills and cases as world models evolve.
Takeaways & Limitations
Out-of-distribution cases may expose evaluator skill gaps when routing finds no skill that can legitimately answer them.
Abstract
from arXiv · showhide
A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.
1 Introduction
HarnessEval-W introduces a hierarchical, agentic benchmark that evaluates context-dependent world-model behavior through decomposed subproblems, specialized tools, evidence validation, and transparent reasoning chains. It organizes 330 cases across observation quality, transition correctness, and world persistence, and is released as a living benchmark that can expand with new skills and cases.
- Motivation: Existing world-model benchmarks do not automate human-like detection of physical causality, geometric consistency, and observation-realism artifacts.The paper characterizes current evaluation as fragile and unable to provide the analysis humans perform naturally.
- Motivation: HarnessEval-W adapts the evaluation-harness paradigm to world models by formalizing evidence gathering, tool use, and reasoning as an agentic workflow.The harness is presented as scaffolding for complex human evaluation workflows rather than merely a code wrapper.
- Method: Because each test case has distinct physical actions, temporal structures, and observation states, HarnessEval-W decomposes questions into measurable subquestions instead of applying a fixed rubric.The hierarchical design assigns appropriate skills and context to each case before decomposition.
- Method: Sub-agents and tools gather diagnostic evidence, while parent agents validate and merge findings into a recursively expandable evidence tree rather than a scalar score.Collision evaluation can involve tracking bounding boxes, checking temporal intersection, and estimating velocity.
- Benchmark and release: 330 evaluation cases cover Observation Quality, Transition Correctness, and World Persistence, and the open-source benchmark is designed to grow new skills and cases as models evolve.Cases are authored by sampling diverse initial worlds, grounding actions in generated scenes, and organizing evaluations around the three core world-model functions.
2 Related Works
Related work progresses from video-generation quality benchmarks to interactive-world evaluation of actions, physics, consistency, and hidden state. HarnessEval-W extends adaptive, tool-grounded evaluation to interactive worlds through reusable skills that select, validate, and aggregate evidence as rollouts unfold.
- Video Generation: Video generation has advanced from U-Net and latent diffusion models to scalable diffusion transformers, improving fidelity, prompt adherence, and temporal coherence.Examples include Video Diffusion Models, Align Your Latents, CogVideoX, HunyuanVideo, and Wan.
- Interactive World Models: Interactive video world models condition future observations on actions in closed loops, spanning latent-dynamics methods and pixel-space systems such as Genie, GameNGen, DIAMOND, and MineWorld.These systems include game-centered and more open-ended action-conditioned models.
- World Evaluation Benchmarks: Existing benchmarks evaluate rendered reliability, realism, action effects, long-horizon consistency, revisitation drift, and offscreen state updates through complementary visual, physical, interactive, and memory-focused tests.VBench and EvalCrafter assess video quality; VBench-2.0, VideoPhy, PhyGenBench, and PhyWorldBench assess realism and physics; WorldScore, WorldMark, revisitation suites, and MemoBench address interactive dynamics and state.
- Adaptive Evaluation: Fixed evaluation pipelines support reproducibility and evidence extraction, but interactive worlds require adaptive applicability, execution, and aggregation because controls and outcomes determine what can be tested next.MUSIQ and related tools extract frame-quality, camera-motion, return-observation, and mask evidence, while VLMEvalKit and related toolkits support reproducible pipelines.
- HarnessEval-W: HarnessEval-W extends adaptive evaluation to interactive worlds through reusable skills defining applicability, evidence, scoring, validation, and aggregation, excluding unsupported measurements while retaining genuine failures as negative evidence.Prior adaptive evaluators select prompts, regions, frames, and tools, but largely target prompt-conditioned image or video generation and editing.
3 HarnessEval-W: Agent Evaluation Harness
HarnessEval-W evaluates interactive world models by decomposing their behavior into three world-state axes and applying context-dependent hierarchical agent reasoning. Its evidence tree records the tests, visual grounding, sub-agent reasoning, and failure diagnoses behind each final score.
- Evaluation axes: HarnessEval-W evaluates interactive world models through observation quality, transition correctness, and world persistence, organized into eight detailed evaluation settings.The axes respectively assess rendered-state reliability, action-conditioned state updates, and coherence as the world evolves.
- World-model formulation: The benchmark accepts models that predict future observations from historical observations and user-specified actions, including bidirectional diffusion and autoregressive video models.Actions may change viewpoint, alter specified entities or events, or intervene physically.
- Hierarchical workflow: Because evaluation cases differ in worlds, actions, temporal structure, and observable state, HarnessEval-W routes each case to context-appropriate reusable skills rather than applying a fixed rubric.Routing interprets the initial image, action prompt, and evaluation setting to select relevant high-level questions.
- Hierarchical workflow: Each high-level skill is decomposed into specialized sub-questions whose sub-agents inspect specific rollout properties and return supporting evidence with their assessments.For Intentional Change Verifier, branches check expected outcomes, target visibility, transition occurrence, and target correctness.
- Evidence-based evaluation: The workflow produces a transparent evidence tree that records what was tested, which tool supplied visual grounding, and the logical chain supporting the final score.This structure provides actionable failure diagnoses for researchers and guides future model development.
4 HarnessEval-W Case Construction
HarnessEval-W constructs scalable, diagnosable evaluation cases by combining diverse scene-taxonomy sampling with probe-family assignment and agentic authoring and validation. The resulting benchmark contains 330 cases spanning six probe families, diverse worlds, and interaction horizons from immediate transitions to long-term persistence.
- Overall Pipeline: The case-construction pipeline combines real-world data and synthetic cases to achieve diverse, realistic, scalable, diagnosable, and verifiable evaluations.It samples world metadata, generates cases, plans actions, and validates each image–action pair.
- Scene Taxonomy Sampling: A structured scene taxonomy samples complete world initializations across six complementary axes covering environments, foreground entities, spatial layouts, and other scene attributes.Environment categories include indoor, outdoor, and transitional settings; foreground entities can be manipulated or serve as landmarks.
- Probe Family Assignment: Six probe families cover immediate state or physical transitions, exploratory rollouts, and longer persistence evaluations of drift resistance, revisit consistency, and offscreen evolution.Observation Quality is evaluated in every case rather than treated as a separate family.
- Agentic Case Authoring: Agentic authoring uses an image generator, image-grounded planner, and case validator to create initial observations, specify feasible evidence-supported actions, and reject invalid cases.The planner cannot change the assigned probe family or introduce entities absent from the image, while the validator checks visibility, feasibility, outcome specificity, and probe evidence.
- Benchmark Composition: 330 cases span diverse initial worlds, interaction patterns, environments, entities, layouts, scene densities, visual appearances, and perspectives.The released benchmark also distributes cases across six probe families and varied intervention horizons.
5 Experiments and Analysis
HarnessEval-W evaluates 18 world models on 330 shared cases using eight normalized metrics and produces auditable reasoning traces. Experiments show strong human alignment, robustness across repeated evaluations, diverse metric-specific strengths, and capability shifts under fine-tuning.
- Experimental Setup: The benchmark evaluates 18 representative world models on the same frozen set of 330 cases through each model’s native conditioning interface.Cases specify an initial observation, interaction, and world-state question, translated into text, camera, or control inputs while preserving evaluation intent.
- Experimental Setup: HarnessEval-W uses eight metrics spanning render quality, observation, transition correctness, drift resistance, revisit consistency, and offscreen evolution, with final scores normalized from 1 to 100.Observation metrics average across all 330 cases, while other metrics use their corresponding probe-family subsets; overall score is the arithmetic average of case-level scores.
- Leaderboard and Reasoning Traces: Seedance 2.0 ranks first with an Overall score of 75.5, followed by Wan 2.7 (75.0), Kling 3.0 (74.4), and MiniMax H3 (74.3).Wan 2.7 leads Intentional and Physical Transition Correctness, while Seedance 2.0, HY-WorldPlay 1.5, and SANA-WM lead Drift Resistance, Revisit Consistency, and Offscreen Evolution, respectively.
- Leaderboard and Reasoning Traces: Each evaluation trace records the case, rollout, planner route, skill measurements, aggregation, and validator decisions, enabling end-to-end auditing of the final score.The planner can select different routes, such as drift analysis for long-horizon navigation or intentional state-change analysis for intervention cases.
- Evaluator Validation: On Physical evaluation, HarnessEval-W raises pairwise accuracy from 31.9% to 71.7% and cuts the draw rate from 52.2% to 1.8% against human judgments.On Intentional, accuracy rises from 60.2% to 77.8% while draws fall from 36.1% to 11.1%; HarnessEval-W also achieves the lower Brier score in both settings.
- Evaluator Validation: Across repeated evaluations at temperature 0, fitted slopes remain within 9.6–10.8 and correlations with human strength within 0.928–0.964.The study repeatedly runs GPT with the same GPT-5.5 backend and applies the same score-to-human fitting procedure to HarnessEval-W and WBench.
6 Future Work: Toward Self-Evolving Harness for Evaluation
The future HarnessEval-W agenda centers on scaling evaluator computation, expanding reusable skill libraries, and recursively improving capabilities when evaluation encounters unsupported scenarios.
- Test-Time Scaling for Agentic Benchmarks: Additional test-time compute enables finer skill decomposition, deeper sub-agent rollout search, multi-step verification, and repeated evidence re-verification.The harness applies the test-time-compute paradigm to evaluation as reasoning traces and tool use become more complex.
- Scaling Skill Libraries: A continuously expanding skill library stores scenario-specific evaluation knowledge, allowing the evaluator to retrieve and compose specialized skills instead of rebuilding bespoke rubrics.Each skill specifies what evidence settles a question and which tool grounds it, supporting assessment across intricate physics and diverse scenes.
- Recursively Self-Improving Agentic Benchmarks: When no existing skill can legitimately answer an out-of-distribution case, the evaluator should expose the gap, acquire the missing capability, and write it back into its library.Proposed routes include external skill expansion and self-driven exploration.
7 Conclusion
HarnessEval-W brings agentified, hierarchical evaluation to world model benchmarking. It replaces fixed rubrics with context-aware sub-question decomposition, specialized sub-agents, and validated evidence aggregation that makes each score examinable and verifiable.
- 7 Conclusion: HarnessEval-W introduces an agentified evaluation pipeline that brings the harness paradigm from LLMs to world model benchmarking.The pipeline is designed specifically for evaluating world models.
- 7 Conclusion: The pipeline interprets each evaluation case’s context and decomposes its evaluation into measurable sub-questions.This replaces the use of a fixed rubric with case-specific evaluation structure.
- 7 Conclusion: Specialized sub-agents answer the sub-questions, while validated evidence is aggregated into a transparent reasoning trace.This hierarchical process allows every score to be examined and verified.