PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

Zhefan Rao, Meng Chu, Xuan Zhang, Jingfan Chen, Guohao Cai, Runyi Li, Shiyi Du, Ning Lu, Xu Huang, Zhixuan Chen, Feng Nie, Harold Haodong Chen, Hanwen Wang, Wen-Jie Shu, Yajing Bai, Xianfeng Wu, Xuanhua He, Shuhan Guo, Ziyue Ma, Ruichuan An, Long Bai, Miaosen Chai, Wenhao Chai, Jingye Chen, Qi Chen, Xuhang Chen, Wei Chow, Minghao Fu, Yunhao Gou, Haokun Gui, Dong Huang, Wei Huang, Zi Cheng Huang, Haibo Jin, Lingdong Kong, Leung Chun Ho, Tianqin Li, Yanheng Li, Jiayuan Liu, Jiyang Liu, Mingyu Liu, Shaoteng Liu, Xiaocheng Lu, Yawen Luo, Zeyu Qin, Tiancheng Shen, Yueqi Song, Zhaolong Su, Hengkai Tan, Tianhao Tang, Yiran Tao, Juncheng Wang, Qinghe Wang, Shangshang Wang, Yingying Wang, Fang Wu, Tiange Xiang, Qiushi Yang, Weicai Ye, Weichen Yu, Jiarui Zhang, Jize Zhang, Liuzhou Zhang, Renrui Zhang, Sheng Zhang, Wenhu Zhang, Junru Zhao, Lingting Zhu, Yuxin Zuo, Yutong Bai, Long Chen, Yang Deng, Ziqi Huang, Yeying Jin, Weijia Li, Kevin Qinghong Lin, Quande Liu, See-Kiong Ng, Hongliang Ren, Hongru Wang, Shujun Wang, Qifeng Chen, Haoxuan Che

arXiv:2610.31066v1cs.AI

TL;DR

Evidence about both action selection and state evolution is needed to assess interaction between Multimodal Agents and World Models, but coupled outcomes alone cannot diagnose component failures. Dual Exam tests the components separately under controlled conditions across four domains. Performance varies across task categories, and the two tracks’ aggregate scores are not directly comparable.

  • Problem

    Coupled outcomes cannot distinguish failures in Agent decisions, World Model responses, or their interaction, despite both components needing reliability for repeated interaction.

    Source: §1 · p. 2

  • Method

    Dual Exam evaluates Agents under fixed environment responses and World Models under fixed action interventions, checking recorded evidence against explicit task goals and constraints.

    Source: §1 · p. 3, §2.4 · p. 8

  • Results

    Performance varies across tasks; GPT 6 Astra leads the case-weighted Agent Index at 31.3, while GPT-Image-2 leads full-coverage WM Overall at 21.7.

    Source: §3.1 · pp. 8–9, §5 · p. 13

  • Takeaways & Limitations

    The separate profiles characterize component performance, but whether they predict coupled success or improvement through joint training remains an empirical question.

    Source: §1 · p. 3

    The leaderboard reports point scores without repeated-run uncertainty estimates or significance tests, and the separate tracks do not establish prediction of coupled performance.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

World Models (WMs) can support Multimodal Agents (MAs) through simulated experience, while Agent trajectories can inform World Model refinement. Evaluating this interaction requires separate evidence about action selection and state evolution. We introduce Dual Exam, which tests Agents under fixed environment response mechanisms and World Models under fixed action interventions across physical, digital, social, and scientific domains. The Agent track compares 10 configurations on 1,088 items in 15 reporting categories. The World Model track contains 400 cases across eight scenarios; 15 configurations have full coverage and one covers six scenarios. GPT 6 Astra leads the case-weighted Agent Index at 31.3, and GPT-Image-2 leads the full-coverage WM Overall at 21.7. These scores use different tasks and metrics and are not directly comparable. A 150-case pilot of one coupled pairing records zero task successes and motivates component-level diagnosis. Results reveal uneven task performance and failures to preserve task-relevant state. Their value for predicting coupled success or sustained reciprocal learning remains to be established.

1 INTRODUCTION

Assessing reciprocal interaction between Multimodal Agents and World Models requires evidence about both action selection and state evolution, while coupled failures alone cannot identify their source. Dual Exam evaluates each component separately under controlled conditions and reports uneven performance across task categories.

1 INTRODUCTION

  • Coupled systems require both reliable action selection and faithful action-conditioned responses, yet end-to-end failures cannot distinguish agent decisions, model responses, or their interaction.Repeated interaction can expose limits in visual grounding, action sequencing, memory consistency, and state similarity; plausible visual fidelity alone does not guarantee task success.Source: §1 · p. 2
  • The 150-case pilot records 0% task success, but its trajectories do not identify which component caused each failure.This motivates separate controlled assessments rather than causal attribution from coupled outcomes.Source: §1 · p. 2
  • GPT 6 Astra leads the case-weighted Agent Index at 31.3, while GPT-Image-2 leads full-coverage WM Overall at 21.7.The tracks use different tasks and metrics, so their aggregate scores are not directly comparable.Source: §1 · p. 3
  • The evaluation spans physical, digital, social, and scientific domains, comparing 10 Agent configurations on 1,088 items and organizing World Model tests into 400 cases across eight scenarios.Fifteen World Model configurations cover all 400 cases; Seedance 2.5 covers six scenarios, or 300 cases.Source: §1 · p. 3, Figure 3 · p. 4
  • Dual Exam separately tests Agents under fixed environment responses and World Models under fixed action interventions, verifying recorded evidence against explicit task goals and constraints.The tracks retain track-specific cases and metrics and need not use matched episodes.Source: §1 · p. 3

2 DUAL EXAM

Dual Exam separates evaluation of Agent action selection from World Model state evolution by fixing the environment response mechanism or action interventions. It covers four domains and uses task-specific cases, protocols, and verification procedures.

2 DUAL EXAM

  • Dual Exam evaluates Agents under fixed reference environment responses and World Models under fixed interventions across physical, digital, social, and scientific settings.The tracks need not use one-to-one matched episodes.Source: §2 · p. 4

2.1 PRELIMINARY STUDY: MOTIVATION FOR DUAL EXAM

  • 26.6%, 9.2%, and 0.0% are the pilot’s World State Consistency, Agent Corrective Response, and Task Success, respectively; commands continued despite unconfirmed acquisition and later used a smaller pouch.The pilot’s retrospective trajectory scores are diagnostics, not isolated model scores; the plotted decline follows from fixed denominators rather than measured monotonic capability degradation.Source: §2.1 · p. 5

2.2 DUAL EXAMINATION FRAMEWORK

  • Both tracks assess specified goals and constraints using recorded execution evidence, but fix different parts of the interaction and do not establish causes of coupled failures or sustained reciprocal improvement.The framework uses shared specification principles without implying matched tasks, numerical scores, or training data.Source: §2 · p. 4, §2.2 · p. 6

2.3 AGENT EXAMINATION: CONTROL THE WORLD, TEST THE AGENT

  • The Agent inventory spans 1,088 tasks in 15 categories across four domains; 0–100 scores may represent strict completion or graded performance, and Edge scores use a study-introduced clipping convention.The four domains are Physical, Digital, Social, and Scientific; the inventory’s selected-task weights describe its composition, not domain importance.Source: §2.3 · pp. 6–7

2.4 WORLD MODEL EXAMINATION: CONTROL THE ACTIONS, TEST THE WORLD MODEL

  • The 400 World Model cases allocate 50 to each of eight scenarios; Overall averages scenario scores, not domains, missing scenarios are not zeros, and text-only models fall outside the visual-observation scope.Configuration histories, duration, budgets, and missing provenance also limit cross-modality interpretation.Source: §2.4 · pp. 7–8

3 EXPERIMENTS

Dual Exam evaluates Agents and World Models separately under controlled conditions, with different tasks and aggregation rules for the two tracks. Results show uneven performance across settings and leave open how component scores predict coupled success.

3 EXPERIMENTS

  • The evaluation compares 10 Agent configurations across 15 categories and 1,088 items, and World Models across eight scenarios and 400 cases.Agent and World Model tracks use separate tasks and metrics; the illustrative trajectories do not estimate failure frequencies.Source: §3 · p. 8

3.1 AGENT RESULTS: RQ1

  • Displayed Agent scores range from 5.6 for Desktop Application Interaction to 47.1 for Game Environment Interaction, under their respective task requirements and scoring rules.Illustrative Navigation and Action Sequencing cases show that recognizing relevant elements can still yield invalid routes or ordering when task constraints are violated.Source: §3.1 · p. 8
  • Across 109 Scientific items, GPT 6 Astra leads the case-weighted score at 50.6, while category leaders differ across Mathematics & Computation, Physical Sciences, Health & Medicine, and Life Sciences.The domain mean combines ALE, SWE-bench repair, ToolChoice, and EdgeBench items; strict binary outcomes and graded quality scores contribute differently.Source: §3.1 · pp. 8–9
  • 31.3 gives GPT 6 Astra the highest case-weighted Agent Index; GPT-Image-2 leads full-coverage WM Overall at 21.7, but the metrics are not directly comparable.The Agent comparison covers 10 configurations and 1,088 items; full-coverage WM evaluation spans eight scenarios and 400 cases.Source: §3 · p. 8, Table 3 · p. 9, §3.1 · p. 9
  • Aggregate ranks depend on task composition: equal-domain weighting moves DeepSeek V4.1 Flash from sixth to second, while GPT 6 Astra remains first.Physical tasks contribute 48.16% of the case-weighted Index, reflecting the inventory rather than equal domain importance.Source: §3.1 · p. 9

3.2 WORLD MODEL RESULTS: RQ2

  • All evaluated World Model configurations score at most 16.5 on Web and 13.3 on Game, identifying digital scenarios as a shared weakness under this protocol.Differences in task difficulty and scoring prevent attributing the gap solely to the domain.Source: Table 4 · p. 10, §3.2 · p. 10
  • Illustrative World Model cases show distinct state-evolution failures: introducing extra cups violates an initial quantity constraint, while an unrequested tray changes unaffected state.The examples distinguish realizing requested changes from preserving task-relevant state and do not establish causes of aggregate ranking differences.Source: §3.2 · p. 10
  • Reliable task progress requires effective Agent decisions and World Model responses that realize action effects while preserving task-relevant state, but the tracks do not establish how their scores predict coupled success.World Model responses also inform the Agent’s judgment of progress.Source: §3.2 · p. 11

4 DISCUSSION

Prior work demonstrates benefits in both directions between Agents and World Models, but bounded-task gains do not establish a stable reciprocal loop. Dual Exam therefore motivates component-level evaluation while identifying limits in measurement reliability, evidence coverage, and comparison.

4.1 RELATED WORK

  • Existing benchmarks assess Agent or World Model competence, while evidence of gains in both directions still does not establish reciprocal improvement or a stable loop.Prior coupled systems and demonstrated gains remain bounded to particular task distributions; broader sustainable improvement and its required capability levels are unestablished.Source: §4.1 · p. 11, §4.2 · p. 11

4.2 TOWARD AGENT–WORLD MODEL CO-EVOLUTION

  • World Model experience can support Agent learning, but its value should be tested against behavior beyond the model used for training.Agent-collected interaction data can improve models or simulators, while Agent task scores alone do not measure the data's training value.Source: §4.2 · p. 11
  • Dual Exam identifies execution and state-consistency failures as possible interaction obstacles, not evidence that either component has limited learnability.The proposed next test compares joint updates with Agent-only, World-Model-only, and no-update controls across fixed tests, rounds, and seeds.Source: §4.2 · pp. 11–12
  • Invalid routes, action orders, state changes, and resource choices motivate checks of global constraints and protected state, but these remain hypotheses for controlled testing.The proposed interventions would measure strict completion and separately defined intermediate progress on the same cases.Source: §4.2 · p. 12

4.3 EVALUATION LIMITATIONS

  • Reported scores depend on task selection, reference environments, verification rules, and execution settings; independent stratified agreement estimates for World Model scores are not reported.The retrospective packing rubric has not been independently adjudicated, so measurement reliability beyond illustrative examples remains to be quantified.Source: §4.3 · p. 12
  • Scientific aggregation includes clipped EdgeBench scores and incomplete endpoint verification, while leaderboard point differences lack uncertainty estimates or significance tests.The separate tracks also do not establish how component scores predict coupled performance; matched interaction tests and repeated evaluations are needed.Source: §4.3 · p. 12

5 CONCLUSION

Dual Exam evaluates Agents under fixed environment responses and World Models under fixed action interventions, producing separate profiles across four domains. The Agent and full-coverage World Model leaders differ, and the tracks' scores are not directly comparable; predicting coupled success requires further experiments.

5 CONCLUSION

  • GPT 6 Astra leads the case-weighted Agent Index at 31.3, while GPT-Image-2 leads the full-coverage WM Overall at 21.7.The Agent comparison covers 10 configurations, 1,088 items, and 15 reporting categories; the World Model evaluation covers 400 cases across eight scenarios.Source: §5 · p. 13
  • The two tracks use different tasks and metrics, so their aggregate scores are not directly comparable or established as predictors of coupled performance.The results characterize model–harness performance under the evaluated specifications and weighting choices, subject to evidence and precision limits.Source: §5 · p. 13

A DETAILS OF AGENT EVALUATION

  • The appendix distinguishes current leaderboard inventory from historical illustrative cases, which explain task requirements and scoring rather than establish current coverage or rankings.The current inventory contains 1,088 items across 15 categories.Source: A DETAILS OF AGENT EVALUATION · p. 18

A.1 DOMAIN SELECTION, TASK CATEGORIES, AND INTERPRETATION

The appendix organizes selected Agent tasks into four domains and reporting categories, while cautioning that these labels are inventory structures rather than independent or exhaustive abilities. It also shows how domain weighting affects rankings and illustrates the goal-group demands of planning demonstrations.

A.1 DOMAIN SELECTION, TASK CATEGORIES, AND INTERPRETATION

  • The four domains organize selected tasks by setting, but are not an exhaustive intelligence taxonomy or evidence that domains measure independent abilities.Categories use different organizing criteria, and tasks can overlap in demands or subject matter.Source: §A.1 · pp. 18–19
  • Physical categories distinguish planning, sequence reconstruction, navigation, and manipulation, while digital categories identify interaction settings that can share underlying demands.The Social and Scientific categories preserve separate resource or subject reporting assignments, not claims of independent abilities or disjoint methods.Source: §A.1 · pp. 19–20
  • The Agent inventory contains 1,088 tasks across 15 categories: 524 Physical, 255 Digital, 200 Social, and 109 Scientific.Scientific tasks include 99 workflows and ten checkpoints; the related provenance does not imply independent observations.Source: Table 5 · p. 19, §A.1 · p. 20

A.2 SENSITIVITY TO DOMAIN WEIGHTING

  • GPT 6 Astra ranks first under both case-weighted and equal-domain mixtures, but DeepSeek V4.1 Flash moves from sixth to second and Kimi K3 from fifth to fourth.The reported Index weights categories by configured item counts; the alternative averages the four within-domain case-weighted scores. This is aggregation sensitivity, not uncertainty estimation.Source: §A.2 · p. 20, Table 6 · p. 20
  • The case-weighted Index reflects the configured inventory rather than a distribution-independent ordering of general Agent ability.Baseline category means are available only to one decimal place, and Social cases share 40 templates, so they should not automatically be treated as independent observations.Source: §A.2 · pp. 20–21

A.3 PLANNING

  • Planning demonstrations combine dependent household actions, and all displayed goal groups are satisfied in D1–D3, while D4 satisfies one group containing 23 conditions.The demonstrations illustrate required behavior rather than evaluated model predictions.Source: §A.3 · p. 21

A.4 MANIPULATION

The appendix describes manipulation, navigation, and action-sequencing evaluations through selected cases, separating recorded outcomes from what sampled frames establish. The examples include successful and failed episodes, route-feasibility errors, and sequence assignment or ordering errors.

A.4 MANIPULATION

  • Pouring and stacking succeed with scores of 1.0, while cloth folding and key insertion fail with scores of 0.2 and 0.15.Scores and success labels are official episode results; sampled frames do not identify precise grasp or contact failures.Source: §A.4 · pp. 21, 23, Figure 7 · p. 23

A.5 NAVIGATION

  • Navigation requires correct feasibility decisions for five agent types and valid endpoint-connected routes; selected cases include correct feasibility with missing-edge routes and a case permitting four infeasible types.N1 and N2 pass all checks; N3 has correct feasibility but an invalid edge, and N4 also permits four infeasible types.Source: §A.5 · pp. 23–24, Figure 8 · p. 24

A.6 ACTION SEQUENCING

  • Action Sequencing requires assigning every clip to one of two tasks and exactly matching both reference orders, without supplied sequence lengths.Selected cases include two exact matches, a case with correct clips in the wrong order, and a five-plus-five assignment where the reference split is four-plus-six.Source: §A.6 · p. 25, Figure 9 · p. 26, Figure 10 · p. 26

A.7 WEB INTERACTION

This section presents selected web-interaction cases, showing how retrieval becomes evaluator-creditable evidence and illustrating visual-feedback control, deadline-aware planning, and a conveyor-kitchen demonstration across windows 5–9. The kitchen demonstration ends at the success and efficiency target but, following prior exposure and an interrupted attempt, is not a new benchmark evaluation.

A.7 WEB INTERACTION

  • Web interaction evaluates sourced product configurations and priced purchase pages, distinguishing filled answer fields from verified claims; desktop and mobile tasks instead use task-specific deliverable checks and verifiers.The current leaderboards contain 95 desktop tasks and 35 mobile tasks.Source: §A.7 · p. 28, §A.8 · p. 31, §A.9 · p. 31
  • 27.50% was the telescope case’s highest score in R3, after 0% in R1 and 4.17% in R2, but the report still lacked required configurations and purchase pages.R3 included 16 configurations and 17 credited pages; tripod evidence was explicit for only five configurations, so the score did not independently verify every hardware condition.Source: §A.7 · p. 29
  • 15 attribute and 15 page credits in NAS R3 improved on R2, but remained far below the 90 requested purchase pages.R3 added nine attribute and ten page credits over R2; the saved F4-424 evidence linked specifications to a $499.99 variant price.Source: §A.7 · p. 29
  • Keyboard verification rose from 3 to 9 to 24 credited items across rounds, while purchase coverage remained incomplete.R3 received 14 attribute and 10 page credits; fewer searches do not establish greater efficiency because attempt budgets differed.Source: §A.7 · p. 30
  • Vacuum credits progressed 8 to 8 to 26, while headphone credits rose 20 to 25 to 28 despite unchanged 100% submission coverage.For vacuums, R2 gained a page credit but lost an attribute credit; for headphones, the remaining 92 items were uncredited, and a multipoint excerpt alone did not establish the complete ANC and battery requirement.Source: §A.7 · pp. 30–31
  • Selected failures included drone scores falling from 20.83% to 6.67%, zero thermal-camera credit in R3, and network-storage cases with full submission coverage but zero verification credit.These examples describe outcomes, not their causes; zero credit alone does not prove every submitted claim false.Source: §A.7 · p. 31

A.10 GAME ENVIRONMENT INTERACTION

  • The game examples cover visual-feedback timing and longer-horizon planning: Flappy Bird cleared one pipe after nine valid actions, while the kitchen run required managing changing demand, recipe limits, and contract deadlines.In the kitchen trace, early salads met the first contract by window 3, and a sandwich met the second deadline in window 6; the run was a non-blind illustration separate from benchmark evaluations.Source: §A.10 · p. 32, §02 · pp. 33–34

02. Key retrieval

  • The kitchen demonstration’s terminal event reports success, but supplies no exact efficiency percentage or post-settlement balance; $230 is the last observed pre-settlement balance.The recorded ingredient selections include tomato in window 2 and pumpkin in window 4.Source: O02: Selected ingredients · p. 34, O11: Selected ingredients · p. 35

O11: Selected ingredients

O14: Selected ingredients

  • At window 6, Avocado Fish Sandwich is listed at +$17 with $132 observed, while at window 8 the trace records net profit of $158.Source: O14: Selected ingredients · p. 35

O23: Selected ingredients

  • The trace reaches game over with the success and efficiency target met after window 9.The agent selects corn, advances to game over, and stops; the passage characterizes this as deadline-aware contract completion, but notes prior exposure and an interrupted attempt.Source: Figure 17 · p. 36, O23: Selected ingredients · p. 36

A.11 DIGITAL WORLD: HISTORICAL CROSS-ATTEMPT RESULTS

The historical multi-attempt results are descriptive trajectories, not controlled estimates of repetition or reflection effects. Coverage varies across attempts, and selected examples do not represent all social templates.

A.11 DIGITAL WORLD: HISTORICAL CROSS-ATTEMPT RESULTS

  • Historical means use all accepted records within each configuration and attempt, but changing coverage makes them unsuitable replacements for current leaderboard scores.The revised inventory also omits pooled statistics and fixed-cohort curves from the previous model set.Source: §A.11 · p. 36
  • The social inventory contains 200 cases in three categories, but selected resource-governance traces are not representative of all social templates.The categories contain 87, 56, and 57 cases, respectively.Source: §A.11 · p. 36
  • Table 8 reports Try 1, Try 2, and Try 3 scores where available; desktop denominators and mobile coverage vary across attempts and models.Unavailable results are marked with a dash.Source: Table 8 · p. 37

A.12 RESOURCE-GOVERNANCE EXAMPLES: GROUNDWATER AND FORESTS

The resource-governance examples are stylized extraction games, and both selected traces deplete their stock after the first round. The surrounding Scientific-task results have configuration and specification constraints that limit interpretation.

A.12 RESOURCE-GOVERNANCE EXAMPLES: GROUNDWATER AND FORESTS

  • The 113 resource-labelled social cases execute either capacity allocation (87) or resource extraction (26), rather than measuring independent governance abilities.The two selected examples illustrate extraction scenes, not representative coverage of all social templates.Source: §A.12 · p. 37
  • Each stylized game has 12 independent operators, a 100-unit initial stock, and a six-round horizon; the industrial operator requests 46 units, leaving four sustainable units for participants.These are not validated hydrological or ecological simulators.Source: §A.12 · p. 37
  • Both groundwater and forest traces reach zero stock after round one, with zero participant extraction in rounds two through six, constraint failures, and raw scores of 0.05.The initial participant requests total 129 units for groundwater and 157 for forest before the industrial request; these are requests, not realized extraction.Source: §A.12 · p. 38, Table 9 · p. 38
  • The Scientific domain has 109 tasks across four subject categories, including 30 SWE-bench repair tasks, 10 derived ToolChoice checkpoints, and six EdgeBench tasks.All 109 enter the Scientific category scores and Agent Index; the ten checkpoints need not be statistically independent of their parent workflows.Source: §A.12 · p. 38, Table 10 · p. 38
  • Added-task scores are strict 0-or-100 outcomes, but execution budgets and tools are not fully matched, so the results are configuration-level observations rather than model-only measurements.The retained records include valid timeout outcomes; Seed 2.1 Pro’s 0/30 SWE result primarily reflects failure to enter the Claude Code/API tool loop.Source: §A.12 · pp. 38–39, Table 11 · p. 39
  • Six ToolChoice checkpoints require a task-id argument absent from the prompt, so their archived strict failures cannot be attributed solely to tool-selection ability.Exact upstream SWE-bench instance IDs and revision links are also needed for complete instance-level provenance.Source: §A.12 · p. 39

A.13 EDGE TASK AGGREGATION AND RAW SCORES

The EdgeBench appendix assigns six tasks to reporting categories and specifies aggregation conventions and evidence limits. A historical tool-selection example illustrates a passing checkpoint decision, not a completed repair.

A.13 EDGE TASK AGGREGATION AND RAW SCORES

  • The six EdgeBench tasks are assigned across Mathematics & Computation and Physical Sciences, but those reporting assignments do not establish disciplinary exclusivity.Wireless Electricity Layout, BipedalWalker Locomotion RL, and Graph Node Classification go to Mathematics & Computation; Borden, Molecular Self Assembly, and DABIC go to Physical Sciences.Source: §A.13 · p. 39
  • For Borden and DABIC, Dual Exam clips raw scores to [0, 100] for aggregation; this is not an official EdgeBench normalization rule or evidence of equal task difficulty.Astra’s Borden raw score of 100.493 contributes 100 to the main table.Source: §A.13 · p. 39
  • Aggregation retains the best valid submission within 43,200 seconds, including valid early-termination results; confirmed absence of a valid submission receives zero, while missing records are not automatically zero.DeepSeek and MiMo Wireless passed 50/50 tests despite stopping at context or resume limits, with normalized scores of 19.9172 and 0, respectively.Source: §A.13 · p. 39
  • The added-task endpoints and evaluator-version equivalence are not all independently verified, and Astra’s native Codex executor means comparisons are between model–harness configurations.The transferred trace archive is truncated.Source: §A.13 · p. 40
  • The workflow examples are qualitative illustrations, not representative samples of the 109 Scientific tasks or current leaderboard results.Their historical source-collection counts and scores are not current leaderboard denominators or results.Source: §A.13 · p. 40

A.14 HISTORICAL WORKFLOW EXAMPLE: TOOL SELECTION

  • The tool-selection checkpoint scores 1.0 for choosing the specified source-reading tool, path, and line range, but the tool is not executed and no repair follows.The checkpoint permits one call and rules out another search or an immediate edit.Source: §A.14 · p. 40

A.15 HISTORICAL WORKFLOW EXAMPLES: EXTENDED EXECUTION

The historical workflow examples illustrate Long Horizon tasks that require dependent actions and verifiable outputs, plus a ToolChoice checkpoint requiring one constrained source read. Their recorded outcomes include successful numerical-file generation and a tested code repair, while the ToolChoice response earns credit for the proposed action without execution.

Long Horizon

  • Long Horizon tasks require carrying dependent actions through to a verifiable output, exemplified by numerical results and a tested software patch.The historical collection contains 147 ALE tasks and 30 scientific code-repair tasks from SWE; both workflows rely on intermediate checks.Source: §A.15 · p. 40

L1: Ising computation from specification to evaluated files

  • The Ising workflow produces four evaluated files after resolving a missing dependency, with an eigenvector residual of 8.093 × 10−13 and probabilities summing to one.A second calculation differs by less than 5.71 × 10−4, within the 10−2 public tolerance; the final files receive 1.0, although the check shares helper code with the generator.Source: §A.15 · pp. 40–41, Figure 19 · p. 42

ToolChoice-S02

  • The ToolChoice response receives 1.0 for one source-read call matching the supplied path and line bounds, but the evaluator does not execute it.The checkpoint says surrounding state transitions are still needed and prohibits another search or editing from the truncated snippet.Source: Figure 18 · p. 41, §A.15 · p. 41

L2: Propagating a tuned parameter into a DFT calculation

  • The DFT repair makes a tuned parameter affect the calculation and earns reward 1 with all three verifier tests passing.The energy difference is approximately −0.34018629 Hartree; the result establishes parameter influence, not comparative physical accuracy.Source: §A.15 · pp. 41, 43, Figure 20 · p. 43

B DETAILS OF WORLD MODEL EVALUATION

The World Model examples use fixed instructions and shared checkpoints to inspect requested effects alongside preservation of identities, objects, and changing rules. The cases show why plausible individual images or actions are insufficient evidence of sustained task-state consistency.

Evaluation scope and comparability

  • Figures 21–26 use six shared instructed-step checkpoints per case, with rows following models and columns matching steps; the starting reference image is not a complete ground-truth trajectory.Image and video configurations provide different temporal evidence, and historical runs with differing histories or rubrics do not support controlled cross-model ranking.Source: B DETAILS OF WORLD MODEL EVALUATION · pp. 43–44

Rule changes: single pendulum

  • The pendulum case changes gravity without requesting immediate motion, then tests continued evolution and apparatus preservation; static checkpoints cannot verify the specified oscillation period.Yume’s archived step-4 image is carried forward from the previous state, not new evidence of its response to the gravity change.Source: B DETAILS OF WORLD MODEL EVALUATION · p. 44, Figure 21 · p. 46

Participant-specific actions: painting-studio mentorship

  • The mentorship case tests whether the named mentor changes gaze and hand while the student responds, then lowers only the hand while retaining gaze toward the student.Framing or appearance changes can make identity and pose checks difficult, and selected states do not establish response timing within a video.Source: B DETAILS OF WORLD MODEL EVALUATION · p. 44, Figure 22 · p. 47

Coordinated actions: dual-arm cup manipulation

  • The dual-arm case checks concurrent actions, hand-over, role exchange, and whether cup identities and assigned arm actions persist; end-state images cannot establish action order.The EchoWM and Yume rows introduce human hands, an identity change distinct from an incorrect cup arrangement.Source: B DETAILS OF WORLD MODEL EVALUATION · p. 44, Figure 23 · p. 48

Memory across interventions: dual trays

  • The dual-tray case tests memory across a 20-step task with changing rules, arm roles, and camera views, including recovery of the first bench’s state after a visit elsewhere.Rule changes at steps 5 and 18 do not request immediate motion, and the final image must still be checked against the standing goal.Source: B DETAILS OF WORLD MODEL EVALUATION · p. 44, Figure 24 · p. 49

Local actions and persistent state: Minesweeper

  • Minesweeper checks whether specified cells change while earlier flags, unrelated cells, and board layout persist; a stable-looking board still needs cell-level inspection.The selected steps omit step 2, which flags h1; displayed failures range from changed geometry to replacement of the interface.Source: B DETAILS OF WORLD MODEL EVALUATION · pp. 44–45, Figure 25 · p. 50

Search and filtering: Google Shopping

  • Google Shopping tests query entry, filtering, and menu opening while requiring earlier values and unrelated interface regions to persist.Readable product imagery alone does not establish correct search, filtering, or preservation of selections.Source: B DETAILS OF WORLD MODEL EVALUATION · p. 45, Figure 26 · p. 51

C EXECUTION PROVENANCE AND EVIDENCE BOUNDARIES

This section distinguishes execution provenance from evidence that supports a score or verdict. It identifies reference-environment audit gaps, configuration and task-record limitations, and additional uncertainty evidence needed for reproducible interpretation.

Reference conditions and validation coverage

  • The evaluation lacks a unified independent error audit across reference environments, and expert specification review provides no agreement estimate for scores assigned to generated outputs.A test pass certifies tested behavior, whereas an observation or task score does not establish every latent property; missing audit coverage is not evidence of zero reference error.Source: C EXECUTION PROVENANCE AND EVIDENCE… · p. 52, Table 13 · p. 52

Scientific configuration identifiers

  • Configuration identifiers do not recover immutable provider versions or establish that the same harness was used across benchmark families.Historical WM illustrations retain recorded configurations and are not evidence of model switching in the fixed-pair pilot.Source: C EXECUTION PROVENANCE AND EVIDENCE… · p. 52

Exclusions and retained failures in the added tasks

  • Retained added-task records include observable tool-use failures and timeouts as scored outcomes, but their audit facts do not determine invalid-run rates across the rest of the benchmark.All 30 Seed runs produced thinking without task tools, and 26 Kimi runs had interrupted tool execution; the extension retains 29 timeouts and one Astra tool-cap termination.Source: C EXECUTION PROVENANCE AND EVIDENCE… · p. 52

Known scoring and provenance limitations

  • Six ToolChoice checkpoints lack a task-id argument required by their strict scoring schema, and the SWE-bench provenance lacks upstream instance identifiers.The original scores are retained for traceability; a corrected prompt would require consistent rescoring.Source: C EXECUTION PROVENANCE AND EVIDENCE… · pp. 52–53

Response, repair, progress, and completion

  • Corrective response records a targeted action, while successful repair requires later evidence of resolution and strict task success requires the complete goal and constraints.The pilot’s 25% Social response value averages eligible-event fractions, whereas 0% success counts fully successful cases; the measures have different denominators.Source: C EXECUTION PROVENANCE AND EVIDENCE… · p. 53, Table 14 · p. 53

Audit and uncertainty still needed

  • The paper reports no independent stratified WM score agreement, repeated-run confidence interval, or full invalid-run audit.The proposed reproducible audit binds cases to sources, specifications, rubrics, configurations, traces, scores, and exclusions; resampling should account for shared templates and parent links.Source: C EXECUTION PROVENANCE AND EVIDENCE… · p. 53

D PRELIMINARY STUDY PROTOCOL AND PACKING-CASE DIAGNOSTICS

The preliminary protocol fixes one model pairing, case distribution, and interaction budget, then retrospectively audits a packing rollout. Its diagnostics separate state consistency and corrective response from task success, while noting limitations of the review and score curves.

Study configuration and interaction budget

  • The pilot uses GPT-5.6 Sol for actions and Seedance 2.5 for observations across ten scenarios, with 15 cases per scenario.Each case runs once for at most 20 interaction steps, without episode-level retries; initialization and technical failures are excluded from the 150-case denominator.Source: D PRELIMINARY STUDY PROTOCOL AND… · p. 53

Response versus repair

  • Corrective response is a targeted action after a visible discrepancy, not evidence that the required state was restored or the task succeeded.Response rates cannot be interpreted as recovery ability because discrepancy types and response opportunities differ; ineffective responses may still receive credit.Source: D PRELIMINARY STUDY PROTOCOL AND… · p. 54

Evidence and scope

  • Figure 2(b) documents one 14-step rollout reviewed retrospectively, not an aggregate over the pilot collection, and its failure assessment relies on later use of a different blue pouch.The rubric was specified after inspecting the case and labels were not independently adjudicated; the review does not certify the original runner’s technical validation.Source: D PRELIMINARY STUDY PROTOCOL AND… · p. 54

Packing-case checks and response windows

  • The packing-case audit assesses backpack identity, acquired-item persistence, and task-conditioned actions, with 6 of 19 assessable checks passing and 2 of 7 eligible discrepancy events receiving a response.Of 42 candidate checks, 22 are unknown and excluded; both responses are navigation retries, which do not establish repair.Source: D PRELIMINARY STUDY PROTOCOL AND… · p. 54

Curve construction and interpretation

  • The plotted scores use fixed full-trajectory denominators and cumulative deductions, so their monotonic decline does not demonstrate monotonic capability degradation.The initial score of 1 is assigned rather than measured; treating two doorway-only navigation outcomes as unknown changes final rates to 6/17 = 0.353 and 1/5 = 0.200.Source: D PRELIMINARY STUDY PROTOCOL AND… · p. 54

Related papers

Loading 2610.31066v1…