Qwen-AgentWorld: Language World Models for General Agents

• The paper frames world modeling as a missing complement to policy learning, enabling scalable, controllable simulation and next-state prediction within agents.Targeted perturbations can expose weaknesses rare in real environments, while prediction before acting can refine action selection and be internalized as future-oriented thinking.Source: §1 · p. 3
• Qwen-AgentWorld achieves superior simulation performance over existing frontier models.The paper also reports gains from simulator-based RL and downstream agent training after world-model warm-up.Source: §1 · p. 3
• Qwen-AgentWorld is presented as the first language world model to simulate seven agent environments through long chain-of-thought reasoning.It uses accessibility trees and UI view hierarchies, rather than pixel frames, for its three GUI domains; training proceeds through CPT, SFT, and RL with hybrid rubric-and-rule rewards.Source: §1 · p. 3
• AgentWorldBench evaluates simulation across seven domains using out-of-distribution interactions from frontier models on established benchmarks.It combines open-ended rubric judging across five dimensions with rule-based verifiers for deterministic checks.Source: §1 · p. 3
• Training with 4k simulated real-world OpenClaw environments yields gains on Claw-Eval and QwenClawBench.The authors also report that controllable simulations complement real-world interaction, with substantial gains on Tool Decathlon, MCPMark, and WideSearch.Source: §1 · p. 3
• Across seven agentic benchmarks, LWM training serves as a warm-up or auxiliary stage that provides a foundation for stronger downstream performance.It acquaints agents with environment dynamics and next-state prediction before downstream agentic RL.Source: §1 · p. 4

2.1 Terminology

• LWM training updates the world model through CPT, SFT, and RL; Sim RL trains a policy using the LWM as simulator, whereas Real RL uses a live environment.Source: §2.1 · p. 4
• One environment trajectory is a sequence of action–observation pairs, while an agentic trajectory is the agent’s full reasoning-and-action completion interleaved with observations.Environment trajectories can be extracted from agentic trajectories by removing agent reasoning, or collected directly from interaction logs.Source: §2.1 · p. 4

2.2 Unified Environment Trajectory Schema

• A shared environment-trajectory schema supports training one world model across seven domains despite varied state representations, with actions and observations encoding agent outputs and environment feedback.The system prompt combines a task description, action space, initial state, optional demonstrations, and simulation instructions for controllable conditions.Source: §2.2 · pp. 5–6

2.3 Language World Model

• An LWM is a conditional text generator that predicts the next environment observation from the system prompt, full interaction history, and current action.The target is the ground-truth next observation.Source: §2.3 · p. 6
• The shared objective spans stateless domains, where state is implicit in conversation history, and stateful domains, where internal state evolves with actions, exercising reasoning, knowledge, and long-context understanding.Source: §2.3 · p. 6

3 Training Recipe

• CPT injects environment knowledge, SFT activates next-state-prediction thinking, and RL sharpens output quality with hybrid rubric-and-rule rewards.The stages are trained end-to-end with environment modeling as the explicit objective from continual pre-training onward.Source: §3 · p. 7, Figure 5 · p. 7

3.1.1 Environment Trajectories Collection

• Training trajectories come from dedicated agent–environment infrastructure, cleaned public interaction traces, and in-house agentic trajectories covering all seven domains.Dedicated infrastructure supplies scalable, controlled, reproducible data; cleaned public traces add long-tail patterns that complement it.Source: §3.1.1 · pp. 7–8

3.1 Training Data

• CPT draws on collected infrastructure and open traces plus specialized-domain corpora, while SFT and RL use only internally accumulated trajectories; the three stage pools are disjoint.Because on-policy RL can amplify data artifacts, the authors invest most data-engineering effort in its downstream pipeline.Source: §3.1.1 · p. 8

3.1.2 Unified Data Processing

• The unified schema expands trajectories into turn-level prediction samples and filters short, invalid, failed, empty-action, retry-cycle, and GUI no-change turns.For training, one turn is randomly sampled per trajectory; retry-cycle pairs are skipped while preserving state history.Source: §3.1.2 · p. 8
• System prompts combine task, action-space, initial-state, demonstration, and simulation-instruction components, with automated optimization producing audited templates for distinct training pools.Simulation instructions can specify controllable conditions, while domain-specific prompt components and initial states vary across trajectories.Source: §3.1.2 · pp. 8–9

3.2 Stage 1: Continual Pre-Training

• CPT applies next-token prediction to environment trajectories and single-turn knowledge corpora, using domain expertise to ground simulations beyond the trajectory data alone.Specialized corpora span industrial control, cybersecurity, law, medicine, finance, current affairs, and encyclopedic knowledge.Source: §3.2 · p. 9
• CPT masks low-information turns from loss using action–observation overlap, novelty, Jaccard similarity, and length ratio, while retaining their tokens as context.Seven statistically defined, tool-agnostic categories determine keep ratios, separating learning the next state from learning boilerplate next tokens.Source: §3.2 · pp. 9–10, Table 3 · p. 10

3.3 Stage 2: Supervised Fine-Tuning

• SFT explicitly activates next-state prediction reasoning to apply CPT-learned state-transition knowledge, reducing hallucinations and improving consistency over long trajectories.Training retains the standard next-token-prediction objective and uses a 256k-token context window.Source: §3.3 · p. 10
• 7,094 trajectories are retained from 10,250 candidate queries, a 69.2% retention rate, after prompt diversification and best-of-three rollout rejection sampling.An independent judge scores candidates pairwise; queries are discarded when the winning trajectory falls below the minimum threshold.Source: §3.3 · p. 10, Table 4 · p. 11

3.4 Stage 3: Reinforcement Learning

• RL uses GSPO despite prompt–output asymmetry: trajectory-history prompts can span tens of thousands of tokens, while predicted observations are typically hundreds to thousands of tokens.The prompt is capped at 128k tokens, covering the vast majority of trajectories across all seven domains.Source: §3.4 · pp. 10–11
• Factuality improves by 11.3% during RL but remains the lowest-scoring dimension throughout training.The authors identify factual world knowledge as the hardest aspect of environment simulation.Source: §3.4 · p. 11

3.4.1 Reward Design

• The reward combines a five-dimension LLM rubric, scaled to [5, 25], with a rule-based binary verifier scaled to [0, 25] at a 9:1 rubric-to-rule ratio.The verifier provides an objective anchor against reward hacking; strict tag extraction ensures the judge sees only the predicted observation.Source: §3.4.1 · p. 11

3.4.2 Training Stability

• Restricting RL-pool expansion to one turn per trajectory prevents collapse caused by samples sharing long prefixes and prediction targets.The resulting samples have unique prediction targets with no shared-prefix overlap.Source: §3.4.2 · p. 11
• The five-dimensional rubric combined with a rule-based verifier converges stably, unlike reference-reward and Turing-test alternatives.The paper attributes this to orthogonal assessment dimensions and a binary anchor for cases with executable ground truth.Source: §3.4.2 · pp. 11–12
• The authors mitigate self-praise reward hacking with verifier-based correctness, content-type-aware judging, and strict separation of predicted content from reasoning.Self-praise in content classified as deterministic earns zero credit.Source: §3.4.2 · p. 12

4.1 Benchmark Construction

• The benchmark converts frontier-agent trajectories on established queries across seven domains into environment trajectories with real ground-truth observations, while source-level splits probe generalization beyond training data.Its construction principles also prioritize widely used queries and complex frontier-agent action sequences, including reasoning chains, tool-call compositions, and error recovery.Source: §4.1 · p. 12
• Five frontier agents ran on nine established query sets across seven domains; GUI coverage added three Qwen-family models, while later text-domain trajectories used Claude Opus 4.6.An early subset of Terminal-Bench 1.0 and the in-house SWE benchmark used Claude Sonnet 4.5.Source: §4.1 · p. 12
• Text trajectories retain five turns—the first, last, and three sampled intermediate turns—while challenging GUI turns are selected; a uniform 50% sample forms the final benchmark.The first turn anchors fidelity before interaction history accumulates, while the last probes the longest context and strongest dependence on prior state.Source: §4.1 · p. 13
• AgentWorldBench contains 2,170 samples: text domains account for 72.4%, with SWE at 21.8% and Search at 21.1%; GUI domains each contribute 9.2%.Average context length ranges from 12,900 tokens for Terminal to 59,300 for MCP, whose samples include full tool-definition schemas.Source: §4.1 · p. 13, Figure 6 · p. 13

4.2 Evaluation Protocol

• An LLM judge scores predictions on Format, Factuality, Consistency, Realism, and Quality, with the five-dimension mean scaled to [0, 100].Judges compare predictions with real observations and apply domain-aware rubrics; deterministic content must match exactly, while pre-existing content and runtime metadata are judged for plausibility or valid format and range.Source: §4.2 · pp. 13–14
• Across seven domains, judge rankings correlated at ρ = 0.92–0.99 (all p < 10−5); all judges ranked Format and Consistency strongest and Factuality weakest, so GPT-5.2 was selected.The judge candidates were Gemini 3 Flash, Claude Sonnet 4.5, and GPT-5.2. They were calibrated using a double-blind Turing test; GPT-5.2 was selected for its highest average Turing-test accuracy.Source: §4.2 · p. 14

5.1 Setup

• The experiments evaluate Qwen-AgentWorld-35B-A3B and Qwen-AgentWorld-397B-A17B against 14 baselines, using Qwen-family checkpoints to isolate the effect of world-model training.The Qwen baselines share the same base architecture but lack the three-stage CPT→SFT→RL pipeline.Source: §5.1 · p. 15

5.2 Main Results

• +8.66 points at 35B (47.73 to 56.39) and an increase from 54.74 to 58.71 at 397B show gains over the corresponding base checkpoints.At 397B, text- and GUI-domain averages rise by 3.35 and 4.92, respectively; Qwen3.6-Plus (50.81) and Qwen3.6-Max-Preview (52.42), which lack world-model training, score below Qwen-AgentWorld.Source: §5.2 · p. 15
• 58.71 is Qwen-AgentWorld-397B-A17B’s highest overall average, ahead of GPT-5.4 (58.25); it leads text-based domains but ranks fifth on GUI domains.Its text-domain average is 58.07 versus GPT-5.4’s 56.84, while Terminal scores are 57.73 versus 53.69 and SWE scores are 68.49 versus 66.29. On GUI domains, Claude Opus 4.8 (60.93) and Claude Opus 4.6 (61.12) lead; Qwen-AgentWorld-397B-A17B scores 59.69.Source: §5.2 · p. 15, Figure 7 · p. 15, Table 5 · p. 16
• Search is the most challenging domain, with a best score of 37.82 versus 68.49 on SWE and 70.10 on MCP; Claude Opus 4.6 and GPT-5.4 are tightly at the top on MCP.The passage attributes Search’s difficulty to evolving web content and challenges maintaining factual consistency across long retrieval chains.Source: §5.2 · p. 16

5.3 Cross-Domain Generalization

• +14.2 points on Terminal and gains on all three held-out text domains result from Stage 3 RL training on Terminal data alone.Held-out gains are SWE +11.5 (52.0 →63.5), Search +11.8 (20.2 →32.0), and MCP +5.0 (53.5 →58.5). Transfer emerges within the first 10 RL steps and remains stable; the authors suggest RL reinforces generalizable world knowledge rather than domain-specific output formats.Source: §5.3 · p. 16, Figure 8 · p. 16

6.1.2 Controllable Simulation

• Controlled Sim RL improved Tool Decathlon by +3.7 and MCPMark by +12.3, whereas uncontrolled Sim RL gave no meaningful gain and dropped Tool Decathlon from 32.4 to 31.5.The passage attributes the difference to controllable, grounded instructions that expose weaknesses such as API errors and incomplete results; controllability is described as a prerequisite for useful Sim RL in this domain.Source: §6.1.2 · pp. 18–19, Table 7 · p. 19
• Controllable fictional-world Sim RL raised 35B-model WideSearch F1 by Item from 34.02 to 50.31 (+16.29) and F1 by Row from 13.72 to 24.21 (+10.49).At 397B, gains were +3.87 for F1 by Item and +6.05 for F1 by Row. The entirely fictional worlds withhold complete answers in snippets, teaching query reformulation, cross-referencing, and iterative aggregation while preventing reliance on parametric memory or confusion with real-world facts.Source: §6.1.2 · pp. 18–19, Table 8 · p. 20
• 50.3% vs. 45.6%: controllable Sim RL matched or exceeded live-search Real RL on WideSearch F1 by Item while increasing web_extractor calls from 2.5 to 4.0 per trajectory.Real RL reduced web_extractor calls from 2.5 to 1.5; simulated snippets withheld details, encouraging full-page extraction and targeted behavior.Source: Figure 9 · p. 20, §6.1.2 · p. 20

6.2 Application II: Agent Foundation Model

• LWM RL warm-up improved all seven agentic benchmarks without subsequent fine-tuning, including three domains absent from world-model training.On in-domain benchmarks, WideSearch F1 by Item rose from 33.38 to 46.17 (+12.79), and Terminal-Bench 2.0 accuracy from 33.25 to 39.55 (+6.30). On out-of-domain benchmarks, Claw-Eval improved from 53.6 to 64.9 (+11.3), QwenClawBench from 39.8 to 49.4 (+9.7), and BFCL v4 from 62.3 to 71.3 (+9.0); SWE-Bench Verified resolve rate rose from 64.5 to 67.9 (+3.4), and SWE-Bench Pro from 42.2 to 47.4 (+5.2).Source: Table 9 · p. 21, §6.2 · p. 21
• Next-state prediction accuracy rose from 69.9% to 78.3% (+8.4%) after LWM RL, as the model learned to mentally predict environment responses before acting.On the mailman task, the warmed-up model correctly predicted that transport_maps alone would not fix Postfix recipient rejection and instead targeted local_recipient_maps; the pre-warm-up model did not.Source: Figure 10 · p. 22, §6.2 · p. 22

7.1 LWM Reasoning Patterns

• Across 129 turns, the model used 1,347 “Wait!” interrupts (10.4 per turn on average) to catch and revise intermediate predictions.Self-corrections include factual, epistemological, and perspective-taking types; the paper describes this as constrained satisfiability search.Source: §7.1 · p. 22
• The model prevents Search answer leakage by distinguishing what the agent knows from what the environment should reveal.When a query is unrelated to the reference answer, it avoids snippets that could accidentally expose that answer.Source: §7.1 · p. 22
• In Terminal, the model correctly chained six system-level causes to predict a specific JSONDecodeError before the agent executed the command.The chain runs from missing Node.js to an unstarted server, silent curl failure, empty piped input, and the resulting Python error.Source: §7.1 · p. 23

7.2 RL Enhances Micro-Level Simulation Fidelity

• By Step 200, RL produced a more realistic Search response, with a changed IMDB identifier, natural source ranking, and query-specific snippets.At Step 100, the identifier was tt2333444; by Step 200, it was tt2988794, with Wikipedia ranked before IMDB, NYT, and Rotten Tomatoes.Source: §7.2 · p. 23
• The model counted 53 bytes exactly by enumerating characters, including invisible newline bytes, and also respected hash invariants and Unix pipeline semantics.Identical files received identical SHA256 hashes, while tee to a nonexistent directory failed without affecting stdout.Source: §7.2 · pp. 23–24
• Across nine sequential MCP calls, the model preserved Notion user identifiers, parent-page references, and the full schema of roughly 20 fields per block.The paper characterizes this as a stateful in-context database maintaining referential integrity across nested JSON objects.Source: §7.2 · p. 24

World Models

• Prior visual and embodied world models learn environment dynamics for planning or policy learning, while Qwen-AgentWorld extends simulation to structured-text observations in agent environments.Examples of structured observations include API responses, terminal outputs, accessibility trees, and UI view hierarchies.Source: §8 · pp. 24–25

Language World Models

• Language-world-model research spans evaluation, trained simulators, and domain-specific agent methods; Qian et al. (2026) find agents fail to use world models strategically for foresight-based planning.This limitation motivates making environment modeling the training objective from CPT onward rather than adding it after training.Source: §8 · pp. 25–26
• Qwen-AgentWorld covers 7 domains with native CPT→SFT→RL world-model training, and studies both controllable Sim RL and LWM pre-training as an agent warm-up.Its CPT objective starts with environment modeling rather than post-hoc fine-tuning; the two paradigms decouple agents from simulators or unify them in one framework.Source: §8 · p. 26

Agent Foundation Models and Continual Pre-Training

• Qwen-AgentWorld shares continual pre-training on agentic data with AgentFounder (Su et al., 2025), but covers seven domains and adds SFT and RL to refine simulation fidelity.The passage contrasts this with AgentFounder’s 30B-parameter model and smaller domain scope.Source: §8 · p. 26

Synthetic Environment Generation

• Programmatic environment synthesis offers deterministic execution and verifiable rewards but is limited to specifiable domains, whereas LWM simulation trades determinism for broader coverage.The passage identifies search engines and real-world MCP servers as domains where code-based synthesis is impractical.Source: §8 · pp. 26–27

Conclusion

• Qwen-AgentWorld spans seven agent-interaction domains at 35B-A3B and 397B-A17B, surpassing uncontrolled simulation and real-environment training on three agentic benchmarks.As a unified agent foundation model, LWM warm-up also consistently improves downstream performance across seven diverse tasks via cross-domain transfer.Source: §9 · p. 27
• The three-stage recipe uses CPT to inject environment knowledge, SFT to activate next-state-prediction reasoning, and RL to sharpen simulation fidelity.AgentWorldBench pairs each sample with a ground-truth observation from a real environment.Source: §9 · p. 27

Future Work

• Future work proposes agent–LWM co-evolution, multimodal GUI-and-text world models, adaptive sim-to-real routing, and on-the-fly tool synthesis.The proposals include self-play that creates novel states and challenging scenarios, and routing each query between the model and real environment to balance cost and fidelity.Source: §9 · p. 27

A Domain Interaction Examples

• Each example shows an agent’s action and Qwen-AgentWorld’s simulated observation for one environment turn across the seven covered domains.Source: A Domain Interaction Examples · p. 35
• For OS and Web, the model predicts the next GUI state in HTML from the current screen and agent action, then renders it as a screenshot.Android examples appear in Section 2.3.Source: Figure 14 · p. 35
• The text-based examples cover Terminal, MCP Tool Use, and Search; Software Engineering examples appear in Section 2.3.Source: Figure 15 · p. 36

B Training Dynamics

• The analysis tracks Format, Factuality, Consistency, Realism, and Quality across 440 RL steps on Qwen-AgentWorld-35B-A3B, evaluating every 10 steps.Source: B Training Dynamics · p. 37
• Consistency gains +0.29 (from 2.81 to 3.10), versus Quality’s +0.11; Format and Quality reach 90% of total improvement within 90 steps, while Consistency and Realism require ∼250.Despite a single scalar reward averaging all five dimensions, improvement rates diverge; surface formatting patterns are reinforced quickly, while cross-turn state tracking develops gradually.Source: Figure 16 · p. 37, B Training Dynamics · p. 37
• Factuality improves by 11.3% relative to its initial value (2.03 to 2.26), the largest relative gain, but remains the lowest-scoring dimension throughout training.The passage interprets this as RL improving response accuracy while factual world knowledge remains the hardest aspect of environment simulation.Source: B Training Dynamics · p. 37

Verifier construction

• Test cases derive from the main evaluation set through trajectory-grounded synthesis and validation, with an LLM judge filtering incorrect outputs that could deceive rule-based verifiers.Candidate verifiers are checked against ground-truth observations and multiple model rollouts, and retained when verdicts agree with reference-grounded rubric labels.Source: C Rule-Based Verification · p. 37

Controllability (Ctrl)

• Controllability tests whether explicit simulation instructions alter generated observations as required while preserving the domain’s normal output format.Cases hold trajectory context fixed while adding control conditions, such as specified command failures, withheld intermediate information, or contrastive GUI conditions.Source: C Rule-Based Verification · pp. 37–38

Results

• 72.93 average score ranked GPT-5.4 first overall, while Qwen-AgentWorld-397B-A17B scored 67.12, ranked second, and outperformed all other frontier models on GUI domains.Table 10 reports per-domain verification accuracy across four text domains and three GUI domains.Source: Table 10 · p. 38, C Rule-Based Verification · p. 38

Error Handling (Err)

• Error handling checks whether the simulator reproduces deterministic failures, such as invalid operations, with an appropriate error signal rather than a plausible success.Search is excluded because its trajectories lack deterministic, executable error outcomes suitable for this axis.Source: C Rule-Based Verification · p. 38

Long-Context Consistency (LC)

• Long-context consistency checks whether later observations respect state changes or revelations from earlier actions across extended interaction histories.Cases include file write–read pairs, environment-variable updates, database mutations, and GUI actions that change later screens.Source: C Rule-Based Verification · p. 38

C.1 Per-Axis Breakdown

• Table 11 reports rule-based accuracy on controllability, error handling, and long-context consistency for text-based domains.Scores are reported as accuracy percentages, with higher values better.Source: §C.1 · p. 38, Table 11 · p. 39
• Table 12 reports rule-based accuracy on controllability, error handling, and long-context consistency for GUI domains.Scores are reported as accuracy percentages, with higher values better.Source: §C.1 · p. 38, Table 12 · p. 39

D Open-Ended Judge Prompts

• Across seven domains, the prompts score Format, Factuality, Consistency, Realism, and Quality, using content-type classification for text and domain-specific anchors for GUI observations.GUI anchors include UI elements, navigation state, and visible text.Source: D Open-Ended Judge Prompts · p. 39

D.1 Terminal Domain

• Terminal evaluation treats Ground Truth as the reference while requiring exact matches for deterministic, session-known output and only plausibility checks for unknown pre-existing state and runtime metadata.The simulator is assumed to know only information shown or created during the current session; plausible content for unknown state is allowed, but contradicting known session state is prohibited.Source: §D.1 · p. 40
• Terminal prompts assess formatting, factuality, consistency, realism, and quality, covering layout and prompts, state continuity, authentic behavior, completeness, and concision.Formatting is scored separately from content correctness, and every commendation or penalty must reference differences or matches with Ground Truth.Source: §D.1 · pp. 40–41

D.2 MCP Domain

• MCP prompts classify objective facts and structural metadata for exact matching, allow reasonable variance for numeric data, and check private or session data for validity and realism.They assess tool execution, hallucination, status accuracy, prior-turn state and resource references, behavioral patterns, completeness, and concision against real tool responses.Source: §D.2 · pp. 41–42

D.3 Search Domain

• Search prompts judge output structure and factuality against Ground Truth, check consistency across repeated searches or extractions, and assess retrieval relevance, ranking, completeness, and concision.Facts and sources present in Ground Truth must match; content not in it is penalized only when clearly contradicted by Ground Truth.Source: §D.3 · pp. 42–43

D.4 SWE Domain

• The available SWE prompt excerpt frames evaluation of simulated tool responses in a command-line and filesystem environment and begins classifying outputs by verification standard.It specifies exact Ground Truth matching for objective facts and reasonableness checks for session or environment-specific data; the excerpt also begins a category for private or unprovided context-dependent data.Source: §D.4 · p. 43
• SWE judging separates exact native formatting from verifiable correctness, requiring simulated outputs to match the Ground Truth on tool execution, errors, and deterministic content without contradicting known state.Plausible values are allowed for unknown environment state.Source: §D.4 · p. 43
• SWE predictions must preserve prior file, environment, and session state while reflecting authentic tool behavior and producing plausible, appropriately complete and concise outputs.The prompt also evaluates tool-specific response style, numeric reasonableness, error formatting, and edge-case handling.Source: §D.4 · pp. 43–44

D.5 Android Domain

• Android predictions are judged against the real screen after the latest action, prioritizing concrete visible anchors such as app, dialog, focus, typed text, controls, and scroll position.The judge penalizes stale or fabricated transitions and favors ground-truth anchors unless they clearly conflict with prior history.Source: §D.5 · p. 44
• Android scores reserve top ratings for the correct primary effect and near-complete anchor matches; wrong primary effects cap factuality and quality at 2, regardless of matching background details.Format, factuality, consistency, realism, and usefulness are scored independently, with realism capped at 3 when factuality or consistency is 1–2.Source: §D.5 · pp. 44–45

D.6 Web Domain

• Web predictions must show the actual post-action browser state, preserving anchors such as URL, page identity, visible text, form values, overlays, and scroll position rather than inventing navigation or content.The judge penalizes stale predictions and broad site-level matches that miss task-critical visible changes.Source: §D.6 · p. 45
• Web scoring reserves top ratings for correct primary effects and nearly all key anchors; when factuality or consistency is 1–2, realism is capped at 3.A wrong or stale primary effect caps factuality and quality at 2, and page-level correctness cannot compensate for incorrect visible facts.Source: §D.6 · pp. 45–46
• In the Web domain, when the state and visible result rows do not match, the total score should normally be <= 3.Source: §D.6 · p. 46

D.7 OS Domain

• For OS predictions, follow the user-provided evaluation protocol and compare the simulated desktop state after the latest operation with the real observation.A strong prediction preserves the trajectory’s concrete state representation and updates the visible state caused by that operation.Source: §D.7 · p. 46
• Treat ground truth as strongest evidence for visible desktop facts, except when it clearly conflicts with prior history, when causal reasoning from the interaction history should guide judgment.Check anchors such as the active window, dialogs, exact visible names, focus and selection, terminal output, file-manager state, viewport, and status text.Source: §D.7 · p. 46
• Penalize unsupported app, window, dialog, focus, or file changes; stale or fabricated states; and generic explanations that describe what should happen instead of the resulting observation.Broad correctness does not offset errors in task-critical anchors or small visible updates after state-changing actions.Source: §D.7 · p. 46
• Reserve top scores for the primary post-action effect and nearly all task-critical anchors; lower scores reflect missing details, plausible but wrong states, or malformed and fabricated outputs.The guidance distinguishes high-but-not-top, middle, low, and minimum scores by transition accuracy and visible-state fidelity.Source: §D.7 · p. 46
• Score format, factuality, consistency, realism, and quality independently, and do not let a well-formed desktop hierarchy inflate factual correctness.Realism is capped at 3 when factuality or consistency is 1–2; quality is low when missing or wrong anchors could lead a downstream agent to act incorrectly.Source: §D.7 · p. 46
• Before assigning 4 or 5, verify the primary action effect and active anchors; if that effect is wrong, stale, or missing, factuality and quality must be <= 2.If exact active text or paths, focus or selection, menus or dialogs, or visible rows and files mismatch, total score should normally be <= 3.Source: §D.7 · pp. 46–47