PaperlayerView exact revision on arXiv

Source-linked AI summary

GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Nicholas Saban, Charvi Bannur, Mariya Vasileva, Darshan Deshpande

arXiv:2610.22708v1cs.AIcs.CL

TL;DR

Text world models could simulate environment transitions, but no standardized benchmark compares their general simulation capabilities or attributes failures to specific skills. GlobeBench evaluates them on execution-derived next states across multi-turn tasks, using deterministic checks and a semantic judge. Accuracy falls sharply when actions must change unnamed, dependent entities, and 631 tasks remain unsolved by any evaluated model.

  • Problem

    Text world models show promise as environment simulators, but no standardized benchmark compares them or attributes failures to specific simulation capabilities.

    Source: §1 · p. 1

  • Method

    GlobeBench provides 1,632 multi-turn tasks across five domains, comparing predicted next states with execution-derived golds using deterministic gates and an LLM judge for unsettled cases.

    Source: §1 · pp. 1–2

  • Results

    Mean success falls from 43.49% at L0 to 5.10% at L3, and no model solves 631 of the 1,632 items.

    Source: §6.1 · pp. 7–8

  • Takeaways & Limitations

    GlobeBench identifies where models depart from environment behavior, including cases where actions require updates to dependent objects or tracking hidden state over long histories.

    Source: §7 · p. 10

    Each item tests only one transition after a fixed prefix, so the benchmark does not measure how a simulator’s errors compound over a rollout.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment's response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent's changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as a world model.

1 INTRODUCTION

Text world models may serve as environment simulators, but evaluating their general simulation capabilities remains difficult because metrics are poorly understood and no benchmark isolates specific failures. GlobeBench addresses this with skill-based, execution-grounded tasks and a hybrid grader; its evaluation finds that no model solves even half the benchmark.

1 INTRODUCTION

  • No benchmark attributes world-model failures to specific simulation capabilities, despite the risk that agents trained on exploitable simulator errors optimize against incorrect dynamics.The paper notes that good evaluation metrics are also poorly understood.Source: §1 · p. 1
  • GlobeBench provides 1,632 multi-turn traces across five domains, each testing one skill against an execution-derived gold next state and quality-checked for consistency, reproducibility, ambiguity, and duplication.The authors assess dataset quality with a human study of 200 items and an LLM-judge audit.Source: §1 · p. 1
  • A hybrid grader settles predictions that deterministically match or contradict the gold and sends the remaining cases, which require judgment, to an LLM judge.The paper studies whether these automated judgments track human judgments.Source: §1 · p. 2
  • No model solves even half the benchmark, and aggregate success rates hide where failures occur and how the leading model’s margin arises.The introduction reports that the lead model’s margin comes almost entirely from items the other models never solve.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates text world models through fidelity, consistency, scaling, robustness, downstream utility, and rubric-based simulation judgments. GlobeBench instead constructs items to isolate one environment skill and records reach-back and impact radius as difficulty-related axes.

2 RELATED WORK

  • Li et al. (2026) evaluate fidelity and consistency, scaling and robustness, and downstream utility, finding usefulness depends on behavioral coverage and environment complexity.Source: §2 · p. 2
  • Deshpande (2026) frames text world modeling as predicting steerable transitions from state, task context, tool schema, domain rules, and steering directives.The work compares output-state diversity from autoregressive and masked-diffusion models.Source: §2 · p. 2
  • Executable sources include stateful tool use, TextWorld games, tool-using conversations, and symbolic backends; the paper evaluates predictions of their behavior rather than agents acting in them.These sources include BFCL, τ 2-bench, and SymPy.Source: §2 · p. 3
  • GlobeBench differs from prior whole-observation evaluations by isolating one environment skill per item, discarding items solvable without exercising that skill, and recording reach-back and impact radius.These axes are used to describe result slices, not as controlled variables.Source: §2 · p. 3
  • AgentWorldBench (Zuo et al., 2026) grades predicted observations across five rubric dimensions, while EnvSimBench (Liu et al., 2026) scores returned text separately from resulting state changes.Source: §2 · p. 3

3 GLOBEBENCH CURATION

GlobeBench items pair a visible interaction history and final action with an execution-derived next-state target, organized by capabilities, skills, domains, reach-back, and impact radius. Quality gates and audits are used to check whether items measure their intended behavior.

3 GLOBEBENCH CURATION

  • Each item is a visible trajectory P and final action a paired with next state g, obtained by executing the action against a fixed backend; the model sees only P and a.The fixed backend keeps the gold target from drifting.Source: §3 · p. 3
  • The four capabilities organize broad world-model behaviors, while each item tests one specific environment skill; skills can map to multiple capabilities.The capabilities are State Coherence, Causal Correctness, Interface Fidelity, and Constraint Enforcement.Source: §3 · p. 3
  • The 1,632 items span five domains, with 1,137 constructed by the authors and 495 adapted mainly from executable open-source backends.Domain counts reflect the available backends rather than a balanced design.Source: §3 · p. 3
  • Trajectory operators target a skill, fix the needed action sequence, and can add filler actions to vary how far back the deciding evidence lies.Operators may build trajectories from scratch or inject a controlled change into an adapted trajectory; final actions are executed against the fixed backend.Source: §3 · p. 4
  • Reach-back measures turns to the earlier event supporting an answer, while impact radius tracks how far an action’s state change extends, from no change at L0 to unnamed entities at L3.Most L3 labels are operator-declared, and nearly all such cases are synthetic long-horizon audits; the axes describe data slices rather than controlled variables.Source: §3 · p. 4
  • Twelve quality gates check item observability, reproducibility, solvability, and ambiguity, followed by human audits of 200 items and LLM-judge ratings for 1,627 items.The 80 items shared by all six human annotators include few long-horizon audits.Source: §3 · p. 4

4 EVALUATION SETUP

The evaluation compares 18 general-purpose LLMs and two specialized text world models, using a hybrid grader that combines deterministic comparisons with an LLM judge. Provider refusals and unusable responses affect the number of scored model-item pairs.

4 EVALUATION SETUP

  • The study evaluates 18 general-purpose LLMs and two specialized text world models, sending each one primary request per item.General-purpose models run at their providers’ high reasoning setting.Source: §4 · p. 4
  • 28,854 of 32,640 model-item pairs have usable answers; provider refusals account for 3,696 of the 3,786 pairs without usable answers.Four Claude models account for the provider refusals.Source: §4 · p. 4
  • Table 1 reports task success across all 1,632 items and 20 models, with provider refusals scored as failures.Specialized text world models are marked separately.Source: Table 1 · p. 5
  • The frozen grader uses deterministic gates for exact matches and contradictions, then sends unsettled responses to GPT-5.6 Terra, with GPT-5.6 Sol as fallback.On 130 provisionally adjudicated boundary examples, Terra agreed with adjudication more often than Sol: 79.2% against 74.6%.Source: §4 · p. 5

5 RESULTS

GPT-6 Astra leads overall, but success varies sharply with task demands and model group. The results also show distinct failure profiles and a stricter LLM audit of State Coherence than human annotators.

Overall performance

  • 49.57% task success makes GPT-6 Astra the leader, ahead of Grok 4.6 (28.13%), Claude Opus 5.5 (26.35%), and Qwen 3.8 Max (25.31%).The specialized models Qwen-AgentWorld and DWM rank 15th and 19th, respectively.Source: §5 · p. 5

Failure types

  • 58.22% of open-weight answered failures involve wrong values, versus 47.69% for closed models; closed models more often omit or add fields, 31.54% versus 16.27%.Across all 20 models, wrong-value, structure-only, and no-comparable-fields failures account for 53.21%, 22.95%, and 23.84% of answered failures, respectively.Source: §5 · p. 5, Table 2 · p. 6

Impact radius

  • 5.10% mean success at L3 contrasts with 43.49% at L0, while L1 and L2 are similar at 27.12% and 27.54%.Every model scores lowest at L3, where an action changes at least one unnamed entity.Source: §5 · p. 6, Figure 3 · p. 7

Human and LLM judgments

  • The LLM audit matches annotators on Interface Realism and Outcome Correctness means but is stricter on State Coherence, rating 9 items below 4 that every annotator rated 5.Annotators agree on every State Coherence rating; most disagreement is on Interface Realism, where 80.8% of annotator pairs agree.Source: §5 · pp. 6–7

6 ADDITIONAL ANALYSIS

Models struggle most when actions must update dependent entities and when answers rely on distant history. Skill and operator mix complicate reach-back comparisons, while qualitative examples locate many leading-model gains in hidden-state tracking.

6.1 QUANTITATIVE RESULTS

  • GPT-6 Astra leads 11 of 19 skills, including 7 of the 9 hardest; six models lead at least one skill, and 631 items are unsolved by all 20.Of the unsolved items, 87.6% are L3, 93.7% exceed 40 turns of reach-back, and 85.1% come from six skills.Source: §6.1 · p. 7
  • 43.49% mean success at L0 falls to 5.10% at L3, where actions must also change unnamed entities; L1 and L2 score 27.12% and 27.54%.Almost all L3 items are synthetic long-horizon audits, so impact radius and reach-back overlap in this subset.Source: §6.1 · p. 8
  • 42.3% pooled success within 40 turns falls to 11.4% beyond 40; in bins beyond 400 turns, the other 19 models stay at or below 8.31%.GPT-6 Astra’s curve rises again beyond 600 turns, a pattern related to operator mix rather than distance.Source: §6.1 · p. 8, Figure 4 · p. 8
  • Reach-back is associated with skill (η=0.69; Cramér’s V=0.54), and controlling for skill shrinks the short–long gap from 30.84 to 25.11 points.The gap disappears within operators (+1.76 points, 95% CI −1.57 to 5.17), but those comparisons omit the longest audits; L3 items show about 40% success on 14 short items versus about 4% on long ones.Source: §6.1 · p. 8
  • GPT-6 Astra passes 34.29% of L3 items and 40.52% beyond 40 turns, versus 8.39% and 16.36% for the next-best model, Claude Opus 5.5.Astra’s long-horizon lead is concentrated in audits requiring hidden-state tracking; it still fails 20 of 76 audits of one weighted-quorum type.Source: §6.1 · p. 8, §6.2 · p. 9

6.2 QUALITATIVE ANALYSIS

  • In a 720-turn audit, Astra derives a hidden capability value of 89 while Grok 4.6 uses 262; the resulting error reflects tracked state, not arithmetic.The capability is changed 49 times without being printed in the intervening trajectory.Source: §6.2 · p. 9
  • Several models produce parseable JSON on at least 99% of JSON items, so their failures are mostly well-formed answers with the wrong state; specialized models show no clear advantage.Qwen-AgentWorld’s margin over the median includes zero, and its results do not isolate the effect of next-observation training because its base model was not evaluated.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion presents GLOBEBENCH as a five-domain, 1,632-item benchmark and reports persistent failures on dependent-state and long-history tasks. It also bounds what its single-transition evaluation establishes and documents the paper’s AI use, ethics checks, reproducibility materials, and capability-skill framework.

7 CONCLUSION

  • GPT-6 Astra solves 49.57% of items, while 631 remain unsolved by all 20 models; accuracy is lowest for dependent-object changes and reach-back beyond 40 turns.The benchmark tests one transition after a fixed prefix, uses binary whole-state success, and does not measure how simulator errors compound over a rollout.Source: §7 · p. 10
  • Because reach-back and impact radius largely follow the skill families used to build items, these analyses locate difficult item families but do not show that distance or breadth causes errors.A high score also does not establish simulator reliability for training.Source: §7 · p. 10, ETHICS STATEMENT · p. 10

AI USE STATEMENT

  • LLM coding agents assisted with the data pipeline, evaluation harness, analyses, and synthetic-item operators, but not with proposing hypotheses or developing the capability and skill taxonomies.Every gold next state came from executing an action against a pinned backend; reported numbers were recomputed from saved outputs with tested scripts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The authors screened released items for personal data and had six annotators audit 200 items for identifying, sensitive, and harmful content.The benchmark items draw on five domains and public sources, with source and license recorded for every item.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The release includes benchmark items, the evaluation harness, reproduction instructions, model outputs, and judge records, enabling rescoring without querying models again.The paper also documents backends, item provenance, model settings, grader prompts, audit procedures, and result intervals.Source: REPRODUCIBILITY STATEMENT · pp. 10–11

A.1 CAPABILITIES AND SKILLS

  • The framework separates four broad capabilities from 19 testable skills, with each skill mapped to one or more capabilities and given construction axes and an item count.Items test one primary skill but may carry overlapping capability and failure labels, so label-slice counts should not be added together.Source: §A.1 · p. 15, Table 5 · p. 15

A.2 DOMAINS AND SOURCE DATA

GLOBEBENCH spans five distinct environment domains, with recipes and operators shaping trajectories that are executed against fixed backends. Across the reported radius and model comparisons, success varies substantially by item grouping and model.

A.2 DOMAINS AND SOURCE DATA

  • Five domains differ in state representation, available actions, errors, and executable backends; Code has 526 items (32.2%) and Dialogue 193 (11.8%).Items may carry several labels but belong to exactly one domain.Source: §A.2 · p. 16
  • Gold next states come from executing the final action against pinned backends, and corpus items must pass deterministic and judge-based quality gates.Container backends are fixed by image digest and run without network access; harness libraries are pinned by dependency or lock-file hashes.Source: §A.2 · p. 16, §A.5 · p. 19

A.3 TRAJECTORY CONSTRUCTION

  • Recipes specify a domain’s backend, entities, trajectory parameters, and allowed actions, while operators arrange action steps and may add filler turns.Independent random streams vary objects and values without changing the tested skill; some operators generate trajectories from seeds, while others adapt testbed trajectories.Source: §A.3 · p. 18

A.4 SOURCES OF IMPACT RADIUS LABELS

  • On measured-radius items, pooled success shows no steady trend from L0 to L3 (43.49%, 49.17%, 36.93%, and 51.88%); declared L3 items pass at 3.92%.The 638 measured-radius items differ from the 953 declared L3 items, which drive the pooled L3 success rate of 5.10%.Source: §A.4 · p. 19

B.1 MODELS AND SERVING

  • Qwen 3.8 Max scores 25.31% versus 23.22% for Flash Next, but each model passes items the other fails.Flash Next passes 112 items that Max fails, while Max passes 146 that Flash Next fails.Source: §B.1 · p. 20
  • GPT-6 Astra leads Grok 4.6 by 350 items, or 21.45 points (95% CI 19.12 to 23.77).Each model answered each item once; the reported bootstrap intervals reflect item choice, not variation across repeated responses.Source: §B.1 · p. 20

B.2 REFUSALS AND NON-RESPONSES

Refusals and unusable completions count as failures, while the hybrid grader combines deterministic checks with judges for unsettled responses. The paper also reports how judge agreement and refusals shape interpretation of the measured scores.

B.2 REFUSALS AND NON-RESPONSES

  • 3,696 provider refusals and 90 empty or errored completions count as failures, leaving 28,854 usable responses out of 32,640 model–item pairs.For 3,674 refusals, providers cited restrictions on violative cyber content; the paper says these safeguards can also block benign work, and the stated reason does not identify the triggering item content.Source: §B.2 · pp. 20–21
  • Items beyond 40 turns score 26.67 points lower within skill after excluding the four refusing models, while the within-operator gap disappears at +2.72 points.The respective 95% confidence intervals are −29.63 to −23.71 and −1.05 to 6.58.Source: §B.2 · p. 21

B.3 HYBRID GRADING

  • The hybrid grader settles equivalence or contradiction with deterministic metrics first and sends responses those metrics cannot settle to an LLM judge.Task success is binary under the frozen grader; domain, skill, reach-back, and radius results are diagnostic slices, not a weighted capability score.Source: §B.3 · p. 21, Table 10 · p. 22

B.4 JUDGE CONFIGURATION

  • GPT-5.6 Terra matched 103 of 130 provisional adjudications (79.2%), while Sol matched 97 (74.6%); the judges agreed on 96.2% of jointly scored responses.The authors retain deterministic decisions where possible and Sol as fallback; the adjudication comparisons used an earlier response prompt.Source: §B.4 · p. 22

B.5 JUDGE PROMPTS

  • Validity prompts test whether gold is determined by the visible state, ambiguity prompts test for equally defensible alternatives, and tell prompts detect surface cues to injected faults.The response prompt asks for semantic equivalence against executed gold and separates semantic correctness from contract compliance.Source: §B.5 · pp. 23–24

C.1 HUMAN EVALUATION

The human evaluation had six industry experts rate execution-derived gold next states across a sampled set of benchmark items. Ratings used four dimensions, while the sample covered the domains and skills unevenly relative to the full corpus.

C.1 HUMAN EVALUATION

  • Six industry experts independently rated 200 sampled items, with each rating 100 and all six sharing 80 anchor items for agreement measurement.The study assesses benchmark gold answers, not model outputs.Source: §C.1 · p. 24
  • The 78 released anchor items cover all five domains and 19 skills, but include only 6 L3 items versus 977 L3 items in the corpus.Their median reach-back is 101 turns, compared with 240 turns for the corpus; the LLM extension rates 1,627 items and covers the audits.Source: §C.1 · p. 24
  • Annotators scored State Coherence, Interface Realism, Outcome Correctness, and Constraint Enforcement from 1 to 5, marking the last dimension not applicable for valid actions.Coherence and realism were rated before showing the expected output; correctness and constraint enforcement were rated afterward.Source: §C.1 · pp. 24–25, Table 12 · p. 26
  • Annotators noted strong constraint adherence in rated next states and raised concerns about error reasoning and whether long distractor-filled trajectories provide a reliable signal.These observations were comments beyond the ratings.Source: §C.1 · p. 24

C.2 FULL-CORPUS LLM EXTENSION

The full-corpus audit found high completion but some judge–annotator disagreement, while benchmark success varied with impact radius and reach-back. Apparent reach-back trends also reflected which operators populated each distance band.

C.2 FULL-CORPUS LLM EXTENSION

  • 1,627 of 1,632 items received complete two-call LLM audit ratings (99.69%); the judge matched annotators on Interface Realism and Outcome Correctness but was stricter on State Coherence.State Coherence had the lowest full-corpus mean (4.452); the judge marked 392 items below 4 on at least one dimension for manual review.Source: §C.2 · p. 26, Table 13 · p. 27

D ADDITIONAL RESULTS

  • Pooled success is the mean of the 20 per-model rates because every model is scored on every item; confidence intervals resample items while retaining all models’ decisions.The benchmark pools 18 general-purpose LLMs and two specialized text world models.Source: D ADDITIONAL RESULTS · p. 27

D.1 DIFFICULTY AND STATE DEPENDENCY

  • 43.49% mean success at L0 fell to 5.10% at L3, where actions change unnamed entities; all 20 models scored lowest at L3.GPT-6 Astra fell from 76.45% at L0 to 34.29% at L3, while only 10 models declined at every step.Source: §D.1 · p. 27, Figure 6 · p. 28
  • 42.28% pooled success within 40 turns fell to 11.44% beyond 40 turns, but success did not decline steadily across reach-back bands.The distance effect is difficult to isolate: 952 of 977 L3 items were also beyond 40 turns.Source: §D.1 · p. 28, Table 14 · p. 28

D.2 REACH-BACK DETAILS

  • GPT-6 Astra’s success dip at 400–600 turns and rebound at longer reaches tracked operator mix; seven of nine operator-weighted bin predictions were within 3.3 points of observed rates.Lifecycle and saga audits dominated bins beyond 600 turns, while four other operators dominated the dip bin; Astra scored 0% on those four at any reach-back.Source: §D.2 · p. 29
  • Astra passed 56 of 76 weighted-quorum audits and 50 of 56 atomic lifecycle-frontier audits, where the other 19 models pooled 0.3% and 0.5%, respectively.Astra also passed all 50 interleaved saga-compensation audits beyond 600 turns, where the other 19 models pooled 12.9%.Source: §D.2 · p. 29

D.3 COMPARISONS WITHIN SKILL AND OPERATOR

The apparent difficulty of long reach-back and large impact radius depends on how items are grouped. Comparisons within operators weaken or reverse some pooled patterns, and the available within-operator comparisons cover only a subset of operators.

D.3 COMPARISONS WITHIN SKILL AND OPERATOR

  • Reach-back varies strongly with item properties: operator group explains 81.64% of the variance in log(1 + reach-back), while reach-back and prompt length correlate at ρ = 0.94.The correlation with gold length is ρ = 0.67.Source: §D.3 · p. 31
  • Within primary skill, the reach-back gap shrank from 30.84 to 25.11 points; within operator it disappeared (+1.76 points, 95% CI −1.57 to 5.17).The operator-level comparison included only 21 of 62 operators with items on both sides of 40 turns.Source: §D.3 · p. 31
  • Within skill, the reach-back gap was concentrated in nine skills whose long items came from audit operators: success was 43.50 points lower beyond 40 turns.In the other nine skills, long items were not harder (+4.02 points, 95% CI −0.59 to 8.62).Source: §D.3 · p. 31
  • Within skill, the L3–L0 gap widened from 38.38 to 47.13 points, but within five operators it reversed: L3 items scored 12.09 points higher.Across those same 71 items, pooled L3 success was lower (51.88% versus 67.23%) because L3 items were more concentrated in lower-success operators.Source: §D.3 · pp. 31–32, Table 17 · p. 32

D.4 PRIMARY-SKILL PERFORMANCE AND COMPLEMENTARITY

Performance varied by primary skill, and model strengths were complementary: some items were solved by just one model, while 631 remained unsolved by all 20. Skill-level rates also reflected the mix of operators used to generate items.

D.4 PRIMARY-SKILL PERFORMANCE AND COMPLEMENTARITY

  • GPT-6 Astra passed 148 of 224 Access Control items, versus 20 for runner-up GPT-5.5, while pooled success was 6.63%.Access Control is dominated by long-horizon lifecycle audits.Source: §D.4 · p. 32
  • GPT-6 Astra led 11 skills, Qwen 3.8 Max led three, and four other models led one or two each; Rule Enforcement had three tied leaders.Counting tied leaders still gives six distinct skill leaders.Source: §D.4 · p. 32, Figure 8 · p. 33
  • At least one model solved 1,001 of 1,632 items (61.34%), while 631 were unsolved by all 20 models.Astra alone was the only solver on 199 items; 274 items were solved by exactly one model.Source: §D.4 · p. 34, Table 19 · p. 34, Table 20 · p. 34
  • Adding complementary models to Astra raised the share of items solved by at least one model from 49.57% to 58.82% after four additions.Claude Opus 5.5 added 79 items, followed by Qwen 3.8 Max, Claude Fable 5.1, and Gemini 3.7 Flash.Source: §D.4 · p. 34
  • Qwen 3.8 Max and GPT-5.5 solved 126 and 124 items, respectively, that the other missed, despite nearly equal totals of 413 and 411.At least one of the two specialized text world models solved 269 items (16.48%).Source: §D.4 · p. 34

D.5 RESULTS BY DOMAIN

Raw domain success differs substantially, but much of the gap tracks how many L3 items each domain contains. Adjusting for item characteristics changes the domain ordering.

D.5 RESULTS BY DOMAIN

  • 27.8% in Games and 25.5% in Enterprise make them the most tractable domains; Math/Science (11.7%), Dialogue (11.8%), and Code (16.2%) are the least.The domain bars cover 11 models; Table 21 gives rates for all 20.Source: §D.5 · p. 34, Figure 9 · p. 35
  • On L3 items alone, every domain scores between 4.3% and 6.9%, while Dialogue and Math/Science contain 84–87% L3 items versus 40–44% in Games and Enterprise.The domain gaps largely follow their share of L3 items.Source: §D.5 · p. 35
  • Enterprise is 9.36 percentage points above Code, which is 4.33 points above Dialogue and 4.43 points above Math and Science; within-tier intervals include zero.The 95% bootstrap intervals are 6.02 to 12.70 for Enterprise minus Code, 0.30 to 7.93 for Code minus Dialogue, and 0.97 to 7.91 for Code minus Math and Science.Source: §D.5 · p. 35
  • Every model performs best in Games or Enterprise, while 17 of 20 perform worst in Dialogue or Math and Science.The three exceptions perform worst in Code; the low Code rates of refusing Claude models mostly reflect refusals. Among 18 general-purpose LLMs, Games leads at 28.96%, followed by Enterprise at 27.50%.Source: §D.5 · p. 35
  • After adjustment for domain, radius, reach-back, and outcome type, Math and Science has the highest predicted rate (23.23%), just above Games (23.19%).Adjusted rates then rank Dialogue (21.42%), Code (19.44%), and Enterprise (16.00%).Source: §D.5 · p. 35

D.6 ERROR AND NORMAL OUTCOMES

Performance varies sharply by outcome type, and the apparent error-item advantage is substantially reduced when comparing similar items. The appendix also examines failure categories, specialized-model results, and a long weighted-quorum audit.

D.6 ERROR AND NORMAL OUTCOMES

  • 53.16% pooled success on error items versus 16.16% on normal items, but the adjusted gap falls to 7.43 points with a 95% CI spanning zero.Error items are mostly L0 probes (153 of 155) with median reach-back of 31 turns, versus 309 for normal items; the within-operator gap is 4.57 points.Source: §D.6 · pp. 35, 37
  • 67.89% of normal-outcome failures predict success but give the wrong state; another 14.08% have no usable response.A normal-outcome gold can still contain nested denials or error fields; heuristic matching found these in 29.38% of normal-outcome golds.Source: §D.6 · p. 37

D.7 FAILURE ANATOMY BY DOMAIN

  • Failure anatomy separates wrong-value errors from structure-only errors and cases with no state fields; GPT-6 Astra’s failures most often involve schema substitutions (30.86%) or wrong values only (30.01%).For refusing Claude models, more than half of failures have no usable response, reaching 86.52% for Claude Fable 5.Source: §D.7 · pp. 37–38, Table 23 · p. 38

D.8 SPECIALIZED TEXT WORLD MODELS

  • Qwen-AgentWorld scores 46.9% on Side-Effect Propagation versus a 39.1% median, but its advantages are not significant and do not extend to L2/L3 items.It exceeds the general-LLM median on five skills; removing refusing Claude models reduces that to two, and it falls below the median on combined L2/L3 items by 2.32 points.Source: §D.8 · pp. 38–39

D.9 WEIGHTED-QUORUM EXAMPLE

  • In the weighted-quorum example, Grok’s incorrect capability yields a suspended state and a stale-state denial, while the gold capability of 89 yields active and allows the request.The stale-state denial aborts Grok’s first batch, so it commits none of the 12 requests execution commits; its capability is wrong for all 12 resources. Across 76 such L3 audits, Astra passes 56, versus 3 for GPT-5.5 and 1 for Grok.Source: §D.9 · pp. 39–41
  • Across the 76 weighted-quorum audits, Grok’s wrong capability matches an earlier true resource value 31.11% of the time, versus 6.69% under a uniform-value comparison.The authors interpret this as consistent with failure to carry updates forward, although the specific example’s erroneous value never occurs in that resource’s history.Source: §D.9 · p. 41

D.10 SAMPLES BEHIND GPT-6 ASTRA’S LEAD

The sampled audits locate much of GPT-6 Astra’s lead in long L3 tasks requiring hidden-state tracking. The qualitative examples also show that plausible or structurally correct answers can fail on exact values and earlier constraints.

D.10 SAMPLES BEHIND GPT-6 ASTRA’S LEAD

  • 272 of Astra’s 350-item lead over Grok comes from 952 L3 items with reach-back above 40, where Astra passes 33.5% against the other models’ 2.2% median.Across quorum audits, Astra reports exact capability for 84% of requests versus 2–8% for peers; when peers get state right, their decisions are correct in 61–91% of cases, compared with over 99% for Astra.Source: §D.10 · p. 41
  • On 51 saga audits, Astra passes all 51 while Grok passes 29, despite peers getting 74–93% of balances right across the audits.A single incorrect balance can fail an item; on the sampled counterfactual and dynamic DAG families, peers get under 6% of revision counts right versus Astra’s 77–80%.Source: §D.10 · p. 42

D.11 QUALITATIVE PATTERNS

  • A plausible answer can miss an eleven-turn-old quota constraint, while a structurally correct database update can still fail on a one-unit value error.The quota item is passed by 13 of 20 models; the database-update item is passed by 14 of 20. In Games, 50.62% of answered failures have no state fields to compare, unlike Math and Science, where 39.32% combine wrong values with missing or extra fields.Source: §D.11 · p. 42

Related papers

Loading 2610.22708v1…