PaperlayerView exact revision on arXiv

Source-linked AI summary

Trace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training

Zezhou Wang, Hanzhang Zhou, Panrong Tong, Quyu Kong, Pengxiang Li, Steven Hoi, Zhuzhong Qian, Yue Wang

arXiv:2610.30792v1cs.LGcs.SE

TL;DR

Partial traces can lead GUI simulators to encode incorrect state transitions, while terminal-value checks may not establish workflow fidelity. Trace2World reconstructs a source-linked app model before implementation and validates packages through source replay; it improves workflow reproduction and retention of simulated gains on real devices.

  • Problem

    Coding agents that build GUI mocks from partial traces can commit to unsupported page identities, states, or action effects, while implementation-derived checks may not establish fidelity to the source app.

    Source: §1 · p. 2

  • Method

    Trace2World reconciles traces into a shared source-linked app model before coding, compiles it into a package, and uses independent source-workflow replay to guide bounded repair.

    Source: §1 · pp. 2–3

  • Results

    91.1% of workflows pass all six source checks versus 59.8% for the strongest matched task-local build, and source validation retains 94.0% of simulated gain on real devices versus 54.7% under self-validation.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    The results support separating evidence-backed app behavior from implementation so source workflows can determine whether a compiled world preserves it.

    Source: §1 · p. 3

    All 2,048 optimization tasks are single-app, and the construction roles use proprietary models.

    Source: §3.4 · p. 6, §E.5 · p. 48

Abstract

from arXiv · show

Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce \textbf{Trace2World}, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.

1 INTRODUCTION

GUI simulators can encode incorrect behavior when coding agents resolve ambiguities in partial traces during implementation, and final-value checks may miss wrong transitions. Trace2World reconstructs supported semantics before coding, then validates implementations against source workflows.

1 INTRODUCTION

  • Partial traces expose only one path from one initial state, so page-transition graphs may miss whether differences reflect distinct pages or state-dependent behavior.Matched builds reach 50.4–94.8% of endpoints but preserve only 16.7–59.8% of source workflows.Source: §1 · p. 2
  • A final-value check can pass even when an implementation skips the source app’s confirmation and saves a setting immediately.In Meituan, the source opens a confirmation after “Disable quick pay,” while the matched reconstruction disables it on that tap.Source: §1 · p. 2, Figure 2 · p. 2
  • Trace2World reconciles shared pages, entities, state, data, and action effects before implementation; unsupported or unresolved behavior remains non-executable.The resulting source-linked model is the semantic commitment boundary between evidence and browser code.Source: §1 · pp. 2–3
  • Bounded compilation and independent replay compare source workflows with the package, while bounded repair edits the package under a frozen model.Learner feedback later guides task composition from a read-only index without redefining app behavior.Source: §1 · p. 3, Figure 3 · p. 3
  • 91.1% of 200 workflows across ten apps pass all six source checks, versus 59.8% for the strongest matched task-local build.The environments are built from 1,570 partial traces across 50 shared worlds.Source: §1 · p. 3

2 RELATED WORK

Prior GUI environments differ in how their behavior is authored and how correctness is checked. Trace2World instead reconciles traces into a shared model before implementation and uses source-workflow replay for acceptance.

Environment construction and semantic fidelity

  • Existing approaches author behavior by hand, synthesize topic-based mocks or task-local sites, or build reusable Android mocks from screens and transitions.Their semantics are committed during implementation; PhoneWorld uses human QA on representative flows.Source: §2 · p. 3
  • Trace2World reconciles every app trace into one source-linked model before code exists, leaves unresolved conflicts outside executable scope, and accepts worlds through independent replay of source workflows.This changes both the source of environment semantics and the evidence used to establish correctness.Source: §2 · p. 4

Experience supply and evolving tasks

  • Online GUI RL, task synthesis, and learnability-driven curricula adapt training experience, while Trace2World connects these lines through a validated app world.Semantics are reconstructed and accepted once per app; learner feedback changes which supported behavior is practiced.Source: §2 · p. 4

3 TRACE2WORLD

Trace2World separates a shared, evidence-linked app model from its executable package and task-authoring index. It reconciles trace evidence, validates implementations against source workflows, and adapts tasks within supported behavior.

3.1 FROM TRACES TO ONE SHARED APP

  • A source trace records one successful real-app workflow, and Trace2World links shared pages, entities, state, data, and action effects to supporting steps in a common app model.Ambiguous or unsupported behavior remains non-executable; the model is reconstructed once and then frozen.Source: §3.1 · p. 4

3.2 RECONCILING APP SEMANTICS ACROSS TRACES

  • When observed state explains differing effects, the model records a condition; otherwise conflicting effects remain unresolved rather than being assigned one interpretation.Bounded shards and reductions preserve source links; across 50 apps the largest model-visible call is 33.2K tokens under a 40K-token budget, with 99.1% audited semantic coverage.Source: §3.2 · pp. 4–5, Figure 4 · p. 4

3.3 BUILDING AND VALIDATING THE ENVIRONMENT

  • Bounded API calls compile the supported model into a DOM-backed browser package whose central state kernel applies guarded updates.Independent replay compares source information, intermediate effects, and terminal state; repair edits only the package, cannot add unsupported behavior, and workflows needing model changes are excluded after three repair waves.Source: §3.3 · p. 5

3.4 LEARNER-GUIDED TASK GENERATION

  • A read-only index of validated surfaces, entities, operations, and dependencies supports learner-adapted task composition; unsupported transitions or effects are rejected.Supported gaps in content or implementation can be instantiated and regression-checked without changing the app model; all 2,048 optimization tasks remain single-app.Source: §3.4 · pp. 5–6, Figure 5 · p. 6

4 EXPERIMENTS

Experiments test reconstruction fidelity, source-grounded validation, learner-guided task generation, and downstream transfer. Trace2World reaches 91.1% workflow fidelity, and training gains extend to external benchmarks and real devices.

4 EXPERIMENTS

  • The experiments evaluate the same 200 source workflows across ten apps, then test training on Trace2World-Bench, four external GUI suites, and 100 paired real-device tasks.Entity/state rebindings and held-out workflows do not enter RL; the 50-app construction set includes ten apps with no policy optimization tasks.Source: §4.1 · p. 6, §4 · p. 6, §4.2 · p. 7

4.1 END-TO-END RECONSTRUCTION FIDELITY

  • 91.1% of workflows pass all six source checks for Trace2World, versus 59.8% for the task-local GUI-GENESIS-style build, despite respective endpoint reach of 98.0% and 94.8%.Across 1,570 workflows, 74.6% pass source replay on the first attempt and all close within the K = 3 automated repair cap without human QA.Source: §4.1 · p. 7, Table 1 · p. 7

4.2 SOURCE-GROUNDED VALIDATION

  • Source validation retains 94.0% of a 16.7-pp simulated gain on real devices, versus 54.7% of 17.7 pp under self-validation, a 6.0-pp paired difference.Source rechecks reject 22/160 self-accepted executions versus 1/176 source-accepted executions; with training matched, simulation-only passes rise from 10.0% to 18.3% under self-validation.Source: §4.2 · p. 7

4.3 LEARNER-GUIDED TASK GENERATION

  • Learner-guided tasks recombine supported capabilities rather than add semantics: 194 of 200 audited accepted candidates are independently solvable by humans.The support gate rejects 77 and 90 of 576 candidates per guided round; 125 ambiguous, invalid, or unresolved terminal-reaching cases remain outside support.Source: §4.3 · p. 8
  • At iteration 52, learner-guided tasks reach 61.1% SR versus 56.9% for static generation, a 4.2-pp difference between the two H = 45-matched augmentation branches.The guided batches have 76.8/75.3% learning-signal tasks versus 47.9/44.0% for static control; only 14.1/14.5% of validated lineages enter bounded refinement.Source: §4.3 · p. 8, Figure 6 · p. 8

4.4 DOWNSTREAM POLICY PERFORMANCE AND REAL-APP TRANSFER

  • 20.2 pp SR and 15.7 pp PR gains over the fixed-backbone base on Trace2World-Bench accompany SR gains of 16.2 pp on MobileGym, 5.4 pp on AndroidWorld, 8.9 pp on MobileWorld, and 11.1 pp on AndroidDaily.On 100 evaluation-only two-app tasks, SR rises 24.7 pp (26.0 to 50.7), including 15.4 pp from the learner-guided stages; the final policy retains 87.5% of simulated gain on real devices.Source: §4.4 · p. 9

5 CONCLUSION

Trace2World separates app-behavior reconstruction from implementation and validates implementations by replaying source workflows. Its conclusion is that scalable GUI RL depends on evidence-grounded worlds, not merely runnable mocks.

5 CONCLUSION

  • Trace2World reaches 91.1% workflow fidelity versus 59.8% for the strongest existing route, and retains 94.0% of simulated gains on real devices versus 54.7% under self-validation.The approach fixes source-linked semantics before coding, leaves unresolved effects non-executable, and uses independent replay to localize later mismatches for bounded repair.Source: §5 · p. 9
  • Validated worlds can be reused for learner-guided tasks while keeping behavior within source-supported bounds.The paper frames the bottleneck as reconstructing app worlds whose state and effects are evidence-fixed, source-workflow checked, and trusted without human QA.Source: §5 · p. 9
  • Global reconstruction and local implementation repair address distinct failure points that agentic reconstruction otherwise conflates.The conclusion distinguishes whether source evidence supports behavior from whether the implementation preserves it.Source: §5 · p. 9

ETHICS STATEMENT

The authors describe consent, review, redaction, and release conditions for trace data and environments. Real-device evaluation uses restorable states and excludes irreversible actions.

ETHICS STATEMENT

  • Source recordings come from consenting operators using accounts they control, with consent covering research use and public release.Recordings and derived trace records are released with generated environments and evaluation artifacts.Source: ETHICS STATEMENT · p. 10
  • Before release, recordings are reviewed for sensitive material; affected regions are redacted when research-relevant evidence remains intact, otherwise traces are withheld and listed.Review includes credentials, personal identifiers, private communications, precise locations, payment or order information, and third-party personal data.Source: ETHICS STATEMENT · p. 10
  • Environments are isolated browser reimplementations with synthetic content, and real-device evaluation starts from restorable states while excluding irreversible actions.Artifacts that fail service-term, redistribution-rights, or trade-dress review are withheld and reported; live-service use requires account-holder authorization and applicable-rule compliance.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

Generative models support multiple Trace2World stages, and AI tools also assisted preparation of the paper. The appendix compares GUI environments by their semantic sources, representations, implementation, and acceptance evidence.

AI USE STATEMENT

  • Generative models support trace interpretation and semantic induction, environment implementation and repair, task generation and refinement, and rollout-based screening and evaluation.The authors retain responsibility for study design, verification and interpretation of results, and the final manuscript and artifacts.Source: AI USE STATEMENT · p. 10
  • Generative AI tools assisted code development, figure preparation, manuscript drafting and revision, and review of methodological and experimental descriptions.Generated environments and tasks are assessed under shared evaluator boundaries and study-specific protocols, with human assessment on sampled evaluation packets.Source: AI USE STATEMENT · p. 10

A SEMANTIC RECONCILIATION ACROSS TRACES

  • The appendix formalizes trace reconciliation from preprocessing through bounded-context scheduling and situates that contract among existing environment builders.Its comparison preserves each system’s native target rather than treating the absence of an external-app fidelity objective as a failed check.Source: A SEMANTIC RECONCILIATION ACROSS TRACES · p. 15
  • Table A1 compares environment builders by behavior source, pre-coding representation, cross-trace rules, implementation, and acceptance evidence.It distinguishes native targets and units; task checks in authored or synthesized worlds do not certify fidelity to an external app.Source: Table A1 · p. 15

A.1 SOURCE-TRACE PREPROCESSING

Trace2World turns retained recordings into trace-linked evidence, reconciles supported app semantics before implementation, and constrains initialization and execution to that support. Its context-bounded process preserves unresolved conflicts rather than compiling unsupported behavior.

A.1 SOURCE-TRACE PREPROCESSING

  • 1,570 traces are selected from 6,541 admitted recordings (24.0%) by clustering task instructions and covering diverse workflows while avoiding duplicate action-signature prefixes.On Meituan, selection reduces 106 admitted candidates to 50 while retaining every occupied workflow bucket.Source: §A.1 · p. 16, Figure A1 · p. 16
  • 12,680 executable frames (77.0%), 1,570 terminal observations (9.5%), and 2,222 pre-semantic removals (13.5%) partition the 16,472-frame corpus.Terminal observations remain as evidence without executable terminal actions; launch or app-entry frames account for 9.2% of the corpus.Source: §A.1 · p. 17, Figure A2 · p. 17

A.2 BEHAVIOR SUPPORT AND IMPLEMENTATION SCOPE

  • The support set contains only Structure, State, and Data relations supported by frozen traces; unsupported or irreducibly ambiguous behavior stays outside executable scope.A visible component outside the task binding cannot change business state and returns NOT INTERACTIVE when acted on.Source: §A.2 · pp. 17–18

A.3 SEMANTIC RECONCILIATION AND SUPPORT VALIDATION

  • RESTORE, RECONCILE, FACTORIZE, and SUPPORTED turn trace-linked transition candidates into source-supported Structure–State–Data relations, recording failures instead of compiling unsupported or incomplete relations.Conflicting consequences remain unresolved until evidence supplies a discriminator; majority agreement cannot settle semantic ownership.Source: §A.3 · pp. 18–19

A.4 TASK-CONDITIONED INITIALIZATION AND RESET

  • Each task starts from supported launch values in a fresh browser context, while Reset restores its fixed snapshot and clears runtime state, events, transactions, and reward.Conflicts become launch fields, supported guards, or unresolved values that remain outside the implemented scope.Source: §A.4 · p. 20

A.5 CONTEXT-BOUNDED RECONCILIATION

  • RECONCILE and FACTORIZE share a hierarchical execution schedule; sharding changes API-call evidence assignment, not semantic mappings, support gates, or contradiction handling.Per-call phase budgets bound the schedule.Source: §A.5 · p. 20

B.1 ENVIRONMENT ARCHITECTURE AND EXPORT

Validated canonical semantics and visual evidence are compiled into browser packages, which are independently replayed against source workflows and repaired only within the frozen model. The process also supports state-based probes, cross-app compositions, and task candidates bound to validated indexes.

B.1 ENVIRONMENT ARCHITECTURE AND EXPORT

  • Each page specification is compiled from validated semantics and visual evidence into a DOM-backed Chromium application, while source traces and evidence maps remain outside the runtime execution path.Export checks build integrity, canonical-consumer coverage, host/reset entry, and smoke execution.Source: §B.1 · pp. 21–22

B.2 SOURCE-TRACE VALIDATION AND REPAIR

  • Independent replay compares source workflows with the exported environment; bounded repair edits the package under frozen specifications, and unrepairable slices are dropped.The default budget permits three repairs; a repair cannot add unsupported pages, state, data, guards, or effects.Source: §B.2 · pp. 22–23

B.3 STATE-BASED VERIFICATION

  • State-based verification compares legal actions’ business-state projections with prescribed transitions and requires illegal attempts to preserve business state without dispatch.A digest must pass GUI execution, independent sequence review, and every applicable probe.Source: §B.3 · p. 23, Table B2 · p. 23

B.4 CROSS-APPLICATION WORKFLOWS

  • Cross-app workflows compose two validated app-local indexes through typed, GUI-visible A→B→A handoffs, with a joint contract checking both local outcomes and the final decision in A.Because no source recording spans the complete episode, the composition is validated against its frozen joint contract.Source: §B.4 · p. 23

C.1 TASK CANDIDATE AUTHORING

  • Task candidates bind goals, launch states, dependencies, and evaluator-visible outcomes to items in a read-only index of the source-validated package; unsupported bindings are rejected.The index references established operations but does not introduce transitions, guards, or effects.Source: §C.1 · p. 24

C.2 TASK IMPLEMENTATION AND SCREENING

Task slices instantiate already-supported behavior and are screened with learner and solver attempts before selection. Frozen batches then guide policy training, while the release, interface, model roles, and evaluation boundaries define the experimental resource and protocol.

C.2 TASK IMPLEMENTATION AND SCREENING

  • Task implementation instantiates supported entities, state instances, transitions, and evaluator-visible consequences, then checks fresh execution and seed-task regression; failed slices are excluded.Learner and fixed-solver attempts distinguish selectable tasks from environment defects, refinements, and held candidates.Source: §C.2 · pp. 24–25

C.3 AUGMENTATION BATCHES AND POLICY TRAINING

  • Each training stage uses a frozen task batch, and task groups are retained only when their eight rewards have population SD above 0.1, excluding all-success and all-fail groups.The learner continues from the preceding weights and consumes each shuffled batch without replacement.Source: §C.3 · pp. 25–26

D SYSTEM CONFIGURATION AND EVALUATION PROTOCOLS

  • The evaluation protocol fixes shared settings and boundaries across matched comparisons unless a study-specific protocol names a changed factor.The section covers resource, agent, partition, optimization, and evaluator roles.Source: D SYSTEM CONFIGURATION AND EVALUATION… · p. 26

D.1 TRACE2WORLD RESOURCE CONTRACT

  • Trace2World is released as an environment-construction resource binding source evidence, reconstructed environments, training tasks, evaluation sets, and fidelity records.The release preserves research-relevant chronology, including retries and corrections, subject to privacy, security, and redistribution screening.Source: §D.1 · p. 26, Table D1 · p. 26

D.2 AGENT INTERFACE AND MODEL ROLES

  • GPT-5.2 handles bounded construction calls, GPT-5.5 handles package verification and repair, and GPT-5.6 Sol reviews fidelity offline; Qwen3.5-9B is the learner and Qwen3.7-Plus the fixed solver.The fidelity reviewer does not supply training rewards or held-out success scores.Source: §D.2 · pp. 27–28

D.3 APPLICATION, TASK, AND EVALUATION SPLITS

The study separates environment construction, policy optimization, and evaluation populations so reconstruction fidelity and policy gains are assessed on explicitly defined sets. Evaluator roles and records are kept distinct to prevent one score from serving as evidence for another task or condition.

D.3 APPLICATION, TASK, AND EVALUATION SPLITS

  • All 50 apps contribute construction traces, but only 40 supply policy-training tasks; ten apps, one per domain, are reserved from policy optimization.The held-out apps still supply construction traces, but no seed or learner-guided task, candidate slot, screening rollout, or RL update.Source: §D.3 · p. 28, Table D4 · p. 28
  • The 1,570-trace quota assigns 50 traces to each of 19 multi-service apps and 20 to each of 31 single-service apps.A service line is defined from the product surface before collection and has its own catalog or service plus a transactional record and lifecycle.Source: §D.3 · p. 28
  • Fidelity is measured on the same 200 construction-exposed workflows across ten apps using six binary diagnostics, not as a held-out policy test.The ten-app roster coincides with the policy-held-out roster, but the estimator uses paired workflow judgments with equal application weighting.Source: §D.3 · pp. 28–29
  • The 2,048-task policy uses 1,280 seed tasks and two 384-task guided batches; a signature-based split holds out 512 workflow identities.Entity-rebinding variants enter only the matched training control, while the held-out slice is defined by complete task and composition signatures absent from training.Source: §D.3 · p. 29
  • Trace2World-Bench comprises four disjoint evaluation slices totaling 1,044 tasks; its 100-task cross-application slice uses policy-training apps, and no benchmark identity informs optimization or learner-guided generation.Source: §D.3 · p. 29

D.5 EVALUATOR ROLES AND NON-INTERFERENCE

  • Construction validation, task-evaluator rewards, and independent fidelity-review labels serve separate purposes and do not feed one another’s evaluation roles.Final evaluation and human annotations do not return to construction, task filtering, or checkpoint selection; every score is bound to its packet, artifact or policy revision, evaluator, seed, and reducer.Source: §D.5 · p. 30, §D.6 · p. 31

E.1.1 FIDELITY PROTOCOL AND EVALUATION PACKETS

  • The fidelity reviewer compares source-trace evidence with generated executions on six P/F diagnostics, and an all-six workflow verdict passes only when every diagnostic passes.Evaluation starts from a native reset or declared preset, permits up to max(80, 3L) actions, and treats missing or contradicted required evidence as failure.Source: §E.1.1 · p. 31

E.1.2 COMPARED BUILDERS AND CONTROLS

  • Builder comparisons assess complete end-to-end contracts, while trace-access pairs and the self-validation control change a declared factor under shared construction conditions.Conditions share models, tools, recording pool, and a six-hour per-app limit; the self-validation control replaces independent source-trace validation with generated verifiers.Source: §E.1.2 · pp. 31–32, Table E1 · p. 32

E.1.3 WORKFLOW FIDELITY ACROSS ENVIRONMENT BUILDERS

Across the ten-app evaluation, Trace2World preserves more source workflows than the compared construction conditions, while diagnostic profiles show that endpoint reach alone does not establish workflow fidelity. The extended results also examine reviewer agreement, scaling, repair, cost, and defect review.

E.1.3 WORKFLOW FIDELITY ACROSS ENVIRONMENT BUILDERS

  • The weakest-diagnostic gap from Reach is 4.3 pp for Trace2World, versus 15.4–21.9 pp for the trace-conditioned builders and 13.5 pp for self-validation.The three shared-app builders without cross-trace reconciliation bottom out on Data or Effect; these profiles locate failed obligations but do not attribute them to a construction stage.Source: §E.1.3 · p. 32
  • Trace2World leads GUI-GENESIS-style by 31.3 pp and self-validation by 12.0 pp in workflow fidelity; it is higher in nine of ten apps in both contrasts.Figure E1b shows the paired app differences and macro-app means; the table reports diagnostic fidelity and reviewer agreement on the primary evaluation set.Source: Table E2 · p. 33, Figure E1 · p. 33, §E.1.3 · p. 34

E.1.4 HUMAN VALIDATION AND AUTOMATED-REVIEWER AGREEMENT

  • Human adjudication raises all conditions by 1.4–3.8 pp, leaves the ordering unchanged, and gives Trace2World margins of 29.9 pp over GUI-GENESIS-style and 12.8 pp over self-validation.Automated workflow-label macro-F1 is 90.2%, with 92.3/87.9% pass precision/recall; the human labels qualify the same 1,200 packets rather than adding a new performance population.Source: §E.1.4 · p. 35, Table E3 · p. 35

E.2.1 CONTEXT AND CALL SCALING WITH LONGER TRACE PREFIXES

  • At 30 and 50 traces, flat inputs exceed the 40K-token budget, while the largest hierarchical call stays within 28.7–33.2K tokens.Across 50 environments, peak model-visible context has median/P95/maximum 28.9/31.7/33.2K tokens, and no call exceeds the budget.Source: §E.2.1 · pp. 35–36, Figure E2 · p. 35

E.2.2 RECONSTRUCTION QUALITY AND EVIDENCE COMPRESSION WITH MORE SOURCE TRACES

  • From Low to Full trace support, supported routes grow from 270 to 512 and visual obligations from 553 to 1,076, while workflow fidelity moves from 92.6% to 91.1%.Evidence compression rises from 16.9% to 38.8%; the nested subsets use 100, 180, and 290 traces and are each scored against their own induced targets.Source: §E.2.2 · p. 36, Figure E3 · p. 36, Figure E4 · p. 37

E.2.3 AUTOMATED VALIDATION AND REPAIR-WAVE ACCOUNTING

  • All seed workflows close within K = 3 repair rounds, while 13.4% of support-passing augmented candidates reach the cap and are dropped.A repair round is one evidence-driven edit followed by fresh verification; the self-validation control averages 0.26 waves per workflow versus Trace2World’s 0.36.Source: §E.2.3 · pp. 37–38, Table E4 · p. 37

E.2.4 CONSTRUCTION COST

  • Trace2World averages 1,160 construction calls and 3.60 hours on the matched ten-app roster; its API pipeline accounts for 602 calls and 0.50 hours.The agentic workflow accounts for 558 calls and an estimated 3.10 hours; a 24-episode API probe reaches 82.6 calls/s with 174/915 ms median/P95 latency, not complete rollout timing.Source: §E.2.4 · p. 38, Table E5 · p. 38, §E.2.5 · p. 39

E.2.6 SOURCE-TRACE EVIDENCE FOR DEFECT REVIEW AND REPRODUCE-BEFORE-REPAIR

  • Adding source traces and a contrastive review rubric raises defect-coverage ranges to 77.4–96.6%, while fresh step-zero re-execution catches 233 of 1,884 repair edits that break prior packets.Stale-suffix reuse would have certified 214 of those regressions; fresh execution incurs 1.9× repair-loop execution overhead.Source: §E.2.6 · pp. 39–40

E.3.2 LEARNER FEEDBACK TARGETS HARDER TASKS WHILE PRESERVING SOLVABILITY

Learner-guided task selection changes the supplied batches toward tasks that are harder for the current policy while retaining fixed-solver solvability. The section examines screening, bounded refinement, and matched training outcomes.

E.3.2 LEARNER FEEDBACK TARGETS HARDER TASKS WHILE PRESERVING SOLVABILITY

  • Learner pass rate falls from 41% to 29% while fixed-solver pass falls only from 71% to 66%, and dependency-heavy tasks rise from 46% to 71%.Both rounds fill the same 384-task batch; the change is in learner-relative difficulty and task composition, not batch size.Source: §E.3.2 · p. 41, Table E8 · p. 41
  • Among audited candidates, 194/200 solver-accepted tasks are independently solvable, while 36 rejected tasks are solvable, a 9.0% conditional rejection error.The stratified audit estimates solvability conditional on acceptance or rejection, not the share of all solvable tasks rejected.Source: §E.3.2 · p. 41
  • Only 14.1% and 14.5% of validated lineages are refinable; bounded hardening raises validated-lineage signal by 5.6 and 5.4 pp in the two rounds.Hard and solver-unsolved lineages are never simplified, and the fixed two-round flow selects 36/60 and 37/62 refinable lineages.Source: §E.3.2 · p. 42, Table E9 · p. 42
  • At iteration 52, MobileGym SR is 19.7 ± 1.2, 21.0 ± 0.8, 22.2 ± 1.3, and 25.2 ± 1.2 for seed continuation, entity rebinding, static augmentation, and learner-guided batches.The longer-schedule controls change optimization budget and therefore test robustness rather than entering the same-budget causal contrast.Source: §E.3.2 · p. 42

E.3.3 NEW TASK COMPOSITIONS STAY WITHIN SOURCE-SUPPORTED BEHAVIOR

Generated tasks can recombine source-supported behavior beyond literal recorded paths, and learner-guided supplies contain more capability probes. The audits keep path and launch-state accounting separate.

E.3.3 NEW TASK COMPOSITIONS STAY WITHIN SOURCE-SUPPORTED BEHAVIOR

  • Of 611 terminal-reaching cases on paths absent from recorded traces, 486 are semantically equivalent, and all are executable or materialized from supported edges.The remaining 125 ambiguous, invalid, or unresolved entries remain outside support; these are attempt-associated cases, not deduplicated paths.Source: §E.3.3 · p. 42, Figure E5 · p. 43
  • Cross-surface tasks rise from 11.9% to 43.6%, while aggregation, lifecycle revisit, and conditional recovery rise from 13.6/6.1/4.5% to 32.4/25.8/21.0%.These are overlapping task-level marginals, so they describe supplied task composition and do not establish capability unions or training outcomes.Source: §E.3.3 · p. 43, Figure E6 · p. 43

E.3.4 COMPLETE TRACE2WORLD-BENCH AND EXTERNAL BENCHMARK RESULTS

The results compare benchmark policies, paired sim-to-real transfer, and remaining policy failures, then use case studies to diagnose how construction choices affect workflow fidelity and reuse.

E.3.4 COMPLETE TRACE2WORLD-BENCH AND EXTERNAL BENCHMARK RESULTS

  • Ours scores 63.2, compared with 60.1 for Stage-1 policy and 57.8 for Base policy.Table E10 reports terminal success rate and mean partial reward across benchmark columns; the supplied result values do not specify which column they represent.Source: §E.3.4 · p. 44, Table E10 · p. 44

E.3.5 HORIZON CALIBRATION AND REMAINING POLICY FAILURES

  • Hard-loop failures drop from 16.1% under the base policy to 8.5% under the final policy; only 18 horizon-reaching failures are rescued by 15 extra actions.Stalled or wandering failures are a larger share of augmented than source-derived failures (28.8% versus 19.9%); 511 of 613 horizon-reaching failures make no new checkpoint in their final ten actions.Source: §E.3.5 · p. 46, Figure E7 · p. 46

E.4.1 BALANCED 100-TASK REAL-DEVICE PROTOCOL AND MATCHED BASELINES

  • The paired real-device panel fixes 100 tasks across 20 policy-training apps, with 50 entity rebindings and 50 held-out workflows evaluated under matched goals, starts, terminals, and step budgets.Five tasks per app do not estimate per-app effects; the two task-family rows are descriptive slices.Source: §E.4.1 · pp. 46–47

E.4.2 SOURCE-TRACE VALIDATION IMPROVES PAIRED SIM-TO-REAL RETENTION

  • Source-trace validation has a 6.0-pp real-device gain advantage over self-validation, with the 8.3-pp reduction in simulation-only passes dominating the decomposition.Both task families improve on device: entity rebindings rise from 54.0% to 73.3%, and held-out workflows from 30.7% to 44.0%.Source: §E.4.2 · p. 47, Figure E8 · p. 47

E.5 SCOPE, LIMITATIONS, AND FUTURE WORK

  • The evaluation uses DOM-backed browser reconstructions of mobile apps from traces collected at one point in time; the paired device panel covers policy-training apps, while unseen-app transfer is measured in reconstructed environments.The authors also identify unreported refresh costs, single-app optimization tasks, and proprietary construction-role models as scope boundaries.Source: §E.5 · pp. 47–48

F CASE STUDIES

  • The case studies cover a shared Meituan workflow, schema-backed augmentation, seed and augmented workflows in 58Tongcheng, a Railway 12306–Didi composition, and reconstructed app pages.The Meituan comparison starts every condition at home and shows complete traces through the pre-commit terminal or the first missing required action.Source: F CASE STUDIES · p. 48, §F.1 · p. 48

F.1 MATCHED BASELINE TRAJECTORIES ON ONE SHARED WORKFLOW

  • In the shared Meituan task, GUI-GENESIS and Trace2World each complete 13 frames, while Direct+Reference, CUA-Gym+Reference, and PhoneWorld stop at unsupported continuations.The workflow applies a CNY 6 coupon to reduce the amount from CNY 38 to CNY 32, selects Alipay, and stops before transaction confirmation.Source: §F.1 · p. 48, Figure F1 · p. 56

Recorded source trace

  • Trace2World and GUI-GENESIS each satisfy all 12 obligations in this case, but Trace2World preserves a shared app world while GUI-GENESIS builds a task-local site.This single case diagnoses mechanisms rather than estimating failure frequencies; it distinguishes app-wide consistency across workflows from reproducing one workflow.Source: Recorded source trace · p. 57

F.2 SCHEMA-BACKED TASK AUGMENTATION

This section describes how Trace2World augments tasks from attributed learner gaps while remaining constrained to the frozen, source-supported app model, with safeguards for authoring, hardening, repair, and two-app evaluation. It presents qualitative examples of workflows, cross-app dependencies, reconstruction, and prompt templates, not estimates of population-level workflow diversity or training effects.

F.2 SCHEMA-BACKED TASK AUGMENTATION

  • The Meituan augmentation composes observed search, selection, item-detail, and checkout behaviors into a 20-step rollout, with the latte as a schema-backed instance rather than a newly observed screen.Saved terminal checks record two vouchers, CNY 14.00 goods, a CNY 1.42 discount, and CNY 12.58 payable; the example demonstrates one supported composition, not population-level diversity or a training effect.Source: §F.2 · p. 57

F.3 SEED AND AUGMENTED WORKFLOWS

  • The 58Tongcheng seed and augmentation are distinct tasks in one environment, with the augmented case composing comparison, revisit, persistent state, and recovery over supported behavior.The examples are qualitative, not estimates of task difficulty or training effects; the augmented execution retains rejected attempts rather than showing only the successful path.Source: §F.3 · pp. 57, 62, Figure F3 · p. 65

F.4 CROSS-APPLICATION RETURN DEPENDENCY

  • The cross-app task requires retaining Didi fare quotes of RMB 30.6 and RMB 34.6 across a return to Railway 12306, then binding the selected quote to the final unpaid order.The fare gate permits the railway order only if the matched fare is no more than RMB 32 and below the unmatched fare; the panels assemble separately recorded app-local executions.Source: §F.4 · p. 68, Figure F5 · p. 72

RMB 824.5

  • The returned Railway 12306 order is unpaid and shows coach 02, seat 09C, and RMB 824.5.The terminal observation binds Yang Qian to the order, with first class and passenger fields visible together.Source: RMB 824.5 · p. 71, Figure F5 · p. 72

F.5 RECONSTRUCTION GALLERY

  • The reconstruction gallery pairs physical-device source observations with archived environment executions at matching task-conditioned page states, while allowing different legitimate paths and non-task visual differences.The gallery spans Xiaohongshu, Bilibili, Railway 12306, and Dingdong Maicai examples.Source: §F.5 · p. 73, Figure F6 · p. 74, Figure F7 · p. 76

G PROMPT TEMPLATES

  • The prompt templates separate trace preprocessing, shared-model reconciliation, source-trace validation, and extraction of a read-only task-facing index; emitted compositions must reference supported relations and source evidence.Reconciliation preserves uncertainty, while validation reports the first mismatch and owner rather than editing the trace or environment.Source: G PROMPT TEMPLATES · pp. 83–85
  • Learner-gap summaries separate non-learner faults from competence evidence and target weak, unsaturated supported motifs while deprioritizing saturated ones.They use rollout outcomes and capability/dependency labels, retain uncertainty when attribution is incomplete, and must not treat environment or solver failures as learner gaps or inspect held-out benchmark outcomes.Source: G PROMPT TEMPLATES · p. 85
  • Single-app candidates use the frozen task-facing index, bind required IDs to source evidence, and reject unsupported interactions or inferred transitions, guards, and effects.Seed authoring uses H=30 and augmented authoring H=45; both omit learner-gap conditioning for seed and static candidates.Source: G PROMPT TEMPLATES · pp. 85–86
  • Hardening applies only to solver-solvable lineages with learner mean reward >0.8, preserves prior obligations, and stops after at most two refinements.Hard or solver-unsolved lineages are held rather than simplified; long tasks require at least seven business subtasks, four decisions, three cross-state dependencies, two evidence revisits, and two evidenced phenomena.Source: G PROMPT TEMPLATES · pp. 85–86
  • Repairs edit the existing package under a frozen app specification, then invalidate dependent evidence and replay affected traces; defects requiring unsupported semantics are dropped.Repair is limited to source-supported implementation mismatches, not changing the app specification or inventing behavior to make a rollout pass.Source: G PROMPT TEMPLATES · p. 86
  • Two-app evaluation composes a GUI-visible typed A→B→A handoff, freezes its oracle before policy evaluation, and rejects hidden, stale, or concatenated transfers.The task is evaluation-only (H=60), requires human GUI completion within 45 public actions, and must preserve A state without adding app-local semantics.Source: G PROMPT TEMPLATES · p. 86

Related papers

Loading 2610.30792v1…