When to Compile a Computer-Use Agent? Measuring Payback and Making Compilation Decisions for Token Efficiency

• Compilation payback is difficult to measure because prior audits use incomparable component costs, while many systems report no payback count.Reported prior counts range from 4 to 17 uses.Source: §1 · pp. 1–2
• Repair may require multiple compilation attempts, and failed attempts still cost tokens; meanwhile, task arrivals may stop or GUI drift may end reuse.Compiling early can waste tokens if tasks do not repeat, while waiting can lose savings on repeated tasks.Source: §1 · p. 2
• PACE measures successful and failed compilation costs and compares agent and program execution on matched inputs to estimate per-use savings and payback.Its protocol also measures fallback costs, deployment extraction costs, and variation across repeated compilation attempts.Source: §1 · pp. 2–3
• PACE compares estimated future savings with compilation costs, including failed attempts, and permits actions only when their possible costs fit the observed-arrival budget.Under the stated cost model, total cost after each arrival is at most 1 + ϵ times the reference cost of serving every task with the agent.Source: §1 · pp. 2–3
• 17.3% lower token costs than ReAct, 24.9% than the AutoRPA adaptation, and 17.3% than the ToolPro adaptation on average in recorded-arrival simulations (ϵ = 0.25).Ablations separate the effects of the savings test, cost budget, and failed-attempt cost.Source: §1 · p. 3

• Tasks in one family share a GUI procedure but differ in inputs, and the objective limits cumulative cost relative to serving every arrival through the agent.The reference charge includes arrivals served by programs; it is a model-based comparison, not an additional observed agent run.Source: §2 · p. 3
• Agent execution has mean cost c, while program execution uses a model call to extract task parameters and then runs the shared procedure without further model calls.Costs use weighted tokens, with cached-input and output tokens weighted by their relative prices.Source: §2 · p. 3
• Compilation attempts use k completed agent traces and incur all compilation-call costs, whether they succeed or fail.The model assigns success probability p and mean charges C and Cfail, but historical means alone do not bound future actual costs.Source: §2 · p. 3
• Program executions can require agent fallback after GUI drift or execution failure; only GUI drift stops further reuse, and routing cost depends on listed programs.The fallback probability is h + (1 − h)q, reducing to q when h = 0.Source: §2 · p. 3
• PACE’s cumulative cost constraint applies after each arrival, and the objective constrains cost without imposing a task-success guarantee.Future tasks that have not arrived add no budget; the main evaluation uses ϵ = 0.25.Source: §2 · pp. 3–4

3 MEASUREMENT

• The protocol records successful and failed compilation costs, verifies generated programs on source tasks, and compares program and agent execution on matched fresh inputs.The broader measurement setup uses recorded agent traces as compilation experience and task prompts for binding extraction during verification.Source: §3 · p. 4, Figure 2 · p. 4

3.1 SERVING COST

• Eleven initial programs passed verification; their extraction costs ranged from 331 to 724 tokens and estimated saving shares from 0.936 to 0.999.The saving calculation includes extraction cost and one mean-cost fallback per failed program run.Source: §3.1 · p. 4
• 157 of 180 matched-input agent runs and 178 of 180 program runs succeeded; saving shares across the six verified Android programs ranged from 0.952 to 0.999.The replay used a different emulator profile and estimated fallback cost as qc, rather than measuring an agent resumed after each failed program run.Source: §3.1 · pp. 4–5

3.2 COMPILATION COST

• Eleven of twenty compilation-stage attempts passed verification, and the reported payback ratios exclude earlier failures and future replacement after drift.The two ratios are C/s and (C + kc)/s, with k = 3 source-task agent runs; rounding up gives uses needed to cover the charge in expectation.Source: §3.2 · p. 5, Table 1 · p. 5
• 1.389–15.488 uses is the estimated post-compilation payback range; failed DeepSeek attempts with complete records cost a median 1701K tokens versus 259K for successful attempts.The failure-cost comparison is across families and does not establish a within-family effect.Source: §3.2 · p. 5

4 ONLINE ALGORITHM

• PACE serves the current task before proposing compilation, using observed arrivals and compilation outcomes to estimate whether compiling will pay off for later tasks.Programs are added for later reuse only after successful verification.Source: §4 · p. 5, Figure 3 · p. 6

4.1 COST-AWARE COMPILATION PROPOSAL

• The savings test proposes compilation when drift-adjusted expected reuse savings exceed estimated compilation costs, including failed attempts, and routing costs.The estimate treats GUI drift as reducing reuse and low verification probability as increasing paid failures; the budget limits spending if estimates are wrong.Source: §4.1 · pp. 5–6

4.2 BUDGET CHECKS

• Before routing, PACE reserves budget for the largest possible service charge, including fallback; it clears the program list and uses the agent if the check fails.A separate compilation check allows a proposal only when its maximum possible charge fits the same cumulative budget.Source: §4.2 · pp. 6–7

4.3 COST GUARANTEE

• Under valid charge bounds and reservation assumptions, PACE keeps cumulative cost within 1 + ϵ times the agent-execution reference after every arrival.The bound covers every history allowed by those bounds; it concerns cost, not task errors or per-family spending.Source: §4.3 · p. 7

5 ONLINE EVALUATION

• The evaluation simulates measured execution and compilation profiles on synthetic streams and recorded arrival orders, comparing PACE with ReAct and adaptations of AutoRPA and ToolPro.The grids were used while developing PACE, so these comparisons are not independent confirmation.Source: §5 · p. 7

5.1 SAVINGS

• Across the two grids, ReAct cost 1.107 and 1.130 times PACE on average, and all 4,920 tested cumulative ratios stayed within the implemented 1.25 bound.The bound held in these runs; the theorem, rather than the empirical check, covers all histories allowed by the model.Source: §5.1 · p. 8, Table 2 · p. 8
• 17.34% lower token cost than ReAct on recorded logs; AutoRPA and ToolPro adaptations had mean cost ratios of 1.362 and 1.262 relative to PACE.PACE lowered cost in nine of twelve log conditions; ToolPro cost less in three, and the comparison concerns decision-rule adaptations rather than the original systems end to end.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 COMPONENT EFFECTS

• Removing the savings test raised mean log cost by 52.68%, while equal estimated costs for successful and failed attempts raised it by 23.34%.Both variants retained the budget, and their largest grid ratios to ReAct stayed near the 1.25 limit.Source: §5.2 · pp. 8–9, Table 4 · p. 8
• Removing the budget changed mean log cost by only 0.0013% but raised the largest tested grid ratio to ReAct from 1.2468 to 1.7563.Thus the reported contribution is protection against larger losses, not lower average cost on those recorded logs.Source: §5.2 · pp. 8–9, Table 4 · p. 8
• When high compilation costs were assigned to frequent, low-verification families, mean PACE-to-ReAct cost ratio rose to 1.015 despite the cost constraint.The sensitivity result illustrates that frequent arrivals alone do not ensure profitable reuse.Source: §5.2 · p. 9

Computer-use agents

• Prior GUI systems reuse trajectories or workflows as guidance, while AutoRPA and PreAct compile interaction traces into programs with verification, repair, or fallback.PACE differs by measuring successful and failed compilation costs and deciding when to compile under a cumulative token budget.Source: §6 · p. 9

Online decision theory

• PACE applies baseline-constrained online decision ideas to token charges, reserving compilation, fallback, and routing costs under valid action-cost bounds.The related work connects this approach to rent-or-buy analysis, prediction-guided purchase timing, and conservative contextual bandits.Source: §6 · p. 9

7 LIMITATIONS

• Measurements cover seven parameterized task families and three model profiles, while the live study tests fallback and call reservations for a fixed program rather than the full online algorithm.Silent program failures and recovery from partially changed interfaces require further evaluation.Source: §7 · p. 9
• The cost guarantee depends on valid action-cost bounds, which historical mean costs do not provide.Model and provider updates can also change measured costs.Source: §7 · p. 9
• The evaluation measures model-token cost, excluding latency, device use, and program-maintenance effort.Its online simulations use fixed cost profiles and given family labels; savings depend on how costs and compilation outcomes are assigned to arriving families.Source: §7 · p. 9

8 CONCLUSION

• PACE saves 17.3% relative to ReAct, 24.9% relative to the AutoRPA adaptation, and 17.3% relative to the ToolPro adaptation in recorded-arrival simulations at ϵ = 0.25.For successful attempts, estimated payback counts excluding source agent runs are 2–16 uses.Source: §8 · p. 9
• PACE measures successful and failed compilation costs and per-use savings, then uses these profiles to make cost-aware compilation proposals.Execution and compilation checks use the budget from observed arrivals to enforce Kt ≤ (1 + ϵ)At under valid action-cost bounds.Source: §8 · p. 9
• The cost bound does not ensure task correctness, so actions with important real-world effects require appropriate checks.The study uses benchmark applications, an Android emulator, desktop applications in a Linux container, and locally hosted web applications.Source: ETHICS STATEMENT · p. 10

A.1 TASK FAMILIES / A.2 COST ACCOUNTING

• The study covers seven parameterized task families across Android, desktop, and web, with three model profiles planned across 21 family–model cells; a refused Qwen web request leaves 20 cells.The cost accounting weights input, cached-input, and output tokens by recorded price ratios, sums serving, compilation, and repair calls, and excludes wall-clock time, device resources, and human effort.Source: §A.1 · p. 12, §A.2 · p. 12

A.3 COMPILATION

• Compilation uses three completed agent traces, parameter extraction, verification on all three source tasks, and bounded repair; unsuccessful attempts retain recorded charges, with missing usage reported as a lower bound.After verification, programs run on 30 fresh bindings; deployment failures include both extraction and execution failures.Source: §A.3 · pp. 12–13

A.4 SERVING PROFILES

• Serving profiles distinguish the three-source-task exploration means from replay means on 30 deployment bindings; the profile table reports agent cost, extraction cost, failure fraction, and saving share.The profile means and replay means differ.Source: §A.4 · p. 13, Table 5 · p. 14, Table 6 · p. 14

A.5 REPEATED COMPILATION

• Of 22 additional compilation attempts, 12 pass verification, six fail program checks, and four exhaust the builder retry budget without a valid program.Three attempts per family–model cell do not establish independent retry outcomes or a stable verification probability; the DeepSeek Contacts failed attempts cost both less and more than its successful repeat under the same accounting boundary.Source: §A.5 · pp. 14–15

A.6 GIVEN AND EXTRACTED PARAMETERS

• On five additional bindings, four verified Android programs pass through both given and extracted parameter paths, while GLM Calendar passes three and GLM Contacts passes none.For each failed task in this check, extracted values match given values and both paths fail; GLM Contacts nevertheless completed all 30 deployment tasks before failing on a rebuilt emulator instance.Source: §A.6 · p. 15

A.7 PROGRAM FAILURES UNDER GUI CHANGES

• GUI-change failures among programs passing cleanly average 0, 0, and 0.100 for appearance changes and 0.842, 1.000, and 0.833 for interruptions on GLM, DeepSeek, and Qwen, respectively.Across all test failures, detectable execution errors account for about 58%, 87%, and 71%; GUI-change rates do not estimate how often deployments encounter those changes.Source: §A.7 · p. 15

A.8 MEASUREMENT CONFIDENCE INTERVALS

• Zero failures in 30 uses still gives a two-sided 95% failure interval of [0, 0.116], and narrow saving-share intervals cannot rule out unseen failure modes.The intervals condition on the recorded program and binding protocol; repeated compilation attempts do not support independent-trial cost intervals.Source: §A.8 · p. 15, Table 8 · p. 16, Table 10 · p. 17

B.1 COMPILATION SPEND / B.2 PROGRAM LIFETIME

• With independent attempts and fixed verification probability p > 0, expected failures before success are (1 − p)/p; PACE estimates compilation spend from past outcomes but assumes a fixed retry distribution.The online algorithm checks execution and compilation charges against the observed-arrival budget, while the lifetime calculation uses an estimated reuse horizon and does not establish optimal replacement under unknown parameters.Source: §B.1 · pp. 17–18, §B.2 · p. 18

B.3 COST OF RESERVING FOR FAILURE

• A passing savings test may still wait for more observed arrivals when the failure-cost reservation exceeds the current allowance.In the example, two additional agent-served arrivals raise the allowance from 75 to 125 tokens, enough to cover either compilation outcome.Source: §B.3 · p. 18
• A failed-attempt reservation can prevent a profitable compilation entirely when its cost exceeds the extra budget available over the task stream.In the example, the reservation never reaches 200 tokens, so PACE serves 300 arrivals with the agent despite a successful compilation being possible at cost 2.Source: §B.3 · p. 18

C.1 COST PROFILES

• For unobserved outcomes, the simulations assign same-model median costs; these are assumptions, not additional measurements.Incomplete Web DeepSeek charges are bounded below by the recorded cost.Source: §C.1 · p. 19

C.2 SYNTHETIC TASK STREAMS

• Two 300-condition grids vary model profiles, verification, arrival patterns, and costs, with 1,200 arrivals and 30 recurring families in the first grid and 1,800 arrivals and 40 in the second.Both grids include sparse patterns; the second uses its own random seeds and parameters.Source: §C.2 · p. 19

C.3 RECORDED ARRIVAL ORDERS

• The recorded streams comprise Sepsis and BPI 2019 process logs plus Wikipedia tool and human edit streams, with family labels defined from completed activity sequences or edit summaries.The process-log labels are assigned after case completion and do not test online family identification; the logs supply arrival frequencies and order rather than live GUI tasks.Source: §C.3 · p. 19
• The evaluation reports paired intervals conditional on tested scenarios and given cost profiles, while its comparisons are exploratory because both grids informed PACE’s development.All conditions use detected failures and one agent fallback; the experiments do not test an online drift detector, and the cost bound itself makes no task-quality claim.Source: §C.3 · p. 20

C.4 COMPARISON RULES

• PACE and comparison policies are evaluated on the same measured compiler and serving profiles, isolating their compilation and reuse decisions rather than reproducing the original systems.The AutoRPA and ToolPro comparisons are adaptations, not reproductions of their original models, code generators, or benchmark scores.Source: §C.4 · p. 20
• The AutoRPA adaptation attempts one build after three completed agent runs, while the ToolPro adaptation requests compilation only when estimated savings within one task exceed construction cost.ToolPro’s adapted rule gives no credit for savings across multiple future tasks.Source: §C.4 · p. 20
• Timing controls vary when they request builds, while budgeted variants add PACE’s cumulative budget check to alternative proposal rules.All methods require three completed agent runs before compiling, serve the current arrival before a build, and share failure outcomes and program-list controls.Source: §C.4 · pp. 20–21

C.5 COST-PROFILE SENSITIVITY

• 0.833 to 0.866 of ReAct is PACE’s mean cost across four missing-cost settings, while the value of separate success and failure costs depends on their assumed difference and its effect on timing.At target ratio r = 5, Equal attempt costs costs 1.141 times PACE on average; PACE is less costly in all twelve conditions.Source: §C.5 · p. 21, Table 12 · p. 22
• 0.671 times ReAct cost is PACE’s mean under cost-dependent verification alone, but assigning costly profiles to frequent families raises the ratio to 1.015.In that joint scenario, PACE costs about 1.5% more than ReAct on average; both tested variants satisfy the implemented budget check in all 1,440 runs.Source: §C.5 · p. 21

C.6 ALTERNATIVE COMPILATION PROPOSALS

• 0.8349, 0.8323, and 0.8312 times ReAct are the recorded-log mean costs for three budgeted timing controls, versus 0.8266 for PACE.Each paired 95 percent interval for control cost divided by PACE cost lies above one, although each mean difference is below 1%.Source: §C.6 · p. 22
• 0.9556 and 0.9536 are Count + budget’s mean cost ratios to ReAct on the two grids, close to PACE’s 0.9563 and 0.9539.On recorded logs, Count + budget is 0.8265 times ReAct cost; the results do not identify the exact reuse-count estimate as a separate source of improvement.Source: §C.6 · p. 22

C.7 COMPLETE COST COMPARISONS

• The complete cost comparisons report ratios relative to PACE across two parameter grids and four recorded arrival sequences, with paired bootstrap intervals conditional on tested scenarios and given profiles.PACE’s ratio is one by normalization; ToolPro matches ReAct in the conditions shown.Source: §C.7 · p. 22, Table 13 · p. 23, Table 14 · p. 24

C.8 COMPONENT COSTS

• Equal attempt costs is cheaper than PACE in 62 and 60 conditions across the two grids, so its average loss is not uniform across conditions.The ablations compare cost ratios relative to PACE; removing the budget has little effect on recorded-log mean cost but increases worst grid cost.Source: §C.8 · pp. 22, 25, Table 15 · p. 25, Table 16 · p. 25

D LIVE GUI EXECUTION

• The live Calendar study uses a fixed program and scheduled compilation attempt, and does not evaluate PACE’s online compilation choice.The program was checked on the unchanged interface and under a renamed submit button or blocking dialog.Source: D LIVE GUI EXECUTION · p. 25

D.1 PAIRED TASK EXECUTION

• 7 of 12 agent tasks succeed, while all four clean program runs succeed without fallback and seven of eight changed-interface program runs succeed after fallback.Only five changed-interface pairs reach the changed part before runs end, and both policies succeed in those five.Source: §D.1 · p. 26, Table 17 · p. 26, Table 18 · p. 26

D.2 RESERVATIONS

• All 221 actual requests pass recorded budget checks; 215 have confirmed charges and six have unconfirmed costs.Known charges total USD 0.121359, with an upper bound of USD 0.363432 including reservations for unconfirmed calls. In a separate four-task test, one task is denied because its binding reservation exceeds the initial allowance, outside the theorem’s default-service assumption.Source: Table 19 · p. 26, §D.2 · p. 27

D.3 COST BOUNDS WITH UNCONFIRMED COSTS

• Across all twelve pairs, mean serving costs are bounded by [8.142, 21.590] milli-USD for the agent and [1.202, 4.564] milli-USD for the program path.The 95% bootstrap envelope for the paired mean difference is [−33.017, 4.621] milli-USD and includes zero; four clusters, one task family, and provider interruptions limit population inference.Source: §D.3 · p. 27