Source-linked AI summary
TomasuLLM: Out-of-Order Speculative Execution for LLM Agents
Jiangnan Yu, Ceyu Xu, Mengming Li, Shiyu Huang, Yiran Xia, Jian Weng, Hui Xue, Haohui Mai, Yuan Xie
TL;DR
Coding-agent harnesses serialize around observations while long-running tools leave future work waiting. TomasuLLM speculatively executes drafted actions in isolated overlays and validates their results before publication. It reports speedups across three benchmarks and zero false accepts in audited commit validation.
Problem
Coding-agent harnesses wait for tool observations before issuing later-turn actions, leaving predictable work unavailable while long-running tools dominate latency.
Source: §1 · p. 1Method
TomasuLLM drafts future tool actions, runs them in copy-on-write overlays, traces dependencies and effects, and commits validated results in trajectory order.
Source: §1 · p. 2Results
TomasuLLM achieves 1.27–1.35× speedups across SWE-bench Verified, Terminal-Bench 2.0, and SWE-Marathon, with zero false accepts across 4,010 audited records.
Source: §7 · p. 12Takeaways & Limitations
As tool latency increasingly dominates agent wall time, the boundary between speculative prediction and validated commit becomes a key design axis for agent runtimes.
Source: §7 · p. 12TomasuLLM targets single-session latency rather than throughput-normalized efficiency, with speculation adding GPU and sandbox costs.
Source: §5.4 · p. 11
Abstract
from arXiv · showhide
Long-running tools can dominate coding-agent latency: compilers, test suites, and repository commands take seconds to minutes while the agent idles. This observation stall presents the same tension that drove out-of-order processors -- asequential interface hides work that can be predicted and started early, but a speculative result may become visible only after it and every earlier step have been validated. We present TomasuLLM, a runtime that executes agent tool calls out of trajectory order while preserving task-execution correctness. It drafts future actions, runs them in isolated copy-on-write sandboxes, traces their dependencies and effects, and commits results in trajectory order only after validation against committed state. Across three benchmarks spanning sub-second to minutes-long tool calls, TomasuLLM improves the reported benchmark means and scales with tool latency: 1.31x on 100 SWE-bench Verified tasks, 1.35x on 28 Terminal-Bench 2.0 tasks, and 1.27x matched progress on 18 SWE-Marathon sessions. Across 4,010 audited commit-validation records, it produces zero false accepts.
1 Introduction
TomasuLLM targets observation stalls by executing drafted tool actions ahead of the main agent while keeping speculative results invisible until validated. It reports speedups across three benchmarks and no false accepts in audited commit validation.
Motivation
- Coding-agent harnesses wait for long-running tools before issuing later-turn actions, leaving predictable future work unavailable to ordinary multi-call dispatch.On published SWE-Marathon serial trajectories, tools account for 9,315 seconds on average, while multi-call dispatch recovers 54 seconds.Source: §1 · pp. 1–2
Approach
- TomasuLLM drafts future actions, executes them in private copy-on-write overlays, and publishes observations only after real-sandbox evidence passes commit validation.The main agent remains authoritative for canonical actions, and Trace IR records actions, file versions read, observations, and prospective effects.Source: §1 · p. 2
Correctness boundary
- Calls whose dependencies or effects cannot be captured remain on the serial path as speculation barriers.The design adapts in-order commit to tool actions whose read sets are undeclared and whose effects may alter files or services.Source: §1 · p. 2
Commit validation
- A drafted action may run out of trajectory order, but it commits only within the validated prefix of consecutive steps beginning at the next uncommitted action.Unlike systems limited to same-turn dispatch or action/output-boundary reuse, TomasuLLM validates dependencies and effects.Source: §1 · p. 2
Evaluation
- 1.31× on 100 sampled SWE-bench Verified tasks, 1.35× on 28 sampled Terminal-Bench 2.0 tasks, and 1.27× matched progress across 18 SWE-Marathon tasks.The evaluation uses two H20 GPUs.Source: §1 · p. 2
2 Agent Harness Background
An agent harness executes tool actions and returns observations before the next model turn, creating an observation boundary that hides cross-turn work. Long trajectories can contain reusable dependence slack that same-turn dispatch cannot reach.
2.1 Execution Model
- A turn groups one model response with its triggered actions and observations, and a trajectory is the complete sequence for one task.The harness builds prompts, invokes the model, executes tool calls, and appends their results to the conversation.Source: §2.1 · p. 2
2.2 Cross-Turn Opportunity
- An average SWE-Marathon trajectory has 31 actions lasting at least 30 seconds; refactor- or training-style tasks have 22–75, while clone-style tasks have almost none.The paper reports a per-candidate break-even of roughly one to twelve seconds of tool time, depending on fork path.Source: §2.2 · p. 3
- In a measured six-call slack-clone chain, actual read and write sets reduce the critical path to four calls, showing trajectory order need not equal execution order.The later-turn actions are unavailable to serial dispatch because they have not yet been decoded, not because all are dependent.Source: §2.2 · p. 3
- Fewer than 10% of turns emit two or more actions, limiting the cross-turn slack that same-turn multi-call dispatch can exploit.The passage also reports similarly limited intra-turn parallelism in GAIA and SWE-bench.Source: §2.2 · p. 3
3 TomasuLLM Design
TomasuLLM applies out-of-order execution to tool calls: it drafts future actions, runs them in isolated overlays, and makes results visible only after in-order validation. Its operand rule determines which calls can run early and which must wait.
3 TomasuLLM Design
- TomasuLLM separates early execution from visible commit, keeping speculative state private while the main agent remains the authority on canonical actions.Source: §3 · pp. 3–4, §3.5 · p. 6
3.1 Drafting Pipeline and Speculation Barriers
- The Action Drafter proposes future actions and the Observation Drafter predicts their results, allowing later drafts to begin before earlier tools return; predictions never validate a commit.Source: §3.1 · p. 4
3.2 The Operand Rule and Scheduling
- 1.73× speedup reduces the twelve-action walkthrough from 91.4 s to 52.9 s, with stale candidates rejected and commit marks remaining in main-agent order.Source: §3.2 · pp. 4–5, Figure 4 · p. 5
3.3 Ready Operands: Workspace Files
- A ready file operand can be copied and read early, but a later committed edit that changes its version causes rejection and serial re-execution.Source: §3.3 · p. 5
3.4 In-Flight Operands: Producer Outputs
- A loaded-version record produced by a service restart is an in-flight operand that cannot be copied, so dependent tests wait for the restart to commit.Source: §3.4 · p. 6
3.5 Isolated Execution
- Each candidate runs in a private copy-on-write overlay; failed validation discards it, and effects that cannot be deferred are blocked before execution.Source: §3.5 · p. 6
3.6 Commit Validation
- Reuse requires matching action identity, fresh dependencies, valid wrapper-canonical observation, and promotable effects; the validator advances the longest consecutive validated prefix.Source: §3.6 · pp. 6–7
3.7 Prediction Consistency and Squash
- A prediction mismatch can squash descendants while the current candidate still commits after reuse validation; run-ahead is bounded by width, depth, operands, and speculation barriers.Source: §3.7 · p. 7, §3.8 · p. 7
4 Implementation
The prototype integrates TomasuLLM at the harness’s tool-dispatch and observation boundaries, using sandbox overlays, tracing, and validation records to run and audit speculative calls.
4 Implementation
- The prototype has approximately 11,200 lines of TypeScript and uses Serial Pi, with chain depth zero, as the serial baseline.Source: §4 · p. 7
- Harness integration intercepts parsed actions at dispatch and validates tool results before observations enter model context; other harnesses need an adapter and compatible wrappers.Source: §4 · p. 7
- E2B copy-on-write overlays take about 0.08 s for read-only calls and 3–5.5 s when tests need a private data copy.Source: §4 · p. 7
- Restart wrappers record the loaded version of the shared server process, while calls unable to prove unchanged in-memory state are speculation barriers.Source: §4 · p. 7
- Riker traces file events in the local process tree into Trace IR dependency and effect sets; calls whose records cannot prove complete are never reused.Source: §4 · pp. 7–8
- The Action Drafter uses a fine-tuned Qwen3-8B, while the Observation Drafter uses an unadapted 35B Qwen-AgentWorld checkpoint.Source: §4 · p. 8
5 Evaluation
TomasuLLM reduces end-to-end latency across three benchmarks, preserves commit order and task outcomes in the reported evaluations, and gains from mechanisms that exploit cross-turn tool execution. Its benefits are bounded by speculation overhead and available sandbox resources.
5.1 End-to-End Latency
- 1.31× on SWE-bench Verified, 1.35× on Terminal-Bench 2.0, and 1.27× matched progress on SWE-Marathon are the reported benchmark means over Serial Pi.Figure 6 reports the three benchmark means; the SWE-Marathon figure uses matched progress for capped runs.Source: §5.1 · pp. 8–9, Figure 6 · p. 9
- Speedups track tool-time structure across tasks, and the DeepSeek V4 Pro configuration reaches the same 1.24× geometric mean as Qwen.Tasks with more than 40% tool time average 1.60× speedup, versus 1.07× below 15%; per-task speedups agree within 0.03× on 17 of 18 tasks.Source: §5.1 · p. 9
5.2 Correctness and Outcome Preservation
- Every committed SWE-Marathon call remained in trajectory order; the 4,010-record audit found no false accepts, and all 20 injected hidden-dependency cases failed before commit.The audit also included 390 barrier records run serially without reuse.Source: §5.2 · pp. 9–10, Table 1 · p. 10
- Both configurations resolved a mean of 41.7% on SWE-bench Verified, with a 0.0-point difference, and paired SWE-Marathon hidden-verifier outcomes matched.The SWE-bench confidence interval for the difference was [−4.3, +4.3] points.Source: §5.2 · p. 10
5.3 Design Attribution
- The full system hides 0.47 of normalized baseline latency, while removing any one mechanism increases latency because mechanisms also exploit one another’s idle slots.The four removal deltas sum to 0.72 because they include interactions, so each over-credits the removed mechanism.Source: §5.3 · p. 10, Figure 8 · p. 10
- Each 10-percentage-point increase in acceptance lowers normalized latency by 0.047, while 82% of speculative chains commit at least one candidate and none commits more than four.Action mismatch and observation mismatch account for 36% and 26% of chain terminations; observation mismatches squash descendants while preserving the candidate’s execution.Source: §5.3 · pp. 10–11, Figure 9 · p. 11
5.4 Costs and Limits
- Read-only candidate preparation typically costs 1–3 seconds, while a private data copy costs 8–12 seconds; end-to-end overhead also consumes GPU and sandbox resources.TomasuLLM adds 1.6M drafter tokens and 1,480 drafter GPU-seconds per task, and raises sandbox CPU time from 2,760 to 4,310 seconds.Source: §5.4 · p. 11
- Measured speedups reach 81–96% of the unlimited-sandbox dependence roofline, with remaining gap attributed to allocation and runtime overhead.The reported summary frames the trade-off as 0.9× one Serial Pi session’s GPU budget and 1.56× sandbox CPU time for a 1.27–1.35× single-session latency reduction.Source: §5.4 · p. 11
6 Related Work
Related work spans hardware-inspired out-of-order execution, speculative tool-call prediction, transactional reliability, and static parallel dispatch. TomasuLLM adapts runtime dependency evidence and ordered commit to generated agent actions.
Out-of-order and speculative execution
- TomasuLLM adapts out-of-order execution with in-order commit to tool calls, using syscall-level file-version evidence and COW-sandbox discard instead of register/cache tracking and hardware rollback.The paper relates this design to Tomasulo’s algorithm and the reorder buffer [29, 23].Source: §6 · pp. 11–12
Speculative drafting in agent systems
- Unlike tool-call speculation systems that discard downstream work after a misprediction, TomasuLLM tracks per-file versions so only calls with changed operands require re-execution.The section contrasts this with SPORK [2], DualSpec [32], AOSpec [6], and Speculative Actions [31], and points to PASTE [26] for the prefetch-only comparison.Source: §6 · p. 12
Transactional and parallel agent execution
- Where transactional agent systems target reliability and static parallelizers require known dependencies, TomasuLLM targets latency and resolves unknown dependencies with runtime evidence at commit.The discussion contrasts Atomix [20], SagaLLM [4], and agentic-transaction proposals [27] with hS and LLMCompiler [14].Source: §6 · p. 12
7 Conclusion
TomasuLLM applies speculative out-of-order issue with in-order commit to overlap cross-turn observation stalls while keeping unvalidated results invisible. Across three benchmarks it reports 1.27–1.35× speedups and zero false accepts across 4,010 audited records.
7 Conclusion
- 1.27–1.35× speedups across SWE-bench Verified, Terminal-Bench 2.0, and SWE-Marathon, with zero false accepts across 4,010 audited commit-validation records.The conclusion reports benchmark speedups of 1.27–1.35× and zero false accepts across 4,010 audited records.Source: §7 · p. 12
- Speculative out-of-order issue with in-order commit turns cross-turn observation stalls into overlap unreachable by same-turn parallel dispatch.COW overlays, Trace IR tracking, and validated-prefix commit keep speculative results from becoming task-visible without justification from committed state.Source: §7 · p. 12
- The boundary between speculative prediction and validated commit is identified as a key design axis as tool latency increasingly dominates agent wall time.Source: §7 · p. 12
A.1 Tool Wrapper API
Tool wrappers implement the commit rule through a typed API over ATIF records, from safety classification and snapshot capture through validation and promotion or replay. Canonicalization preserves declared test-result fields while schema mismatches and untraceable effects prevent reuse.
A.1 Tool Wrapper API
- The typed wrapper API classifies call safety, captures an input snapshot, executes in an overlay, canonicalizes observations, emits Trace IR, validates commit, and promotes or replays validated effects.Source: §A.1 · p. 14
- Test canonicalization retains exit_code, failed_tests, error_types, source locations, and artifact hashes, while dropping duration, process identifiers, and absolute temporary paths.Fields not declared droppable must match exactly after canonicalization.Source: §A.1 · p. 14
- The prototype implements no remote-endpoint wrapper, leaving unversioned and non-idempotent network calls as speculation barriers.Source: §A.1 · p. 14
A.2 ATIF Records
The adapter normalizes harness-native actions and observations into ATIF, which downstream runtime components consume. It also records transition-causing harness events so candidate dispatches can be distinguished from committed observations.
A.2 ATIF Records
- ATIF normalizes harness-native records into a shared action/observation format consumed by the drafters, wrappers, tracing, validation, recovery, and evaluation components.Source: §A.2 · p. 14
- The dispatch adapter runs after Pi parses an action and before execution; the observation adapter runs after tool return and before the result enters model context.The observation side also decides whether dependent speculative work remains eligible.Source: §A.2 · p. 14
- ATIF actions carry tool name, canonical JSON arguments, working directory, and stable environment; observations carry model-visible results, schema version, and timing metadata.Source: §A.2 · p. 14
- The adapter records the harness event causing each transition, distinguishing candidate dispatch from committed observation, and the same converter normalizes 212 offline reference trajectories.Source: §A.2 · p. 14
A.3 Conservative Dependency Tracing
Riker traces file dependencies and effects for wrapped tools, with conservative rules for accesses and explicit barriers when relevant inputs or effects cannot be captured. Trace IR records provide the evidence used for commit, replay, and hazard checks.
A.3 Conservative Dependency Tracing
- Tests, compilation, search, and general bash commands are reused only when their dependency, projected-field, and effect records are complete.Riker records file events that TomasuLLM lowers into read, absence, write, and effect sets in Trace IR.Source: §A.3 · p. 14
- Each read-set entry stores a sandbox-state digest that Vdep recomputes on committed state and compares at the commit frontier.Digests cover file bytes and mode, directory state, or path absence.Source: §A.3 · p. 14
- Tracing covers the local process tree but not detached daemons, external database internals, sandboxed multiprocess browsers, remote services, or network state.Inputs such as environment, process identity, enumeration, time, or randomness become barriers when they affect results but cannot be captured; Figure 11 shows a measured Trace IR record and its commit-check fields.Source: §A.3 · p. 14, Figure 11 · p. 15
- Failed lookups enter the absence set, parent directories are tracked, and symbolic-link use records both link and target paths when they can affect results.Paths count as accesses whether operations such as open, stat, or rename succeed or fail.Source: §A.3 · p. 14
A.4 Overlay Promotion and Replay
Overlay promotion reuses a validated candidate sandbox; when promotion is unavailable, replay can apply a stored effect, with fresh execution as the fallback.
A.4 Overlay Promotion and Replay
- A matching frontier action promotes its overlay only after dependency, result, and effect checks pass; otherwise TomasuLLM tries validated replay or ordinary execution.Matching predicted and sandbox observations keeps descendants eligible; if they differ, only the current result is reused. Replay checks the post-state digest and rejects RAW/WAW hazards.Source: §A.4 · p. 15
A.5 Trace IR Schema and Validation
The appendix details the trace-validation record, process-load dependencies, scheduler and drafter settings, and additional evaluation results. Those results show task-level variation and ablations that clarify the roles of operand classification, drafter acceptance, and resource allocation.
A.5 Trace IR Schema and Validation
- The Trace IR appendix maps record fields to commit checks to make the record format reproducible.Table 4 is described as an implementation-level mapping; the supplied passage does not enumerate the individual field checks.Source: §A.5 · p. 15, Table 4 · p. 16
A.6 Process Loads and Holds
- In the 263-call audit, all 46 shared-process tests bound to the most recent restart, while two isolated-workspace tests bound to none.Candidates with unresolved bindings fork an overlay and wait; release, loaded-version publication, and execution start share a timestamp in the traces.Source: §A.6 · p. 15
A.7 Scheduling Inputs and Resource Settings
- The controller uses observed tool-class durations and fixed dispatch weights to rank candidates, within a six-step chain and eight-sandbox resource limit.The weights are heuristic latency estimates, not measurements or a learned policy; classes without registry entries are treated as non-speculatable.Source: §A.7 · p. 15, Table 5 · p. 16
A.8 Drafter Training and Runtime Settings
- The deployed main agent is Qwen3.8-27B; its Action Drafter uses a fine-tuned Qwen3-8B checkpoint trained on 177 trajectories, while the Observation Drafter uses unadapted Qwen-AgentWorld-35B-A3B.The corpus excludes TomasuLLM-produced trajectories; training uses one epoch, and no validation-based early stopping was used.Source: §A.8 · pp. 15–16
A.9 Additional Evaluation Results
- Every SWE-Marathon task is at or above Serial Pi, with speedups from 1.03× to 2.07×; tool-heavy tasks show larger geometric-mean gains than tasks with little tool time.The six tasks spending at least 40% of Serial Pi wall clock in tools average 1.60×, versus 1.07× for the nine below 15%; timeout-capped matched-progress ratios do not lift the distribution.Source: §A.9 · pp. 16–17, Table 6 · p. 17
- The DeepSeek V4 Pro configuration shares the task ranking and 1.24× geometric mean with the reported target model, while 17 of 18 task rows agree within 0.03×.It replaces the main agent and Action Drafter adapter while leaving the Observation Drafter, wrappers, sandbox, and scheduler unchanged.Source: §A.9 · p. 17, Table 7 · p. 17
- The single-version operand alternatives are safe but slower: permissive latency is 0.68 and conservative latency is 0.85, versus 0.58 for the two-class model.Permissive classification raises stale rejects and serial re-executions; conservative classification eliminates stale rejects but holds more dependencies and reduces overlap. False accepts remain zero in all three variants.Source: §A.9 · p. 18, Table 10 · p. 18
- The trace-driven allocation sweep estimates that speedup rises from 1.50× at K=1 to 2.05× at K=4, then stays flat through K=6.The reported configuration is the only measured point; the rest are replay estimates. The appendix also reports a measured-range drafter-acceptance slope of 0.47× per ten percentage points.Source: §A.9 · pp. 17–18, Figure 14 · p. 18, Table 8 · p. 18