Source-linked AI summary
Executing as You Generate: Hiding Execution Latency in LLM Code Interpreters
Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo
TL;DR
LLM code interpreters serially generate complete programs and then execute them, leaving avoidable idle time. The paper formalizes overlapping generation, detection, and execution, and implements it in EAGER. Across broad evaluations, overlap hides most execution and reduces error-free end-to-end latency by up to 37.3%, while early interruption improves repair success by up to 44.3 percentage points.
Problem
Serial LLM code-interpreter workflows generate the complete program before execution, leaving the executor and generator idle in turn and increasing end-to-end latency.
Method
The paper formalizes parallel execution as a three-stage generation, detection, and execution pipeline and implements EAGER with AST-based chunking, dynamic batching, and early error interruption.
Results
Across four benchmarks, seven LLMs, and three execution environments, EAGER reduces error-free end-to-end latency by up to 37.3% and improves error resolution rates by up to 44.3 percentage points.
Takeaways & Limitations
Executing finalized code chunks during later token generation can hide most execution time, while earlier interruption provides earlier error feedback for repair.
Takeaways & Limitations
The implementation and evaluation target Python; compiled languages require a different backend, and reported results exclude 46 Docker cold-start outliers.
Abstract
from arXiv · showhide
Current LLM systems are increasingly equipped with a code interpreter that executes generated code to obtain results. This works serially: the model first generates the complete code, then an interpreter executes it. This sequential workflow leaves the executor idle during generation and the generator idle during execution, resulting in unnecessary end-to-end latency. Our key observation is that an LLM, unlike a human developer, emits code tokens left to right and does not backtrack over what it has already written. This makes it possible to start executing a piece of code while later tokens are still being generated. We formalize this parallel execution paradigm, modeling it as a three-stage pipeline of generation, detection, and execution, and derive closed-form latency bounds that characterize its speedup potential and operating regimes. We then present EAGER, a concrete implementation featuring AST-based chunking, dynamic batching with gated execution, and early error interruption. We evaluate EAGER across four benchmarks, seven LLMs, and three execution environments. The overlap mechanism hides almost all execution behind generation, reducing the non-overlapped portion of execution time by up to 99.8% and cutting end-to-end latency by up to 37.3% on error-free runs.
I. INTRODUCTION
LLM code-interpreter workflows generate a complete program before execution, leaving generation and execution idle in turn. The paper proposes overlapping execution of finalized code prefixes with later generation and implements this idea in EAGER.
- Motivation: Code-interpreter systems commonly make users wait for complete program generation and subsequent execution.The serial workflow leaves the interpreter idle during generation and the model idle during execution.
- Parallel execution: Autoregressive models emit tokens left to right without rewriting prior tokens, making completed statements safe to execute immediately.Parallel execution dispatches each complete statement while later code is still generated.
- Parallel execution: Parallel execution changes latency from roughly Tgen + Texec to Tgen + Ttail, where only the final chunk’s execution remains on the critical path.This overlaps earlier execution with later generation.
- EAGER: EAGER implements the paradigm with AST-based chunking, gated dynamic batching, and early error interruption.The framework supports script-style Python code generation and execution.
- Evaluation: EAGER is evaluated across four benchmarks, seven LLMs, and three execution environments, measuring latency and repair benefits separately.The evaluation examines the conditions under which overlap and early interruption apply.
II. PARALLEL EXECUTION
The paper replaces sequential code execution with a streaming pipeline that overlaps generation, detection, and execution. Complete executable chunks are detected from the token stream and run while later tokens are generated.
- Workflow: Current LLM code execution generates the entire program before invoking the execution environment, creating idle time on both sides.This sequential separation increases user-perceived latency.
- Workflow: The proposed streaming pipeline overlaps generation, detection, and execution across executable code chunks.The LLM produces tokens, a detector identifies chunks, and an execution engine runs them concurrently.
- Notation: The model tracks total tokens L, generation speed vgen, time-to-first-token TF T, and executable chunk count N.The model also includes residual detection delay δi, per-chunk setup overhead Tsetup, and chunk execution time Texe,i.
- Notation: Each executable chunk is defined by residual detection delay δi, per-chunk setup overhead Tsetup, and execution time Texe,i.These quantities characterize costs that affect the pipeline’s critical path.
1) Serial Execution:
The paper contrasts monolithic serial execution with chunked parallel execution. In the parallel formulation, chunk generation and detection feed execution subject to readiness and completion constraints, with final-chunk completion determining latency.
- 1) Serial Execution:: Serial execution generates the complete program and then runs it as a monolithic block.The serial baseline avoids repeated per-chunk setup and streaming detection overhead.
- 1) Serial Execution:: The chunked formulation approximates execution using per-chunk generation and execution terms while distinguishing it from the serial baseline.The serial baseline incurs one-time setup rather than repeated per-chunk setup overhead.
- 2) Parallel Execution:: In parallel execution, chunk i becomes executable only after its generation and detection complete, while generation proceeds cumulatively across preceding chunks.The model defines chunk-ready timing through online detection and residual detection delay δi.
- 2) Parallel Execution:: Chunk i starts only after detection and completion of the previous chunk, and overall latency is the completion time of the final chunk.The recurrence can be unrolled into a closed-form characterization.
3) Latency Bounds:
The paper derives closed-form latency bounds for the parallel generation–detection–execution pipeline, identifying structural constraints and the regimes that determine latency. The bounds show how detection and repeated setup overhead limit overlap.
- The closed-form latency expression yields upper and lower bounds for the parallel execution pipeline.The lower bounds follow from generation and execution constraints, while the upper bound corresponds to zero-overlap execution.
- The upper bound captures added overhead from streaming detection and repeated per-chunk setup.Detection residuals are typically milliseconds, so cumulative setup cost can dominate the overhead.
- When setup overheads are negligible, the parallel scheme introduces no regression relative to serial execution.
- Generation and execution constraints provide complementary lower bounds on parallel latency.The system must generate all tokens before the final chunk completes and execute all chunks in order after the first becomes available.
- The lower-bound terms separate generation-dominated and execution-dominated latency regimes.The first term dominates when generation is the bottleneck; the second dominates when execution is the bottleneck.
4) Speedup Bounds:
The speedup analysis bounds the benefit of parallel execution and identifies when it is guaranteed not to regress below serial execution. The guarantee depends on cumulative chunk overhead remaining within the serial setup cost.
- When final-chunk detection and execution are small, speedup approaches 1+(T(full)exe)/(TF T + L/vgen).
- S ≥1 whenever the one-time serial setup cost exceeds cumulative chunk overhead.The condition is T(full)setup ≥ δ̄ + N Tsetup.
- The no-regression condition depends on the execution environment.Persistent REPL sessions can keep per-call overhead small, whereas container cold starts or cross-process IPC may make cumulative overhead non-negligible.
5) Regime Analysis under Uniform Chunks:
Under uniform chunks, the pipeline has generation-dominated, execution-dominated, and balanced regimes determined by per-chunk generation and execution times. Chunking improves overlap only up to a critical count, beyond which cumulative setup overhead degrades latency.
- R1: Generation-dominated: Generation-dominated execution occurs when α > β, hiding every chunk except the last behind generation.
- R2: Execution-dominated: Execution-dominated execution occurs when α < β, hiding generation for every chunk except the first behind execution.
- R3: Balanced: The balanced regime occurs when α = β, so the pipeline is perfectly paced.Here α is per-chunk generation time and β is per-chunk execution time including setup.
- Critical chunk count: For N ≤ N∗, additional chunks improve overlap; beyond N∗, cumulative setup overhead dominates and latency degrades.The critical chunk count is positive whenever total generation time exceeds total execution time.
III. IMPLEMENTATION
EAGER implements parallel execution with an AST-based chunker and an executor connected through a pending queue. The executor batches and gates chunks, while runtime errors interrupt generation early.
- EAGER instantiates parallel execution as a concrete implementation of the theoretical framework.Its design targets lower detection and per-chunk setup overhead without affecting execution outcomes.
- Detected chunks enter a pending queue, where the executor applies gating and batching before running them in a persistent session.The architecture consists of a chunker and executor connected through queued dispatch.
- Runtime errors trigger an interrupt signal that terminates LLM generation immediately.
- Producer: AST-Based Chunker: The chunker incrementally identifies complete Python statements from streaming LLM tokens using AST parsing.It recognizes boundaries when the buffer forms a complete top-level statement, while accounting for cases where syntactic validity does not yet imply completion.
- Dispatched chunks are designed to be both syntactically complete and semantically independent.
B. Consumer: Gated Executor with Dynamic Batching
The executor preserves state across chunks, dynamically batches pending work, and gates chunks that are unsafe or unprofitable to execute eagerly. Runtime errors can interrupt generation early, although interruption is optional.
- A persistent execution session preserves imports, variable bindings, and function definitions across successive chunk executions.
- Dynamic batching merges pending chunks when the executor becomes available, adapting to differences between detection and execution pace.Batching also reduces setup overhead when execution is slower than detection.
- The gating policy defers low-yield chunks and falls back to serial execution for chunks with external side effects, timing sensitivity, or dynamic execution.
- A static AST name-based gate uses a conservative denylist of standard-library modules with common external effects, extensible by the deployer.
- Error handling: EAGER catches runtime errors after the failing chunk finishes, terminates generation, and returns the error with code generated through failure.
- Error handling: Early interruption reduces wasted generation and provides earlier feedback that improves repair success rates in most scenarios.
- Error handling: Interruption can be disabled so generation continues and the error is reported after the full program, while prefix execution still overlaps generation.
IV. EXPERIMENT SETUP
The experiments test EAGER’s latency and repair effects across simulated and real generation conditions, using four Python benchmarks and seven diverse LLMs. The setup targets latency savings, generalization, and error-repair outcomes.
- The evaluation asks how latency savings vary with token generation speed and execution environment.
- A second question tests whether simulated-generation latency savings generalize to real LLM code generation.
- A third question tests whether earlier error feedback helps or hinders subsequent LLM code repair.
- Benchmarks: The study uses four benchmarks requiring executable Python scripts, covering data analysis, visualization, and version-specific code generation.
- LLMs: The evaluation spans seven open-weight and proprietary LLMs with varied scales, speeds, and architectures.
C. Execution Environments
EAGER is evaluated in local, Docker-based, and Open Interpreter environments, while latency is measured through non-overlapped execution and end-to-end wall-clock time. The setup also validates chunk reconstruction and execution equivalence.
- Execution environments: The three environments are local execution, isolated Docker execution, and the Open Interpreter sandbox.They represent progressively different execution abstractions and allow testing across practical deployment settings.
- Metrics: Non-overlapped Execution Latency measures execution time outside generation, so lower values indicate more execution hidden from user-perceived delay.
- Metrics: End-to-End Latency measures wall-clock time from the LLM call’s start through code-execution completion.
- Implementation details: Timing experiments isolate CPUs and use persistent executors that preserve program state across chunks.Local and Docker runs use persistent Python REPL subprocesses; Open Interpreter uses its native Jupyter-kernel backend.
- Integrity validation: EAGER validates program integrity by concatenating emitted chunks and comparing the reconstruction with the original generated code.
- Integrity validation: For deterministic benchmark programs, lossless reconstruction implies execution equivalence because one persistent session carries state across chunk boundaries.
B. RQ1: Latency Savings Across Generation Speeds and Environments
EAGER’s latency savings are evaluated by replaying benchmark programs at controlled token rates and by measuring real LLM outputs. Overlap reduces non-overlapped execution substantially, while early interruption improves error-path latency and often repair outcomes.
- RQ1: The simulated study replays benchmark solutions at 20, 50, 100, and 200 tokens per second across three execution environments.
- RQ1: 89.8% of execution time is hidden behind generation on average across tested configurations, with E2EL savings reaching 34.9% at 200 TPS under Docker.
- RQ2: For error-free runs, EAGER often reduces NEL to near zero, while faster models obtain larger E2EL savings because execution occupies more of total latency.
- RQ2: Real LLM outputs generalize the simulated savings across seven models and four benchmarks, reducing E2EL by up to 37.3% on error-free executions.
- RQ2: Early interruption delivers error-path savings of up to 75.0% and can substantially reduce waiting even when post-generation NEL is unavailable.
- RQ3: Repair comparisons include a stochasticity caveat: 37% of cases showed identical input code, with differences attributed to LLM repair stochasticity.
- RQ3: Partial-code repair improves error resolution by 2.1 to 44.3 percentage points on three data-centric benchmarks, but GitChameleon is an exception.The truncated suffix there contains version-specific API context needed for repair.
VI. THREATS TO VALIDITY
EAGER’s validity is bounded by its evaluation scope, execution assumptions, and conservative statement classification. The authors also report that excluding Docker cold-start outliers has little effect on aggregate results.
- Language generalizability: Python is the evaluation target; extending EAGER to compiled languages requires a different backend and raises compilation-overlap questions.The pipeline is described as language-agnostic, but compiled-language execution is left to future work.
- Language generalizability: EAGER avoids regression relative to serial execution when cumulative per-chunk overhead stays within the one-time serial setup cost.This condition was met in the evaluated REPL setup, where no configuration regressed.
- Benchmark representativeness: The method targets single-file scripts, while its broader applicability depends on execution backends and scheduling assumptions.The discussion distinguishes the scheduling mechanism from backend-specific compilation questions.
- Docker cold-start outliers: The reported results exclude 46 of 990 tasks with abnormal Docker virtual-environment cold-start overhead.Removing this filter changes aggregate results by less than 1 percentage point.
- Execution assumptions: EAGER preserves serial behavior for effect-bearing statements, but its static gating check conservatively runs unclassifiable statements serially.The completeness of this classification is treated separately from the latency mechanism.