PaperlayerView exact revision on arXiv

Source-linked AI summary

Substrate-Aware AI Agents: Execution Context as a First-Class Input

Manu Agrawal

arXiv:2609.05232v1cs.AI

TL;DR

Agents may lack execution-environment information needed to choose suitable actions, a condition the paper calls substrate blindness. Through controlled numerical code-generation comparisons, it tests whether disclosing RAM and time changes generated implementations and resource behavior. Contract disclosure reduced memory in 13 of 14 executable comparisons and lowered mean wall time in every cohort, while tighter-contract outcomes remained model-dependent.

  • Problem

    Agents can receive complete task specifications while lacking execution-environment information that determines whether a plan is suitable.

  • Method

    The paper compares single-turn numerical code generation from the task alone versus the same task with a RAM/time contract, without prescribing an algorithm or using post-failure repair.

  • Results

    13 of 14 executable comparisons had lower measured peak memory with disclosure, and mean wall time fell in all three cohorts, by up to 3.1x.

  • Takeaways & Limitations

    The controlled result supports treating execution context as part of an agent’s planning state because disclosure shifted generated implementation choices and resource-time profiles.

  • Takeaways & Limitations

    The RAM/time experiment does not test runtime-version disclosure; the Python 3.9.6 compatibility incident is illustrative rather than a controlled treatment.

Abstract

from arXiv · show

Autonomous AI agents increasingly select actions in environments whose memory, execution-time, runtime, compute, and operational constraints determine what counts as a suitable plan. We call the absence of this execution context from an agent's planning state substrate blindness. We test this general proposition through numerical code generation, where selected implementation choices and operational consequences are directly observable. Three frontier model configurations--Anthropic Claude Opus 5, OpenAI GPT-5.6-Sol, and Google Gemini 3.7 Flash--generate code for a high-dimensional pairwise Euclidean-distance task either from the task alone or with a 128 MB RAM and 10.0 s wall-time contract. Contract disclosure reduced measured peak process memory in 13 of 14 executable index-aligned task-only versus contract-disclosed comparisons and reduced mean wall time in all three cohorts, making execution up to 3.1x faster. Across the audited corpus, disclosure produced structural code changes including bounded blocking, float32 retention, upper-triangle traversal, and in-place or memory-mapped buffers. At a tighter 96 MB contract, independently sampled contract-disclosed cohorts achieved correct-and-within-budget outcomes of 4/5 for Claude Opus 5, 5/5 for GPT-5.6-Sol, and 3/5 for Gemini 3.7 Flash, compared with task-only outcomes of 0/5, 1/5, and 0/5; cohort mean MaxRSS and wall time were 49-74% and 35-64% lower than their task-only references. These results establish a controlled proof of concept for substrate-aware agent planning: a minimal execution contract induces proactive structural adaptation in generated programs, shifting computation away from unconstrained allocations and substantially improving observed resource-time profiles before execution.

1. Substrate blindness

Agents can be unsuitable when planning omits the execution environment, not merely task requirements. The paper names this failure substrate blindness and tests whether a minimal operating contract changes generated implementations.

  • Execution context includes operational constraints such as memory limits, runtimes, timeouts, permissions, and quotas that may be available before planning.
  • Substrate blindness is the failure to condition an agent’s plan on execution context that determines solution suitability.
  • The study supplies an operating contract without prescribing an algorithm, fine-tuning a model, or waiting for failed execution.
  • Pre-execution RAM/time disclosure changes generated implementations and improves observed resource-time profiles across three provider-configured model cohorts.
  • The preserved programs, execution profiles, and source-linked audit expose implementation adaptation rather than superficial acknowledgement of the disclosed budget.
  • The paper broadens the agenda to runtime, accelerators, tools, quota, reliability, and cost, grounded by the demonstrated memory/time intervention.

2. From task context to execution context

Task context specifies what an agent should do, whereas execution context specifies where and under which operating contract it must act. The intervention isolates this information by disclosing RAM and time without prescribing an implementation, then evaluates generated programs and their resource behavior.

  • A substrate-aware agent receives both task context and execution context while deciding which program or action to produce.
  • The prompt provides a RAM/time contract but no algorithm, block size, data-type instruction, or post-failure repair loop.
  • Single-turn numerical code generation is evaluated as an implementation-selection step before multi-turn tool feedback adds further dynamics.

3. Controlled demonstration

The controlled demonstration uses an inspectable pairwise-distance benchmark and compares task-only generation with generation given explicit RAM and execution-time limits. It records correctness, timing, and peak process memory under a fixed execution setup.

  • An 8,000 by 1,024 float32 matrix requires 256,000,000 bytes for an 8,000 by 8,000 float32 distance intermediate.Bounded block algorithms provide correct alternatives with much lower peak memory.
  • Each model cohort contains five task-only and five contract-disclosed generations across Claude Opus 5, GPT-5.6-Sol, and Gemini 3.7 Flash.The cohorts are diverse provider configurations, not tier-matched controls or provider-wide capability rankings.
  • The task-only condition supplies the task specification, while the contract-disclosed condition adds a 128 MB RAM limit and 10.0-second execution-time limit.
  • The experiment compares generated programs under two information conditions without prescribing a preferred implementation.
  • Executions run in isolated macOS subprocesses with Python 3.9.6, NumPy 2.0.2, pinned single-thread settings, and measurements of correctness, wall time, and MaxRSS.
  • Measured MaxRSS is converted from bytes to MiB, with the archived 128 MB scorer using a strict <128 MiB threshold and the descriptive 96 MB classification using <=96 MiB.
  • One blind Claude program failed because it used Python 3.10-style union syntax under the pinned Python 3.9.6 runtime.Its continuous RSS and wall-time measurements are excluded from executable-only means.

4. Results: context changes the plan

Contract disclosure changed generated implementations and improved observed memory and wall-time profiles across independently sampled model cohorts. A tighter 96 MB contract continued to reduce resource use and time relative to task-only references, although exact budget fit remained model-dependent.

  • 4.1 Memory and time improve together: 13 of 14 executable index-aligned comparisons had lower measured peak memory under contract disclosure, while mean wall time fell in all three cohorts.Mean execution was 2.52x faster for Claude, 1.68x for GPT, and 3.09x for Gemini.
  • 4.2 How generated code adapts: Disclosure changed implementation choices involving block sizing, precision handling, traversal extent, temporary buffers, and input mapping.The audited source links the aggregate resource changes to inspectable generated-code adaptations.
  • 4.2 How generated code adapts: The adaptation was not a single fixed recipe: programs varied in blocking, block geometry, float32 retention, precision promotion, temporary-array reuse, and pairwise traversal.Execution context shifted the distribution of generated strategies rather than imposing one canned response.
  • 4.3 Tighter contracts reveal graded responsiveness: All 15 retained executable 96 MB programs were numerically correct and complete within the 10-second target, while 96 MB-aware cohorts lowered mean MaxRSS and wall time versus task-only references.Tightening from 128 MB to 96 MB increased mean wall time within aware configurations by 5.3% for Claude, 9.1% for GPT, and 11.9% for Gemini; exact measured-budget fit remained model-dependent.
  • 4.3 Tighter contracts reveal graded responsiveness: Figure 2 indexes executable observations to each model's task-only-reference mean, with peak process memory above wall time and 96/128 MiB reference lines supplied separately.The 96 MB observations are independently sampled condition-level cohorts.

5. Related work

Prior work studies agent action planning, execution-centered software evaluation, efficiency benchmarking, and feedback-based repair. This paper examines the earlier implementation-selection moment by supplying execution context before generation rather than execution feedback after failure.

  • Existing agent and software evaluation: ReAct interleaves reasoning traces with task-specific actions, while SWE-bench evaluates repository issue resolution through codebase and test coordination.These examples place reasoning and execution environments at the center of agent evaluation.
  • Efficiency and execution feedback: EffiBench and Mercury evaluate generated programs for correctness and execution efficiency, including time and memory use.These benchmarks assess resource outcomes alongside functional correctness.
  • Efficiency and execution feedback: Reflexion and Self-Refine use later execution feedback to improve subsequent generations, whereas this work supplies the operating contract before implementation selection.The controlled two-condition design targets first-generation implementation choice rather than post-failure repair.

6. The broader substrate-awareness agenda

Execution context belongs in an agent’s planning state because it determines whether a computational plan is suitable for deployment. The paper’s controlled result and broader agenda show how minimal contracts can change generated implementations and support tighter resource provisioning.

  • The broader substrate-awareness agenda: A minimal RAM/time contract shifted generated implementation choices without prescribing an algorithm or waiting for execution failure.The intervention changed what evaluated frontier configurations generated while isolating execution context from algorithmic guidance and repair loops.
  • The broader substrate-awareness agenda: Execution context that materially determines plan suitability belongs in an agent’s planning state.The result remains relevant even when a task-only program works in one environment, because deployment suitability depends on the actual operating environment.
  • The broader substrate-awareness agenda: Substrate-aware agents could condition plans on GPU memory, CPU, runtime versions, tool constraints, permissions, quotas, and cost.The proposed agenda extends pre-execution planning inputs beyond the RAM/time contract tested here.
  • The broader substrate-awareness agenda: 67-90% lower measured MaxRSS × wall-time products were observed across the 96 MB condition than task-only cohort means.This duration-weighted observed process footprint is not a cloud-billing measurement, though it may support tighter provisioning when configured tiers are safely cleared.

7. Research agenda and artifact availability

The paper frames its result as a controlled demonstration in numerical code generation and identifies extensions to other operating dimensions. An evaluation archive preserves the study’s prompts, artifacts, measurements, and generation code.

  • Research agenda and artifact availability: The paper establishes a controlled demonstration of substrate-aware planning in a numerical-code setting.The stated next studies include runtime-version-aware generation, accelerator-aware multimodal computation, constrained data pipelines, and dynamic tool telemetry.
  • Research agenda and artifact availability: Future studies will test the same intervention against runtime versions, accelerators, constrained data pipelines, and dynamic tool telemetry.These extensions target operating dimensions that matter in their respective settings.
  • Research agenda and artifact availability: The evaluation archive preserves prompts, dataset hashes, raw responses, generated scripts, numerical profiles, source-linked audits, and figure-generation code.Historical artifacts are retained for provenance and are not combined with the fresh cohort.

Appendix A. Absolute resource profiles

Appendix A presents retained executable MaxRSS observations in native MiB, using panel-specific scales and reference thresholds to show absolute resource profiles. Task-only and 128 MB contract-disclosed results are reference distributions, while 96 MB results are independently sampled.

  • Appendix A. Absolute resource profiles: Every retained executable MaxRSS observation is shown in native MiB.The figure reports observed peak process memory rather than a normalized resource index.
  • Appendix A. Absolute resource profiles: Each panel uses its own MiB scale to reveal within-configuration distributions.The panel heading states the scale used for that panel.
  • Appendix A. Absolute resource profiles: Dashed lines mark the prompt-labelled 96 MB and 128 MB boundaries at corresponding 96 MiB and 128 MiB observed-RSS thresholds.These lines provide absolute reference thresholds for interpreting the distributions.
  • Appendix A. Absolute resource profiles: Task-only and 128 MB contract-disclosed results provide reference distributions, whereas 96 MB programs are independently sampled condition-level observations.The figure therefore distinguishes reference conditions from the independently sampled tighter-contract cohort.

Appendix B. Reproducibility record

The reproducibility record specifies the frozen numerical task, the added execution contract, model-generation settings, and trial-level audit contents. These details document how the two prompt conditions and their outputs were recorded.

  • Appendix B. Reproducibility record: The frozen task asks for an executable NumPy-only Python script computing total pairwise Euclidean distance for an 8,000 x 1,024 float32 matrix.The script must print TOTAL_DIST:<value> and may use only NumPy and standard-library modules.
  • Appendix B. Reproducibility record: The contract-disclosed prompt adds a 128 MB RAM limit and a 10.0-second execution-time limit to the identical task text.No algorithmic instruction is included in the contract itself.
  • Appendix B. Reproducibility record: The archive records model-specific API identifiers, output limits, timestamps, and configured sampling parameters.Claude Opus 5 used provider-default sampling because its configured API rejected explicit temperature and top-p controls.
  • Appendix B. Reproducibility record: Each retained trial records its condition, numerical result, exit status, MaxRSS, wall time, source hash, and observable structural features.The cohort tables summarize these records without collapsing heterogeneous implementations into a single strategy label.
Loading 2609.05232v1…