Source-linked AI summary
Metis: Typed Runtime Mediation for Tool-Using Software Agents
Jun Yu
TL;DR
Software agents can turn probabilistic model output into changes to repositories, processes, networks, and graphical applications, but existing approaches leave runtime admission and execution handling under-specified. Metis converts provider streams into typed events with explicit permission, scheduling, result, and lifecycle edges, then evaluates those mechanisms on frozen artifacts. Across 30 matched real-I/O pairs, four-class mediation was faster than forced serialization, while the studies also exposed bounded authority and identifier limitations.
Problem
Existing agent approaches enable tool use and repository interaction but do not by themselves specify which component admits, orders, authorizes, and closes proposed calls before external effects.
Method
Metis converts provider streams and model-proposed calls into typed runtime events and an inspectable execution trace with explicit permission, scheduling, terminal-result, and lifecycle edges.
Results
Four-class mediation produced a 14.146 ms median versus 25.958 ms under forced serialization across 30 matched real-I/O pairs, faster in 30/30 pairs.
Takeaways & Limitations
The evidence supports bounded claims about dispatch, permission routing, child authority, and provider-valid trace closure.
Takeaways & Limitations
The studies do not generalize to remote networks, large repositories, GUI operations, competing runtimes, other tasks, or broader model competence and semantic safety.
Abstract
from arXiv · showhide
Software agents connect probabilistic model output to operations that change repositories, processes, networks, and graphical applications. We present Metis, a multi-provider runtime that converts provider streams into typed events before admitted calls reach external effects. Its execution path makes permission decisions, interference classes, terminal results, and lifecycle transitions explicit and inspectable. We evaluate these mechanisms on frozen source artifacts. Across 30 matched real-I/O pairs, four-class mediation reduced median elapsed time from 25.958 ms under forced serialization to 14.146 ms. The mean paired difference was -12.295 ms (95% bootstrap interval [-12.968, -11.694]), with mediation faster in all pairs. A ten-case fault matrix exposed duplicate-identifier and rollback limits. In a child-boundary ablation, the full gate-plus-registry condition blocked the declared unauthorized effect and hid all five escape tools. Removing both protections reversed both observations. A decision-only permission oracle matched all ten declared cases across five invocation routes. Five model conditions also completed a fixed Read-marker protocol in 3/3 trials each. These results support bounded claims about dispatch, permission routing, child authority, and provider-valid trace closure. They do not establish model competence, semantic safety, rollback, or superiority over another runtime.
1 Introduction
Metis treats runtime mediation as a typed, inspectable boundary between probabilistic model output and external effects. It makes permission, scheduling, result closure, lifecycle, and bounded child authority explicit while evaluating mechanisms on frozen artifacts.
- Software-engineering agents can alter repositories, processes, remote services, and graphical applications, making the intervening runtime a software-engineering object.
- Existing agent research enables tool use and repository interaction but leaves admission, ordering, authorization, and terminal-result handling to the surrounding environment.
- Metis converts model-proposed calls into typed runtime events and a derived execution trace with explicit permission, scheduling, terminal-result, and lifecycle edges.
- The runtime routes interactive, protocol, and headless permission handling through one gate and resolves built-in, plugin, and MCP tools through one registry.
- Metis specifies a five-mode permission order and four-class dispatcher, including batch preflight before effects.
- The evaluation uses paired real-I/O, fault, child-boundary, route-level permission, and model-protocol studies on frozen artifacts, with explicit validity limits.
2 Related Work
Related work places Metis in the systems layer between model policy and environmental execution. The paper distinguishes runtime mediation from reasoning, harness design, recursive orchestration, memory, tool representation, and security evaluation.
- ReAct and Toolformer address how models propose or exploit actions, while Metis addresses admission, interference, and terminal-result handling.
- SWE-agent, OpenHands, and related work treat repositories, commands, and feedback as a first-class harness, whereas Metis occupies the runtime mediation layer.
- Recursive harnesses and Metis children both distinguish model policy from execution environment, but Metis traces authority narrowing without establishing improved task success.
- Metis uses event graphs for inspectable runtime execution, unlike orchestration-trace work focused on optimization and credit assignment.
- The paper contrasts Metis with memory, tool-representation, and attack studies, emphasizing architectural coverage rather than factual recall, skill acquisition, or transferred defense rates.
- Table 1 compares overlapping research lineages by where control and failures reside, not by benchmark performance.
3 Problem Formulation and Threat Model
Metis models provider output and tool execution as a typed event graph governed by authorization, interference ordering, result closure, and bounded continuation. Its threat model includes malformed calls, failures, context pressure, and child propagation.
- A provider stream is normalized into an ordered sequence of shared content blocks before tool-use events enter runtime mediation.
- Each tool-use block contains an identifier, tool name, and structured input; the runtime decides admission, schedules execution, and returns a paired result.
- A run induces a typed directed graph whose vertices are runtime events and whose edges encode order, decisions, pairing, or lifecycle dependence.
- The target properties are authorization before effects, ordered interference, terminal-result closure, and structurally valid continuation after context control or repair.
- Table 2 separates runtime assumptions from properties that the evidence does not establish at each trust boundary.
- The threat model covers malformed or adversarial calls, misleading output, policy errors, concurrency metadata errors, tool failures, cancellation, interruption, context pressure, and child propagation.
4 Runtime Design
Metis mediates provider output through typed events, ordered permission decisions, interference-aware scheduling, terminal-result closure, lifecycle repair, context control, and constrained child authority. The design makes runtime transitions explicit while documenting boundaries around fail-open classification, identifier ambiguity, semantic preservation, and process isolation.
- Typed events and provider normalization: Provider adapters accumulate complete calls, incremental arguments, and thinking blocks into ordered normalized blocks before dispatch.The invariant is ordering and parseable shared structure for implemented adapter cases, not semantic identity between provider APIs.
- Loop state machine: The agent loop alternates context preparation, provider streaming, assistant persistence, and permissioned tool dispatch, with terminal events for turns without calls.Turns with calls append ordered results and begin another iteration; return paths include repair handling.
- Four-class scheduling: Input-sensitive classes permit Safe overlap, FIFO Queue execution, Exclusive barriers, and Background handshakes before detached work completes.The idealized critical path assumes sufficient resources and no undeclared Safe–Queue dependency; evaluation instead uses paired real-I/O observations.
- Terminal-result closure and repair: Supported in-process failures and successes produce result blocks in input-call order, preserving identifier multiplicity and order when batch handling returns.This property neither establishes identifier uniqueness nor rolls back external effects, and it does not survive arbitrary host termination.
- Permission semantics and batch preflight: Permission resolution orders plan checks, bypass-immune checks, rules, path scope, and mode fallback, while batch preflight resolves every pending question before any admitted effect.Classifier failure in explicit bypass mode remains a documented fail-open boundary, although hard checks remain.
- Lifecycle repair and context control: Interruption repair restores identifier coverage with stubs, while context controls escalate from pruning and snipping to offload, collapse, compaction, and retry.These mechanisms prioritize protocol structure and acceptance; lossy transformations do not imply semantic equivalence.
- Sub-agents and boundaries: Child surfaces intersect parent visibility, profile and call-site allowlists, then subtract denylists, using a cloned gate and filtered registry.Optional worktree isolation separates repository writes but does not provide process isolation or isolate network access, credentials, host resources, or provider state.
5 Evaluation
The evaluation uses frozen artifacts and controlled studies to examine runtime conformance, scheduling, terminal closure, authority boundaries, and protocol behavior. Results support bounded mechanism claims while exposing important scope limits.
- Evaluation design: The evaluation combines frozen logs, paired real-I/O timing, injected faults, child-boundary ablation, permission decisions, and model protocol trials.The studies use different evidence levels, and the decision oracle does not execute proposed effects.
- RQ1: Snapshot conformance: The baseline recorded 4,998 pass, 2 fail, and 31 skip leaf events, but its 63.5% coverage profile was not a whole-tree certificate.The companion recorded 297/6/1 leaf pass/fail/skip events with a partial 64.2% profile.
- RQ2: Execution mechanisms: 14.146 ms median elapsed time under four-class mediation compared with 25.958 ms under forced serialization across 30 matched real-I/O pairs.The mean paired difference was −12.295 ms, with a 95% bootstrap interval of [−12.968, −11.694] ms; mediation was faster in 30/30 pairs.
- RQ2: Execution mechanisms: The fault matrix repaired a normal orphan and paired a Background handshake with its effect, but duplicate identifiers and partial mutation prevented stronger closure guarantees.The observed limits were no duplicate-ID uniqueness, no transactional rollback, and no one-to-one terminal closure after restart with a duplicate ID.
- RQ3: Authority boundaries: The full child gate-plus-registry condition blocked the declared unauthorized effect and exposed 0/5 escape tools, while removing both protections admitted the effect and exposed 5/5.Because both protections were removed together in one deterministic case per condition, their independent effects remain unidentified.
- RQ3: Authority boundaries: The permission oracle matched all ten declared authorized and unauthorized decisions across five routes, while five model conditions each passed the fixed Read-marker protocol in 3/3 trials.The model protocol retained 15/15 trials, requiring unique identifier closure, marker presence in the result, and marker return in final text.
6 Discussion
Metis separates runtime mediation from model policy by making permission, dispatch, failure, and lifecycle behavior explicit. The discussion emphasizes that this improves inspectability without establishing semantic authorization, task effectiveness, or isolation.
- Metis isolates runtime decisions from policy proposals, distinguishing action proposal, environment exposure, and effect mediation.
- The event graph gives failures typed locations, including decision edges for denials, error results for panics, and lifecycle events for compaction.
- Terminal closure and task success are separate measurements because paired identifiers can coexist with an incorrect task result.
- External effects remain coverage-dependent because equivalent actions may traverse built-ins, shells, plugins, aliases, or MCP services.
- Context compaction can preserve structural validity while omitting evidence needed for a good decision.
- Child authority is filtered through a cloned gate and registry, but shared host, network, provider, budget, and spill state prevent claims of process isolation.
7 Threats to Validity and Required Evaluation
The validity analysis limits Metis’s evidence to frozen, mechanism-specific workloads and identifies unresolved risks in generalization, rollback, permission classification, and host-level isolation. Further studies are required beyond the completed narrow evidence.
- Threats to Validity: Terminal events are log records, not independent observations, and the studies measure declared decisions, protocol structure, or task oracles rather than broad agent capability.
- Threats to Validity: The paired dispatcher design reduces stable host and order effects, but caches, scheduling, transport, and programmed delay remain part of the workload.
- Threats to Validity: The real-I/O probe excludes remote networks, large repositories, GUI operations, competing runtimes, and broad task generalization.
- Threats to Validity: Scheduling assumes accurate tool declarations, terminal closure is not transactional rollback, permission checks can misclassify intent, and compaction can remove critical evidence.
- Required Evaluation: Table 6 identifies studies required beyond the completed mechanism evidence.
8 Conclusion
Metis makes tool-mediated runtime behavior explicit through typed events and supports bounded mechanism claims across dispatch, permission routing, child authority, and provider-valid trace closure. The evidence does not establish product-level safety or cross-runtime superiority.
- Metis converts model-proposed calls into typed events with explicit permission, scheduling, terminal-result, and lifecycle edges.
- The completed studies support mechanism-specific claims while retaining the maintenance trace as treatment-activation evidence rather than an effectiveness estimate.
- Repeated randomized tasks with a predeclared whole-tree baseline remain necessary for an effectiveness claim.
Artifact Availability
A de-identified replication artifact is being curated for public release, but it is not part of this preprint version.
- The planned artifact will contain normalized study records, task definitions, exclusion decisions, and deterministic analysis scripts.
- No human-participant or private user dataset was used.