Source-linked AI summary
Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints
Saransh Dhage
TL;DR
LLM agents can vary across identical runs, creating a deployment problem for regulated, auditable workflows. This paper evaluates deterministic execution harnesses across two synthetic tasks and two open-weight models, finding that Structured Planning removes residual plan variance but introduces model-dependent latency costs.
Problem
Run-to-run agent variance is acceptable for exploratory use but blocks predictable, auditable procedures in regulated domains, while prior work lacked a powered mechanism-level account of constraint effects.
Method
The study compares baseline agents, a first-pass deterministic harness, and Structured Planning using trace-level decomposition across two tasks and two open-weight models.
Results
The first-pass harness has mixed reproducibility effects, whereas Structured Planning closes the residual variance gap and achieves perfect or near-perfect reproducibility with near-perfect task success across all four cells.
Takeaways & Limitations
Harness engineering can improve agent reliability, but evaluation should diagnose per-layer effects and report model-specific latency rather than rely on aggregate claims.
Takeaways & Limitations
The study evaluates only two synthetic linear four-state tasks and two open-weight model classes, so its conclusions should not be assumed to generalize to branching, judgment-based, proprietary, or reasoning-specialized settings.
Abstract
from arXiv · showhide
Large Language Model (LLM) based agents exhibit substantial run-to-run execution variance even when given identical tasks and tools -- acceptable for exploratory use but unacceptable in regulated domains such as finance and compliance. We study harness engineering: wrapping an agent in a deterministic execution layer (finite-state control, forced tool selection, output validation, bounded retry, and structured planning) and measuring its effect on execution determinism and task success. Across two synthetic tasks (finance and legal) and two open-weight models (Qwen-2.5-7B-Instruct, Gemma-3-27B), a first-pass harness produces a mixed result: it significantly improves reproducibility in one of four model-task cells, significantly degrades it in two, and has no effect in the fourth. A trace-level diagnostic finds the cause: once tool sequence, state sequence, and output are already highly consistent, an unconstrained free-text planning step becomes the dominant remaining source of variance. Adding Structured Planning -- validating the plan against a fixed schema before any tool is invoked -- eliminates the effect entirely: three of four cells reach a Reproducibility Rate and Determinism Index of 1.000 at N=100, and task success rises to 100% in three of four cells. The gain is not free: token cost falls in every cell, but latency shows a genuine, sample-size-robust split by model -- one becomes faster under the constraint, the other markedly slower. We argue harness engineering is a distinct, effective discipline for agent reliability, but its cost is model-dependent and must be measured, not assumed.
1 Introduction
The paper asks whether deterministic execution harnesses can reduce agent variance without harming success, and uses trace-level diagnosis to identify which constraint is needed and what it costs.
- Motivation: The study targets run-to-run agent variance that blocks predictable, auditable procedures in regulated domains.Prior work measured inconsistency but did not intervene on it.
- Research question: The paper asks whether wrapping an agent in a deterministic harness reduces variance without reducing task success.
- Findings: The first-pass harness significantly improved reproducibility in one model×task cell, degraded it in two, and had no detectable effect in the fourth.The harness used finite-state execution, forced tool selection, output validation, and bounded retry.
- Contributions: Trace-level diagnosis decomposes reproducibility into plan, tool-sequence, state-sequence, and output layers instead of treating it as one composite number.
- Contributions: Structured Planning schema-validates plans before execution, closing residual variance and achieving perfect measured reproducibility and near-perfect task success across two tasks and two models.
- Cost: Token cost is uniformly favorable, while latency cost is model-dependent and must be measured per model.
2 Related Work
Related work measures agent consistency, constrains workflow execution, adapts runtime interfaces, and distinguishes orchestration determinism from inference-level numerical nondeterminism.
- Consistency measurement: Prior observational work evaluated repeated-run consistency across six models and 19 tool-calling tasks without execution-level constraints.
- Constrained execution: R-LAM combines fixed ordering, schema-validated actions, and provenance logging, reducing trace variance to zero while leaving task accuracy unchanged in one workflow.
- Runtime harness adaptation: Life-Harness adapts the runtime interface rather than model weights and improves most evaluated deterministic model–environment settings.Its harnesses transferred from one small model’s trajectories to seventeen others.
- Inference-level determinism: The paper scopes its claims to behavioral orchestration determinism and controls inference-level noise with fixed low-temperature decoding and single-provider routing.
- Structured execution: Prior finite-state reliability work and procedure-execution studies motivate explicit failure taxonomies and structural constraints for longer workflows.
3 The Harness
The harness inserts a controllable layer between the agent and tools, combining fixed state progression, authorized tool calls, validated outputs, bounded retries, and optional structured planning.
- Architecture: The harness is implemented as an independently toggleable control layer between a LangGraph agent and its tools.
- Finite-state execution: Finite-state execution restricts the agent to a predefined sequence of named states.The example sequence is LOAD DATA → VALIDATE DATA → CALCULATE → GENERATE REPORT.
- Forced tool selection: Forced tool selection binds each state to exactly one authorized tool through API-level tool choice="required".
- Output validation: Output validation checks each tool result’s shape and type before the state machine can advance.
- Bounded retry: Bounded retry permits only a fixed number of attempts after validation failure before escalation halts the run.
- Structured Planning: Structured Planning requires a JSON array of state–tool objects, validates it against the finite-state graph, and blocks tool calls until a valid plan exists.Canonical schema-validated pairs are logged instead of raw free text, removing plan wording as measured variance.
- Conditions: Baseline is unconstrained, Harness omits Structured Planning, and Harness+SP includes the full harness.
4 Determinism Metrics
The paper measures determinism, exact trace reproducibility, task correctness, and resource cost, using pairwise comparisons and statistical tests across runs.
- Metric framework: The study computes three metric families over pairwise comparisons across all runs within each condition.
- Determinism Index: Determinism Index averages Plan Stability, Tool Path Consistency, State Transition Stability, and Output Consistency on a 0–1 scale.
- Reproducibility Rate: Reproducibility Rate is the fraction of runs exactly matching the modal complete trace.Even one differing plan-text token breaks an exact match despite identical execution behavior otherwise.
- Task success: Task Success Rate measures correctness against deterministic ground truth separately from determinism.A run can be deterministic and wrong, or correct and non-reproducible.
- Resource cost: Token count and wall-clock latency quantify whether reproducibility gains require materially greater resource cost.
- Statistical analysis: Each contrast uses a bootstrap confidence interval, permutation test, and Cohen’s d, with significance at α = 0.05.Both resampling procedures use 10,000 iterations.
5 Experimental Setup
The experiment evaluates two deterministic synthetic pipelines on two open-weight models, with controlled infrastructure and matched 100-run conditions.
- Tasks: The finance ecl and legal clause tasks are four-state linear pipelines with deterministic ground truth, designed to isolate execution variance from task difficulty.Finance ecl uses a 12-loan portfolio with four deliberately invalid loans; legal clause classifies 10 contract clauses by fixed-priority keywords.
- Models: The study evaluates qwen/qwen-2.5-7b-instruct and google/gemma-3-27b-it through OpenRouter.
- Infrastructure and confound control: Provider pinning, fixed low-temperature decoding, and response reissuance after provider-side failures control infrastructure and inference confounds.Malformed responses caused by a provider outage were discarded and reissued rather than scored.
- Sample sizes: Baseline and Harness use N = 100 runs per model×task cell, while Harness+SP is extended from N = 50 to N = 100 for matched comparisons.The extension tests whether early Structured Planning estimates were sample-size artifacts.
6 Results
The first-pass harness has a mixed reproducibility effect because unconstrained plan wording remains variable, whereas Structured Planning resolves this variance with favorable token costs but model-dependent latency.
- 6.1 Stage 1: The first-pass harness produces a mixed effect: Of four model×task cells, the first-pass harness improves reproducibility in one, degrades it in two, and has no detectable effect in one.Task Success Rate remains 0.97–1.00 across conditions, so the inconsistent reproducibility effect does not reflect a correctness trade-off.
- 6.2 Diagnosis: plan-text is the only unconstrained axis: Tool Path Consistency, State Transition Stability, and Output Consistency are near ceiling, but Plan Stability remains low and variable because free-text plans are unconstrained.Plan wording can mechanically move exact-match Reproducibility Rate away from 1.0 through phrasing differences such as “load document” versus “Load Document.”
- 6.3 Stage 2: Structured Planning eliminates the effect: Structured Planning drives three of four cells to Reproducibility Rate and Determinism Index values of 1.000 at N = 100, with p < 0.001 versus baseline in all three.The remaining finance ecl/Qwen cell reaches 0.980 because of two genuine per-state tool-calling failures, not residual plan-text variance.
- 6.3 Stage 2: Structured Planning eliminates the effect: Task success reaches 100% in three of four cells under Harness+SP, while finance ecl/Qwen improves from baseline 0.97 to 0.98–1.00 across the two extensions.
- 6.4 Cost: token savings are universal, latency cost is model-dependent: Structured Planning adds 2–5% tokens over plain Harness but remains 15–17% below baseline in every cell.The harness’s tool-call savings outweigh the schema-plan round-trip overhead.
- 6.4 Cost: token savings are universal, latency cost is model-dependent: Latency splits by model: Qwen is 12–21% faster than plain Harness, whereas Gemma is 15–24% slower.At N = 100, Gemma’s latency tax grows relative to N = 50, while Qwen’s advantage holds in direction and magnitude.
7 Discussion
Structured Planning resolves residual planning variance, but its benefits and costs differ by model. Trace-level diagnosis and cost accounting are therefore essential for interpreting harness results.
- Three of four cells reached perfect Reproducibility Rate and Determinism Index after Structured Planning, creating a ceiling that limits metric discrimination.Once saturated, these metrics cannot distinguish effective interventions from overengineering; token and latency costs remain informative.
- Structured Planning improved Qwen on reproducibility, task success, token use, and latency, while Gemma incurred a robust latency cost for comparable gains.The two models therefore exhibit materially different cost profiles under the same intervention.
- Trace-level decomposition showed that the first-pass harness constrained execution layers but left free-text plan wording as the dominant variance source.Without per-layer analysis, mixed composite scores could be mistaken for evidence that harnessing is fundamentally unreliable.
8 Limitations
The study’s conclusions are bounded by its synthetic linear tasks, two open-weight model classes, third-party inference infrastructure, and incomplete mechanistic analysis.
- The evaluation covers two synthetic linear four-state tasks, excluding branching, data-dependent, and judgment-based execution paths.The authors caution that conclusions should not be assumed to generalize to tasks whose correct paths are not fixed in advance.
- Only 7B- and 27B-class open-weight models were tested, not proprietary frontier or reasoning-specialized models.The model-dependent latency result demonstrates dependence exists but does not identify which other models will show it.
- OpenRouter introduced shared-infrastructure noise, especially for latency measurements, although reproducibility metrics were structural rather than timing-based.
- The study did not mechanistically investigate why Gemma required more plan-validation retries than Qwen.
9 Conclusion
A first-pass deterministic harness alone did not reliably improve reproducibility, but schema-validated Structured Planning closed the residual variance gap in this setting. The intervention reduced token cost universally while producing model-dependent latency effects.
- The first-pass harness produced mixed reproducibility effects across four model-task cells, including two statistically significant reversals.
- Structured Planning, enforced before tool calls through schema validation, achieved perfect or near-perfect reproducibility and task success across all four cells at N = 100.
- Trace-level diagnosis identified unconstrained free-text planning as the dominant remaining variance source after execution behavior became highly consistent.
- Token cost decreased in every cell, whereas latency improved for one model and worsened robustly for another.
- Harness engineering should be evaluated by decomposing both the responsible constraint and the resource axis it trades against.