PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Haiqing Li, Xin Ma, Yinhao Wu, Wenliang Zhong, Feng Jiang, Thao M. Dang, Xiao Hu, Hehuan Ma, Yuzhi Guo, Junzhou Huang

arXiv:2609.29921v1cs.AIcs.MA

TL;DR

LLM agents follow external specifications that often remain context for the same system that acts and declares completion, leaving execution and state-authority gaps. SpecHarness compiles verifiable requirements into obligations and commits state only from admissible evidence; it improves all seven agents on artifact and guideline tasks.

  • Problem

    External specifications often remain agent-readable context, leaving a gap between understanding requirements and executing them and between an agent’s completion claim and established required state.

    Source: Introduction · p. 1

  • Method

    SpecHarness compiles verifiable requirements from agent-visible materials into source-linked obligations and lets an external runtime commit specification-governed state only from admissible evidence.

    Source: Introduction · p. 2

  • Results

    SpecHarness improves all seven agents on SkillsBench and GuideBench, raising SkillsBench macro Pass from 61.1% to 73.1%.

    Source: Results · p. 6

  • Takeaways & Limitations

    Specifications can govern which verifiable states and outcomes are accepted, rather than serving only as behavioral guidance.

    Source: Conclusion · p. 7

    SpecHarness governs the grounded, observable mandatory portion of specifications, not the full natural-language specification.

    Source: Ablation and Operational Analysis · p. 7

Abstract

from arXiv · show

Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Introduction

Agents must follow external specifications, yet those specifications often remain context for the same agents that act and declare completion. The paper identifies execution and authority gaps, measures them, and proposes an external runtime that governs specification-based state.

Introduction

  • The understanding–execution gap separates correctly interpreted requirements from incorrect execution, while the state–authority gap separates completion claims from established required state.The paper asks who may establish specification-governed state and authorize completion.Source: Introduction · pp. 1–2, Figure 1 · p. 1
  • Across seven representative language models, only 79.6%–86.4% of the directions were satisfied, while agents’ completion-claim rates exceeded official pass rates by 28.7–37.9 percentage points.Source: Introduction · p. 2
  • Post-hoc verification checks after execution, completion gating checks claims before acceptance, and runtime enforcement constrains actions during execution; these approaches do not jointly govern execution and state commitment.The intervention paradigms differ in when and how they intervene.Source: Introduction · p. 2, Figure 2 · p. 2
  • SpecHarness operationalizes the State Authority Principle: agents propose actions and completion, while an external runtime commits state from admissible evidence for source-linked obligations.Ambiguous, subjective, conflicting, or unverifiable requirements remain advisory; the architecture is evaluated across GuideBench and SkillsBench.Source: Introduction · p. 2

Related Work

Prior approaches verify completed work, gate completion, or constrain execution, but leave a gap in jointly governing execution and authoritative state. The paper’s figure also quantifies specification-following and completion-claim discrepancies across seven models.

Related Work

  • Post-hoc methods assess completed answers, trajectories, or artifacts, whereas completion gates reject unsupported claims at terminal admission but primarily leave preceding execution unconstrained.These approaches provide evidence or control acceptance without governing intermediate execution and state.Source: Related Work · p. 2
  • 79.6%–86.4% of 509 source-grounded directions were satisfied across seven models, while reported completion exceeded official verifier pass rates by 28.7–37.9 percentage points.Figure 3 presents direction satisfaction above and the completion-claim versus verifier-pass gap below.Source: Figure 3 · p. 3
  • Runtime-enforcement systems constrain behavior or state transitions during execution, while SpecHarness compiles visible specifications into source-linked obligations and uses qualified evidence to govern versioned state and finalization.Its preventive mediation is limited to closure-audited surfaces.Source: Related Work · p. 3

SpecHarness: Runtime Specification Authority

SpecHarness makes specifications govern execution and completion by separating agent proposals from authoritative state. It compiles visible requirements into obligations, then relies on authorized evidence and versioned state to control actions and finalization.

SpecHarness: Runtime Specification Authority

  • Only admissible evidence can commit versioned obligation state; agents may plan and act, but their proposals and self-assessments cannot establish authoritative state.A commit may record validated success or failure; satisfaction is determined separately.Source: SpecHarness: Runtime Specification… · p. 3
  • SpecHarness compiles visible context into source-addressable directions and task-specific obligations, using development annotations to select and freeze a compiler before execution.Held-out official verifiers are reserved for post-hoc alignment and final evaluation, not runtime enforcement.Source: SpecHarness: Runtime Specification… · p. 3
  • The obligation IR separates mandatory blocking obligations, nonblocking guidance, abstentions, and residual context; directions need not map one-to-one to obligations.Advisory, abstained, and residual directions create no independent blocking rules.Source: SpecHarness: Runtime Specification… · p. 3
  • Finalization requires all fresh mandatory obligations to be satisfied, but the guarantee covers grounded mandatory obligations, not the full natural-language specification.No-bypass applies only to closure-audited actions; isolated channels use post-effect validation.Source: Figure 4 · p. 4, SpecHarness: Runtime Specification… · p. 5
  • Only mandatory, source-grounded obligations with authorized evidence providers and qualified validators enter the hard set; hard classification alone does not ensure preventive mediation.Otherwise, an obligation remains advisory or triggers abstention.Source: SpecHarness: Runtime Specification… · p. 4
  • Closure audits identify actions eligible for bounded mediation; safely isolated declared channels use validate-and-commit, while unsupported channels are rejected.Unmatched proposals may raise an error when mediation is expected or follow the base policy when explicitly out of scope.Source: SpecHarness: Runtime Specification… · p. 4
  • Trusted observers and bound validators supply admissible evidence for atomic commits; agent claims, validator errors, and effects without admissible evidence cannot create authoritative state.Both passed and failed validation results may be committed.Source: SpecHarness: Runtime Specification… · p. 5
  • Committed results are fresh only when dependency versions match; mutations mark affected entries stale or unknown until revalidation, and recommitment requires new admissible evidence.SpecHarness returns source-linked feedback so the agent may propose repairs.Source: SpecHarness: Runtime Specification… · p. 5

Experiments

The experiments test SpecHarness against Raw and alternative methods across artifact-producing SkillsBench and guideline-constrained GuideBench, then probe architecture, freshness, and enforcement scope.

Experiments

  • SkillsBench evaluation uses 87 tasks and 509 source-grounded directions compiled from agent-visible materials; GuideBench uses 1,042 tasks and 5,817 task-level obligation instances.The frozen compiler was selected using development annotations; held-out verifiers and oracles were reserved for evaluation.Source: Experiments · p. 5
  • Raw and SpecHarness are compared across the same seven agents, with GPT-5.6 Sol fixed for comparisons against post-hoc verification, completion gating, and runtime-enforcement alternatives.GuideBench also compares against a computation substrate and evidence-authorized commitment method.Source: Experiments · p. 5

Results

  • 61.1%→73.1% Macro Pass, 17.4%→9.3% U–E, and 32.8%→12.8% S–A under SpecHarness on SkillsBench.The macro Pass gain has a 95% paired task-bootstrap interval excluding zero; the complete governed runtime includes online validation, feedback, and repair.Source: Table 1 · p. 6, Results · p. 6
  • 86.2%→91.3% macro Pass, 10.4%→5.0% U–E, and 13.8%→6.9% S–A under SpecHarness on GuideBench.The results extend the obligation–evidence–commit abstraction from artifact state to rule-local decision state.Source: Table 3 · p. 6, Results · pp. 6–7

Ablation and Operational Analysis

  • Removing mediation primarily increases U–E, removing effect validation increases unsupported acceptance, and removing commitment produces the largest S–A degradation.Removing blocking qualification lowers S–A only by reducing Raw-pass preservation; the variants do not isolate commitment under matched trajectories or compute.Source: Ablation and Operational Analysis · p. 7, Table 5 · p. 7
  • 95.8% of affected entries are restored through revalidation, and repair recovers 95.6% of tasks across 248 targeted dependency mutations.Without invalidation, every mutation leaves outdated evidence admissible for completion.Source: Table 6 · p. 7, Ablation and Operational Analysis · p. 7
  • 44.0% of SkillsBench hard obligations use mediate-and-commit, while the remaining 56.0% use validate-and-commit; GuideBench uses validate-and-commit exclusively.These allocations bound governance to the grounded, observable mandatory portion of the specification.Source: Table 7 · p. 7, Ablation and Operational Analysis · p. 7

Conclusion

SpecHarness makes state authority, rather than reasoning or verification alone, central to specification following. It operationalizes this boundary through source-linked obligations and admissible evidence, within the scope of grounded, monitorable conditions.

Conclusion

  • SpecHarness prevents agent proposals, actions, and self-assessments from directly establishing specification-governed state.Only admissible evidence can update that state.Source: Conclusion · p. 7
  • The obligation–evidence–commit architecture applies this boundary across action authorization, effect validation, freshness, and finalization.It compiles heterogeneous agent-visible specifications into source-linked conditions whose state changes only from admissible evidence.Source: Conclusion · p. 7
  • The experiments evaluate the complete architecture, including its feedback-and-repair loop, rather than isolating commitment under matched compute.Source: Conclusion · p. 7
  • The claim is scoped to grounded and monitorable conditions, where specifications can serve as an external basis for execution and acceptance.Source: Conclusion · p. 7

A Specification Compilation and Coverage

The compiler-selection protocol compares source-grounded directions against frozen development annotations, selects by development coverage, and freezes the chosen compiler before evaluation. Structural metrics audit completeness but do not determine selection.

A.1 Compiler Selection and Freezing Protocol

  • The development set contains 14 task-disjoint tasks, 65 source units, and 81 annotated directions; task-family and skill overlap remain allowed.All candidates receive the same visible materials and fixed protocol, and annotations are frozen before comparison.Source: §A.1 · p. 9
  • A direction matches an annotation only when it uses the same source unit and preserves the annotated behavior or acceptance condition.Each annotation can match once; duplicates and unmatched predictions do not increase coverage.Source: §A.1 · p. 9
  • Development coverage is the prespecified selection criterion; structural-completeness metrics are audits, and machine-checkability alone does not qualify a requirement for hard enforcement.Source: §A.1 · p. 9
  • GPT-5.6 Sol achieves the highest development coverage and is selected, after which its instructions, output schema, decoding settings, and construction procedure are frozen.Official tests, runtime results, and experimental comparisons do not revise the selection.Source: §A.1 · p. 9

A.2 Post-Selection Full-Task Coverage Audit

A post-selection audit compares frozen compiler outputs across 87 SkillsBench tasks with source-unit coverage and official-test correspondence. The selected 509 directions are a common measurement surface, not an exhaustive reconstruction of specifications or evaluator logic.

A.2 Post-Selection Full-Task Coverage Audit

  • The post-selection audit applies all seven unchanged candidates to 87 SkillsBench tasks and compares extracted directions and source-unit mapping with 585 official test functions.It does not affect compiler selection, the selected output, the obligation surface, or experimental settings.Source: §A.2 · p. 9
  • GPT-5.6 Sol is the only candidate with complete task-level direction accounting and source-unit mapping, and it also has the highest broad and fine-grained correspondence.Gemini 3.1 reaches 95.90% broad correspondence but remains incomplete on both construction measures.Source: §A.2 · p. 9
  • Official tests are inspected only after candidate outputs are frozen and are never provided to compilers, agents, runtime validators, or the repair loop.The audit does not revise extractions, construct obligations, or transfer evaluator logic into runtime.Source: §A.2 · p. 9
  • Construction completeness covers each compiler’s extracted direction surface, not complete recall of natural-language specifications.The selected 509 directions form a common measurement surface; broad correspondence does not establish recovery of complete verifier logic.Source: §A.2 · p. 9

B Runtime Implementation and Assurance

Runtime validators are built from visible specifications, observable workspace state, and frozen compiler criteria, then qualified before task-agent execution. Only qualified, admissible, fresh evidence can establish authoritative obligation state or permit finalization.

B.1 Validator Qualification and Commitment Protocol

  • A validator qualifies for blocking only when inputs are observable, criteria are deterministic and source-linked, and identical workspace and dependency states reproduce identical outputs.Qualification tests satisfying cases, targeted violations, malformed or missing inputs, unavailable providers, and execution failures.Source: §B.1 · p. 10
  • Requirements without a qualified provider remain advisory, abstained, or residual, and validators are frozen before task-agent execution.Qualification results are obtained before task-agent evaluation.Source: §B.1 · p. 10
  • SkillsBench uses 250 qualified validator instances, each bound one-to-one to a hard obligation.Shared implementations with different grounded parameters are counted separately.Source: §B.1 · p. 10
  • A qualified, admissible, fresh passed result may establish satisfaction, while a failed result may establish non-satisfaction and trigger repair.Errors do not establish satisfaction or failure, update authoritative state, or permit finalization while a mandatory obligation remains affected.Source: §B.1 · p. 10
  • Agent actions, self-reports, completion claims, tool-return strings, and unqualified observations cannot modify authoritative state.Commitment is atomic and checks the obligation, provider, validator identity and version, scope, result type, and dependency versions.Source: §B.1 · p. 10

B.2 Trusted Boundary, Closure Audit, and No-Bypass Tests

Closure-qualified mediation screens out paths with detected bypasses, while freshness invalidation prevents outdated evidence from supporting finalization. Both guarantees are conditional on the audited, observable system boundary.

B.2 Trusted Boundary, Closure Audit, and No-Bypass Tests

  • The trusted boundary includes the obligation ledger, authorization service, qualified validators, dependency-version store, and audited capability wrappers.The agent, prompts and memory, proposals, self-reports, and natural-language tool-return strings remain outside it.Source: §B.2 · p. 10
  • All 11 detected bypasses occurred on paths excluded from closure-qualified mediation, leaving no unresolved bypass on surfaces receiving the preventive no-bypass claim.The stress tests and obligation allocation use different denominators: one action surface can support multiple obligations.Source: §B.2 · p. 11, Table 11 · p. 11
  • Preventive mediation qualifies only when every reachable effect-equivalent path is represented in the capability graph and mediated within the trusted boundary.Unresolved-bypass surfaces use validate-and-commit only if their effects remain safely isolated and observable; otherwise they remain outside hard enforcement.Source: §B.2 · p. 11

B.3 Freshness Invalidation and Recovery Protocol

  • Dependency mutations invalidate affected commitments; stale or unknown entries need new admissible evidence, and errors do not update authoritative state.Revalidation can create a fresh satisfied commitment or record current non-satisfaction and trigger repair.Source: §B.3 · p. 11
  • 570 of 595 invalidated entries were restored by revalidation, and 218 of 228 affected tasks recovered after repair.Without freshness invalidation, all 248 mutations left previously committed evidence eligible for finalization.Source: §B.3 · p. 11

C.1 Tasks, Models, and Execution Settings

The experiments use paired benchmark conditions and independent post-run evaluation to compare runtime approaches. The reported SkillsBench improvements are significant across all seven task-agent models, but the comparison does not isolate commitment from other SpecHarness components.

C.1 Tasks, Models, and Execution Settings

  • All SkillsBench conditions share the same 87 tasks, workspaces, agent-visible prompts, injected skills, tool interfaces, and official evaluation procedure.Paired runs share nominal budgets and environment settings, but realized calls, tokens, repair steps, and time are not matched.Source: §C.1 · pp. 11–12

C.2 Baseline Adaptation, Runtime Access, and Completion Rules

  • Raw and post-hoc baselines accept terminal claims without an additional gate, while VeriMAP gates completion and SpecHarness uses versioned commitments and ledger-derived finalization.The compared adaptations preserve each method’s defining intervention point; official evaluation occurs independently after execution.Source: §C.2 · p. 12
  • The comparison evaluates the complete SpecHarness architecture rather than isolating the causal effect of commitment.Validator definitions and measurement criteria are shared, but condition-specific acceptance rules, feedback, repair trajectories, and realized compute are not matched.Source: §C.2 · p. 12

D.1 Significance Tests and Confidence Intervals

  • Uncertainty estimates use 10,000 paired task-bootstrap samples, resampling tasks while keeping each task’s measurement units clustered.For SkillsBench Pass, exact two-sided McNemar tests use 87 paired outcomes per model, with Holm correction across seven tests.Source: §D.1 · p. 12
  • SpecHarness improved SkillsBench Pass by 12, 11, 15, 6, 10, 10, and 9 tasks across the seven models; all comparisons remained significant after Holm correction.The tests establish paired end-to-end differences between evaluated conditions, not the isolated effect of commitment.Source: §D.1 · pp. 12–13, Table 17 · p. 13

D.2 Category-Level Results and Residual Failures

This section defines category-level measurement and residual-failure accounting, then reports runtime costs and diagnostic traceability. The cost comparison reflects the complete online validation, feedback, and repair architecture rather than commitment alone.

D.2 Category-Level Results and Residual Failures

  • Requirement categories partition the 509 SkillsBench directions and 5,817 GuideBench rule-local instances, without changing runtime decisions or obligation state.Residual-failure categories describe the immediate unresolved failure, not a unique causal explanation.Source: §D.2 · p. 13, Table 18 · p. 13, Table 19 · p. 13
  • Token and wall-clock overhead are cost ratios to Raw, while call overhead is the paired mean difference in task-agent calls per task.The final row macro-averages model-level quantities without weighting models by token volume or execution time.Source: §D.2 · p. 14

D.3 Absolute Runtime Cost and Diagnostic Traceability

  • SpecHarness averages 1.53× task-agent tokens, 1.24× condition-execution time, and 0.36 additional task-agent calls per task relative to Raw.The wall-clock ratio covers condition execution only; task-specific preprocessing is reported separately.Source: §D.3 · p. 14
  • A diagnostic is actionable when it identifies the source requirement, failed or unresolved condition, and repair target.The comparison uses each condition’s native diagnostics, so Raw lacks SpecHarness’s ledger and source-linked runtime evidence.Source: §D.3 · p. 14

E Benchmark Instantiations and Execution Traces

SpecHarness applies the same obligation–evidence–commit discipline to artifact requirements and guideline decisions, with benchmark-specific validators. Representative replays show that a selected obligation’s satisfaction is not enough for completion: all active mandatory obligations must be satisfied.

E.1 SkillsBench and GuideBench Instantiations

  • SkillsBench compiles visible task and skill requirements into source-linked obligations over workspace state, using mediation for closure-audited action surfaces and validation for safely isolated channels.Evidence binds observed artifact state to the corresponding dependency versions.Source: §E.1 · p. 14
  • GuideBench compiles visible guidelines into rule-local obligations whose applicability depends on decision context, with validators checking applicability and agreement with the guideline.Because no closure-audited physical action surface is exposed, hard obligations use validate-and-commit.Source: §E.1 · p. 14

E.2 End-to-End Execution Traces

  • The SkillsBench replay follows an artifact obligation through authorization, observation, commitment, invalidation, and revalidation; a changed artifact requires fresh evidence at the current dependency version.The example is a protocol replay, not a verbatim naturally occurring trajectory, and omits other task obligations.Source: §E.2 · p. 14
  • The GuideBench replay’s bound validator checks both rule applicability and agreement of the proposed option with the same active obligation before committing evidence.The examples omit other active obligations; finalization requires fresh satisfaction of every active mandatory obligation.Source: §E.2 · pp. 14–15, Table 25 · p. 16

Related papers

Loading 2609.29921v1…