PaperlayerView exact revision on arXiv

Source-linked AI summary

Engineering Reliable Commit Gates for Agentic AI: Cost-Aware Verification Portfolios under Common-Mode Data Failures

Zihao Zheng, Baichuan Li, Junyi Yao, Jiayu Long

arXiv:2609.10969v1cs.SE

TL;DR

Agentic commit gates need to handle verifier agreement caused by shared upstream faults and evidence that becomes stale before execution. VP-CONTROL benchmarks evidence and model diversity, selects verification portfolios from observable metadata, and tests live transactional safeguards. Source diversity has the larger benchmark effect, while portfolio calibration and commit-time enforcement remain conditional on fault coverage, verifier competence, and the tested tool setting.

  • Problem

    Agent commit gates require verification that remains reliable when verifiers share upstream evidence and when state changes between checking and execution.

  • Method

    VP-CONTROL combines a deterministic benchmark, a fixed-call model/source factorial comparison, an observable-metadata portfolio controller, and live HTTP/SQLite commit-gate experiments.

  • Results

    Source diversity reduces unsafe approval more than model diversity, while the calibrated portfolio reaches 1.9% unsafe execution and 38.2% automated safe coverage at a nominal 5% target on the locked test.

  • Takeaways & Limitations

    Reliable commit gates require explicit evidence lineage, cost-aware verification selection, and enforcement of checkable predicates at commit time.

  • Takeaways & Limitations

    Transfer is conditional: unseen faults, weak verifiers, approximate calibration, and broader distributed or long-horizon settings remain outside the demonstrated scope.

Abstract

from arXiv · show

Agentic systems commit state-changing actions, but additional verifiers can inherit the same upstream fault. We present VP-CONTROL, a runtime-assurance design and deterministic benchmark for cost-aware commit gates. Its 48 task templates yield 2,880 scenarios across six fault regimes. A fixed-call 2 x 2 experiment separates verifier-model diversity from evidence-source diversity. On frozen proposals from two local actor families, a cross-model vote over shared evidence approves 62.9% of unsafe proposals, versus 22.9% with an independent source. The source effect is 40.9 percentage points, compared with 11.3 for model diversity. A portfolio controller selects verification plans using only deployment-observable metadata. Approximate cluster-adjusted calibration at a nominal 5% per-task target yields 1.9% unsafe execution and 38.2% automated safe coverage on the locked test. Matched-budget portfolios also improve on fixed verification policies. Transfer remains conditional: unseen fault families yield 16-26% risk, and a FinQA check fails to reproduce the source effect with the tested small verifiers. A preregistered live HTTP/SQLite study tests concurrent writes and lost responses. After-check races defeat verifier-only gates; transactional partial guards prevent only covered failures, while a full atomic guard records no unsafe effects across 216 episodes. Idempotent request identifiers prevent duplicate effects after lost responses. The results motivate explicit evidence lineage, cost-aware selection, and commit-time enforcement, while exposing the limits of approximate calibration and local-tool generalization.

I. INTRODUCTION

Agent commit gates must account for shared evidence failures and changes between checking and execution. VP-CONTROL evaluates evidence lineage, model diversity, cost-aware selection, and enforcement at this boundary.

  • Motivation: Shared upstream faults can make multiple verifier models agree on the same unsafe state, while changes between checking and writing can invalidate correct evidence.The design therefore distinguishes evidence diversity from model diversity and considers atomic enforcement and deferral.
  • Research questions: The study asks how evidence source and model diversity affect verification, whether portfolios improve risk–cost trade-offs, where transfer fails, and which live-tool safeguards survive.The questions cover benchmark behavior, runtime control, boundaries, and concurrent tool integration.
  • Contributions: The contribution jointly evaluates commit-gate alternatives and complements, including the conditions under which verification portfolios stop helping.The main experiment uses isolated proposals, while replan and live studies cover narrower operational settings.
  • Contributions: The benchmark isolates evidence lineage from model identity, the portfolio uses observable metadata for risk–cost selection, and live tests establish limits from races and partial guards.These results motivate explicit lineage recording, commit-time enforcement, and calibration or deferral for incomplete evidence.

II. RELATED WORK

Prior work motivates verifier aggregation, source-aware selection, risk-controlled intervention, and tool contracts. This paper places those ideas at the commit boundary and varies model identity, evidence lineage, and enforcement mechanisms explicitly.

  • Verifier dependence: Correlated verifier errors and duplicated evidence roots limit the reliability gained from aggregating verifier outputs.The paper instantiates this distinction at an agent commit boundary by varying model and source independently.
  • Source-aware and selective verification: Source-aware routing, selective verification, cost-aware retrieval, and workflow-node verification motivate choosing mechanisms by evidence source and cost.The paper extends these ideas by jointly pricing same-lineage checks and independent-source acquisition.
  • Risk-controlled action gates: Risk-control methods and agent action gates provide precedents for calibrating interventions, probing before commitment, and falling back when risk requirements are unmet.VP-CONTROL differs architecturally by selecting among mechanisms with distinct evidence lineages and explicit source-level common modes.
  • System model: The system model defines proposals, preconditions, effects, and commit, block, or defer outcomes, with only commit invoking the target system.It exposes actor, same-lineage, and independent evidence views plus exact guards, costs, and human deferral.
  • Illustrative case: A schema-migration example shows that stale same-lineage views can approve a revoked operation while a fresh independent source exposes the problem.Guard expressiveness also matters because the illustrated guard checks compatibility and duplicate execution but not approval.
  • Observable interface: The observable controller view exposes deployment metadata while excluding authoritative state, injected faults, true staleness, and unsafe labels.This separates policy inputs from evaluation-only information.

B. Verification plans

VP-CONTROL represents verification as auditable decision-tree plans selected from heterogeneous mechanisms under observable context and either calibrated risk or matched-cost objectives.

  • Plan library: 37 decision trees span same-lineage re-checks, independent reads, transactional guards, and deferral, with leaves that commit, block, or defer.The library includes unanimity votes, resampling, escalation, and guard-then-source plans; duplicate mechanisms are charged once.
  • Context selection: Observable contexts bucket evidence ages, tool status, health flags, and guard availability across a six-level back-off hierarchy.A context level requires at least 24 training rows from three templates, preserving an auditable lookup-table policy.
  • Calibration: Risk-target selection tests policies in increasing λ order with a design-effect-adjusted Wilson upper bound and stops at the first failing policy.If none passes, the controller defers everywhere; the bound is explicitly treated as an approximation rather than a finite-sample guarantee.
  • Budget matching: Matched-cost variants minimize br−0.04q+µc while choosing µ so mean training cost stays within the baseline budget.These variants answer a cost-matching question rather than the calibrated risk-target question and carry no risk guarantee.

IV. IMPLEMENTATION AND TRACEABILITY

The implementation makes policy inputs, execution boundaries, and decision traces explicit. Audits and staged evaluation improve inspectability while leaving external validity unresolved.

  • Implementation boundary: The reference implementation separates generation, decision logic, execution, and analysis, with templates declaring predicates, effects, guard clauses, blast radius, and reversibility.The controller receives a sanitized observable view and maps it to a named executable plan.
  • Traceability: Every decision records the selected plan, mechanism bindings, verdicts, confidence, costs, disposition, and scored outcome for auditability.Append-only resumable runners and cache keys bind results to model, source, scenario, proposal, seed, and prompt variant.
  • Evaluation staging: Staged evaluation separates feasibility work, prompt repair, frozen locked-test evaluation, and explicitly labeled stress, external, and live datasets.Preregistration deviations and later-stage replay or stress analyses are logged by stage.
  • Audits: Static leakage, dynamic reproducibility, and recomputation audits make policy choices and reported strategy rates mechanically inspectable.The recomputation audit sampled 900 plan entries and found zero mismatches across 33 reported strategy rates.

V. EVALUATION METHOD

The evaluation freezes a diverse benchmark, defines task-level risk and coverage, and compares verification policies using locked proposals, costs, and fault regimes.

  • Benchmark design: 48 templates instantiate six fault regimes, with ten hash-seeded worlds per template and hazardous or benign latent changes in non-clean regimes.The template is the unit of analysis, and B independently lags with probability 0.10.
  • Factorial comparison: The 2×2 experiment holds two calls fixed while independently varying verifier model identity and evidence source.It compares same or cross-model calls on A′ with same or independent-source calls using B.
  • Metrics: Risk counts unsafe executions per evaluated proposal, while safe automated coverage counts satisfied goals excluding deferrals; modeled cost is neither latency nor tokens.The nominal risk target therefore requires coverage and cost to be reported alongside it.
  • Models and locking: Actors and verifiers use frozen quantized local checkpoints, with one resident model at a time and constrained proposal and verdict schemas.The main run locks fault rates, costs, plans, splits, and statistical procedures after a single prompt correction.
  • Replay and inference: The main run freezes 5,760 proposals and caches 28,362 verifier calls, replaying every controller over identical proposals and verdicts.Confidence intervals use cluster bootstrap samples, while paired differences use template-cluster sign-flip permutations with Holm correction for three primary comparisons.

C. Boundary checks

Boundary checks extend the locked benchmark to external transfer and live transactional tools, injecting races, stale evidence, degraded sources, and ambiguous responses.

  • Transfer checks: External checks include held-out domains or fault families, prompt shifts, weaker verifiers, role swaps, seed and cost changes, bounded replanning, and 960 FinQA scenarios.These analyses are separated from the locked main policy and include explicitly labeled stress and external datasets.
  • Live integration: The live arm tests 12 business contracts across schema migration, snapshot promotion, and failed-job restart, each under four predicate or guard-coverage variants.Episodes use a loopback HTTP service, per-episode SQLite databases, a separate writer, and local checkpoints.
  • Fault injections: Six injected conditions cover clean behavior, benign changes, common stale caches, after-check races, ambiguous responses, and degraded evidence.The after-check race commits between the gate’s last check and the action; ambiguous responses commit before closing the connection and trigger a retry.
  • Policy comparisons: Eight live policies compare plain commit, shared-cache and independent-source votes, partial guards, portfolio selection, and a full atomic guard.Unsafe execution is evaluated at the transaction’s linearization point, and three preregistered comparisons are Holm-corrected.

VI. RESULTS

The benchmark shows that independent evidence matters more than verifier-model diversity, especially under shared stale data, but loses value when all sources degrade.

  • RQ1: Source diversity has the larger effect in the benchmark: 22.9% false-approve rate results from changing both verifier family and source, versus 62.9% for changing only the verifier family on unsafe locked-test proposals.Holding the model fixed while changing to an independent source yields 33.3%, establishing the main source-related reduction.
  • RQ1: Source diversity has the larger effect in the benchmark: The paired source effect is -0.409, compared with -0.113 for model diversity, and the difference-in-differences does not detect an interaction.The reported intervals are [-0.479, -0.331] for source and [-0.215, -0.038] for model effects; p = 0.9056 for the interaction.
  • RQ1: Source diversity has the larger effect in the benchmark: Under common stale data, cross-model approval falls from 81.7% on the shared source to 7.3% when one call uses B.Under all degraded conditions, the cross-model independent-source cell remains at 41.4%, limiting the benefit of independent evidence.
  • RQ1: Source diversity has the larger effect in the benchmark: The rule-based arm reproduces the qualitative source effect, with cross-model approval of 65.9% on shared evidence and 22.8% on independent evidence.Because model-side errors are independent by construction in this arm, the result is attributed to evidence plumbing rather than one LLM pair.

B. RQ2: A portfolio improves the risk–cost trade-off

The portfolio reduces unsafe commits at matched verification budgets while improving or preserving coverage, but calibration and transfer remain conditional.

  • 1.9% unsafe execution with 38.2% safe automated coverage is achieved by the calibrated 5% portfolio on the locked test.The result uses an approximate cluster-adjusted bound; it is empirical evidence rather than a finite-sample safety certificate.
  • At the cross-model vote budget, the portfolio lowers unsafe commits from 27.8% to 12.7% and raises coverage from 40.6% to 51.7%.
  • At the independent-source budget, the portfolio reduces unsafe commits from 16.5% to 5.7% with nearly unchanged coverage.
  • Fault-family holdouts reach 26% and 16% realised risk at the nominal 5% target when ambiguous commit or partial update is unseen.These folds intentionally violate exchangeability and demonstrate failure to maintain the nominal target.
  • Returning better evidence to the actor is insufficient: after one replan, 69.8% of initially unsafe proposals remain unsafe even with independent evidence.The result supports commit-time enforcement rather than relying on actor response to evidence.

D. RQ4: Live execution exposes the enforcement boundary

Live HTTP/SQLite experiments show that verification alone cannot close interleaving races, whereas atomic guards and idempotent requests address distinct execution hazards.

  • The live study supports an enforcement boundary rather than a second demonstration of portfolio risk superiority.The guarded portfolio completes more tasks than the guarded independent vote but has slightly higher observed risk, while the full atomic guard performs best.
  • The full-predicate atomic guard records no unsafe effect across 216 episodes and completes 81.0% of tasks safely.The zero count has a cluster-adjusted 95% upper bound of 1.4%; safe completion ties the atomic partial guard for the highest rate.
  • All verifier-only policies commit the same twelve unsafe effects under the forced after-check race because evidence is read before the writer changes state.The partial guard reduces this to nine when the same clauses are rechecked inside the write transaction, but uncovered predicates remain unsafe.
  • The frozen portfolio has 4.2% unsafe rate and 68.5% safe completion, compared with 2.8% and 41.2% for the independent vote with the same guard.All nine portfolio unsafe cases occur in the race condition, and the observed unsafe difference is not significant after correction.
  • Idempotent request identifiers replay stored responses for all 96 retried requests without duplicate effects after lost responses.Without idempotency, snapshot promotions duplicate audit rows and pointer switches, while SQLite constraints reject duplicates in two other families.
  • On a 16 GB machine, median gate latency is 3.44 s for the portfolio versus 6.09 s for the shared-cache vote and 6.38 s for the independent-read vote.Excluding model-load time, the medians are 1.34 s, 2.44 s, and 2.43 s, respectively.

VII. ENGINEERING IMPLICATIONS AND VALIDITY

Reliable deployment requires explicit evidence lineage, commit-time enforcement, and calibration matched to the deployment population. The study’s validity is bounded by approximate clustered calibration, controlled constructs, local tooling, and limited external evaluation.

  • Engineering implications: Full atomic enforcement is preferred when available; portfolio selection addresses cases where guards are partial or unavailable and evidence sources differ.The live results report no observed safety or completion advantage from additional model checks when the entire predicate is enforced within the write transaction.
  • Engineering implications: Evidence lineage, freshness, completeness, health, and observation time should accompany tool results so shared failure domains are not mistaken for independent redundancy.Checks sharing lineage should count as one failure domain even across different verifier models.
  • Validity: The observed 1.9% test rate supports a nominal 5% setting only in this evaluation population, not as a formal guarantee across unseen fault families.Operational use requires recurring calibration for each new tool, lineage, and incident family.
  • Engineering implications: A check separated from the write transaction cannot close an after-check race; only an atomic guard can enforce the covered clauses at commit time.The live integration should bind decisions to the exact proposal, resource, evidence version, and policy version.
  • Validity: Absolute risk and cost values do not transfer because tasks, fault probabilities, telemetry noise, unit costs, and live contracts are controlled design choices.The supported claim is the ordering of mechanisms under the stated fault model, with human deferral modeled as safe at fixed cost.
  • Validity: The evaluation primarily covers quantized 4–10B models, local services, twelve live contracts, and one writer, while larger, distributed, adversarial, and drifting settings remain open.An external numeric benchmark also fails to reproduce the main effect.
  • Validity: Reproducibility is supported by released prompts, model digests, parameters, frozen records, audit archives, live artifacts, and regeneration scripts.The frozen-record analysis does not require model access.

VIII. CONCLUSION

The paper concludes that reliable commit gates need both evidence selection and enforcement. Source-aware portfolios improve the risk–cost trade-off under the tested conditions, but unseen faults, weak verifiers, and concurrent changes limit transfer.

  • Conclusion: Reliable commit gates require both evidence selection and enforcement, with source diversity reducing false approval more than model diversity in the controlled benchmark.The observable portfolio also improves the risk–cost trade-off over fixed checks on frozen proposals.
  • Conclusion: Atomic guards protect only the clauses they express, while a full guard is preferred when the entire safety predicate is checkable.With partial guards, the portfolio improves completion relative to a guarded independent vote but does not establish lower risk.
Loading 2609.10969v1…