Cross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection Study

1 Introduction

• In a multi-subscription workstation, reviewing every completed turn spends a distinct allowance even for status updates, direct questions or refusals, while never reviewing forfeits possible error detection.The paper motivates both a reliable independent review and evidence about where review can be spared.Source: §1 · p. 1
• Fault injection exposed false-success paths, including cancellation during reaping and nonzero exit with a well-formed verdict; the latter was found after collection began, leaving exposure unscoped.The introduction describes three implementation-specific paths and explicitly does not claim selective gating already works.Source: §1 · pp. 1–2
• The setting is OpenTerminalAgent, a terminal-agent runtime that runs coding CLIs behind one interface while each provider draws on its own allowance.The paper names Codex, Claude and Kimi among the supported CLIs.Source: §1 · p. 2
• A cross-provider review contract makes separate pools, bounded execution, complete input, non-mutation, typed failures and durable per-attempt evidence part of the interface.A process exit alone cannot establish complete delivery, usable output or provider-and-pool independence; review is advisory and cannot rewrite the repository.Source: §1 · pp. 1–2
• The controlled pilot completed 20 paired turns across two provider directions with turn-level blinded source checking, measuring when uniform review yielded no material finding.It demonstrates that review can surface material issues in this workload, not improved final software quality.Source: §1 · pp. 1–2
• The study is a systems-and-evaluation contribution, not a new routing algorithm, calibrated risk model or claim of broad generality from 20 turns.Its third contribution is a deterministic fault-injection case study and preregistered shadow allocation protocol.Source: §1 · p. 2

2 Related Work and Boundary of the Contribution

• Prior verification and compute-allocation work includes selective verification and solver-state or solve-versus-verify approaches, while this paper studies review from a separate pool after answer delivery.The paper says its V3 gate is zero-shot and has no demonstrated advantage.Source: §2 · p. 2
• Coding-agent studies report directional asymmetry, regressions after review changes, partly distinct planted errors, and plausible but incorrect security findings from missing payloads (Xiang et al. (2026); Song (2026); Shaw (2026)).The cited work motivates a narrower runtime question: whether an advisory review is a valid observation.Source: §2 · p. 2
• The three false-success paths here are implementation-specific; the paper does not generalize them beyond its runtime.This boundary distinguishes the study from broader harness-observability and agent-lifecycle work.Source: §2 · p. 2
• The Claude-family pilot judge saw Claude-family text in both directions, so identity blinding and source checks establish neither model-family independence nor human ground truth.The paper also notes known adjudicator biases including self-recognition, self-preference and position bias (Panickssery et al., 2024; Shi et al., 2024; Chen et al., 2024).Source: §2 · p. 2

A. Multi-agent interface

• The Reviewer critiques a completed turn; in shadow evaluation, the metadata-only Allocator advises but does not suppress the review call.The gate maps to OTA’s Allocator role and the reviewer to its Reviewer role.Source: Figure 1 · p. 3, §3 · p. 4

3 A Review Contract for Heterogeneous Coding Agents

• A review counts as complete only when all applicable contract checks pass; a zero process exit alone is insufficient.The runtime also requires a usable semantic outcome, not just a successful process exit.Source: §3 · p. 3
• The contract makes pool disjointness, bounded execution, complete input delivery, non-mutation, typed failure, and per-attempt evidence observable.These properties define the measurable interface for cross-provider advisory review.Source: §3 · p. 3
• Missing, truncated, or same-pool reviews become explicit failures or exclusions rather than silent no-finding observations.Such cases cannot supply valid labels for the later gate study.Source: §3 · p. 4

4 Controlled Pilot: Method

• 20 paired complete answers and reviews from 23 attempts entered the pilot denominator, covering 20 controlled read-only codebase tasks.The two provider directions each contained ten paired turns, and task order was not randomized between directions.Source: §4 · p. 4
• One fresh-context Claude Opus 5 subagent labeled each turn MATERIAL, NON_MATERIAL, or UNCERTAIN after checking factual claims against source.Packets were shuffled and opaque; the adjudicator was not an independent human audit, and no inter-adjudicator agreement estimate was available.Source: §4 · p. 4
• The study recorded latencies, process outcomes, repository mutation, and leaked-process checks, but these checks do not estimate user-perceived latency or effects on final code.The reviewer prompt and decision rule remained fixed throughout.Source: §4 · p. 5

5 Controlled Pilot: Findings

• 8/20 turns had a material reviewer finding (95% exact interval 19.1–63.9%); 12 had none, and none was UNCERTAIN.The interval is wide on this selected controlled workload; the result does not establish a population rate or that calls could safely have been skipped.Source: §5 · p. 5
• The 2/10 versus 6/10 direction contrast runs opposite to simple Claude-family self-preference predictions, but does not rule out judge bias or other confounding.Directions differed in task and answer mix, order, and runtime behavior, so the count difference is not a provider-quality ranking.Source: §5 · p. 5
• 23 reviewer launches yielded 20 completed paired reviews, with no recorded mutation, leaked process, timeout, or cancellation across attempts.One primary allowance exhaustion left the other provider’s reviewer able to run; this is one operational example, not a reliability estimate.Source: §5 · p. 5
• Partial input delivery could previously be mistaken for a successful review if the reviewer replied; the repaired contract requires complete delivery.The audit treats input completeness as a prerequisite for counting a review.Source: §5 · p. 5

6 Fault Injection and CLI-Level Write Probe

• The synthetic lifecycle scan exercised both parsers with 300,000-character input and distinct input, cancellation, and deadline boundaries; no V3 data or provider call entered it.Each synthetic boundary was one deterministic case; the exact-deadline case depended on scheduling.Source: §6 · pp. 5–6, Table 3 · p. 6
• Across both backends, four historical truncation boundaries and cancellation during reaping produced false success; a nonzero exit with a valid verdict was also counted complete before repair.The post-repair candidate treats any nonzero exit as an operational failure, regardless of output.Source: §6 · pp. 5–6, Table 3 · p. 6
• Claude had no write tools, while Codex attempted file-change and shell-write operations in 5/5 trials; all failed and disposable repositories stayed unchanged.The event stream did not identify the lower-level rejection mechanism, so the probes do not establish a universal non-mutation guarantee.Source: Table 3 · p. 6, §6 · p. 6

7 Prospective Shadow Study: Review Allocation

• V3 uses pinned Qwen3 models on task metadata, excluding answers, findings, source text, paths, and credentials; every eligible turn still receives review in shadow mode.Nine frozen thresholds define offline would-SKIP sets, and scoring failures cannot recommend a skip.Source: §7 · p. 6
• The primary descriptive endpoint is MATERIAL among adjudicated 0.6B would-SKIP turns at t=0.90, with fixed outputs including model/stratum curves and activity/rule baselines.The frozen random-skip benchmark expects K*M/N misses for K skips among N eligible turns with M MATERIAL labels.Source: §7 · p. 6
• Collection stops at 80 adjudicated reference would-SKIP turns or 400 eligible adjudicated measurement-valid turns; rare skips may leave the safety endpoint empty or imprecise.No V3 gate result is reported.Source: §7 · p. 7
• After the nonzero-exit defect was found, 25 formal and two pending records remained an audit cohort, while measurement-valid collection restarted at zero with per-attempt exit evidence.As of 2026-10-07, eight post-repair turns had been formalized, without any adjudication label inspected.Source: §7 · p. 7

8 Discussion and Limitations

• The pilot found material reviewer findings, but neither it nor the synthetic scans establishes general security or a causal benefit of review.Recognition of the twelve non-material turns remains unknown, motivating shadow scoring and matched-count baselines.Source: §8 · p. 7
• The 20-pair pilot used one codebase and two provider directions; wide intervals, nonrandom order, one judgment per item, and partial adjudicator independence limit comparisons.Natural-use tasks are not currently collected, and post-repair turns draw from one fixed product revision.Source: §8 · p. 7
• V3 gates share one model family, the adjudicator backend revision is undisclosed, and independent human audit is unavailable; the gate has not yet been validated.Proprietary conversations and reviewer instructions cannot be released unchanged.Source: §8 · p. 7
• The study supports observable failure semantics and a design capable of registering a negative result; independent human audits and calibrated decision models remain future work.Source: §8 · p. 7

9 Conclusion

• Cross-provider review is measurable when the runtime checks input delivery, separate pools, bounded execution, restricted capabilities, explicit failures, and durable per-attempt evidence.Source: §9 · p. 7
• The pilot and probes provide bounded evidence, not general security or a causal benefit of review; no V3 effectiveness claim is made.Real read-only CLI probes observed no repository writes in five trials per backend.Source: §9 · p. 7

Competing Interests

• The runtime developer may benefit from adoption, and the source code, prompts, and orchestration logic are proprietary and unreleased.The paper describes interfaces, the review contract, and the evaluation protocol.Source: Competing Interests · p. 8

A Appendix: Post-repair requalification and operational record

• As of 2026-10-07, the record reports checks and counts only, with no post-repair adjudication label or allocation result.Source: A Appendix: Post-repair… · p. 10

A. Defects found during qualification and operation

• An exact-runtime regression found that a nonzero reviewer exit with a well-formed verdict could pass; because exit status was not durable per attempt, collection restarted with explicit exit evidence.Source: A. Defects found during qualification… · p. 10
• A model-provenance audit found configured choices did not prove which models the CLIs used; CLI-reported identities and evidence hashes were added before post-repair collection.Source: A. Defects found during qualification… · p. 10
• Pre-collection repairs addressed unattended error dialogs, blocked subprocess output readers, and timezone parsing that could shift the collection window.Source: A. Defects found during qualification… · p. 10
• Further fixes covered selector-response parsing, an unhandled disposition, synthetic bootstrap data, and supervisor row accounting; existing records were preserved.The early bootstrap-file hashes were unavailable for four files.Source: A. Defects found during qualification… · p. 10

• Attempt receipts record exit status, termination reason, STARTED and terminal times, deadline and cancellation state, output validity, and input-delivery completeness and eligibility.A missing or mismatched required field makes the attempt ineligible.Source: B. Durable evidence for each attempt · p. 10
• Receipts capture each CLI’s reported main model, all reported model identifiers, hashes of supporting metadata fragments, and hashes of key runtime source files.Source: B. Durable evidence for each attempt · p. 10
• These checks establish recorded process and model provenance, not the correctness of the answer or critique.Source: B. Durable evidence for each attempt · p. 10

C. Administrative counts and stopping

• Eight post-repair turns were formalized; all had zero reviewer exit status, matching pinned and CLI-reported main models, strict-cohort receipts, and matching source-file hashes.One attempt was excluded because the Claude reviewer allowance was blocked; there was no further recorded attrition in the two post-repair operating windows.Source: C. Administrative counts and stopping · p. 10
• The controlled stratum ends after three consecutive NO_DISTINCT_TASK returns without a previously frozen task-source extension, leaving the cohort at its actual formal count.The checkpoint is descriptive, reports strata separately, uses matched coverage at 10%, 20%, and 25% of that count, and computes no Phase-2 threshold below 50; integrity checks and sealed label-blind scoring precede any label join.Source: C. Administrative counts and stopping · pp. 10–11

D. Known limits

• The selection schema omits duplicate-idea contents and links, SELECTION_BLOCKED has no reason field, and Kimi records no resolved backend model identifier.Historical artifacts were present in candidate workspaces; a path-level inventory found no adjudication label or outcome files but cannot show that other artifacts were never accessed.Source: D. Known limits · p. 11
• The eight formal turns were linked through live workspace/session association, turn identifiers, and sealed-source evidence, but the later pending captures remain unformalized with candidate-to-turn mapping unverified.Chronological order was not accepted as proof of identity.Source: D. Known limits · p. 11