Source-linked AI summary
TwinCheck: Evidence-Grounded Negative-Twin Verification for Stateful Tool Agents
Jiaxuan Dai, Tianyi Huang
TL;DR
Stateful tool agents can fail because of a single action, but suspicion alone is insufficient grounds for repair when a replacement can create failure. TwinCheck requires trace-grounded evidence for a constrained alternative and counterbalanced verifier agreement; on 159 complete exact-replay pairs, it increased task success with no observed regressions.
Problem
A single locally plausible tool call can derail a stateful agent trajectory, while unnecessary repairs can turn success into failure.
Source: §1 · p. 1Method
TwinCheck replaces an agent proposal only when trace evidence supports a specific failure hypothesis, a structurally valid negative twin exists, and a pairwise verifier prefers it in both candidate orders.
Source: §1 · p. 1, §4.3 · p. 4Results
Takeaways & Limitations
Across the evaluated BFCL trajectories, execution-boundary repair should establish a case for the replacement, not merely a case against the proposal.
Source: §8 · p. 8Evaluation covers stored trajectories in four BFCL V4 multi-turn categories under one actor model; robustness to model/API resampling and generalization to other models and environments remain untested.
Source: §7 · p. 8
Abstract
from arXiv · showhide
A single locally plausible tool call can derail an otherwise successful agent trajectory. Suspicion alone does not justify intervention, because the replacement itself can introduce the very failure verification is meant to prevent. We introduce TwinCheck, an inference-time verification policy that considers replacement only when the trace satisfies an evidence condition tied to a trace-local failure hypothesis. It constructs a trace-grounded counterfactual alternative, a negative twin, and replaces the agent's proposal only if the twin passes structural checks and the pairwise verifier prefers it in both candidate orders. For paired evaluation, exact replay holds the agent's parsed responses and actions fixed until the first accepted replacement, separating intervention effects from resampling. In the primary analysis of 159 multi-turn BFCL V4 tasks with complete exact-replay pairs, the complete policy raises task success for GPT-5.6 Sol from 45.3% to 58.5% (95% task-bootstrap CI [8.2, 18.8]), with no observed success-to-failure regressions. Together, these findings recast execution-boundary repair as a constrained comparison, making the counterfactual action itself the object of verification.
1 Introduction
Stateful tool agents can fail through a single locally plausible action, while an unnecessary repair can turn success into failure. TwinCheck therefore requires trace-grounded evidence for a constrained alternative and uses exact replay to evaluate interventions separately from resampling.
1 Introduction
- Because an unnecessary repair can create failure while declining one preserves the proposal, intervention requires evidence that the action is locally responsible and its alternative addresses the defect.Source: §1 · p. 1
- TwinCheck constructs a trace-grounded negative twin for a specific failure hypothesis and replaces the actor’s proposal only after evidence checks and pairwise comparison.Source: §1 · p. 1
- Operational evidence matches do not prove semantic error or recover user intent, so the evaluation concerns this intervention policy rather than arbitrary tool-agent repair.Source: §1 · p. 2
- Exact replay fixes the evaluator-visible actor sequence until the first accepted replacement, enabling paired rescue-and-harm accounting apart from resampling.Source: §1 · p. 2
2 Related work
Prior work studies verification, feedback-based repair, safeguards, and tool-agent evaluation. TwinCheck focuses on trace-local evidence for a specific pre-execution substitution, with a pairwise judge required to preserve its preference across candidate orders.
Verification and model judging
- Model judges can show candidate-order effects, so TwinCheck consults one only after a trace supports a specific error hypothesis and a valid alternative exists.Source: §2 · p. 2
Feedback and agent repair
Selective safeguards for tool agents
Tool-agent evaluation
- BFCL evaluates function calling through stateful multi-turn tasks, where its official evaluator checks state and responses after each turn.Source: §2 · p. 2
3 Problem formulation
The formulation treats tool-action repair as an intervention before execution: an actor proposal may be executed or replaced, and final task success is evaluated. TwinCheck’s objective is selective intervention that yields more rescues than harms.
3 Problem formulation
- For task j at step t, the actor proposes an action from visible context, and the wrapper either executes it or substitutes a counterfactual action before final success is evaluated.Source: §3 · p. 2
- A continuous trace-level anomaly score ranks possible actor failures after a proposal is available and before it executes.Source: §3 · p. 3
- Intervention requires an evidence condition, a structurally valid twin, and verifier consensus; a frozen threshold determines whether the switch score triggers replacement.Source: §3 · p. 3
- The objective is to produce more rescues than harms while keeping the intervention set selective, rather than merely maximizing detector recall.Source: §3 · p. 3
4 Method
TWINCHECK uses trace-local evidence to constrain a candidate repair, then switches only when structural checks and counterbalanced verification support the twin. Calibration freezes the gate, and exact replay holds actor behavior fixed until an accepted replacement.
4.1 Evidence ledger and continuous detector
- The detector applies three trace-local eligibility conditions, but its score ranks possible failures and cannot by itself authorize repair.The conditions cover lexical inverse actions, repeats after errors, and unique stale arguments; only severities of at least 0.90 qualify.Source: §4.1 · p. 3
4.2 Constructing and validating the twin
- A certificate determines a constrained edit or guides a generated twin; every nonidentical twin must pass schema and structural checks before judgment.Authorized inverse-action deletion and repeat-after-error repair may change action kind or call count; otherwise, generated twins are constrained in distance and evidence-backed values.Source: §4.2 · pp. 3–4
4.3 Counterbalanced contrastive verification
- The verifier compares the actor and twin in both orders, and switching requires both judgments to prefer a nonfatal twin over a fatal actor.Each judgment must also meet confidence ≥0.98 and quality margin ≥0.50; omitted or duplicated orientations yield no switch.Source: §4.3 · pp. 4–5
4.4 Calibration, intervention, and exact replay
- Calibration checks a frozen gate on a shadow run; test-time exact replay preserves evaluator-visible actor responses and actions until the first accepted switch.In the 32-task calibration, 17 trajectories succeeded and their maximum G_j was 0, so the prespecified floor remained γ = 0.50; at most one switch is accepted per trajectory.Source: §4.4 · p. 5
5 Experimental setup
The experiments use four stateful multi-turn BFCL V4 categories and the official evaluator, with category-stratified calibration and primary samples. They compare GPT-5.6 Sol actor and repair policies using paired task outcomes and stratified uncertainty estimates.
Benchmark and splits
- Four BFCL V4 stateful categories each contribute 200 release tasks, and the official evaluator checks state and responses after each turn.The categories are multi_turn_base, multi_turn_miss_func, multi_turn_miss_param, and multi_turn_long_context.Source: §5 · p. 5
- After excluding 401 development-touched IDs, category-stratified sampling selected 32 calibration tasks and planned 160 primary tasks.A later 160-task conditional extension followed a passed exact-replay launch check, with scoring, threshold, and policy unchanged.Source: §5 · p. 5
Models and execution
- GPT-5.6 Sol served as actor, pairwise verifier, and model-based twin generator, with high, medium, and low reasoning effort, respectively.At most two decision points per trajectory could reach the verifier, and at most one action could be replaced.Source: §5 · p. 5
Baselines
- The controls were the unwrapped actor, selective self-revision, and a trigger-matched selective absolute verifier.Self-revision accepted changed schema-valid actions passing structural checks; the absolute verifier triggered below health 0.60 and required a 0.02 health increase.Source: §5 · p. 6
Metrics and uncertainty
- Task success is the unweighted mean of category-level rates, with paired 95% task-bootstrap CIs and exact two-sided McNemar tests.The 20,000 category-stratified bootstrap resamples task pairs within category; cluster resampling is a dependence sensitivity.Source: §5 · p. 6
- One prespecified primary task lacked an actor trajectory after a transient API error, leaving 159 complete exact-replay pairs for the primary analysis.A regenerated trajectory for that task enabled a 160-task completed-primary sensitivity; task-pair tests treat pairs as independent.Source: §5 · p. 6
6 Results
On 159 complete primary pairs, TwinCheck raises task success from 45.3% to 58.5%, with 21 rescues and no observed harms. Results also characterize failure ranking and the selectivity of the frozen gate, alongside trigger-matched controls.
6 Results
- 13.2 percentage points: TwinCheck raises GPT-5.6 Sol task success from 45.3% to 58.5% on 159 primary pairs (95% task-bootstrap CI [8.2, 18.8]), with 21 rescues and no observed harms.The exact-replay comparison uses the 159 available complete primary pairs.Source: §6 · p. 6, Table 1 · p. 6
- 11.6 percentage points: pooling the completed-primary sensitivity and conditional extension raises task success from 46.6% to 58.1%, with 37 rescues and no observed harms.The pooled task-bootstrap interval is [8.1, 15.3]; the pooled gain is positive in every category, but the study is not powered for separate category-level claims.Source: §6 · pp. 6–7, Table 2 · p. 6
- The pooled gain is positive in every BFCL category, but the study is not powered for separate category-level claims.Table 2 reports pooled results by category, with 80 tasks each and intervals conditional on extension launch.Source: Table 2 · p. 6, §6 · p. 7
6.1 Failure ranking and the conservative switch gate
- Pooled failure ranking reaches ROC-AUC 0.759 (numeric-suffix-cluster 95% CI [0.686, 0.824]) and average precision 0.829, against a pooled failure prevalence of 0.534.Figure 2 presents ROC and precision–recall curves for the completed-primary sensitivity, conditional extension, and pooled set.Source: §6.1 · p. 7, Figure 2 · p. 7
- The frozen gate selects 71 interventions across 3,665 action decisions (1.94%); none targets an actor success, and 37 interventions rescue failed trajectories.The 71 interventions yield a 52.1% repair yield; the policy's selection statistics measure failure targeting, not repair.Source: §6.1 · p. 7
6.2 Trigger-matched controls and attribution limits
- On the completed-primary sensitivity, task success rises by 1.2, 1.9, and 13.1 percentage points under selective self-revision, the selective absolute verifier, and the full protocol, respectively.Across the completed-primary sensitivity and conditional extension, the verifier accepts all 71 structurally valid candidates, and forward-order judgments agree with counterbalanced decisions.Source: Table 3 · p. 7, §6.2 · p. 8
7 Limitations and broader impacts
The evaluation is limited to stored trajectories from four BFCL V4 multi-turn categories under one actor model, and does not estimate robustness to model or API resampling. The paper also recommends safeguards for high-impact deployments.
Scope and attribution
- The evaluation covers stored trajectories from four BFCL V4 multi-turn categories under one actor model, with one stored trajectory per task.Exact replay fixes evaluator-visible behavior through the first accepted switch but does not estimate robustness to model or API resampling.Source: §7 · p. 8
- Positive cluster-resampling intervals are a sensitivity analysis and do not remove split-selection uncertainty; actor and verifier errors may also be correlated.Generalization to other models and interaction environments remains to be tested.Source: §7 · p. 8
Potential impact
- High-impact deployments should preserve audit trails, restrict tool privileges, and require external confirmation and independent monitoring.TWINCHECK is a verification layer, not authorization for autonomous consequential action.Source: §7 · p. 8
8 Conclusion
The conclusion frames execution-boundary repair as an asymmetric decision, with preservation as the default. Across the evaluated BFCL trajectories, the complete policy improved task success without observed success-to-failure regressions.
8 Conclusion
- Across the evaluated BFCL trajectories, the complete policy improved task success with no observed success-to-failure regressions.The passage reports no numeric effect size.Source: §8 · p. 8
- Declining a replacement preserves the actor’s proposal, whereas an unnecessary replacement can create a new failure.TWINCHECK makes preservation the default and requires trace-grounded evidence for a specific local alternative.Source: §8 · p. 8
- When verification can change what an agent executes, it must establish a case for the replacement, not just a case against the proposal.Source: §8 · p. 8
A Additional evaluation diagnostics
Figure 3 presents additional exact-replay outcomes: task-success gains by BFCL category for two test sets, alongside pooled paired transitions across 320 tasks.
Split-resolved outcomes
- Figure 3 separates task-success gain over the actor by BFCL category for the completed-primary sensitivity and conditional extension.Each category cell contains 40 tasks.Source: Figure 3 · p. 10, A Additional evaluation diagnostics · p. 10
- Figure 3 also summarizes pooled paired transitions across 320 tasks.Source: Figure 3 · p. 10, A Additional evaluation diagnostics · p. 10
B Illustrative certificate-to-twin constructions
Figures 4–6 illustrate three certificate-to-twin constructions, including a synthetic account-control case and retail-pattern composites. They are schematics, not sampled trajectories or additional evaluation results.
Illustrative constructions
- Figures 5 and 6 are manually constructed retail-pattern composites, not sampled τ-bench trajectories or additional evaluation results.A nonidentical candidate executes only after structural checks and the counterbalanced verifier gate.Source: B Illustrative certificate-to-twin… · p. 11
Figure 4
- After a successful lock_account call, an unauthorized unlock_account proposal matches a fixed-table inverse, so the deterministic twin deletes that call.With no unresolved later failure, the twin is the fixed completion response; “semantic inverse” denotes a prespecified lexical relation.Source: Figure 4 · p. 11, B Illustrative certificate-to-twin… · p. 11
Figure 5
- After a failed modification and an identical repeat, the certificate permits structural change when no later successful current-turn tool record intervenes.The generator proposes an evidence-gathering call instead of reissuing the failed write.Source: B Illustrative certificate-to-twin… · p. 11, Figure 5 · p. 12
Figure 6
- For a uniquely requested card identifier, the deterministic twin replaces only the stale payment_method_id and preserves the rest of the proposed call.The current identifier is established in context omitted from the schematic.Source: B Illustrative certificate-to-twin… · p. 11, Figure 6 · p. 12