PaperlayerView exact revision on arXiv

Source-linked AI summary

ContractRL: Shielded Group-Relative Policy Optimization for Auditable Tool-Call Repair

Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Yina Sa, Daren Zha

arXiv:2610.00328v2cs.AIcs.CLcs.LG

TL;DR

Structured tool calls can fail on a small number of schema or contract violations, while regenerating the whole object risks changing unrelated intent-bearing fields. ContractRL treats repair as a bounded, verifier-guided process with masked actions and deterministic validation. Under identical verifier information, it achieves higher semantic success with fewer generated tokens than Patch-SFT and full regeneration.

  • Problem

    Structured-generation methods commonly optimize what to emit from scratch, leaving the decision of how to repair a failed tool call—what field to change, whether to retry, or when to abstain—as the central problem.

    Source: §1 · p. 1

  • Method

    ContractRL uses a bounded repair MDP with typed verifier feedback, a contract-derived action mask, deterministic validation, and a ledger frozen before hidden targets are joined.

    Source: §1 · p. 1

  • Results

    0.9362 semantic success with 34.4 generated tokens for ContractRL, versus 0.9076 and 44.9 tokens for Patch-SFT and 0.9148 and 137.2 tokens for full regeneration under identical verifier information.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The protocol prevents invalid objects from reaching the executor and keeps canonical targets and semantic labels outside the online repair state.

    Source: §1 · p. 1

    Evaluation uses controlled corruptions and a deterministic sandbox with frozen tool responses; deployment against external services requires service-specific permissions and side-effect monitoring.

    Source: ETHICS STATEMENT · p. 9

Abstract

from arXiv · show

Structured tool calls often fail after only a small number of fields violate a schema or an execution contract. Regenerating the complete object enlarges the action surface and makes repeated repair difficult to audit. We introduce ContractRL, a contract-constrained sequential repair protocol that models verifier-guided JSON repair as a bounded decision process. At each step the policy observes the candidate, typed verifier feedback, JSON Pointer, immutable repair history, and remaining budget; a contract-derived action mask filters malformed or prohibited RFC-6902 operations before a deterministic validator performs the transition. We specify a contract-constrained group-relative objective for patch, retry, and abstention decisions while keeping canonical targets and semantic labels outside the online state until trace freeze. Under identical verifier information, ContractRL attains 0.9362 semantic success with 34.4 generated tokens, compared with 0.9076 and 44.9 tokens for Patch-SFT and 0.9148 and 137.2 tokens for full regeneration over 192 cases per seed and five seeds. Policy optimization improves semantic success from 0.9186 for supervised ContractRL to 0.9375. A separate three-seed paired evaluation against Patch-SFT yields a semantic difference of +0.0396 (95% CI $[+0.0137,+0.0662], p=0.0039$). Feedback, action-mask, budget, and schema-shift analyses connect these gains to localized correction, while adversarial and multi-turn evaluations characterize the remaining failure modes.

1 INTRODUCTION

ContractRL frames failed tool-call repair as a bounded decision over an already meaningful object, rather than regenerating the whole object. It combines verifier-guided actions with a deterministic validator and reports higher semantic success with fewer generated tokens in the five-seed sweep.

1 INTRODUCTION

  • ContractRL treats repair as a bounded transaction over an existing object, using typed verifier state, admissible local actions, and deterministic transitions before tool execution.This addresses the risk that broad regeneration changes unrelated intent-bearing fields.Source: §1 · p. 1
  • 0.9331 semantic success with 34.8 generated tokens for ContractRL, versus 0.8924–0.9041 semantic success and 46.9–148.1 tokens for the listed baselines in the five-seed 2B/r8 sweep.The baseline range covers Patch-SFT, full regeneration, and constrained decoding under their recorded information conditions.Source: §1 · p. 2
  • The paper formulates a CR-MDP, specifies a contract-aware policy objective and action mask, and freezes the repair ledger before joining hidden targets and semantic labels.Its evidence includes same-information controls, policy optimization and ablations, factorized analyses, paired uncertainty, and a blinded audit.Source: §1 · pp. 1–2

2 RELATED WORK

The related work spans generation-time constraints, post-processing, verifier-guided repair, tool-use agents, and policy optimization. ContractRL is positioned as a sequential repair policy that uses a contract-derived action space and bounded retry or abstention.

2 RELATED WORK

  • Structured-output repair connects program synthesis, tool-use agents, and conservative feedback updates, with precedents in iterative correction, stateful tool interaction, repository-level testing, and localized verifier supervision.The cited examples include ReAct and Toolformer, ToolSandbox (Lu et al., 2024), SWE-bench, Math-Shepherd, and ProcessBench (Zheng et al., 2024).Source: §2 · p. 2
  • Offline RL work motivates attention to support and split integrity under distribution shift (Fu et al., 2020), while tau-bench and AgentDojo add multi-turn and prompt-injection evaluation contexts (Yao et al., 2024).Broader policy-optimization and agent-evaluation work also supplies complementary baselines for training objectives, tool-use breadth, selective prediction, and reward auditing (Schulman et al., 2017; Rafailov et al., 2023; Ouyang et al., 2022; Wang et al., 2023; Zhou et al., 2024; Liu et al., 2024; Chen et al., 2021; Janner et al., 2022; Shao et al., 2024; Geifman & El-Yaniv, 2019; Ong et al., 2025; Chen et al., 2023; Skalse et al., 2022; Lambert et al., 2024; Xie et al., 2024; DeepSeek-AI, 2024; 2025).Source: §2 · p. 2
  • The closest comparisons span validator loops, schema-grounded patching, schema-conformant generation, post-processing, and generation-time constraints.The passage distinguishes BFCL as a function-calling benchmark rather than a repair algorithm.Source: §2 · p. 2
  • ContractRL's stated distinction is a verifier-conditioned sequential repair policy with a contract-derived action space, bounded retry and abstention, post-freeze oracle separation, and measured policy optimization.The related-work discussion also separates the BFCL structural mechanism screen from matched sandbox and adversarial semantic evaluation.Source: §2 · p. 2

3 METHOD

The method models verifier-guided repair as a bounded process over a public candidate and verifier state, with repair history and budget. A contract-derived mask and deterministic validator constrain actions, while the ledger separates online repair from offline scoring.

3 METHOD

  • Online inputs exclude canonical targets and oracle labels, which are joined only after trace decisions are frozen; serialized inputs contain the public state.The public contract also keeps gold candidates and target patches in offline scoring fields.Source: §3.1 · p. 3
  • Exact-patch match compares an ordered patch with the canonical operations, whereas final-object match compares the resulting object with the hidden target.The 64-case screen uses only add or replace operations and excludes nested, array, multi-field, and ambiguous or noisy-verifier cases; its zero-collateral result is construction-bound.Source: §3.1 · pp. 3–4
  • The CR-MDP state includes the current candidate, typed verifier code and JSON Pointer, immutable repair history, and remaining attempt and operation budgets.Its fail-closed transition checks syntax, path applicability, immutable paths, operation count, and schema prohibitions before validation; invalid actions consume an attempt and never reach the executor.Source: §3.1 · p. 3
  • The action mask removes malformed, unavailable, immutable-path, invalid-source, and over-budget operations, while locality remains a soft policy preference.The group-relative objective rewards semantic success and constrains collateral edits, retries, unsafe actions, and token cost under explicit budgets.Source: §3.1 · p. 3
  • The repair loop parses a bounded patch or ABSTAIN, filters it, validates applicable patches, records the outcome, and freezes the ledger before offline target scoring.The mechanism replay uses precomputed patches, so its one-attempt result is not a learned retry-policy test.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

4 EXPERIMENTS

The experiments evaluate ContractRL through structural diagnostics, same-information controls, case-matched semantic execution, and analyses of repair transitions and operating boundaries. Under identical verifier information, bounded patching achieves higher semantic success with fewer generated tokens and collateral edits than the listed alternatives.

4.1 EXPERIMENTAL SETUP

  • The 64-case BFCL replay records 64/64 structural-valid outcomes using precomputed single-operation repairs, so it audits protocol behavior rather than learned retry policy.The separate nine-case audit reaches 7/9 final structural validity and accepts 6/8 repairable cases.Source: §4.2 · p. 5, §4.1 · p. 5

4.2 MAIN RESULTS

  • With typed feedback, the 2B adapter reaches 0.9141 object match and 0.9727 schema validity, versus 0.0742 and 0.0820 for context-only.Because context-only removes the complete validation block, this contrast cannot identify which encoding component accounts for the difference.Source: §4.2 · p. 5
  • 0.9362 semantic success, 34.4 generated tokens, and 0.0169 collateral edits for ContractRL compare with 0.9076, 44.9, and 0.0488 for Patch-SFT and 0.9148, 137.2, and 0.1876 for full regeneration.All four arms receive identical public verifier information across five seeds and 192 cases per seed.Source: Table 2 · p. 6, §4.2 · p. 6
  • 0.9314 semantic success and 0.9768 schema validity are reported for ContractRL in the three-seed, case-matched held-out evaluation.The same ledger reports 34.6 generated tokens and 0.0547 mean retries; its nested/array cohort reaches 0.8516 semantic success.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 ANALYSIS AND ABLATIONS

  • ContractRL reaches 0.9453 first-attempt validity and 0.9768 final validity with 0.0547 mean retries, versus Patch-SFT’s 0.8897, 0.9446, and 0.1176.Full regeneration and constrained decoding are single-shot controls at 0.9392 and 0.9901 first-attempt validity, respectively.Source: §4.3 · p. 7
  • In the mixed-family ledger, ContractRL reaches 0.9314 semantic success, above full regeneration’s 0.9027 and Patch-SFT’s 0.8918.Across ContractRL case families, semantic success is 0.9502 for single-field, 0.8516 for nested/array, and 0.7935 for ambiguous/multi-field episodes.Source: §4.3 · pp. 7–8
  • The policy-optimization row reaches 0.9375 semantic success, compared with 0.9362 for the same-information ContractRL control.The controls and policy-component ablations are reported separately.Source: §4.3 · p. 8
  • On the adversarial suite, malformed feedback records 0.7168 semantic success and wrong pointers 0.7449.Injected tool text records 0.8989 semantic success; the report treats these as prompt and model-shift boundaries.Source: §4.3 · p. 8

5 LIMITATIONS

The mechanism screen covers single-field corruptions with precomputed patches, while semantic success declines on nested, array, ambiguous, and multi-field cases and in the multi-turn study.

5 LIMITATIONS

  • Semantic success falls from 0.9502 on single-field cases to 0.8516 on nested/array cases and 0.7935 on ambiguous/multi-field cases; eight-step multi-turn success is 0.6457.The 64-case mechanism screen uses single-field corruptions and precomputed patches; generalization to changing services and longer trajectories requires evaluation with corresponding tool states and failure costs.Source: §5 · p. 8
  • Removing the action mask or repair history can permit unsafe dispatch, so the complete contract is part of the evaluated system.Source: §5 · p. 8

6 CONCLUSION

ContractRL specifies a bounded patch, validator, and ledger interface for structured tool-call repair, with an auditable evidence boundary. In the case-matched sandbox and same-information control, it achieves higher reported semantic success than the named comparisons, while wrong pointers, malformed feedback, and multi-field corruption remain failure boundaries.

6 CONCLUSION

  • In the case-matched three-seed sandbox, ContractRL reaches 0.9314 semantic success with 34.6 generated tokens and 1.30 p95 latency, versus 0.9027 for full regeneration and 0.8663 for constrained decoding.Source: §6 · p. 8
  • Policy optimization reaches 0.9375 semantic success versus 0.9186 for supervised ContractRL-SFT; the same-information control reaches 0.9362.Source: §6 · p. 8
  • Wrong pointers, malformed feedback, and multi-field corruption remain explicit failure boundaries.The paper also reports ablation costs from removing the action mask, repair history, learned stopping, or collateral objective.Source: §6 · p. 8

ETHICS STATEMENT

The evaluation is limited to public tool definitions, controlled corruptions, and a deterministic sandbox with frozen tool responses; external-service deployment requires service-specific permissions and side-effect monitoring.

ETHICS STATEMENT

  • The evaluation uses public tool definitions, controlled corruptions, and a deterministic sandbox with frozen tool responses; external-service deployment requires service-specific permissions and side-effect monitoring.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

The supplementary material documents the evaluation contract, evidence ledger, and distinct structural and complete-output diagnostics. These records preserve separate populations and denominators while supporting audit of structural and semantic outcomes.

REPRODUCIBILITY STATEMENT

  • The supplementary material provides code, configurations, evaluation records, and descriptions of the task contract, validator transitions, split checks, denominators, and resource accounting.Canonical targets are joined only after traces are frozen, allowing structural and semantic scoring to be checked separately.Source: REPRODUCIBILITY STATEMENT · p. 9

A.1 REPAIR TRANSACTION AND TRACE IDENTITY

  • The repair transaction separates public observations, patch actions, validator responses, and offline scoring; learned patch arms can take a second attempt within the bounded budget.The target patch and object are held out of the online loop, and validator feedback is the only retry signal.Source: §A.1 · p. 12, Table 4 · p. 12

A.2 LEDGER INTERPRETATION

  • The evidence ledger freezes online traces before joining canonical targets or semantic labels, keeping structural definitions and later scoring fields distinct.Rows are keyed by case identifier and source manifest, and validator responses, patch sequences, retry counts, and resource fields become immutable after the digest is written.Source: §A.2 · p. 13
  • Split counts and assessed overlap checks use the same manifest, enabling case-level comparison while retaining separate denominators for each measurement.Source: §A.2 · p. 13

A.3 STRUCTURAL DIAGNOSTICS

  • The independent nine-case audit reaches 7/9 final validity, with 6/8 repair acceptance distinguishing recovery from rejection and rollback.The adapter comparison separates schema validity from final-object match and retains model-specific seed units.Source: §A.3 · p. 13, Figure 4 · p. 14

A.4 PARSER AND COMPLETE-OUTPUT DIAGNOSTIC

  • The eight-case complete-output diagnostic yields 8/8 parseable objects, 6/8 schema-valid objects, 6/8 exact final-object matches, and no token-cap hits.Figure 5 records parser and termination endpoints; token-cap hits are plotted separately from successful-output rates.Source: §A.4 · p. 14, Figure 5 · p. 14

B DATA AND EVALUATION PROTOCOLS

The protocols compare repair methods under recorded information and resource conditions, then test case integrity, semantic outcomes, retries, and validator boundaries. The appendix extends these records to patch operations and fail-closed transitions.

B DATA AND EVALUATION PROTOCOLS

  • The comparator protocol matches case identifiers, output caps, retry budgets, sandbox, and seeds, while public verifier information differs by arm.The accounting records each arm’s inputs, outputs, online feedback, and resources; Table 2 fixes public information across the four conditions.Source: B DATA AND EVALUATION PROTOCOLS · pp. 14–15, Table 7 · p. 15

B.1 CASE CONSTRUCTION AND SPLIT INTEGRITY

  • All listed overlap checks are zero, including semantic-equivalence clusters, normalized schema shapes, tool names, and candidate-pool membership.The held-out families use disjoint record, schema, tool, and candidate sources.Source: §B.1 · p. 15, Table 8 · p. 15

B.2 SEMANTIC AND ADVERSARIAL OUTCOMES

  • Single-field cases define local repair, while multi-field and nested-array cases expose structure-shift penalties and wrong-pointer or malformed-feedback cases mark rejection boundaries.Semantic outcomes are measured after accepted objects execute in the sandbox, with separate scenario denominators.Source: §B.2 · p. 15, Table 9 · p. 16

B.3 RETRY AND RESOURCE ACCOUNTING

  • ContractRL has the highest semantic endpoint and lowest generated-token and p95-latency costs among the recorded case-matched arms.The comparison is conditional on the fixed sandbox, three-seed ledger, and arm-specific public inputs.Source: §B.3 · p. 16, Table 10 · p. 16

B.4 TRANSITION AND RESOURCE STRATA

  • Transition and resource comparisons retain recorded case families rather than pooling them across cohorts.The tables report typed-feedback outcomes and resource-normalized comparisons by cohort.Source: §B.4 · p. 16, Table 11 · p. 16, Table 12 · p. 17

C IMPLEMENTATION AND BASELINES

  • Canonical targets and post-hoc semantic outcomes remain scoring fields, not online repair inputs.The evidence contract binds public inputs and ledger data, while canonical patches and objects remain offline.Source: C IMPLEMENTATION AND BASELINES · p. 17

C.1 PATCH VOCABULARY AND VALIDATOR TRANSITIONS

  • The validator checks each operation against the current object and schema before another retry; invalid or inapplicable operations consume an attempt and return a typed error without execution.Wrong paths and ambiguous reports consume the full retry cost, whereas add/replace repairs stay local on the selected screen.Source: §C.1 · p. 18, Table 15 · p. 18

C.2 FULL SPLIT AND CONTAMINATION AUDIT

The appendix audits split integrity, semantic execution, failure boundaries, resources, and repair components. Across these analyses, outcomes depend on case family and the recorded public inputs.

C.3 SEMANTIC EXECUTION SANDBOX

  • ContractRL’s accepted-object executor yields 0.9502 semantic success on single-field cases and 0.8516 on nested/array cases; semantic success is distinct from final-object match.The sandbox executes only structurally accepted objects and checks task outcome, tool error, and side-effect trace.Source: §C.3 · p. 19, §C.10 · pp. 21–22

C.4 ADVERSARIAL AND WRONG-PATH CASES

  • Immutable-pointer and malformed-output families are fully rejected, while wrong-path, ambiguous, nested/array, and multi-field families quantify the remaining acceptance and semantic-success boundary.The extended suite includes these cases and the validator fails closed when no safe patch is available.Source: §C.4 · p. 19, Table 18 · p. 19

C.5 TOKEN, RETRY, AND LATENCY LEDGER

  • ContractRL emits 34.6 tokens on average versus 138.9–147.8 for complete-object controls, while its semantic endpoint remains 0.9314.The resource ledger records generated tokens, validator calls, retries, latency, and semantic endpoint on matched case and seed units.Source: §C.5 · p. 19, Table 19 · p. 20

C.8 POINTER SEMANTICS AND COLLATERAL EDITS

  • Nested/array and multi-field rows show nonzero collateral and semantic degradation, bounding pointer-local repair; the exact single-field row is only a construction check.Collateral is measured on changed leaf paths outside the verifier pointer.Source: §C.8 · pp. 20–21, Table 22 · p. 21

C.9 RETRY-POLICY COMPARISON

  • ContractRL improves final validity over its first-attempt rate with 0.0547 mean retries, while Patch-SFT has a larger retry tail.Complete-object controls are single-shot references, not evidence that they would retry safely.Source: §C.9 · p. 21, Table 23 · p. 21

D.1 SCALING, CAPACITY, AND FEEDBACK ATTRIBUTION

  • Semantic success rises from 0.8124 at 0.8B/r4 to 0.9331 at 2B/r8 and 0.9446 at 7B/r16, while generated tokens remain between 33.1 and 35.6.The 7B row reports statistical uncertainty of 0.0009 and p95 latency of 1.68 s; reference arms use 139.4 and 148.1 tokens for full regeneration and constrained decoding.Source: §D.1 · p. 22, Table 25 · p. 22, Table 26 · p. 22
  • The full typed block reaches 0.9317 semantic success, versus 0.7815 for pointer-only, 0.8236 for code-only, 0.8001 for text, and 0.0678 for context-only feedback.The full block also has the lowest retry and collateral rates and the highest fail-closed rate.Source: §D.1 · p. 23, Table 27 · p. 23

D.2 SAME-INFORMATION CONTROLS AND POLICY COMPONENTS

  • With the public block fixed, masked JSON Patch with abstention has the highest semantic success and lowest token, p95-latency, and collateral costs; policy optimization improves outcomes and costs, while component removals reduce success or increase unsafe-action costs.Removing the action mask produces 26 unsafe dispatches and semantic success of 0.8971.Source: §D.2 · pp. 23–24, Table 28 · p. 23, Table 29 · p. 24

D.3 BUDGET, STRUCTURE, AND ROBUSTNESS BOUNDARIES

The analyses map the accuracy-cost frontier, structural and adversarial boundaries, longer-horizon and cross-family performance, selective abstention, and independent audits. They show where repair remains effective and where its measured limits emerge.

D.3 BUDGET, STRUCTURE, AND ROBUSTNESS BOUNDARIES

  • The two-operation, two-attempt contract raises semantic success from 0.8794 to 0.9314 with 34.6 tokens and 0.0189 collateral; larger budgets raise success further but also increase collateral and tokens.The measured frontier supports the selected bounded budget as the lower-cost point rather than the maximum-accuracy point.Source: §D.3 · p. 24, Table 30 · p. 24, Figure 7 · p. 25
  • Flat single-field cases reach 0.9506 semantic success, declining to 0.7951 for multi-field cases as structure and pointer depth grow.Nested objects, arrays, unions, and long pointers reach 0.8798, 0.8469, 0.8302, and 0.8134, respectively; retries and p95 latency increase.Source: §D.3 · p. 24, Table 31 · p. 25
  • All six verifier-noise conditions report zero leakage; stale schemas, wrong or ambiguous pointers, and malformed reports reduce semantic success while preserving fail-closed rates above 0.95.Injected tool text retains 0.9007 semantic success and 0.9911 fail-closed behavior.Source: Table 32 · p. 25, §D.3 · p. 26

D.4 LONG-HORIZON EXECUTION AND TRANSFER

  • Over eight tool calls, completion declines from 0.9761 at one step to 0.8073, and semantic success falls from 0.9321 to 0.6457.Cumulative retries, tokens, latency, and collateral rise with horizon.Source: §D.4 · p. 26, Table 33 · p. 26
  • Held-out semantic success ranges from 0.9024 on file tools to 0.8429 on code-edit tools, with database and code-edit families showing the largest collateral rates.Those collateral rates are 0.0368 and 0.0417, respectively.Source: §D.4 · p. 26, Table 34 · p. 26

D.5 SELECTIVE RISK, UNCERTAINTY, AND INDEPENDENT AUDIT

  • Raising the threshold from 0.50 to 0.90 reduces semantic risk from 0.0719 to 0.0174 while coverage falls from 0.9482 to 0.4047.Fail-closed behavior rises from 0.7821 to 0.9782; thresholds were fixed before held-out scoring.Source: §D.5 · pp. 26–27, Table 35 · p. 27, Figure 8 · p. 27
  • Paired differences favor ContractRL on schema validity and semantic success, with lower retries, tokens, collateral, and p95 latency; all six reported 95% intervals exclude zero.The largest standardized reduction is in generated tokens (effect size −0.91).Source: §D.5 · p. 27, Table 36 · p. 27

D.6 QUALITATIVE FAILURE AUDIT

  • ContractRL has the highest recorded human-audit agreement (0.9417) and lowest false-accept rate (0.0167); the qualitative audit retains local successes alongside nested-index misses and wrong-pointer or immutable-field refusals.The blinded audit keeps human labels separate from deterministic scoring, and the case-level sample includes both successes and failures.Source: §D.5 · p. 28, Table 37 · p. 28, §D.6 · p. 28

E.2 RESULT AND METRIC INDEX

The result and metric index links measurement families to their definitions and reported results, while preserving the distinct populations and seed units used across records.

E.2 RESULT AND METRIC INDEX

  • Table 39 links each measurement family to its definition and result.The index keeps structural diagnostics, controlled policy tables, and paired uncertainty tied to their separate populations and seed units.Source: §E.2 · p. 29, Table 39 · p. 29
  • Structural diagnostics retain small fixed populations, while controlled policy tables retain five training seeds.Paired uncertainty uses a separate three-seed case-matched ledger.Source: §E.2 · p. 29
  • The measurement index is a reproduction aid that distinguishes families rather than pooling their denominators.This separation preserves each measurement’s recorded result context.Source: §E.2 · p. 29, Table 39 · p. 29

Related papers

Loading 2610.00328v2…