PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning When to Withdraw: Curriculum Coverage and Rule-Specific Failures in Fine-Tuned Language Agents

Mohamed Abouzahra

arXiv:2610.21045v1cs.AI

TL;DR

When evidence or authority changes, agents must withdraw only actions whose support has failed while preserving supported work. This paper compares supervised fine-tuning curricula and prompt framings in NAQD-Env; SFT checkpoints outperform instruction-tuned baselines on agreement and withdrawal recall, but rule-level behavior remains uneven.

  • Problem

    When evidence changes or authority is revoked, agents must suspend unsupported actions while preserving eligible work and resuming only after repair.

    Source: Introduction · p. 1

  • Method

    The study compares original and enriched supervised fine-tuning curricula and uses separate diagnostics to measure withdrawing dependent authority and preserving independent authority.

    Source: Introduction · p. 2, Curricula and Supervised Targets · p. 5

  • Results

    SFT checkpoints show substantially higher reference-policy agreement and withdrawal recall than baselines, but rule-level performance remains uneven across curriculum coverage, composition, delegation, and reporting.

    Source: Surviving Routes and Curriculum Coverage · p. 10, Conclusion · p. 15

  • Takeaways & Limitations

    The four-arm rule-diagnostic account motivates evaluating what curricula distinguish, what training preserves, and what aggregate scores hide.

    Source: Conclusion · p. 15

    Each arm is one training run and the enriched curriculum bundles multiple changes, limiting estimates of run-to-run variance and attribution to individual curriculum features.

    Source: Surviving Routes and Curriculum Coverage · p. 10

Abstract

from arXiv · show

Tool-using language agents must revise prepared actions when evidence changes or authority is revoked. Selective withdrawal means suspending affected actions, preserving supported work, and resuming only after sufficient repair. We study supervised fine-tuning in NAQD-Env, a synthetic environment with explicit dependencies and a deterministic reference policy. Four LoRA checkpoints of Qwen2.5-3B-Instruct compare an original curriculum with an enriched one, the latter under three system-prompt framings with identical views and targets. Three instruction-tuned baselines and all four checkpoints are evaluated under three inference prompts on fresh validation cases, held-out dependency families, and held-out structural combinations. Under the naqd prompt, SFT checkpoints reach decision agreement of 0.77-0.92 and withdrawal recall of 0.46-0.86, versus 0.45-0.63 and at most 0.06 for the baselines. Rule-level diagnostics qualify these gains. The original-curriculum checkpoint wrongly withdraws 26 of 34 actions with surviving alternative support; enriched checkpoints withdraw 0-1. Conversely, all SFT checkpoints narrow prohibited combinations less often than the 7B and 8B baselines, a case absent from both curricula. Delegation results show more withdrawal but poorer preservation of independently authorized actions, and all four checkpoints omit valid event reports. Framing contrasts vary across metrics and splits and do not establish a reproducible advantage or equivalence. One training run per arm, jointly changed curriculum properties, prior evaluation exposure, and baseline-SFT quantization differences limit causal attribution. The contribution is a method for detecting rule-specific regressions that aggregate accuracy can conceal, not evidence that fine-tuning alone ensures agent safety.

Introduction

Selective withdrawal requires suspending actions whose support fails while preserving eligible work and resuming only after repair. The paper tests what task-specific fine-tuning preserves or loses using a controlled dependency-policy environment and rule-level diagnostics.

Introduction

  • The study compares supervised fine-tuning in NAQD-Env, where explicit dependencies, authority relations, events, and a deterministic reference policy make policy application observable.The benchmark evaluates application of a stated policy, not trustworthy-evidence discovery, natural-language attack recognition, or real-tool containment.Source: Introduction · p. 2
  • The paper examines what task-specific SFT preserves, acquires, and loses, including whether agents withdraw dependent actions while preserving independently supported work.Enriched curriculum arms share views and assistant targets but differ in system-prompt framing; all checkpoints are evaluated under all three inference prompts.Source: Introduction · p. 2

Relationship to the companion paper.

  • The study reuses the companion benchmark’s baselines and exploratory probes, including the 26/34 versus 0/34 survivor contrast, as comparison points rather than independent discoveries.This paper adds the full four-arm training comparison, neutral and checklist checkpoints, expanded rule diagnostics, and a historical development iteration.Source: Relationship to the companion paper · p. 2

Contributions.

  • The contribution is a rule-diagnostic account showing that higher aggregate agreement can coexist with inappropriate withdrawal, missed narrowing, and lost reporting.The protocol checks both required withdrawal and required preservation rather than crediting a general tendency to refuse or persist.Source: Contributions · p. 2

Related Work

The paper situates selective withdrawal among dependency revision, agent evaluation, and behavioral testing, while distinguishing its controlled policy task from security and compositional-generalization claims.

Dependency revision and agent evaluation.

  • NAQD-Env applies a specified dependency policy to action assessments; unlike truth-maintenance and AGM work [2, 6, 8], it proposes no new revision algorithm or AGM compliance claim.The study asks how task-specific SFT changes behavior under explicit dependency and authority labels, complementing stale-state evaluation [5].Source: Dependency revision and agent evaluation · p. 2
  • Prompt-injection, harmful-task, sandbox, shutdown-resistance, and AI-control benchmarks address related risks, but this synthetic assessment benchmark cannot establish real or interactive tool security.Its gate results concern replayed proposals and privileged simulator state.Source: Dependency revision and agent evaluation · pp. 2–3

Shortcut learning and compositional generalization.

  • Held-out family labels exclude generators and recombine templates, but enrichment teaches some component motifs, so the study does not claim transfer to wholly unseen rules or graph structures.The survivor-blindness pattern is consistent with a dependency-mention shortcut, but behavior alone does not identify an internal representation or learning mechanism.Source: Shortcut learning and compositional… · p. 3

Fine-tuning and behavioral regressions.

  • The paper treats lost composition-specific narrowing as an observed output regression, not proof that a rule representation was erased.It applies targeted behavioral tests alongside accuracy to separate withdrawal, preservation, repair, composition, and reporting requirements.Source: Fine-tuning and behavioral regressions · p. 3

Environment and Measurement

NAQD-Env represents prepared actions with explicit premises, authority, policy constraints, and event sequences; its reference policy specifies action options and event reports at each step. The evaluation uses several distinct metrics and rule-specific diagnostics rather than treating agreement as a complete account.

Scenarios and Reference Policy

  • Each scenario includes premises, prepared actions, a policy, and events; actions require ordinary conditions and absent impediments, with at least one complete alternative support set surviving when alternatives are listed.Sub-agent grants fall with their parent, while co-agent grants remain independently valid with respect to the parent's continued authority.Source: Scenarios and Reference Policy · p. 3
  • The reference policy resolves event blockers, safety, authorization, evidence, composition, and eligibility in a prescribed order, and separately requires current-step illegitimate event identifiers.Agreement measures adherence to this specification, not a universal definition of the safest response.Source: Scenarios and Reference Policy · pp. 3–4

Metrics and Scoring Boundaries

  • Mean episode agreement weights episodes equally, while decision-exact episodes match every decision and fully exact episodes also report correct event identifiers at arrival steps.Each episode's decision count is its number of prepared actions multiplied by its assessment steps.Source: Metrics and Scoring Boundaries · p. 4
  • Task score credits full and safe partial action outcomes net of experimental option prices, while reporting score separately rewards correct illegitimate-event reports and penalizes other reports or missed events.The option prices are experimental utilities, not calibrated deployment costs; unique-event reporting credit differs from arrival-step exactness.Source: Metrics and Scoring Boundaries · p. 4
  • Withdrawal recall and precision pool true positives, false positives, and false negatives at eligible invalidation steps; exact withdrawal also requires no false withdrawal.Backend or parsing failures are skipped by the withdrawal analyzer but count as accuracy mismatches, so the metrics have different inclusion rules.Source: Metrics and Scoring Boundaries · p. 4
  • Rule diagnostics separately measure withdrawal despite surviving alternatives, correct withdrawal of parent-dependent grants, preservation of independent grants, and narrowing of prohibited combinations.Evidence narrowing and composition narrowing are distinct diagnostic subsets.Source: Metrics and Scoring Boundaries · p. 4

Experimental Design

The experiment compares four LoRA checkpoints and three instruction-tuned baselines across controlled prompt framings, curriculum variants, and held-out evaluation splits. Its single-run, bundled-curriculum, and evaluation-history constraints limit causal interpretation.

Models, Prompts, and Arms

  • Four Qwen2.5-3B-Instruct LoRA checkpoints are compared with three instruction-tuned baselines; the v1 control uses the original curriculum and named prompt, while three v2 arms use enriched data with different framings.Every prompt already contains the full policy and worked examples; neutral means no appended procedure, not no policy instructions.Source: Models, Prompts, and Arms · p. 5

Curricula and Supervised Targets

  • The enriched curriculum conditionally attempts alternative-route examples, delegated grants, and initial narrowing cases, while neither curriculum includes prohibited combinations.Enrichment retains development-family identifiers and introduces motifs also used by held-out generators; the held-out configurations are excluded from training, but their concepts are not all unseen.Source: Curricula and Supervised Targets · p. 5

Training and Serving

  • V2 framing comparisons use identical user views and reference completions, whereas v1 and v2 differ in views and targets where enrichment changes scenarios.Each arm has one training run; the final epoch-two adapter is evaluated, and identical completions do not prevent system context from changing probabilities and gradients.Source: Curricula and Supervised Targets · p. 5, Training and Serving · pp. 5–6

Evaluation Splits and Units

  • Evaluation covers fresh-seed validation and shared family-held-out and structure-held-out splits, with every model tested under three inference prompts.The recorded evaluation-view check finds no exact overlap, but does not establish semantic independence or exclude prior evaluation exposure; the model receives its own prior assessments as state.Source: Curricula and Supervised Targets · p. 5, Evaluation Splits and Units · p. 6

Statistical Interpretation

  • Intervals describe variation across cases for observed checkpoints, not training seeds, and cannot estimate run-to-run variance or establish a training-treatment effect.The curriculum comparison changes several conditions together, and evaluation development alongside earlier inspection means the splits are not a preregistered one-shot external test.Source: Evaluation Splits and Units · p. 6, Statistical Interpretation · p. 7

Results

SFT checkpoints improve agreement and withdrawal recall over instruction-tuned baselines, but rule-level diagnostics reveal distinct failures in preservation, composition-specific narrowing, delegation, and event reporting. Framing contrasts and recovery results remain qualified by evaluation design and unmeasured training-run variation.

Aggregate Gains with a Different Error Profile

  • 0.768–0.920 SFT decision agreement and 0.46–0.86 withdrawal recall exceed baseline ranges of 0.452–0.628 and at most 0.06, yet unsafe-attempt rates remain 30.0–53.0%.The named v2 checkpoint’s held-out task values exceed Qwen-7B, but its unsafe rates are similar; case-bootstrap intervals do not measure training-seed uncertainty.Source: Aggregate Gains with a Different Error… · p. 7, Table 1 · p. 7, Figure 1 · p. 10

Surviving Routes and Curriculum Coverage

  • 26 of 34 F5 survivor actions were wrongly withdrawn by the v1 control, versus 0–1 by enriched checkpoints; each arm is one training run and v2 bundles curriculum changes.The contrast is consistent with the alternative-route-coverage hypothesis, but the bundled treatment does not isolate a component’s effect.Source: Surviving Routes and Curriculum Coverage · p. 10, Figure 2 · p. 11

Composition-Specific Narrowing Regresses

  • 0/19 and 0/21 designated prohibited-combination actions were narrowed by the named v2 checkpoint, while Qwen-7B narrowed 14/19 and 15/21.Neither curriculum included this combination case; the evidence-based narrowing cases appeared in training, but omission was not independently manipulated.Source: Composition-Specific Narrowing Regresses · p. 11

Framing Differences Remain Inconclusive

  • Framing rankings vary by metric and split, and evaluation-case intervals do not establish a reproducible training-treatment advantage, no effect, or practical equivalence.Across inference prompts, checkpoints retain qualitative survivor, composition, and reporting patterns; the largest shown agreement range is 0.017, a descriptive range.Source: Framing Differences Remain Inconclusive · pp. 11–12

Structural Transfer and Delegation Trade-offs

  • Enriched checkpoints show higher agreement on the shared structure split, while delegation diagnostics pair more correct withdrawals with poorer preservation of independently authorized actions.On F7, enriched checkpoints correctly withdrew 26–41 of 55 parent-dependent actions but kept only 6–9 of 18 independently authorized actions; Figure 3 separates these action subsets.Source: Structural Transfer and Delegation… · p. 12, Figure 3 · p. 12

Conditional Recovery and Reporting Omissions

  • 31–39 valid resumptions occurred in 31–40 v2 validation opportunities created by prior withdrawals, while all four SFT checkpoints omitted every legitimate target event report.Resumption counts are conditional on model history, not a common repair-opportunity set; reporting failure is not explained by empty supervision, and its cause remains undiagnosed.Source: Conditional Recovery and Reporting… · pp. 12–13

Gate replay.

  • Gate-on replay yields zero simulated-executed violations while retaining the same unsafe proposals, but does not validate controller soundness or real-tool containment.The model is not tested on interactive denial and replanning.Source: Gate replay · p. 13

Discussion

Aggregate agreement can coexist with rule-specific failures, so evaluation should test whether agents restrict invalid work while preserving eligible actions, recovering after repair, and reporting events. The discussion proposes matched counterexamples as a curriculum hypothesis, while cautioning that this study does not establish curriculum content as the sole determinant of behavior.

Why the rule-level view matters.

  • Higher agreement can coexist with inappropriate withdrawal, missed composition narrowing, and lost reporting, motivating evaluation of the conjunction of restricting invalid work, preserving eligible work, recovering on repair, and reporting events.The survivor test, delegation modes, and narrowing causes distinguish outputs that aggregate scores can conceal.Source: Why the rule-level view matters · p. 13

Curriculum coverage as a hypothesis.

  • Matched cases requiring opposite treatments are proposed for testing whether curricula need counterexamples to superficially adequate policies.The proposed contrasts cover surviving routes, grant dependencies, evidence- versus composition-based narrowing, and valid reporting; the study does not establish curriculum content as the sole determinant.Source: Curriculum coverage as a hypothesis · p. 13

Training targets and procedures.

  • The framing ablation tests a procedural reminder as context, not supervised intermediate applications of the seven checks.Future comparisons could train explicit rule applications, contrastive preferences, or corrective examples, with matched token budgets, coverage, quantization, and replication.Source: Training targets and procedures · p. 13

Implications for control.

  • Before consequential tool authority is granted, trained agents should be checked; learned compliance alone does not demonstrate an enforceable governing architecture.An external controller must independently validate allowed actions and mediate effects; denying useful work and oracle-replay execution scores answer different questions.Source: Implications for control · p. 14

Limitations and Next Experiments

The study’s causal and generalization claims are constrained by single-seed, bundled curriculum changes, serving differences, evaluation exposure, and a synthetic, explicitly specified domain. The authors recommend matched and replicated follow-up studies, an untouched evaluation set, and deployment-oriented tests with uncertain inputs and interactive enforcement.

Training replication and bundled changes.

  • One seed per arm and simultaneous changes to several curriculum properties prevent treatment-level uncertainty estimates and attribution of gains to any individual enrichment.A matched-budget factorial or targeted counterexample study with multiple independent training runs is needed for causal credit.Source: Training replication and bundled changes · p. 14

Serving confounds.

  • Baseline–SFT comparisons change quantization, and no matched-quantization counterfactual identifies training-only effects.One fine-tuned base model and three 3–8B baselines do not establish behavior in larger or different model families; serving and protocol factors may also affect measured behavior.Source: Serving confounds · p. 14

Evaluation exposure and leakage scope.

  • The record does not establish curriculum independence from prior evaluation inspection or a final test untouched after curriculum choices.The authors recommend freezing the curriculum and analysis plan before evaluating newly constructed cases with independently checked labels.Source: Evaluation exposure and leakage scope · p. 14

Specification and domain.

  • Synthetic procurement cases use explicit, trusted states and do not test raw-evidence graph inference, ambiguous policies, naturalistic attacks, real-user authorization, or long-running tools.Reference agreement can penalize a safe alternative, and utility prices are not calibrated to actual harm.Source: Specification and domain · p. 14

Statistics and reporting.

  • Case-bootstrap intervals and unadjusted exploratory p-values omit seed variation and full-comparison correction, while reporting failure remains undiagnosed.Recommended next steps include multiple seeds, one-rule-at-a-time equal-budget curricula, an untouched test set, and report-field diagnostics; deployment extensions should add uncertain inputs and interactive enforcement.Source: Statistics and reporting · pp. 14–15

Conclusion

In controlled NAQD-Env evaluation, SFT checkpoints outperform instruction-tuned baselines on policy agreement and withdrawal recall, but show uneven rule-level behavior. The study offers a four-arm diagnostic account, not evidence that learned compliance alone establishes autonomous authority.

Conclusion

  • Task-specific SFT checkpoints show substantially higher agreement with the reference policy and higher withdrawal recall than instruction-tuned baselines.These results are from a controlled NAQD-Env study.Source: Conclusion · p. 15
  • Rule-level diagnostics show uneven behavior across surviving-route preservation, composition narrowing, delegation, and required event reporting.The checkpoints show omitted composition cases, poorer preservation of independently authorized actions alongside delegation-related withdrawal, and omitted valid reports.Source: Conclusion · p. 15
  • Framing contrasts do not establish a reproducible training advantage or absence of an effect; the contribution is a four-arm rule-diagnostic account of observed checkpoints.The paper motivates checking what curricula distinguish and what aggregate scores hide before treating learned compliance as evidence for autonomous authority.Source: Conclusion · p. 15

Artifacts and Ethics

The paper reports a public repository with source, scripts, manifests, outcomes, and logs, and states that its benchmark uses synthetic scenarios without external tool execution. It cautions that these findings do not certify deployment safety.

Artifacts and Ethics

  • The public repository includes the inspected source snapshot, generators, training and evaluation scripts, manifests, outcomes, and logs for the seven main models and checkpoints.The v5 release provides the advertised adapter/prepared-input bundle; regeneration instructions do not guarantee that every historical development artifact is archived.Source: Artifacts and Ethics · p. 15
  • Training wall-time was 13.6 GPU-hours for four arms on one RTX 5090, while evaluation time and total development-iteration GPU-hours were not established.Evaluation wall-time was not separately recorded.Source: Artifacts and Ethics · p. 15
  • All scenarios are synthetic and no external tools execute in the benchmark, so the findings do not certify deployment safety.“Illegitimacy” is a simulator label, not a general judgment about real messages or users.Source: Artifacts and Ethics · p. 15

System Prompts

The system prompt defines a least-restrictive decision policy for prepared actions, with explicit rules for dependencies, authority, events, repairs, and reporting.

System Prompts

  • The prompt distinguishes proceed, verify, ask_permission, escalate, narrow, decline, withdraw, and resume, with precedence from event blockers through safety, authorization, evidence, composition, and proceed.Narrowing cannot bypass missing or revoked grants, recoverable evidence gaps, or safety constraints.Source: System Prompts · p. 18
  • Ordinary dependencies must hold alongside at least one listed alternative route, while failed sources invalidate derived premises and co-agent grants remain independent of parent validity.Sub-agent grants, unlike co-agent grants, fall with their parent.Source: System Prompts · p. 18
  • At event steps, withdraw actions blocked by stops, invalidations, or impediments, and resume only when an authenticated repair clears the last blocking cause.Grant revocation and external-action stops remain active; evidence repairs cannot clear them.Source: System Prompts · pp. 18–19
  • Report only current-step illegitimate event IDs, independently of action eligibility, and do not report legitimate invalidations or repairs.Unauthenticated approval claims are reportable, while pressure reminders are not.Source: System Prompts · pp. 18–19

Additional Results

Additional results cover inference prompts, family outcomes, response frequencies, curriculum composition, and a historical development iteration; the historical checkpoints are not additional main-experiment seeds.

Additional Results

  • Table 5 pairs unsafe-episode percentage with mean episode agreement across inference prompts, evaluating each checkpoint under all three prompts.The baseline prompt includes the full policy and examples; checklist and NAQD append the same procedure under different labels, and the v1 control validation content differs.Source: Table 5 · p. 20
  • Table 6 reports per-family outcomes under NAQD with 30 scenarios per development family and 33–34 per held-out family.The control validation content differs.Source: Table 6 · p. 20
  • Table 7 reports event-step option usage as a percentage of all decisions, not counts of valid uses.These outcomes are under NAQD with the gate off; control validation content differs.Source: Table 7 · p. 21
  • Each curriculum contains 3,000 procurement scenarios from development-family generators at seeds 1000–3999.Training-set percentages use the stated scenario or step denominators, while option mix uses decision targets.Source: Table 8 · p. 21
  • Table 9’s historical iteration uses a different held-out-family draw (n=100) and repeats the survivor-blindness pattern under changed conditions, not as a matched multi-seed replication.Its SFT checkpoints use a 3,000-example stratified subsample, one epoch, and fp16.Source: Table 9 · p. 22

Related papers

Loading 2610.21045v1…