PaperlayerView exact revision on arXiv

Source-linked AI summary

Making Agents More Consistent: Skills Should Form Habits for Repeat Tasks

Travis Weber, Rohit Taneja

arXiv:2609.25299v1cs.AIcs.CR

TL;DR

Agents are inconsistent on repeated tasks and spend much of their generated output re-deriving plans. The paper mines execution histories for coverage-bounded skill variants, admitting them through gates that include trace comparison to a retained reference. The habit-formed variant reproduced on all 456 repeated dispatches and was non-inferior to the replaced arms, while the routing guard’s inability to defer inputs remained a limitation.

  • Problem

    Agents disagree on repeated tasks, and much generated output goes to re-deriving plans, limiting consistency and wasting tokens.

    Source: §I · p. 1

  • Method

    The paper mines execution histories for competing skill variants with declared coverage regions and tests trace conformance against a retained reference’s variability.

    Source: §I · p. 2, §VI · p. 4

  • Results

    A habit-formed variant reproduced on all 456 repeated dispatches, was non-inferior to every replaced arm, and cut tokens by 14% to 56% per served request.

    Source: §XIII · p. 11

  • Takeaways & Limitations

    Separating routing from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.

    Source: §XIII · p. 11

    The routing guard cannot abstain, so it serves every input and makes false admissions without reasoning available to catch the error.

    Source: §XI · p. 10

Abstract

from arXiv · show

On repeated work, agents are inconsistent. We ran 42 tasks three times each and found that, depending on the model, 38% to 74% returned answers that did not agree. Consistency is what a buyer, an auditor, or a regulator requires, and agents do not have it. They are wasteful too: 95.3% to 97.2% of what an agent generates goes to re-deriving a plan the system already knows. We propose skill habit formation. An agent mines its own execution history for candidate skills, deterministic variants that compete against the incumbent rather than replacing it. A candidate declares the region of input space it claims, so the common case runs as a script and the rest falls through to reasoning. Four gates of ascending cost admit candidates; the central one tests a candidate's execution trace against a retained reference, within a tolerance measured from that reference's own run-to-run variability. On text-to-SQL, three of four reasoning arms reproduced their own output on 11 to 13 of 42 repeated questions and the fourth on 26 of 42, while a habit-formed variant reproduced on all 456 dispatches we repeated and was non-inferior to every arm it replaced (p<0.0001). It also used 14% to 56% fewer tokens, turning net positive after 7 to 53 reuses. We measured what this costs in accuracy. The guard admitted work it should have deferred on 2.6% of natural paraphrases and 26% of inputs near its boundary, and 11 of 13 such failures were invisible to the trace-conformance gate at any threshold. Deterministic errors repeat exactly: a bad habit is as reliable as a good one, and that is the price of the property that makes the system auditable. Separating routing from parameter extraction raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost.

I. INTRODUCTION

The paper frames output inconsistency and repeated planning as deployment and operating-cost problems. It proposes habit formation: mined skill variants compete with retained reasoning paths under coverage limits and staged admission.

I. INTRODUCTION

  • Agents disagree with themselves on 38% to 74% of repeated tasks, while planning accounts for 95.3% to 97.2% of generated tokens on the measured workload.The planning share is reported across three arms that meter reasoning separately; much of it re-derives an unchanged workflow.Source: §I · p. 1
  • Habit formation mines execution history for competing skill variants that claim a bounded input region, leaving other inputs to the reasoning path.The retained reasoning path stays available for the tail, allowing reversion by pointer operation.Source: §I · pp. 1–2
  • The paper cannot prove that a fast variant means the same thing as the reasoning it replaces, and adopts pseudo-oracle testing with acknowledged limits.Prior autonomy work addresses whether an agent may act, while this paper addresses verification of a candidate before it receives traffic.Source: §I · p. 2
  • Candidate skills compete with an incumbent through four ascending-cost gates, including trace conformance calibrated to the reference’s own variability.The contribution statement also reports preliminary evidence on two workloads, failure modes in admission, and a measurement plan for outstanding claims.Source: §I · p. 2

II. RELATED WORK

Prior work supplies ideas for guarded specialization, differential testing, reusable routines, and deferral, but generally does not retain competing skill variants for governance and revocation. The lineage model orders variants by inference use and dispatches through coverage guards while retaining an anchor.

II. RELATED WORK

  • Tracing JITs guard compiled paths and deoptimize on failure, but their deterministic or verified interpreter assumptions do not transfer to stochastic reasoners.This design keeps the slow path running, rather than merely recoverable.Source: §II · p. 2
  • The trace-conformance gate uses differential testing against a retained ancestor as a pseudo-oracle, comparing execution traces rather than outputs.Related work also supplies specialization, rejection-based deferral, repeated-reliability measurement, and evidence of planning-token duplication.Source: §II · p. 2
  • Earlier agent-improvement systems optimize artifacts in place or grow libraries monotonically, rather than retaining competing variants that a governance principal can revoke.Publicly documented coding-agent skills are described as single mutable artifacts updated in place.Source: §II · p. 2

III. THE LINEAGE MODEL

  • A skill is a declared capability contract with ordered variants ranging from model-inference-heavy instructions to deterministic scripts.The anchor is the slowest retained ancestor, terminal fallthrough, and reference for conformance.Source: §III · pp. 2–3
  • Coverage-gated dispatch serves an input with the first admitted variant, while the anchor admits everything and false positives risk confident errors without reasoning.The guard is tuned for precision over recall; coverage also measures the traffic share served without reasoning.Source: §III · p. 3
  • Deterministic variants are auditable but brittle: they cannot notice a changed world, and partial coverage bounds exposure by leaving the remaining traffic in contact with novelty.A variant claiming eighty percent of traffic leaves twenty percent in continuous contact with whatever is novel.Source: §III · p. 3

A. The lineage economy

The lineage economy generates candidates in batches while selectively retaining useful variants. Candidate generation mines execution traces and corrections, using bounded simulation only as a lower-trust source.

A. The lineage economy

  • The system protects the anchor and variants currently serving traffic, prunes other variants by non-selection, and controls false discovery across candidate batches.Batch-level false discovery control addresses the increased false-admission risk of testing many candidates against one key set.Source: A. The lineage economy · p. 3

IV. CANDIDATE GENERATION

  • Candidate generation mines historical traces, perturbed or counterfactual replay, and short simulations branched from real execution states, in descending trust order.Replay freezes near-deterministic decision points, distills high-entropy but learnable ones, and excludes the rest from the coverage predicate.Source: §IV · p. 3
  • Corrections can yield proposed deterministic guardrails by marking boundaries; a cluster of corrections in one region induces an assertion that blocks corrected outputs.The guardrail enters the tenant set only through the same human authorization required for a promotion.Source: §IV · p. 3

V. ADMISSION: FOUR DERIVED GATES

Candidates pass four gates of ascending cost, from recoverability and governance checks through trace conformance and production evaluation. The trace gate compares candidate behavior with retained-reference variability, but its projection discards payload values.

V. ADMISSION: FOUR DERIVED GATES

  • Admission authoring is per capability or tenant, not per candidate, covering guardrails, reversibility, non-inferiority margin, and dependency pinning.The stated authoring cost is O(capabilities), while candidate volume scales separately.Source: §V · p. 3
  • C0 probes whether withdrawal can return traffic to an executable variant, while C1 statically checks guardrail subsets and requires partial coverage to narrow.A failed C0 probe blocks promotion; re-anchoring onto a current model invalidates the prior C2 apparatus.Source: §V · p. 3
  • C3 tests non-inferiority on in-coverage traffic through shadow execution, live A/B, and human review before authorization.Shadow execution runs both paths while only the incumbent output takes effect.Source: §V · p. 3

VI. TRACE CONFORMANCE

  • The trace projection records invocation and action classes but discards free text and payload values, so a candidate can use the right tool with wrong arguments and still conform.The event spine already emits the recorded events for provenance, making the gate cheap.Source: §VI · pp. 3–4
  • The reference-derived key set includes symbol coverage, irreversible actions, guardrail events, escalations, refusals, corrections, and variant divergences, and sharpens with later corrections.The key set is versioned.Source: §VI · p. 4
  • C2 requires exact agreement on consequential actions and guardrail outcomes, then tests procedural divergence against the retained reference’s own run-to-run variability.The soft test uses weighted edit distance and a one-sided permutation test with batch-level false discovery control; opaque regions are marked C2-inapplicable.Source: §VI · p. 4
  • C2 can pass, decline, or escalate a candidate; splitting key traces into forge and sealed sets helps detect candidates that overfit exposed traces.Escalation sends a candidate with procedural divergence to C3 as an improvement hypothesis rather than automatically rejecting it.Source: §VI · p. 4

VII. RECOVERY

Recovery withdraws coverage to return traffic to reasoning, while monitoring and simulation check for divergence and goal insensitivity.

VII. RECOVERY

  • Narrowing or revoking a variant’s coverage predicate returns its traffic to the next level up without a rebuild.Source: §VII · p. 4
  • Recovery can follow a fired guard, sustained shadow divergence, detected concept drift, or human revocation.Source: §VII · p. 4
  • Periodic simulated outcome devaluation checks whether a variant stops when its objective loses value.A healthy habit remains goal-sensitive; a compulsive one does not.Source: §VII · p. 4

VIII. GOVERNANCE

Governance bounds unattended admission and makes promotion a human-authorized, reversible act, while monitoring reviewer behavior.

VIII. GOVERNANCE

  • Humans set the policy envelope for candidate classes, maximum unreviewed coverage, and C2/C3 parameters before admission.C0–C2 then run unattended within that envelope.Source: §VIII · p. 4
  • A human authorizes each promotion over a legible delta, bounding its blast radius with a coverage predicate.The paper makes no claim that this architecture complies with Article 14 of the EU AI Act.Source: §VIII · p. 4
  • Reviewer time, rejection rates, and evidence-panel expansion are monitored, with sustained degradation suspending unattended admission.Source: §VIII · p. 4

IX. WHEN HABIT FORMATION PAYS

Promotion is framed as an execution-cost trade-off whose break-even depends on in-coverage reuse and maintenance, but its auditability rationale remains untested.

IX. WHEN HABIT FORMATION PAYS

  • Promotion pays when expected in-coverage executions times reasoning cost exceed admission, monitoring, and fallthrough-maintenance costs.The condition is NcR > cA +NcM +cK.Source: §IX · p. 4
  • At design-partner scale, tenant-owned procedural skills with hundreds of yearly executions may not meet the cost inequality.Source: §IX · p. 4
  • No experiment tests whether auditors fare better, faster, or more accurately with habit-formed traces than reasoned traces.The paper therefore treats auditability as an argument, not a result.Source: §IX · p. 4

A. Personalization erodes the economics that justify it

Personalized coverage fits observed phrasing, but specialization fragments the shared execution base that supports amortization.

A. Personalization erodes the economics that justify it

X. PRELIMINARY EVIDENCE

On text-to-SQL, habit-formed dispatch reproduced every served output and was non-inferior to the reasoning arms, while reducing tokens after sufficient reuse. Follow-up tests show that coverage depends strongly on guard design and input scope; retrieval routing also improved on unseen intents, with distinct limits around boundary errors and admission.

Text-to-SQL results

  • All 456 served dispatches reproduced identical routes and parameters across five repeats, and the candidate was non-inferior to all four reasoning arms (p < 0.0001 throughout).The arms reproduced their own answers on 11–13 of 42 questions, except Sonnet on 26 of 42.Source: §X · p. 5
  • 888 tokens per served request reduced token use by 13.9%–55.5% versus the reasoning arms, with break-even at 7–53 reuses after formation costs.The net-saving curves cross zero at 7 (Kimi), 16 (Qwen), 17 (GLM), and 53 (Sonnet) reuses.Source: §X · pp. 5–6, Figure 1 · p. 5

Text-to-SQL guard tests

  • The regular SQL pilot’s held-out pass rate stayed at 18/18 for harder shapes versus 14/14 for easy shapes, but literal-regex guards covered 0/40 natural paraphrases.The pilot’s held-out instances varied literal values, not phrasing, so the initial clean result did not establish paraphrase generalization.Source: §X · p. 6
  • A semantic guard covered 39/40 held-out paraphrases at 97.4% accuracy among covered, with a dispatch cost of $0.00102 and break-even at 5.4–24.1 uses per template.The SQL template body stayed unchanged; only the coverage predicate changed.Source: §X · p. 6

Guard failure analysis

  • The semantic guard’s false-admission rate was 2.6% on natural paraphrases and 26.0% on boundary-probed inputs; 11 of 13 boundary failures were invisible to the trace-conformance gate.The gate’s soft trace test records argument schemas, not values, so wrong-parameter serves can produce identical traces.Source: §X · p. 7

Retrieval-routing results

  • At 100 mined traces, routing accuracy improved by +0.157 overall and +0.157 on unseen intents; at 200 traces, gains were +0.145 overall and +0.132 unseen.The reported intervals are over runs, and the pre-specified t=100 → t=200 contrast was −0.003 [−0.028, +0.022].Source: §X · pp. 7–8, Figure 2 · p. 8

Bundle scaling

  • Adding distractor templates cost $0.0000564 per dispatch per template, while false admissions arose only when semantic adjacency coincided with bundles of at least 20 templates.There were no wrong-template admissions; the observed scale-related decline involved out-of-scope questions.Source: §X · pp. 8–9

Retrieval-routing results

  • For retrieval routing, the false-admission rate fell from 0.382 to 0.237, but the severe direct-answer error rose from 0.000 to 0.071, below the 0.118 replication floor.The aggregate improvement came from fewer unnecessary retrievals; the new severe-error direction appeared in all six runs.Source: §X · p. 9

A. Which half of dispatch needs a model

Separating template routing from parameter extraction improves accuracy at lower cost, while routing proves more reliable than extraction. The results also show that routing costs rise with catalog size and parameter errors remain the main fault.

A. Which half of dispatch needs a model

  • The study separates template routing from slot filling using mined templates at catalog sizes 5 and 20, evaluated at reuse depths 5, 20, and 50.The frozen evaluation set contains 76 items, including questions no template should serve.Source: A. Which half of dispatch needs a model · p. 9
  • 456 of 456 routing decisions were correct, but all 51 false admissions came from wrong parameters, not wrong templates.The largest fault was an exclusive-window date error: the guard got 0 of 51 slots right, while an extractor given the comparison operator got all 51 right.Source: A. Which half of dispatch needs a model · pp. 9–10
  • 0.952 end-to-end accuracy at 43% of the cost, versus 0.888 for the joint call, follows from splitting routing and extraction.A deterministic extractor reaches 0.925 at zero marginal cost, but recovers 19 wrong answers, converts 16 more into declines, and costs 2 answers the joint call got right.Source: A. Which half of dispatch needs a model · p. 10
  • 0.776 routing accuracy for the embedding classifier trails the model guard's 1.000 despite ranking the correct template first on all 53 in-catalog items.Its abstention threshold discards 30% of already-correct argmaxes to decline 22 of 23 questions it should refuse; oracle threshold fitting recovers about a third of the gap.Source: A. Which half of dispatch needs a model · p. 10
  • At 20 templates, the guard spends 2360 tokens per dispatch, exceeding the reasoning arms' 1032 to 1998 tokens.Its prompt carries every competing template description, so the current design stops paying for itself on tokens alone beyond that catalog size.Source: A. Which half of dispatch needs a model · p. 10

XI. LIMITATIONS

The evaluated coverage guard is not itself gated, and one routing guard cannot defer uncertain inputs. The limitations also include catalog-level risk, weakening reversion safety in deep lineages, and unmeasured production-scale quantities.

XI. LIMITATIONS

  • The coverage guard is ungated, so a false admission quietly skips reasoning; proposed guard checks and escalation mechanisms were not evaluated.Boundary-weighted shadow sampling, held-out conformance testing, and margin-gated escalation are named as possible responses.Source: §XI · p. 10
  • The routing guard emits a route for every input and deferred none of 30,240 scored decisions, despite the design calling for escalation on ambiguous questions.It defaults to a route when parsing fails and computes a confidence value that no caller reads; one of the two domains therefore lacks the intended safety property.Source: §XI · p. 10
  • A set of narrow-coverage candidates can tile the input space while each passes gates on weak evidence, even though risk accrues across the chain.The authors identify a minimum coverage floor and chain-level evidence accounting as needed safeguards.Source: §XI · p. 10
  • Reversion safety weakens as lineage depth grows because intermediate variants absorb most declined work and the anchor sees only the deepest tail.The stated warmth argument holds for shallow chains.Source: §XI · p. 10
  • Trace conformance filters candidates but cannot certify better work, and its assurance relies on a baseline nobody audited.Generated candidates also carry the risks of generated code.Source: §XI · p. 10
  • Four framework quantities remain unmeasured, and the two-workload evaluation leaves the lineage architecture at production scale untested.The unmeasured quantities are achievable coverage, trace-opaque fraction, forge yield through C0–C3, and realized cK across a model deprecation cycle.Source: §XI · p. 10

XII. FUTURE WORK

The paper identifies failure modes that current admission gates do not reliably catch, especially silent parameter faults and risks that emerge as catalogs change. It calls for better held-out evidence and production monitoring.

XII. FUTURE WORK

  • A bad habit can be as reliably repeated as a good one, and the paper says mechanisms to detect and correct bad habits remain future research.The central gate's schema-based comparison misses most faults caused by wrong argument values.Source: §XII · p. 10
  • C2 misses dominant wrong-parameter faults in read-only queries, while a five-point C3 margin admits the measured 2.6% natural-paraphrase defect rate.The same non-inferiority test can also reject a cold memory-based learner, leaving a flat curve that looks like method failure.Source: §XII · p. 10
  • 18.8% false admissions appeared only when semantically adjacent templates co-occurred in catalogs exceeding twenty; adjacency or size alone produced none.Admission is not rerun for existing candidates when a sibling enters, motivating catalog-change re-admission and chain-level evidence accounting.Source: §XII · p. 11
  • Production lacks a gold answer, and the study has not measured what shadow-sampling detection rate its savings-funded budget buys at a 2.6% defect rate.The authors also identify a legible divergent-case delta as a prerequisite for trusting human review.Source: §XII · p. 11
  • Discovery success falls from 1.000 at one recorded execution to 0.900 at fifty because the gate tests traces used to mine the template, not generalization.The proposed repair is to split mined evidence into fitting and held-out portions; it has not been run.Source: §XII · p. 11

XIII. CONCLUSION

Habit formation improved repeatability and reduced token use in the evaluated workloads, but guards still admitted work they should have deferred. The conclusion highlights dispatch separation and catalog-level risk as broader findings, while leaving detection and correction mechanisms for future research.

XIII. CONCLUSION

  • All 456 repeated habit-formed dispatches reproduced their output and were non-inferior to replaced arms, which reproduced on only 11 to 26 of 42 questions.The variant reduced tokens 14% to 56% per served request and turned net positive after 7 to 53 reuses.Source: §XIII · p. 11
  • False admissions affected 2.6% of natural traffic and 26% of inputs near the guard boundary, and most such failures were invisible to the central gate.The gate compares argument schemas while the faults concern argument values; a bad habit repeats exactly like a good one.Source: §XIII · p. 11
  • Separating procedure selection from its inputs raised end-to-end accuracy from 0.888 to 0.952 at 43% of the cost, while catalog changes can make a previously safe template unsafe.The authors present both findings as applicable to systems that cache procedures beyond their own architecture.Source: §XIII · p. 11
  • Production deployment needs mechanisms to detect and correct bad habits, which the paper leaves for future research.Habit formation can produce a bad habit as readily as a good one.Source: §XIII · p. 11

Related papers

Loading 2609.25299v1…