PaperlayerView exact revision on arXiv

Source-linked AI summary

From Migration to Calibration: Aligning Agents with Deployment Requirements

Yaxiao Liu, Pengbo Liu, Yiwen Liu, Yihua Guan, Jiaxing Song, Xingjian Li

arXiv:2609.35149v3cs.AIcs.MAcs.SE

TL;DR

Agents can change substantially across deployment conditions, making explicit destination acceptance criteria important beyond simply replacing a model. The paper proposes a standards-linked calibration process and reports limited pilot evidence, while noting that confirmatory efficacy remains unestablished.

  • Problem

    Deployment changes can affect an agent’s model, support system, and use context, while successful model replacement alone does not show that it meets destination requirements.

    Source: §1 · p. 2

  • Method

    The paper specifies three destination-standard families and uses consultant supervision, configurable reward channels, and bounded adaptation around a frozen foundation executor.

    Source: §6.5 · p. 26, §6.6 · p. 28, §11 · p. 59

  • Results

    C-lite minus T0 execution was approximately −0.7 percentage points (pp), with a reported 95% interval of [−4.2, 2.3] pp; C-lite had eight gains and ten induced failures over T0.

    Source: §8.13 · p. 48

  • Takeaways & Limitations

    The proposal treats destination qualification and held-out improvement as separate outcomes and specifies independent gates and matched-budget tests for evaluating adaptation.

    Source: §6.5 · p. 26, §6.6 · p. 28, §8.6 · p. 43

    Pilot evidence is only partially reconciled: supplied labels and summaries retain provenance limits, and confirmatory validation remains outstanding.

    Source: §8.16 · p. 56, §10 · pp. 57–58

Abstract

from arXiv · show

Deploying, migrating, or scaling an agent changes its model, support system, application, and users. Calibration should align the agent with destination requirements without requiring source imitation or universal superiority. We propose three standard families: basic capability and operational reachability; technical and professional precision, including terminology, units, reporting periods, program fidelity, and report dependencies; and user context covering roles, use cases, authorized values, permissions, and budgets. A versioned mapping distinguishes requirements to preserve, transform, add, or retire. Four instruments---benchmarks, positive/negative examples, skills, and role-conditioned cost/utility---supply tunable reward channels within a fixed acceptance contract. Consultants annotate methods, branches, bounded loops, checkpoints, relative comparisons, and report refinement; a registry records applicable evidence. Adaptation revises configurations or trains a separate controller or skill while freezing the foundation executor. Independent hard gates govern qualification; RL and preference methods optimize candidates without supplying factual truth. Qualification and held-out improvement over the unchanged target are separate outcomes. Supplied intent labels permit a limited paired recount: small prompt/retry gains coexist with regressions and failed gates. FinQA supplements match prior summaries and retain provenance conflicts. These observations motivate destination-specific evaluation but do not establish calibration efficacy. We specify matched-budget tests, clean reload, complete-denominator reporting, and dependency revalidation. This is a methodological proposal with partially reconciled pilot evidence; confirmatory validation and the greater-than-80% zero-shot task-context acceptance target remain outstanding.

1. Introduction

Deployment changes can alter an agent’s model, support system, and use context, so a successful model replacement alone does not establish destination readiness. The paper proposes explicit destination standards, configurable supervision, and a bounded diagnose–adapt–recheck process, while treating qualification and improvement as separate judgments.

1. Introduction

  • The central question is how to adapt an agent to future use conditions using explicit acceptance criteria, without treating the source agent as a perfect standard or capability ceiling.An unchanged configuration may already satisfy a new contract, while equal average accuracy can conceal bidirectional drift.Source: §1 · p. 2
  • Calibration means defining destination standards, diagnosing gaps, applying adaptations, and rechecking the resulting agent against those standards.Initial deployment can use independent task references and user examples when no source agent exists.Source: §1 · p. 2
  • Three proposed standard families cover basic competence and operational reachability, technical and professional precision, and user context.User context includes roles, use cases, dependencies, authorized values, permissions, and resources; passing a question-answer benchmark does not establish report usability.Source: §1 · p. 2
  • Benchmarks, positive/negative examples, skills, and role-conditioned cost/utility serve as both evaluation evidence and configurable reward channels.Consultants annotate task methods, dependencies, branches, bounded repair loops, checkpoints, comparisons, and report refinement; the registry records selectable resources, but its existence or size does not establish coverage or accuracy.Source: §1 · pp. 2–3
  • A versioned deployment specification maps requirements to information, harness, and user-facing output, then diagnoses, revises, and rechecks affected components.Hard requirements such as disclosures and budgets cannot be traded away through reward optimization.Source: §1 · p. 3
  • The proposed contribution integrates deployment standards, consultant-to-agent supervision, and equal-budget tests; it does not claim the integrated specification is an established priority.The paper distinguishes destination qualification from calibration efficacy and requires strong target-native controls; the greater-than-80% transfer target remains unmeasured.Source: §1 · p. 3

2. Related Work

Related work establishes precedents for harness optimization, skill adaptation, evaluation contracts, process rewards, and personalization. The paper narrows its proposed contribution to their integration around independently grounded deployment requirements, dependency-aware repair, and destination qualification—claims that still require comparative evidence.

2.1. Prompt transfer and harness optimization

  • PromptBridge studies model-switch drift and transferable prompt mappings [1], while HarnessDev and HarnessOpt-Bench motivate model-sensitive harness design and held-out optimization controls [2] [3].Beating an unchanged source harness would not establish superiority to target-specific tuning.Source: §2.1 · pp. 4–5

2.2. Recovery, personalization, and evaluation

  • Recovery, personalization, and evaluation precedents motivate separating tool invocation from adequate output, user preferences from group norms, and optimization rewards from independent factual checks [4] [5] [6] [7] [8] [9] [10] [11] [12] [13].The cited review is focused rather than exhaustive.Source: §2.2 · p. 5

2.3. Agent training and frozen-backbone adaptation

  • Frozen-backbone controller training and fine-grained credit assignment have precedents [14] [15] [16] [17] [18] [19] [20] [35] [36], but they do not establish deployment compliance or recipient acceptance.The paper treats transfer from coding settings to user-specific report calibration as untested.Source: §2.3 · pp. 5–6

2.4. Recent harness transfer and judge-validity evidence

  • Recent harness-transfer and judge-validity work motivates matched target-native controls, independent correctness evaluation, and audits of task validity, recipient preference, and close rankings [21] [22] [23] [24] [25].Numerical findings from conversational benchmarks are not assumed to transfer to report or tool calibration.Source: §2.4 · p. 6

2.5. Process supervision and human-feedback adaptation

  • Process-supervision and human-feedback systems motivate step-level labels and budgeted user trajectories, but generalization to new users and deployment environments remains empirical [26] [27] [28].Math-trained process reward models do not certify financial reporting or infrastructure correctness.Source: §2.5 · p. 7

2.6. Retrospective adaptation, portable specifications, and runtime enforcement

  • Retrospective adaptation, portable specifications, runtime enforcement, and preference learning are established approaches [29] [31] [32]; the proposed distinction is independently grounded obligations with explicit dependency revalidation.A same-contract enforcement control can separate enforcement benefits from reusable adaptation.Source: §2.6 · p. 7, §2.7 · p. 7

2.7. Personalization and trainable agent controllers

  • Personalized and component-level RL systems are prior art [33] [34], so the proposed extension must test scarce-feedback efficiency and noncompensable domain requirements.Outcome-reward training is distinct from validated process supervision.Source: §2.7 · pp. 7–8

2.8. Expert rubrics, process rewards, and evaluator exploitation

  • Expert rubrics and process rewards have precedents [38] [39] [40], but evaluator audits and independent qualification are needed because training rewards do not certify executed tools or recipient-specific reports.Consultant reputation or rising training reward cannot replace that check.Source: §2.8 · p. 8

3. Problem Formulation

The formulation separates a fixed destination acceptance contract from the candidate agent and distinguishes qualification from improvement over an unchanged target. It combines versioned requirement mapping, independent checks, and configurable optimization while keeping mandatory gates noncompensatory.

3. Problem Formulation

  • Qualification requires passing mandatory tests, verifying required end-to-end paths, satisfying hard predicates and task/user minimums, and staying within budget.Fail, unknown, not-run, or mocked results cannot certify qualification; a finite suite is an operational baseline, not a guarantee for future tasks.Source: §3 · p. 11
  • The destination contract groups acceptance requirements into basic capability and operation, technical and professional precision, and user context.The contract is separate from the candidate; its requirements can constrain multiple implementation layers and require explicit applicability, authority, thresholds, and validity periods.Source: §3 · pp. 10–11
  • Table 1 frames requirement mapping from source deployment to destination contract; record that mapping before optimization and test inspection.The destination contract determines what must be retained or changed; source behavior is evidence of useful capability.Source: §3 · p. 11, Table 1 · p. 12
  • A candidate can qualify without beating the source, while a higher-scoring candidate can still fail a mandatory requirement; efficacy separately tests held-out improvement over the unchanged target.Both candidates face the same destination contract, and efficacy comparisons use matched budgets.Source: §3 · p. 12
  • The gap report distinguishes source-to-target drift, the unmodified target’s destination-contract shortfalls, and the adapted target’s changes against that same contract.For paired source-target outcomes, report harmful, beneficial, gross, and net drift with planned-denominator coverage and unresolved pairs.Source: §3 · pp. 12–13

3.1. Three standards and four instruments

  • The three standard families are distinct from four instruments—benchmarks, examples, skills, and role-conditioned cost/utility—that provide evidence and configurable reward channels.Hard acceptance remains noncompensatory: invalid trajectories cannot pass because another reward channel is large, and GRPO or DPO does not independently establish correctness.Source: §3.2 · pp. 14–15

3.2. Configurable reward design inside a fixed acceptance contract

  • Reward weights, sampling, checkpoint weights, and anchored transformations are frozen for each training round and before hidden evaluation.The registry can select versioned datasets, references, examples, annotations, and skill contracts; the synthetic consultant package is an interface illustration, not a populated database or trained calibrator.Source: §3.2 · p. 14, §3.3 · p. 15

3.3. Selectable data and reference registry

  • The registry distinguishes calibration-only learning resources from permitted runtime evidence, excludes hidden test information, and treats missing, stale, incompatible, or unauthorized data as explicit gaps.A larger registry alone is not evidence of generalized coverage.Source: §3.3 · p. 15

4. Information Calibration: Verify Required Content and Diagnose Gain or Loss

Information calibration uses traceable answer summaries and independent task requirements to check applicable content. It reports gains and losses separately rather than treating aggregate improvement as proof of preservation.

4.1. Traceable summaries and independent validity

  • Each answer is decomposed into traceable information units—claims, conclusions, conditions, evidence, and recommendations—with qualifications attached to the claims they constrain.Summaries support comparison, not truth determination; the original answer remains available for audit.Source: §4.1 · p. 15
  • Source units are labeled for validity and relevance, but independent task requirements remain necessary to detect omissions shared by both agents.Incorrect, irrelevant, and unresolved source content does not become a preservation obligation or get silently accepted.Source: §4.1 · p. 15

4.2. Gain and loss are separate outcomes

  • Report valid-unit loss, critical omissions, valid coverage, correct additions, and new errors separately; average gain cannot compensate for a preservation violation.Semantically equivalent compression is permitted, but zero observed violations on a finite test set does not imply zero future loss.Source: §4.2 · p. 16
  • Paired binary outcomes distinguish source-correct regressions, source-error reversals, and net accuracy change; calibration effects require a separate target-to-calibrated comparison.Unknown gold is not a correctness label, and grouped tasks require group-aware uncertainty estimates.Source: §4.2 · p. 16

5. Harness Calibration through Semantic Checkpoints

The harness represents tasks as semantic checkpoints with explicit artifact and dependency contracts, then validates, repairs, and revalidates observable outputs. It extends these controls to target-native actions, end-to-end deployment routes, and application-specific process supervision.

5.1. A shared semantic contract

  • Tasks are represented as semantic checkpoints with validated artifacts and dependencies, permitting different tool sequences without equating unlike calls.Repeated stages can be represented by finite, budget-bounded unfolding; automatically generated dependency graphs remain proposals for expert validation.Source: §5.1 · pp. 16–17

5.2. Validation and continuation criteria

  • Continuation requires hard predicates plus independently grounded task quality; source-relative gates are optional, and unknown judgments block work for repair or escalation.Validation coverage should also be audited against independent labels, including missed errors, false blocks, and checker overhead.Source: §5.2 · pp. 17–18

5.3. Diagnose, repair, and revalidate

  • Changed artifacts invalidate affected downstream outputs and cached verdicts, which must be recomputed or explicitly revalidated before final acceptance.Diagnoses record evidence, failure class, affected dependencies, permitted actions, uncertainty, and resource use; repairs repeat validation, while unknowns trigger bounded evidence collection or abstention.Source: §5.3 · pp. 18–20, Figure 4 · p. 20

5.4. Constructed execution example

  • The subscription example repairs missing tax-basis evidence with verified seller terms, but reuse requires configuration improvement on held-out tasks.Source: §5.4 · p. 21

5.5. Proposed actions are not verified artifacts

  • A proposed action is not verified until its observed artifact passes the contract; distillation should pair executed actions with deployment-visible history.Private reviewer deliberation and source-only evidence are excluded from student inputs unless equivalent retrieval is available at deployment.Source: §5.5 · p. 21

5.6. End-to-end deployment qualification

  • Qualification requires the actual end-to-end route, not HTTP success or mocked tools; fault tests may appropriately end in retry, abstention, or escalation.Connectivity gates precede behavioral comparisons, and unsafe continuation after a failed prerequisite counts as failure.Source: §5.6 · pp. 21–22

5.7. Application-specific process supervision

  • Financial-report checkpoints can supervise evidence, terminology, basis, calculations, disclosures, and role-specific synthesis without assuming a trained PRM exists.A learned process scorer requires task-specific training and held-out error-location audits, and cannot replace authorized compliance references.Source: §5.7 · p. 22

6. User Calibration: Output Contracts and Human Feedback

This section specifies user- and time-aware calibration procedures, including recipient-specific output contracts, gated human feedback, and reporting-period protocols. It covers adaptation workflows and consultant-method provenance while distinguishing preference learning from factual validation and bounding reaggregation claims to available evidence.

6.1. User experience extends beyond cultural expression

  • A recipient’s usable output depends on its report purpose, requirements, structure, evidence, units, and review workflow, not just factual accuracy or assumed national style.The US–China financial-report example requires authoritative, dated requirements and qualified review; it does not assume either jurisdiction has one universal template.Source: §6.1 · p. 22

6.2. Reference-guided, section-level preference calibration

  • Section-level preferences should compare alternatives under shared evidence and context, while hard-gate failures receive validity or repair supervision rather than a forced preference winner.The proposed section-level DPO application is not a new derivation or evaluated algorithm; accepted edits require whole-document revalidation.Source: §6.2 · p. 23

6.3. Four implementation routes and a critical distinction

  • Four routes adapt different artifacts: distill a smaller policy, search frozen-weight configuration, train a separate controller, or train a bounded skill policy while retaining a frozen executor.Agent-level human-feedback optimization covers distinct implementations; reports should identify changed parameters or artifacts, feedback sources, and training and inference costs.Source: §6.3 · pp. 23–25

6.4. Role, values, culture, and individual change

  • Context records should capture role, task, recipients, authorized preferences, verified access grants, dependencies, and resource limits; role adaptation cannot override factual or domain constraints.Nationality alone does not determine individual values, and role descriptions do not themselves grant access.Source: §6.4 · p. 26

6.5. A practical frozen-backbone controller protocol

  • The frozen-backbone controller acts on observable history through legal, budget-masked actions, while an external contract gate—not the controller—decides release.Offline improvement depends on useful target-native action support; feedback budgets do not guarantee successful adaptation, and qualification precedes held-out evaluation after clean reload.Source: §6.5 · pp. 26–27

6.6. Consultant-authored standards and dual-level trajectory supervision

  • Consultants define versioned standards and observable workflow checkpoints, while training compares outcome-only, process, and component-polish feedback under matched budgets and fixed supervision conditions.The initial calibrated, frozen-and-reloaded agent Ainit is the main baseline for testing subsequent controller improvement; failed initial qualification instead calls for a recovery experiment.Source: §6.6 · pp. 27–28

6.7. Trajectory distillation into a reusable skill API

  • A reusable skill has an explicit contract and bounded execution; its proposed training sequence verifies expert trajectories, distills public actions, then trains the student through target-environment interaction.For the manufacturing pilot, geometry tools, drawing evidence, expert review, and sandbox receipts distinguish a valid route proposal from executed production; generalization benefits remain hypotheses.Source: §6.7 · pp. 29–30

6.8. Consultant-annotated workflow and component-level Polish

  • Consultant workflow annotations represent dependencies, branches, bounded loops, alternatives, and unresolved exits, while component-level report edits trigger checks of affected cross-references and the whole document.Checkpoint records preserve original defects, repairs, and revalidation; cosmetic fluency cannot compensate for factual changes or broken dependencies.Source: §6.8 · p. 30
  • The synthetic package demonstrates workflow and report-polishing components, but is neither a completed consulting engagement nor a live financial-compliance specification.An extension protocol defines what interns must implement and return before an empirical claim is made.Source: §6.8 · p. 30

6.9. Temporal abstraction of consultant methods and reports

  • Consultants can version method libraries with workflow, evidence, and checkpoint annotations while recording provenance, disagreements, and distinct evidence and requirement timelines.A plausible abstraction does not prove the source process was correct; later-created methods cannot silently enter earlier deployment simulations.Source: §6.9 · pp. 30–31
  • Historical-as-of qualification admits only then-available methods and evidence, while retrospective upgrading applies a later method to fixed earlier evidence; unverifiable pretrained future knowledge remains a limitation.Changed professional obligations require separate contract strata, with shared predicates compared separately rather than ranking different compliance targets as one preference.Source: §6.9 · p. 31
  • Matched method comparisons can ground preference or controller training, but require independently justified labels, shared resources, and time/family splits; a timeline index alone demonstrates no RL improvement.Record ties, incomparable or invalid groups, and unknown labels; retain absolute anchors and measure both local continuation effects and whole-report acceptance.Source: §6.9 · p. 31

6.10. Reporting-period calibration and evidence-preserving reaggregation

  • Reporting-period conversion must fix calendar boundaries and accounting basis; exact reaggregation requires authorized daily records, while weekly totals alone cannot identify daily allocations.Under an actual-amount contract, insufficient evidence calls for unknown or abstention; approved estimates must be labeled and accompanied by uncertainty. Equal weekly totals can yield different recipient-period contributions.Source: §6.10 · pp. 31–32
  • Separate checkpoints verify evidence decisions, method applicability, period calculations, and dependency closure; changed period artifacts require dependent reports to be recomputed and revalidated.Reward weights cannot turn an unidentified daily distribution into truth, and DPO/GRPO comparisons need independently grounded labels under common evidence and context.Source: Figure 7 · p. 32, §6.10 · pp. 32–33

7. Joint Procedure and Reliable Relative Evaluation

The procedure fixes destination standards, adapts against diagnosed gaps within registered limits, and evaluates candidates using evidence distinct from optimization preferences.

7.1. Define standards, diagnose gaps, generate adaptations, and recheck

  • Register and freeze destination requirements, references, qualification tests, data access, repeat rules, budgets, and acceptance checks before adaptation.Diagnose requirement-level gaps, package each proposed revision with its evidence and checks, then recheck the same standards; unresolved gaps and exhausted budgets exit explicitly.Source: §7.1 · p. 33
  • Reload the selected package without calibration-time context, qualify its frozen implementation, and assess generalization on independent hidden tasks.Hidden results cannot guide selection; repairs require affected-dependency revalidation, and feedback-driven reusable updates require a fresh held-out assessment.Source: §7.1 · pp. 33–35, Figure 8 · p. 35

7.2. Evidence before preference

  • Evaluate outputs against executable tests, trusted records, traceable sources, or human-verified labels before making randomized, identity-blinded comparisons.Disagreement or missing support triggers retrieval or adjudication rather than a forced winner; report coverage, erroneous acceptance, order sensitivity, and human-annotation agreement.Source: §7.2 · p. 35
  • Audit whether metrics measure intended properties and whether evaluators rank candidates reliably; preference or judge agreement alone cannot establish factual correctness.Keep task correctness separate from recipient preference, and test closely performing pairs, ties, fluent factual errors, abstention, coverage, and order sensitivity against independent labels.Source: §7.2 · pp. 36–37

7.3. DPO and GRPO are optimizers, not truth criteria

  • DPO and GRPO can optimize trainable policies, but relative rewards do not validate factual judgments or turn an all-incorrect group into acceptable supervision.The reference policy in DPO is an optimization distribution, not an independent factual answer; labeling and training costs belong in the comparison.Source: §7.3 · p. 37

7.4. One framework, three deployment scenarios

  • The framework applies to model replacement, recipient-contract changes, and global data collection, with each scenario retaining its own required checks and constraints.Cross-border preferences cannot establish compliance; global collection preserves raw observations and provenance, while scaling requires per-site and per-market results rather than averages alone.Source: §7.4 · pp. 37–38

8. Experimental Protocol and Falsifiable Hypotheses

The protocol proposes controlled tests of calibration across deployment shifts, matched alternatives and budgets, and report-specific feedback and training routes.

8.1. Research questions and controls

  • Vary model, tools, recipient contracts, site heterogeneity, and workload independently before combining shifts, with same-model repeats to distinguish switch-related drift from run variation.Test at least two model families in both directions and preserve independently correct target capabilities rather than defining success as source imitation.Source: §8.1 · p. 38

8.2. Data, budgets, and baselines

  • Compare the full framework and layer ablations with direct substitution, fixed retries, and a strong target-native optimizer under matched task standards and tuning budgets.Charge source-trace collection, search, tools, evaluation, and adjudication; repeat full calibration searches across seeds and report cost sensitivity and held-out performance distributions.Source: §8.2 · pp. 39–40

8.3. Testing output contracts and training routes

  • Test report feedback and training routes on complete-document acceptance, hard-gate failures, contradictions, and transfer to held-out users and templates.Compare section feedback with dependency-aware revalidation and test credit assignment: local section rewards may polish a component while harming global correctness.Source: §8.3 · p. 40

8.4. Three application tracks

The protocol defines three application tracks and separates qualification from held-out effectiveness, with paired, clustered analysis and explicit failure accounting.

8.4. Three application tracks

  • Model-replacement tests hold tools and contracts fixed initially, measure critical capabilities, and distinguish cross-family from foundation-to-post-trained changes.Where matched training lineage is unavailable, report observational comparisons without attributing differences causally to post-training.Source: §8.4 · p. 40
  • Cross-border tests reuse underlying records under reviewer-specified destination contracts and measure field compliance, factual retention, whole-document acceptance, and unresolved conflicts.The outcome is domain-reviewed task conformance, not broad legal compliance.Source: §8.4 · p. 40
  • Global-scale tests hold out site-market combinations and report semantic accuracy, provenance, missingness, throughput, cost, and worst-group performance.A volume-weighted average cannot conceal failure of a mandatory site or market contract.Source: §8.4 · pp. 40–41

8.5. Outcomes and statistical analysis

  • Qualification requires every applicable mandatory check, while held-out completion is reported on its full denominator; aggregate gains cannot erase hard failures.Use paired comparisons and cluster-aware uncertainty, preserve failures and abstentions, and keep endpoint-specific intervals and estimands distinct.Source: §8.5 · pp. 41–42

8.6. Minimal experiment that can refute the proposed advantage

The proposed minimal test compares a structured diagnosis-and-enforcement package with matched alternatives, then evaluates cleanly reloaded candidates on hidden tasks.

8.6. Minimal experiment that can refute the proposed advantage

  • The three-arm study compares ordinary failed-requirement feedback, requirement-to-artifact localization, and localization plus dependency invalidation and fresh downstream checks.All arms share the optimizer, evidence regime, edit space, references, acceptance rules, and calibration and deployment ceilings.Source: §8.6 · pp. 42–43
  • Reload frozen candidates without builder state or online source access, then use an independent checker for the primary complete-task comparison of C versus A.B-versus-A and C-versus-B are component contrasts; C-versus-A alone cannot establish a diagnosis-only effect.Source: §8.6 · p. 43
  • The efficacy claim is unsupported if the package fails to improve independent held-out completion at comparable cost, causes unacceptable regressions, or only improves judge scores.Qualification and auditability may remain useful without an optimization gain, but constitute different contributions.Source: §8.6 · p. 43

8.7. Minimum qualification, drift, deployment paths, and repeated changes

  • Repeated deployment-change tests measure transfer before further adaptation and require fresh held-out evaluation after each newly budgeted calibration.The protocol measures declared shifts, not open-ended future adaptability; changed checkers or requirements require reassessment.Source: §8.7 · p. 44

8.8. Limited-feedback automatic calibration extension

The proposed extensions test limited-feedback adaptation, consultant-supervised controller improvement, compact skills under manufacturing shifts, checker portability, and industrial precision and context transfer. They specify matched-resource comparisons, independent qualification, and held-out evaluation; the proposed acceptance target is not a demonstrated result.

8.8. Limited-feedback automatic calibration extension

  • Limited-feedback tests compare frozen and trainable adaptation controls under matched references, export constraints, and declared budgets, while treating low-data failure and qualification without modification as valid outcomes.Proposed trajectory counts are learning-curve budgets, not power guarantees; training compute, feedback, and deployment costs should be reported separately.Source: §8.8 · pp. 44–45

8.9. Consultant-supervised improvement over initial calibration

  • Consultant-supervised improvement is measured against a frozen, reloaded, already-qualified Ainit using paired hidden-task hard-gate pass rates and complete-denominator reporting, not training-judge scores alone.The design separates budgets, controls resource access across feedback conditions, and reports uncertainty, worst groups, false releases, repair, dependency revalidation, user acceptance, and cost–quality.Source: §8.9 · p. 45

8.10. Compact skill APIs under manufacturing shifts

  • Manufacturing-shift tests compare the qualified initial agent, deterministic skill code, retrieval, and trained skills under a shared executor and deployment cap, with all calls and fallback included in evaluation.The desired improvement over the initial agent and SFT-only route is an objective to test, not a guarantee or demonstrated physical-production result.Source: §8.10 · pp. 45–46

8.11. Checker stress tests and qualification portability

  • Checker stress tests audit false acceptance, false blocking, abstention, and coverage against independent references, while keeping evaluator sensitivity separate from adaptation efficacy.Checker changes require fresh audit material; replacement stress tests assess which qualification claims survive checker or dependency changes, not universal reliability or actual financial compliance.Source: §8.11 · p. 46

8.12. Industrial precision, context transfer, and reward controls

  • The industrial precision extension tests terminology, formulas, program fidelity, units, recipient-specific reports, dependencies, and reporting-period handling using controlled fixtures; actual compliance requires authoritative dated rules and qualified review.Evaluation distinguishes exact daily reaggregation, indicator-assisted estimation, and weekly-only non-identifiability.Source: §8.12 · p. 46
  • The greater-than-80% target is prospective joint acceptance across applicable standards on feasible new-task and context instances, not a current result or a claim about arbitrary industrial use.Zero-shot transfer excludes new held-out-specific demonstrations, annotations, policy updates, or hyperparameter choices; reward controls and improvement are evaluated under fixed budgets against the initial agent.Source: §8.12 · pp. 46–47

8.13. Exploratory intern reports and the resulting protocol revision

The exploratory FinQA reports and intent-routing pilot show bidirectional correctness changes and limited rule-based gains, but unresolved provenance and measurement constraints preclude confirmatory claims. The revised protocol calls for independent evidence recovery, matched-budget tests, and separate measurement of acceptance, drift, and cost.

8.13. Exploratory intern reports and the resulting protocol revision

  • FinQA evidence remains report-derived: raw responses, complete score records, frozen specifications, and a verified provider/configuration/code chain remain missing or unreconciled.The October 13 supplements match earlier test aggregates, but matching summaries do not establish independent replication.Source: §8.13 · p. 47
  • C-lite minus T0 execution was approximately −0.7 pp in FinQA R1 (95% interval [−4.2, 2.3] pp); C-lite and T2 had equal reported final-answer correctness, 59.0%.The report’s claimed p < 0.05 is not reproduced by the inspected summary code, which computes bootstrap intervals but not a corresponding p-value.Source: Table 4 · p. 47, §8.13 · p. 48
  • In FinQA R1, source-to-raw-target execution differed on 29/300 task–repeat outcomes (9.7%); C-lite’s eight gains and ten induced failures yield a net change of −2.Five of 17 source-harm outcomes were reported repaired, but same-model variation and raw traces are needed to attribute disagreement to model replacement.Source: §8.13 · p. 48
  • C-lite minus T0 execution was 0 pp in FinQA R2 (reported interval [−5.8, 5.3] pp), while C-lite minus T2 was +2 pp ([−4.3, 8.1] pp).The +2 pp final-answer and +1 pp strict-success differences over T0 have no supplied endpoint-specific intervals; the reported direction is not a verified reverse-direction replication.Source: Table 5 · p. 48, §8.13 · p. 48
  • The revised protocol requires provenance recovery, independent checker audits, fresh grouped holdouts, matched-resource comparisons, and only then tests of a trained controller or skill.The reports motivate testing destination-specific requirements but do not establish calibration efficacy or the unmeasured greater-than-80% held-out transfer objective.Source: §8.13 · p. 49, §8.14 · p. 52

8.14. Industrial intent-routing pilot: aggregate gain, retention loss, and a no-effect rule set

  • C-lite adds zero percentage points over T0 on Test, and its predicted labels match T0 for all 100 IDs.The rules reportedly did not activate, while 21 Test intent errors remained above the confidence threshold; complete response artifacts and incremental checking costs are unverified.Source: §8.14 · p. 50
  • The raw intent target gains 4 percentage points overall while losing 3/75 (4%) of source-correct outcomes, based on the reported paired recount.The 100-item figure’s equal-sized cells are categorical, not area-proportional; original responses and independent gold remain unverified.Source: §8.14 · p. 50, Figure 10 · p. 51
  • The intent pilot has unresolved count, split, ontology, context, and provenance discrepancies; its stated zero-harm criterion is failed by three harmful pairs.Listed token and time differences do not establish monetary or end-to-end delivery savings without a complete resource ledger.Source: §8.14 · pp. 49, 52

8.15. Proposed staged validation across the three standards

The staged plan separates qualification from held-out improvement and tests basic, professional, and user-context requirements under frozen configurations and explicit evidence rules.

8.15. Proposed staged validation across the three standards

  • Candidates P, H, K, and R are cumulative, so adjacent gains do not isolate component effects; R restarts the full trajectory after failure.The proposed total is 2,070 test episodes, or 2,270 with S_K; new runners, evidence isolation, and monetary hard stops remain implementation milestones.Source: §8.15 · pp. 53–54
  • The staged workloads are proposed, not completed experiments or power-justified confirmatory sample sizes.The plan calls for paired family-level intervals, preregistered power, multiplicity control, and retention margins; episodes are not independent samples.Source: §8.15 · pp. 53–54
  • E1 distinguishes numerical correctness from a verified pass requiring evidence, valid units and periods, report consistency, reachability, completion, and budget compliance.Source answers are not independent references; optional source-model comparisons require running the frozen target configuration on that model.Source: §8.15 · pp. 53–54
  • E2 fixes eligibility from evidence and contract: daily data supports exact reaggregation, approved proxies permit labeled estimates, and unidentified daily amounts require a data request.A deterministic baseline enters the full-report comparison only if it can produce the same complete report; otherwise it is limited to numerical subtasks.Source: §8.15 · p. 54
  • E3 holds out a role–template combination to test composition transfer, with two blinded reviewers assessing each of 120 reports and adjudicating disagreements.Generic demonstrations do not imply strict zero-shot performance; configurations and contracts must be frozen jointly before shared tests are inspected.Source: §8.15 · p. 54

8.16. October 13 evidence reconciliation

The reconciliation narrows the empirical claim to supplied-label patterns: small prompt or retry increments coexist with regressions, and the evidence does not establish qualification or calibration efficacy.

8.16. October 13 evidence reconciliation

  • The new delivery supports reproducing supplied-label counts and reconciling reported aggregates, but not independently verifying responses against gold.Table 11 separates accuracy from structural compliance and identifies which endpoints are recounts versus summary-level evidence.Source: §8.16 · p. 54, Table 11 · p. 55
  • For B-to-A, T1 versus T0 repairs two errors and introduces one, a net +1/100, or +1 pp.This is a supplied-label recount, not independent adjudication or a controlled estimate of prompt efficacy.Source: §8.16 · p. 55, Table 11 · p. 55
  • For A-to-B, T2’s +3 pp over T0 is an aggregate difference, yet it loses 2/79 source-correct items and passes the structural contract on only 97/100.It fails the full-passage contract gate and stated zero-source-loss objective; other technical, user, permission, and cost requirements remain unqualified.Source: §8.16 · p. 55
  • FinQA’s supplied test JSON matches earlier summaries but adds no independent replication; unequal realized costs also preclude an efficiency claim.A-to-B has 300 task–repeat episodes per method and B-to-A has 100, despite 100 questions per direction.Source: §8.16 · p. 55
  • Run metadata conflict across archives, while the 80-row review worksheet has empty reviewer fields and only 20 unique task IDs.The receipt artifacts do not establish a common verified run chain or evidence user/value calibration.Source: §8.16 · pp. 55–56
  • Supplied labels show bidirectional redistribution of successes and failures, but do not establish full qualification or controlled calibration efficacy.The recount and reconciliations are not confirmatory experiments; original responses and independently adjudicated gold remain unverified.Source: §8.16 · pp. 54, 56

9. Future Work: An Automatic Calibration Tool

The proposed automatic tool would qualify deployment-specific standards, apply bounded adaptations, and recheck candidates; its integrated acceptance and transfer benefits remain research questions.

9. Future Work: An Automatic Calibration Tool

  • The proposed tool registers versioned standards, diagnoses gaps, applies adaptations, and rechecks the same requirements before exporting a frozen, reloaded candidate.It accepts a source agent when available, a target specification, qualification tests, and authorized user trajectories; an audit record alone is insufficient.Source: §9 · p. 56
  • Consultant packages connect recipient-approved materials and workflow annotations to selectable evidence and four adjustable reward channels.The initially calibrated agent remains a qualified baseline, and subsequent gains must account for expert labor and total optimization cost.Source: §9 · p. 56
  • The implementation must declare whether it trains policy, controller, or skill parameters, or only changes external configuration around a frozen backbone.Prompt search alone is distinct from RL, which requires a trainable policy, transitions, reward/credit assignment, and actual parameter updates.Source: §9 · p. 56
  • Hard acceptance predicates remain noncompensable gates; rewards rank admissible candidates, while hidden task and user outcomes remain unavailable to optimization.Fresh held-out evaluation is required after revisions, alongside uncertainty-aware selection and rollback to a qualified checkpoint.Source: §9 · p. 56
  • The main test is whether the integrated specification improves independently verified industrial acceptance and held-out transfer under matched resources.The roadmap and CPU starter are not an integrated automatic calibrator or evidence of RL-based improvement.Source: §9 · p. 57

10. Limitations and Responsible Use

The proposal remains vulnerable to faulty standards, measurement and evaluator errors, overfitting, cost, and limited evidence; its conclusions must remain within the evaluated scope.

10. Limitations and Responsible Use

  • Incorrect or incomplete standards and checkers can falsely qualify or unnecessarily reject agents, and audits do not certify completeness.Opaque tools limit observability, while deployment or checker updates expire affected claims; staged shifts do not prove unlimited transfer.Source: §10 · p. 57
  • Calibration evidence is vulnerable to measurement error, and conclusions from verifiable tasks should not extend to subjective preferences without suitable evidence.Summary extraction, validity labeling, checkpoint alignment, and the tension between strict preservation and concision affect measurement.Source: §10 · p. 57
  • Joint calibration can be expensive, and the framework neither identifies internal knowledge organization causally nor guarantees every target model will outperform its source.Locally adequate artifacts may still produce a globally inadequate answer, while service and tool changes complicate reproducibility.Source: §10 · p. 57
  • Reviewer and consultant errors can be correlated, while rewards can double-count evidence or encourage ritual checking and evaluator exploitation.Distilled skills can inherit teacher errors and restricted coverage; a skill API cannot supply missing engineering specifications or ensure correct composition.Source: §10 · p. 58
  • Development-data overfitting and expert labor, registry, data-access, and review costs may undermine the proposed approach.The greater-than-80% zero-shot target is prospective and scoped to the declared population; test-guided revisions require fresh held-out assessment.Source: §10 · p. 58
  • Pilot evidence remains partial: supplied-label recounts and aggregate reconciliation do not verify original responses, gold labels, or controlled efficacy.A-to-B transitions remain aggregate-only, and FinQA records retain conflicting metadata, unequal denominators, an empty manifest, and an incomplete worksheet.Source: §10 · p. 58

11. Conclusion

The conclusion presents destination-specific agent calibration as an integrated deployment specification, while emphasizing that exploratory observations do not establish its efficacy. Confirmatory validation remains outstanding.

11. Conclusion

  • Calibration specifies destination acceptance across operational capability, professional precision, and user context, using configurable evidence and reward instruments with consultant annotations.A versioned registry supplies applicable resources.Source: §11 · p. 59
  • Higher raw-target accuracy can coexist with source-correct regressions and zero incremental benefit from tested calibration rules; these observations do not establish efficacy.The proposed matched-budget experiments are needed to test calibration efficacy.Source: §11 · p. 59
  • Its bounded workflow maps requirements, diagnoses gaps, reconstructs the agent, and rechecks standards; adaptation can leave the foundation executor frozen and qualification uses separate hard gates.The objective is acceptable destination-context operation, not universal model ranking or source-format imitation.Source: §11 · p. 59
  • Industrial usefulness, training gains beyond initial calibration, and the greater-than-80% zero-shot task/context acceptance target still require matched-budget experiments.The proposal also calls for clean reload, complete-denominator reporting, and dependency revalidation.Source: §11 · p. 59
Loading 2609.35149v3…