Measure First, Then Allocate: Calibrated Rollout Budgets for MCP Agents in Reinforcement Learning

1 INTRODUCTION

• The paper treats harness evaluation and policy training as complementary routes, directing interaction budget toward outcomes that can affect retention or supply a group-relative learning signal.DISCORD concerns harness edits; BRACE allocates rollouts for the policy operating within the harness.Source: §1 · pp. 1–2
• DISCORD uses exact paired inference and decision-preserving stopping to retain harness edits only when observed disagreements support improvement.It evaluates edits without changing model weights.Source: §1 · p. 2
• BRACE combines bounded historical evidence, task-success uncertainty, and within-group dependence to forecast informative groups and select training tasks prospectively.The forecast guides generation before the next rollout group is sampled.Source: §1 · p. 2
• Across three model scales, BRACE reaches the strongest plotted baseline peaks with fewer new rollouts and transfers to BFCL and NESTFUL without further training.Historical-bank collection is counted separately.Source: §1 · p. 2

2.1 GROUP DEGENERACY IN RLVR

• When all rollout rewards agree, the centered GRPO reward advantage is zero for every rollout, regardless of within-group normalization.The condition is a degeneracy of the reward-based term, not necessarily of every update component.Source: §2.1 · p. 2
• Figure 2 compares predicted and observed inert-edit admission across AWM, BFCL, and two API-Bank tool surfaces, with residuals within ±0.024.The predictions use independently measured tie rates.Source: Figure 2 · p. 3
• 85.2% of 92,706 reconstructed groups were degenerate in a 482,510-rollout AWM bank snapshot spanning 1,493 verifier-scored tasks.The degeneracy concerns the reward-based policy-gradient term; other regularizers may still contribute.Source: §2.1 · p. 3

2.2 PAIRED OUTCOMES IN HARNESS EVOLUTION

• Paired harness evaluation controls for task difficulty; matching outcomes cancel, so only discordant pairs favor the incumbent or candidate.The discordant counts are candidate wins and incumbent wins on identical tasks.Source: §2.2 · p. 3
• Figure 3 reports that 18 of 21 paired harness-recovery loops favor DISCORD across four callers.The dashed line marks equal recovery.Source: Figure 3 · p. 4
• Table 1 classifies accepted live-AWM edits as good or bad by measured effect and reports realized objective and episode counts.The live setting withholds 10% or 40% of tool names; the full-surface objective is 0.2813.Source: Table 1 · p. 4, §2.2 · p. 4

3 DISCORD: EXACT HARNESS RETENTION

• DISCORD evaluates proposed harness edits with model weights fixed and retains a candidate only when paired outcomes provide sufficient evidence of improvement.DISCORD evaluates edits but does not generate proposals.Source: §3 · p. 4

3.1 WHY A POSITIVE MEASURED GAIN IS INSUFFICIENT

• When ties are rare, strict-improvement retention accepts nearly half of inert edits, so a positive measured gain alone is insufficient evidence.For inert edits, apparent improvements and regressions are equally likely under exchangeable scores.Source: §3.1 · p. 4

3.2 TESTING WHETHER THE CANDIDATE WINS RELIABLY

• Conditioning on discordant pairs, DISCORD retains a candidate when the exact one-sided probability is at most the prespecified α.With b = 12 and c = 4 on 100 paired tasks, the 8 pp improvement has probability 0.0384 and passes at α = 0.05.Source: §3.2 · p. 4
• Exact curtailment stops when every possible completion yields the same terminal decision, preserving the result full evaluation would return.Ambiguous comparisons may still use the full task budget.Source: §3.2 · pp. 4–5

4 BRACE: CORRELATION-AWARE ROLLOUT ALLOCATION

• BRACE selects tasks before generation to favor groups predicted to contain both successes and failures, while reserving uniform exploration and keeping group size fixed.The score targets the chance of a group-relative reward signal rather than the highest expected solve rate.Source: §4 · p. 5

4.1 ALLOCATION OBJECTIVE AND BANKED EVIDENCE

• Banked success and failure counts initialize task-level Beta evidence, distinguishing tasks before current-run rollouts; capped, discounted counts limit historical evidence as new outcomes arrive.The bank is read once, and every rollout used for a policy update is generated by the current policy.Source: §4.1 · p. 6

4.2 MODELING CORRELATED ROLLOUT GROUPS

• The exchangeable Beta–Binomial model represents within-group dependence through shared task difficulty and sampling conditions, separating group-level variation from posterior uncertainty in task success probability.It uses mean pi and concentration ν = (1 − ρ)/ρ, where ρ is within-group correlation.Source: §4.2 · p. 6
• With k = 5 and ν = 0.285 at pi = 0.5, the mixed-reward probability is 0.23 under correlation versus 0.94 under independence.The independent case is recovered as ν → ∞.Source: §4.2 · p. 6

4.3 RANK PRESERVATION UNDER CORRELATION CORRECTION

• At fixed k and shared ν, the correlation-corrected score preserves the task ordering induced by p(1 − p), but nonlinear score transformations can still change sampling weights.Posterior integration may also change rankings when tasks have different evidence counts; the proposition concerns fixed group sizes.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETUP

• Training uses Qwen3-VL-Instruct at 2B, 4B, and 8B on 1,127 verifier-scored tasks; evaluation covers 295 held-out AWM tasks and uses four seeds.BFCL and NESTFUL supply no AWM training data; the main AWM results use step 100.Source: §5.1 · pp. 6–7

5.2 DISCORD: HARNESS RETENTION

• Strict-improvement retention accepts 43.3% of 1,440 inert edits on AWM, and 43.6% of accepted edits fail to replicate.Independently measured tie rates predict inert-edit admission within ±0.024 across AWM, BFCL, and two API-Bank tool surfaces.Source: §5.2 · p. 7
• DISCORD recovers more handicap than the single-round gate in 18 of 21 paired loops; under harmful proposals, degradation occurs in 0/17 DISCORD seeds versus 9/17 strict-improvement seeds.The live MCP experiments find that DISCORD accepts no harmful edits in either regime but uses more evaluation episodes and reaches a lower final objective than the deployed rule.Source: §5.2 · pp. 7–8

5.4 GENERALIZATION

• BRACE leads plotted methods at each scale on BFCL and NESTFUL without further training; Figure 5 reports gains over untrained bases, not absolute rates.NESTFUL scores executed answers rather than exact call-sequence matches, and its prompt and parsing conventions support within-study comparisons only.Source: Figure 5 · p. 8, §5.4 · p. 8

5.3 HELD-OUT MCP PERFORMANCE

• 12.5%, 14.9%, and 16.5% solve rates let BRACE lead all baselines at 2B, 4B, and 8B, reaching strongest plotted peaks with 4,000, 8,000, and 4,000 new rollouts.The corresponding strongest-baseline peaks use 12,000, 16,000, and 12,000 new rollouts.Source: Table 2 · p. 7, §5.3 · p. 8

5.5 ABLATION STUDY

• Every component ablation lowers solve rates at all three scales; removing the bank costs up to 4.3 pp and removing correlation correction costs up to 4.6 pp.The full method has the highest solve rate at all three scales in Table 3.Source: §5.5 · p. 8, Table 3 · p. 9
• On 83 prospective arm-cycles, correlation-aware forecasts reduce mean absolute degeneracy-prediction error from 0.591 to 0.056.The forecast evaluation excludes cycles used to fit the correlation parameter; calibration tests forecasting, while held-out solve rates assess learning.Source: §5.5 · p. 8

• BRACE selects tasks for fixed-size groups using capped historical evidence, posterior uncertainty, and a correlated reward model, unlike selectors based on difficulty, replay, or predicted gradient variance.Its correlation correction improves forecast calibration while preserving known-probability task ranks at fixed group size.Source: §6 · p. 9
• DAPO filters all-equal groups after generation, whereas DISCORD uses exact conditional paired inference and decision-preserving curtailment to decide whether to retain harness edits.Harness-search systems propose edits to prompts, workflows, or agent code; DISCORD addresses the subsequent retention decision.Source: §6 · p. 9
• Beta–Binomial models of clustered binary outcomes motivate BRACE’s within-group forecast, while its task selection relates to active learning and adaptive replay.The allocation objective is non-degenerate group probability.Source: §6 · p. 9

• DISCORD applies exact paired inference and decision-preserving stopping to retain harness edits when observed disagreements support improvement; BRACE forecasts mixed-reward groups before allocating generation.The methods direct interaction budget toward harness comparisons that can change retention and rollout groups that can supply a group-relative training signal.Source: §7 · p. 9

• Real-world use requires appropriate permissions, execution isolation, monitoring, and safeguards beyond the reward verifier.The results do not establish equitable performance across users or application domains.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Appendix B specifies model and training settings, task splits, evaluation procedures, and checkpoint selection; Appendix D.2 documents historical evidence and initialization.Allocation forecasts are recorded before rollouts are generated, and paired tests use matched task indices.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

• BRACE forecasts groups with nonzero centered reward, not whether their gradients improve the policy; historical evidence also carries a separate collection cost.Cold-start or shifted task pools may provide less useful prior evidence, and other rewards, group sizes, and tool environments are untested.Source: A LIMITATIONS · p. 14
• DISCORD controls a prespecified paired comparison, not unrestricted adaptive edit sequences, and conservative retention can cost more or miss useful edits.The live MCP comparison illustrates this trade-off.Source: A LIMITATIONS · p. 14

B.1 MODELS, TRAINING DATA, AND OPTIMIZATION

• Training uses Qwen3-VL-Instruct at 2B, 4B, and 8B with GRPO on 1,127 tasks across 305 live servers; 295 held-out tasks are disjoint.The substrate is AWM with SQLite-backed MCP tools and a binary verifier.Source: §B.1 · p. 14, Table 4 · p. 14

B.2 BASELINES AND COMPARISON SCOPE

• The comparison covers uniform sampling, replay and curriculum selection, tool retrieval, post-generation filtering, and pre-generation allocation under matched model and budget settings.Current-run rollout budget is distinct from historical data acquisition and from groups ultimately used for optimization.Source: §B.2 · pp. 14–15

B.3 STATISTICAL REPORTING

• Paired solve-rate differences use candidate-only and reference-only wins over all task–seed pairs; exact conditional McNemar inference assumes independent pairs and an exchangeable paired null.Holm correction is applied within registered comparison families, and paired significance is reported separately from absolute performance.Source: §B.3 · p. 15

B.4 EVALUATION AND AGGREGATION ACROSS RUNS

• Repeated evaluations describe checkpoint-level stochasticity, not variability across independently trained policies; main results use final plotted checkpoints.Learning curves retain intermediate evaluations.Source: §B.4 · p. 15

B.5 TRANSFER BENCHMARKS AND PROCESSING

• Percentage-point changes on BFCL pass rate and NESTFUL win rate are measured relative to each scale’s untrained base model, with no benchmark training or task-selection feedback.The rates should be interpreted within each benchmark because their task structures and success criteria differ.Source: §B.5 · p. 15

C.1 INERT-EDIT ADMISSION IDENTITY

• Strict-improvement admission under exchangeability depends on the probability of a non-tie, and changing how ties are treated changes admission probability.Tie and admission rates are measured separately across AWM, BFCL, and API-Bank; the table gives values underlying Figure 2.Source: §C.1 · p. 16, Table 5 · p. 16

C.2 CALIBRATION AND EVALUATION-BUDGET PLANNING

• Permutation procedures remain calibrated in the displayed diagnostic, whereas both t-test variants exceed nominal null false-positive rates for three of four callers; pairing increases power.DISCORD’s error control follows from its exact conditional test under stated assumptions.Source: §C.2 · p. 16, Figure 6 · p. 17

C.3 EXACT CURTAILMENT

• Exact curtailment stops only when every feasible completion yields the same terminal decision, preserving the full-evaluation decision and its error control.Ambiguous comparisons can still require the full evaluation set.Source: §C.3 · p. 17

C.4 CONTROLLED EVOLUTION LOOPS

• DISCORD recovers more injected handicap in 18 of 21 paired comparisons and preserves performance in all 17 harmful-proposal matched seeds, versus degradation under strict-improvement retention in 9 of 17.The mean changes are small; the harmful-proposal result concerns frequency of degradation rather than a large mean-improvement difference.Source: §C.4 · p. 18

C.5 LIVE MCP COMPARISON

• The deployed live rule reaches a higher final objective with fewer evaluation episodes, but overstates it by 32% and 47% across the two regimes; DISCORD costs more to evaluate.The comparison does not show dominance on both terminal objective and reliability of reported improvement.Source: §C.5 · p. 18, Table 6 · p. 18

D.1 POSTERIOR EVIDENCE AND SAMPLING

• Posterior integration over task success probability distinguishes tasks with the same posterior mean but different evidence counts, particularly early in training.The implementation uses a capped historical bank at initialization and then updates evidence from current-run outcomes.Source: §D.1 · pp. 18–19

D.2 BANK PROVENANCE AND COST ACCOUNTING

• The bank reuses episodes from other experimental runs without a dedicated collection run, but historical generation, current-run generation, and the one-time bank read remain separate costs.A current allocation cycle requests 1,280 episodes.Source: §D.2 · p. 19

D.3 CORRELATION CORRECTION AND EXACT BETA MOMENTS

• BRACE models within-group dependence by drawing a latent task-level probability before conditional binary rewards, inducing correlation distinct from uncertainty about the task mean.The correlation is ρ = 1/(ν + 1).Source: §D.3 · p. 19

D.4 EXPECTED INFORMATIVE-GROUP YIELD

• Under the model, selecting tasks with the largest known informative-group probabilities maximizes expected informative groups for a fixed-size set; BRACE instead uses estimated scores, weighted sampling, and exploration.Eligibility checks determine which tasks can be selected, while posterior evidence ranks and weights eligible tasks.Source: §D.4 · p. 20, §D.5 · p. 20

E.1 PROSPECTIVE FORECAST EVALUATION

• The forecast diagnostic compares likelihood assumptions prospectively but does not establish improved task ordering or downstream policy quality.Those questions require separate rank analysis and held-out comparisons, respectively.Source: §E.1 · p. 20
• 0.591 versus 0.056 mean absolute error: the independent model’s forecast error exceeds the correlated model’s across the 83 evaluated arm-cycles.Figure 7 reports signed mean errors of −0.591 and +0.006, respectively; the diagnostic covers 21 arms.Source: §E.1 · p. 20, Figure 7 · p. 21

E.2 PROOF OF RANK PRESERVATION

• Both availability scores rise strictly with p(1 − p), establishing rank preservation across success probabilities on either side of 1/2.The proof uses symmetry around p = 1/2 and establishes strict monotonicity in the variance-like quantity u = p(1 − p).Source: §E.2 · p. 21