PaperlayerView exact revision on arXiv

Source-linked AI summary

What Confidence Routing Is Actually Doing: Auditing Routing, Calibration, and Commitment in Multi-Agent Deliberation

Jingyan Jiang, Huihuo Zheng, Rajeev Thakur, Chih-Hsuan Yang

arXiv:2609.27822v1cs.MAcs.CL

TL;DR

Confidence-routed deliberation uses one self-reported score both to select a speaker and to estimate uncertainty, although the public answer may differ from the poll answer. This paper audits routing, calibration, and commitment separately, finding their behavior varies across settings. It concludes that each must be measured independently before raw confidence guides deployment decisions.

  • Problem

    Confidence-routed designs use one verbalized score both to select who speaks and to estimate uncertainty, while poll and public answers can differ.

    Source: §1 · p. 1

  • Method

    The paper audits broadcast traces by measuring candidate routing, confidence ranking and probability calibration, and changes between selected poll answers and public speech.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Across the grid, raw confidence and commitment effects vary by setting, while fixed routers differ by at most 1.1 pp on gpt-oss/math.

    Source: §12 · p. 7

  • Takeaways & Limitations

    Routing discrimination, probability calibration, and public commitment should be audited separately because scale repair cannot create missing routing signal.

    Source: §12 · p. 7

    The audit examines one broadcast protocol, anchored on gpt-oss-120b math traces and a 2×2 extension, and does not establish profiles for all systems or actors.

    Source: §11 · p. 7

Abstract

from arXiv · show

A common multi-agent design asks agents to report confidence and lets the highest-scoring agent speak next, implicitly using one scalar both to route the conversation and to estimate uncertainty. We audit this confidence-routed broadcast protocol by separating three trace-level questions: whether it selects the right candidate (routing), whether reported confidence behaves like a probability (calibration), and whether the selected agent publicly states the answer that won the turn (commitment). Our primary study covers 4,181 gpt-oss-120b olympiad-math traces; we repeat the audit on a 2-by-2 actor-by-benchmark grid that adds gemma-4-31B-it and a biology multiple-choice benchmark. In the primary cell, confidence discriminates correct from wrong candidates (AUROC 0.72) but is strongly overconfident (79% mean stated confidence versus 52% accuracy). A cross-fitted, tier-stratified isotonic procedure reduces Expected Calibration Error from 0.278 to 0.008 on held-out candidates, but it does not recover missing discrimination: raw AUROC is only 0.537 and 0.440 in the two Gemma cells. Routing is likewise setting-dependent. Fixed routers differ by at most 1.1 percentage points on gpt-oss/math, whereas raw-confidence argmax performs 5.6 and 11.2 points below random-valid selection in the Gemma cells. Commitment is distinct again: in the primary cell, poll and spoken answers diverge in 20.4% of valid pairs, 62.4% of those revisions are fresh generations, and the unconditional correctness shift is -1.7 points; the other three cells instead range from +0.9 to +12.2 points. The transferable lesson is procedural: routing discrimination, probability calibration, and public commitment must be measured separately before raw confidence is used for deployment decisions.

1 Introduction

Confidence-routed deliberation uses one self-reported score both to choose who speaks and to estimate uncertainty, but the selected poll answer may differ from the public answer. The paper audits routing, calibration, and commitment as separate trace-level questions.

2 Protocol under audit

  • The poll candidate allocates the turn, but a separately generated free-form message supplies the answer that enters the public transcript.The selected agent's spoken answer can match, differ from, or be less extractable than its polled candidate.Source: §1 · p. 1, §2 · p. 2

1 Introduction

  • The audit separates routing, calibration, and commitment into distinct trace-level measurements with explicit denominators.It asks whether a correct candidate is selected, whether confidence ranks correctness and behaves like a probability, and whether the public answer preserves the polled answer.Source: §1 · p. 1, §2 · p. 2
  • Across settings, raw confidence can discriminate above chance yet remain overconfident, while its routing value varies by actor and task.The introduction reports held-out ECE falling from 0.278 to 0.008 in the primary cell, while confidence argmax can underperform random-valid selection in Gemma cells.Source: §1 · pp. 1–2
  • The primary audit covers gpt-oss-120b broadcast math traces and extends to a Gemma actor and biology benchmark, motivated by a 12.8× token cost premium over the single-agent comparator.The cost comparison is context, not a preference claim.Source: §1 · pp. 1–2

2 Protocol under audit

  • The protocol makes routing, confidence, and public commitment observable on the same broadcast trace, including whether the selected agent's answer changes at the speak step.All agents see the shared transcript; an approval sub-protocol can endorse a system incumbent between rounds.Source: §2 · p. 2

3 Metrics and audit machinery

The audit defines separate, validity-gated measurements for candidate selection, confidence quality, and poll-to-speech changes, with explicit denominators and dependence handling.

Metrics and audit machinery

  • Correctness-based measures apply only to extractable candidates, and the audit reports validity rates because extraction failures can otherwise confound routing and confidence assessments.Equivalence is checked with numeric and symbolic fast paths, then an LLM judge for residual cases.Source: §3 · p. 2
  • Routing measures include selected-candidate accuracy, hit rate when a correct candidate exists, missed-correct rate, within-block confidence alignment, and confidence tie rate.The audit retains CRA, HRWP, MCR, PCCA, and CTR; PCCA and AUROC use different stratifications.Source: §3 · p. 2
  • Calibration is assessed with ECE, Brier score, mean stated confidence versus accuracy, and AUROC, which distinguish probability scale from ranking ability.ECE compares confidence and empirical accuracy across equal-width bins; pooled calibration intervals use a problem-cluster bootstrap.Source: §3 · p. 2
  • Commitment measures compare valid selected poll–speak pairs, tracking divergence, whether changes help or hurt, and conditional versus unconditional correctness shifts.The conditional shift is averaged over divergent pairs; the unconditional shift is averaged over all valid pairs.Source: §3 · pp. 2–3, Figure 1 · p. 3
  • Residual equivalence cases are self-judged and cross-checked with two Gemma variants, but this does not test fast-path decisions or errors shared across judges.Across the three judge variants, conditional NPCS ranges from −0.083 to −0.060.Source: §3 · p. 3
  • Problem UID prevents calibration leakage across folds, while cluster ID defines resampling and dependence units for uncertainty estimates.Pooled confidence intervals use a bootstrap over 421 problem clusters; regressions use cluster-robust standard errors at that level.Source: §3 · p. 3

4 Experimental setting

The primary audit uses a fixed corpus of gpt-oss-120b broadcast math traces. In that setting, tested routing rules cluster closely, while the candidate pool limits attainable success.

4 Experimental setting

  • The corpus contains 4,181 problems and traces from 64 source collections, yielding 23,391 valid poll candidates and 9,436 valid selected poll–speak pairs.It spans ten math-reasoning difficulty tiers and 421 problem clusters; labels were extracted from answers and checked against verifier or dataset references.Source: §4 · p. 3

5 Routing differences are small in this setting

  • The router comparison is specific to a homogeneous pool of three gpt-oss peers at temperature 0, with mixed-correctness blocks comprising about 1,184 of 11,122 selected blocks.The passage cautions that this within-pool similarity does not generalize across actors; confidence can be harmful for routing on a different actor.Source: §5 · p. 4
  • A learned cross-fitted logistic router matches argmax at 49.9% despite higher candidate-level AUROC, because its strongest feature is constant within each block.The model's AUROC is 0.806 versus 0.721 for the raw score; the round index cannot break within-block ties among candidates.Source: §5 · p. 4
  • In 1,184 mixed-correctness blocks, PCCA is 0.570 against a 0.500 no-signal reference, while selected-invalid rate falls from 14.9% overall to 0.7% when a correct alternative exists.The separate missed-correct rate is 5.8% over 7,970 eligible blocks.Source: §5 · p. 4
  • Five fixed routers fall within 1.1 pp on 9,653 all-routeable gpt-oss/math blocks, while the oracle ceiling is 4.8–5.9 pp above them.The learned router lands in the same band; the table describes counterfactual router performance on all-routeable blocks.Source: Table 1 · p. 4, §5 · p. 4

6 Confidence is a poor probability, but the gap is cheaply fixable

Confidence ranks candidates better than it estimates their probability of correctness, and held-out recalibration improves the probability scale without restoring discrimination. Public answers can diverge from selected poll answers, with the direction and source of revisions varying across settings and revision types.

6 Confidence is a poor probability, but the gap is cheaply fixable

  • AUROC is 0.721, but mean stated confidence is 0.790 against 0.518 accuracy, a 27.2 pp overconfidence gap.The values are for 23,391 valid candidates in the primary cell.Source: §6 · p. 4
  • ECE rises from 0.10 on tier 01 to 0.31–0.41 on tiers 06–10, while rank signal weakens more gradually as confidence flattens on harder problems.The pooled confidence-correctness regression has a strongly negative score-by-difficulty interaction.Source: §6 · p. 4
  • Cross-fitted isotonic regression maps stated confidence to empirical accuracy with a nondecreasing function, using problem_uid folds stratified by tier.No question appears in both the fit and evaluation data; the map can merge score levels but cannot reverse a strict ordering.Source: §6 · p. 4
  • ECE falls from 0.278 to 0.008 and Brier score from 0.302 to 0.208 after isotonic calibration, while candidate-level AUROC changes only from 0.721 to 0.720.These are held-out candidate-level estimates, not a rerun of the router on calibrated values.Source: Table 2 · p. 5, §6 · p. 5
  • At a 0.7 threshold, raw confidence retains 75% of candidates at 59% accuracy, while isotonic-calibrated confidence covers 14% at 91% accuracy.This is a descriptive risk–coverage sweep, not evidence of an end-to-end threshold policy effect.Source: §6 · p. 5

7 The poll is not a public commitment, and many revisions are fresh regenerations

  • 20.4% of valid selected poll-speak pairs diverge in gpt-oss/math, with a −1.7 pp unconditional correctness shift; the other three cells range from +0.9 to +12.2 pp.Under conservative NaN-as-divergent treatment, divergence is 20–27% across the three cells.Source: §7 · p. 5
  • 62.4% of divergent spoken answers are novel generations matching no candidate visible in prior context, and this class contributes most aggregate harm.Novel revisions have a −9.1 pp per-pair shift; self-reverts are more negative per pair at −11.4 pp.Source: §7 · p. 5, Table 3 · p. 5
  • Pool-solvable rate has the largest same-trace association with solving, while mean selected confidence is small and nonsignificant, with sensitivity-refit signs unstable.The outcome logit is exploratory and observational.Source: Table 4 · p. 6

8 Outcome linkage: what best predicts broadcast success?

Pool-solvable rate has the largest association with broadcast success in the outcome model. Across actors and benchmarks, confidence calibration, routing performance, and commitment effects vary by setting.

8 Outcome linkage: what best predicts broadcast success?

  • OR ≈121 (95% CI [32, 456]) makes pool-solvable rate the largest coefficient, descriptively linking successful traces to more correct polled candidates.Pool-solvable rate is the fraction of poll blocks in an evolving trace that contain at least one valid correct polled candidate.Source: §8 · p. 6
  • The unconditional improvement-event rate is positively associated with outcome (OR ≈3.7), while divergence and degradation-event rates are imprecise.Source: §8 · p. 6
  • Mean selected confidence has OR 0.84, p = 0.35, and sensitivity OR 0.84–1.11; same-trace, non-preregistered coefficients are descriptive and hypothesis-generating.The sign of the mean-confidence association is not stable across sensitivity specifications.Source: §8 · p. 6

9 Ablation: cross-actor and cross-benchmark

  • Overconfidence recurs in all four cells and isotonic calibration lowers ECE to ≤0.023, while raw AUROC falls to 0.537 and 0.440 in Gemma.The scale correction does not establish a common ranking signal.Source: §9 · p. 6
  • Argmax scores 0.399 versus 0.455 and 0.362 versus 0.474 in Gemma; commitment shifts are −1.7 pp on math and +0.9 to +12.2 pp elsewhere.The grid’s decomposition generalizes, but the direction and practical importance of each effect must be measured in the deployment setting.Source: Table 5 · p. 6, §9 · p. 7

10 Discussion

The discussion finds that routing, calibration, and commitment behave differently across settings, so they should be measured separately. It recommends basing design decisions on those measurements while treating the audit’s scope and comparisons cautiously.

10 Discussion

  • In gpt-oss/math, fixed-router counterfactuals show little selection-rule headroom, while successful traces are associated with frequent availability of correct polled candidates.The same-task broadcast pass rate is 0.892 versus 0.788 for the available single-agent comparator, but broadcast uses 12.8× tokens, so this is not an equal-compute preference claim.Source: §10 · p. 7
  • The discussion recommends measuring candidate availability and selection headroom, testing within-block selection, recalibrating scores, and evaluating commitment scaffolding as an intervention.It cautions that pooled AUROC alone is insufficient for routing and that local poll-to-speak shifts do not establish system-level effects.Source: §10 · p. 7

11 Limitations

  • This is a trace audit of one broadcast protocol, anchored on gpt-oss-120b math and a 2×2 grid, not a claim that all actors or topologies share its profile.The direction of the commitment effect varies across cells.Source: §11 · p. 7
  • No commitment-lock ablation, calibrated-router rerun, or equal-compute single-agent baseline is run; the available comparators therefore provide context, not a matched-cost frontier.The audit’s equivalence judge is gpt-oss-120b in a self-judge configuration, and the novel-versus-borrow split depends on excluding peer polls from speaker context.Source: §11 · p. 7
  • A commitment lock’s end-to-end effect cannot be inferred from local shifts because changing a public message would alter later transcripts and generations.The outcome analysis also depends on the audited visibility model, and training-data overlap is unmeasured.Source: §11 · p. 7

12 Conclusion

  • Held-out ECE falls from 0.278 to 0.008, yet raw-confidence argmax trails random-valid selection by 5.6 and 11.2 pp in Gemma cells, while commitment shifts range from −1.7 to +12.2 pp.The conclusion also reports that fixed routers differ by at most 1.1 pp on gpt-oss/math.Source: §12 · p. 7

Technical Appendix

The appendix documents headline denominators and worked examples, then details protocol estimands, robustness choices, and limits on interpreting local commitment shifts. Its analyses distinguish observed associations from causal or system-level effects.

A Headline denominators and a worked trace

  • Table 6 reports headline metrics with explicit denominators and problem-cluster bootstrap 95% confidence intervals, including conditional and unconditional commitment measures.Figure 2 traces a triple-tied routing decision in which the polled answer differs from the selected agent’s spoken answer.Source: Table 6 · p. 9, Figure 2 · p. 9
  • Figure 2 instantiates the block, pair, candidate, and pool abstractions on one trace, where first-seen tie-breaking selects A1 despite a different spoken answer.The same trace later succeeds in round 1.Source: Figure 2 · p. 9

B Protocol and analysis details

  • The protocol defines valid selected poll–speak pairs and divergent pairs, while outcome regressions use unconditional event rates and a separate block-level pool-solvable predictor.Pool-solvability depends on candidate content, not which candidate the router selected.Source: B Protocol and analysis details · pp. 9–10
  • The system incumbent persists until an approval event updates it; incumbent-pull is the only divergence class with a non-negative point estimate, but endorsement is not shown to cause it.The reported point estimate is +1.7 pp; broadcast variants without a tracked incumbent will not exhibit this class.Source: B Protocol and analysis details · p. 10
  • Difficulty tiers are ordinal rather than equal-interval, and regression p-values are Wald-asymptotic without CR2/CR3 refits, multiplicity adjustment, or preregistered predictors.The authors emphasize effect sizes and uncertainty intervals for headline descriptive comparisons and treat marginal regression p-values as hypothesis-generating.Source: B Protocol and analysis details · p. 10
  • The adoption logit is observational: source-poll confidence is analyst metadata, not confidence shown to or acted on by the speaker.It does not establish that confidence caused adoption.Source: B Protocol and analysis details · p. 10
  • The −1.7 pp local commitment shift is not an estimate or upper bound on a lock’s end-to-end effect, which requires an intervention or rerun.Replacing a public message can change the later transcript, generations, and final answer; positive shifts in other cells caution against a blanket lock.Source: B Protocol and analysis details · p. 10

C Notation: metrics referenced throughout the paper

This appendix section indexes metric abbreviations and supporting analyses, then reports per-tier descriptive patterns and points to regression tables. The per-tier view complements pooled headline estimates rather than replacing them.

C Notation: metrics referenced throughout the paper

D Per-tier descriptive metrics

  • Table 10 supplies per-tier routing, ranking, commitment, and calibration metrics as a complement to pooled estimates and trends underlying the calibration regression.Tiers 03 and 10 have small denominators, and Figure 3 highlights that calibration worsens faster than rank signal across harder tiers.Source: D Per-tier descriptive metrics · p. 10, Figure 3 · p. 11

E Regression highlights

  • Table 7 summarizes the audited runtime protocol surface, whose structured private poll and free-form public speak stages underlie the poll/speak commitment gap.The stated asymmetry is that the poll is private to the router, while the speak step is public.Source: Table 7 · p. 11

D Per-tier descriptive metrics

  • Figure 3 shows PCCA above the 0.5 no-signal floor across most tiers but weakening toward harder slices, while ECE reaches 0.31–0.41 on tiers 06–10.Tiers 03 and 10 are marked for small denominators; the curves are not asserted to be perfectly monotone.Source: Figure 3 · p. 11

F Robustness checks for the main claims

The appendix checks synthetic routing behavior, judge sensitivity, and outcome-model robustness. These checks qualify the audit’s metrics and inferences without making the observational analyses causal.

G Sanity-check constructions

  • Within-block confidence shuffling drives PCCA to its no-signal floor, while noisy-oracle scores drive PCCA toward 1 and CTR toward 0; CRA remains limited by candidate content.These constructions use the same trace data without rerunning deliberations.Source: G Sanity-check constructions · p. 11

H LLM-judge sensitivity audit (two-judge cross-check)

  • Both Gemma judges label at least 90% of residual pairs, and agreement with gpt-oss is substantial, but correlated judge bias and lack of human adjudication prevent identifying ground truth.On the stratified sample, the Gemma judges agree with each other at 94.4%.Source: H LLM-judge sensitivity audit… · p. 11
  • Conditional NPCS remains negative across judge variants (−0.083, −0.061, and −0.060), although divergent-pair denominators and magnitudes vary.Table 17 uses tiers 02–09 rather than the full headline population, and judge parse failures are conservatively treated as divergent.Source: H LLM-judge sensitivity audit… · p. 11

I Outcome-linkage decomposition: pool-solvable rate split into router-picked and routing-slack pieces

  • The outcome-linkage decomposition separates router-picked correctness from remaining routing slack using separately standardized predictors, so it is diagnostic rather than algebraically equivalent after standardization.Pool-solvable rate is the per-trace mean share of blocks containing any valid correct candidate.Source: I Outcome-linkage decomposition:… · p. 12
  • Pool-solvable rate nearly separates the modeled outcome, so Firth and dropped-predictor refits check sensitivity; improvement-event estimates stay positive while mean selected-confidence estimates span both sides of 1.The improvement-event odds ratios are 3.5 and 9.1 across the alternative specifications; selected-confidence OR spans 0.84–1.11.Source: I Outcome-linkage decomposition:… · p. 12
  • Table 10 marks tiers 03 and 10 as small-denominator cases, and Figure 4 shows the dominant residual-pair disagreement direction without determining which judge’s label is correct.The marked tiers each contain two problem clusters.Source: Table 10 · p. 13, Figure 4 · p. 13

J Cross-actor and cross-benchmark replication

The replication crosses two deliberator actors with math and biology benchmarks, using a common extraction pipeline and cell-specific calibration. Across the grid, calibration repairs probability scale more consistently than confidence supplies useful ranking or routing.

J Cross-actor and cross-benchmark replication

  • The 2 × 2 grid crosses gpt-oss-120b and Gemma 4-31B-it with math and LAB-Bench biology, while holding the equivalence judge fixed.The preprint gives descriptive point estimates but no per-cell confidence intervals or underlying analysis files.Source: J Cross-actor and cross-benchmark… · p. 13
  • Across cells, raw confidence is overconfident, and cell-specific cross-fitted isotonic calibration reduces held-out ECE to ≤0.023 without supplying missing ranking discrimination.Raw AUROC is 0.537 on Gemma/biology and 0.440 on Gemma/math; calibrated AUROC is 0.540 and 0.530, respectively.Source: J Cross-actor and cross-benchmark… · p. 13
  • Raw-confidence argmax is nearly tied with random-valid selection on gpt-oss/math but trails it on the other cells, including by 5.6 and 11.2 pp on Gemma biology and math.The oracle ceilings show that candidate availability and selection quality can both constrain performance; per-cell significance intervals are not reported.Source: J Cross-actor and cross-benchmark… · p. 13
  • Poll–speak divergence recurs across cells, but its unconditional correctness shift varies from −1.7 pp on gpt-oss/math to positive estimates of +0.9 to +12.2 pp elsewhere.Revision is therefore not uniformly harmful in these point estimates.Source: J Cross-actor and cross-benchmark… · p. 13

K Task-level single-agent context

The appendix places the audited broadcast protocol alongside same-task and cross-setting comparisons, then reports robustness and scope checks. It also notes that training-data overlap was not measured, limiting interpretation of tier-stratified calibration patterns.

K Task-level single-agent context

  • The appendix reports cluster-robust pooled regressions and same-trace descriptive specifications, while explicitly distinguishing these associations from intervention estimates.The correctness and divergence models use 23,391 valid poll candidates and 9,436 selected poll-speak pairs; source confidence was not visible to speakers.Source: Table 11 · p. 14, Table 12 · p. 14, Table 14 · p. 15
  • Peer answers match 28.5% of divergences, but only 12.4% count as primary peer-echo after precedence rules; the other 312 also match the speaker’s earlier answer or incumbent.Self-revert and novel flag counts are unchanged by precedence, bounding peer anchoring under a maximally permissive attribution rule.Source: Table 13 · p. 14
  • The permutation null places runtime PCCA above its no-signal floor, while a noisy oracle shows PCCA saturating under a clean signal.CRA remains capped because selected-valid accuracy cannot exceed the underlying content ceiling.Source: Table 15 · p. 15
  • When gpt-oss flags divergence, Gemma judges more often classify the pair as equivalent (311 / 433) than the reverse disagreement occurs (62 / 68).Both Gemma judges show this asymmetric disagreement pattern.Source: Table 16 · p. 15
  • Conditional NPCS remains negative across three residual-case label variants, a sensitivity check rather than ground-truth adjudication.The variants use the same 9,263 valid pairs from tiers 02–09.Source: Table 17 · p. 16
  • Table 19 reports descriptive point estimates across the 2 × 2 actor×benchmark grid, with the primary cell first and no per-cell confidence intervals reproduced.Routing rows use per-block success over all-routeable blocks; net commitment shift is the unconditional poll-to-speak correctness change in percentage points.Source: Table 19 · p. 16
  • Broadcast reached a 0.892 final pass rate versus 0.788 for single_agent, at 12.8× tokens, 7.5× model calls, and 4.0× wall time.These are weighted aggregates from the same 4,181-problem evaluation, not equal-compute frontiers.Source: Table 20 · p. 16

L Future work: source-stratified contamination probe

  • Training-data overlap was not measured, so rising ECE across difficulty tiers may reflect intrinsic difficulty, uneven pre-training coverage, or both.The proposed post-cutoff, low-public-overlap audit is outside this study.Source: L Future work: source-stratified… · p. 17

M Analysis and material availability

The paper distinguishes analysis materials from materials available in the arXiv package and introduces a trace-audit card for reporting protocol measurement properties. The card records protocol configuration, headline metrics, denominators, and unavailable information rather than implying that missing artifacts are available.

M Analysis and material availability

  • The arXiv package omits trace data, benchmark text, runtime code, prompts, per-cell bootstrap files, and an artifact locator; separate redistribution requires review of source licenses and participant, coauthor, and institutional concerns.Aggregate compute totals are reported, but per-example accounting records are not included.Source: M Analysis and material availability · p. 17

N Trace-Audit Card v0.1

  • The trace-audit card reports a deliberation protocol’s measurement properties, rather than describing a single model or dataset.Its required fields are minimum reporting, and unavailable prompts, counts, or uncertainty estimates must be stated.Source: N Trace-Audit Card v0.1 · p. 17, Table 21 · p. 18
  • Card guidance calls for explicit denominators, unavailable-item labels, per-model calibration for heterogeneous pools, and adapted routing and visibility fields.Peer-poll visibility should be stated, as should each agent model in a heterogeneous pool.Source: N Trace-Audit Card v0.1 · p. 17
  • The recorded protocol uses three homogeneous openai/gpt-oss-120b peers, confidence_argmax with a deterministic first-seen tie rule, and public broadcast visibility.It specifies up to three outer rounds, a [1,100] confidence field normalized to [0,1], and notes that verbatim prompt templates are not distributed.Source: N Trace-Audit Card v0.1 · p. 19
  • The card requires an extractable boxed or numeric answer for validity and checks equivalence through numeric and symbolic fast paths, followed by an openai/gpt-oss-120b judge cross-check.The Gemma cross-check reports κ=0.82 (n=4048 decided pairs).Source: N Trace-Audit Card v0.1 · p. 19
  • The card reports routing.cra as 0.506 (n=9467), routing.hrwp as 0.906 (n=5295), routing.mcr as 0.058 (n=7970 eligible blocks), and routing.pcca as 0.570 (block-macro; 1184 blocks).The filled card marks prompt, source-count, and overlap information not distributed in the arXiv package; headline metrics have explicit denominators.Source: N Trace-Audit Card v0.1 · p. 19, Figure 5 · p. 19
  • The corpus comprises 4,181 traces, 421 clusters, and 64 sources, while train_data_overlap was not evaluated and complete per-source counts are unavailable.The two largest listed sources are HMMT_2=1321 and HMMT_11=860.Source: N Trace-Audit Card v0.1 · p. 19

Related papers

Loading 2609.27822v1…