Source-linked AI summary
Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation
Xiaofei Feng
TL;DR
The paper asks whether correctness-gated multi-teacher distillation improves decisions without conflating decision correctness with rationale grounding. In a fixed multi-arm experiment, it finds decision redistribution and lost label functionality, while the available human grounding audit remains inconclusive.
Problem
Decision correctness and rationale grounding are distinct objectives, so aggregate decision metrics may not establish grounded reasoning.
Method
The study performs a controlled fixed-matrix comparison of correctness-weighted and unfiltered multi-teacher distillation with matched decision inputs, per-label checks, and a human grounding audit.
Results
The weighted arm improved accuracy and macro-F1 versus unfiltered MTD, but lost label functionality, while source-label SFT had the highest mean macro-F1.
Takeaways & Limitations
The fixed experiment supports interpreting the outcome as decision redistribution rather than grounding improvement or a demonstrated incremental benefit over hard filtering.
Takeaways & Limitations
One student, domain, teacher pool, decoder, held-out set, and three seeds, together with a non-paired amended audit, preclude population or scaling claims.
Abstract
from arXiv · showhide
Candidate decision correctness and rationale grounding are different objectives. We examine correctness-gated multi-teacher distillation in a fixed experiment. Eight arms share 4,330 sources, a 63.9M-parameter student, 12,990 optimization rows, 406 updates, evidence inputs, and a decoder; seven teacher-based arms use one fixed three-response pool. Three seeds are evaluated on 267 held-out examples. Relative to unfiltered distillation, the correctness-weighted arm differed in accuracy by +0.1660 (95% observed-matrix interval [0.0670, 0.2455]), five-label macro-F1 by +0.1323 ([0.0916, 0.1731]), and task-defined conditional unsafe-action rate by -0.4979 ([-0.5926, -0.3686]). These shifts do not imply uniformly better behavior. Source-label SFT had the highest mean macro-F1 (0.586). The weighted arm had zero Refuted recall in every seed, and two seeds assigned NotEnoughInfo to all 167 claim examples. In an availability-amended audit at one reference seed, weighted and unfiltered outputs had 0/20 versus 1/20 evidence-supported positives and 20/20 versus 19/20 positives containing unsupported material. Samples were non-paired, source overlap was not serialized, and the amendment followed automatic summarization but preceded annotation. The audit therefore cannot estimate a common-source grounding effect and is inconclusive about system-level improvement or harm. Hard filtering already achieved 0.660 accuracy, 0.530 macro-F1, and 0.135 conditional unsafe rate. The implemented weighted arm showed no demonstrated incremental decision benefit over hard filtering. This fixed-matrix failure analysis shows decision redistribution with lost label functionality; the available human audit does not establish a grounding gain.
1 Introduction
The paper distinguishes candidate decision correctness from rationale grounding and tests this distinction in a controlled correctness-gated multi-teacher distillation experiment. It frames the study as a bounded failure analysis rather than a generally superior method.
- Motivation: Evidence-grounded decisions require Supported or Refuted labels only when evidence licenses that polarity; otherwise the system should output NotEnoughInfo.
- Approach: The experiment compares correctness-gated candidate selection and weighting with seven supervision arms around a fixed three-response teacher pool.
- Research questions: The study asks whether decision-endpoint shifts coexist with functional recall for every label and evidence-supported positive outputs.
- Contribution: The contribution is a controlled fixed-matrix failure analysis combining matched hard-versus-weighted comparisons, per-label checks, and a system-label-masked human audit.
- Scope: The conclusions are bounded to one student, preserved teacher pool, English restaurant-evidence domain, held-out set, and decision policy.
- Objective alignment: Decision-correctness gates may improve decision targets without improving accompanying language-model targets such as rationale faithfulness or evidence support.
3 Task, Data, and Teacher Materials
The experiment evaluates evidence-grounded claims and recommendations over fixed evidence inputs, using three claim labels and two recommendation decisions. Its teacher materials combine human-gold, weak or silver sources and an archived, non-regenerated response pool.
- Tasks and labels: Claim verification uses Supported, Refuted, and NotEnoughInfo, while recommendation decisions use Answer and Abstain.
- Task setup: Each positive output contains a decision, answer or verdict, rationale, and cited evidence identifiers, while retrieval and evidence inputs remain fixed.
- Evaluation: The conditional unsafe-action rate counts Supported or Refuted on gold-NotEnoughInfo claims and Answer on gold-Abstain queries, not general harmful language.
- Data: The held-out test contains 167 claims and 100 queries, with 44 Supported, 34 Refuted, 89 NotEnoughInfo, 33 Answer, and 67 Abstain labels.
- Training sources: All arms share 4,330 training sources, including 353 human-gold and 3,977 weak or silver sources that are not equivalent to human annotation.
- Teacher materials: The 12,990 teacher candidates come from three fixed diagonal responses per source, and later metadata changes did not regenerate teacher-visible messages.
4 Correctness-Gated Supervision
A composite gate scores candidate correctness, filters candidates, and assigns retained-set supervision weights to the student. The hard and weighted arms share selected candidates and materialized decisions but differ in soft targets and conversations.
- Gate: The gate labels a candidate positive when it is structurally valid and its decision equals the adjudicated human training label.
- Gate specification: The no-role gate uses grouped five-fold logistic regression, but its broader-group dependence makes q an engineering score rather than an out-of-cluster estimate.
- Filtering: Thresholding retains candidates with qij ≥ τ, while empty retained sets fall back to the highest-scoring valid candidate.
- Weighting: The hard arm assigns equal within-family mass, whereas the weighted arm uses scores to allocate retained candidate mass.
- Retained supervision: The hard and weighted arms retain 8,578/12,990 candidates, including 321 deterministic best-candidate fallbacks, with no downstream rationale fallbacks.
- Matched comparison: Selected IDs and materialized decisions match for 4,330/4,330 families, but soft targets match for 4,324/4,330 and conversations for 4,202/4,330.
- Interpretation: Because retained candidates are matched but some soft targets and conversations differ, the comparison does not isolate a scalar-weight-only effect.
5 Experimental Design and Analysis
The experiment fixes the training matrix and evaluates decision endpoints, descriptive uncertainty, and a separate human grounding audit. Its analysis explicitly limits interpretation because the audit is non-paired and the observed-matrix intervals do not generalize to future seeds or settings.
- Experimental matrix: 12,990 physical rows, 406 optimizer updates, and a common matrix define the experiment, but objective families are not token- or compute-matched.The student architecture and training configuration are fixed across arms, while target lengths and supervised language-model exposure differ across source-label, answer-only, and rationale objectives.
- Outcomes: The three decision summaries are accuracy, five-label macro-F1, and task-defined pooled conditional unsafe-action rate, treated as descriptive co-primary endpoints without multiplicity adjustment.Macro-F1 assigns zero F1 to a label with no predicted positives.
- Human grounding audit: The availability-amended human audit evaluates answer support, rationale faithfulness, unsupported material, and citation validity among emitted positives at one seed.The audit reports raw counts because its system-specific positive samples are small and non-paired.
- Statistical analysis: The primary contrast is the correctness-weighted arm minus uniform unfiltered multi-teacher distillation, estimated from paired seed-specific differences across 10,000 crossed-bootstrap replicates.The procedure resamples three observed seeds and 22 leakage groups, reconstructs paired rows, and reports the 2.5th and 97.5th percentiles.
- Interpretation limits: Observed-matrix intervals summarize the current three-seed by 22-group matrix and are not calibrated confidence intervals for future seeds, model families, or domains.Secondary arm comparisons are likewise descriptive rather than a multiplicity-adjusted confirmatory family.
- Human grounding audit: The audit cannot estimate a common-source contrast or establish system-level improvement or harm because it is post-summary, one-seed, and non-paired.Source overlap was not serialized, and the audit amendment occurred after automatic summarization but before annotation.
6 Results
Correctness weighting improved aggregate decision endpoints relative to unfiltered MTD, but redistributed decisions toward abstention and did not demonstrate an incremental benefit over hard filtering. Per-label functionality was lost, while the non-paired human grounding audit remained inconclusive.
- 6.1 Primary decision contrast: 0.135 conditional unsafe-action rate for correctness-weighted MTD versus 0.632 for unfiltered MTD.The observed accuracy values were 0.634 and 0.468, respectively.
- 6.1 Primary decision contrast: 0.586 was the highest mean five-label macro-F1, achieved by source-label SFT, so rationale-bearing supervision was not monotonically beneficial in this matrix.Relative to SFT, single-teacher rationale KD differed in macro-F1 by −0.2460, with 95% observed-matrix interval [−0.3386, −0.1889].
- 6.1 Primary decision contrast: +0.1660 accuracy difference for correctness-weighted MTD minus unfiltered MTD, with 95% observed-matrix interval [0.0670, 0.2455].The corresponding five-label macro-F1 difference was +0.1323, with interval [0.0916, 0.1731].
- 6.2 Primary and component contrasts: 0.660 accuracy for hard filtering, compared with 0.634 for correctness weighting; the weighted arm showed no demonstrated incremental decision benefit over hard filtering.Hard filtering also had 0.530 macro-F1 and 0.135 conditional unsafe-action rate.
- 6.3 Per-label behavior at the fixed operating point: 0 Refuted recall occurred in every weighted seed, while two seeds assigned NotEnoughInfo to all 167 claim examples.Weighted recall was 0.030 for Supported, 0 for Refuted, 0.955 for NotEnoughInfo, 1.000 for Answer, and 0.746 for Abstain.
- 6.4 Inconclusive human grounding audit: 0/20 weighted positives were evidence-supported versus 1/20 unfiltered positives, and 20/20 weighted positives contained unsupported material versus 19/20 unfiltered positives.The audit used one reference seed and non-paired samples; it cannot establish a population-level grounding benefit or harm.
7 Discussion
The weighted arm redistributed decisions while losing label functionality, and the non-paired audit cannot establish a grounding effect. Hard filtering already showed the aggregate pattern, so the weighted arm has no demonstrated incremental decision benefit.
- Refuted recall was zero in every seed, while two seeds emitted no positive claim decisions.These findings accompany the weighted arm’s decision shifts but indicate lost label functionality.
- The non-paired audit precludes a grounding-effect claim in either direction.The evidence does not isolate a common-source grounding effect.
- Hard filtering already exhibited the aggregate pattern, while weighting changed mass and 128/4,330 conversations despite matched IDs and decisions.The passage states that causal isolation therefore requires independent manipulation.
8 Limitations
The study’s conclusions are bounded by a single student, domain, teacher pool, decoder, 267 examples, and three seeds. Additional limits affect causal, scaling, and retraining claims.
- One student, domain, teacher pool, decoder, 267 examples, and three seeds preclude population or scaling claims.
- Broader-group dependence, weak or silver provenance, runtime and phase gaps, and threshold sensitivity further limit causal and retraining claims.
- The amended non-paired audit further limits causal and retraining claims.
9 Ethics and Responsible Research
The study recognizes harms from unsupported or stale claims in restaurant reviews, while its narrow unsafe metric omits several relevant risks. Human-subject documentation is also incomplete.
- Unsupported or stale claims in restaurant reviews can remain harmful despite separating restricted text.
- The narrow unsafe metric omits harmful language, fairness, privacy, retrieval error, and downstream consequences.
- Recruitment, compensation, consent, privacy, and institutional determination for the human corpus and audit are unrecorded.The authors make no claim, and state that this blocks submission.
10 Reproducibility and Data Availability
An internal record reproduced aggregate arm-seed summaries and contrasts, while the source package supports aggregate inspection rather than row-level recomputation. The reported conclusion remains decision redistribution with lost label functionality.
- Aggregate arm-seed summaries and contrasts were reproduced within 10^-12, but training was not reproduced.
- The source package includes code, figures, bibliography, aggregate audit, and protocols, but excludes predictions, text, labels, raw reviews, identity mappings, and weights.
- The package supports aggregate inspection, not row-level recomputation.
- The weighted arm shifted decisions but lost label functionality, while the hard arm already showed the pattern without a weighted increment.
A Study Units and Label Distributions
The study uses separated human-gold and weak/silver sources, fixed teacher-generation and gating procedures, and multiple distillation arms. Diagnostics document substantial differences in automatic support and role-feature effects across the evaluated configurations.
- Study units and labels: The student pool contains 4,330 sources: 353 human-gold and 3,977 weak/silver sources, including 3,598 claim and 732 query sources.Reference-label totals are 1,294 Supported, 1,122 Refuted, 1,182 NotEnoughInfo, 348 Answer, and 384 Abstain.
- Study units and labels: Teacher generation uses one recommendation, one verification, and one abstention model with fixed seeds, temperatures, token caps, and task-restricted JSON requests.Serialized requests include task, claim or query, evidence and identifiers, time bin, entity, role, and evidence-only instructions.
- Study units and labels: The gate uses 16 no-role features and five source-grouped folds, while retained candidates are materialized into hard and weighted training sets with identical decision totals.The supplied passage reports the feature design and matching decision totals but truncates the full totals after NotEnoughInfo.
- Diagnostics: Automatic support is 0.304 for unfiltered outputs and 0.034 for weighted outputs, conditioned on each arm’s emitted positives.The supplied passage reports this comparison as an automatic-support diagnostic.
- Diagnostics: Explicit-role minus no-role weighted changes student macro-F1 by +0.0033 and accuracy by 0, while the reported unsafe-action change is +0.0021 with intervals spanning zero.Model identity remains a role proxy, so this is an explicit-role-feature ablation rather than complete role removal.
C Complete Contrast Ledger
The contrast ledger defines six declared comparisons and a controlled audit procedure, while recomputation verifies the reported numerical results. Several upstream and legacy-metric boundaries limit what the audit and manuscript can establish.
- Declared contrasts: The six declared comparisons cover teacher rationale transfer, unfiltered MTD, hard filtering, correctness weighting, role-aware weighting, and the primary weighted-versus-unfiltered contrast.Each comparison uses 10,000 crossed seed-by-group replicates with a fixed recorded random-number seed.
- Audit design: The audit samples positives and safe controls within variant and stratum using stable SHA-256 ordering without replacement, with 160 positives and 154 safe rows.Gold labels and automatic scores do not determine eligibility or order.
- Audit design: Annotators judge support, rationale faithfulness, unsupported material, and citation validity using only the displayed task, evidence, decision, answer, rationale, and citations.A rationale is faithful only when every material proposition is evidence-supported and decision-consistent.
- Audit design: System variant, training seed, gold label, and automatic scores are hidden from two independent annotators and a disagreement-only adjudicator.Agreement statistics use all 314 paired pre-adjudication rows.
- Numerical audit: The internal recomputation reproduces all 24 arm-seed summaries, six declared contrast jobs, and 906 comparable parsed result leaves within 10^-12.Of 906 parsed values, 889 are exactly equal and 17 differ only within floating-point tolerance.
- Limitations: The audit validates the current observed inputs-to-report path rather than the complete upstream custody chain because teacher generation, gate fitting, supervision construction, and student training were not rerun.Two historical Windows manifests are unavailable.
- Limitations: Pooled F1 and risk-coverage metrics are excluded from submission-level inference because their estimands, names, qualifications, or provenance remain unresolved.Neither metric is reported numerically or used in the manuscript’s principal claims.