PaperlayerView exact revision on arXiv

Source-linked AI summary

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

Dipto Sumit, Sakib Ul Haque, Farig Sadeque

arXiv:2608.27729v1cs.CL

TL;DR

Small-model function routing is attractive under latency and cost constraints, but reported distillation gains may be artifacts of single-seed evaluation. This paper examines seed variation across KD configurations and finds bimodal collapse with distinct failure modes, while progressive and rank-based KD avoid collapse across observed seeds.

  • Problem

    Small language models are attractive for latency- and cost-constrained function routing, but KD improvements are often reported from single seeds despite unknown seed variance.

  • Method

    The paper compares multiple KD configurations using a student candidate-ranking head and teacher and student candidate distributions, while evaluating seed behavior across configurations.

  • Results

    Three of seven KD configurations exhibit bimodal collapse below 55% accuracy in some seeds, with wrong-function and output-truncation modes; progressive_kd and rank_kd avoid collapse across observed seeds.

  • Takeaways & Limitations

    Single-seed evaluation can miss central KD failure modes, whereas delaying or weakening the full-strength KL gradient avoided collapse across the observed seeds.

  • Takeaways & Limitations

    Seed counts vary across configurations, ranging from three to six, because experiments were run incrementally and seed allocation prioritized controlled comparisons.

Abstract

from arXiv · show

Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD.

1 Introduction

The paper argues that single-seed knowledge-distillation evaluations obscure large seed variance and distinct collapse modes in small-model function routing. A controlled multi-seed study finds that only delayed-onset and order-only KD avoid collapse across observed seeds, while a prompt-enrichment gain reverses under within-split retesting.

  • Study design: The study compares eight KD variants with supervised cross-entropy on a 740-instance healthcare API routing task using three to six seeds per condition.The student is a 1.5B Qwen model and the teacher is a 20B model.
  • Motivation and findings: σ=2.8–48.7 percentage points across conditions, making seed variance comparable to or larger than reported between-method gaps.The paper argues that this variance can swallow claimed KD improvements of 1–5 percentage points over supervised baselines.
  • Motivation and findings: At least one in three to five seeds falls below 55% accuracy in three KD configurations despite identical data, models, and hyperparameters.Other seeds train within a point or two of the baseline, producing bimodal rather than gradual degradation; gated_kd has elevated variance without crossing 55%.
  • Collapse modes: ce_Knowledge Distillation and ce_paraphrase produce high-confidence wrong-function selections, whereas reasoning_kd truncates before emitting a function name.The reasoning_kd truncation mode yields 0.9% accuracy and is qualitatively distinct from wrong-function selection.
  • Stability: Only progressive_kd and rank_kd avoid collapse across observed seeds, with σ = 2.8 pp and σ = 3.9 pp, respectively.Both remove the full-strength KL gradient at training onset by delaying it or replacing it with a weaker ranking signal.
  • Evaluation reliability: +3.78 pp from input enrichment across splits reverses to −2.70 pp under controlled within-split retesting.The reversal is presented as a subset-selection artifact and an example of why multi-seed reporting is needed.

2 Related Work

Prior work develops knowledge-distillation objectives, function-calling benchmarks, and warnings about seed variance, but this paper targets multi-seed small-model KD in a fixed, modest routing catalog.

  • Knowledge distillation: Knowledge distillation trains students to match a larger teacher’s softened distribution using hard cross-entropy and soft KL-divergence terms.Extensions include compressed language models, sequence-level distillation, and reasoning distillation through chain-of-thought rationales.
  • Knowledge distillation: Progressive and curriculum-style KD methods delay or anneal teacher signals, while confidence-based methods modulate or selectively apply distillation.The paper tests variants from these families and reports delayed-onset KD as the only curriculum approach eliminating collapse without sacrificing accuracy.
  • Function calling and tool use: Function-calling research often emphasizes catalogs of thousands of APIs, whereas this paper studies per-query routing accuracy in a fixed, modest catalog under linguistic variation.This positions the study as complementary to catalog-breadth benchmarks such as Gorilla and ToolLLM.
  • Seed variance: Earlier small-data evaluation studies report seed-to-seed deviations of several points and substantial changes in conclusions after random restarts.The cited literature includes BERT fine-tuning, sentence classification, and sequence labeling.
  • Seed variance: The paper states that no published small-model KD comparison had reported bimodal collapse, distinct collapse modes, or variance as high as σ up to 48.7 pp.This frames the study’s claimed novelty within the cited literature.

3 Setup and Methods

The study evaluates candidate function selection on synthetic paraphrase-family splits using a 1.5B student, a 20B teacher, multiple input formats, and diverse auxiliary-loss configurations.

  • Task and data: The task predicts one correct function from K candidates, with K ranging from 2 to 4, and excludes argument generation and multi-step tool use.The benchmark therefore isolates candidate function selection.
  • Models: The student is a 1.5B Qwen2.5-Instruct model fine-tuned with QLoRA, while the teacher is a 20B open-weight mixture-of-experts model with function-calling pretraining.Teacher annotations are generated once and stored offline.
  • Inputs and annotations: The main comparison uses names-only prompts on V1, with a controlled within-split test toggling candidate descriptions.Descriptions add one-line explanations and typed parameter lists.
  • Loss formulations: Ltotal combines causal-LM cross-entropy with a configurable candidate-selection loss implemented through a two-layer ranking head.The configurations include CE-only, KL-based KD, paraphrase consistency, combined losses, pairwise ranking, confidence gating, progressive schedules, reasoning KD, contrastive KD, and logit KD.
  • Metrics: The study evaluates Accuracy, Macro F1, and strict paraphrase consistency, defined as identical function predictions across every member of each paraphrase family.The test split contains 14 paraphrase families, making the consistency measure noisy at this sample size.

4 Results

Multi-seed V1 results show that variance and bimodal collapse dominate several KD comparisons, with wrong-function and output-truncation failures, while progressive_kd and rank_kd remain stable.

  • Reporting: Per-seed values are reported in the appendix, supporting inspection of the distribution behind each condition’s mean.The supplied results passage introduces Table 1 as the multi-seed comparison.
  • Multi-seed V1 results: Three of seven KD configurations exhibit collapse, defined as at least one seed below 55% accuracy; gated_kd has elevated variance without a sub-55% seed.Table 1 ranks multi-seed V1 test accuracies for names-only prompts by mean.
  • Multi-seed V1 results: Only progressive_kd and rank_kd are stable across all observed seeds.The table caption identifies these as the only configurations without collapse.
  • Collapse modes: reasoning_kd seed 7 reaches 0.9% through output truncation rather than wrong-function selection.The table caption marks this as a qualitatively distinct collapse mode.

Appendix A.

Multi-seed results show that auxiliary-loss KD can fail bimodally through distinct mechanisms, while progressive and rank-based variants remain stable across observed seeds. The controlled prompt-format experiment also reverses a naive cross-split improvement.

  • Seed instability: 31.5% and 48.7% accuracy collapses occurred on 2 of 5 ce_Knowledge Distillation seeds, while three remaining seeds clustered near baseline.ce_paraphrase showed one seed at 52.3% and four between 77% and 86%; reasoning_kd reached 0.9% on one of three seeds.
  • Stable variants: 79.3–85.6% across four progressive_kd seeds and 77.5–84.7% across four rank_kd seeds were accompanied by σ = 2.8 and 3.9 pp, respectively.Both methods avoided collapse, although their mean accuracies did not significantly exceed baseline.
  • Distinct collapse modes: Wrong-function collapse preserved valid function names but confidently aliased related intents in ce_Knowledge Distillation and ce_paraphrase.Examples included return_equipment→get_equipment_list and view_medical_records→search_patients.
  • Distinct collapse modes: 0.9% accuracy in reasoning_kd reflected output truncation: 107 of 111 predictions returned None before the required function name.The model produced reasoning, then terminated on the learned EOS surrogate U+622A.
  • Interpretation: Single-seed V2 results should be treated as samples from high-variance distributions rather than stable per-method effects.The paper explicitly declines to interpret the 19 pp drop of full on V2 as a stable method effect.
  • Prompt-format test: +3.78 pp in the naive cross-split comparison reversed to −2.70 pp in the controlled within-split re-test, with both arms’ standard deviations exceeding the mean difference.The controlled test fixed the V1 split and varied only prompt format across three seeds per arm.

5 Discussion

The discussion tests whether teacher noise explains collapse and instead examines the timing and strength of the KL gradient. Controlled prompt testing also shows that an apparent enrichment gain does not survive multi-seed within-split evaluation.

  • Prompt-format evidence: −2.70 pp replaced the naive +3.78 pp prompt-format result under controlled within-split testing, so the data support neither effect direction at this scale.The arm standard deviations were larger than their mean difference.
  • Mechanism: 0.9% accuracy and σ = 48.7 in reasoning_kd show that agreement-conditioned, denoised KD was not more stable than ce_Knowledge Distillation.This argues that teacher noise alone is insufficient to explain the collapse pattern.
  • Mechanism: Delaying KL or replacing it with a weaker pairwise margin signal avoided collapse across all observed progressive_kd and rank_kd seeds.The authors hypothesize that early interaction between full-strength KL and the randomly initialized ranking head may drive bimodal collapse.
  • Caveats: Four normal full seeds do not establish stability: the authors estimate a 30–50% chance that all four would avoid collapse even under a shared failure distribution.Alternatively, mixing KL with other auxiliary objectives may be protective; the evidence does not isolate the cause.
  • Practical implications: Practitioners should use ce_only as a strong baseline, prefer progressive_kd or rank_kd when using KD, and inspect raw outputs alongside multi-seed accuracy.The discussion recommends at least three seeds before reporting improvements below 5 pp.

6 Conclusion

The study finds that small-model KD can fail bimodally across seeds, with distinct collapse modes, while progressive_kd and rank_kd avoid collapse in observed runs. It also shows that an apparent prompt-enrichment gain reverses under controlled multi-seed re-testing.

  • Three to six seeds per condition expose bimodal collapse in auxiliary-loss KD, while progressive_kd and rank_kd avoid collapse across observed seeds.One in three to five seeds can fall below 55% accuracy while others train normally.
  • Wrong-function selection affects ce_Knowledge Distillation and ce_paraphrase, whereas reasoning_kd exhibits output truncation before producing a function name.The reasoning_kd truncation mode is qualitatively distinct from wrong-function selection.
  • The common property of the successful methods is removing the full-strength KL gradient from the start of training, while agreement-conditioning does not prevent collapse.The paper links this pattern to early-training KL interaction with the randomly initialized ranking head.
  • A naive cross-split prompt-format gain of +3.78 pp reverses to −2.70 pp under controlled within-split multi-seed re-testing.The reversal is identified as a subset-selection artifact.
  • The stability-accuracy tradeoff is monotone: no method achieves both higher mean accuracy and lower variance than ce_only.progressive_kd and rank_kd match the baseline mean while sitting at or below its variance.

Limitations

The evidence is constrained by uneven and generally small seed counts, a single narrow task domain, and indirect mechanism analysis. Several measurements and output inspections also have limited coverage.

  • Seed counts range from n = 3 to n = 6, making σ estimates such as reasoning_kd’s σ = 48.7 preliminary rather than tight population estimates.All collected seeds were reported, but allocation was uneven across configurations.
  • Three seeds document that collapse exists but cannot establish precise collapse-rate distributions or stable quantitative rankings.Even the best-sampled conditions use n = 6, below recommendations of n ≥10 for stable variance estimates.
  • The study covers one healthcare/insurance/fitness domain with approximately 700 training examples and a 1.5B student, limiting transfer to larger catalogs or other domains.Teacher disagreement with gold is 16.9%, so a more capable teacher could shift the comparison.
  • The early-training KL-interaction hypothesis is inferred from ablations rather than direct measurements of gradient trajectories or loss-landscape geometry.Two converging ablations and one null result support the hypothesis, but two counterexamples complicate it.
  • Paraphrase consistency uses only 14 families and can reach 1.0 vacuously, while manual output inspection covered only the most extreme collapsed checkpoints.Subtler output pathologies may therefore remain undetected.

Ethics Statement

The work uses synthetic, non-personal data and frames its findings as a deployment-safety concern for small-model function routing. It recommends multi-seed evaluation and raw-output inspection before deployment.

  • The dataset contains synthetic instruction-tool pairs without personal information, and its healthcare-themed schemas are illustrative rather than connected to clinical systems.The trained models are not intended for clinical or patient-facing deployment.
  • Single-seed deployment of tested auxiliary-loss KD variants could produce fluent reasoning without a function call or confidently route requests to the wrong API.The paper identifies these outcomes as a direct safety implication in function-routing settings.
  • The authors recommend multi-seed evaluation and raw-output inspection, not only argmax accuracy, before shipping KD-trained small students.
  • The full multi-seed sweep used approximately 30 GPU-hours, and the code, logs, and prompt templates are released for replication.

A Per-seed accuracy values

Table 5 reports per-seed accuracies for all multi-seed conditions, using a base seed set and additional seeds added incrementally.

  • Per-seed accuracies are reported for all multi-seed conditions, with seed assignments drawn from a base set and incrementally added seeds.The base set is 42, 123, 7; additional seeds include 456, 789, 1729, 2024, and 2025.
Loading 2608.27729v1…