PaperlayerView exact revision on arXiv

Source-linked AI summary

Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers

Haifeng Wu, Srinivasan Manoharan, Jian Wan, Fangbo Tu, Junhua Zhao, Xin Chen

arXiv:2610.02516v1cs.LGcs.AI

TL;DR

Scoring every request against all labels makes zero-shot NLI teacher labeling costly. This paper uses a student to select candidates for iterative teacher reranking, and reports preliminary router throughput measurements. A complete held-out evaluation remains in progress.

  • Problem

    A zero-shot NLI teacher evaluates each request-label pair separately, making labeling costly when the taxonomy has 60 labels and many requests.

    Source: §1 · p. 2

  • Method

    A student predicts the fixed taxonomy, selects top-k candidates for teacher reranking, and is retrained on resulting labels to guide later rounds.

    Source: §1 · pp. 2–3

  • Results

    At batch size 32 and concurrency 32, WAN throughput reached 242.6 requests/s at p50 latency 111.5 ms; server-local throughput was approximately 5,200–5,250 requests/s.

    Source: §6.3 · p. 9

  • Takeaways & Limitations

    The student-guided pipeline is intended to reduce teacher labeling cost while preserving the teacher’s final classification behavior.

    Source: §1 · p. 2

    The complete comparison across k, embedding retrieval, and a larger human-reviewed held-out sample remains in progress.

    Source: §7 · p. 10, §9 · p. 10

Abstract

from arXiv · show

Zero-shot classifiers are useful for routing user requests to specialized LLM tasks, but scoring every request against a large candidate set is expensive: a zero-shot NLI classifier must evaluate one premise-hypothesis pair per label, so cost scales linearly with taxonomy size. We study a student-guided teacher distillation pipeline for a fixed taxonomy of 60 LLM task categories: a compact ModernBERT classifier predicts the full category distribution in one forward pass and retrieves a small top-k candidate set, and a larger DeBERTa-v3 zero-shot NLI classifier reranks only those candidates rather than all 60 labels; the resulting teacher labels iteratively improve the student, which produces sharper candidates for the next round. Unlike generic embedding retrieval or clustering-derived shortlists used in extreme multi-label classification, our candidate generator is trained end-to-end on the target taxonomy and is the same model serving production traffic, distinguishing it from LLM-routing work that routes between candidate models, and from concurrent System-1 encoder-classifier proposals (e.g. TypeSafe AI's Jev and the open-source Laya project) whose training methodology is undocumented or RL-based. Our best student checkpoint reaches 77.5% teacher agreement on a 200-example evaluation set, and preliminary coverage measurements show Coverage@16 of 91-100%, suggesting top-k sets retain most of the teacher's decision-relevant information. We further show truncated top-k teacher scores should not be treated as full 60-class soft targets for KL distillation: zeroing untruncated classes destroys the dark knowledge soft-label distillation depends on, introducing systematic bias rather than a harmless sparse approximation. A complete evaluation, including coverage at multiple k on a held-out set, an embedding-retrieval baseline, and a larger human-reviewed test set, remains in progress.

1 Introduction

Zero-shot NLI routing requires a separate premise-hypothesis evaluation for each label, making teacher labeling costly at 60 labels. The paper proposes a taxonomy-trained student that narrows teacher scoring and iteratively improves its candidate set.

1 Introduction

  • The teacher evaluates each request against every label as a separate premise-hypothesis pair, making repeated computation the dominant dataset-creation cost at 60 labels.The introduction situates this cost in settings with hundreds of thousands of requests.Source: §1 · p. 2
  • The paper distinguishes its fixed-taxonomy teacher-labeling pipeline from Jev-style System 1 models and Laya, whose training methods or evaluations are undocumented, RL-based, or unrelated.It frames the contribution as a reproducible pipeline using existing open-weight encoders, rather than a new frontier model class.Source: §1 · pp. 2–3
  • A direct end-to-end classifier over the fixed taxonomy supplies task-aware top-k candidates, unlike generic embedding retrieval or clustering-derived shortlists.The student predicts all 60 classes in one pass, and the teacher reranks only the selected subset.Source: §1 · pp. 2–3
  • The student-guided loop uses teacher-reranked candidates to retrain the student, which proposes candidates for later rounds and can use smaller k once coverage is confirmed.The stated goal is to reduce teacher labeling cost while preserving its final classification behavior.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The evaluation framework jointly measures candidate recall, teacher agreement, classification quality, and labeling throughput.The paper also analyzes why truncated top-k teacher scores are not full-taxonomy KL targets, relating the issue to partial-label learning and non-top-class “dark knowledge” [6].Source: §1 · p. 3

2 Related Work

Related work supplies coarse-to-fine precedents across cascades, extreme multi-label classification, NLI distillation, and self-training. This paper distinguishes its fixed-taxonomy, student-guided teacher-labeling setting from those approaches and from LLM-choice routing.

2 Related Work

  • Cascade classifiers and retrieve-then-rerank systems share a coarse-to-fine structure, but generic embedding retrieval does not classify over the same fixed label set as the second stage.Speculative decoding [10] [11] is a conceptual analogue because the expensive stage verifies proposals in the same output space, though its mechanism differs.Source: §2 · p. 3
  • Extreme multi-label classification narrows large label spaces, but methods often use clustered label trees or embedding retrieval; this work instead classifies directly over its 60-class taxonomy.XR-Transformer recursively fine-tunes one model on finer partitions rather than using two models of different capacity.Source: §2 · p. 4
  • Efficient NLI work improves the NLI classifier, but does not distill its scores into a single-pass fixed-K-class student; truncated scores zeroed outside top-k bias full-distribution KL targets.The paper connects the missing-label issue to partial-label learning and the loss of non-top-class “dark knowledge” to Hinton et al. [6].Source: §2 · p. 4
  • Prior self-training rounds still have the teacher score the full label space, whereas this loop uses evolving student top-k predictions to prune per-example labels and reduce k as coverage permits.Classical active learning selects examples for oracle labeling rather than candidate labels within each example.Source: §2 · p. 4
  • LLM-routing systems choose among models rather than classify requests into a semantically meaningful task taxonomy for dispatch to agents or tools.The authors state that this fixed-taxonomy routing setting with iterative teacher-guided distillation has not been directly studied in prior published work.Source: §2 · pp. 4–5

3 Problem Formulation

The formulation uses a student distribution to select a small candidate set for the teacher, reducing full-taxonomy scoring while focusing on preserving the teacher’s top-1 decision. It measures candidate coverage and teacher agreement, while noting that agreement alone does not establish correctness.

Problem formulation

  • The teacher’s top-1 label can be recovered from student-guided labeling only if it belongs to the selected candidate set.The student selects k labels from its distribution over the fixed 60-class taxonomy; the teacher otherwise scores the full label space.Source: §3 · p. 5

Evaluation metrics

  • Coverage measures whether the teacher’s top-1 label is retained in the student-selected candidates.The section also identifies teacher agreement as the primary behavioral metric.Source: §3 · p. 5
  • Human-reviewed accuracy remains necessary because teacher agreement measures imitation, not correctness.Source: §3 · p. 5

4 Method

The method pairs a full-taxonomy student with a zero-shot teacher, using student-selected candidates for teacher labeling and iterative retraining. Its deployment uses replicated student instances behind a load balancer.

4.1 Teacher

  • The DeBERTa-v3 teacher scores all 60 labels when full distillation requires a complete teacher distribution.It uses the Hugging Face zero-shot-classification pipeline with the hypothesis template “The user’s task is {}.”Source: §4.1 · p. 5

4.2 Student

  • ModernBERT emits a 60-class probability vector in one encoder pass instead of evaluating separate prompt-label pairs.A newly initialized 60-class classification head produces logits that are converted with softmax.Source: §4.2 · p. 5

4.3 Distillation objective

  • The student combines hard and soft teacher supervision, but top-k scores with omitted classes zeroed are not valid full-distribution KL targets.Top-k reranking supplies hard pseudo-labels; KL uses the normalized teacher distribution over all 60 classes.Source: §4.3 · p. 6

4.4 Iterative procedure

  • Each iteration selects student top-k categories, has the teacher score those candidates, filters by confidence, mixes data, and fine-tunes the best checkpoint.The procedure evaluates on a fixed independent test set; k should be reduced only after candidate coverage is measured.Source: §4.4 · p. 6

4.5 Deployment architecture

  • Production serves full student replicas across four GPU instances behind Nginx, with throughput scaling close to additively at the cost of 4× GPU memory.The described architecture is used for live-endpoint measurements.Source: §4.5 · p. 6

5 Data

The experiments use a fixed 60-category taxonomy and combine seed, public, and internally generated examples. The synthetic examples are manually reviewed, and no real production request records are used.

Data sources

  • The dataset combines seed examples, public request datasets, and synthetic human-reviewed examples across a fixed taxonomy of 60 task categories.The synthetic taxonomy dataset ranges from 1,000 examples in early experiments to 2,218 in later full-60 teacher-scored runs.Source: §5 · p. 6

Serving measurement context

  • The deployed v4 serving architecture uses four GPU replicas and is the setup measured in the server-local throughput benchmark.The caption reports approximately 5,200–5,250 requests per second.Source: Figure 2 · p. 7

Data scope

  • No real user traffic or production request records are used for training or evaluation.Source: §5 · pp. 6–7

Synthetic data

  • Synthetic requests are generated by an LLM and manually reviewed for category fidelity, quality, and duplication; prompts over 8,000 characters are excluded.Source: §5 · p. 7

6 Experimental Results So Far

The best measured checkpoint is v5-tuned, while v4 remains deployed; preliminary live-endpoint measurements show high Coverage@16 in two samples and substantial throughput. The evaluation remains incomplete, and agreement varies with the evaluation sample.

6.2 Observations

  • 77.5% teacher agreement makes v5-tuned the best checkpoint measured so far, but v4 remained deployed during the live-endpoint measurements.The checkpoint results use a 200-example evaluation set and are preliminary, not a final benchmark.Source: §6.3 · p. 8, Table 1 · p. 8

6.3 Preliminary coverage, agreement, and latency measurements

  • Coverage@16 was 91.3% on the likely in-sample seed set and 100% on a best-effort fresh SWE-rebench sample.Top-k rerank agreement matched coverage at each tested k in both samples.Source: §6.3 · pp. 8–9
  • The seed-set student–teacher agreement was 39.0%, versus 62.5% on the fresh SWE-rebench sample, and the authors attribute the gap primarily to distribution shift.The seed set was likely part of v4 training; the fresh sample was best-effort held out, not certified disjoint.Source: §6.3 · pp. 8–9
  • At the highest WAN load, throughput reached 242.6 requests/s (7,763.2 classifications/s) at p50 latency of 111.5 ms.This single-external-client benchmark had 7 failed requests out of 3,646 and is not a server-capacity ceiling.Source: §6.3 · p. 9
  • Server-local throughput was approximately 5,200–5,250 single-example requests/s, with 12.15–12.29 ms average latency across four vLLM instances.These in-cluster, unbatched measurements are not directly comparable to the WAN batch-of-32 results.Source: §6.3 · p. 9
  • Coverage@16 reached 91–100% across both samples, but agreement depended on distribution proximity and a labeling-cost reduction ratio remains unmeasured.The authors call for a larger held-out, human-reviewed evaluation and comparable full-60-teacher in-cluster throughput measurements.Source: §6.3 · p. 9

7 Required Evaluation

The required evaluation is a multi-baseline comparison on a properly held-out, human-reviewed sample, and the complete comparison plan is still being executed.

7 Required Evaluation

  • The evaluation must compare student-guided selection at several k values against full-teacher and student-only bounds, as well as embedding retrieval.The comparison is intended to test the pipeline’s central claim.Source: §7 · p. 10
  • The proposed evaluation uses a properly held-out, human-reviewed sample and reports candidate recall, agreement, accuracy, latency, and cost.Source: §7 · p. 10
  • The complete comparison matrix and metric list remain in the repository punch list because the plan is still being executed.Source: §7 · p. 10

8 Reproducibility and Release Plan

The repository has scripts covering the pipeline, but model artifacts remain outside it; the planned reproducible release specifies code, data, model, evaluation, and hardware details.

8 Reproducibility and Release Plan

  • Existing scripts cover seed preparation, teacher labeling, synthetic data generation, prompt filtering, student-guided retrieval, student training, and evaluation.Source: §8 · p. 10
  • Model artifacts remain outside the public repository.Source: §8 · p. 10
  • The planned reproducible release includes the label schema, example prompts, model and tokenizer revisions, hyperparameters, split hashes, reports, and hardware measurements.Source: §8 · p. 10

9 Conclusion

The pipeline uses a student-guided teacher distillation approach for 60-class LLM task routing, with preliminary agreement and candidate-coverage results. Its central accuracy–labeling-cost comparison remains for a later revision.

9 Conclusion

  • 77.5% teacher agreement is reported for the v5-tuned student, compared with 76.5% for the deployed v4 checkpoint.These are preliminary experiments.Source: §9 · p. 10
  • Coverage@16 is 91–100% across two convenience samples for the deployed v4 checkpoint, with agreement sensitive to evaluation–training distribution shift.The passage describes this as a first, best-effort measurement.Source: §9 · p. 10
  • The completed paper’s central question—whether iterative candidate selection improves the accuracy–labeling-cost trade-off against three alternatives—remains unanswered pending planned evaluation.The planned measurements include coverage across k, an embedding-retrieval baseline, and a larger human-reviewed set.Source: §9 · p. 10

Related papers

Loading 2610.02516v1…