PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Textual Teaching for LLMs

Zhanyi Lu, Huan Wang

arXiv:2610.12114v1cs.AI

TL;DR

Most LLM distillation updates Student parameters, binding knowledge to specific weights and limiting inspection and reuse. UTT identifies Teacher–Student knowledge gaps and builds a natural-language Primer through parameter-free, multi-role interactions. It improves results across math and code tasks, and Primers can transfer across Students without resynthesis.

  • Problem

    Most existing LLM distillation trains the Student, binding knowledge to specific architectures and checkpoints and limiting inspection, reuse, and applicability to API-only or costly-to-train models.

    Source: §1 · p. 2

  • Method

    UTT identifies representative Teacher–Student gaps and synthesizes a natural-language Primer through Student attempts, Prompter feedback, Teacher demonstrations, and Synthesizer integration without parameter updates.

    Source: §1 · p. 2

  • Results

    Across math and code tasks, UTT improves source Students and transfers across Students without resynthesis; KernelBench accuracy rises from 9.4% to 48.6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The Primer provides an interpretable knowledge representation not tied to particular model weights and directly transferable to other models.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Knowledge distillation (KD) transfers knowledge from stronger Teacher models to weaker Student models, but most methods require training the Student parameters, thereby binding the distilled knowledge to a specific architecture and checkpoint. This implicit representation is difficult to interpret or reuse across models and limits KD for API-only or costly-to-train models. This paper studies knowledge transfer for large language models (LLMs). We introduce Universal Textual Teaching (UTT), a parameter-update-free framework that distills observed Teacher-Student knowledge gaps into a textual, interpretable, and reusable natural-language artifact called Primer. Specifically, UTT first identifies representative gap cases through paired evaluations, and iteratively updates the Primer via multi-role interactions: the Student attempts each task, the Prompter turns evaluation feedback into a teaching instruction, the Teacher provides a targeted demonstration, and the Synthesizer consolidates validated lessons. Empirically, on the challenging math (Omni-MATH-2) and code generation (KernelBench) tasks, extensive results confirm the effectiveness of the method: UTT remarkably raises the Student's accuracy from 9.4% to 48.6% and Fast1 accuracy from 9% to 35% on KernelBench, while increasing mathematical reasoning accuracy from 27.6% to 51.7%. UTT also performs better than representative prompt engineering and parameter-based KD methods. Of note, UTT is shown to be generalizable across different Teachers and Students: a Primer synthesized for one Teacher-Student pair can generalize to other Students that do not participate in the synthesis.

1 INTRODUCTION

Most LLM distillation updates Student parameters, tying knowledge to particular weights and limiting inspection and reuse. UTT instead distills observed Teacher–Student gaps into a reusable Primer through parameter-free, multi-role teaching interactions.

1 INTRODUCTION

  • Most existing LLM distillation methods train the Student, binding knowledge to a specific architecture and checkpoint and limiting inspection, transfer, and use with API-only or costly-to-train models.The cited routes include supervised fine-tuning on Teacher outputs and preference optimization or reinforcement learning guided by Teacher evaluations.Source: §1 · p. 2
  • UTT identifies representative gap cases, then uses Student attempts, Prompter feedback, Teacher demonstrations, and Synthesizer integration to build a deployable Primer without parameter updates.The framework relies on LLM interactions and supports open-source and API-only commercial models.Source: §1 · p. 2
  • KernelBench accuracy rises from 9.4% to 48.6%, Fast1 from 9% to 35%, and mathematical reasoning accuracy from 27.6% to 51.7%; UTT also outperforms representative prompt engineering and parameter-updating KD methods.Primers improve source Students and transfer to other Students without resynthesis across evaluated math and code settings.Source: §1 · p. 2
  • UTT represents the Teacher–Student knowledge gap in natural language as an interpretable Primer, rather than encoding transferred knowledge in Student parameters.The paper argues reliable transfer should identify problems solved by the Teacher but not the Student, extract reusable knowledge, and avoid degrading existing capabilities.Source: §1 · p. 2

2 RELATED WORK

Related work includes parameter-updating distillation, prompt optimization, and other textual knowledge representations. UTT distinguishes itself by selecting and synthesizing teaching content from an observed Teacher–Student performance gap, then examining cross-Student transfer.

Knowledge distillation for LLMs

  • LLM distillation commonly uses supervised Teacher-generated outputs or on-policy trajectories and Teacher feedback, but typically stores transferred knowledge in Student parameters.Parameter binding limits independent inspection, editing, cross-model reuse, and use when Student parameters are not trainable.Source: §2 · p. 3

Prompting techniques and engineering

  • Unlike automatic prompt optimization, which refines prompts for target-task performance, UTT selects teaching content from a designated Teacher’s observed advantage over a Student.UTT organizes that knowledge into an inspectable Primer intended for reuse across Students.Source: §2 · p. 3

Textual knowledge representations

  • Prior textual representations and experience or skill libraries address model parameters, dataset explanations, trajectories, environmental rules, or general-purpose skills, rather than Teacher–Student performance gaps.UTT uses paired performance to choose what to transfer and tests whether the resulting knowledge transfers without resynthesis.Source: §2 · p. 3

3 UNIVERSAL TEXTUAL TEACHING

UTT reformulates distillation as optimizing a natural-language knowledge carrier instead of Student parameters. It constructs gap-focused training data, synthesizes a Primer through multi-role interactions, and validates candidates before deployment.

3 UNIVERSAL TEXTUAL TEACHING

  • UTT reframes distillation as optimizing a natural-language knowledge carrier rather than model parameters, then instantiates it through gap construction and multi-role interaction.The framework overview proceeds from Teacher–Student gap evaluation to Primer synthesis and evaluation, with model parameters frozen.Source: §3 · p. 4, Figure 2 · p. 5

3.1 FROM PARAMETRIC TO TEXTUAL KNOWLEDGE DISTILLATION

  • Textual distillation fixes both models and optimizes a Primer that enhances the Student through context, rather than optimizing Student parameters as conventional KD does.The discrete text objective makes distilled knowledge auditable and reusable across models; UTT uses gap-construction training data, task-specific evaluation feedback, and evaluative plus generative Teacher signals.Source: §3.1 · p. 4

3.2 THE UTT FRAMEWORK

  • The UTT framework first partitions data through knowledge-gap construction, then synthesizes a Primer through multi-role LLM interaction.The resulting Primer is evaluated on the source Student and transferred to other Students, while model parameters remain frozen.Source: §3.2 · p. 4, Figure 2 · p. 5

3.2.1 KNOWLEDGE-GAP CONSTRUCTION

  • Paired Teacher–Student evaluations divide data into distillation, retention, and frontier subsets; only the distillation training set feeds Primer synthesis.Distillation examples are those where the Teacher outperforms the Student; retention and frontier subsets evaluate capability preservation and potential improvement beyond the Teacher. Training and test sets do not overlap.Source: §3.2.1 · pp. 4–5

3.2.2 CLOSED-LOOP PRIMER SYNTHESIS VIA MULTI-ROLE INTERACTION

  • In batches, Student attempts expose gaps, Prompter feedback frames instructions, Teacher demonstrations supply lessons, and the Synthesizer consolidates validated records into a global Primer.Only records whose Teacher demonstrations pass evaluation are retained, and a candidate Primer passes the validation gate only if it performs no worse than the current Primer on the same batch; the final Primer can be deployed without parameter updates.Source: §3.2.2 · pp. 5–6

4 EXPERIMENTAL RESULTS

Across math and code benchmarks, UTT improves Student performance, transfers Primers across evaluated model configurations, and compares favorably with prompt-engineering and parameter-updating KD methods. The analysis also examines generalization to unseen problems, retention of existing capabilities, and frontier extension.

4.2 PRIMER EFFECTIVENESS AND CROSS-MODEL TRANSFERABILITY

  • 39.2, 26, and 24.1 percentage points are the Pro→Flash gains in KernelBench accuracy, Fast1, and Math accuracy, respectively.The scores rise from 9.4% to 48.6%, 9% to 35%, and 27.6% to 51.7%.Source: §4.2 · p. 7, Table 1 · p. 7
  • Without resynthesis, exchanging the Pro-based Primers between Flash and Qwen improves the receiving Student on KernelBench accuracy, Fast1, and Math accuracy.The Flash-derived Primer gives Qwen gains of 32.0, 25, and 25.8 points; the Qwen-derived Primer gives Flash gains of 34.4, 21, and 16.1 points.Source: §4.2 · p. 7
  • An Opus-derived Primer improves Flash, then transfers unchanged to Qwen, which did not participate in synthesis.The transferred Primer yields 51.0% KernelBench accuracy, 22% Fast1, and 65.8% Math accuracy on Qwen.Source: §4.2 · p. 7

4.3 ANALYSIS: GENERALIZATION, RETENTION, AND FRONTIER EXTENSION

  • On the distillation test set, Pro→Qwen math accuracy rises from 9.7% to 64.8%, supporting generalization to unseen problems.The analysis also reports Pro→Flash KernelBench accuracy rising from 6.7% to 65.2% on the distillation training set.Source: §4.3 · p. 8
  • On the retention subset, KernelBench accuracy improves in all three configurations, while Omni-MATH-2 accuracy decreases for Pro→Qwen and Opus→Flash.The math decreases are 9.2 and 14.2 percentage points, respectively; Pro→Flash math accuracy rises from 58.0% to 69.0%.Source: §4.3 · p. 8
  • On the frontier subset, primed Students achieve nonzero accuracy in five of six configurations, including problems neither Teacher nor Student solved at baseline.The three KernelBench configurations reach 11.7% to 18.6%; Pro→Flash and Pro→Qwen reach 5.3% and 30.0% on Omni-MATH-2.Source: §4.3 · p. 8

4.4 COMPARATIVE EXPERIMENTS

  • UTT performs best across all six prompting combinations; GEPA is stronger on Omni-MATH-2, while MIPROv2 performs better on KernelBench.The paper suggests the inconsistent Teacher-summary results indicate that Teacher-Student interaction is important for constructing an effective Primer.Source: §4.4 · pp. 8–9, Table 2 · p. 9
  • UTT exceeds the strongest parameter-updating KD baseline by 7.2 points on Omni-MATH-2 accuracy, 8.2 on KernelBench accuracy, and 4 on Fast1.The comparison is under the Pro→Qwen setting, with Student parameters fixed.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION

The conclusion presents UTT as parameter-update-free knowledge distillation that turns Teacher–Student gaps into a textual Primer. Across math and code tasks, it reports improved Student performance and frames the Primer as an interpretable knowledge carrier transferable across models.

5 CONCLUSION

  • UTT identifies Teacher–Student knowledge gaps and synthesizes validated teaching records into a textual Primer through multi-role LLM interactions, without parameter updates.The paper reports substantial Student performance improvements across Teacher–Student settings on Omni-MATH-2 and KernelBench.Source: §5 · p. 9
  • UTT substantially improves Student performance across Teacher–Student settings on the math and code-generation benchmarks Omni-MATH-2 and KernelBench.The conclusion also reports that UTT outperforms representative prompt-engineering and KD methods.Source: §5 · p. 9
  • The Primer is interpretable and not tied to particular model weights, supporting the paper's view of natural language as a deployable knowledge carrier across models.The paper presents this as a parameter-update-free perspective on LLM knowledge distillation.Source: §5 · p. 9

A.1 DATASETS AND EVALUATION

The appendix specifies the evaluation pools and protocols, model-role assignments, and the prompts used to conduct and consolidate UTT’s teaching interactions.

A.1 DATASETS AND EVALUATION

  • Omni-MATH-2 uses 200 problems from difficulty levels 5–10, while KernelBench uses all 100 Level 1 problems; each problem receives five model responses.The Omni-MATH-2 set is balanced across categories, and the resulting initial pools contain 1,000 and 500 samples, respectively.Source: §A.1 · p. 12

A.2 MODELS AND ROLE ASSIGNMENTS

  • The four roles use distinct prompts and runtime inputs, while the Teacher model also serves as Prompter and Synthesizer; model parameters remain unchanged.The target Student model performs the Student role; all calls to the same underlying model share decoding settings.Source: §A.2 · p. 12, §A.3 · p. 12

A.3 ROLE PROMPT TEMPLATES

  • The role templates combine system instructions with runtime fields, including task-specific requirements and, for batched synthesis, repeated case-level fields.The implementation sends the system instruction and runtime fields as separate system and user messages.Source: §A.3 · p. 12

Student Role Prompt

Prompter Role Prompt

  • The Prompter diagnoses the primary gap from task and evaluation feedback, then gives the Teacher concise, actionable guidance grounded in transferable principles.It is instructed to distinguish an underlying cause from symptoms and not solve the task itself.Source: Prompter Role Prompt · p. 13

Teacher Role Prompt

  • The Teacher follows the instruction to emphasize the identified knowledge, reasoning strategy, or validation steps in a complete worked solution.Its output must follow the task’s format and contain no additional commentary.Source: Teacher Role Prompt · p. 13

Synthesizer Role Prompt

  • The Synthesizer maintains one evolving, compact Primer by merging transferable lessons, preserving useful knowledge, and removing redundancy or material that is incorrect or too vague.It generalizes beyond individual tasks, avoids retaining complete solutions or edit history, and observes a specified word limit.Source: Synthesizer Role Prompt · p. 14

A.4 METHOD AND BASELINE CONFIGURATIONS

The appendix describes how UTT partitions knowledge-gap samples and sets its synthesis and inference budgets, alongside the prompt-optimization and parameter-updating baselines.

A.4 METHOD AND BASELINE CONFIGURATIONS

  • UTT constructs distillation, retention, and frontier subsets from paired Teacher–Student performance, then splits distillation data 7:3 for Primer synthesis and evaluation.The distillation-training portion is used for synthesis; the distillation-test portion is used for evaluation.Source: §A.4 · p. 14
  • UTT uses batches of 16, prepends the Primer to inference inputs, and applies a default soft ceiling of 1,000 words to limit Primer growth.The optimization budgets are 100 sample-level evaluations for KernelBench and 200 for Omni-MATH-2.Source: §A.4 · p. 15
  • Prompting baselines use benchmark templates or three examples, while Teacher summary condenses distillation-training problems into a global prompt.KernelBench one-shot and few-shot templates contain one and three demonstrations; Omni-MATH-2 examples are sampled outside the selected task pool.Source: §A.4 · p. 15
  • APE, MIPROv2, and GEPA use standard implementations, validation-selected prompts, and the same total optimization budget as UTT.Their prompts are generated and refined by Pro; MIPROv2 also has a demonstration-augmented setting.Source: §A.4 · p. 15
  • Under Pro→Qwen, SeqKD, Fine-tune-CoT, RSR, and LUFFY train LoRA adapters on a frozen Qwen backbone using UTT’s distillation-training set.The baselines use 200 updates on Omni-MATH-2 and 100 on KernelBench; losses had largely stabilized by the end of those budgets.Source: §A.4 · p. 15

B.1 EXTENSION TO MULTIMODAL REASONING

On MathVerse visual geometry problems, UTT Primers improved source and transfer Students across the reported configurations, with transfer gains varying by target model.

B.1 EXTENSION TO MULTIMODAL REASONING

  • The evaluation uses 100 Vision Dominant plane-geometry problems, two responses per problem, and three Teacher–Student configurations.The synthesis procedure follows the main experiments.Source: §B.1 · p. 15
  • Primers improved every source and transfer Student; cross-Student gains were positive, but their magnitudes differed between Qwen3.6-27B and DeepSeek-V4.1-Flash.For example, Qwen3.6-27B rose from 41.5% to 76.0% with Qwen3.8-Max as Teacher and from 41.5% to 80.5% with Claude Opus 5.Source: §B.1 · pp. 15–16, Table 5 · p. 16

B.2 PRIMER EFFECTIVENESS UNDER LARGER TOKEN BUDGETS

Across evaluated output-token budgets, UTT outperforms the unprimed Student, with a larger advantage at smaller budgets and continued gains as performance approaches saturation.

B.2 PRIMER EFFECTIVENESS UNDER LARGER TOKEN BUDGETS

  • To isolate inference-budget effects, UTT uses the same Primer synthesized at 32k tokens at every evaluated budget, without resynthesis.The evaluation compares the unprimed Student, prompt-optimization methods, UTT, and the Teacher.Source: §B.2 · p. 16
  • Accuracy generally rises with output-token budget while gains diminish; UTT beats the unprimed Student at every budget, especially at smaller budgets, and retains its advantage near saturation.The plotted curves compare the unprimed Flash Student, Pro Teacher, APE, GEPA, and UTT.Source: §B.2 · p. 16
  • Truncation errors decrease as budgets grow; at the largest tested budget, UTT remains between baseline and Teacher, while APO approaches or slightly underperforms baseline.UTT does not reach Teacher performance in the reported range.Source: §B.2 · p. 17

B.3 RESOURCE COMPARISON BETWEEN KD TRAINING AND UTT PRIMER SYNTHESIS

UTT synthesizes a Primer without parameter training or local GPUs, using API calls instead. The reported resource measurements are not strictly equivalent costs because the methods consume different resource types.

B.3 RESOURCE COMPARISON BETWEEN KD TRAINING AND UTT PRIMER SYNTHESIS

  • The comparison assumes fixed, available task data and covers only each method’s optimization stage.The authors caution that GPU training compute and API inference tokens are different resource types, so the measurements are not strictly equivalent costs.Source: §B.3 · p. 17
  • UTT completed Primer synthesis in 1.6 hours through 281 API model calls, using approximately 0.68 million input tokens and 5.57 million output tokens.The paper reports that UTT outperformed all KD baselines without parameter updates, but does not interpret GPU training compute and API inference tokens as strictly equivalent costs.Source: §B.3 · p. 17
  • 1.2–7.5 hours of parameter training were required by the three supervised KD baselines; LUFFY required 22.0 hours on eight GPUs.SeqKD, Fine-tune-CoT, and RSR used four RTX A6000 GPUs; LUFFY used eight.Source: §B.3 · p. 17

C.1 EFFECTS OF CORE COMPONENTS AND BATCH SIZE

The ablations test knowledge-gap selection, the Prompter, validation gating, and batch size; the same transferred Primer also works across evaluated Qwen quantization and thinking settings.

C.1 EFFECTS OF CORE COMPONENTS AND BATCH SIZE

  • 65.3%, 56.0%, 72.5%, and 5.0% accuracy with the full method exceeded the unprimed Student’s 33.7%, 33.3%, 60.0%, and 0.0% across four subsets, respectively.The subsets are distillation-training, distillation-test, retention, and frontier.Source: §C.1 · p. 18
  • Knowledge-gap selection improved distillation-training and distillation-test accuracy by 9.7 and 6.0 percentage points over random selection, with unchanged retention accuracy and one additional frontier response correct.Both variants used 98 synthesis problems; random selection removed selection based on the Teacher–Student gap.Source: §C.1 · p. 18
  • Removing the Prompter raised distillation-training accuracy to 72.4% but lowered distillation-test and retention accuracy from 56.0% and 72.5% to 48.8% and 70.0%.The source describes this as a trade-off between stronger fit to synthesis problems and transfer to problems not used for synthesis.Source: §C.1 · p. 18
  • Without validation gating, distillation-training, distillation-test, and frontier accuracy fell from 65.3%, 56.0%, and 5.0% to 59.2%, 39.3%, and 3.8%, while retention rose from 72.5% to 80.0%.The ungated variant accepted candidate updates regardless of the comparison between Primer versions.Source: §C.1 · p. 18
  • Batch size 16 performed best on distillation-training and distillation-test and tied size 32 on retention and frontier; size 8 performed worse across all four subsets.At size 32, distillation-training and distillation-test accuracy decreased to 59.2% and 45.2%.Source: §C.1 · p. 18

C.2 EFFECTS OF QUANTIZATION AND THINKING MODE

  • The same Primer improved accuracy and Fast1 in all four evaluated Qwen3.6-27B quantization and thinking settings without resynthesis.The tested settings included unquantized Qwen with thinking enabled, two Int4 RTN configurations, and unquantized Qwen with thinking disabled; gain magnitudes varied.Source: §C.2 · p. 19, Table 8 · p. 19, §C.1 · p. 19

D PRIMER ANALYSIS AND EXAMPLES

The examples show task-specific Primers organizing mathematical problem-solving, CUDA kernel engineering, and multimodal geometry guidance. They preserve generated content as examples, including potentially inaccurate or incompletely qualified statements.

D PRIMER ANALYSIS AND EXAMPLES

  • The examples organize math problem-solving, CUDA kernel generation, and multimodal geometry into task-specific guidance and strategies for avoiding failures.The displayed Primer texts preserve model-generated content without factual correction, so they may include inaccurate, incompletely qualified, or problem-specific statements.Source: D PRIMER ANALYSIS AND EXAMPLES · p. 19
  • The mathematical Primer groups Olympiad records by topic and includes practices such as checking boundary cases, verifying constructions, and testing counterexamples.Its examples cover geometry, number theory, polynomial values, graph constructions, combinatorics, and probability.Source: D PRIMER ANALYSIS AND EXAMPLES · p. 19

1 Geometry

  • Geometry guidance uses supporting lines and signed foot positions, since an obtuse-case foot can fall outside the segment.It cautions that unsigned distances fail in obtuse cases.Source: §1 · p. 20

Core Requirements

Symbol Visibility Before Binding (Critical)

  • Kernel extension bindings require referenced functions to be visible before registration; the examples give monolithic and multi-file patterns.The host function signature must also be compatible with the PyTorch wrapper.Source: Symbol Visibility Before Binding… · p. 22

General Kernel Design Patterns

0. Non-negotiable output rule

  • The geometry Primer emphasizes identifying what labels denote, tracing angle relations, and stopping once the target is determined.It recommends an answer-first output and a limited number of figure interpretations to avoid spending the reasoning budget without a final answer.Source: §0 · p. 24, §1 · p. 24, §2 · p. 25

E LIMITATION DISCUSSION AND FUTURE WORK

The paper identifies Primer length as an inference-time cost and presents weight-based distillation as a possible next step, while framing text-only distillation as its proof of concept.

E LIMITATION DISCUSSION AND FUTURE WORK

  • Because the Primer is prepended to every task input, UTT increases inference-time token usage; compression or retrieval-based selection could reduce this overhead.Source: E LIMITATION DISCUSSION AND FUTURE WORK · p. 26
  • The authors identify distilling Primer knowledge into model weights through subsequent training as a natural next step.They describe this direction as most practical for resource-rich organizations with model-weight access and substantial training infrastructure.Source: E LIMITATION DISCUSSION AND FUTURE WORK · p. 26
  • The work’s primary goal is a proof of concept that LLM knowledge distillation can occur in text form without weight updates.Source: E LIMITATION DISCUSSION AND FUTURE WORK · p. 26

Related papers

Loading 2610.12114v1…