PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

Sanket Badhe, Deep Shah

arXiv:2602.21103v2cs.CLcs.IR

TL;DR

Chain-of-Thought reasoning can deliver high accuracy but imposes prohibitive latency and compute costs, while smaller models struggle to provide comparable reasoning. Prompt-Level Distillation transfers teacher-derived reasoning instructions into a student’s system prompt without parameter updates, and the authors report frontier-level reasoning with zero-shot-like latency on compact models.

  • Problem

    Chain-of-Thought prompting improves accuracy but creates prohibitive latency and compute costs, while smaller models struggle to deliver comparable reasoning depth efficiently.

  • Method

    Prompt-Level Distillation mines, synthesizes, and validates teacher-derived reasoning instructions from labeled data into a smaller model’s system prompt without updating parameters.

  • Results

    PLD enabled compact Gemma-3 4B and Mistral Small 3.1 models to match frontier-level reasoning with the low latency of zero-shot inference.

  • Takeaways & Limitations

    PLD offers a non-parametric way to transfer reasoning capabilities while retaining transparent, system-prompt-based instructions and efficient inference.

  • Takeaways & Limitations

    PLD may be limited on dynamic runtime computation tasks, and its instruction set may outgrow the context window or increase prompt-processing latency as task complexity rises.

Abstract

from arXiv · show

Advanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative, fine-tuning smaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introduce Prompt-Level Distillation (PLD). We extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model's System Prompt. Evaluated using Gemma-3 4B, PLD improved Macro F1 scores on StereoSet (57\% to 90.0\%) and Contract-NLI (67\% to 83\%), while increasing LogiQA accuracy to 70\%. Similar results on Mistral Small 3.1 demonstrate cross-architecture generalizability, enabling these compact models to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.

1 Introduction

The introduction presents Prompt-Level Distillation (PLD) as a non-parametric alternative that transfers teacher reasoning into a smaller model’s system prompt without updating parameters. It targets the latency, scalability, maintenance, and reasoning-transfer limitations of Chain-of-Thought prompting and traditional fine-tuning while preserving zero-shot inference speed.

  • Motivation: Chain-of-Thought prompting improves reasoning accuracy but generates verbose traces that increase inference latency and computational cost.CoT has become the default strategy for high-stakes and reasoning-intensive applications, while its traces can run hundreds of tokens long.
  • Limitations: Traditional knowledge distillation fine-tunes smaller students to mimic larger teachers, but text-only imitation can fail to transfer underlying reasoning capabilities.Fine-tuning also creates maintenance debt when the teacher or domain logic changes, requiring student retraining and management of model artifacts.
  • Contribution: PLD transfers a large teacher’s reasoning skills into a smaller student’s system prompt without updating model parameters.The framework uses a labeled training dataset and aims to combine Chain-of-Thought-level accuracy with zero-shot inference speed.
  • Methodology: PLD’s modular pipeline combines supervised instruction extraction, clustering, and closed-loop Conflict Resolution to synthesize contradiction-free reasoning heuristics.This approach compiles teacher logic into the student’s system prompt rather than compressing knowledge into weights.
  • Performance: PLD enables compact models such as Gemma-3 4B and Mistral Small 3.1 to match frontier-level reasoning with zero-shot inference latency.The claimed result decouples reasoning depth from computational cost.

2 Related Work

Prior work improves LLM reasoning through Chain-of-Thought prompting, knowledge distillation, and automated prompt optimization, but these approaches incur inference or training costs and often leave reasoning implicit. PLD instead offers non-parametric distillation by transferring reasoning capabilities into the context window through prompt injection and semantic compression.

  • Chain-of-Thought Prompting: Chain-of-Thought prompting improves multi-step reasoning by decomposing complex problems into intermediate steps, but requires lengthy autoregressive rationales that increase inference latency and computational overhead.It has become a primary method for eliciting multi-step reasoning in LLMs and improves symbolic and arithmetic benchmark performance.
  • Knowledge Distillation: Knowledge distillation transfers capabilities from teacher models to efficient students by minimizing divergences between teacher and student logic distributions.In opaque settings, related methods fine-tune students on teacher-generated samples so smaller models can internalize reasoning patterns from supervised traces.
  • Non-Parametric Distillation: Parametric distillation requires weight updates, extensive training corpora, and hyperparameter tuning, whereas PLD transfers reasoning into the context window without modifying model weights.The passage contrasts compressing intelligence into weights with prompt-based transfer to closed-weight models.
  • Prompt Optimization: Existing automatic prompt optimization searches for improved instruction wording, while PLD semantically compresses reasoning and shifts its computational cost from runtime generation to offline compilation.Prompt Compression instead reduces latency by pruning low-information tokens, whereas PLD differs in objective and retains reasoning logic as explicit prompt content.

3 Methodology

Prompt-Level Distillation transfers reasoning capabilities without parameter updates by mining, synthesizing, validating, and deploying explicit reasoning instructions in a student model’s system prompt. The method extracts generalized instructions from labeled examples, clusters and consolidates them, resolves conflicts iteratively, and performs zero-shot inference without external retrieval latency.

  • Framework: PLD treats the System Prompt as a comprehensive reasoning-instruction set mined from labeled training data, synthesized to remove redundancy, and rigorously validated.The framework transfers reasoning capabilities from reasoning-optimized models to efficient inference models without parameter updates.
  • Supervised Instruction Extraction: A teacher model jointly performs chain-of-thought problem solving and instruction abstraction, producing generalized natural-language instructions that preserve causal mechanisms while removing entity-specific details.This single-inference strategy yields an augmented dataset D = {(x_i, y_i, I_i)} and avoids multi-stage data-generation pipelines.
  • Semantic Synthesis: DBSCAN groups embedded micro-instructions by cosine-distance similarity, lets the natural number of logical-rule clusters emerge, synthesizes each dense cluster, and discards noise outliers.Unlike partition-based clustering, DBSCAN avoids forcing every instruction into a cluster, reducing contamination by non-generalizable rules.
  • Conflict Resolution: The conflict-resolution loop deploys the student on training data, isolates instruction-following errors, uses a teacher-equivalent model to refine instructions, and repeats until validation error converges.Successful examples accompany failures during adversarial refinement to help prevent performance degradation.
  • Zero-shot Inference: For deployment, the refined consolidated system prompt is injected into the student model, enabling coverage of training-defined logical constraints without external retrieval latency.The complete instruction set is provided directly to the student for zero-shot inference.

4 Experimental Setup

The experiments construct PLD instructions with capable teacher models, clustering and synthesis, then evaluate them across compact student models, prompting and fine-tuning baselines. Evaluation spans datasets with different reasoning demands, including legal entailment, stereotype-bias assessment, and complex logical deduction.

  • Teacher Models and Instruction Construction: Gemini 3 Flash provided supervised instruction extraction, while Gemini 3 Pro handled instruction synthesis and conflict resolution in thinking mode.The paper selected Gemini 3 Pro because consolidating instructions and resolving conflicts required a highly capable model.
  • Teacher Models and Instruction Construction: DBSCAN clustered extracted instructions using Gemini Embedding vectors, after which Gemini 3 Pro synthesized the clusters into unified heuristics.Full hyperparameters and prompt details are provided in Appendix C.
  • Student Models: PLD was evaluated with Gemma-3 4B, Mistral Small 3.1 24B, and Gemini 2 Flash, selected for compactness or lower inference cost.Gemma-3 4B and Mistral Small 3.1 24B were selected for efficient inference, while Gemini 2 Flash offered lower inference cost than current frontier models.
  • Baselines and Ablations: Comparisons included Zero-shot, Few-shot (k = 5, randomly sampled), TextGrad, and LoRA fine-tuning on teacher-generated reasoning traces.An ablation evaluated intermediate instructions immediately before closed-loop conflict resolution to isolate that phase’s impact.
  • Datasets: The benchmarks comprised Contract NLI for three-class legal logical relationships, StereoSet for stereotypic biases across four domains, and LogiQA for expert-exam logical deduction.These datasets represent varying reasoning complexity, from relatively simple classification to nuanced legal reasoning and categorical, conditional, and disjunctive inference.

5 Results and Analysis

PLD consistently outperformed zero-shot and few-shot baselines across StereoSet, Contract-NLI, and LogiQA, with especially strong gains on compact models. Its refinements resolved contradictions in complex tasks while externalizing reasoning through explicit logical heuristics.

  • Overall Performance: PLD consistently outperformed standard zero-shot and few-shot baselines across StereoSet, Contract-NLI, and LogiQA.The comparison validates non-parametric reasoning transfer across all three datasets.
  • Compact-Model Results: 0.90 macro-F1 on StereoSet was achieved by Gemma-3 4B with the fully refined PLD prompt, a +0.33 absolute improvement over zero-shot prompting.Gemma-3 4B is described as 25 times cheaper and 80 times faster than Gemini-3 Flash.
  • Compact-Model Results: 0.83 macro-F1 on Contract-NLI was achieved by Gemma-3 4B, outperforming its zero-shot base, while Mistral Small 3.1 showed similar trends across architectures.The results indicate that PLD-distilled instructions transfer across diverse model architectures.
  • Teacher-Model Results: 0.86 macro-F1 on ContractNLI was achieved by Gemini 3 Flash with explicit reasoning distillation, compared to its 0.77 zero-shot baseline.The teacher model also benefited from the distillation process.
  • Conflict Resolution: 2.5% macro-F1 improvement on Contract-NLI resulted from the closed-loop conflict-resolution phase, whereas StereoSet gains were negligible.The loop converged on the first iteration for StereoSet and the second iteration for Contract-NLI.
  • Reasoning Externalization: PLD externalizes pre-mined logical heuristics in the prompt, capturing complex nuances and contradictions while reducing the student model’s inference burden.This contrasts with fine-tuning, which embeds reasoning patterns in model weights, and TextGrad, which still relies on inference-time reasoning.

6 Conclusion

The paper introduces Prompt-Level Distillation as a general-purpose prompt-optimization framework that distills teacher-generated per-example instructions without fine-tuning or model training. It reports improved benchmark performance and identifies conflict resolution as especially valuable for complex edge cases, while suggesting future improvements to failure sampling and clustering.

  • 6 Conclusion: Prompt-Level Distillation automatically optimizes LLM prompts by distilling teacher-generated per-example instructions into the prompt instead of fine-tuning the model.The framework is presented as simple and general-purpose.
  • 6 Conclusion: The method significantly improves performance on two benchmarks without fine-tuning or model training.The conclusion reports evaluation against two benchmarks but does not provide their names or numerical results here.
  • 6 Conclusion: Conflict resolution provides the most value on complex tasks involving tricky edge cases.This observation identifies task complexity and edge cases as conditions where the loop is especially useful.
  • 6 Conclusion: Future work could improve conflict-resolution failure sampling and replace embedding-based clustering with in-context clustering to produce more cohesive consolidated instructions.The proposed clustering direction is motivated by rising model context windows and the emergence of in-context clustering.

7 Limitations

The evaluation focuses on reasoning-intensive classification with complex, static decision boundaries. PLD may be limited on dynamic runtime computation and as task complexity increases, because reasoning may require intermediate tokens and prompt-scaling limits were not explicitly modeled.

  • PLD is evaluated on reasoning-intensive classification tasks involving complex, static decision boundaries such as regulatory compliance.
  • PLD may struggle with dynamic runtime computation, including complex arithmetic or symbolic proofs, when reasoning cannot be condensed into a concise summary.Such tasks may require generating intermediate tokens instead of fully externalizing reasoning.
  • The system’s scaling limits for increasingly complex tasks were not explicitly modeled.

8 Ethical Considerations … C.1 Clustering Implementation details

The paper documents ethical risks, learned instruction patterns, conflicts among extracted rules, and a clustering-consolidation pipeline that organizes instructions into executable heuristics. The approach uses semantic clustering to retain generalizable logic while discarding outliers.

  • 8 Ethical Considerations: The Teacher may hallucinate or amplify training-data biases, so practitioners must ensure the Consolidated Instruction Set does not encode discriminatory logic.StereoSet is included because it measures stereotypic biases, while the Student model follows extracted instructions faithfully.
  • A Learned instructions: StereoSet instructions classify the noun preceding or modifying the blank as profession, race, gender, or religion based on semantic category.The rule maps job titles, geographic origins, gendered terms, and religious groups to corresponding bias categories.
  • A Learned instructions: Contract-NLI instructions distinguish Entailment, Contradiction, and NotMentioned by checking explicit retention permissions, prohibitions, exceptions, and silence.Examples cover archival or legal retention, mandatory return or destruction, and absent or weakly stated retention conditions.
  • B.1 Supervised Instruction Extraction conflicts: Supervised extraction can produce conflicting Contract-NLI rules, including disagreements over return triggers, broad-form information, and retention claims.The examples contrast Entailment with NotMentioned and Contradiction under similar contractual wording.
  • C.1 Clustering Implementation details: 768-dimensional dense vectors from Gemini Embedding are clustered with DBSCAN using cosine distance, ϵ = 0.4, and min_samples = 6.The configuration partitions instructions into semantic clusters and automatically discards non-generalizable outliers.
  • C.1 Clustering Implementation details: 17 distinct semantic clusters are formed for Contract-NLI and 4 broad clusters for StereoSet under the stated DBSCAN configuration.The clustering stage partitions the embedding space before instruction consolidation.
  • C.3 Instruction Consolidation Prompt: The consolidation prompt synthesizes noisy micro-instructions within each cluster into one high-fidelity heuristic while preserving causal logic.It identifies a short topic and produces a comprehensive Master Instruction from raw instructions.
  • C.3 Instruction Consolidation Prompt: Consolidation generalizes away specific entities unless they are critical thresholds, preserves If X then Y mechanisms, and outputs clear natural-language instructions.The requested output contains a short topic name and synthesized master instruction text.

C.2 Ablation Study of DBSCAN Clustering Parameters

The ablation identifies ϵ = 0.4 and min_samples = 6 as the optimal DBSCAN configuration, balancing cluster count and prompt length while yielding the highest accuracy. Deviations fragment or merge reasoning patterns, or overfit and omit long-tail logic, reducing performance.

  • Impact of Epsilon: ϵ = 0.2 created 27 fragmented clusters and a 4,200-token prompt, making instructions overly specific and lowering F1.The clustering required vectors to be extremely close, increasing fragmentation.
  • Impact of Epsilon: ϵ = 0.5 merged distinct reasoning paths into 14 broad clusters, producing generic instructions and a lower F1 score.Although the prompt became shorter, it handled nuanced edge cases less effectively.
  • Impact of Minimum Samples: min_samples = 3 formed 24 clusters and a 5,914-token prompt, capturing niche patterns but lowering accuracy through overfitting to minor variations.The threshold allowed smaller and more specific rules to form clusters.
  • Impact of Minimum Samples: min_samples = 9 reduced clustering to 15 groups, filtering for frequent patterns and losing long-tail logic that reduced overall performance.Raising the threshold forced recognition of only highly frequent reasoning patterns.

D Scalability Ablation · E Conflict resolution model · E.1 Examples of Contract-NLI Conflict resolution instructions

The ablation shows that expanding Contract-NLI data stabilizes the prompt structure after the main logical topics are identified, while conflict-resolution instructions refine entailment, contradiction, and NotMentioned decisions through explicit exceptions and obligations. Examples demonstrate broader semantic matching beyond keywords for retention and reverse-engineering hypotheses.

  • D Scalability Ablation: Increasing Contract-NLI data steadily improved F1 macro as existing clusters were refined.The supplied passage reports improvement from 0, but does not provide the ending value.
  • D Scalability Ablation: 1,030 examples established 16 logical clusters, while more than 7,000 examples plateaued at 18 clusters and kept prompt length highly stable.Additional data refined existing clusters rather than generating new topics.
  • E.1 Examples of Contract-NLI Conflict resolution instructions: The conflict-resolution model treats explicit permission to retain, keep, store, or preserve confidential-information copies for specified purposes as Entailment.It also covers retention tied to legal counsel, audits, regulatory requirements, record-keeping, or defending claims.
  • E.1 Examples of Contract-NLI Conflict resolution instructions: Retention is labeled Entailment when exceptions permit legal, archival, backup, or compliance retention, but Contradiction when all copies must be eliminated without retention exceptions.The after-model instructions also distinguish NotMentioned when the contract is silent about retention.
  • E.1 Examples of Contract-NLI Conflict resolution instructions: Reverse-engineering hypotheses are Entailed only when the contract explicitly prohibits the act without an exception, while exceptions that permit it produce Contradiction.The expanded instruction vocabulary includes reverse assemble, reverse compile, source-code discovery, and chemical-structure analysis.
  • E.1 Examples of Contract-NLI Conflict resolution instructions: The model assigns NotMentioned when specific reverse-engineering terms are absent, even if the contract generally prohibits unauthorized use or copying.This replaces a narrow keyword-only interpretation with explicit semantic conditions and exceptions.

F Model Latency · G Model Pricing

The paper reports model latency measured over 1,000 requests and presents comparisons among Gemma-3 4B, Gemini 2 Flash, and Gemini 3 Flash, alongside a model-pricing comparison.

  • F Model Latency: Latency measurements cover 1000 requests across the evaluated models.The latency experiment monitors model behavior over 1000 requests.
  • F Model Latency: Figure 2 compares end-to-end latency for Gemma-3 4B, Gemini 2 Flash, and Gemini 3 Flash.The comparison concerns E2E Model Latency across the three named models.
  • F Model Latency: Gemma-3 4B is included in the end-to-end latency comparison.Figure 2 explicitly names Gemma-3 4B as one of the compared models.
  • F Model Latency: Gemini 2 Flash is included in the end-to-end latency comparison.Figure 2 explicitly names Gemini 2 Flash as one of the compared models.
  • F Model Latency: Gemini 3 Flash is included in the end-to-end latency comparison.Figure 2 explicitly names Gemini 3 Flash as one of the compared models.
  • G Model Pricing: Figure 3 presents a comparison of model pricing.The supplied passage identifies Figure 3 as a model-pricing comparison but provides no numerical prices.
Loading 2602.21103v2…