PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Failure, Conservative Repair: Textual Knowledge Distillation from Cross-Model Failures

Andrew Ren, Haokun Liu, Chenhao Tan

arXiv:2609.25400v1cs.AI

TL;DR

Failure-derived rules may expose missing knowledge, yet source-only refinements can underperform baseline cheat sheets on non-source model families. RFCR builds rules from shared failures, sharpens their activation boundaries with boundary cases, and abstains when validation finds no useful rule; it improves CS-ICL on a 400-item BBH task set without degrading previously correct cases.

  • Problem

    Failure-derived rules can encode source-specific patches that hurt other models, so source-model errors alone do not establish transferable task knowledge.

    Source: §1 · p. 1

  • Method

    RFCR constructs rule atoms from shared failures, constrains activation with boundary cases, and accepts only rules supported by conservative validation.

    Source: §1 · p. 2

  • Results

    +2.75 percentage points over CS-ICL on a 400-item BBH task set, without degrading any previously correct case.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Improving cheat sheets through textual distillation requires both extracting new knowledge and controlling when that knowledge is applied.

    Source: §8 · p. 10

    The main evaluation uses four BBH tasks with eligible routed atoms, so broader benchmark-scale claims require more tasks and stronger held-out validation.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Failure-based textual knowledge distillation aims to discover gaps in a model's knowledge by examining its task errors. The distilled knowledge can be useful for the reasoning of both this model ("source model") and other models. However, this transfer of knowledge may not be stable. We define a rule atom to be a standalone rule injected into a model's textual input at inference time. A rule atom can encode transferable task knowledge or model-specific reasoning patches that can confuse other models. Also, the injected rule atoms can be misapplied to unrelated cases, causing the model to incorrectly flip its answer based on irrelevant information. Building on a pipeline that distills training examples into task-specific cheat sheets that aid model reasoning, we examine when failure-derived rules can improve these cheat sheets. Our early experiment shows rule distillation from a single model's failures underperforms the baseline cheat sheet on non-source model families. This motivates Robust Failure, Conservative Repair (RFCR), a textual distillation procedure that derives rules from failures shared across models, sharpens their application boundaries using boundary cases, and abstains when no useful rule is found. On a 400-item BIG-Bench Hard task set, RFCR improves the baseline cheat sheets from 68.50% to 71.25% (+2.75 pp; 95% CI [+1.25,+4.50]) without performance degradation on previously correct cases. Ablations and cross-model diagnostics support that accuracy gains come from both new knowledge injection and strict rule-application control.

1 Introduction

RFCR addresses the risk that failure-derived rules may help their source model but harm other models. It derives rules from shared failures, constrains their application with boundary cases, and accepts them conservatively.

1 Introduction

  • Source-model failures can reveal missing knowledge, but failure-derived patches may help the source model while hurting other models.Source-only refinement underperforms the static CS-ICL baseline across all tested non-source model families.Source: §1 · p. 1
  • RFCR derives rule atoms from failures shared across models and constrains their activation using boundary cases.It accepts a rule only when evidence supports cross-model accuracy gains without degrading other cases, and leaves the cheat sheet unchanged if none passes validation.Source: §1 · pp. 1–2
  • On a 400-item BBH task set, RFCR improves CS-ICL by 11 items (+2.75 pp) without degrading previously correct cases.Figure 1 also reports gains for the four models shown.Source: Figure 1 · p. 2, §1 · p. 2

2 Failure-Derived Distillation Can Produce Model-Specific Patches

The source-only RFR diagnostic shows that refining a cheat sheet on one model’s failures can produce model-specific changes that do not transfer to other model families.

2 Failure-Derived Distillation Can Produce Model-Specific Patches

  • RFR runs three source-model-only phases on a training split, accepting patches through source-model quality gates without multi-model evaluation.The pipeline builds a global cheat sheet, patches it using source errors, then adds local ACTIVATE-IF/WHY rules for residual source failures.Source: §2 · p. 2
  • As refinement targets increasingly narrow residual error sets, its modifications become more model-specific because other models may not share those errors.The RFR diagnostic motivates evaluating failure-derived changes on non-source models.Source: §2 · pp. 2–3
  • The full RFR pipeline falls below static CS-ICL on every non-source family across five non-ceiling BBH tasks.Table 1 reports mean accuracy over those five tasks.Source: §2 · p. 3, Table 1 · p. 3

3 Transfer Theory for Failure-Derived Textual Knowledge Distillation

The transfer analysis frames a rule’s effect as dependent on where it activates relative to a model’s failures under a particular anchor. It identifies why source gains alone do not establish transfer and why inactive cases need protection.

3 Transfer Theory for Failure-Derived Textual Knowledge Distillation

  • A rule atom combines rule text with a positive activation predicate and an activation-boundary predicate; both reduce to keyword matching at inference.The same atom’s effective active region is evaluated relative to the anchor cheat sheet.Source: §3 · p. 3
  • Activation precision is the share of activated cases that are anchor failures needing help; conditional fix and degradation rates determine the atom’s accuracy effect.The proposition gives a threshold for positive marginal accuracy effect under an anchor.Source: §3 · p. 3
  • A rule can help a source model and hurt a target even when both have the same conditional fix and degradation rates.The models can differ in the proportion of baseline failures within the atom’s active region.Source: §3 · pp. 3–4
  • The same rule atom can have positive marginal gain under one anchor and negative gain under another for the same model.Atoms selected on a raw prompt therefore need revalidation when deployed on a stronger cheat-sheet anchor.Source: §3 · p. 4
  • A routed atom cannot change predictions outside its effective activation region, so inactive examples follow the anchor unchanged.If no atom activates, all examples use the anchor cheat sheet.Source: §3 · p. 4
  • Shared failures and source-private failures provide an overlap risk screen, but high overlap alone does not guarantee safe transfer.Wording, routing, fix rate, and degradation risk still matter.Source: §3 · p. 4

4 Robust Failure, Conservative Repair

RFCR reframes failure-derived distillation as routed repair rather than a global prompt update, accepting shared-failure rules only under conservative validation and returning to the baseline when none qualify.

4 Robust Failure, Conservative Repair

  • RFCR frames failure-derived knowledge distillation as routed repair rather than a global prompt update.Source: §4 · p. 4

4.1 Rule Atom Construction

  • Each atom pairs rule text with a positive activation predicate and a boundary predicate, then uses RULE, USE WHEN, DO NOT USE WHEN, and CHECK fields.Source: Figure 2 · p. 5, §4.1 · p. 5
  • Candidates come from clustered failures shared by source and proxy models, while source-private failures serve as negative boundary cases and a disjoint gate set is reserved for validation.Easy examples check for degradation; cases where the source is correct but a proxy fails are set aside because the acceptance gate would be too strict for any candidate generated from them.Source: §4.1 · pp. 4–5
  • Acceptance requires U(m) > 0, balancing lower-bound shared-failure fixes against upper-bound degradation penalties on private and easy cases plus a rule-length penalty.Strict main-table deployment additionally requires positive net improvement and no observed degradation on items CS-ICL already answered correctly.Source: §4.1 · p. 5
  • Boundary repair changes firing triggers, not rule content, using source-private failures and anchor-correct cases as negative controls; the 400-item result is controlled development because test items inform repair.For held-out validation, the frozen atom set is evaluated on 295 official BIG-Bench items unseen during generation and boundary repair.Source: §4.1 · pp. 5–6

4.2 Rule Atom Routing at Inference

  • At inference, keyword-matched predicates route atoms whose activation conditions fire and boundaries do not; if none qualify, RFCR returns unchanged CS-ICL.Source: §4.2 · p. 6

5 Experiments and Results

Experiments evaluate RFCR against the CS-ICL anchor on BBH and examine held-out results, cross-model transfer, and the effects of activation control. The reported gains are concentrated in a subset of tasks.

Evaluation setup

  • The main evaluation covered four RFCR main-eligible BBH tasks totaling 400 examples, using a GPT-4.1-mini-generated CS-ICL cheat-sheet anchor scored by GPT-4.1-mini; inactive RFCR items copied it exactly.Example selection did not rely on prior knowledge of either our method’s or baselines’ performance or expected outcomes.Source: §5 · p. 6

Main controlled result

  • 71.25% versus 68.50% for CS-ICL on the 400-item BBH set (+2.75 pp; 95% CI [+1.25,+4.50]), with no paired degradations.The comparison uses the GPT-4.1-mini CS-ICL anchor.Source: §5 · p. 6, Table 2 · p. 7

Cross-model diagnostics

  • Repaired atoms had positive net transfer on Claude Haiku, Mistral, and Qwen, but Gemini Flash Lite had negative net transfer and some models showed degradation.The cross-model rerun used no target-model route tuning.Source: Table 3 · p. 7, §5 · pp. 7–8

Held-out confirmation

  • On 295 held-out cases, the frozen atom set scored 67.46% versus 66.10% for CS-ICL (+1.36 pp), with four fixes and zero observed regressions.Only 16 cases activated atoms; the other 279 used CS-ICL passthrough.Source: §5 · pp. 6–7

Supplementary 18-task check

  • The supplementary 18-task check rose from 75.58% to 76.22%; RFCR added no atoms on 14 tasks and was presented as a passthrough sanity check.The object-counting candidate was rejected by the acceptance gate.Source: §5 · p. 7

Task-level decomposition

  • The gain was sparse: geometric shapes added 7 items and disambiguation QA added 4, while formal fallacies was neutral and object counting remained a no-op.Table 5 reports counts out of 100 items per task.Source: Table 5 · p. 8, §5 · p. 8

Activation control

  • Boundary repair raised accuracy to 71.25% without degradations while reducing test activations from 35 to 15 and increasing fixes from 1 to 4.Boundary-routed exposure also improved accuracy over global and raw-routing exposure, eliminating degradation.Source: §5 · p. 8

Validation and prompt-optimization comparisons

  • Generation-only produced more fixes but five paired degradations; ULCB filtering avoided degradations at the same net gain, and boundary repair further improved net gain and activation precision.ProTeGi recorded 32 fixes and 24 regressions, while GEPA recorded 16 fixes and 28 regressions on the 400-item set.Source: §5 · pp. 7–8

6 Analysis

RFCR’s gains are sparse and depend on routing rules to cases where they are useful. Its analysis highlights activation precision, abstention, and the risk that source-model boundaries may not fit other models.

6 Analysis

  • Only 39 rule applications activate on 400 items, accounting for +11 net fixes without source-model performance degradations.The analysis describes the gain as sparse and routed across subsets with similar knowledge demands.Source: §6 · p. 8
  • Activation precision and fix precision are positively correlated with net gain, while total activations are not.The geometric atom has both high activation precision and the largest net gain; the formal-fallacies atom is safe but too underactive to affect the test result.Source: §6 · p. 8, Table 9 · p. 9
  • Object counting receives no active applications despite eligible coverage, and inactive passthrough does not degrade performance in the supplementary full-18 result.The pipeline treats abstention as a no-op decision, not a deployment failure.Source: §6 · p. 8
  • A rule can transfer positively yet regress target-correct items when the target’s activation boundary differs from the source’s.This motivates leave-one-family-out proxy gating and per-target activation-boundary diagnostics.Source: §6 · pp. 8–9

7 Related Work

RFCR is positioned alongside cheat-sheet distillation and prompt optimization, but focuses on whether failure-derived rules transfer across model families without becoming model-specific patches.

7 Related Work

  • CS-ICL compresses many-shot demonstrations into a reusable task-specific cheat sheet, while RFCR asks when failure-derived knowledge transfers without becoming a model-specific patch.Dynamic Cheatsheet instead studies persistent, evolving memory for black-box models at test time.Source: §7 · p. 9
  • Prompt optimization methods generally optimize inputs for a measured development distribution, whereas RFCR targets cross-model harm from rules activating on examples another model already solves.The cited methods use data, feedback, search, evolution, or textual gradients.Source: §7 · p. 9
  • BBH is used to test robustness to prompt wording and reasoning format; AGIEval remains diagnostic rather than a main claim pending a locked nonnegative-anchor protocol.BBH is described as a challenging reasoning subset of BIG-Bench.Source: §7 · p. 9

8 Conclusion

The conclusion frames failure-derived rule refinement as a transfer-risk problem and presents RFCR as a conservative approach. On BBH, it improves the CS-ICL baseline without degrading previously correct cases.

8 Conclusion

  • RFCR constructs rule atoms from shared failures, refines activation boundaries with source-private failures and anchor-correct examples, and applies conservative acceptance checks.The procedure can leave the cheat sheet unchanged when no candidate passes validation.Source: §8 · p. 9
  • Rules distilled from one model’s failures can fix that model’s errors yet hurt other models on unrelated or already-correct cases.The conclusion presents this transfer risk as motivation for conservative refinement.Source: §8 · p. 9
  • RFCR improves the CS-ICL baseline accuracy by 2.75 percentage points without degrading any previously correct case.The conclusion says cheat-sheet refinement requires both extracting new knowledge and controlling when it is applied.Source: §8 · p. 10

Ethical Considerations

The work studies textual distillation on reasoning benchmarks and treats cross-model reliability as a concern. It reports paired degradations, retains a no-op fallback, and states that the experiments do not require personal data.

Ethical Considerations

Limitations

The reported improvement is modest and concentrated in geometric shapes and disambiguation QA, supporting a sparse no-degradation gain over a strong same-generator CS-ICL anchor rather than broad benchmark optimization. The implementation uses shared failures, boundary cases, conservative validation, and abstention when no rule passes.

Limitations

  • The main improvement is modest and concentrated in geometric shapes and disambiguation QA, not broad benchmark optimization.The evidence is framed as a sparse no-degradation improvement over a strong same-generator CS-ICL anchor.Source: Limitations · p. 10
  • The zero-degradation guarantee applies only to the locked source-model protocol; cross-model degradations remain for some targets because activation predicates were not validated against every target family.The retrospective leave-one-family-out proxy diagnostic did not regenerate atoms with each family excluded.Source: Limitations · p. 10
  • The main evaluation includes four BBH tasks with eligible routed atoms, so broader benchmark-scale claims require more tasks, learned routers, and stronger held-out validation.Most items in the supplementary full-18 result use inactive passthrough.Source: Limitations · p. 10
  • The 400-item result is a controlled development result, because activation boundaries were repaired using item-effect evidence from an atom-enabled pilot before the route was frozen and rerun.The protocol used a fixed task set, CS-ICL cache, selected-atom file, and paired bootstrap with 10,000 samples.Source: Limitations · p. 10, A Reproducibility Statement · p. 11

A Reproducibility Statement

  • The reproducibility statement reports a single locked run for the main result and ablations, with cross-model 95% confidence intervals computed per model using paired bootstrap resampling.The paper also states that code, the frozen atom set, and experimental CS-ICL cheat sheets are released.Source: A Reproducibility Statement · p. 11

B RFCR Implementation Details

  • RFCR identifies shared failures using failures from at least one proxy model, while source-private examples provide boundary cases for rule generation.Candidate rules are generated from shared-failure examples and source-private examples that the rule should not trigger on.Source: B RFCR Implementation Details · p. 11
  • Atoms pass the ULCB gate only when its bound-based score is positive, and strict deployment additionally requires zero observed degradations on CS-ICL-correct items across the panel.After acceptance, boundary repair narrows over-triggering predicates and selects a narrowing that maximizes net gain while retaining zero observed degradations.Source: B RFCR Implementation Details · p. 12

C Baseline Method Execution Details

The baseline comparisons use a locked, leakage-controlled protocol against the same frozen CS-ICL anchor. The appendix describes a source-only diagnostic pipeline and supplementary audits that delimit the evidence and test transfer across model families.

C Baseline Method Execution Details

  • Both baselines were evaluated against the same frozen CS-ICL anchor after optimization on 150 training items, with one prompt per task frozen before a single 100-item test evaluation.The audited protocol reported zero verbatim test-item matches and a 100% verdict parse rate for both methods.Source: C Baseline Method Execution Details · p. 12
  • ProTeGi optimized the cheat-sheet text but selected its final prompt by full-train accuracy; GEPA returned the seed unchanged on three of four tasks and its object-counting rewrite had 11 fixes and 12 regressions.On disambiguation QA, ProTeGi retained the seed cheat sheet unchanged.Source: C Baseline Method Execution Details · pp. 12–13

D RFR Diagnostic Pipeline

  • The RFR diagnostic refines a cheat sheet through source-only patching and case-study generation, so its accepted content accumulates the source model’s error profile rather than shared cross-model errors.Its gates evaluate source-model fixes and regressions only, without multi-model evaluation during development.Source: D RFR Diagnostic Pipeline · pp. 13–14

E Supplementary Audits

  • The supplementary AGIEval analysis excludes the benchmark from the main claim because its available CS-ICL anchor underperformed raw prompting on all three tasks.A development-only rebuild recovered nonnegative anchors for LogiQA and LSAT-LR, but not LSAT-AR.Source: E Supplementary Audits · p. 14
  • The retrospective leave-one-family-out proxy diagnostic found positive net transfer for Anthropic, Mistral, and Qwen, while Gemini had net degradation.The atom set remained fixed from the source protocol, and Gemini-specific activation boundaries were not validated.Source: E Supplementary Audits · pp. 14–15, Table 13 · p. 15

F Exploratory Holistic Rewrite Results

The exploratory holistic rewrite integrates repairs into the cheat-sheet text rather than appending routed atoms, with transfer varying by task and target model. Gains on object counting for smaller open-weight models coexist with a substantial loss for GPT-4.1, so the approach remains exploratory.

F Exploratory Holistic Rewrite Results

  • Holistic rewrite integrates repairs into the cheat-sheet text rather than appending a routed atom, but it is not yet robust enough to be a main contribution.The authors suggest it may help when a repair requires structural integration rather than a localized addition.Source: F Exploratory Holistic Rewrite Results · p. 15
  • Holistic rewrite substantially hurt gpt-4.1 on object counting (−15.7 pp), while causal-judgment results were mixed across target models.The reported pattern is exploratory and requires replication with tighter controls.Source: F Exploratory Holistic Rewrite Results · p. 15
  • On object counting, holistic rewrite improved accuracy by 24.0 pp for qwen2.5-7b and 11.7 pp for mistral-7b, while effects varied across other models and causal judgment.Table 15 reports exploratory cross-model transfer accuracy, averaged over seeds 1000, 2000, and 3000.Source: F Exploratory Holistic Rewrite Results · p. 15, Table 15 · p. 16

Related papers

Loading 2609.25400v1…