PaperlayerView exact revision on arXiv

Source-linked AI summary

COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Kavinda Kehelella, Sanghamitra Dutta

arXiv:2610.12837v1cs.CLcs.AIcs.LG

TL;DR

Distillation for resource-constrained LLM use remains data-intensive, and few-shot performance can worsen with samples near the teacher’s decision boundary. COFFEQA uses CFD-validated factual-counterfactual pairs for generative open-book QA, and experiments report improved knowledge transfer across multiple datasets and architectures under limited supervision.

  • Problem

    LLM distillation often relies on large-scale datasets or extensive teacher supervision, while few-shot performance depends heavily on sample choice and boundary-near samples can deteriorate performance.

    Source: §1 · pp. 1–2

  • Method

    COFFEQA uses factual-counterfactual pairs for few-shot open-book QA distillation and validates counterfactuals with CFD based on target-answer agreement and dependence on modified context.

    Source: §1 · pp. 2–3

  • Results

    Experiments across multiple datasets and LLM architectures demonstrate improved knowledge transfer under limited supervision.

    Source: §4 · p. 10

  • Takeaways & Limitations

    The findings highlight the potential of counterfactual supervision for improving knowledge transfer in generative QA under limited supervision in the few-shot setting.

    Source: §4 · p. 10

    The geometric analysis assumes exact student-teacher agreement on paired samples, while stochastic generation may fail to elicit the intended alternative answer.

    Source: §2.2 · p. 6

Abstract

from arXiv · show

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.

1 INTRODUCTION

COFFEQA addresses costly LLM distillation by using CFD-validated factual-counterfactual pairs to add supervision near difficult teacher boundaries. The introduction positions this approach for generative open-book QA, a setting not addressed by prior counterfactual or few-shot distillation work.

1 INTRODUCTION

  • Few-shot distillation depends heavily on sample choice, and samples near the teacher’s decision boundary can significantly deteriorate performance.This motivates adding supervision around challenging regions without increasing the limited few-shot budget.Source: §1 · pp. 1–2
  • COFFEQA pairs challenging factuals with counterfactuals that expose changes in the teacher’s response after minimal context modifications.For generative tasks, a counterfactual is a minimally modified input that changes the expected output while preserving semantic consistency.Source: §1 · p. 2
  • Factual-counterfactual pairs probe the teacher’s local decision-boundary geometry, with deviation bounded by counterfactual perturbation magnitude and boundary coverage.Near-boundary factuals may need smaller perturbations to change the teacher’s response, potentially leading to tighter teacher-student boundary alignment.Source: §1 · pp. 2–3
  • CFD validates counterfactuals by checking whether the teacher gives the intended alternative answer and whether that answer depends on the modified context.This targets generated examples that fail to elicit the alternative or allow the teacher to infer it without relying on the modified context.Source: §1 · p. 3
  • 8.77 average EM points and 9.98 average F1 points over the KD baseline on TriviaQA are reported for COFFEQA (T5, 32 few-shot pairs).Across five in-domain and three out-of-domain benchmarks, COFFEQA outperforms factual-only baselines under the same few-shot budget.Source: §1 · p. 3

1.1 RELATED WORK

  • Prior counterfactual QA and few-shot distillation studies do not address few-shot knowledge distillation for generative open-book QA with free-form answers.The cited related work instead includes classification settings or QA counterfactuals not used to facilitate knowledge distillation.Source: §1.1 · pp. 3–4

2 COFFEQA: COUNTERFACTUAL-GUIDED FEW-SHOT DISTILLATION

COFFEQA pairs factual QA examples with minimally modified counterfactuals to provide additional supervision near the teacher’s decision boundary. It analyzes how these pairs relate to teacher–student boundary alignment and validates generated counterfactuals before distillation.

2 COFFEQA: COUNTERFACTUAL-GUIDED FEW-SHOT DISTILLATION

  • COFFEQA pairs boundary-near factual examples with counterfactuals and uses CFD-validated pairs for few-shot distillation.The framework selects pairs that pass a CFD threshold and trains the student on factual and counterfactual examples.Source: §2 · p. 4, §2.1 · p. 4, §2.3 · p. 7

2.1 COUNTERFACTUAL DEFINITION AND GENERATION

  • Counterfactual generation preserves the question while minimally editing context to support a different answer; the described process restricts answers to five entity types.GPT-4o-mini classifies the original answer, generates a distinct alternative of the same type, then edits answer mentions and related references in the context.Source: §2.1 · p. 4

2.2 GEOMETRIC ANALYSIS OF COUNTERFACTUAL-GUIDED KNOWLEDGE TRANSFER

  • The analysis defines context-response boundaries between answer regions and uses Hausdorff distance to quantify worst-case teacher–student boundary deviation.Factual-counterfactual perturbation size and coverage of the boundaries govern the stated alignment bound; smaller perturbations and better coverage tighten it.Source: §2.2 · pp. 5–6
  • The geometric analysis assumes student–teacher agreement on paired samples and that the modified context induces the intended alternative answer.The paper notes that stochastic generation can violate the latter condition, motivating validation before distillation.Source: §2.2 · p. 6

2.3 CONTEXT FOLLOWING DIFFICULTY (CFD) FOR COUNTERFACTUAL VALIDATION

  • CFD combines answer agreement with contextual loss ratio to favor counterfactuals that elicit the intended answer and depend on the modified context.Lower CFD scores are preferred; the components capture intended-answer agreement and the context’s contribution to predicting that answer.Source: §2.3 · p. 7

3 EXPERIMENTS

Experiments compare COFFEQA with three distillation baselines under the same few-shot budget across in-domain and out-of-domain QA benchmarks. COFFEQA improves task performance and teacher-student agreement across model families, and the ablations examine CFD validation, threshold choice, and computational cost.

3 EXPERIMENTS

  • Experiments cover five in-domain and three out-of-domain benchmarks, two model families, and KD, SeqKD, and LWD baselines; both methods use the same few-shot budget.COFFEQA uses half the budget for boundary-near factuals and half for their corresponding counterfactuals.Source: §3 · pp. 8–9

3.1 MAIN EXPERIMENTAL RESULTS

  • Across datasets and k-shot settings, Figure 5 shows generally higher teacher-student F1 agreement under COFFEQA; T5 average agreement rises from 68.51 to 71.50 with standard KD.The passage reports a similar trend for Qwen2.5.Source: Figure 5 · p. 9, §3.1 · p. 9
  • T5 average F1 rises from 66.42 to 69.27 with standard KD, from 66.37 to 69.28 with LWD, and from 68.63 to 70.52 with SeqKD under COFFEQA.The averages span all datasets and k-shot settings.Source: §3.1 · p. 9
  • Under Qwen2.5, COFFEQA raises average F1 from 66.21 to 67.72 with standard KD, from 66.77 to 68.00 with LWD, and from 67.16 to 67.88 with SeqKD.The paper reports improvements across both T5 and Qwen2.5 model families.Source: §3.1 · p. 9
  • Out-of-domain F1 with standard KD rises from 53.35 to 54.52 for T5 and from 57.09 to 57.77 for Qwen2.5 under COFFEQA.These are overall averages across the three out-of-domain datasets and k-shot settings.Source: §3.1 · pp. 9–10, Table 2 · p. 10
  • With modified contextual evidence, average F1 agreement over standard KD rises from 66.72 to 69.55 for T5 and from 49.23 to 53.55 for Qwen2.5.The test replaces context answers with randomly selected answers of the same entity type and evaluates against the modified target.Source: §3.1 · p. 10

3.2 ABLATION STUDIES

  • Ablations compare CFD-verified counterfactuals with unverified examples, vary the CFD threshold, and measure counterfactual generation and training costs.The passage describes the ablation design but does not report its outcomes.Source: §3.2 · p. 10

4 CONCLUSION

The conclusion presents a few-shot, counterfactual-guided distillation method for open-book QA and reports improved knowledge transfer across datasets and LLM architectures. The accompanying reproducibility statement describes where experimental details and geometric-analysis proofs are provided.

4 CONCLUSION

  • COFFEQA uses factual-counterfactual pairs to supervise challenging regions and CFD to validate counterfactuals before few-shot distillation; experiments report improved transfer across datasets and LLM architectures.The conclusion frames counterfactual supervision as having potential to improve knowledge transfer in generative QA under limited supervision.Source: §4 · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement directs readers to appendices for dataset, training, evaluation, baseline, and geometric-proof details.Appendix F covers datasets, teacher fine-tuning and evaluation, and baselines; Appendix E provides assumptions and the complete proof.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

  • The appendix contents list sections on limitations and future work, boundary-near factuals, counterfactual prompts, the geometric proof, and experimental setup.The listed sections include B through F.Source: A APPENDIX · p. 17
  • The appendix contents also list ablations on CFD validation, threshold, and cost, plus additional experimental results on agreement, EM, and modified contextual evidence.Additional results are organized in Appendix H.Source: A APPENDIX · p. 17

B LIMITATIONS AND FUTURE WORK

COFFEQA’s current counterfactual generation is limited to entity-answer QA, and its formulation is designed specifically for open-book question answering. The authors identify broader answer types and generative tasks as future directions.

B LIMITATIONS AND FUTURE WORK

  • Counterfactual generation is limited to QA samples with entity-based answers, constraining the answer types covered by the current framework.The authors suggest more general generation strategies as future work.Source: B LIMITATIONS AND FUTURE WORK · p. 18
  • COFFEQA is formulated specifically for open-book QA, where counterfactuals modify supporting context while preserving the question.The authors identify extending counterfactuals to other generative tasks as future work.Source: B LIMITATIONS AND FUTURE WORK · p. 18

C KD PERFORMANCE ON BOUNDARY-NEAR FACTUAL SAMPLES

With the same 100-sample budget, factual examples with the smallest teacher prediction margins yield worse distillation performance than larger-margin examples. The section also describes prompts that generate minimally edited counterfactual contexts.

C KD PERFORMANCE ON BOUNDARY-NEAR FACTUAL SAMPLES

  • Smaller average teacher prediction margins indicate greater ambiguity and serve as a proxy for boundary proximity; each margin bin uses 100 factual samples.Samples are sorted by margin into three bins, with Bin 0 smallest and Bin 2 largest.Source: C KD PERFORMANCE ON BOUNDARY-NEAR… · p. 18
  • Bin 0 yields lower EM and F1 across all five datasets, with smaller differences on SQuAD than on HotpotQA, NewsQA, TriviaQA, and NQ.The pattern is reported across datasets rather than being driven by one benchmark.Source: C KD PERFORMANCE ON BOUNDARY-NEAR… · pp. 18–19
  • 56.94% mean EM in Bin 0 rises to 67% in Bins 1–2, while mean F1 rises from 65.99% to 72–73%.Distillation uses 100 factual samples per bin; the difference between Bins 1 and 2 is relatively small.Source: C KD PERFORMANCE ON BOUNDARY-NEAR… · p. 18, Figure 6 · p. 19

D COUNTERFACTUAL GENERATION PROMPTS

  • Prompts classify the answer entity, generate a plausible same-type alternative, then minimally edit the context; generated counterfactuals undergo CFD evaluation before distillation.The editing prompt replaces answer references while leaving the rest of the passage unchanged.Source: D COUNTERFACTUAL GENERATION PROMPTS · pp. 19–20, Figure 7 · p. 20

E PROOF OF PROPOSITION 1

The proof bounds the Hausdorff distance between teacher and student context-response boundaries by α + ϵ. It establishes the two directed-distance bounds separately before combining them.

E PROOF OF PROPOSITION 1

PART 1: BOUNDING THE DEVIATION FROM Mt TO Ms

PART 2: BOUNDING THE DEVIATION FROM Ms TO Mt

FINAL RESULT

  • The Hausdorff distance between teacher and student boundaries is at most α + ϵ, placing the student boundary within an (α + ϵ)-tube of the teacher boundary.The bound combines the two directed-distance results.Source: FINAL RESULT · p. 21

F.1 DATASETS

The evaluation uses eight open-book QA benchmarks, split into five in-domain and three out-of-domain datasets. The section describes their dataset properties and the entity-answer filtering used to form distillation data.

F.1 DATASETS

  • The evaluation includes five in-domain benchmarks—NQ, SQuAD, NewsQA, TriviaQA, and HotpotQA—and three out-of-domain benchmarks: DROP, DuoRC, and RelationExtraction.Source: §F.1 · pp. 21–22
  • Table 3 reports statistics for teacher datasets from original training and validation splits and student datasets containing available factual distillation and evaluation instances.Source: Table 3 · p. 22, §F.1 · p. 22
  • The benchmarks span varied question styles and sources, heterogeneous passage domains, and answerable questions whose answer spans occur within the first 800 passage tokens.Source: §F.1 · p. 22
  • Counterfactuals are generated only for entity-answer samples, and student distillation data and in-domain test sets are filtered to entity-based answers.Out-of-domain test counts are DROP 496, DuoRC 450, and RelationExtraction 912.Source: §F.1 · p. 22

F.2 TEACHER MODEL FINE-TUNING AND EVALUATION

This section describes teacher fine-tuning and the student distillation baselines, then reports that CFD verification generally provides larger gains in QA performance and teacher-student agreement.

F.2 TEACHER MODEL FINE-TUNING AND EVALUATION

  • Teacher fine-tuning used up to five epochs with validation-loss early stopping; the models converged within 1 to 2 epochs on average.T5-Large used batch size 4 and learning rate 1 × 10−4; Qwen2.5-7B-IT used batch size 1 and learning rate 2 × 10−5.Source: §F.2 · p. 22

F.3 ADDITIONAL DETAILS OF THE BASELINES

  • Standard KD combines ground-truth cross-entropy with token-level teacher-student distribution matching.The reported temperature is 1 and the loss trade-off α is 0.5.Source: §F.3 · pp. 23–24
  • LWD adds intermediate-representation supervision by aligning paired teacher and student layers, alongside the standard KD objective.In these experiments, decoder layers were uniformly mapped and self-attention distributions were aligned using KL divergence.Source: §F.3 · pp. 23–24
  • SeqKD trains the student on complete teacher-generated sequences rather than adding token-level distillation loss.The teacher sequence is approximated with beam search; the experiments use beam size 4 and cross-entropy on those sequences.Source: §F.3 · p. 24

G.1 EFFECT OF CFD VALIDATION

  • CFD verification generally yields larger gains in downstream QA performance and teacher-student agreement, especially at k = 32, 64, and 128.The comparison uses identical factual splits and training settings, differing only in whether generated counterfactuals are CFD-verified.Source: Figure 9 · p. 25, §G.1 · p. 25

G.2 CFD THRESHOLD ABLATION

COFFEQA’s gains on TriviaQA depend on the CFD threshold: moderate thresholds perform best, while a very low threshold yields negligible improvement.

G.2 CFD THRESHOLD ABLATION

  • At τ = 0.1, COFFEQA yields negligible improvements over standard KD on TriviaQA.The evaluation uses T5 across few-shot settings and reports accuracy and teacher-student agreement.Source: §G.2 · p. 26, Figure 10 · p. 26
  • Gains over standard KD peak around τ = 0.2–0.3 and decrease as the CFD threshold becomes more permissive.Figure 10 averages improvements across k ∈ {16, 32, 64, 128, 256} and five random seeds, showing accuracy and teacher-student agreement for EM and F1.Source: Figure 10 · p. 26, §G.2 · p. 26

G.3 COST ANALYSIS

The cost analysis reports counterfactual-generation costs and compares distillation accuracy, duration, and energy use; the section also identifies the scope of the in-domain EM comparison.

G.3 COST ANALYSIS

  • Counterfactual generation requires approximately 2.88 seconds per candidate, with relatively small CPU and GPU energy consumption.The reported per-candidate cost is averaged across all generation attempts under the T5 and Qwen2.5 settings.Source: §G.3 · p. 26, Table 5 · p. 26
  • For standard KD and LWD, COFFEQA achieves higher accuracy with less distillation time and energy across all k-shot settings under both model families.With SeqKD, accuracy improves across settings, but computational costs are higher at k = 16 and 32 and become comparable to baseline from k = 64 onward.Source: §G.3 · p. 26, Table 6 · p. 27, Table 7 · p. 28

H.1 IN-DOMAIN EM RESULTS

  • Table 8 compares in-domain EM scores across k-shot settings for T5 and Qwen2.5 against the baselines.Mean scores and standard deviations are reported, with aggregate results highlighted.Source: §H.1 · p. 28, Table 8 · p. 29

H.2 IN-DOMAIN AGREEMENT RESULTS

The section presents in-domain teacher-student agreement results, out-of-domain EM comparisons, and performance comparisons on modified-context test sets.

H.2 IN-DOMAIN AGREEMENT RESULTS

  • COFFEQA generally achieves higher teacher-student F1 agreement than KD, LWD, and SeqKD across in-domain datasets and k-shot settings for both T5 and Qwen2.5.The figures show k-shot settings on the spokes, with radial F1 ranges adjusted independently by dataset.Source: §H.2 · p. 30, Table 10 · p. 31, Figure 11 · p. 32

H.3 OUT-OF-DOMAIN EM RESULTS

  • Table 11 reports out-of-domain EM comparisons across k-shot settings for T5 and Qwen2.5 using KD.The table caption indicates comparison of COFFEQA with KD and reports means with standard deviations.Source: Table 11 · p. 33, §H.3 · p. 33

H.4 PERFORMANCE UNDER MODIFIED CONTEXTUAL EVIDENCE

  • Tables 12 and 13 compare EM and F1 on modified-context test sets, while Tables 14 and 15 report corresponding teacher-student agreement scores.The comparisons cover k-shot settings for T5 and Qwen2.5 using KD.Source: §H.4 · p. 33, Table 12 · p. 33, Table 13 · p. 34
Loading 2610.12837v1…