PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Prior Barriers: Supervised Fine-Tuning under Long-Tail Distribution

Haohui Wang, Jiahao Xu, Wangzhi Zhan, Tong Zeng, Dongqi Fu, Hong Li, Swastik Roy, Naren Ramakrishnan, Chris North, Jian Kang, Yujun Yan, Dawei Zhou

arXiv:2610.12345v1cs.AIcs.CLcs.LG

TL;DR

Downstream concepts can receive substantially different pretrained support, leaving rare concepts less well supported. The paper defines prior barriers, analyzes how they interact with SFT evidence, and proposes PASS to estimate instruction evidence and adaptively allocate a limited selection budget. PASS outperforms seven state-of-the-art instruction selection methods in all four backbone-budget settings.

  • Problem

    Downstream concepts differ in pretrained support, making instruction selection under a limited SFT budget a challenge across concepts with unequal starting points.

    Source: §1 · pp. 1–2

  • Method

    PASS constructs reference-derived concepts, estimates instruction-specific distinguishing evidence and prior barriers, then allocates the limited budget toward concepts that remain under-represented.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    PASS outperforms seven state-of-the-art instruction selection methods in all four backbone-budget settings, and ablations show the benefit of adaptive allocation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Under long-tail prior barriers, instruction selection can account jointly for the evidence instructions provide and concepts’ differing needs for additional supervision.

    Source: §1 · p. 2

    The theoretical guarantee is conditional on the stated assumptions, including target identifiability under the SFT instruction distribution when ε(r) > 0.

    Source: §3 · pp. 4–5

Abstract

from arXiv · show

Supervised fine-tuning (SFT) adapts pretrained large language models (LLMs) to downstream tasks, but the required concepts can receive substantially different levels of pretrained support. Frequent concepts are more likely to be well learned, whereas rare concepts may remain weakly represented. We introduce a novel notion named prior barrier to quantify how strongly the pretrained model supports competing concepts over the target concept. We observe that prior barriers follow a long-tail distribution, placing head and tail concepts at different starting points for SFT: head concepts face lower prior barriers, whereas tail concepts require additional instructions to overcome their higher prior barriers. Our theoretical analysis further derives a predictive risk bound for SFT under long-tail prior barriers, explicitly characterizing how the prior barrier and accumulated SFT evidence jointly determine predictive performance. Motivated by this prior barrier-dependent demand, we propose PASS, an adaptive SFT instruction selection method that constructs reference-derived concepts and estimates the distinguishing evidence provided by each instruction, and adaptively allocates the selection budget toward concepts that remain insufficiently covered under the current selection. In this way, PASS jointly considers which instructions can provide useful evidence and where additional supervision is needed under a limited budget. Experiments show that our method consistently outperforms seven state-of-the-art instruction selection methods on four backbone-budget settings. An ablation study further shows that PASS's adaptive allocation consistently improves over uniform allocation.

1 INTRODUCTION

Downstream concepts receive unequal pretrained support, and their prior barriers follow a long-tail distribution. The paper analyzes how those barriers shape SFT evidence needs and proposes PASS to select instructions under a limited budget.

1 INTRODUCTION

  • Prior barriers quantify pretrained support for competing concepts over a target, and their long-tail distribution leaves head concepts with lower barriers than tail concepts.Tail concepts consequently require more instructions to overcome their higher barriers.Source: §1 · pp. 1–2
  • The instruction-selection challenge is to identify useful evidence in a large candidate pool and allocate limited SFT supervision across concepts with different support.The available instruction pool can exceed the amount usable for fine-tuning, and candidate instructions can vary in quality.Source: §1 · pp. 1–2
  • The theoretical analysis derives a predictive risk bound connecting predictive performance to prior barriers and distinguishing evidence from selected instructions.Tail concepts need accumulated evidence to overcome their pretraining disadvantage.Source: §1 · p. 2
  • PASS outperforms seven state-of-the-art instruction selection methods in all four backbone-budget settings, and ablations show the benefit of adaptive budget allocation.PASS jointly estimates useful instruction evidence and where the limited budget should go.Source: §1 · p. 2

2 PRELIMINARY

The preliminary section models downstream behaviors as concepts in a space of input–response relations, with pretraining represented as a distribution over concepts. It then frames SFT instruction selection as choosing a fixed-size subset from a candidate pool to perform well on the downstream task.

2 PRELIMINARY

  • Each concept describes an input–response relation through a conditional distribution, and π represents pretrained support across concepts, with larger values indicating stronger support.The target concept is denoted θ⋆.Source: §2 · pp. 2–3
  • SFT uses supervised instruction–response pairs to increase the likelihood of target responses conditioned on their inputs.For autoregressive language models, the conditional probability is expressed as a product over response tokens.Source: §2 · p. 3
  • In the Bayesian concept-space view, the SFT dataset provides log-likelihood evidence that updates the pretrained prior over concepts.The prior represents pretrained support, while the updated distribution represents support after observing SFT data.Source: §2 · p. 3
  • The selection problem is to choose n instructions from a candidate pool using a reference set of desired downstream behaviors, aiming for strong overall task performance.The reference set represents a task with a long-tail distribution of prior barriers across concepts.Source: §2 · p. 3

3 THEORETICAL ANALYSIS

The analysis defines prior barrier as the target’s pretrained disadvantage relative to competing concepts and studies how it combines with distinguishing SFT evidence to shape posterior concentration and predictive risk.

Prior barrier

  • Prior barriers have a long-tail distribution: many concepts face relatively small barriers, while few face substantially larger ones; B(r) ≤ 0 defines the head case.In the head case, the target is at least as well supported as all concepts in Ar; when B(r) > 0, at least one competitor receives stronger pretrained support.Source: §3 · p. 3

Distinguishing evidence

  • Instruction evidence K(θ) measures expected separation of the target from a competitor, while ϵ(r) is the least such evidence among competitors outside the target’s r-neighborhood.A larger K(θ) means instruction-response pairs provide stronger expected evidence for separating the target from that competitor.Source: §3 · p. 4

Posterior concentration

  • Posterior concentration depends on B(r), the pretrained disadvantage to overcome, and nϵ(r), the distinguishing evidence accumulated through SFT instructions.Accumulated evidence grows with the number of SFT instructions and can overcome the disadvantage.Source: §3 · p. 4
  • Once the target is identifiable, sufficient SFT examples move posterior mass from competing concepts toward the target, with concentration error decreasing exponentially as supervision increases.The required supervision differs between head and tail cases.Source: §3 · p. 4

Predictive risk bound

  • If posterior mass outside the target’s r-neighborhood is at most δ, predictive risk is bounded by α(r) + M_rδ; tail targets generally need more SFT instructions to reach the same risk.The bound connects concept-level concentration to predictive performance.Source: §3 · p. 5

4 METHOD

PASS estimates distinguishing evidence for reference-derived concepts, then allocates a limited SFT budget toward concepts whose prior barriers and accumulated evidence indicate greater need.

4 METHOD

  • PASS jointly estimates instruction evidence and allocates the selection budget adaptively across concepts that remain under-represented.Its two modules estimate distinguishing evidence and allocate instructions according to concept-level demand.Source: §4 · p. 5

4.1 M1: DISTINGUISHING EVIDENCE ESTIMATION

  • M1 clusters reference-response representations into coherent concepts and represents each concept's desired and wrong-response competitors with normalized centroids.The clustering recursively splits representations until they satisfy a coherence criterion.Source: §4.1 · p. 6
  • For each candidate and concept, M1 calibrates the similarity difference between the desired representation and its closest competitor as empirical distinguishing evidence.The output is an evidence matrix across candidates and reference-derived concepts rather than a single score per candidate.Source: §4.1 · p. 6

4.2 M2: PRIOR BARRIER-AWARE INSTRUCTION ALLOCATION

  • M2 estimates concept-level demand from the prior barrier and pretrained target support, assigning more instruction to concepts with higher barriers and weaker support.It incrementally allocates instructions as useful evidence accumulates.Source: §4.2 · pp. 6–7
  • M2's objective rewards positive distinguishing evidence with diminishing returns while penalizing evidence favoring competitors and redundant selections.The redundancy term encourages selected instructions to cover distinct regions of the response representation space.Source: §4.2 · p. 7

5 EXPERIMENTS

The experiments evaluate PASS on a five-capability downstream task with long-tail prior barriers. PASS leads across all four backbone-budget settings, and its adaptive allocation outperforms uniform allocation; performance varies by less than 2% across tested hyperparameter settings.

5 EXPERIMENTS

  • The experiments also report complexity sensitivity and the composition of instructions selected by different methods.These analyses are reported in Appendix C.2 and Appendix C.3, respectively.Source: §5 · p. 7

5.1 EXPERIMENT SETUP

  • The reference-set concepts exhibit a long-tail distribution of prior-barrier scores.The reference set covers five capabilities, supporting evaluation across concepts with varied prior barriers.Source: Figure 3 · p. 8, §5.1 · p. 8
  • The setup uses a 300,932-example candidate pool and evaluates five capabilities across two backbones and two selection budgets.The capabilities are broad knowledge, multi-step reasoning, mathematics, truthfulness, and multilingual extractive QA; the budgets are 5,000 and 10,000 records.Source: §5.1 · p. 8

5.2 PERFORMANCE ANALYSIS

  • PASS achieves the best overall performance in all four backbone-budget settings.On Mistral-7B, 5,000 selected instructions score 51.5, above every competing selector using twice the budget, whose strongest score is 50.9.Source: §5.2 · p. 8, Table 1

5.3 ABLATION STUDY

  • Budget-aware allocation consistently outperforms uniform per-capability allocation across both backbones and both budgets.The uniform variant assigns an equal n/5 budget to each capability; reported gains for PASS are 0.32 and 0.26 points on Mistral-7B, and 0.29 and 0.56 points on Qwen2.5-7B, at budgets of 5,000 and 10,000, respectively.Source: §5.3 · p. 8, Table 2 · p. 9, Method · p. 9

5.4 PARAMETER ANALYSIS

  • Across the tested settings, performance fluctuates by less than 2%, indicating low sensitivity to the two studied hyperparameters.The harm coefficient λh weights the penalty on negative distinguishing evidence, while the redundancy coefficient λr penalizes redundant selections.Source: §5.4 · p. 9, Table 3 · p. 9

6 RELATED WORK

Prior instruction-selection work scores examples by quality, model-dependent utility, or target relevance, while some methods adapt supervision across skills. This paper positions prior barrier as a way to connect candidate evidence with budget allocation.

6 RELATED WORK

  • Instruction-selection methods have used response quality, complexity, diversity, model uncertainty, difficulty, and likelihood as selection signals.Examples include DEITA's combination of quality, complexity, and embedding diversity, and ADG's use of divergence among generated answers.Source: §6 · pp. 9–10
  • Reference-based methods estimate target relevance through gradient influence, pretrained representations, distribution matching, or proxy labels.The paper characterizes these methods as generally summarizing candidate utility through similarity, influence, or distribution matching.Source: §6 · p. 10
  • Skill-It, MIG, and BIDS address supervision distribution across skills or targets; this work connects allocation to unequal pretrained support through prior barrier.PASS combines prior barrier with distinguishing evidence to adaptively allocate the SFT budget.Source: §6 · p. 10

7 CONCLUSION

The conclusion frames SFT as learning concepts with unequal pretrained support, which follows a long-tail pattern. It presents PASS as combining concept-specific evidence estimates with adaptive budget allocation and reports best overall performance across four settings.

7 CONCLUSION

  • The paper introduces prior barrier to characterize unequal pretrained support, which it observes to have a long-tail distribution.The analysis links prior barrier and instruction evidence to instruction requirements: head concepts need relatively limited added supervision, while tail concepts need more evidence.Source: §7 · p. 10
  • PASS estimates concept-specific distinguishing evidence and adaptively allocates a limited SFT budget according to prior barrier.The conclusion states that effective selection should account for both candidate evidence and adaptive allocation.Source: §7 · p. 10
  • PASS achieves the best overall performance against seven instruction-selection baselines in all four backbone-budget settings, and adaptive allocation consistently improves over uniform allocation.Source: §7 · p. 10

A DETAILS OF THEORETICAL ANALYSIS

The stated theoretical setup assumes a compact concept space with a full-support reference measure and a strictly positive continuous prior density. It also specifies regularity conditions for likelihoods and a finite, measurable local approximation term.

A.1 ASSUMPTIONS

  • The concept space is compact and equipped with a full-support Borel probability reference measure; the prior has a strictly positive continuous density.The input and output spaces are standard Borel, and SFT examples are assumed i.i.d. from P⋆.Source: §A.1 · p. 14
  • The assumptions include bounded log-likelihood ratios on Ar and finite local Lipschitz bounds for log likelihoods.The displayed bounds apply to pairs of concepts in Ar and to concepts near θ⋆.Source: §A.1 · p. 14
  • The local approximation term α(r) is assumed finite, with a measurable pointwise supremum.Source: §A.1 · p. 14

A.2 PROOF OF THEOREM 1

The proof establishes uniform empirical separation over A_r, then uses that separation to bound posterior mass on A_r.

A.2 PROOF OF THEOREM 1

  • The proof establishes uniform empirical separation over A_r using Hoeffding’s inequality and a finite cover.The cover has radius ϵ(r)/(8Lr) and size Nr.Source: §A.2 · pp. 14–15
  • For n ≥ nsep(r, η), the stated bound is at most η, and Assumption 3 relates each θ ∈ A_r to a cover center.Source: §A.2 · p. 15
  • The proof writes the posterior mass of A_r and restricts the denominator of (27) to U_r before concluding.Source: §A.2 · p. 15

A.3 PROOF OF COROLLARY 1

The appendix proves the head and tail cases separately, bounds predictive risk using KL divergence, and gives PASS’s two-stage selection procedure.

A.3 PROOF OF COROLLARY 1

  • The corollary proof separates head concepts with B(r) ≤ 0 from tail concepts with B(r) > 0.Source: §A.3 · p. 16

A.4 PROOF OF THEOREM 2

  • Theorem 2’s proof splits the KL integral over Θ \ A_r and A_r, bounding the former by α(r) and the latter using Assumption 2 and Mr.It uses convexity of KL divergence and then takes expectation over x.Source: §A.4 · p. 16

B PSEUDO CODE

  • PASS constructs reference-derived concepts and estimates each candidate instruction’s distinguishing evidence for them.It forms an evidence matrix across candidate instructions and concepts.Source: B PSEUDO CODE · p. 17
  • PASS estimates each concept’s prior barrier and existing support, then allocates budget and selects instructions by marginal gain.The selection objective also includes a redundancy term.Source: B PSEUDO CODE · p. 17

C.1 IMPLEMENTATION DETAILS

The appendix reports evaluation setup, selection runtime and memory, and how selected instruction sources vary across models.

C.1 IMPLEMENTATION DETAILS

  • Evaluation combines reference partitions from five capability benchmarks and uses the remaining examples for evaluation.The benchmarks are MMLU, BBH, GSM8K, TRUTHFULQA, and TYDIQA; their evaluation sets contain 11,250, 1,080, 200, 632, and 885 examples, respectively.Source: §C.1 · p. 17

C.2 RUNTIME AND MEMORY CONSUMPTION

  • PASS requires considerably less selection time than LESS, TSDS, and ADG, with the lowest CPU memory among non-random selectors and comparable GPU memory.Source: §C.2 · p. 18, Figure 4 · p. 18

C.3 SELECTION COMPOSITION

  • Under the 5,000 budget, PASS shifts source proportions on both backbones, and its selected mix varies between Mistral-7B and Qwen2.5-7B.It increases FLAN V2, OASST1, and Dolly shares and reduces WizardLM relative to the candidate pool.Source: §C.3 · p. 18, Figure 5 · p. 19, Figure 6 · p. 19
Loading 2610.12345v1…