PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?

Hansi Zeng, Kai Hui, Honglei Zhuang, Zhen Qin, Zhenrui Yue, Hamed Zamani, Dana Alon

arXiv:2504.12491v2cs.CL

TL;DR

Whether pre-training indicators can reliably predict fine-tuning performance among same-size LLMs remains unclear, despite perplexity’s correlation with scaling-driven gains. The study uses five pre-training proxies in a binary classifier to predict which of two LLMs performs better after SFT. Proposed unsupervised and supervised proxies reduce relative performance prediction error by over 50% compared with perplexity.

  • Problem

    Whether pre-training indicators can reliably predict fine-tuning performance among same-size LLMs remains unclear, despite perplexity’s correlation with scaling-driven gains.

    Source: §1 · p. 1

  • Method

    The study uses five pre-training proxies in a binary classifier to predict which of two LLMs performs better after SFT.

    Source: §4.1 · p. 5

  • Results

    Proposed unsupervised and supervised proxies reduce relative performance prediction error by over 50% compared with perplexity.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    The proxies are useful particularly for predicting outcomes between models with large performance gaps and identifying top-performing candidates.

    Source: §7 · p. 11

    Generalizability to larger model scales, broader downstream tasks, and other fine-tuning paradigms remains for future study.

    Source: §7 · p. 11

Abstract

from arXiv · show

While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development. To address this gap, we formulate the task of selecting pre-training checkpoints to maximize downstream fine-tuning performance as a pairwise classification problem: predicting which of two LLMs, differing in their pre-training, will perform better after supervised fine-tuning (SFT). We construct a dataset using 50 1B parameter LLM variants with systematically varied pre-training configurations, e.g., objectives or data, and evaluate them on diverse downstream tasks after SFT. We first conduct a study and demonstrate that the conventional perplexity is a misleading indicator. As such, we introduce novel unsupervised and supervised proxy metrics derived from pre-training that successfully reduce the relative performance prediction error rate by over 50%. Despite the inherent complexity of this task, we demonstrate the practical utility of our proposed proxies in specific scenarios, paving the way for more efficient design of pre-training schemes optimized for various downstream tasks.

1. Introduction

The paper asks whether pre-training signals can predict fine-tuning outcomes among same-size LLMs, where conventional perplexity’s scaling-study correlations may not hold. It evaluates proxy metrics on systematically varied 1B-parameter models and finds that alternatives, especially learning-to-compare, reduce prediction errors.

1. Introduction

  • Perplexity’s predictive efficacy for fine-tuning outcomes within a constant model size remains uncertain despite its scaling-based correlations.Dependable predictors could help avoid the expense of fine-tuning numerous checkpoints.Source: §1 · p. 1
  • The study varies pre-training configurations across 50 distinct 1B-parameter LLMs, then evaluates their performance after SFT on diverse tasks.The tasks include commonsense reasoning, retrieval-augmented generation, and closed-book question answering.Source: §1 · p. 2
  • The prediction task compares pairs of pre-trained models that differ in pre-training and predicts which will perform better after SFT.The setup targets selection among candidate models rather than predicting an absolute performance score.Source: §1 · p. 2
  • Over 50% lower relative prediction error is achieved by proposed proxies compared with perplexity, while learning-to-compare further decreases predictive error.The introduction reports a 69.4% to 31.3% error drop for commonsense reasoning when using the best proxy instead of conventional perplexity.Source: §1 · pp. 2–3

2. Problem Definition and Setup

The setup evaluates whether pre-training indicators can predict post-SFT performance among LLM variants. It creates 50 variants, scores them on three SFT task groups, and evaluates proxies through pairwise comparisons.

2.1. LLM Variants and Target SFT Tasks

  • The study generates 50 1B-parameter variants through continuous pre-training on 100B tokens, varying objectives, domain mixtures, filtering, and tagging.Seven objectives and multiple data-mixture and length-based subsets contribute to the variant set.Source: §2.1 · p. 3
  • The target SFT tasks are commonsense reasoning, retrieval-augmented generation, and closed-book question answering, with dataset scores averaged within each task.CMS uses five datasets, RAG four, and CBQA two.Source: §2.1 · p. 3

2.2. Prediction Proxies

  • The five pre-training proxies are causal and span-corruption perplexity plus k-shot scores for CMS, RAG, and CBQA.Perplexities are inverted so higher proxy values align with better SFT performance; k-shot uses one shot for CMS and five for RAG and CBQA.Source: §2.2 · p. 4

2.3. Pairwise Accuracy as a Measure of Predictive Power

  • Proxy effectiveness is measured by predicting which model in each of the 1,225 unique pairs from 50 variants achieves better aggregated task-level SFT performance.This pairwise evaluation reflects choosing among a small pool of candidate models.Source: §2.3 · p. 4

3. Predictive Power on SFT Tasks

Individual proxies and their combinations improve on conventional perplexity, but no proxy is universally reliable. Prediction accuracy varies with the pre-training difference and target SFT task.

3. Predictive Power on SFT Tasks

  • Kshot-RAG reaches approximately 0.7 accuracy on SFT-CMS and SFT-RAG, while no individual proxy is universally reliable across tested tasks.PPL-CLM reaches 0.3 accuracy on SFT-CMS, and PPL-SC exceeds 0.5 accuracy.Source: §3 · p. 4
  • Averaging all five proxies underperforms Kshot-RAG, while combining PPL-SC and Kshot-RAG matches Kshot-RAG and surpasses PPL-SC.Even the best individual or combined proxies retain pairwise error rates around 30%.Source: §3 · pp. 4–5
  • Proxy accuracy is higher for large objective-driven performance gaps than for smaller variations, and the same model comparison can vary across target tasks.SC-versus-CLM comparisons often reach at least 0.6 accuracy, while SC-versus-SC+CLM is 0.2 on SFT-CMS and at least 0.6 on SFT-RAG and SFT-CBQA.Source: §3 · p. 5

4. Learning to Compare

Learning-to-compare uses pre-training proxies to predict which of two models will perform better after SFT. Combining proxies with LightGBM improves prediction beyond the best individual proxy and generalizes across SFT tasks.

4.1. Formulation

  • The comparator uses five pre-training proxies to form 20 pairwise features and predict which model performs better after SFT, trained with binary cross-entropy.Source: §4.1 · pp. 5–6

4.2. Experiment Setup

  • LightGBM models are trained separately for each SFT task, using 20 runs with random 60%/40% splits of the 50 LLM variants.Source: §4.2 · p. 6

4.3. Results

  • LightGBM improves predictive power by 10% over Kshot-RAG on SFT-CMS and SFT-CBQA while maintaining comparable performance on SFT-RAG.Table 2 reports pairwise prediction accuracy for unsupervised baselines and supervised classifiers across the three SFT tasks.Source: §4.3 · p. 6, Table 2 · p. 7
  • Source-task-trained classifiers typically perform within 2–3% of classifiers trained directly on the target task.The generalization test trains on one SFT task and evaluates on another, using all five proxies as input.Source: §4.3 · p. 7
  • Kshot-RAG is the most influential proxy across tasks, while PPL-CLM contributes despite lower standalone accuracy, possibly through its negative correlation with SFT performance.PPL-SC ranks second for SFT-CMS and PPL-CLM ranks second for SFT-CBQA.Source: §4.3 · p. 7

5. Can Post SFT LLM Performance be Reliably Predicted?

Learning-to-compare is more reliable than perplexity for predicting relative post-SFT performance, but accuracy depends on how different the models are. It can identify top-performing candidates in small pools despite a persistent pairwise error rate.

5. Can Post SFT LLM Performance be Reliably Predicted?

  • Despite doubling perplexity’s prediction accuracy, learning-to-compare retains a 25% pairwise error rate, limiting general applicability.Its reliability also decreases for subtle performance distinctions, though it can help filter an initial candidate set.Source: §5 · p. 7

5.1. Impact of Performance Gaps on Prediction Reliability

  • Accuracy for Kshot-RAG and learning-to-compare rises from near 0.5 for the smallest performance-gap quantile to approximately 0.9 for the largest, while PPL-CLM accuracy declines as gaps widen.Learning-to-compare outperformed both baselines across quantiles, particularly on SFT-CMS and SFT-CBQA.Source: §5.1 · p. 8

5.2. Recall the Best Model from a Small Candidate Set

  • Perfect top-1 recall was achieved on SFT-CMS, SFT-RAG, and SFT-CBQA when considering the top 7, 7, and 8 predicted models, respectively.This supports use for narrowing a candidate pool; Kshot-RAG also showed strong performance.Source: §5.2 · p. 9

6. Related Work

Prior work links pre-training choices and perplexity to downstream capabilities, while also noting settings where perplexity may be unreliable. This study evaluates proxies for post-SFT outcomes among same-size, same-compute LLMs.

Related work

  • Pre-training choices include objectives and corpus composition, and lightweight predictors are desirable given the range of design options.The reviewed work describes pre-training as shaping reasoning, knowledge, and tool use.Source: §6 · p. 9
  • Scaling studies associate lower perplexity with improved downstream capabilities, but token-level perplexity may mask rare or semantically critical token deficits and may not predict some tasks reliably.The cited limits include long-context generation and many-shot in-context learning.Source: §6 · pp. 9–10
  • The study tests perplexity among same-size LLMs trained with the same pre-training compute, finding it unreliable for downstream SFT prediction.It introduces pre-training-accessible proxies and a learning-to-compare framework that combines them to rank candidates.Source: §6 · p. 10

7. Conclusion and Future Directions

The study finds conventional perplexity unreliable for predicting post-SFT performance and reports that alternative proxies reduce prediction error by over 50%. The experiments use 1B-parameter models, and broader generalization remains future work.

7. Conclusion and Future Directions

  • Over 50%: novel unsupervised and supervised proxies reduced relative prediction error compared with perplexity in pairwise prediction across 1B-parameter variants.The proxies were particularly effective between models with large performance gaps and for identifying top candidates.Source: §7 · p. 11
  • Generalization to larger model scales, broader downstream tasks, and additional fine-tuning paradigms remains future work.The authors also call for investigation of more pre-training strategies and proxy metrics.Source: §7 · p. 11

B. Pretraining and LLMs

  • The appendix describes 50 one-billion-parameter LLMs pretrained on 100 billion tokens, varying objectives, dataset composition strategies, and learning rates.Each model’s configuration is provided in Table 4.Source: B. Pretraining and LLMs · p. 18

C. Proxy Predictive Accuracy

The analysis compares proxy accuracy across paired model groups that differ in domain re-weighting or tagging and length filtering, on three SFT tasks.

C. Proxy Predictive Accuracy

  • Predictive accuracy is evaluated for proxies on SFT-CMS, SFT-RAG, and SFT-CBQA using model pairs that share pre-training settings except for domain re-weighting or tagging and length filtering.The results are reported in Figures 6 and 7.Source: C. Proxy Predictive Accuracy · p. 18

D. Classifier Implementation Detail

The learning-to-compare classifier is trained with binary cross-entropy over ordered model pairs. Logistic Regression and Neural Networks are also evaluated alongside unsupervised baselines, with implementation settings specified for both and LightGBM.

D. Classifier Implementation Detail

  • The classifier uses binary cross-entropy over model pairs, with C equal to the number of training pairs, |M_train|(|M_train|−1).The supplied loss passage assumes distinct ordered pairs.Source: D. Classifier Implementation Detail · p. 18
  • Logistic Regression and Neural Networks are compared with unsupervised baselines for predicting SFT-CMS, SFT-RAG, and SFT-CBQA.The supplied passage identifies Table 5 as the results location but gives no values.Source: D. Classifier Implementation Detail · p. 18
  • Figure 6 evaluates PPL-CLM, PPL-SC, and Kshot-RAG accuracy for pairs differing in domain reweighting while other pre-training configurations are fixed.DC-0 through DC-5 denote dataset variants detailed in Table 3.Source: Figure 6 · p. 19
  • The Logistic Regression setup uses L2 regularization with C=1.0, while the neural network has two 32-unit ReLU hidden layers; other settings follow defaults.LightGBM uses binary_logloss and default settings including 31 leaves, learning_rate=0.1, and 100 estimators.Source: D. Classifier Implementation Detail · p. 19

E. Proxy Normalized Importance Score for LightGBM

LightGBM proxy importance is based on loss reduction from feature splits, aggregated across each proxy’s four dimensions and normalized across proxies. The section also points to prompt examples and a figure comparing prediction accuracy across data-length and filtering configurations.

E. Proxy Normalized Importance Score for LightGBM

  • LightGBM feature importance sums the binary-log-loss reduction from splits using a feature across all trees.Proxy importance groups the four dimensions associated with each proxy, sums their feature gains, then normalizes the proxy totals across all proxies.Source: E. Proxy Normalized Importance Score… · pp. 19–21
  • Figure 7 compares PPL-CLM, PPL-SC, and Kshot-RAG accuracy across length and filtering configurations, with other pre-training settings fixed.Tags indicate whether domain tags are added; All, Mid, and Max retain all samples, the 25–75% length quantile, and the longest 25%, respectively.Source: Figure 7 · p. 20

F. Prompts

  • The paper presents example prompts used for the Kshot-CMS, Kshot-RAG, and Kshot-CBQA tasks, showing them in Figure 8, Figure 9, and Figure 10, respectively.Source: F. Prompts · p. 21

G. Supervised Finetuned, Perplexity and Kshot Results of LLMs

The section points readers to the per-model results for supervised fine-tuning, perplexity, and K-shot evaluation. It also includes example prompts used for Kshot-CMS, Kshot-RAG, and Kshot-CBQA.

G. Supervised Finetuned, Perplexity and Kshot Results of LLMs

Learning To Compare

G. Supervised Finetuned, Perplexity and Kshot Results of LLMs

Loading 2504.12491v2…