PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecasting Downstream Performance of LLMs With Proxy Metrics

Arkil Patel, Siva Reddy, Marius Mosbach, Dzmitry Bahdanau

arXiv:2605.18607v1cs.CLcs.LG

TL;DR

Reliable forecasting is needed for model-development decisions, but loss is poorly aligned with downstream capability and direct evaluation is costly and weak at early stages. The paper builds proxy metrics from token-level predictions over expert solutions, which outperform common baselines across model selection, data selection, and training-time forecasting.

  • Problem

    Model-development decisions lack reliable early signals because cross-entropy loss poorly reflects downstream capability, while direct evaluation is expensive and often uninformative.

  • Method

    The paper aggregates token-level statistics from a candidate model’s next-token distribution while processing expert-written reasoning trajectories.

  • Results

    Across three settings, proxies outperform loss- and compute-based baselines, reaching ρ = 0.81 for cross-family model ranking versus ρ = 0.36 for cross-entropy loss.

  • Takeaways & Limitations

    Expert trajectories provide a broadly useful, task-conditioned signal for assessing capabilities and forecasting downstream performance throughout model development.

  • Takeaways & Limitations

    Pretraining-data selection and downstream extrapolation were not demonstrated on hard reasoning benchmarks, and generative, long-context, and agentic tasks remain untested.

Abstract

from arXiv · show

Progress in language model development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which training recipe to apply. Making these decisions well requires reliable performance forecasts, yet the two commonly used signals are fundamentally limited. Cross-entropy loss is poorly aligned with downstream capabilities, and direct downstream evaluation is expensive, sparse, and often uninformative at early training stages. Instead, we propose to construct proxy metrics by aggregating token-level statistics, such as entropy, top-k accuracy, and expert token rank, from a candidate model's next token distribution over expert-written solutions. Across three settings, our proxies consistently outperform loss- and compute-based baselines: 1) For cross-family model selection, they rank a heterogeneous population of reasoning models with mean Spearman Rho = 0.81 (vs. Rho = 0.36 for cross-entropy loss); 2) For pretraining data selection, they reliably rank 25 candidate corpora for a target model at roughly $10{,}000\times$ less compute than direct evaluation, pushing the Pareto frontier beyond existing methods; and 3) for training-time forecasting, they extrapolate downstream accuracy across an $18\times$ compute horizon with roughly half the error of existing alternatives. Together, these results suggest that expert trajectories are a broadly useful source of signal for assessing model capabilities, enabling reliable performance forecasting throughout the model development life cycle.

1 Introduction

The paper proposes expert-trajectory proxy metrics that use candidate models’ next-token distributions to forecast downstream performance when cross-entropy is task-agnostic and direct evaluation is costly or uninformative. Across model selection, data selection, and training-time forecasting, these proxies provide smooth, task-conditioned signals and outperform existing alternatives.

  • Motivation: Cross-entropy scales smoothly but is poorly aligned with downstream capability, while direct evaluations can be expensive, inaccessible, or indistinguishable across small models and intermediate checkpoints.These limitations motivate proxies that combine smoothness with task conditioning; the approach also avoids generating from the candidate model.
  • Method: The method computes token-level proxy metrics from a candidate model’s next-token predictive distribution while processing an expert solution.Expert trajectories expose many local decisions, allowing partial capability signals even when a model cannot yet solve the task.
  • Empirical results: Mean Spearman ρ = 0.81 versus 0.36 for cross-entropy loss when ranking heterogeneous models on held-out reasoning benchmarks.The best proxy targets cross-family model selection without direct evaluation.
  • Empirical results: Roughly half the extrapolation error is achieved across an 18× compute horizon compared with existing alternatives.This result concerns forecasting downstream accuracy during training.

2 Related Work

Related work extends classical scaling laws toward downstream forecasting and uses small proxy models to guide pretraining-data decisions. The paper’s proxy library comprises token-level core metrics and weighting schemes.

  • Scaling laws and downstream forecasting: Scaling-law research predicts pretraining loss from compute, parameters, and data, while later work extends predictability toward downstream task performance.Approaches include fitting accuracy against compute and mapping validation perplexity to downstream error.
  • Small-scale proxies for pretraining decisions: Small-scale proxy studies rank candidate pretraining corpora before target-scale training by selecting domain weights or data mixtures using inexpensive runs.DataDecide evaluates twenty-five pretraining corpora across fourteen proxy scales in a controlled testbed.
  • Proxy metric library: The proxy metric library combines core token-level metrics with alternative weighting schemes.Table 1 defines p as the model’s next-token distribution conditioned on the input and preceding expert tokens, with y_t denoting the expert token.

3 Method

The method builds cheap downstream-capability proxies from a candidate model’s predictive distribution over expert reasoning trajectories. It aggregates token-level alignment statistics into a library of 80 complementary metrics computed from a single forward pass per instance.

  • Method motivation: The proxies assess how closely a candidate model’s predictive distribution matches expert reasoning at each step, even when the model’s own generation fails.The design targets signals that are both indicative of downstream capability and cheap to evaluate.
  • Core metrics and weighting: Ten core metrics capture agreement, distribution concentration, and confidence when disagreeing, while eight weighting schemes emphasize different notions of token importance.Each metric is aggregated as a weighted average because token positions are not equally diagnostic.
  • Proxy construction: Combining 10 core metrics with 8 weighting schemes yields 80 proxy metrics, each assigning a scalar score to a candidate model on a task.The sign convention ensures that higher proxy values indicate a better model.
  • Proxy construction: The full proxy library is extracted from a single forward pass per instance, providing 80 complementary views of candidate–expert alignment at low computational cost.The task-level proxy is obtained by averaging instance-level values.
  • Computing proxies in practice: In experiments, proxies use the last 1,000 tokens of each expert trajectory, retain trajectories with incorrect answers, and average across experts and instances when available.Using the last 1,000 tokens empirically outperforms using the full trace.

4 Cross-Family Model Selection: Ranking LLMs on Unseen Tasks

Across 18 reasoning models from six families and six post-training recipes, proxy metrics rank downstream performance on six challenging benchmarks substantially better than loss-based baselines. The full linear RankSVM proxy achieves mean Spearman ρ = 0.81, while generic-text CE loss and rBridge reach only 0.36 and 0.33.

  • Experimental setup: The evaluation covers 18 reasoning-capable models spanning six families, six post-training recipes, and 0.6B–70B parameters across AIME 2025, HMMT, GPQA, USACO, MMLU-Pro, and SuperGPQA.The benchmarks cover competition math, graduate-level reasoning, coding, and broad knowledge evaluation.
  • Analysis: No single proxy metric is universally predictive: in an oracle selection setting, the best metric reaches mean ρ = 0.62, with per-task correlations from 0.43 to 0.81.A linear combination is therefore evaluated alongside individual metrics and sparse combinations.
  • Results: ρ = 0.36 for CE loss on generic FineWeb text and ρ = 0.33 for rBridge, showing that loss-based baselines fail to rank reasoning models.CE uses 10M randomly sampled FineWeb tokens, while rBridge reweights likelihood along an expert reasoning trace using expert logprobs.
  • Results: ρ = 0.81 for the full linear RankSVM proxy, tied by the RBF variant and exceeding the 3-sparse proxy at 0.78, the univariate proxy at 0.54, and both loss-based baselines.These correlations are aggregated across 15 leave-2-tasks-out folds and 20 model-subsampling seeds; the relationship is monotonic across all six tasks in a held-out fold.
  • Analysis: The strongest ranking signal concentrates on a few metrics: inverse-frequency-weighted top-1 accuracy dominates univariate selection, while entropy-weighted entropy and frequency-weighted top-5 accuracy are most frequent in 3-sparse selection.These metrics capture agreement with experts on rare tokens and model uncertainty at individual positions.

5 Pretraining Data Selection: Ranking Datasets with Smaller LMs

The study ranks 25 pretraining corpora using small proxy models and expert-trajectory token statistics. Frequency-weighted top-5 accuracy exceeds 0.85 decision accuracy at roughly 10^-5 of target compute, establishing a new DataDecide Pareto frontier.

  • Experimental setup: The DataDecide testbed contains 25 corpora, with rankings defined by mean OLMES accuracy from corresponding 1B-parameter target models.Proxy models range from 4M to 90M parameters, while OLMES comprises ten multiple-choice benchmarks.
  • Evaluation: Ranking quality is measured by decision accuracy, the fraction of corpus pairs whose proxy ordering agrees with the target-scale ordering.Compute cost is expressed as the fraction of target training FLOPs, using FLOPs = 6ND; 4M-parameter proxies require roughly 10^-5 of target compute.
  • Results: At roughly 10^-5 of target compute, frequency-weighted top-5 accuracy exceeds 0.85 decision accuracy and dominates rBridge at almost every matched budget.Figure 1 compares decision accuracy against compute budget; the downstream baseline requires more than 10^-1 of target compute for comparable accuracy.
  • Why proxies work: Token-level compatibility with expert CoT trajectories varies across corpora before benchmark accuracy leaves its noise floor, enabling discrimination at small scales.A model may track an expert-written solution even when it cannot solve the underlying problem or produce a correct answer.

6 Training-time Forecasting: Proxy Metrics Facilitate Extrapolation

Proxy metrics support reliable training-time forecasting by following smooth power-law trajectories and outperforming cross-entropy loss and compute baselines when extrapolating downstream accuracy far beyond the fitting horizon.

  • Motivation and baselines: The approach addresses the need to forecast end-of-training capability from early checkpoints, requiring proxy metrics that evolve predictably with training compute.Prior approaches fit exponentials over validation loss or sigmoids over log-compute, but most such fits reportedly fail to extrapolate reliably.
  • Power-law trajectory fitting: The study evaluates OLMo-3-7B pretraining and OLMo-3-7B-Think post-training checkpoints, selecting proxies through an inner split before fitting the trajectory.Pretraining uses six reasoning benchmarks, while post-training excludes SuperGPQA because the model shows negligible improvement there.
  • Power-law trajectory fitting: In pretraining, the selected proxy followed a smooth power law across all six reasoning benchmarks, achieving mean NMAE 0.03 while extrapolating to approximately 4× compute.The fit used checkpoints through 350K steps; post-training extrapolation reached nearly 2× compute with mean NMAE 0.038.
  • Extrapolation against existing predictors: Mean RMSE was 0.024 for proxy-based fits across ten OLMES benchmarks, roughly half the CE-loss (0.059) and compute-based (0.055) predictors.The evaluation fits predictors through 80K steps and forecasts accuracy at 1.4M steps, roughly an 18× horizon.

7 Conclusion

Proxy metrics derived from expert reasoning trajectories forecast downstream capability across model selection, pretraining-data selection, and training-time extrapolation. The conclusion also identifies open questions about generalization, expert quality, and learned aggregation.

  • Main findings: ρ = 0.81 versus ρ = 0.36 for cross-entropy loss in cross-family selection across 6 benchmarks and 18 models spanning 6 base families.A linear ranker over token-level proxy statistics was evaluated with leave-two-tasks-out cross-validation.
  • Main findings: Above 0.85 decision accuracy at roughly 10^-5 of target compute when ranking 25 pretraining corpora with a single univariate proxy.The method requires only the expert’s tokens, not its logprobs, and displaces the prior Pareto frontier on DataDecide.
  • Main findings: Across an 18× compute horizon, proxy-to-accuracy fits extrapolate downstream performance along OLMo-3-7B’s pretraining trajectory with roughly half the error of loss-based or compute alternatives.
  • Limitations and future work: Whether proxy metrics generalize to mixture-of-experts architectures, long-context retrieval, agentic evaluation, and model scales beyond a single training trajectory remains unresolved.The conclusion also flags uncertainty about how signal degrades with weaker or noisier experts.
  • Limitations and future work: The fixed library of eighty proxy metrics was selected post hoc, leaving end-to-end learning of the aggregation function as an open direction.

8 Limitations … A.2 Model details

The paper’s evidence is limited by model and task coverage, and its appendix defines the proxy metrics, weighting schemes, evaluated models, and standardized generation settings. These details frame the results as setting-dependent rather than universally transferable.

  • 8 Limitations: Training-time forecasting uses only OLMo-3-7B and OLMo-3-7B-Think checkpoints, while cross-family ranking covers 18 models across six families and six post-training recipes.The cross-family population spans most major open-weight reasoning-capable models available during the study, but remains modest.
  • 8 Limitations: No single proxy metric dominates across cross-family ranking, data selection, and benchmark extrapolation, because the best choice depends on benchmarks, model populations, and selection hyperparameters.Relevant hyperparameters include the inner-split fraction and training-window size.
  • 8 Limitations: Task coverage differs by experiment: cross-family ranking uses reasoning benchmarks, whereas data selection and downstream extrapolation use the non-reasoning OLMES suite.This split reflects resource constraints, including DataDecide’s controlled support for OLMES and the lack of comparable reasoning infrastructure.
  • A.1 Proxy metric definitions: The appendix defines seven core token-level metrics spanning expert agreement, predictive peakedness, and error severity, including loss, top-k accuracy, entropy, rank, reciprocal rank, margin, and wrong-confidence mass.Top-k accuracy uses k ∈ {1, 2, 3, 5}, and entropy is vocabulary-normalized.
  • A.1 Proxy metric definitions: Per-token weighting downweights uninformative positions and aggregates metrics through eight schemes, including entropy, inverse entropy, frequency, inverse frequency, and a Gaussian-NLL kernel.The motivation is that function words and punctuation are predicted well by most models and can drown out signal.
  • A.2 Model details: The cross-family experiment evaluates 18 reasoning-capable models from 0.6B to 70B parameters across six base families and six post-training recipes.The evaluated families include Qwen3, Qwen2.5, Llama3, Ministral, OLMo3, and SmolLM3.
  • A Details of Experiments: All models use maximum 28000-token generations, generally with temperature T = 0.6 and top-p = 0.95, while benchmark evaluations average five random seeds.Top-k sampling is disabled, with model-specific overrides where needed, such as Ministral ≤8B using T = 0.7 and a special system prompt.

A.3 Details of Benchmarks

The experiments use six challenging reasoning benchmarks and the OLMES suite, which aggregates ten established general-purpose evaluation datasets. These benchmarks cover mathematical, scientific, programming, commonsense, and other reasoning tasks.

  • A.3 Details of Benchmarks: Six challenging reasoning benchmarks cover AIME 2025, HMMT, GPQA, USACO, and computer-science and engineering subsets of MMLU-Pro.The benchmark collection contains 30 AIME 2025 problems, 93 HMMT problems, 448 GPQA problems, 307 USACO problems, and 1379 MMLU-Pro problems.
  • A.3 Details of Benchmarks: The benchmark choices span mathematical, scientific, competitive-programming, computer-science, engineering, commonsense, and social reasoning evaluations.The six challenging reasoning benchmarks are used in Sections 4 and 6.1, while OLMES is used in Sections 5 and 6.2.
  • A.3 Details of Benchmarks: OLMES comprises MMLU, HellaSwag, ARC Challenge, ARC Easy, PIQA, CommonsenseQA, SocialIQA, OpenBookQA, BoolQ, and WinoGrande.OLMES is used in experiments in Sections 5 and 6.2.

A.4 Other Experimental Details · A.5 Software and Compute Requirements · B Additional Results and Discussion

The appendix documents experimental settings, reproductions, expert and prompt choices, and the software and hardware used to compute proxy metrics and benchmark results. Experiments used open-weight experts for reliable logprob access and ran primarily on a four-GPU H100 cluster.

  • A.4 Other Experimental Details: Experiments in Section 5 use DataDecide evaluation results from default-seed checkpoints, with downstream-task baseline results taken directly from DataDecide.
  • A.4 Other Experimental Details: The rBridge (Koh et al., 2026) comparison reproduces their approach while replacing GPT-4o (OpenAI et al., 2024) with the open-weight Llama-3.3-70B expert for reliable logprob access.
  • A.4 Other Experimental Details: Section 6.1 uses MiniMax-M2.5 and Qwen3Next-80B as experts, fitting pretraining extrapolation through step 350000 and testing it at step 1300000 with an inner-split fraction of 0.5.
  • A.4 Other Experimental Details: For pretraining checkpoints, proxy metrics use a standard chain-of-thought prompt that appends “Let’s think how to answer this question step by step.” before the expert trajectory.
  • A.5 Software and Compute Requirements: The implementation uses PyTorch (Paszke et al., 2019), HuggingFace Transformers, and vLLM (Kwon et al., 2023) for local efficient inference.
  • A.5 Software and Compute Requirements: Benchmark evaluation and proxy-metric calculations run on a cluster with 4 NVIDIA H100 GPUs with 80 GB memory, while Kimi-K2.5 experiments use the Together API.

B.1 Cross-Family Model Selection

Cross-family analysis shows that a small set of proxy metrics captures most of the ranking signal, with sparse proxies remaining accurate under task and model subsampling while loss-based baselines provide weak, non-monotonic rankings.

  • Which proxy metrics carry the ranking signal?: Inverse-frequency-weighted top-1 accuracy accounts for 32% of univariate selections, while entropy-weighted entropy and frequency-weighted top-5 accuracy account for 0.16 and 0.14 of 3-sparse selections.Selection mass concentrates in a small cluster of metrics, emphasizing rare-token correctness, uncertainty at difficult reasoning positions, and frequency-adjusted accuracy.
  • Oracle upper bounds: ρ = 0.88 for the 3-sparse oracle, versus ρ = 0.62 for the univariate oracle, with per-task 3-sparse correlations between 0.78 and 0.97.These oracle values upper-bound cross-validated performance because the proxies are fit and evaluated on the same benchmarks and model population.
  • Loss-based baselines: FineWeb cross-entropy yields ρ = −0.52, uniform expert-trajectory cross-entropy ρ = −0.27, and rBridge ρ = −0.33 against MMLU-Pro accuracy.The three loss-based signals produce overlapping, weak, and non-monotonic rankings across model families and post-training recipes.
  • Robustness to held-out tasks and model fraction: ρ = 0.81 with K = 2 held-out tasks and 60% of models, while ρ = 0.74 remains achievable with K = 3 and 50% of models.Performance degrades gracefully as fewer tasks and models are available, and variance decreases with more models.

B.2 Training-time Forecasting

Proxy metrics extrapolate downstream performance across pretraining and post-training, including an approximately 18× compute horizon, while tracking downstream accuracy rankings. They outperform direct sigmoid and cross-entropy baselines, though post-training extrapolation is noisier on AIME.

  • Pretraining extrapolation: NMAE = 0.007 on AIME and 0.035 on SuperGPQA for pretraining extrapolation, with power-law fits reaching held-out checkpoints despite a larger SuperGPQA gap.The fits have train-window R2 = 0.93 on AIME and R2 = 0.97 on SuperGPQA.
  • Post-training extrapolation: NMAE = 0.110 and R2 = 0.88 on post-training AIME, the noisiest setting and roughly three times the post-training average on the other four benchmarks.The section attributes this difficulty to AIME’s task noisiness and its only 30 problems.
  • Proxy-accuracy correspondence: ρ = 0.84 across five post-training benchmarks, with proxy values strongly tracking downstream accuracy rankings across checkpoints.The relationship is strongly monotonic on USACO, with Spearman ρ = 0.91; Figure 12 provides the checkpoint-level sanity check.
  • Proxy selection: Nine of ten OLMES tasks select top-k accuracy proxies, with k ranging from 1 to 5; PIQA instead selects probability-weighted reciprocal rank.The selections come from the inner-split procedure and use different weighting schemes.
  • Baseline comparisons: RMSE = 0.01 for proxy-based Winogrande extrapolation and RMSE = 0.02 for ARC Challenge, outperforming sigmoid and cross-entropy fits at the held-out checkpoint.For Winogrande, sigmoid and cross-entropy achieve RMSE = 0.02 and 0.08; for ARC Challenge, they achieve RMSE = 0.07 and 0.13.

C Extended Related Work

Prior work forecasts downstream performance from compute or pretraining loss, but realistic constraints expose limited reliability. Other research instead uses continuous, token-level, cross-distribution, small-scale, and reasoning-trace signals to improve prediction or guide pretraining decisions.

  • Compute-based scaling laws for pretraining loss: Classical scaling laws relate compute, parameters, and tokens to pretraining cross-entropy loss, providing the foundation for downstream forecasting pipelines (Hoffmann et al., 2022, Kaplan et al., 2020).These laws are powerful planning tools when loss itself is the target, using power laws with additive terms in N and D.
  • Forecasting downstream task performance: Downstream forecasting maps pretraining loss or compute to task accuracy, but only about 39% of tasks in Gadre et al. (2025) showed predictably linear accuracy scaling (Lourie et al., 2025).Proposed approaches include exponential mappings, two-stage task-loss forecasts, latent-skill models, and direct two-parameter compute laws (Gadre et al., 2025; Bhagia et al., 2025; Krajewski et al., 2026).
  • Emergence and the role of continuous metrics: Continuous metrics can replace apparent emergent jumps: Schaeffer et al. (2023) report that emergence largely evaporates under smooth measures such as log-likelihood, Brier score, and edit distance.Wei et al. (2022a) coined the emergent-abilities framing, which many interpreted as evidence against forecastability.
  • Loss-to-loss and cross-distribution prediction: Loss-to-loss prediction extrapolates shifted power-law relationships across datasets and up to roughly twenty times training compute, with robustness across architectures and tokenizers (Brandfonbrener et al., 2025; Mayilvahanan et al., 2025).Related cross-family evidence reports Pearson correlation approximately −0.95 between bits-per-character and downstream scores across thirty-one LLMs and twelve benchmarks (Huang et al., 2024).
  • Token-level likelihood signals and fine-grained weighting: Token-level and small-scale proxies use prompt or document perplexity, reference-model signals, or smaller proxy models to predict tasks and inform data or recipe choices (Gonen et al., 2023; Ankner et al., 2025; Thrush et al., 2025; Wortsman et al., 2024).Reasoning traces also serve as inference or training signals through chain-of-thought, self-consistency, STaR, Quiet-STaR, and process reward models (Wei et al., 2022b; Wang et al., 2023; Zelikman et al., 2022, 2024; Uesato et al., 2022).
Loading 2605.18607v1…