Source-linked AI summary
Likelihood Ranking doesn't Scale Like Prompting in LLMs
Alessandro Bondielli, Lucia Passaro, Davide Bacciu, Alessandro Lenci
TL;DR
Prompted answering and likelihood-based scoring are common evaluation paradigms, but standard likelihood-based MCQA retains a task-conditioned answer-selection interface. This paper compares prompting with declarative-statement likelihood ranking and finds that their accuracy patterns diverge across scale and instruction-tuning. The findings indicate that these protocols measure related but distinct properties and should not be used interchangeably.
Problem
How declarative-statement likelihood ranking diverges from prompting with scale and instruction-tuning is underexplored, despite the distinction between task-conditioned answer selection and likelihood preferences.
Source: §1 · pp. 1–2Method
The study compares prompted MCQA with likelihood ranking of declarative statements constructed from the same question–answer pairs.
Source: §2 · pp. 2–3Results
Takeaways & Limitations
Prompting, standard likelihood-based MCQA, and declarative-statement ranking measure related but distinct properties and should not be used interchangeably.
Source: §5 · p. 4APS relies on automatically generated statements, and residual surface-form differences, paraphrase competition, or stylistic preferences may affect its likelihood estimates.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
LLM evaluation is commonly performed either by prompting models to produce answers or by scoring candidate outputs with likelihood-based metrics. In multiple-choice QA, however, standard likelihood-based scoring is still conditioned on the question and answer set, and can therefore leverage the same task-conditioned answer-selection interface used in prompting. We study a complementary protocol based on likelihood ranking of declarative statements constructed from the same question--answer pairs. Across 95 decoder-only models, ranging from 0.1B to 104B parameters, and 10 MCQA datasets, we find a systematic divergence between declarative-statement likelihood ranking and prompted answering. Statement-likelihood accuracy remains comparatively stable across scale, whereas prompted answering improves sharply with scale and instruction-tuning. These results suggest that likelihood preferences over controlled declarative alternatives and task-conditioned answer selection probe distinct aspects of model behavior, and should not be treated as interchangeable.
1 Introduction
Prompting and likelihood-based evaluation use different interfaces: prompting asks models to act on task-framed choices, while declarative-statement ranking scores alternatives in a fixed linguistic form. The paper examines how these signals diverge with model scale and post-training, finding that prompted accuracy improves much more sharply than statement-likelihood accuracy.
1 Introduction
- Prompting asks models to produce answers, while likelihood-based evaluation scores candidate outputs with perplexity or related metrics.Prompting commonly uses a multiple-choice format.Source: §1 · p. 1
- Standard likelihood-based MCQA scores options in the original question–answer context, whereas declarative-statement ranking removes that explicit answer-selection interface.Likelihood scores can also be sensitive to formulation and surface form, so the protocols should be distinguished.Source: §1 · p. 1
- How prompting-based and likelihood-based signals diverge with scale and instruction-tuning remains underexplored.Source: §1 · p. 2
- Declarative-statement likelihood accuracy improves slowly with scale, while prompted accuracy rises sharply and instruction-tuning moves their crossover to smaller models.The paper reports this divergence as a function of model scale and post-training.Source: §1 · p. 2
2 Evaluation Setup
The evaluation compares prompted MCQA with likelihood ranking of declarative statements derived from the same question–answer data. It uses two accuracy metrics, 10 MCQA datasets, and 95 open-weights decoder-only models, with statement generation and evaluation procedures specified for the comparison.
2 Evaluation Setup
- APX measures correct answers to explicit prompts, while APS measures cases where the correct declarative statement receives higher likelihood than distractors.The authors distinguish APS from standard likelihood-based MCQA, which scores candidate answers in the original question–option context.Source: §2 · p. 2
- The evaluation covers 10 HuggingFace MCQA datasets unified into a common question, choices, correct-label, and choice-index format.The dataset statistics are summarized in Table 1.Source: §2 · p. 2, Table 1 · p. 2
- For each question, the pipeline generates one affirmative declarative statement per answer option and filters cases without exactly the required number of statements.Validation on a stratified 1% sample reports meaning preservation of 99% under two LLM judges and shared structural templates in 95% of cases.Source: §2 · p. 2
- The study tests 95 open-weights decoder-only models from around 0.1B to 104B parameters, including base and instruction-tuned variants when available.Mixture-of-Experts and reasoning-enabled models are excluded to avoid confounding factors.Source: §2 · p. 2
- Prompted models receive task instructions, the question, and its options; declarative statements are evaluated independently, with the lowest-perplexity statement selected.Prompt responses are parsed to obtain one answer for accuracy scoring.Source: §2 · p. 3
3 Results
Across datasets and model families, declarative-statement likelihood accuracy scales weakly while prompted accuracy rises sharply and eventually surpasses it. Instruction-tuned models reach that crossover earlier, though the crossover varies by dataset.
3 Results
- The scaling curves use the monotonic function minimizing AIC for each metric and model type, with uncertainty estimated by bootstrap and shown as 95% confidence intervals.The fitted curves are reported separately for APS and APX in base and instruct models.Source: §3 · p. 3
- APS improves gradually and remains below the best prompted results, while APX rises sharply with scale and larger models outperform APS.Figure 2 shows this pattern across model scales and families; instruction-tuning has limited effect on APS but substantially benefits APX.Source: Figure 2 · p. 3, §3 · p. 3
- Instruction-tuned models’ APX reaches the APS break-even point substantially earlier than base models, while APS and APX are best fit by rational and log-linear functions, respectively.The break-even point is the threshold where APX surpasses APS.Source: §3 · p. 3
- The APX–APS break-even point varies by dataset: SCIQ requires larger models than OPENBOOKQA, while SCIQ’s APS remains high at 0.65–0.85.The paper suggests this SCIQ pattern reflects more stable likelihood support for correct statements, with lower and more stable PPL.Source: §3 · p. 4
4 Interpreting the mismatch
The paper interprets the mismatch as a difference between diffuse likelihood preferences and the task-conditioned policy needed to turn them into discrete answers. It cautions that statement ranking remains sensitive to surface form and candidate wording.
4 Interpreting the mismatch
- Most models show near-zero PPL differences between their first- and second-preferred answers, suggesting diffuse likelihood support over declarative alternatives.A few outliers appear notably in the 4–10B parameter range.Source: §4 · p. 4
- Prompting requires interpreting instructions, comparing alternatives, and committing to a discrete answer, so prompted accuracy reflects both answer support and the learned policy for expressing it.The paper characterizes that policy as mapping diffuse likelihood preferences to task-appropriate outputs.Source: §4 · p. 4
- Instruction-tuning is interpreted as strengthening the interface between likelihood preferences and task-conditioned behavior more than reshaping the underlying pretraining distribution.Because model size, training data, and post-training procedures are often opaque, the authors attribute the observed gap to their combined effect on the answer-selection interface.Source: §4 · p. 4
- APS is a controlled likelihood signal rather than a calibrated truth estimator, and remains sensitive to surface form, paraphrase competition, and candidate wording.Preserving question wording and structural similarity mitigates these confounds, while further normalization could isolate statement-conditioned preferences more precisely.Source: §4 · p. 4
5 Conclusion
The paper concludes that prompting, standard likelihood-based MCQA, and declarative-statement likelihood ranking measure related but distinct properties. It argues that they should not be treated as interchangeable.
5 Conclusion
- Prompting improvements, particularly those associated with instruction-tuning, reflect behavioral alignment and answer selection at least as much as likelihood preferences over semantically matched alternatives.Source: §5 · p. 4
- Prompting, standard likelihood-based MCQA, and declarative-statement likelihood ranking measure related but distinct properties and should not be used interchangeably.Source: §5 · p. 4
Limitations
The observed divergence is robust across datasets and model families, but the study is bounded by its MCQA focus, open-weight models, scale-only analysis, and generated declarative alternatives.
Limitations
- The study covers only MCQA, and uneven model-family representation across scale may leave residual family-specific effects.MCQA enables a controlled comparison but represents a limited class of tasks.Source: Limitations · p. 5
- The findings come from open-weights models, while training-token exposure is not analyzed and could independently affect results.The authors note that missing training-data information, especially for smaller variants, prevented this analysis.Source: Limitations · p. 5
- Automatically generated statements may retain surface-form or stylistic effects, and fuller comparisons with other likelihood and truth-evaluation protocols remain future work.APS is therefore interpreted as a likelihood signal over controlled declarative alternatives, not a direct or exhaustive measure of knowledge.Source: Limitations · p. 5
- The divergence between prompting-based and declarative-statement likelihood evaluation is robust across datasets and model families.The authors describe this as highlighting fundamental distinctions between evaluation paradigms.Source: Limitations · p. 5
A.1 Declarative Statements Generation
- The protocol uses gpt-oss-20b to generate affirmative declarative statements from the multiple-choice questions.The authors use its HuggingFace implementation with vLLM.Source: §A.1 · p. 7
System prompt
- For blank-containing questions, the system prompt fills each blank with an answer choice; otherwise, it forms an affirmative statement by appending the answer.The prompt requests one affirmative statement per option and preserves the question wording as much as possible.Source: System prompt · p. 7
User prompt
- The user prompt preserves choice order and question wording, uses the default chat template, and sets medium reasoning effort, temperature 0.7, top_p 0.95, and a 1,500-token limit.Generation is requested as one statement per choice.Source: User prompt · p. 7
- Generation errors are checked by counting parsed statements against the available choices; errors are relatively uncommon except on MMLUPRO, which averages 9.45±1.5 choices.Figure 3 reports examples and errors by dataset.Source: User prompt · p. 7, Figure 3 · p. 7
A.2 Statement Validation
The appendix checks semantic fidelity and structural consistency of generated statements, then describes answer distributions across datasets. The validation reports high semantic equivalence and mostly uniform answer distributions, with two dataset exceptions.
A.2 Statement Validation
- 99% of sampled statements were judged semantically equivalent, with raw agreement of 0.98 between the two judges.The sample comprised 900 items, approximately 1% of the dataset, stratified by dataset and length bin.Source: §A.2 · pp. 7–8
- BERTScore F1 averaged 0.68/0.73 mean/median and BLEU 0.69/0.75 mean/median across question–answer pairs.Figures 4 and 5 show the respective score distributions.Source: §A.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
- The median statement was roughly one token shorter than its source question–answer pair, with an average within-item median difference of -1.01 tokens.The mean within-item difference was -9.1 tokens.Source: §A.2 · p. 8
- 95% of items were judged to have structurally consistent statements across answer options.The authors note that the validation results are limited by the sample size but support the generated dataset’s viability.Source: §A.2 · p. 9
A.3 MCQA Answer Distribution
- Answer distributions were uniform in most datasets; BIGBENCHHARD and TRUTHFUL_QA were exceptions associated with greater variability in answer counts per question.Both datasets had much higher answer-count standard deviations than the others.Source: §A.3 · p. 9, Figure 6 · p. 9
B Standard MCQA Likelihood
A control experiment compares prompted answering with standard likelihood-based MCQA, where each option is scored conditioned on the question and full answer set. Their performance trends are similar in this experiment.
B Standard MCQA Likelihood
- Standard likelihood-based MCQA scores each answer option as a perplexity continuation conditioned on the question and the full answer set.The prompt was kept identical for base and instruction-tuned models to limit variation in the comparison.Source: B Standard MCQA Likelihood · pp. 9–10
- Across 51 models, APX and standard option-likelihood scores were moderately correlated (ρ = 0.5444, p = 3.025×10−5), with no significant difference between their scaling slopes (p = 0.58).The models ranged from 0.135B to 67B parameters; Figure 7 visualizes performance across model size and groups.Source: B Standard MCQA Likelihood · p. 10, Figure 7 · p. 10
C Model Inventory
The appendix inventories evaluated model families and describes the evaluation setup and prompts. The prompts use a shared system instruction, with distinct question formulations for instruction-tuned and base models.
C Model Inventory
- The model inventory groups evaluated models by family, and Table 2 reports base and instruction-tuned variant counts and parameter scales.The appendix provides the full model list grouped by family.Source: C Model Inventory · p. 10, Table 2 · p. 10
D. Details of Evaluation Setup
- The evaluation setup keeps prompts short and general rather than tailoring prompting techniques to individual models.When available, prompts are wrapped in each model’s chat template; otherwise, system and question prompts are concatenated.Source: §D.1 · p. 11
System
- The shared system prompt instructs models to provide only the letter of the correct option, without the answer text or commentary.The instruction applies across models.Source: System · p. 11
Question - Instruct
- The instruction-tuned question prompt presents the options and asks, “Which is the correct answer?”The question prompt follows the shared system prompt.Source: Question - Instruct · p. 11
Question - Base
- The base-model prompt presents the question and options, then cues completion with “The letter corresponding to the correct answer is:”.The prompts omit trailing whitespace because several base models struggled with whitespace after the colon.Source: Question - Base · p. 11
D.2 Implementation
The implementation describes the evaluation hardware and response handling for prompted MCQA, then explains how likelihood-based evaluation ranks declarative statements. Figure 8 presents accuracy comparisons across datasets, scales, and model tuning status.
D.2 Implementation
- Models were evaluated locally on A100 80GB GPUs using FP-16 variants and vLLM, with one, two, or four GPUs allocated according to model size.Models smaller than 20B parameters could fit on one GPU; 70–100B models required at least four.Source: §D.2 · p. 11
- Prompted MCQA generates up to 15 tokens at temperature zero, then uses regex matching to extract a single answer and marks multiple-answer responses as errors.Possible matches include an answer letter or the answer text.Source: §D.2 · pp. 11–12
- Figure 8 presents prompting-based and declarative-statement likelihood accuracies by dataset, model scale, and instruction-tuned versus base status.Source: Figure 8 · p. 12
- For likelihood evaluation, each declarative statement is scored without further prompting, and statements are ranked from most likely to least likely.The implementation computes log probabilities for the whole sequence.Source: §D.2 · p. 12
E.1 Per-dataset results
Across datasets, APS scales weakly while APX scales sharply and benefits from instruction-tuning; supporting PPL analyses show correct-answer PPL generally falls with model size, while declarative-choice margins are usually near zero.
E.1 Per-dataset results
- APS scales weakly and is less sensitive to instruction-tuning than APX, whose sharp scaling and instruction-tuning gains recur across datasets rather than reflecting one benchmark.Absolute performance varies by dataset, but the divergence is generally consistent.Source: §E.1 · p. 12
E.2 PPL of correct answer
- Average PPL for the correct answer tends to decrease with model size, though middle-sized models show variability.Figure 9 breaks the average correct-answer PPL down by dataset.Source: §E.2 · p. 12, Figure 9 · p. 12
E.3 PPL delta
- Declarative-statement PPL differences between the lowest and second-lowest scores are mostly near zero, with notable exceptions especially in the 4–10B parameter range.Figure 10 plots this delta against model size.Source: §E.3 · p. 12, Figure 10 · p. 12