Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs

1 Introduction

• MCQA prompts commonly end with “Answer:” so answer extraction can use next-token probabilities for option labels.Source: §1 · p. 1
• Prior practice splits between including the leading space in the prompt and tokenizing it together with the answer letter, with no community-wide convention.Even widely used evaluation frameworks differ in their convention.Source: §1 · p. 1
• Tokenizing the space with the answer letter yields consistent, statistically significant accuracy and calibration gains, with accuracy shifts reaching 11%.The reported accuracy effect exceeds previously observed prompt-formatting variations such as option-order permutation.Source: §1 · p. 1
• Changing the space-tokenization convention reshuffles leaderboard rankings: Llama 3.1 70B Instruct leads without-space tokenization, while Qwen 2.5 72B leads with it.Source: §1 · p. 2
• The paper recommends tokenizing the space together with the answer letter and comparing model rankings only under that configuration.It also calls for unified evaluation frameworks and greater transparency, particularly for closed-source models.Source: §1 · p. 2

2 Related Work

• MCQA is widely adopted for quantitative evaluation because open-ended LLM outputs make evaluation challenging and human or LLM-as-judge assessments can be subjective and unreliable.Source: §2 · p. 2
• Next-token probabilities for answer-label tokens provide a commonly used automated extraction method, while freeform generation plus a secondary LLM can be costly and inconsistent across secondary models.Source: §2 · p. 2
• Although prior studies examine prompt sensitivity and label or answer-order biases, they give little attention to tokenization of the space before the answer label.Source: §2 · p. 2
• Prior studies use both separate-space and space-with-letter tokenization, and evaluation frameworks have not converged on one convention.The passage names Lighteval as an example of differing conventions across datasets.Source: §2 · p. 2

3 Space or No Space?

• The literature has not converged on a convention, and widely used evaluation frameworks can tokenize the leading space differently across datasets.Source: §3 · p. 2
• MCQA prediction selects the answer-label token with the highest conditional next-token probability given the prompt.This supports efficient, automated answer extraction for performance assessment.Source: §3 · p. 2
• With “Letter Token Without Space,” the space stays in the prompt and the model scores a single-letter token matching the label token in the options.Source: §3 · p. 2
• With “Letter Token With Space,” the space is included in the answer token, matching the tokenization produced when the final answer letter is included in the prompt.Source: §3 · p. 2

4 Experimental Setup

• The experiments compare accuracy and calibration under single-letter versus space-plus-letter answer-token scoring.Accuracy is assessed with McNemar’s test and calibration with paired bootstrap resampling on ECE; differences are considered significant at p < 0.05.Source: §4 · p. 3
• The main benchmark is MMLU, supplemented by five other MCQA datasets to test whether findings extend beyond one benchmark.Source: §4 · p. 3
• The study evaluates 15 LLMs with greedy decoding and varies prompt formulations across zero-shot, few-shot, chain-of-thought, option formats, and languages.Source: §4 · p. 3
• Calibration is measured by ECE, the weighted average discrepancy between prediction confidence and accuracy across confidence bins.The experiments use 10 bins.Source: §4 · p. 3
• For each prompt, next-token logits for the option letters are normalized into probabilities, allowing analysis of predictions and confidence distributions.Source: §4 · p. 3

5 Results

• Accuracy improves for all models with space–letter tokenization, with statistically significant gains except for Gemma 3 12B and Mistral 7B.Table 1 reports zero-shot MMLU performance for the two tokenization schemes.Source: §5 · p. 3, Table 1 · p. 4
• Changing only the space tokenization flips the top model from Llama 3.1 70B Instruct to Qwen 2.5 72B.The former leads when the space is tokenized before the letter; the latter leads when it is tokenized with the letter.Source: §5 · pp. 3–4
• ECE is lower for most models with space–letter tokenization, often significantly, with calibration gains from the 30% confidence bin onward.For Gemma 3 4B, the affected confidence bins become closer to perfect calibration.Source: §5 · p. 4, Figure 2 · p. 4

5.1 Few-Shot and Chain-of-Thought Results

• Few-shot prompting preserves the accuracy and calibration improvements from space–letter tokenization.The prompts include five example question-answer pairs using the evaluated tokenization.Source: §5.1 · p. 4
• Under CoT prompting, calibration still improves significantly, but absolute accuracy gains are not significant.The paper attributes lower accuracy sensitivity to easier answer-label extraction after the reasoning chain.Source: §5.1 · p. 4

5.2 Robustness to Prompt Variations

• Across prompt variations, the tokenization effect is consistent and exceeds differences from changing option labels or their order.The reported variants include parentheses, placing choices before the question, and option permutations.Source: §5.2 · pp. 4–5
• The findings remain robust across the tested languages, including some not natively supported by the models.Source: §5.2 · p. 5

5.3 Results on Other Datasets

• Across five additional MCQA datasets, space–letter tokenization systematically raises accuracy and usually lowers calibration error; Qwen 2.5 72B gains over 11% accuracy on HellaSwag.Figure 3 summarizes average accuracy improvement and ECE reduction by dataset.Source: §5.3 · p. 5, Figure 3 · p. 5

• Tokenizing the space with the option letter consistently improves accuracy and calibration, with performance gains reaching up to 11%.Source: §6 · p. 5
• This tokenization change can alter leaderboard rankings, raising concerns about the comparability of prior LLM evaluation results.Source: §6 · p. 5
• The authors encourage careful treatment of low-level evaluation details to support fair and meaningful model comparisons.Source: §6 · p. 5

• The evaluation focuses on open-weight models because proprietary API models do not provide all next-token logits.Source: Limitations · p. 5
• The experiments use small- to medium-sized LLMs up to 72B to accommodate computational constraints, with similar trends observed across them.Source: Limitations · p. 5
• Testing whether the findings hold for large-scale LLMs remains future work.Source: Limitations · p. 5

Ethics Statement

• LLMs remain susceptible to errors despite the reported MCQA evaluation improvements, so critical uses such as medical advice warrant human oversight and domain-specific validation.Source: Ethics Statement · p. 5
• The authors caution against relying solely on LLMs in critical settings such as medical advice.Source: Ethics Statement · p. 5

A Space or No Space? Detailed Motivation of Both Approaches

• The section provides a more in-depth analysis of the rationale for both approaches to tokenizing the space before the answer letter.Source: A Space or No Space? Detailed… · p. 7

A.1 Letter Token Without Space: Token Similarity

• ≈0.6 versus ≈0.2 cosine similarity for space-prefixed correct and incorrect options contrasts with 1.0 versus ≈0.3 for letter-only tokens compared with the correct letter.Source: §A.1 · p. 7, Figure 4 · p. 8
• Using the same letter token as in the option list is argued to make labels more distinguishable and could clarify decision boundaries when the model is uncertain.Source: §A.1 · pp. 7–8

A.2 Letter Token With Space: Model’s Default Tokenization

• The space-plus-letter approach matches the model’s default final token for a prompt ending in “Answer: X.”Source: §A.2 · p. 8

B.1 Datasets

• The evaluation uses each dataset’s default Hugging Face test set, with dataset sizes listed in Table 4.Source: §B.1 · p. 8, Table 4 · p. 8

B.2 Models

• The study evaluates models downloaded from the Hugging Face model hub, with model sizes listed in Table 5.Source: §B.2 · p. 8, Table 5 · p. 8

B.3.1 Main Prompt Templates

• Base-model prompts differ from instruction-tuned prompts by the inclusion of model-specific system, user, and assistant special tokens.The special tokens align instruction-tuned prompts with the models’ expected conversational usage; probabilities come from next-token logits after the arrow.Source: §B.3.1 · p. 8, Figure 5 · p. 9, Figure 6 · p. 9

B.3.2 Prompt Variations

• Prompt robustness tests vary spacing before options, parenthesized or numeric labels, and whether the option list precedes the question.The appendix provides the exact prompts used for these variations.Source: §B.3.2 · p. 9, Figure 7 · p. 9, Figure 8 · p. 9

B.4 Statistical Tests

• Accuracy significance is tested with McNemar’s test, while calibration differences are assessed by paired bootstrap resampling of ECE.Source: §B.4 · p. 10

C.1 Effect of Option Order Permutations

• Tokenizing the space with the answer letter improves results consistently across five option shufflings, with effects larger than differences caused by changing option order.The five-shuffling average appears in Table 3, with individual results in Table 6.Source: §C.1 · p. 10

C.2 Performance across Languages

• Across Spanish, German, French, Hindi, and Chinese, space–letter tokenization yields higher accuracy and lower ECE; in Chinese, the gains exceed 4 accuracy points and reduce ECE by 5 points.The paper notes that Chinese is not natively supported by Llama 3.1.Source: §C.2 · p. 10, Table 7 · p. 10

C.3 Performance across Datasets

• 11.7 accuracy points separate the two tokenizations for Qwen 2.5 72B on HellaSwag, and the advantage appears across datasets and model families.The comparison is between tokenizing the space before the letter (“X”) and together with it (“␣X”).Source: §C.3 · p. 10, Table 8 · p. 11
• Across datasets and model families, the per-model results show a consistent trend favoring space–letter tokenization.Source: §C.3 · p. 10, Table 8 · p. 11
• Table 8 reports full results for all models and datasets; an asterisk marks a significantly better strategy, and bold marks each model’s top-performing strategy.Source: Table 8 · p. 11