PaperlayerView exact revision on arXiv

Source-linked AI summary

Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation

Jiaxuan Guo, Kejia Zhang, Shuo Xin, Jingxin Yang, Youran Sun, Haizhao Yang

arXiv:2609.33179v1cs.CLcs.AIcs.LG

TL;DR

Constrained-option scoring can return a score even when the model is about to emit something outside the declared answers, making it unclear whether that score tracks the intended target. This paper measures option support and tests the quoted instruction’s role, then finds that an answer stem improves the forecast’s ranking of reader correctness while the original-position forecast remains at chance.

  • Problem

    Constrained-option scoring returns a score over permitted answers even when the model’s next token may be something else, so the score’s relation to the intended target needs testing.

    Source: §1 · p. 1, §7 · p. 9

  • Method

    The paper measures in-context probability mass on declared options’ distinct first tokens, tests the quoted instruction by deletion, and compares the original and stemmed forecast positions.

    Source: §1 · p. 2, §3 · p. 4, §6 · p. 8

  • Results

    0.4929 to 0.6152: the Qwen3-averaged AUC rises after an answer stem, while the original-position forecast is indistinguishable from chance.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Option mass flags low support without labels, but only checking the intended target shows whether a score still ranks correctly.

    Source: §7 · p. 9

    The main prompt and answer-stem result are scoped to the tested prompt; measurements require log-probabilities at a chosen position and do not address API-only models.

    Source: §7 · p. 9

Abstract

from arXiv · show

Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study a prompt that quotes a multiple-choice item, ending in the item's own answer instruction, and asks for a forecast of whether a reader model given only a short note will answer it correctly. The model can begin answering the quoted item: on three Qwen3 checkpoints an answer letter is the most probable token on nearly every passage, and the forecast ranks the reader's correctness indistinguishably from chance. Beyond prior work's argmax check and prefill repair, we contribute a label-free diagnostic of the declared options' support, their first-token mass in context; a deletion test tracing the Qwen3 answer-letter start to the quoted instruction; and rendering and tokenisation checks for dropped separators and coinciding first tokens. Prefilling an answer stem lifts the median option mass from below to above one half on all nine checkpoints scored with this prompt and, with options and renormalisation unchanged, raises the Qwen3-averaged AUC from 0.4929 to 0.6152. lm-polygraph's default P(True) estimator shows a related failure: it reads True where Qwen3 without thinking favours an answer letter (MMLU) or the prompt's (A)/(B) label (TriviaQA), and its Qwen3-averaged AUROC is indistinguishable from chance on MMLU and below chance on TriviaQA. Scoring and renormalising the (A)/(B) labels after an answer stem raises it to 0.632 and 0.868, renormalising True against False in place reaches 0.660 and 0.860, and on TriviaQA both exceed all fourteen default single-answer library estimators. The original forecast is renormalised too, yet stays indistinguishable from chance: option mass flags low support in both settings without labels, and only checking the intended target shows which score still ranks.

1 INTRODUCTION

A constrained-option score can report a forecast even when the model first answers the multiple-choice item quoted in the prompt. In this setup, the original forecast ranks reader correctness indistinguishably from chance, while an answer stem improves the Qwen3-averaged AUC.

1 INTRODUCTION

  • The prompt quotes an item with its own letter-answer instruction, then asks for a forecast over permitted numbers; the model can answer the quoted item first.The scorer reads the position after the assistant turn opens and returns a distribution over the permitted forecast numbers.Source: §1 · pp. 1–2
  • 0.4929 [0.4390, 0.5458] Qwen3-averaged AUC at the original position is indistinguishable from chance; an answer stem raises it to 0.6152 on 240 shared passages.Meta-Llama-3-8B-Instruct’s AUC rises from 0.5231 to 0.6044.Source: §1 · p. 1
  • The contributions include an option-mass diagnostic, deletion tests localising Qwen3’s answer-letter start to the quoted instruction, and rendering and tokenisation checks.The introduction also reports that the default lm-polygraph P(True) score ranks correctness at chance on MMLU and below chance on TriviaQA.Source: §1 · p. 2

2 RELATED WORK

Related work distinguishes answer correctness from string probability, studies instruction confusion and prompt formatting, and proposes readout and calibration checks. This paper places its failure at the scored position, where an unlisted answer can take the next-token position.

2 RELATED WORK

  • Prior work distinguishes sentence probability from answer correctness and shows that a model can assign its highest probability to a choice symbol rather than the intended answer.The paper’s quoted item adds its own labels and instruction, which can take over a position scored only for forecasts.Source: §2 · p. 2
  • Related work addresses option ranking, response perturbations, and answer-mode failures, but the paper’s setting concerns a quoted item whose instruction can control the scored position.Its numeric options retain ordinal structure, unlike repairs that use arbitrary symbols.Source: §2 · p. 3
  • Studies of instruction-like input and indirect prompt injection examine data–instruction confusion, while this paper reads the scorer’s side of that interaction.Here, the quoted instruction and scorer’s question share one user turn, and answering the quoted item first can put its answer letter at the scored position.Source: §2 · p. 3
  • Prior prompt-format studies report whitespace-related accuracy and ranking changes and recommend tokenising the space with the answer letter.For numeric labels, their two-element split uses the final element, where digits remain distinct, rather than the shared first element.Source: §2 · p. 3
  • Prior readout studies compare first-token scores with generated text, while this paper examines a declared set that excludes the token at the scored position.The paper frames its case as a gating failure: the model’s next token is not among the scored candidates.Source: §2 · p. 3
  • Unlike calibration and selection-bias methods that assume the scored position is correct, this work measures probability mass outside the declared option set.Prior work also measures whether the unrestricted highest-probability token belongs to the option set; restricting the argmax would remove that diagnostic while leaving option mass measurable.Source: §2 · p. 3

3 TASK, MEASUREMENTS AND EXPERIMENTS

The experiments test whether a forecasting model’s score tracks the correctness of a note-only reader. They measure both the next-token distribution at the rendered prompt and the reader-correctness forecast derived from full-option log-probabilities.

3 TASK, MEASUREMENTS AND EXPERIMENTS

  • The study evaluates forecasts of a note-only reader’s correctness using items from RACE-H passages of at least 200 words and multiple samples and note conditions.The main text includes 64 items with a fixed note, 120 constructed-note rows on Qwen3-4B, and 240 passages with checkpoint-written notes.Source: §3 · pp. 3–4
  • Option mass sums the probabilities of the options’ distinct first tokens when encoded as continuations of the rendered prompt; it upper-bounds probability on the full option sequences.The unrestricted argmax is recorded separately, and the answer-letter rate counts items whose argmax is an A–D letter.Source: §3 · p. 4
  • A median option mass below 0.5 is called collapse, but whether a score ranks the intended target remains a separate measured question.Shared first tokens cannot distinguish options even at high mass; the unrestricted continuation shows what the model writes.Source: §3 · p. 4
  • The note-only reader selects the lettered answer with the highest full-sequence log-probability, and matching gold defines the reader’s correctness outcome.The forecast is evaluated as a probability of that outcome.Source: §3 · p. 4
  • The forecast combines full-option log-probabilities into a probability of reader correctness, then evaluates ranking with AUC and probability error with Brier score.Renormalisation produces a weighted mean even when the options’ total probability is small.Source: §3 · p. 4

4 WHICH INSTRUCTION CONTROLS THE SCORED POSITION?

The quoted item’s answer instruction can control the scored position, so the model starts answering the item instead of forecasting. Deleting that instruction removes the answer-letter start on five checkpoints, while lm-polygraph’s default P(True) readout also scores tokens the model is not about to emit.

4 WHICH INSTRUCTION CONTROLS THE SCORED POSITION?

  • On Qwen3-4B, an answer letter is the unrestricted argmax on 118 of 120 constructed-note rows, while no forecast option’s first token is the argmax.Answer-letter starts also occur on 58 of 60 rows whose notes concern an unrelated passage.Source: §4 · p. 4
  • Deleting the quoted instruction reduces the answer-letter rate to 0.000 on the three Qwen3 checkpoints and from 0.469 to 0.016 on Mistral.On Phi-3, the rate falls to 0.500 and reaches 0.000 only when the choice list is also removed.Source: §4 · pp. 4–5
  • Three judges label all 32 Qwen3 continuations per checkpoint as item answers with the quoted instruction intact and as forecasts after its deletion.Two simple rules—matching the replacement instruction’s opening word and predicting its first response token—do not explain the changed outputs.Source: §4 · pp. 5–6
  • At the original prompt position, lm-polygraph’s P(True) confidence ranks correctness indistinguishably from chance on MMLU and below chance on TriviaQA.Qwen3-4B’s respective library values are 0.423 [0.369, 0.477] and 0.185 [0.147, 0.226].Source: §4 · p. 6

5 HOW OFTEN THE READOUT FAILS ACROSS CHECKPOINTS

Across checkpoints, constrained-option readouts often assign little probability to the declared options, while answer letters or formatting tokens can occupy the scored position. An answer stem raises all nine reported checkpoint medians above the 0.5 floor for this prompt.

5 HOW OFTEN THE READOUT FAILS ACROSS CHECKPOINTS

  • All nine checkpoint medians start below the 0.5 floor, and the answer stem lifts them above it.The nine include the seven checkpoints in Table 3, Llama-3-8B-Instruct, and Qwen3-32B.Source: §5 · pp. 6–7
  • On six of seven checkpoints, every item is below the floor and no unrestricted argmax is a forecast option’s first token.SmolLM2 is a partial reproduction: its median is 0.3895, 50 of 64 items are below the floor, and 18 argmaxes are option first tokens.Source: §5 · p. 6
  • Answer letters occupy the scored position on most items for the three Qwen3 checkpoints, Phi-3, and SmolLM2; missing final whitespace instead puts a space token there on Mistral and falcon.Supplying the space reduces Mistral’s answer-letter rate to 1 of 64; falcon’s remains 0.000.Source: §5 · p. 7
  • Phi-3’s median option mass of 5.79 × 10^-4 does not establish separability because its options share a first token in context.Its answer-letter rate of 0.984 independently indicates the readout failure.Source: §5 · p. 7
  • Two model-written forecast templates remove letter pull, but each leaves three checkpoint medians below the floor for other reasons.The remaining cases involve missing separators, shared first tokens, or an option at the argmax despite low mass.Source: §5 · p. 7

6 MOVING THE SCORED POSITION AND CHECKING THE READOUT

Moving the scored position with an answer stem improves forecast ranking on the reported Qwen3 average, but the improved Brier scores remain worse than a constant 0.5 forecast. Readout reliability also depends on how the prompt ending and options tokenize.

6 MOVING THE SCORED POSITION AND CHECKING THE READOUT

  • Across the same 240 passages, the paired Qwen3-averaged AUC gain from original to stemmed position was 0.1223 [0.0449, 0.1993] under one readout; Qwen3-14B’s own interval included zero.The stemmed Qwen3-average Brier score was 0.2544, worse than 0.2500 for a constant 0.5 forecast.Source: §6 · p. 8, Table 4 · p. 8
  • Llama-3-8B-Instruct’s AUC rises from 0.5231 to 0.6044, a paired difference of 0.0813 [0.0046, 0.1574].The two readouts use the same model, note, item, and option first-token IDs.Source: §6 · p. 8
  • On five checkpoints, medians rise from at most 0.390 to at least 0.888 after adding the stem, even though context ends in whitespace both before and after.On Phi-3, the stem also restores distinct first tokens by ending the context in a space rather than a newline.Source: §6 · p. 8
  • Encoding options as continuations of the rendered prompt avoids isolated-encoding errors, but a prompt ending can still make their first tokens coincide.Among the tested forms, Phi-3’s six unst emmed combinations share a first token; its three stemmed combinations have distinct first tokens.Source: §6 · pp. 8–9
  • Adding a space to G-Eval’s options raises Qwen3-4B’s first-token mass from 1.26 × 10^-5 to 0.99449, but all five options then share the space token.Thus the prefix trades a separator defect for a shared-first-token defect.Source: §6 · p. 9
  • The six checks cover in-context option mass, unrestricted argmax and continuation, distinct option tokens, prompt separators, and retesting after a stem.The final check includes whether downstream ranking improves, since increased option mass alone does not establish that result.Source: §6 · p. 9

7 LIMITATIONS AND CONCLUSION

The main comparisons are scoped to RACE-H, one forecast template, 64 items per checkpoint, and one fixed note; the method requires chosen-token log-probabilities and does not address API-only models. The paper recommends checking the unrestricted continuation, tokenised options, and rendered ending before testing whether the score tracks its intended target.

7 LIMITATIONS AND CONCLUSION

  • The main mass and token-class comparisons use RACE-H, one forecast template, 64 items per checkpoint, and one fixed note; the nine-checkpoint stem result is scoped to Table 3’s prompt.Scale comparisons vary only within Qwen3 and establish no size trend across families; SmolLM2 tests specificity to Qwen3, not to small models.Source: §7 · p. 9
  • The method requires log-probabilities for chosen tokens at a chosen position, so its measurements do not address API-only models.Two observations also remain unexplained: deleting Mistral’s imperative lowers median option mass, and token-class rules do not predict which class replaces answer letters.Source: §7 · p. 9
  • Inspect the unrestricted continuation, tokenised options, and rendered ending to identify what is answered and scored, then test whether that score tracks the intended target.The configuration can arise in self-evaluation and judge scoring when quoted material carries its own answer instruction but the scorer reads a different declared set.Source: §7 · p. 9

A DEFINITIONS AND SAMPLING

This appendix specifies the option-mass measurement and sampling, then documents prompt renderings, tokenisation and boundary checks, and readings across checkpoints and prompt variants. These details show how the measured support depends on the rendered context and whether options have distinct first tokens.

A DEFINITIONS AND SAMPLING

  • When option token sequences are prefix-free, their full probabilities sum to at most µ(c); shared first tokens make list-based sums misleading and scoring after a shared prefix measures a different position.On Qwen3-4B’s 16 ladder items, the list-based median is 4.9850 versus a set sum of 0.997; Phi-3’s median is 0.0368 rather than 0.9334 on the 64-item stemmed prompt.Source: A DEFINITIONS AND SAMPLING · p. 12

A.1 SAMPLING

  • The RACE-H sample keeps 19,566 passages of at least 200 words, shuffles them with random.Random(0), and selects one question for each of the first 3,052 passages.Every sample is a prefix of this list, including 64 items per checkpoint and the judge’s first 32 shared by all 49 checkpoint–ending combinations.Source: §A.1 · p. 13

A.2 REPRODUCTION SNAPSHOTS

  • The reproduction snapshots pin model and tokenizer files to recorded revisions; four checkpoints reproduce item for item on their original GPU model, while three were re-read on a second GPU model.The cited harness counts and source locations use commit d6de81643928d653435c431bae19945d41d32520.Source: §A.2 · p. 13

B THE PROMPT, RENDERED

  • The rendered forecast prompt ends after the quoted item’s letter-answer instruction, while the original scored position is after the assistant opening; the stem adds 11 tokens before a different scored position.On the shown item, the stemmed argmax is ’5’ at 0.98907 and the declared set holds 0.9999999 of the mass.Source: B THE PROMPT, RENDERED · pp. 13–14

C.1 CROSS-FAMILY READINGS

  • On 58 of 60 rows with unrelated notes, an A–D letter remains the unrestricted argmax; when notes state the answer, forecast and reader-task argmaxes match on 59 of 60 rows.Across two prompts and three Qwen3 checkpoints, option mass and the argmax check disagree on 13 of 1,024 readings; on the other 14 checkpoint–prompt pairs they agree on every item.Source: B THE PROMPT, RENDERED · p. 14, §C.1 · p. 14

C.2 DETAILS OF ANSWER-STEM AND BOUNDARY CHECKS

  • The answer stem restores distinct first tokens on Phi-3 and supplies the missing separator on Mistral and falcon; it lifts medians on five other checkpoints from at most 0.390 to at least 0.888.Deleting the quoted instruction reduces Mistral’s answer-letter rate from 30 of 64 to one; supplying the omitted separator with the instruction kept reduces it to one as well.Source: §C.2 · pp. 14–15, Table 5 · p. 14

C.3 SCORED-POSITION PROBE ON 120 ROWS (SECTION 4)

  • On Qwen3-4B’s 64 fixed-note items, median option mass rises from 2.71 × 10^-15 for the quoted item to 3.74 × 10^-3 with fencing and 0.99940 with the stem.The corresponding option-argmax counts are 0, 21, and 62 of 64; on the 120-row probe, A–D is the unrestricted argmax on 118 rows and median option mass is 1.42 × 10^-15.Source: §C.4 · p. 15, §C.3 · p. 15

C.5 TWO MISLEADING READINGS FROM ISOLATED ENCODING

  • In-context scoring differs sharply from misleading isolated readings: Phi-3’s stemmed prompt gives 0.9331 with 16 of 16 argmax hits, while Mistral’s original ending gives 3.93 × 10^-6 with zero hits.Across 11 checkpoints and seven vocabularies, (A)/(B) shares a first token on all 11, while yes/no stays distinct on all 11.Source: §C.5 · p. 15, Table 6 · p. 15, §C.6 · p. 15

C.7 THE READER TASK WITH SPACED AND UNSPACED OPTIONS

The reader-task comparison contrasts spaced and unspaced answer strings on the same 240 rows. Unspaced strings greatly increase option mass while usually preserving the selected answer, and accuracy rises modestly.

C.7 THE READER TASK WITH SPACED AND UNSPACED OPTIONS

  • Unspaced strings raise median option mass to 1.000 and overall accuracy from 0.6958 to 0.7208, while 96.25% of picks remain unchanged.With spaced strings, median mass is 5.21 × 10^-14 when the note contains the answer and 9.67 × 10^-12 when it does not; the two group accuracies are 55.8 points apart.Source: §C.7 · p. 16, Table 7 · p. 16

D THE INSTRUCTION EXPERIMENT

The experiment varies instructions inside the quoted item and tests whether models follow those instructions instead of the forecast task. Deletion tests, token-level checks, and human and judge labels distinguish answer starts from instruction execution.

Deleting the instruction and the choice list

  • Deleting the quoted multiple-choice instruction raises median option mass above the floor on Qwen3 and SmolLM2, but template defects and shared tokens constrain interpretation elsewhere.Qwen3 median mass rises by factors of 3.7 × 10^14, 1.2 × 10^13 and 7.3 × 10^7; SmolLM2 rises from 0.3895 to 0.58023.Source: D THE INSTRUCTION EXPERIMENT · p. 16

Token-level rules

  • The opening-word rule and token-class rule measure different properties: instruction execution can begin with a response word rather than the instruction’s opening word.On Qwen3-4B, continuations begin with “Keywords” on 17 of 64 list-instruction items and “Note” on 30 of 64 translation items.Source: D THE INSTRUCTION EXPERIMENT · pp. 16–17

The judges and the hand labels

  • Judges label all 126 cross-paired rows non-executing and all 13 heuristic-selected real rows executing, while bare-number continuations count as forecasts.The three judges label the same 1862 records.Source: D THE INSTRUCTION EXPERIMENT · p. 17

Execution rates, item answers and forecasts

  • With the quoted instruction intact, Grok labels 31, 32 and 31 of 32 Qwen3 continuations as item answers; deleting it yields forecast labels on 32, 32 and 32.Counts are for Qwen3-4B, Qwen3-8B and Qwen3-14B, respectively.Source: D THE INSTRUCTION EXPERIMENT · p. 17

Agreement

  • Grok matches the first unanimous judge majority on all 99 rows and the human majority on all 45 digit-opening rows rejected by the first-token heuristic.Both validation sets give identical execute-or-not labels at 64 and 128 generated tokens.Source: D THE INSTRUCTION EXPERIMENT · p. 18

E WHAT OCCUPIES A PUBLISHED SELF-EVALUATION READOUT

The published P(True) setup can score a different answer set from the one the model favors. Tests on MMLU and TriviaQA compare its default readout with label-based alternatives, an answer-stem readout, and other library estimators.

A maintained implementation

  • The library’s default P(True) configuration reads True while Qwen3 often favors an answer letter on MMLU; its 23 enabled configurations make this a shipped setting.The reported revision enables P(True) in 23 of 30 evaluation configurations.Source: E WHAT OCCUPIES A PUBLISHED… · p. 18

Correctness ranking on MMLU

  • 0.632 [0.595, 0.669] AUROC: the answer-stem readout raises Qwen3-averaged MMLU correctness ranking, while thinking-enabled checkpoints return only a thinking token and newline.Deleted instruction raises the library readout’s Qwen3-averaged AUROC by 0.085 [0.042, 0.127].Source: E WHAT OCCUPIES A PUBLISHED… · p. 18, Table 9 · p. 19

Mass and split at the library’s scored position

  • At MMLU’s scored position, log P(True) combines declared-set mass and within-set rank; the mass term can reverse the ranking carried by the conditional ratio.On Qwen3-4B, the mass term’s AUROC is 0.353 [0.302, 0.405], while the ratio alone ranks correctness at 0.674.Source: E WHAT OCCUPIES A PUBLISHED… · p. 18

The floor and the ranking

  • At the MMLU scored position, answer letters are the unrestricted argmax on 387, 496 and 500 of 500 Qwen3 items, but the library’s True/False set is below the mass floor.Moved scores the prompt’s (A)/(B) labels after an answer stem, with median mass above 0.999 on the three Qwen3 runs.Source: E WHAT OCCUPIES A PUBLISHED… · pp. 18–19

Leaving out one checkpoint

  • Dropping any one Qwen3 checkpoint leaves Moved’s gain over the library readout above zero on both MMLU and TriviaQA.The reported gain ranges from +0.1007 [+0.0434, +0.1578] to +0.1648 [+0.1024, +0.2266] on MMLU and +0.2424 [+0.2065, +0.2771] to +0.5846 [+0.5351, +0.6332] on TriviaQA.Source: E WHAT OCCUPIES A PUBLISHED… · p. 20

Single-answer estimator comparison on TriviaQA

  • On TriviaQA, Renormalised reaches 0.857 [0.833, 0.880] Qwen3-averaged AUROC, while Moved exceeds the best of fourteen default library estimators by +0.053 [+0.024, +0.082].Moved has the highest AUROC point estimate among the seventeen readouts compared.Source: E WHAT OCCUPIES A PUBLISHED… · pp. 20–21, Table 10 · p. 20

Four further checkpoints

  • Across all eight tested checkpoints, True and False have less than half the mass at the library position on every TriviaQA item and nearly every MMLU item.After the answer stem, the library’s (A)/(B) labels hold at least half the mass on every TriviaQA item on all eight.Source: E WHAT OCCUPIES A PUBLISHED… · p. 20

A 72B checkpoint

  • Qwen2.5-72B-Instruct also assigns True and False little mass, while the stemmed (A)/(B) labels have at least 0.847 mass on every item.The True/False median masses are 9.85 × 10^-7 on MMLU and 4.27 × 10^-6 on TriviaQA.Source: E WHAT OCCUPIES A PUBLISHED… · p. 20

F PUBLISHED SPECIFICATIONS, CELL BY CELL

The paper tests whether published prompt specifications provide usable declared-set first-token readouts. Results vary by prompt rendering, checkpoint, and whether the options have distinct first tokens.

The four specifications, and how far the boundary carries

  • Among the four specifications, AlpacaEval 2.0 is the only chat-rendered prompt and the only one that can discriminate; its boundary condition is necessary but not sufficient.SmolLM2 ends at whitespace yet has median mass 0.42963, while the two prompts not ending at whitespace hold at most 7.01 × 10^-4.Source: F PUBLISHED SPECIFICATIONS, CELL BY CELL · pp. 21–22, Table 11 · p. 22

Published specifications

G-Eval’s chat rendering

  • G-Eval’s repository chat rendering falls below the floor on all four tested checkpoints with final whitespace and distinct option first tokens.The four are Qwen3-4B, Qwen3-8B, Qwen3-14B and SmolLM2.Source: F PUBLISHED SPECIFICATIONS, CELL BY CELL · p. 22

The repair’s mechanism, in the same harness

  • The harness’s assistant-prefilling mechanism places the scored position after assistant content, but byte-identical rendering across implementations was not tested.The paper appends the stem after rendering the generation prompt.Source: F PUBLISHED SPECIFICATIONS, CELL BY CELL · p. 22

G SHARED FIRST TOKENS ON FOUR PUBLISHED PROMPTS

The paper evaluates first-token scoring across four G-Eval dimensions and seven checkpoints. Although the original options have distinct first tokens, the argmax is never a declared option and the space-prefixed score correlations are constant.

Constant scores and defined correlations

  • Space-prefixed first-token scores are constant on all 28 cells, leaving zero documents with a defined rank correlation for that score.The original-option correlations range from −0.0598 to +0.6306.Source: G SHARED FIRST TOKENS ON FOUR… · p. 23

Argmax at G-Eval’s own scored position

  • At G-Eval’s scored position, the argmax is a declared option on 0 of 44,800 rows across four dimensions and seven checkpoints.The original options have distinct first tokens on all 28 cells.Source: G SHARED FIRST TOKENS ON FOUR… · p. 23

The 28 differences

  • 19 of 28 differences are negative and 9 positive, spanning −0.2194 to +0.0831; 14 bootstrap intervals exclude zero.Here ∆ρ is the bare-first-token correlation minus the space-prefixed-sequence correlation.Source: G SHARED FIRST TOKENS ON FOUR… · pp. 22–23

H WHETHER THE ORIGINAL POSITION’S ORDERING CARRIES THE FORECAST

The study forecasts a note-only reader’s correctness from the note and quoted item, while scoring the forecasting model at the prompt’s original next-token position. It compares that readout with generated values and answer-stem repairs.

H WHETHER THE ORIGINAL POSITION’S ORDERING CARRIES THE FORECAST

  • With options and renormalisation unchanged, the answer stem shifts the scored position; the stemmed Qwen3-averaged and Llama Brier scores each exceed both descriptive constant baselines.The five options have the same first-token IDs at both positions on all four checkpoints.Source: H WHETHER THE ORIGINAL POSITION’S… · p. 24
  • At the original position, the Qwen3-averaged AUC interval contains 0.5, indicating that its item-level ranking is not distinguishable from chance.Source: H WHETHER THE ORIGINAL POSITION’S… · p. 24
  • The two positions select the same grid value on only 4.6%, 9.2% and 12.9% of Qwen3 rows, versus 96.67%, 96.25% and 98.33% for the reader’s spaced and unspaced options.At the original position, 10 dominates the grid; the repaired readout spreads forecasts across it.Source: H WHETHER THE ORIGINAL POSITION’S… · p. 24
  • The generated-value readout has four-checkpoint averaged AUC 0.5356 [0.5083, 0.5628], with no significant improvement over the original and worse AUC than the stemmed readout.Its averaged Brier score is 0.2448, lower than the stemmed readout’s by 0.0089 [−0.0040, +0.0222] but higher than the fitted constant’s 0.2220.Source: H WHETHER THE ORIGINAL POSITION’S… · pp. 24–25
  • Number: and Forecast: lift Qwen3-averaged AUC by +0.1218 and +0.1145 over the original readout, respectively, and are statistically indistinguishable from the answer-stem readout.Their differences from the answer-stem readout are −0.0011 and −0.0084, respectively.Source: H WHETHER THE ORIGINAL POSITION’S… · p. 25

I THE READING UNDER EIGHT PROMPT CONDITIONS

The eight-condition probe varies the forecast template, prompt placement, quoted instruction, dataset, and answer stem while measuring in-context option mass and unrestricted next-token behavior. Across conditions, the findings distinguish instruction-driven answer starts from rendering and tokenisation issues.

I THE READING UNDER EIGHT PROMPT CONDITIONS

  • The probe tests eight conditions on 64 items per checkpoint, varying the note, turn placement, dataset and item format, inserted sentences, and whole forecast templates.The forecast options remain 10, 30, 50, 70, 90; the conditions use the checkpoint’s chat template and float16 forward pass with float32 log-softmax.Source: I THE READING UNDER EIGHT PROMPT… · pp. 25–26
  • T1 and T2 remove letter pull on every checkpoint, and no item-letter argmax remains with the primary stem across the eight conditions or the additional Qwen3-32B reading.The stemmed medians reach the floor except for three below-floor cells in each template: Mistral’s omitted separator and a shared first token for Phi-3 or SmolLM2.Source: I THE READING UNDER EIGHT PROMPT… · pp. 28, 30
  • Deleting the quoted imperative reduces Qwen3-32B’s letter-argmax rate to 0.000 and raises median option mass to 0.99085; all 64 items then have a digit at the argmax.At the original position, its median mass is 3.79 × 10−3 and 35 of 64 items have an answer letter at the argmax.Source: I THE READING UNDER EIGHT PROMPT… · p. 29
  • Phi-3’s raw option mass is not a separability reading because all options share first token 29871; its stemmed first tokens are distinct in every condition.SmolLM2 under T2 also remains below the floor on all 64 items, despite an option at the argmax on 56.Source: I THE READING UNDER EIGHT PROMPT… · p. 30
  • Supplying a separator clears the mass floor for Mistral in all eight conditions, while Falcon remains below it in seven despite an option at its argmax on most items.Falcon’s S1 median is 0.69728; the condition medians and option-argmax counts are reported in the condition order.Source: I THE READING UNDER EIGHT PROMPT… · p. 30
Loading 2609.33179v1…